Why the PRM will not work

This guest blog is by Philip Gillingham. Dr. Gillingham is a Senior Research Fellow at the University of Queensland. He is a qualified social worker who has spent 27 years working in and conducting research about child protection services. Recent publications can be viewed at http://researchers.uq.edu.au/researcher/2576.

Serious ethical concerns have been raised about the development of the Predictive Risk Model (PRM) to identify children at the highest risk of maltreatment as they enter the public welfare benefit system. However, there are also serious practical problems with how it was developed which mean that it is seriously flawed. What follows is a brief and jargon-free explainer as to why it will not work, based on an analysis of the documents released about its development.

In the report prepared by the team at the University of Auckland (CARE, 2012), it is explained that data about substantiated cases of maltreatment was used to train the algorithm. This data was taken to mean that the children identified in this group had been maltreated. The development of a predictive risk model is reliant on using a source of data that is an accurate representation of what it is that is to be predicted. In health care settings, for example, relatively objective data about the presence or otherwise of particular diseases or conditions is used. However, substantiated cases in child protection in Aotearoa/New Zealand includes children who have been maltreated but also many more who have not, such as those deemed to be at risk of maltreatment, the siblings of children who have been maltreated and those deemed at risk and children assessed as having behavioural problems. The PRM will therefore identify many more children as being at high risk of maltreatment than actually are and will be highly inaccurate. A more detailed explanation has been published (see Gillingham, 2015).

Further details about the development of the PRM were released in May 2015 (see MSD, 2015a and b). While it is clear from these documents that the University of Auckland team went to considerable lengths to test and develop the PRM, analysis reveals that there are further serious concerns about the data used to train the algorithm and the effects this will have on its accuracy. In a report prepared by external reviewers, this concern is neatly summarized: “if the primary covariate with child welfare is poverty, in incorporating a multitude of poverty indicators (i.e. corrections data, benefit data, care and protection history, etc.), this tool serves merely to identify a large number of poor families in need rather than to justify a call to action that would involve referral to the child welfare system (TCC Group, 2013, p. 5)”. And indeed, the “primary covariate”, or strongest predictor identified by the PRM is poverty, identified as the length of time spent by the main caregiver on public welfare benefits (MSD, 2015b). In order of strength, the other main predictors are the relationship status of the main caregiver as “single parent” and the caregiver’s care and protection history as a child and the care and protection history of other children in the family (MSD, 2015b).

These predictors are important as they are the main factors the PRM uses to calculate risk scores for children. Given that the correlation between poverty and child maltreatment have been acknowledged for decades, any hope that the PRM might provide new insights into the causes of or circumstance that lead to child maltreatment is gone. Rather ironically, a tool which has been criticized for providing the means to individualize social problems draws attention to the need to tackle the structural problem of poverty if child maltreatment is to be prevented. That previous maltreatment is also a strong indicator is not a surprise either but it does suggest that developing interventions to assist both adults and children to overcome its effects must be part of child protection reforms if the aim is to prevent future maltreatment.

In short, the PRM cannot be used to identify children at the highest risk of maltreatment with any level of accuracy. Neither does delving into the detail of how it was developed and operates reveal any new insights into the causes of child maltreatment. Not surprisingly, addressing poverty and the long term effects of maltreatment in both children and adults emerge as priorities for the reform of child protection services.

If you like this, you might also like: Safety in Numbers

References

CARE (2012). Vulnerable Children: can administrative data be used to identify children at risk of adverse outcomes? Centre for Applied Research in Economics, University of Auckland.

Gillingham, P. (2015), Predictive risk modelling to prevent child maltreatment and other adverse outcomes for service users: inside the “black box” of machine learning. The British Journal of Social Work. Published online 9 April 2015. DOI: 10.1093/bjsw/bcv03.1

MSD (2014a). Final report on feasibility of using predictive risk modelling. Wellington, New Zealand: Ministry of Social Development.

MSD (2014b). The feasibility of using predictive risk modelling to identify new-born children who are high priority for preventive services – companion technical report. Ministry of Social Development, Wellington.

TCC Group (2013) Peer Review Report 1. Wellington, New Zealand: Ministry of Social Development.