Why the PRM will not work

This guest blog is by Philip Gillingham. Dr. Gillingham is a Senior Research Fellow at the University of Queensland. He is a qualified social worker who has spent 27 years working in and conducting research about child protection services. Recent publications can be viewed at http://researchers.uq.edu.au/researcher/2576.

Serious ethical concerns have been raised about the development of the Predictive Risk Model (PRM) to identify children at the highest risk of maltreatment as they enter the public welfare benefit system. However, there are also serious practical problems with how it was developed which mean that it is seriously flawed. What follows is a brief and jargon-free explainer as to why it will not work, based on an analysis of the documents released about its development.


In the report prepared by the team at the University of Auckland (CARE, 2012), it is explained that data about substantiated cases of maltreatment was used to train the algorithm. This data was taken to mean that the children identified in this group had been maltreated. The development of a predictive risk model is reliant on using a source of data that is an accurate representation of what it is that is to be predicted. In health care settings, for example, relatively objective data about the presence or otherwise of particular diseases or conditions is used. However, substantiated cases in child protection in Aotearoa/New Zealand includes children who have been maltreated but also many more who have not, such as those deemed to be at risk of maltreatment, the siblings of children who have been maltreated and those deemed at risk and children assessed as having behavioural problems. The PRM will therefore identify many more children as being at high risk of maltreatment than actually are and will be highly inaccurate. A more detailed explanation has been published (see Gillingham, 2015).

Further details about the development of the PRM were released in May 2015 (see MSD, 2015a and b). While it is clear from these documents that the University of Auckland team went to considerable lengths to test and develop the PRM, analysis reveals that there are further serious concerns about the data used to train the algorithm and the effects this will have on its accuracy. In a report prepared by external reviewers, this concern is neatly summarized: “if the primary covariate with child welfare is poverty, in incorporating a multitude of poverty indicators (i.e. corrections data, benefit data, care and protection history, etc.), this tool serves merely to identify a large number of poor families in need rather than to justify a call to action that would involve referral to the child welfare system (TCC Group, 2013, p. 5)”. And indeed, the “primary covariate”, or strongest predictor identified by the PRM is poverty, identified as the length of time spent by the main caregiver on public welfare benefits (MSD, 2015b). In order of strength, the other main predictors are the relationship status of the main caregiver as “single parent” and the caregiver’s care and protection history as a child and the care and protection history of other children in the family (MSD, 2015b).

These predictors are important as they are the main factors the PRM uses to calculate risk scores for children. Given that the correlation between poverty and child maltreatment have been acknowledged for decades, any hope that the PRM might provide new insights into the causes of or circumstance that lead to child maltreatment is gone. Rather ironically, a tool which has been criticized for providing the means to individualize social problems draws attention to the need to tackle the structural problem of poverty if child maltreatment is to be prevented. That previous maltreatment is also a strong indicator is not a surprise either but it does suggest that developing interventions to assist both adults and children to overcome its effects must be part of child protection reforms if the aim is to prevent future maltreatment.

In short, the PRM cannot be used to identify children at the highest risk of maltreatment with any level of accuracy. Neither does delving into the detail of how it was developed and operates reveal any new insights into the causes of child maltreatment. Not surprisingly, addressing poverty and the long term effects of maltreatment in both children and adults emerge as priorities for the reform of child protection services.

If you like this, you might also like: Safety in Numbers

References

CARE (2012). Vulnerable Children: can administrative data be used to identify children at risk of adverse outcomes? Centre for Applied Research in Economics, University of Auckland.

Gillingham, P. (2015), Predictive risk modelling to prevent child maltreatment and other adverse outcomes for service users: inside the “black box” of machine learning. The British Journal of Social Work. Published online 9 April 2015. DOI: 10.1093/bjsw/bcv03.1

MSD (2014a). Final report on feasibility of using predictive risk modelling. Wellington, New Zealand: Ministry of Social Development.

MSD (2014b). The feasibility of using predictive risk modelling to identify new-born children who are high priority for preventive services – companion technical report. Ministry of Social Development, Wellington.

TCC Group (2013) Peer Review Report 1Wellington, New Zealand: Ministry of Social Development.

3 thoughts on “Why the PRM will not work

  1. Refreshing to read a jargon-free account of exactly what is wrong with the reliance on this approach as I think I am not alone in finding the language of the ‘science’ of the risk discourse acts to silence those of us who have a growing sense of disquiet about the growing attention given to it.

    I’m interested too, that what we now know about a focus on protective factors being a far better predictor of outcomes, has not featured much in these conversations.
    Early studies of resilience (Werner et al, 1971) found that even in the worst scenarios, where children experienced persistent and multiple risks, most of them still achieved good developmental outcomes. Subsequent work has been able to provide more detail about the factors themselves and the complex interactions between individuals and their environment. We now know more exactly what helps protect children living in difficult circumstances and how these things can be harnessed to allow the normative processes of resilient development to occur. Whereas ‘the identification of risk factors are predictive of outcomes only between 20 – 40% of the time, the identification of protective factors, those supports and opportunities which buffer the effect of adversities and enable development to proceed are predictive of positive outcomes in anywhere between 50 – 80% of any given high risk population’. (Benard, 2004, p. 8)

    What this suggests to me is that no matter how brilliantly predictive of risk the tool, the attention of social workers should not be diverted away from the practice of locating and actively supporting those protective factors which we know to be of much greater value than knowing about ‘risks’ when assessing the likelihood of children making it through.

    A universal welfare system which supports all families and whānau to care for their vulnerable members and service provision which is responsive to its users through the life course is a better place to start.

Leave a Reply

Your email address will not be published. Required fields are marked *