what is it about child protection plans and statistics?

Ofsted warns of abuse risks to disabled children” says the BBC headline about the report entitled Protecting disabled children; thematic inspection published last week. And indeed the report opens by reminding us that research “indicates that disabled children are more at risk of being abused than non-disabled children”. It goes on:

However, they are less likely than other children in need to become the subject of child protection plans. This suggests either that risks to disabled children are not well identified or that support effectively reduces risks and helps to keep them safe.

The thematic inspection was undertaken to find out more. I’ll come back to the findings later, but in this post I want to take a closer look at the data about risk that appear to have prompted Ofsted’s interest. The research cited in support of Ofsted’s opening claim consists of one paper (Sullivan and Knutson, 2000) describing a large-scale study of children in Omaha, Nebraska during 1994-95. Disabled children were found to be 3.4 times more likely to have experienced maltreatment than non-disabled children.

By contrast, Ofsted points out;

The children in need census as at 31 March 2011 showed that there were 382,400 children in need in England of whom 54,100 (14.2%) were recorded as having a disability. At that time 42,700 (11%) children were subject to a child protection plan of whom 1,600 (3.8%) were recorded as having a disability. Children with a recorded disability were therefore less likely to be the subject of a child protection plan than other children in need.” (p.8)

If I’ve understood this correctly, it’s saying that if disabled children are at a higher risk of maltreatment than non-disabled children, you’d expect to see a higher rate of child protection plans (CPPs) amongst disabled children. But the frequency of CPPs amongst disabled children is lower (3.8% as compared to 11%), prompting concerns about children falling through the net.

There was something that didn’t look quite right about Ofsted’s analysis and I think I’ve figured out what it is. (I should point out that statistical analysis isn’t my forte, so please let me know if I’ve got it wrong.) I think there are three problems with Ofsted’s data;

– the level of risk to disabled children is based on an assumption
– the figures for children in need and disabled children are unreliable
– the level of risk and level of CPPs are derived from qualitatively different populations.

Assumptions about disabled children and the risk of maltreatment

First, let’s look at the figures for the risk of maltreatment. Little relevant research has been undertaken. Sullivan and Knutson’s study did indeed find that disabled children were 3.4 times more likely than non-disabled children to experience maltreatment. But they also cite an earlier finding that the risk was 1.7 times higher (p.1258). It’s clear from previous research that determining the prevalence of maltreatment is dependent on factors such as the quality of available data, quality of support services, definitions of disability and maltreatment, and whether disability has led to maltreatment or maltreatment has led to disability. This means that the prevalence of maltreatment is likely to vary widely between communities. The figure Ofsted took as its baseline for England was one that emerged from data gathered almost 20 years ago and 5,000 miles away in a community with a very different healthcare system. We don’t actually know the risk of maltreatment for disabled children in England.

Who qualifies as a child in need or a disabled child?

Let’s assume for the moment that disabled children are at a higher risk of maltreatment than non-disabled children. Whatever the level of increased risk, we’re still left with the question of the lower proportion of disabled children with a CPP.

Sullivan and Knutson’s sample consisted of 50,278 children enrolled in the Public (OPS) and Archdiocese schools of Omaha, Nebraska during the 1994–95 school year. It included children in other educational programmes and therefore covered an age range between 0-21. So the sample represented almost all children in a specific geographical area. But the sample Ofsted refers to consists, not of all children in a geographical area (in this case England), but of children in need in that location.

Section 17 (10) of the 1989 Children Act defines children in need as follows:

“For the purposes of this Part a child shall be taken to be in need if—
(a) he is unlikely to achieve or maintain, or to have the opportunity of achieving or maintaining, a reasonable standard of health or development without the provision for him of services by a local authority under this Part;
(b) his health or development is likely to be significantly impaired, or further impaired, without the provision for him of such services; or
(c) he is disabled”

and s. 17 (11) defines ‘disabled’ as;

For the purposes of this Part, a child is disabled if he is blind, deaf or dumb or suffers from mental disorder of any kind or is substantially and permanently handicapped by illness, injury or congenital deformity or such other disability as may be prescribed; and in this Part—
* “development” means physical, intellectual, emotional, social or behavioural development; and
* “health” means physical or mental health.”

The first problem that arises is one of classification. The Children Act appears to class all disabled children as ‘children in need’, whether or not they are considered to be otherwise at risk. But local authorities don’t necessarily do that. DirectGov site says ‘local councils must identify the extent of need in their area and make decisions about levels of service they provide’, and it’s clear from local authority websites that their criteria for ‘children in need’ and criteria for ‘disabled children’ vary somewhat. One local authority officer told me that her LA classifies all disabled children as children in need whereas Cheshire East, for example, carries out an assessment to determine whether a disabled child qualifies as a child in need.

Then there’s the issue of the reliability of local authority records. The Citizen’s Advice Bureau tells us that ‘the local authority must keep a register of children with disabilities in its area but does not have to keep a register of all children in need’. Local authorities might be required to keep a register of children with disabilities, but that doesn’t mean that all children with disabilities will be registered. A recurring complaint from my own local authority is that although they have a register, it’s perpetually out of date and incomplete. In short, if LAs are unclear about how many disabled children and how many children in need are living in their area, the census data cited by Ofsted might not be entirely reliable. I did try to find out what data were collected by the census, but failed. If you can figure it out, let me know.

Qualitatively different populations

A key problem with Ofsted’s data is that if some LAs are treating all disabled children as children in need, but are treating non-disabled children as children in need only if their health or development is at risk, this would result in non-disabled-children-in-need being at a higher risk of maltreatment than non-disabled children-in-the-general-population. Ofsted is comparing data drawn from two qualitatively different populations – the general population vs children in need.

Child protection plans

What difference does that make to Ofsted’s concern about CPPs? It means that you’d expect non-disabled-children-in-need to have a disproportionately high level of CPPs compared to non-disabled-children-overall. That would widen the gap between CPPs for non-disabled children and CPPs for disabled children, making the proportion of CPPs for disabled children look disproportionately low. Some calculations might clarify the point:

Sullivan and Knutson set out their data (for all children) in a 2 x 2 matrix as shown below:

If Ofsted’s data (for children in need) are set out in the same way, they look like this:

In Table 2, the proportion of non-disabled children with CPPs is 4 times the proportion of disabled children with CPPs, exactly the opposite of Sullivan and Knutson’s trend, and that’s what Ofsted is concerned about.

But Sullivan and Knutson’s finding was based on data from all children in a location, not just children in need. If we put Ofsted’s figures into the context of all children in a location (in this case, England), a different picture emerges. (The total figures for all children and for all disabled children are estimates – the figure for disabled children is the Aiming High estimate).

If my calculations are correct, the proportion of non-disabled children with CPPs in the whole population of non-disabled children is only 0.39%, whereas the proportion of disabled children with CPPs in the whole population of disabled children is slightly lower at 0.28%, considerably less than the 4-fold ratio if children in need only are taken into consideration. When the population differences are corrected for (as far as is possible with the data provided), the discrepancy between CPP rates for disabled and non-disabled children almost disappears.

*********

There’s something about child protection plans that seems to have an adverse effect on statistical analysis. UK readers who were home-educating their children during the Badman review in 2009 will probably recall the disagreement between Graham Badman and Graham Stuart (later to become Chair of the Education Select Committee) about the proportion of home-educated children subject to CPPs. The central issue was that the total number of home-educated children in the UK was unknown; estimates had varied between 20,000 and 150,000. You can read the full account here, but I thought it was worth reproducing some of the exchange in full:

Graham Badman: I reflected a great deal on our exchange of views, I promise you. I did go back and look at the figures and I came up with exactly the same conclusion. In fact, if you want me to qualify it, when surveying the number of child protection plans in the authorities that we covered in the last survey, we came up with a figure of 0.4 per cent., which is double that within the normal population. In fact, that figure could be fractionally higher, because we discounted any child protection plan that was there as a consequence of disability. I know you argued that because I did not know about the other half, that therefore negated it and made it even again. My argument in return, as you will recall, was that because we did not know about the other half, that did not mean to say that they were all safe either. It could be exactly the same figure or more.

Q 86Mr. Stuart: May I interrupt you there? The question was not about the letter to the Select Committee in which you suggested that I had said that any child who was not known about was automatically safe. I said no such thing, so you have your facts wrong there—again. What we were establishing was the rate of child protection plans in the home-educating community. We went through it slowly, and you want to rehearse it, so I will rehearse it.

If you have a child protection plan, automatically you are known to the local authority, so children previously not known to the local authority who become subject to a child protection plan are then known to the local authority. It does not by any means guarantee that children who are not known to the local authority and do not have a child protection plan are necessarily safe. But what you do know is that every child with a child protection plan is known to the local authority, and you claimed, completely erroneously—and although you have been corrected, here you are months later still getting it wrong, which is pretty frightening—that the percentage of children who are home educated who have a child protection plan is double the national average, when your own report suggests that the numbers are at least double the 20,000, which is the number registered with the local authority, and the Secretary of State said on Second Reading that the Government estimated the number was 70,000. They do not know for sure, but they think it is 70,000.

If you take the number of home-educated children with a child protection plan and you see that as a percentage of 70,000, it comes out at a great deal less than the average for the population, and that I believe is the definitive statement of what we know. It does not tell us everything we need to know, which is why I would not overstate my reliance on it. Could you please comment on whether you believe that we can truly say, as you have repeatedly said, that there is double the rate of the most serious level of child protection plan among children who are home educated?

Graham Badman: I fear we are in danger of going round in the same circle. I am afraid I fundamentally disagree with you. You think I am wrong; I think you are wrong.

Mr. Stuart: It is maths.

Graham Badman: Well, fine; we might want to debate that later—perhaps you went to a better school than me. The fact remains that I do not agree with your assumptions. On the basis of the data collected from the local authorities and from two surveys that have been validated by DCSF statisticians, I believe that we are quite safe in saying that on the basis of the child protection plan analysis that they carried out, there are twice as many young people on a child protection plan known to local authorities within that population as are within the general population. That is a fact. I am sorry that you do not agree with me, but we could go on for ever disagreeing.

I also have concerns about other aspects of the report, which I’ll discuss in the next post.

Reference

Sullivan, P.M & Knutson, J.F (2000). Maltreatment and disabilities: A population based epidemiological study. Child Abuse & Neglect, 24, 1257–1273,

If it wasn’t for those pesky feminists…

This report argues that domestic abuse is a shocking and disturbingly prevalent hallmark of social breakdown – yet it exists inside every community.”

This sentence opens the executive summary of the Centre for Social Justice’s report on domestic abuse Beyond Violence, published, coincidentally, the day after my last post criticizing two previous CSJ publications for assuming that a correlation between two phenomena indicated a causal link between them. I was curious about the new report’s take on correlations and causality.

Correlations and causality

Beyond Violence, co-authored by a clinical psychologist and a policy expert, provides an overview of the legal, sociological and psychological aspects of domestic abuse from a public policy perspective. It recognizes the complex nature of domestic relationships and abuse and that perpetrators, as well as victims, need understanding and help. However, there’s an implicit assumption running through the paper, as in the CSJ’s papers about marriage and child development, that if A and B are positively correlated and A happens before B, then A must cause B.

That isn’t necessarily the case, of course. A and B might each have a different, independent cause, or both be caused by C – or the causes might be more complex. Take mental health for example. There’s no doubt that domestic abuse can cause mental health problems – for both victims and perpetrators. But mental health problems – in victims or perpetrators – can also be causal factors in abuse. Similarly, marriages might tend to result in stable relationships, or it might be that couples in stable relationships tend to get married. The social problems associated with poor attachment in infancy might not be caused by poor attachment; poor attachment and the associated social problems might have independent causes. What is certain is that interventions that make incorrect assumptions about causes run the risk of being ineffective.

Some other aspects of this report caught my eye and gave me cause for concern.

Domestic abuse and the law

The report deals at some length with the legislation relevant to domestic abuse and points out that

“…fundamentally the law and legal system were not designed with domestic abuse in mind and they still both misapply understandings of other sorts of crime to it.” (p.15).

In response, it recommends that a new crime of coercive control is recognized

“…whereby a prosecution can be brought on the basis of a ‘course of conduct’ in which a person has acted strategically to control, isolate, intimidate and/or degrade their victim.” (p.19)

and the report bemoans

the high standard of proof required before there is an acknowledgement of wrongdoing.” (p.15)

It’s quite likely that the law relating to domestic abuse might need to be changed; however, I have reservations about the direction in which this line of argument is moving. UK law puts the burden of proof on the plaintiff. This means that there’s a greater risk of criminals getting away with it than of innocent people being wrongly convicted. There are good reasons for that. It’s more difficult to prove that someone didn’t commit a crime than that they did, wrongful conviction causes long-term damage and reduces respect for the law, and if the burden of proof were on the defendant the courts would be full of vexatious litigants. UK law also requires evidence to show beyond reasonable doubt that someone committed a crime. A ‘high standard of proof’ is required for good reason. I couldn’t find any evidence that the report had tackled the thorny question of what sort of evidence would be required to prove that a course of conduct involving coercive control had taken place.

Involvement of public sector services

Understandably, the CSJ wants to prevent domestic abuse before it starts. In order to do so, it recommends a ‘skill-based module’ in schools ‘backed up by a supportive school culture and learning across other subjects’ (p.17).

It also recommends the involvement of health visitors, Sure Start Centres and GP surgeries and makes recommendations for the criteria used to commission interventions in domestic abuse cases. It’s clear that the authors envisage reallocating existing funding but since education, health and social care services are currently struggling to cope with a) demand b) reduced budgets and c) structural changes, these recommendations are likely to go on the ‘to do later’ list.

Payment by results

The report recommends several interventions funded by the creation of payment-by-results commissioning frameworks, including using social impact bonds for ‘domestic abuse services’(p.17). The report doesn’t explain what a social impact bond is, nor could I find any indication of what results would trigger payments. According to Wikipedia a social impact bond depends on

“…specified social outcomes being achieved and therefore in terms of investment risk Social Impact Bonds are more similar to that of a structured product or an equity investment.”

Social impact bonds have also been mooted in the USA and Australia, although there doesn’t appear to be any evidence yet for their efficacy. The idea is that the savings resulting from interventions pay for the interventions themselves. Of course until the interventions result in savings, there’s no money to pay to the agencies doing the intervening. That’s where the ‘investment’ of the social bond comes in. If my experience is anything to go by, this model would result in agencies focusing on meeting required outcomes, rather than on outcomes of maximum benefit to the client. It also assumes that the savings to the exchequer would be greater than the money needed to fund the agencies. But savings wouldn’t go back to the taxpayer – at least not during the life of the social impact bond – they would go to private investors. So much for limiting public sector borrowing.

Feminism

The authors are critical of what they describe as a ‘power, control and patriarchy’ model of domestic abuse (pp.15, 52ff) originating in feminist thinking (feminists get 24 mentions in this report). They are careful to acknowledge the part played by feminists in putting domestic abuse firmly on the political agenda. They also acknowledge that the model is oversimplified in government policy and in practice. But when they say;

However, as movements move from the margins to the mainstream, they need to confront and adapt to the nuances and complexities of the problems they are aiming to address. This has not happened in the domestic abuse field; the ‘patriarchy, power and control’ analysis remains more or less intact despite its incompatibility with emerging findings about domestic abuse” (p.54).

The implication is that the feminist model of domestic abuse is itself simplistic, lacks nuance and doesn’t recognize the complexities of the problem, despite Janice Haaken – ‘psychologist, feminist, activist and film-maker’ – being quoted as saying of domestic abuse;

‘There is something quite vital – and respectful – in acknowledging this complexity, and the challenges we face in bringing about a more humane world’” (p.23).

I wouldn’t describe myself as a feminist – I’ve always questioned the assumptions implicit in feminist thinking – and I’m quite sure many feminists do subscribe to a simplistic model of domestic abuse framed only in terms of power, control and patriarchy. But most of the feminist writers I’ve read (mainly academics) have been clear about their framework, explicit about their assumptions, and have understood the complexity of the phenomena they are dealing with. Their emphasis has been on power, control and patriarchy because of the extent to which such issues are taken for granted.

My experience with ‘marginal movements’ suggests that it’s when ideas rather than movements move to the mainstream that mainstream agencies such as governments and policy-makers adopt simplistic versions of the ideas that lack nuance and don’t recognize the complexities involved.

Underlying assumptions

What troubled me most about this report was something that wasn’t made explicit – its underlying assumptions. Let’s go back to the sentence that opens the executive summary. “This report argues that domestic abuse is a shocking and disturbingly prevalent hallmark of social breakdown – yet it exists inside every community.”

An accusation often leveled at researchers is that their writing is too dense and full of jargon, rendering it inaccessible to the average reader. There’s no reason why researchers’ writing should be impenetrable, but there are reasons for it being like it is. One is the space constraints of academic journals – authors often have to work within strict word limits and therefore use technical terminology for brevity. Another reason is that scientific research is about the objective evaluation of evidence. No one can be truly objective, of course, which is why, over the centuries, scientists have developed a toolbox of research methods that go a long way toward mitigating the effects of the errors and biases inherent in human thinking. A good academic paper will be grounded in evidence – authors will avoid personal opinion and emotive language because these could bias the conclusions of readers – or of the authors themselves.

Beyond Violence isn’t an academic journal paper, so it doesn’t have to conform to the same level of rigour. However, opening a report with personal opinion and emotive language makes it clear what the authors want readers to think. Some people, unfamiliar with domestic abuse or the full range of human behaviour might find the statistics on the prevalence of domestic abuse ‘shocking and disturbing’. Others, more familiar with both, might mutter about no one knowing about what goes on behind closed doors. The report refers to domestic abuse as a ‘hallmark of social breakdown’, but I couldn’t find any evidence to support this assertion. There have been times of social breakdown in English history; the Viking invasions, the wars of the roses, the civil war, the aftermath of the industrial revolution and the depression spring to mind. It’s quite likely that domestic abuse escalated during those periods, but they were characterized by far more obvious ‘hallmarks’ than domestic abuse.

Also, the authors of the report emphasise the fact that domestic abuse has only achieved widespread recognition as a social problem in the last few decades. The implication, coupled with historical evidence, is that the prevalence of domestic abuse is probably lower now than at any other time during our history. So what ‘social breakdown’ does it indicate? In the report’s Foreword, the CSJ’s Managing Director, Christian Guy, links ‘disadvantage’ to ‘family breakdown’.

In all of our work with community-based organisations fighting poverty on the frontline, we are reminded of the close association between family breakdown and other drivers of disadvantage – particularly drug and alcohol addiction, welfare dependency, educational failure and serious personal debt.” (p.4)

The causal chain underpinning the CSJ’s view of social problems appears to look something like this;

individual behaviour
leads to
family breakdown
leads to
social problems/social breakdown

The CSJ appears to imagine that if all couples got married, engaged appropriately with their children and acted responsibly (and if the pesky feminists stopped propagating simplistic ideas) social problems would melt away. Certainly the behaviour of a small number of individuals can cause a disproportionate number of social problems, but the fact that individual behaviour results in those problems is a different issue from what causes the individual behaviour in the first place. Individuals are often severely constrained as to what they can do about their genes, physiological make-up, health problems, disabilities, learning difficulties, education, income, housing and family responsibilities. They can’t change national or international economic and social conditions, nor the quality of education, health or social care that’s available to them. That doesn’t absolve individuals of moral or legal responsibility for their own actions. What it does mean that locating the root cause of individual behaviour in individuals only is short-sighted and that interventions aimed at changing individual behaviour without addressing the complex causes of that behaviour are likely to fail.

Decades of research have shown that the causes of individual behaviour are complex. In order to change the behaviours that cause social problems, it’s necessary first to tease out the causes and then to address them all. Simultaneously. A rigorous review of several research literatures, some very careful planning and a great deal of joined-up working will be required. The task isn’t an impossible one, but it’s challenging. It might be a while before social impact bond investors get their money back. Taxpayers might need to wait a bit longer.