Notes on: Connolly, P., Taylor, B., Francis, B., Archer, L., Hodgen, J., Mazenod, A, & Tereschenko, A (2019). The misallocation of students to academic sets in maths: a study of secondary schools in England. British Educational Research Journal. 45 (4): 873 – 97. DOI: 10.1002/berj.3530 This study is large scale and it compares actual allocation in maths with the 'counterfactual position' where allocation to sets is based solely on prior attainment at the end of KS2. Overall, 31% of students had been misallocated to lower or higher sets. School setting practices were found to exacerbate these differences in relation to gender and ethnicity but not SES. Girls had 1.5 times higher a chance of being misallocated than boys; Black students 2.4 times higher than White students, Asian students 1.7 times higher than White students. The conclusion is that setting by attainment in secondary school exacerbates already est patterns of educational inequality in gender and ethnicity. There has been a lot of research on setting, streaming and class grouping [defined and compared to American tracking]. In UK primary schools, it is common to group children at ability tables, although it is hard to establish accurately how prevalent this is. It seems to be most common in maths and English — one estimate for the DFE says that 34% of schools had '"introduced/improved setting or streaming"' as a strategy to try and close the gap between kids receiving pupil premium and their peers (874). The benefits of such grouping remain 'highly contested'. Some argue that it enables stretch and challenge for the able, and support for the struggling, the better allocation of resources and the design of suitable learning activities. However, 'it is well established'' that there is little or no positive impact on student outcomes' (874) — The highest groups may make small gains, but those in lower groups 'experience a greater negative effect'' both on attainment and on measures such as self-confidence. There is evidence in primary schools too that attainment grouping may widen the gap in achievement between 'students from disadvantaged backgrounds and their peers'. These are long-standing concerns, and teacher judgements and recommendations have been studied together with attainment data, which has become more important recently [loss of other studies are cited, including some American ones 875]. It has however been difficult to disentangle the various factors between class, ethnicity and gender. Inequalities do seem to emerge early, although gaps in attainment widen as students go through schooling. The decisions made in schools do seem to contribute to patterns of inequality [citing Gillborn and Youdell]. Wright has also been influential to understand the pattern affecting Black and Asian students. There has been no large-scale quantitative study up until now. This one has data on 9301 students in 46 secondary schools in England [but see the actual sample size,below] The method set out to control for the differences in educational attainment that students entered secondary schools with, and did this by looking at their level of attainment in maths at the end of primary schooling, KS2. The schools chosen were part of 'a broader cluster randomised controlled trial study of the effectiveness of schools' (876). The data on KS2 scores were derived from the National Pupil Database and their subsequent allocation to maths sets collected from schools. The characteristics of the students were 'broadly reflective of the national population… Well balanced in terms of gender and also broadly representative of the national population in relation to ethnicity… [And]… The proportion of disadvantaged students [FSM]' (877). Other characteristics of the sample of schools were 'broadly reflective of a national sample'. However, some schools had already expressed an interest in best practice and thus were not 'fully representative'— but they might be more interested in allocating schools equitably. The counterfactual case takes students KS2 scores and then the subsequent allocation to maths sets, and then the relationship between the two variables is examined. There are imperfections, such as 'instances where students have obtained same KS2 scores but was subsequently allocated to different sets'. An estimate was made of students who 'could be considered to have been correctly allocated to a set based solely on their KS2 scores' [the top scoring students were assumed to be allocated to the top set, but some students got the same scores and there were only limited places, so there was some imprecision and 'borderline' students. Borderlines were examined further to see if they were equally likely to be allocated to upward or downward sets. Actual data is set out on 878 and 879]. After applying this process to each of the 46 schools, and allowing for different number of sets in each school, students could be assigned to categories: correctly allocated, those allocated to a set below and those allocated to a set above that which they should have been allocated if prior attainment alone determined the allocation. This assumes that this should have happened of course. The KS2 tests are reasonably assumed to be 'a comprehensive and nationally bench-marked measure for use in secondary schools in assessing pupils prior attainment' (880 – 81), but there is controversy and many secondary school teachers do not trust them, mostly because they think that 'primary schools teach pupils to the test and/or otherwise manipulate results'. As a result 'many schools additionally purchase alternative tests': a particular study has shown, however that 'KS2 test results remain among the most accurate (and more so than some paid for tests)' (881). [Bit weaselly here — accurate in what sense? And if teachers persist in believing that primary school teachers teach to the test, their own judgements might be more accurate still] There might well still be 'minor discrepancies' or particular cases schools might take into account, but we would expect that these would be small and randomly occurring. We seem to have instead 'broader systemic trends and patterns… a notable level of misallocation', and one which is not randomly spread. It is still the case that there might be different approaches to set allocation and may be other divergences between sets and KS2 attainment, and another article discusses those possibilities. In more detail, the table shows a clear correlation between social class background and levels of attainment in maths. Social class was measured by household socio-economic background and FSM eligibility. There is a pattern in relation to gender, boys achieving better than girls on average. With ethnicity, Black students attained the lowest score, Chinese and Indians the highest. Comparisons between sets is difficult because there are different numbers of sets in different schools — so set 2 might be the bottom set in one school but the middle in another, which is why they use three categories — top, bottom and middle, and the same patterns persisted. So the pattern show that there is 'a degree of consistency between overall levels of attainment at KS2 in maths… and subsequent allocations to maths sets' (833). However actual set allocations can be compared with the counterfactual case which considers only KS2 attainments. In the initial analysis, borderline students were admitted and then they were analysed specifically. For the first sample 'nearly 1/3 of students were misallocated to maths sets… Boys are slightly more likely to be misallocated upwards and downwards' [the converse for girls] (884). The pattern is 'less clear in relation to socio-economic background where the proportions misallocated upwards or downwards tend to be similar. With ethnicity a slightly higher proportion of White students would appear to be misallocated upwards and the opposite for 'many, but not all, of the Asian and Black subgroups' (885). We need to go beyond descriptive statistics, however, for example Bangladeshi and Pakistani students are also more likely to come from lower socio-economic backgrounds than White students so the factors need to be disentangled. Simple patterns can also mislead — for example eligibility for FSM means a greater likelihood of misallocation both upward and downward. There are also problems associated with smaller subgroups, and in some cases these had to be collapsed into broader categories. For this reason they turn to 'an ordered logistic multilevel regression model', with an ordinal dependent variable [the three categories], a number of independent variables [gender ethnicity and SES], and clusters of students within schools. They fitted this model to the data, and arrived at odds ratios suggesting that 'overall, male students do tend to fare better than female students whilst, conversely, Asian and Black students tend to fare less well than their White counterparts' (886) while FSM eligibility 'has no noticeable impact on set allocation'. However odds ratios are also problematic [for technical reasons, 886], leading to the need for the development of two models. Even then, 'FSM eligibility was found not to have a statistically significant influence on the odds of a student being misallocated to a higher set but there was evidence that the odds of being  misallocated to a lower set were 1.2 2 higher'. The odds of such male students being misallocated to a higher set were '1.3 times higher', while their being misallocated to a lower set were 'not.0.65 times lower' than for female students'. (886). Black students had odds of 2.43 times higher than White students of being misallocated to a lower set, Asians being misallocated downwards were 1.65 higher than White counterparts, and conversely, Black and Asian students being misallocated to a higher set were 0.48 and 0.58 times lower than White students respectively (887). However there is a note of caution because ethnic groups are a combination of different groups 'whose chances of being misallocated may vary' and further research is needed. Overall, 'approximately 27.3% in the variation in the tendency for schools to be misallocated can be attributed to variations in school level factors (what is commonly termed "intra-class correlation")' [not sure what this means --it seems to be a statistical term assesing the similarity of groups to each other but I still cannot see what they are getting at]. However it suggests that there is 'significant potential for schools to have an impact on reducing levels of misallocation'. Once we control for gender, SES and ethnicity of the pupils, the proportion of the variance associated with school still only drops marginally to 27%. This suggests that schools do not vary in the way in which they associate gender and ethnicity with misallocation, especially in terms of the different proportions of Black and Asian students there might be in particular schools [this is the intra-class variation?] , 'although further research would be needed'. There is however 'a high proportion of missing data', they only got a sample of 4609 from a total of 9301. Second, they are not happy with FSM as a measure for SES. They conducted further research with another model. One included 'a further dummy variable for ethnicity which included all other students including mixed or those where ethnic data was missing to get a higher sample size'. Another used SES background instead of FSM again including 'dummy variables' [estimates] for those in intermediate categories. Even with these corrections the estimated odds remain fairly consistent except for the one relating to FSM which became nonsignificant: the evidence now suggests that those eligible for FSM were more likely to be allocated to lower sets but not less likely to be allocated to higher sets. Turning to borderline students, there is only a small proportion of those, about 6% and this was produced by the authors arising from their attempts to produce a kind of pure model [that is they were not a teacher category]. The issue is whether they were likely to be allocated to higher or lower sets and whether there was a pattern according to gender, social class and ethnicity. It seems that 'similar proportions of borderline students were allocated upwards and downwards' (890), usually to positions adjacent from those that would be determined by KS2. It is generally difficult to compare the students with others in the sample. They are generally only a small sample anyway and so they were redistributed for analytic purposes. Before that though, there was some variation and it was not clear that there were patterns. They tried two multilevel binary logistic regression models, where the binary was allocated either to the lower or higher set and there was no statistically significant difference. So this study confirms earlier studies where teachers have said that they take into account both ability and 'wider behaviour and attitudes as well as… Actual prior attainment' (892). The study shows the extent of misallocation, and the extent to which differences in schools have an impact — 27% of variations in misallocation 'was found to be associated with school level factors'. There is some evidence that the setting practices 'exacerbate existing educational inequalities' in relation to 'socio-economic background and ethnicity, clear patterns were evident' confirming a long-standing trend. [That long-standing stuff includes a finding that 'Chinese and Indian heritage students outperform the White majority, while those from Black and minority ethnic groups underperform' (893). There is also some data about allocation to sets in English from the long-standing stuff]. However, this study showed that there was no exacerbation of differences with regard to socio-economic background, although it found that those from lower SES were both more likely to be misallocated downwards and upwards, with a higher proportion of downwards, which fits 'existing research findings that stereotyping and labelling lead working class students to be misallocated downwards' [with reference to a very old study by Jackson 1964], although upward misallocation is 'previously undocumented and intriguing. It is possible that there could be some application of a "deserving scholarship" impetus operating here — further research would be required'. There is no suggestion here that SES has no bearing on set allocation — 'inequalities with regard to socio-economic background and education attainment are evident at the end of KS2 in the test scores reported', which suggests that SES background has already been internalised and is then carried forward. However, there is no evidence that secondary schools exacerbate these patterns of inequality. There is also a need to remember that setting can still have a subsequent impact on inequalities, and 'exacerbate existing attainment gaps over time', perhaps via 'inequality of resources or pedagogy' which will have a disproportionate impact on this group of students. With gender and ethnicity, the findings suggest that schools 'do have a role to play in exacerbating existing inequalities, specifically through this setting practices'. Again there are existing differences at KS2 with boys attaining slightly higher scores, so we would expect some gender differences in set allocations in secondary schools, but even after taking this into account, 'boys are still more likely to be misallocated to higher sets in maths compared to girls' and the converse. This is one example where boys are not disadvantaged by school practices. Black and Asian students are 'more likely to be misallocated to lower sets in maths than White students' (894), with the greatest differences for Black students. This confirms 'mainly qualitative evidence' but this is a more reliable estimate. Again there are problems. There is missing data about ethnicity 'although this does not appear to significantly affect the results' [how do they know?]. There may be 'complexity across different ethnic subgroups', especially concerning Black African students: 'a relatively high proportion of these students are misallocated downwards… Yet in general Black African students achieve much better GCSE outcomes at age 16 than White British students' however the number of Black African students is small so we need to treat the data with caution. The third problem relates to school size which is not taken into account in the model: large schools have more sets and thus more misallocated students, and since larger schools are in urban areas 'higher proportions of ethnic minority students, this may be a confounding factor'. The study has also identified the problem with borderline students. Actual setting practice is obviously constrained by class size decisions and timetable, so there will always be a group of borderlines who cannot be fitted into the proper sets, and this requires 'an arbitrary decision… Automatically precipitating injustice (given that the decision is arbitrary) and opening up the possibility of bias' [so lots of assumptions here, given that they have not observed such decisions]. In the present study they did not find evidence of 'any systematic bias in allocation practices with respect to gender, socio-economic background, or ethnicity for students identified as "borderline"' (895) and further research is needed.[We would expect to find such bias if it were widespread and extended to those non-borderline, those more securely allocated?] Their findings do require 'urgent reflection and action on the part of schools' and 'lend credence' to the view that setting should be applied 'purely on the basis of prior attainment' to avoid 'misallocation and the creeping prejudice [strong!] that our findings suggest'. Overall 'the decision to adopt setting should never be taken lightly' and there should be a review, especially of borderline students and how decisions are taken.