University Entrance: Part II - Standards Based Testing

This post is part of a multi-part series on University Entrance and whether it is set at the right standard. For the previous part, click here - Introduction.

What is University Entrance?

We start by investigating the context in which the UE debate is held. The New Zealand Qualifications Authority (NZQA) states that:

University Entrance is the minimum requirement to go to a New Zealand university. To qualify, you will need:

-          NCEA Level 3 [60 credits at Level 3 or above + 20 credits at Level 2 or above]

-          14 credits in each of three approved subjects at Level 3

-          Literacy requirement: 10 credits at Level 2 or above, with at least 5 credits in writing and 5 credits in reading

-          Numeracy requirement: 10 credits at Level 1 or above, made up of either achievement standards or all three of the unit standards 26623, 26626, 26627

It is important to note that this is the minimum requirement to be admitted to a university; each programme can then have additional or higher requirements. Often, the credits are converted into another score, such as a rank score (Auckland) or guaranteed entry score (Victoria), for the purposes of comparing NCEA to other high school qualifications like Cambridge (CIE) and International Baccalaureate (IB). It’s probably worth mentioning here that these blog posts will only deal with entrance for domestic students.

What existed before University Entrance (as we know it now)?

Before 2003/2004, the system had two parts. Year 12 and 13 students worked towards the School Certificate, which led towards the New Zealand University Bursary, or Bursary for short. Most students (Forms 6 and 7 at the time) completed internally assessed coursework and sat externally assessed exams in the same way as they do now for NCEA.

Marks for up to five subjects are added together; with each subject scored out of 100, an aggregate score of 300 was an A Bursary, and a score between 250 and 299 was a B Bursary. This influenced how much cash was provided by the government for tertiary study. On a side note, scholarship grades were separate and awarded to the top 3-4% of each subject. Paradoxically, scholarship grades did not have any monetary award attached to them, although Top Scholars (top in each subject or scholarship in five or six subjects) did receive additional money.

Critically, the exams were norm-referenced; students received grades dependent on their performance in comparison to other students, with a letter awarded based on a pre-determined distribution.

What does norm-referenced mean?

Students are compared to other students – you don’t have to get a high number of questions correct, you just have to do better than everyone else (or a large majority of them). It was very popular around the world because the outputs are predictable, mainly because the proportion of students who pass, and the spread of those students, are pre-determined. It allows easy comparison between two students, as you can easily say which one is “better”. It’s useful if you’re a government and you have to plan for the right number of students to enter universities because there are only so many spaces and only so much funding available.

Scores are often given in percentiles instead of percentages – the higher the percentile, the more highly you scored in comparison to other students. Alternatively, grade boundaries are used to divide students into broader groups, acknowledging that a student who scores 85% in a test is probably roughly at the same level as a student who scores 86%. The grade boundaries move year-to-year to ensure that the number of students in each group is roughly proportional to the desired distribution. This is the approach used in the Cambridge and IB examinations. For Bursary, it's a combination of both - marks were scaled to fit a distribution and then letter grades assigned based on fixed grade boundaries. The grade boundaries looked something like this:

For School Certificate, it was widely known that roughly half (or maybe 46%) of the students would fail. The results were scaled to fit a standard distribution, because it was decided that a standard distribution was the desired distribution. These days it seems weird that we once forced half of all students to fail. Over a number of decades, there were increasing calls for educational reform. Ultimately, we scrapped School Certificate and Bursary, and replaced it with NCEA, a standards-based system.

So what does standards-based mean?

The big problem with norm-referenced testing is that it doesn’t ensure that any particular student has proficiency in a particular skill or knowledge about a particular subject; only that they are better/worse at the skill or know more/less about the subject than another student. Standards-based testing sets targets (also known as criteria, as in criterion-based marking), and assess whether or not the student can achieve those targets. The idea is that every student who passes a standard is known to be capable of achieving that target, regardless of whether they are in the top 50% or not. In fact, you shouldn’t care about what percentile they are in comparison to other students; just whether they achieved the standard or not. 

NCEA breaks the targets up into different subjects, different areas within those subjects (achievement standards), and divides those subjects/areas into levels (the numerical levels 1, 2, and 3 as well as the Achieved, Merit, Excellence sublevels within each numerical level). Since we aren’t focused on making students better than other students, we can focus on getting students across each line and ensuring that every student can achieve as highly as they can. This seems to make a lot more sense to most people, and educational reform around the world is moving in this direction.

What’s the problem with standards then?

The challenge is where to set the targets. If you set them too low, then they’re too easy, so students and teachers are not incentivised to put in much effort. If you set them too high, then they’re too hard, so students and teachers are demoralised (and see no point in trying). On top of the psychological effects, setting the targets at appropriate levels has implications on sectors of society that rely on the standards to judge ability and competency, such as employers and universities. Holding a particular qualification, such as NCEA Level 2 with Merit, has to communicate to other people that this student will be capable of completing some task. If the standard isn’t quite in the right place, then the value of the qualification becomes more questionable. Additionally, there are issues surrounding “teaching to the test/standard” and students working only just hard enough to get across the line, rather than continuously working to be the best that they could be as you would in a norms-referenced system like School Certificate. But let’s get back to getting the targets right.

How are the standards set?

If we purely followed the philosophy of standards-based testing, then you would set the standards at some level, and leave them there. If you need to, you make more standards higher or lower, and you make the standards public so people can look up what each standard means. However, there is a trade-off between the granularity/number of standards and the communicability of those standards. In a hypothetical world, you can have a hundred different targets under NCEA Level 1 math, and that would very accurately describe what the student can or can’t do. But when you tried to communicate that ability to someone else, it would be a very painful process; it makes life much easier to be able to say that the student has the ability to do geometry at an NCEA Level 1 Merit level. So in order to make things easier to understand, the system has to sacrifice some granularity and create broader bands.

When the standards exist in bands, then it’s much harder to get them right. Over time, you have to move them up and down slightly to ensure that you have the boundaries at the right levels. Accepting that you have to move them up and down means that the standards can be more responsive to the needs of society and the abilities of students; for example if students are getting smarter over time (as we hope they are), then the standards can drift upwards. So the NZQA monitors the results, consults with stakeholders, and decides whether standards need to move. In fact, the NZQA updates lists of standards that are being revised or reviewed on a monthly basis.

So what makes a standard “right”?

And here we arrive at what I think is probably the biggest issue. When the NCEA standards were first set, they were drafted, consulted upon, redrafted, and consulted upon again many times until teachers and government were mostly okay with things. It was a process that took many years, but they wanted to get things right. NCEA Level 1 was introduced in 2002, Level 2 in 2003, and Level 3 in 2004. As students sat the exams and went through the system, NZQA adjusted the standards as they deemed necessary. This is to be expected for any new system – some calibration is needed when you move from the theory to the real world. When the States Services Commission (SSC) investigated the performance of the NZQA in delivering secondary school qualifications in 2005, we found patterns such as this:

The targets were moved and they became harder. This didn’t happen to every achievement standard, but it occurred enough that one of the recommendations from the SSC report was that expectations with tolerances should be introduced to ensure that standards are appropriately set. Assessment should be consistent year to year, and assuming that the performance of students does not change significant year to year, then roughly the same proportions of students should be scoring achieved, merit, and excellence each year. If assessment is unnecessarily harsh one year, then it would fall outside of the expected range and the alarm bells would ring.

To be clear, officially, the NZQA says that if 100% of students get enough credits to meet the requirements for NCEA certificates, then they pass. Education Minister Hekia Parata has set an explicit target that by 2017, at least 85% of 18 year olds will have passed NCEA Level 2 (or equivalent). Pass rates for NCEA are generally increasing. But when we look at the individual standards, it becomes clear that something else is going on. For example, ENGLISH 90951 (NCEA Level 1 Unfamiliar Texts after ENGLISH 90057 was expired in 2011), has this “Profile of Expected Performance” published for 2014:

The NZQA expects that something between 20% and 25% of students who sit ENGLISH 90951 will fail. Exam papers are usually remarked in order to force the grade distribution into these expected “profiles”, although the NZQA denies that this is scaling. In fact, the NZQA says “there is no predetermined distribution of grades”. The NZQA deems claims that profiles of expected performance (PEP) are scaling or norm-referenced to be myths. Perhaps regardless of the intention, the outcome is the same; a certain percentage of students are expected to fail.

So once PEPs were set, that dictated how to set standards. The “right” standard is the one that achieves the expected grade distribution. There are some tolerances (as there should be), but overall the standard is expected to be set at a level that causes some students to fail. There’s the problem – you can’t expect a 100% pass rate if you set a standard that forces some students to fail. It all seems reminiscent of… norm-referencing (even if NZQA calls that suggestion a myth). It’s important to note that you can still pass NCEA overall even if you fail individual achievement standards; that’s why NCEA overall pass rates are increasing.

What does this have to do with University Entrance?

Universities require students to be at a particular level of competency before they enter university; otherwise they would struggle to be able to provide degrees that are of sufficient quality to be accredited. So Universities NZ, the body that represents all universities in New Zealand, along with NZQA and other stakeholders, set a minimum bar. It makes sense to use NCEA to set the height of the bar, and thus University Entrance is standards-based. It causes UE perceptions to be subject to the psychological implications of standards. It causes UE to be susceptible to the fluctuations of changing standards levels. It causes UE to be affected by the use of Profiles of Expected Performance and their suggested fail rates. Ideally, universities would be able to say “an incoming student must be able to…” but the way that standards are used in New Zealand makes that tricky.

So it makes answering the question of “Is University Entrance in New Zealand set at the right standard?” just a little bit harder, because the very system used to measure University Entrance has some flaws. So far we’ve provided some context about what we used to do, why that was bad, what standards are, why we have standards-based testing, and what some of the shortcomings are. In the next section, we’ll have a look at the more recent history of University Entrance, why it was changed recently, and the perspective of government.

If you’re further interested in the problems of NCEA and steps taken to improve it, I suggest this 2007 policy paper from the Maxim Institute as a starting point. There are probably additional issues surrounding internal vs. external assessment, vague standards, and marking fairness, but that’s outside the scope of this discussion on University Entrance. It’s interesting to see recommendations for NCEA taken on, with initiatives such as Grade Score Marking, which provides some additional granularity to the system without affecting overall communicability of the qualification. Ultimately, I think very few people would say that we should go back to the School Certificate and Bursary system; NCEA is mostly working and still better.

Part III of this series - History, Reviews, and the Perspective of Government, is available here.