Monday 16 February 2015

University Entrance: Appendix - Chronicling an OIA journey

Awhile ago, I was wondering about the pass rate of courses at different universities, and whether conclusions could be drawn about any given students' chances of success at a university once they have been accepted. Obviously there are a multitude of variables that affect pass rates, but there may be some overall patterns at play. There are many rumours about whether some courses are easier or harder to pass than others, whether universities have (un)official policies about pass rates, and whether teaching staff are "encouraged" to "massage" the numbers to get students to pass.

So on March 13th 2014, I sent the following e-mail to each of the eight universities in New Zealand:

To the Registrar [or other OIA designee],
Under the Official Information Act, I would like to request a list of the:
1. Course Code
2. Course Title
3. Number of students enrolled
4. Completion Rate
5. Pass Rate
6. Mean, median, and range of the marks (in %)
7. Whether the final scores were scaled from the raw scores
8. Whether the course is externally moderated
for every taught paper delivered by the University in the past three years.

I would strongly prefer this in a digital format (such as an Excel spreadsheet) if possible. Please let me know when this this request has been received. Please let me know in advance if there is a cost associated with this request.

Below are the responses that I received, in chronological order:

University of Waikato (17/03/14)
Please be advised that you can find this information in the Annual Report released yearly. Please visit http://www.waikato.ac.nz/annualreport/annualreport.pdf for more information. The 2013 Report will be released soon. You can also visit http://calendar.waikato.ac.nz/

This unfortunately did not satisfy my request. The Annual Report (which all universities produce) does have the total number of students enrolled, the completion rate, and pass rate, but I was interested in the data for each individual taught paper. I replied to let them know, but received no response after that.

University of Auckland (17/03/14)
I was called by the General Counsel of the University, who informed me that while 1-5 of my request would be relatively easy to source, 6 and 7 would be almost impossible as that would be managed at a Faculty level and it was unlikely that the information was even stored (which is reasonable). With regards to 8, most courses are not externally moderated as there are 5000 courses and it would be impossible to moderate them all. I was advised to send in an amended request so that the request wouldn't be refused overall (as per 18B of the OIA Act). I sent a new request for just 1-5 and a list of externally moderated courses on the same day.

University of Otago (19/03/14)
I received a response that my request had been passed on by the Registrar to the Manager, Policy and Compliance in the Academic Services Department for co-ordination. I replied to thank them for responding.

University of Auckland (01/04/14)
I got another call from the General Counsel, who said that they had the data for 1-5 ready and was able to send it to me. Unfortunately she had to refuse the request with regards to information about externally moderated courses - it wasn't an issue about funding, it was simply a time and resources issue because it would be difficult to co-ordinate the collection of that information. The e-mail I then received read as follows:

Please find attached the information you requested in points 1 to 5 in your amended request for information as stated in your email dated 17 March 2014. The information is provided in both your preferred format, and excel spreadsheet, and in a  pdf.

All courses taught in 2011 to 2013 are included in the list. Please note that the student enrolments are provided in EFTS as that is how the University reports on pass rates and Course Completions.  

As I mentioned to you during our telephone conversation this afternoon, we do not have a central repository of information relating to the papers that are externally moderated, and a list of such papers cannot be made available without substantial collation and research.

Accordingly, as I have already advised, your request for this latter information is refused under s18(f) of the Official Information Act. I am obliged to advise you have the right, by way of a complaint to the Ombudsman under s28(3) of the Official Information Act, to seek an investigation and review of this refusal.
University of Otago (02/04/14)
I recevied the following e-mail from the Registrar:
I refer to your request under the Official Information Act for information relating to student grades by course subject code.  Unfortunately, the statistical data in the particular format you describe are not readily available, as the University does not standardly report to the level of individual courses.  However, we have now had the opportunity to identify existing reports we have that could provide you with meaningful data in response to your request.

In the University of Otago’s Annual Report, we publish a table entitled Examination Pass Rates -  http://www.otago.ac.nz/about/otago045429.pdf (see page 102).  We have been able to break this down to a greater level of detail into pass rates by subject for the five years 2009 – 2013.  This report is attached for you.

We must emphasize that these data do not take into account the nature of student cohorts in the various subject areas, variations in course content, variations in the nature of course delivery, differences between professional programmes and general areas of study, and a range of other variables all affecting pass rates.  

However, bearing all these limitations in mind, we trust that it is of interest to you.

The data I received was a PDF, in a similar format to that seen on Kiwiblog here. Given that it is unfortunately quite different to the University of Auckland data, it will be difficult to do direct comparisons. In particular, the fact that the data was given for courses/subjects rather than papers, with no distinction for number of students enrolled or level of study, drawing conclusions may not be possible.

Victoria University of Wellington (11/04/14)
I received an e-mail from the Assistant In-house Solicitor, which included an official response (printed on the letterhead and then scanned):

I refer to your request of 13 March 2014 to Victoria University of Wellington ("Victoria") under the Official Information Act 1982 ("the Act") for the following information:

[The OIA request as stated above]

In response to parts 1) to 5) of your request, a spread sheet containing all courses for the 2011 - 2013 academic years, including title, code and the number of students enrolled is attached. The pass rate is the percentage of students who passed the course. The completion rate is the percentage of students who completed the requirements of the course, this may include some students who did not necessarily pass the course.

Where a course had 10 or fewer students I have withheld the information regarding pass and completion rates under section 9(2)(a) of the Act on the basis that it is necessary to protect the privacy of natural persons (the students) and that this is not outweighed by any other consideration that would make it desirable, in the public interest to release this information.

The information regarding 6) to 8) of your request is not held centrally and would require collating the information across all 29 schools at Victoria. Further, providing the information in part 6) of your request would, in some cases, involve staff undertaking calculations. As the requested information covers 6112 courses over three academic years, collating the requested information would take a significant amount of time and involve multiple staff members.

I have considered whether extending the timeframe or fixing a charge to provide the information would enable your request to be granted. However, I estimate that Victoria would need to extend the timeframe for response by 1-2 months and any charge fixed would be substantial due to the number of staff hours required, therefore, I do not consider these to be viable. As such, I am refusing these parts of your request under section 18(f) of the Act on the basis that this information could not be made available without substantial collation and research.

You have a right, under section 28(3) of the Act to seek review by an Ombudsman of my decision to refuse your request.

I replied with a gracious thank you, accepting the reasons for the refusal of the request. However the data was provided in PDFs, which is a little unwieldy to work with, particularly when there are thousands of courses, so I asked for an Excel spreadsheet. This was denied to "ensure that the data I had withheld was not inadvertently provided within the excel spreadsheets". I ultimately converted the PDFs into spreadsheets myself (it was a bit more than just a copy-paste job).

Lincoln University (14/05/14)
About two months after the original request, I received an e-mail from the Director, Governance. It included a letter that had been printed, signed by the Vice-Chancellor Dr Andrew West, and scanned back in. I hadn't really anticipated that the question would be escalated so highly.

Thank you for your request under the Official Information Act 1982, received by Lincoln University on 14 March 2014. Apologies for the delay in responding to the request.

You requested [the OIA request as above]. 

The data is provided in the attached Excel spreadsheet. There is a data page and a notes page with the applicable definitions or assumptions. The data does not include courses with four or fewer enrolled students (by head count) in order to protect the privacy of individual students who could potentially be identified. The number of such courses is as follows:

2011 - 148
2012 - 146
2013 - 159

For each taught course, you also sought information on any applicable scaling and/or external moderation. There are very few cases of scaling being applied at the University. The Academic Administration Committee scrutinizes examinations reports from faculties in which scaling would be identified if it was carried out, but scaling is very rarely applied by examiners and would most likely only have been applied when results fell outside the range of historical norms, and there was a specific reason for this. The small number of cases of scaling have therefore not been identified as it would be a substantial task to do so across the three year period.

Taught courses are not currently externally moderated, although theses and dissertations are externally examined. Certain courses will be moderated by professional bodies as part of accreditation activities.

Please note that you are entitled, under section 28 of the Official Information Act 1982, to have this response reviewed by the Office of the Ombudsmen.

I received no response from Massey University, Auckland University of Technology, or Canterbury University, and as indicated above an unsatisfactory response from Waikato University. I could go to the Ombudsman about this, but at this point it's been far too long and I just can't be bothered anymore. I'm happy to just work with the data that I got and try to go from there.

I am choosing not to attach the raw datasets and release them - I believe that the information would likely not be used appropriately by students and lead to consequences that I don't really want to deal with in the long-term. In some cases the class sizes are small enough that individuals would be identifiable (and then people would know if they passed or failed a particular paper). Also, teaching staff are people too, and I don't want to embarrass or cause undue stress to any staff members by releasing the pass rates of courses they teach. If anyone is motivated enough to want to find out the pass rates, they are welcome to submit an OIA request to the appropriate university themselves.

1 comment: