Official information kept secret too long

Correction 2020-06-02 When creating the codebook for this article almost two years after it was published, I had to re-write the code to generate all of the graphs because I no longer had the original code. In this process, I found that I had made mistakes when looking at how many requests were extended by the Ministry of Social Development. These mistakes extended to showing MSD to have had more overdue requests than they actually had, because I had not correctly recognised that some requests had been extended. When re-creating the graphs, there were also some small differences due to different choices being made when cleaning the data. For all details of the changes made when this article was corrected, please see the log of changes made to this page on GitHub.

For the past few years, I have been keeping track of how long it takes for government agencies to respond to Official Information Act requests.

The OIA sets a time limit of 20 working days to complete Agencies can extend this, but they have to tell the requester how much longer they need and give a reason why the extension is necessary. Technically the time limit is for sending a notice of the agency's decision about whether or not to release the requested information, rather than applying to the release of the information itself. Section 28(5) of the OIA essentially requires that, when a decision is made to grant a request, the information must be released without "undue delay". However, in the vast majority of cases, information will be released at the same time as a requester is told the agency has decided to grant their request. these requests. As well as this due date, the OIA also requires that requests must be completed "as soon as reasonably practicable". However, I've found that it's been very common for responses to my requests to arrive on the day they are due.

Of course, this might be because it wasn't reasonably practicable for a response to be sent any sooner than the due date. But when less than half of the responses arrive before the due date, things start to look a bit suspicious.

In late 2014, then Prime Minister John Key admitted that, when handling OIA requests:

Sometimes we wait the 20 days because, in the end, Government might take the view that's in our best interest to do that

John Key

However, it's been a few years since then, and things might have changed. Later that year, then Chief Ombudsman Beverley Wakem started a year-long review of the operation of the OIA across 12 government agencies, which resulted in the report Not a game of hide and seek. In response to Key's admission of the government gaming the OIA, she said:

It's pretty clear. It couldn't be much clearer than that... As soon as you have made a decision as to whether you're going to respond to the request or how you're going to respond to it, you ought to convey that.

Beverley Wakem

This view was backed up in April 2018 by the current Chief Ombudsman, Peter Boshier, who referred to the report in a speech given to managers at MBIE:

Some [agencies] were frank about their practice of 'gaming' the OIA, particularly by interpreting the 20-day time limit as an opportunity [to] 'put it off until the 20-day deadline' to either delay release or delay having to deal with the request.

This is clearly not acceptable in an open and participatory democracy.

Peter Boshier

One thing that has changed since 2014 is that the State Services Commission has started collecting and publishing statistics on OIA compliance.

This includes a measure of how many requests were completed "within the legislated timeframe". Most agencies seem to be doing fairly well, with few falling below 90% for this measure of compliance. But were those responses sent "within the legislated timeframe" also sent as soon as reasonably practicable, or did they tend to be put off until the due date arrived?

To find out, I sent requests for six months' of data on OIA response times to 11 government agencies, many of which had also been included in the review by the previous Chief Ombudsman.

I also sent an OIA request to the State Services Commission beforehand, in the hope of both refining my request and getting some basis for comparison, asking how quickly they responded to OIA requests in 2016/17.

There are various ways in which agencies can delay OIA requests, or deter requesters. Some of these can be examined by looking at the data that has been released, others I've observed anecdotally but don't have data to show how common or rare they are. Whether or not it's ever an agency's intention to delay or prevent the release of official information isn't something I can tell from that, but in either case the effect is the same.

You can find links to the data released by each agency at the start of their section. There's more information there than I have analysed in this article, and I'd encourage you to explore the data yourself if you're interested.

State Services Commission

Data

The day before my request to them was due, the SSC extended it by eight working days. I received the data the day before that extended due date.

The SSC told me they dealt with The criteria used for my requests weren't identical to those used in the statistics published by the SSC, so the numbers won't always line up perfectly. For example, the 2016/17 OIA statistics published for the SSC show they completed 97 requests in that timeframe, all of which were completed on time. 96 OIA requests in 2016/17. Looking at the number of working days remaining when they completed these requests, a very clear pattern emerges:

Requests that didn't have their due date extended have a fairly even spread, with all but one being completed before the due date. But over a third of all requests were extended, and all but four of those were completed on the day before the due date.

However, looking at the lengths of those extensions the spread is also fairly even:

So instead of waiting until the day before requests are due before responding, it looks like the SSC is probably just good at estimating the amount of work remaining, and they tend to add an extra day just in case.

There is one more thing worth checking, however, and that's whether or not the SSC has a practice of not sending notices of extension until a request's original due date arrives:

There's a fairly even spread here as well, though there was one request where a notice of extension wasn't sent until the day after the request was originally due. That request was extended by 16 working days, and was one of those that received its final response the day before its extended due date.

Given the SSC's involvement in publishing OIA guidance for other agencies and monitoring OIA compliance, I'm not surprised to find that they seem to be doing pretty well here.

Ministry for Primary Industries

Data

I was pleasantly surprised by the quick response I received from MPI. In contrast with the other agencies I contacted, MPI was quick to get in touch with me to discuss refining my request. In the end, they took only 6 working days to respond, which was one of the fastest responses I've ever received to an OIA request.

We reached a compromise where they would release only some of the data I had requested, because the rest would be difficult to retrieve. However, the quick turnaround and the fact that they'd talked to me about how best to manage my request were both refreshing changes from my typical experience with OIA requests.

Unfortunately, the data are not so complimentary:

There is a huge spike in responses sent on the due date. There is also a long tail of requests that were not responded to in time. Nineteen more late requests don't fit on the graph, with the latest being over five months overdue.

Though the SSC's stats report that MPI completed 87% of requests in this period within the legislated timeframe, that number obscures the fact that fewer than half of their responses were sent before the due date.

Of those requests that were late, they weren't clustered soon after the due date as you might expect. Rather they were spread out over the next few months. Of the overdue requests, the median number of working days late they were was 15.5 — over three weeks.

MPI didn't tell me how many days were remaining when they sent notices of extension, so I can't tell if these were also left until the last minute. I can, however, look at the durations of these extensions:

(The request which was apparently extended by 0 days was reported by MPI as having been extended, but the due date they reported for it was 20 working days after the date on which it was received.)

Unlike the SSC, MPI's extensions are clustered around multiples of five. Extensions of 20 working days, in particular, are very common.

Those requests extended by a multiple of 5 working days did tend to be answered a little less late On average, requests extended by a multiple of five working days were 2.1 working days overdue, whereas other extended requests were 6.4 working days overdue on average. There were some outliers though, requests that were months overdue which can skew the average. By far the most common outcome for each group of extended requests was for the response to be sent on the day it was due. on average.

Whether they intend to delay OIA requests or not, that certainly seems to be the effect of the way MPI handles them.

MPI have also recently made the news for dragging their feet in releasing information to the Herald. MPI refused a request the Herald made in March, but reconsidered their decision following an investigation by the Ombudsman. MPI also extended a separate request from the Herald by a further 25 working days, only notifying the Herald of the extension once the due date had arrived.

Ministry of Justice

Data

The Ministry of Justice handled 446 OIA requests The numbers reported by the SSC — 436 requests completed with 85% being on time — are slightly different due to different interpretations of the criteria for which requests to include. within Q3-4 2017, and 88% of them were answered within the legislated timeframe.

Of those 87%, about a third of them were only completed either on the due date or the day before it:

The Ministry of Justice only extended a relatively small number of their requests compared with agencies like MPI and the SSC. It's quite possible that different agencies will typically receive OIA requests with different degrees of complexity, so without more information I wouldn't want to read much into that. The Ministry of Justice may not have extended many OIA requests because it just didn't need to.

However, it seems the Ministry of Justice also avoided sending notifications of an extension after a request's due date. They didn't include the date on which extension notices were sent in the data they released, but many of their overdue requests The Ministry of Justice extended 44 OIA requests within this period, whereas 36 unextended requests were at least two working days late. had not been extended. This is despite there being time between the due date and when the request was completed during which the requester could have been told how much longer was still necessary.

The Ministry of Justice was about twice as likely to respond to extended requests on the due date (34%) as they were for unextended requests (18%). This could be in large part due to the durations of the extensions being accurate reflections of the time remaining, but looking at the distribution of these durations there is a pattern of extensions being most often taken for whole numbers of weeks:

With such a small number of extended requests, it's hard to conclude much from this pattern for the Ministry of Justice. The largest spikes are at 5, 10, and 20 working days. Those extensions account for just over a third of all extensions (36%). This could be worse, but it still seems likely that this practice of often extending requests by a number of weeks rather than a number of days may be leading to responses being unnecessarily delayed.

Ministry of Foreign Affairs and Trade

Data

The Ministry of Foreign Affairs and Trade handled just over 100 OIA requests in this timeframe, and the SSC's statistics report that 93% of them were completed within the legislated timeframe. However, as with other agencies, a large number of those requests were completed on the due date. In the case of MFAT, about a quarter of OIA requests they completed received a response on the due date:

When I asked on Twitter about agencies with a history of handling OIA requests poorly, I was told that the Ministry of Foreign Affairs and Trade were "serial delayers". Sure enough, more than half of the requests MFAT handled in this period had their original deadlines extended.

Unfortunately, MFAT didn't release information on how much time was remaining when a notice of extension was sent, so I can't tell if it was also common for this to happen on the due date.

Those requests that were extended were about as likely to receive a response on the due date as unextended requests. However, looking at the lengths of the extensions taken by MFAT, every one of them was a multiple of 5 working days:

This implies that MFAT was only estimating the remaining time in terms of weeks, rather than days. Their extended requests were no more likely to be responded to before the due date than those that weren't extended, so it seems either some of these requests were delayed or Parkinson's Law may have been at play: "work expands so as to fill the time available for its completion".

If that was the case, then MFAT's practice of always extending OIA requests by a number of weeks instead of a specific number of days may be effectively delaying these OIA requests more than is necessary.

MFAT's OIA practices underwent a fair amount of criticism during the negotiations of the Trans-Pacific Partnership Agreement (TPPA). In 2015, they lost a court case to Jane Kelsey and several others, where it was found that their blanket refusal to release information was unlawful.

Accident Compensation Corporation

Data

I had some trouble with ACC's response, at first. The reason being that, though I had explicitly requested that the data be released as a spreadsheet, their response (which arrived on the afternoon of the due date) only included a PDF of the data. Though it thankfully wasn't one of the dreaded image-based PDFs that are so commonly used, it still wasn't useful to me until it was converted into a spreadsheet.

Section 16(2) of the OIA requires that agencies must release data in the format preferred by the requester unless there is a specific good reason not to. ACC had not done this, and when I asked them again to release it as a spreadsheet they took a further 8 working days to do so.

This is an example of the kind of response that, while it could be considered overdue, will certainly have been recorded as having been completed within the legislated timeframe. Because of how it avoids being recorded, there's no real way to know how common or how rare this sort of case is, within ACC or across other agencies.

Looking at the data they have recorded, however, it looks like ACC have generally done very well at responding to requests within the legislated timeframe.

Out of over 400 OIA requests, only seven were completed outside the legislated timeframe. Unfortunately, some of those seven were significantly outside the legislated timeframe. The worst of these requests was over six weeks overdue, and ACC did not record sending a notice of extension to the requester to let them know it would be late and how much more time was necessary.

ACC extended about 11% of requests within this period. Looking at the pattern of extension lengths, there are clear spikes at 10 and 15 working days. Combined, these account for about half of all extensions within this period:

ACC was far more likely to complete extended requests on their due date (53%) than unextended requests (25%). This could reflect well-estimated extensions, like in the case of the SSC, but given the spikes at extensions of two and three weeks long it seems more likely to me that ACC has similar issues to MFAT in this regard. If the duration of each extension better reflected the actual amount of work remaining, it's quite possible that would have led to responses being sent earlier.

Like the Ministry of Justice, ACC never sent a notice of extension after any request's original due date had passed. Though to be fair only three unextended requests passed their due date without being extended:

In July 2016, Mark Honeychurch from the Society for Science Based Healthcare I am the chair of the Society for Science Based Healthcare, and have made various OIA requests regarding issues such as ACC's funding of acupuncture, though I wasn't directly involved in these particular discussions with ACC. was told by ACC that, instead of asking for information through an Official Information Act request, he should make a "media request".

This revealed a lack of understanding on ACC's part of what the Official Information Act does, which likely stemmed from the fact that they have a specific internal process for correspondence labelled as "OIA requests".

The OIA applies to every request for information received by a government agency. It doesn't matter who asked for it, how they asked for it, or whether or not they mentioned the OIA. In all cases, the agency has exactly the same legal obligations.

In the Not a game of hide and seek report, it was noted that many agencies treated requests differently depending on who the requester was. Recommendation 27 of the report was that:

Agencies should review their practices to ensure that the identity of the requester, their mode of engagement, or any practices do not impinge on the requirements to make a decision that is appropriate under the OIA and communicate it to the requester 'as soon as reasonably practicable'.

When ACC was asked for information as a "media request", their media advisor responded with the requested information within an hour. While that response time is certainly laudable, there's no reason why such a quick response should be restricted to only certain classes of requesters. It's also not clear whether or not ACC includes these requests as OIA requests for the purpose of their records and the statistics published by the SSC.

Ministry of Business, Innovation and Employment

Data

MBIE extended my request by 20 working days, and sent the eventual response on the afternoon of the due date. The SSC's data showed they completed 782 OIA requests within Q3-4 of 2017, 97% of which were completed within the legislated timeframe.

In their response to me, over 100 requests didn't have a completion date recorded, due to being either transferred (24 requests) or withdrawn (79 requests). There were also a dozen or so rows where there had clearly been errors in data entry (e.g. requests that were reported as being extended before they had been received, or after they had been completed). These requests, which I've had to exclude from my analysis due to the missing information, made up 14% of the data reported by MBIE.

Of the remaining 710 requests, most of them were completed no earlier than the day before the due date. The spike on the due date itself represents over a third of all MBIE's responses.

Very few requests — only 28 — were overdue. However, some of those requests appear to have been very overdue. The worst offender in that regard is a request that was received in October 2016, and extended by 33 working days in November that same year so that its new due date The Official Information Act effectively takes a break over summer, as dates from the 25th of December to the 15th of January aren't considered to be "working days". This means requests received near the end of the year typically take longer to be completed. was the 20th of January 2017. The actual response didn't arrive until the 29th of August, over seven months overdue.

As well as this rather extreme example, MBIE also had requests overdue by 117, 62, 51, 44, and 27 working days. The three requests that were the least overdue were not extended, even though MBIE had plenty of time after their due date during which they could have told the requester how much time was remaining.

MBIE extended a quarter of all their requests, and extended requests were far more likely to receive a response on the due date itself: 53%, compared with 30% of unextended requests.

Of course, this could just be due to MBIE being good at accurately estimating the time remaining when extending a request. Here's the distribution of the lengths of extensions they granted:

As seen with many other agencies, there are large spikes at extensions of a whole number of weeks. Extensions of 20 and 10 working days were by far the most common, comprising 42% of all extensions. Requests extended by these durations were less likely than other extended requests to receive a response on the due date (40% vs 60%).

As with the Ministry of Foreign Affairs and Trade, it seems likely that MBIE's practices of often extending OIA requests by a whole number of weeks is delaying their responses more than necessary.

Looking at the amount of time remaining when MBIE sent requesters a notification of extension, it's clear that this is also often left until the last day:

As with the final responses to requests, MBIE sent more than half of their notices of extension no earlier than the day before the due date.

With requests requiring extensions almost always requiring many weeks more time to complete, it seems very unlikely that MBIE has been notifying requesters of these extensions as soon as reasonably practicable, as might be expected of them. If you need another four weeks to complete something when the due date arrives, either you've known for a while that you'll need more time or you've just let it sit until the last minute.

Ministry of Social Development

Data

The statistics published by the State Services Commission show that MSD completed 747 OIA requests within Q3-4 of 2017, 97% of which were completed within the legislated timeframe.

Unfortunately, they only record detailed statistics for one third of these requests, as the remainder are considered "media requests" by MSD and handled through a different internal process. They also do not record data on OIA requests that were answered within 24 hours, so they only released information on 241 OIA requests to me.

In analysing this data, I found that MSD uses an unusual method of responding to OIA requests. The approach is uncommon and unexpected enough that I've seen many people, including journalists familiar with the process of making OIA requests, question its legality when they first encounter it.

The thing is, the OIA's 20 working days due date technically doesn't apply to the information that's been requested. Instead, it applies to sending the requester a notice of the decision made on the request. This means it's legal for agencies to tell the requester that they have decided to provide the information they requested, but not actually provide that information until a later date.

Just over a third of the OIA requests included in the data released by MSD seem to have used this method. Technically, the legal countdown to the due date stops once a notice of decision is sent, so keep in mind when looking at this graph of the number of days remaining when MSD "completed" their requests that it doesn't entirely reflect when they released information that had been requested of them: