Police Review on Emergent Technologies

This article was first published on The Spinoff, co-authored with Kristiann Allen.

The idea of “emergence”, in a philosophical sense, is the notion that a system can have properties, behaviours and naturally forming rules or patterns that individual parts of the system do not have by themselves – the interactions between the components create something new. Snowflakes demonstrate this phenomenon, where individual ice crystals form and grow as they circulate in the air, leading to unpredictable but complex patterns. This perspective considers how seemingly independent parts of a system co-evolve.

Earlier this month, RNZ reported on New Zealand Police’s Review of Emergent Technologies. It included a stocktake of technologies “tested, trialled or rolled out” by NZ Police, from locating where 111 calls are coming from to facial recognition technology for finding people in CCTV footage. In total, 20 technologies were identified with a further nine “under consideration”. The review was urgently commissioned after the police commissioner was caught unaware that the High Tech Crime Unit had trialled the controversial Clearview AI product – understandably, he wanted to know if there were any other unknown surprises on the horizon.

The use of the term “emergent” in the review’s title is interesting. Police intend for it to mean “new” or “in development”, but the similarity to “emergence” also reminds us that none of these technologies exist on their own – they form part of a wider police organisation and can have interactions and synergies that are less predictable than when they are considered independently. In what way could automatic number plate recognition be combined with drones? How might a facial recognition tool currently restricted for use by the investigations team broaden in scope to other teams over time? What information might a smartphone analysis tool uncover that is incidental to an investigation? There is perhaps a missed opportunity for the review to consider the emergence of these types of interactions and combined effects (which could be both good or bad). There are also other, arguably less digital technologies and scientific tools that are, or could be, part of the system as well, such as DNA sampling, chemical analysis of substances, and/or measuring brain waves.

We can consider properly these scenarios when we have an understanding of what technologies are being used. But it is concerning that some of the technologies described in the police review had not previously been disclosed to the public. While NZ Police could argue that disclosing the use (or proposed use) of a particular technology might compromise its effectiveness against clever criminals, this argument is somewhat undermined when vendors and distributors publicly say they have NZ Police as a customer or that they offer the same product that NZ Police uses. Being transparent would go a long way towards avoiding the vacuum that will inevitably be filled with misconceptions and misinformation.

Many of the technologies in the review aren’t too alarming. For instance, the online form for non-emergency reports has keyword scanning to check if there is actually something high-priority that needs more urgent attention. It’s probably good that land and marine search and rescue can get GPS information for people who call in to say they’re lost. Most New Zealanders would agree that it’s appropriate for police to use software tools to automatically detect child abuse material so that human officers don’t have to trawl through millions of disturbing images. However, the risks and negative consequences are definitely much more severe in some areas than in others and should not go unexamined. We’ve written about the issues presented by facial recognition technologies in the past, but shouldn’t there be more scrutiny about, for instance, the fact that NZ Police executive endorsed the use of “remotely piloted aircraft systems (aka drones)” in June 2019? This particular technology is described in the review with a single sentence, yet it significantly changes the way that police can engage in surveillance activities and how that might impact the public.

Nor does this review offer much detail about process. How does a new technology make it from being a potential tool, to trial and testing, to operationalisation? Who is responsible for making the relevant decisions, and what criteria are they considering? What is the role of consulting external experts or the public, and listening to them seriously? The review says “privacy, legal, and ethical implications have been appropriately considered”, but this statement by itself doesn’t offer much confidence to people who can’t see those considerations to evaluate for themselves. What public guarantees are there that this work has been done to a high standard, and that the technology will be used in ways that actually improve public safety? Establishing a Police Technology Ethics Group or similar with a diverse range of members, including external experts, could help improve public confidence that broad impacts are being considered and that harms are mitigated. The New Zealand public service has not previously engaged much in “technology assessment” that is common Europe and the US, but recent bodies like the Data Ethics Advisory Group for StatsNZ and the Data Science Review Board for Immigration NZ provide examples for how this could work.

After the emergent technologies review was conducted but before it was released to the public, NZ Police developed some new rules requiring formal approval from one of two governance groups before trialling emergent technologies. This is a good step in that it helps us understand the approval processes and clearly identifies the group who is accountable for decisions. But we have no visibility into what trials these groups might be considering, what they have approved, or by what criteria they make their decisions. Better transparency would be helpful for improving confidence that these processes are taken seriously and that decisions are robust and of high-quality. It could also give stakeholder groups and the public an opportunity to highlight potential risks or harms that may have been overlooked by police. Collaborative processes with a diversity of inputs can be instrumental in revealing implicit biases and unrecognised risks.

Something police could consider are moratoriums on the use of more controversial technologies, or “red lines” that they will not cross. This is particularly important where technologies have been shown to have significant error rates or bias that disproportionately affects certain groups of people, and where combining that with the power of the police may be dangerous. Police repeatedly state in the review that facial recognition technologies are currently only used on stored footage in an investigative setting, not on live CCTV feeds in a real-time or frontline policing context. If NZ Police intend to maintain this policy on an ongoing basis, then it may help public confidence and understanding if they make such distinctions more obvious and pledge to not cross these lines, even if that decision is to be reviewed as the technology advances. A policy position like “police pledge not to use facial recognition on live CCTV feeds, to be reviewed every five years” might offer some confidence without unduly limiting NZ Police’s ability to consider the role new technologies might have in their toolkit as those technologies continue to improve over time. Similar provisional positions could be taken about predictive policing and online surveillance, which are high-risk and controversial topics. This approach could also help NZ Police avoid the risk of scope creep that could erode rights protections through incremental changes to policy over time.

Lastly, all of the above assumes that NZ Police are internally motivated, equipped, and enabled to change the way they approach emergent technologies and be a lot more transparent about their plans and actions. To their credit, it has been helpful for NZ Police to release this review, and there are encouraging statements about strengthening governance and oversight of emergent technologies, considering privacy and ethics, and ensuring better public and stakeholder engagement. But if the recommendations are not ultimately adopted, then the key method to realise the beneficial effects would be through government regulation.

To be sure, regulation is a powerful tool, which is used much more sparingly today than in the past, but these emergent technology challenges are not merely “operational issues” that are beneath regulatory scrutiny. Providing clarity to both NZ Police and the public about what is and isn’t acceptable use of these emergent technologies is an appropriate role for government to play. The Search and Surveillance Act 2012 needs to be updated to reflect the changes in technology that now allow for far more invasive and automated actions, rather than relying on police interpreting the intention of the Act in a way that protects the rights and freedoms of individuals. This direction is also needed for the judiciary, so they can understand how parliament wants these new technologies to be treated and where the limits on warrants are. It’s important to note that once a technology or tool is approved for use by police and operationalised, the process of stopping its use is much harder than not starting in the first place.

The review is a good start in terms of shining the light on emergent technologies that NZ Police are experimenting with and in some cases operationalising. But it also leaves us with many more unanswered questions about how these technologies are being used, why they are deemed to be necessary, what the oversight and audit mechanisms are, and whether these technologies risk infringing on human rights or defying privacy principles. The review leaves us wondering how these technologies are being incorporated into the organisation that is the police and into the process that is policing, and what new assumptions, behaviours, and patterns may emerge from the use of individual or combined technologies over time. As the review notes, the government’s principles for the safe and effective use of data and analytics say that “guidance, oversight, and transparency are essential to fostering trust, confidence, and integrity around the use of data the government holds on behalf of New Zealanders.” This trust, confidence and integrity doesn’t come automatically – it must be earned and constantly maintained by all government agencies, perhaps especially NZ Police.

Thank you to Mark Hanna, Anne Bardsley, Jill Rolston, Nessa Lynch, and Colin Gavaghan for providing review and feedback.

Shouting Zeros and Ones: Digital Technology, Ethics and Policy in New Zealand

I edited a book! Thanks to Bridget Williams Books, we worked on this book with a variety of authors to put together this Text, centering the global digital technology debates in Aotearoa New Zealand and focusing on the local context. This summary article first appeared in the August 2020 Tohatoha newsletter.

Data flows around the world as computers trade in 0s and 1s, silently making decisions that affect all of us, every day. Meanwhile, we humans shout louder than ever before, spreading news and opinions through social media, polarising society and pitting ourselves against each other. What are the quiet, hidden impacts that technology is having on New Zealand and our collective psyche? How can we mitigate these impacts so that we can leverage technology sustainably, securely, and safely? A new book in the BWB Texts series, Shouting Zeros and Ones, aims to explore this dichotomy of quiet and loud, off and on in the way we use technology.

A lot has been written globally about how digital technology is changing our societies, but this book focuses on Aotearoa and is a call to action for New Zealanders. Our government has the Integrated Data Infrastructure, which holds information about almost every person that steps foot in our country. We are making progress towards Indigenous and Māori Data Sovereignty, setting an example for the rest of the world. The Christchurch massacre happened on our soil, and our people have taken a leading role in reducing online harm and fighting online fascism so that real-world attacks like that may never happen again.

And we are not immune to the digital challenges that face the globe. Misinformation and disinformation run rampant online, threatening the integrity of information. The environmental impacts of globalised computing are too often swept under the rug in the name of convenience and cost. Predictive risk modelling is used by our government in a variety of sectors, including to inform parole decisions in criminal justice. Online voting is regularly posed a panacea to voter engagement, even though the evidence of effectiveness is weak at best. Digital inclusion/exclusion remains a significant challenge – a symptom of ongoing structural inequality in our society.

The book is not all doom and gloom though. A diverse group of contributors reveals hidden impacts of technology on society and on individuals, and explores policy change and personal action to keep the internet a force for good – home to a careful balance of freedom of expression and the safety and well-being of people in the digital and the real worlds. Where there are challenges, there is also a lot of hope that we can find a pathway forward as a society.

Shouting Zeros and Ones: Digital Technology, Ethics and Policy in New Zealand (ed Andrew Chen) from BWB Texts is available in bookstores in August 2020, and is available for order at https://www.bwb.co.nz/books/shouting-zeros-and-ones

Digital Contact Tracing

Over the last couple of weeks I've written quite a bit about digital contact tracing in the context of COVID-19 and the potential to introduce technology to help make contact tracing more effective. Below are some links to these opinion pieces, which I'll update when I remember to do it:

31 March 2020: The trade-offs for digital data and contact tracing, in which I talk about the subjectivity of proportionality and the difficult trade-off between public health and privacy.

6 April 2020: Hard decisions in digital contact tracing, and a similar later piece in The Spinoff, discussing the myriad of difficult questions that have to be answered, including uptake/adoption rates, usability, privacy, errors, and groups of people who might miss out.

11 April 2020: Digital giants enter digital contact tracing, which talks about the Apple-Google proposal to embed Bluetooth-based contact tracing protocols into all Android and iPhone devices.

13 May 2020: On Fragmentation of Digital Contact Tracing Registers, which was at a time when there were many QR code systems from a variety of different vendors, leading to confusion and frustration for users. This also led to a more complete, retrospective article published in ACM Interactions.

Can we eliminate bias in AI technologies?

The bias problem is common for all artificial intelligence applications – that these systems appear to work really well for some segments of the population, but really poorly for others. This can be a significant problem because it erodes trust in these systems. Imagine your customer having an interaction with an AI system, and it “doesn’t work” – who will they blame? Will they blame the computer for being wrong, or the company for deploying a product that doesn’t work? Will they be willing to give the system another chance, or will they write it off and try their luck somewhere else?

There are only two key places in an AI system where this bias can originate – the architecture of the AI system, and the training data used by that architecture to form a model of the world. AI architectures are increasingly complex, with millions of nodes and connections, and it’s quite difficult for any human to understand exactly how these architectures work or which parts of the architecture to tweak. So it’s much easier to treat it as a black box, and to just assume that it’s probably doing as well as it can.

That leaves us with the training data, and this is where most people have put their efforts. More data gives you a more comprehensive view of the world, because you are providing more examples for the AI system to learn. Imagine showing a toddler hundreds of pictures of Sphynx cats and telling them “this is a cat” – would they then correctly identify a Norweigian Forest cat as a cat? The groups that have been most successful in AI are also those with the largest amount of data – the big tech companies or governments with millions or bilions of samples available. In the boom of AI-powered computer vision, the big object detection competitions weren’t being won by smart research groups at universities with better or smarter tech – they were being won by Google, Facebook, Microsoft, Alibaba using really similar methods but with access to more data.

However, that data is valuable, so they typically aren’t going to release it publicly for free. It can be costly to acquire more samples of the environment, and there is a limit to how much time and effort companies are willing to spend. The recent controversy around Clearview, who breached the Terms and Conditions of multiple social media sites to collect a billion-sample facial recognition dataset, shows how difficult it is to otherwise legally build a truly large dataset.

There are some proposals that synthetic data (i.e. data that has been artificially constructed based on existing datasets to try and make the dataset bigger) can help solve this problem. For example, if we have a facial recognition system, deepfake technology could be used to help generate a wider variety of faces, with a bit of randomness to help create new data points that previously did not exist. I think the challenge with these proposals is that they are unlikely to eliminate bias – worse, they may further hide the bias by pretending that it isn’t there. This is because the deepfake technology is trained using a dataset as well, and if that underlying dataset is biased, then it becomes encoded in the synthetic data too. Some have argued that you can control how the synthetic data is generated to address specific biases. But if the designer already knows where those biases are, then they should really invest the time and resources into collecting the right data to address those deficits, rather than trying to create fake data to compensate.

The other missing element is that we currently do not have particularly systematic ways of identifying or testing for bias. There are arguments that AI transparency will resolve this, but transparency can make it easier for malicious actors to fool, or worse, reverse engineer the underlying AI models. Instead, auditing and certification by trusted parties are important for ensuring that AI systems meet a high-standard of performance. This relies on there being a way to define those standards. Some vendors are providing auditing services as part of their product offering to determine if bias is present, but they are also incentivised to claim that there is no bias.

We need independent researchers and government agencies to invest resources into developing these standards and tests, alongside appropriate regulation and investigative powers to ensure that these standards are being met. There are predecents of government investigations of technologies, particularly in safety-critical applications such as aircraft and vehicles. For example, road transport and highway authorities generally have wide-ranging powers to investigate how car crashes happen - in the wake of an Uber autonomous vehicle killing a pedestrian, the US National Transportation Safety Board engaged technical experts to go through the hardware systems and software code to understand what the sensors in the car saw and how decisions about stopping the vehicle were made.

The trouble for the people and companies using AI systems is that very few would intend for their systems to be biased. Perhaps this is a limitation of the technology that we have available to us at the moment, and bias just exists without anyone putting it there. But there are two key actions that can mitigate and minimise that bias as much as possible – collecting as much real-world data for training as possible, and testing/auditing the system widely including exposing the system to edge cases. These may both seem like costly actions, but it’s worth comparing this to the cost of losing customers because the system doesn’t work for them. Understanding the cost of errors in the system is critical to understanding how much effort needs to be put into avoiding them.