The integration of Artificial Intelligence (AI) into criminal investigations is no longer a theoretical exercise for legal scholars. While much of the recent discussion around generative AI (genAI) in legal practice has centred on workplace efficiency and the risks of hallucinated case law, a far more fundamental challenge has arrived in the courtroom: the admissibility of evidence derived from AI systems.
As law enforcement agencies increasingly leverage automated analysis, such as connecting common cases together through series linking, matching biometric information such as facial recognition, and inferring behavioural analytics such as keystroke analysis, the criminal justice system must grapple with tools that are inherently probabilistic and opaque.
At the same time, as technology advances and people find more utility in the outputs of AI systems, societal expectations around the use of AI-derived evidence will shift. If AI-derived evidence can help establish the truth, holding offenders or responsible parties accountable, then we must consider if that type of evidence can help uphold an effective and credible system of justice.
In August 2025, this challenge was put to a rigorous practical test through a series of mock trial hearings hosted in the High Courts of Christchurch, Auckland, and Wellington. Presided over by real judges and argued by experienced counsel, the hypothetical case of R v Popper examined the evidential and statutory friction points created when Police rely on AI-derived intelligence to obtain warrants.
The Fictional Facts of R v Popper
The moot proceedings were based upon an invented AI software tool and an invented scenario: an application to exclude evidence seized during the execution of a search warrant, when the warrant had been obtained using information ultimately provided by AI.
Following the proliferation of MDMA pills bearing a distinctive "Goodnight Kiwi" logo at various music festivals, New Zealand Police identified an anonymous vendor operating on a dark web marketplace. Lacking investigative leads to uncover the account holder's identity, Police deployed a (hypothetical) novel, commercially-available AI software tool named TraceLex.
TraceLex utilises advanced "lexical analysis" to compare input writing samples against a massive, continuously updated database of publicly available online text, also referred to as Open Source Intelligence (OSINT). The system outputs candidate text matches along with a ranked list of prospective authors accompanied by confidence scores.
When Police put the writing samples from the dark web marketplace into TraceLex, the tool flagged matching content on Reddit. Police intelligence analysts examined the Reddit content for that account and discovered a screenshot of a utility invoice displaying Mr Popper's name and address, as well as a vehicle photograph containing a partial facial reflection matching Mr Popper. Analysts concluded there was a "high likelihood" (defined in intelligence standards as 75–85%) that Mr Popper operated the dark web account.
Relying primarily on this intelligence analysis, Police applied for and was granted a Surveillance Device Warrant under s 51 of the Search and Surveillance Act 2012 to track Mr Popper’s vehicle. Officers subsequently conducted a controlled drug purchase, observing Mr Popper leaving a package at a dead drop location. This observation formed the basis for a search warrant executed at Mr Popper’s residence, where Police seized substantial quantities of MDMA powder and pressed pills bearing the "Goodnight Kiwi" stamp. Mr Popper was charged with possession of a Class B controlled drug for supply.
The pre-trial hearing focused on two primary questions:
- Was the Surveillance Device Warrant validly issued given its reliance on TraceLex?
- If the warrant was invalid, making the evidence improperly obtained, should the resulting drug evidence nonetheless be admitted under the balancing test in s 30 of the Evidence Act 2006?
Designing TraceLex: The Technical Anatomy of AI-derived Evidence
To reflect the reality of modern machine learning tools without impugning existing proprietary software, and to mirror the legal pain points seen in the jurisprudence to date, TraceLex’s design in the R v Popper moot problem incorporated features of leading commercial AI applications which have been tested in international case law.
1. The "Black Box" and Commercial Sensitivity
TraceLex operates similarly to generative AI detectors, evaluating complex stylistic markers such as word choice, sentence structure, and punctuation syntax. However, the exact weightings behind its neural network remain an inscrutable "black box". Unlike traditional forensic handwriting or linguistic analysis, where an expert witness can explain the precise stylistic anomalies pointing to common authorship, an AI model cannot articulate why it reached a specific conclusion.
Furthermore, because TraceLex is a cloud-based, proprietary third-party product, its vendor invoked trade secrets and refused to supply source code or technical representatives for cross-examination. This mirrors controversial US jurisprudence such as State v Loomis 881 NW2d 749 (Wis. 2016), where the Wisconsin Supreme Court allowed the use of the COMPAS risk-assessment algorithm in sentencing despite its underlying code remaining a proprietary secret, so long as the risk-assessment report warned judges of its algorithmic limitations. In R v Popper, the Court had to rely on an independent expert who could only conduct external "black box" user testing.
2. Probabilistic Inconsistency and Non-Deterministic Models
Unlike deterministic, rules-based software that produces identical outputs for identical inputs, advanced AI systems are probabilistic and stochastic. This introduces variability and unpredictability. In the US decision Matter of Weber 2024 NY Slip Op 24258, an expert witness used Microsoft Copilot to produce his damages calculations. When the court performed its own cross-check by re-entering the same damages calculation prompt three times, Copilot produced three distinct figures, none of which matched the expert’s evidence. In R v Popper, defence counsel highlighted that TraceLex could produce variable and inconsistent results. They also pointed out that cloud-side updates by software developers could alter model behaviour over time, severely compromising the scientific requirement of repeatability.
3. Misinterpretation of Confidence Scores
TraceLex outputs a numerical “confidence score”, which (as seen in the R v Popper moot) humans often erroneously interpret as a linear percentage. In fact, AI scoring models are non-linear: an increase from 0.67 to 0.68 does not represent the same quantitative increment as a jump from 0.66 to 0.67. Inappropriately setting thresholds can become subject to challenge, as demonstrated in R (Bridges) v South Wales Police [2020] EWCA Civ 1058, where the UK Court of Appeal noted the dangers of a live facial recognition system’s similarity score threshold being set in an unclear way.
4. Mass OSINT Data Sourcing and Privacy
TraceLex relies on continuous scraping of public domain data, which may not be reliable in itself. Beyond accuracy, continuous scraping raises severe privacy concerns under s 21 of the NZBORA 1990. Defence counsel drew parallels to third-party data aggregation systems, such as automated number-plate recognition networks, where private databases are accessed by Police for law enforcement purposes.
Core Evidentiary and Legal Arguments
Common themes emerged across the three moot hearings.
1. Human-in-the-Loop vs. "Rubber-Stamping"
The prosecution contended that TraceLex was never used as a sole determinant of guilt, but merely as an investigative lead reviewed by experienced Police analysts. This "human-in-the-loop" framework was argued to cure any underlying technical opacity.
Conversely, defence counsel argued that human oversight is illusory if the analysts inspecting the output lack the technical literacy to understand the tool's error rates and failure modes. Although human analysts endorsed TraceLex’s match, manual lexical analysis across such volumes is challenging for humans. The technology ethics literature refers to the common consequence of this as “automation bias”, where humans tend to agree with the automated computer systems over time. Relying on TraceLex’s unvalidated vendor claims without independent evaluation before applying for a warrant was characterised by the defence as a breach of the applicant's duty of candour to the issuing officer.
2. Assessing Reliability under the Evidence Act 2006
Under New Zealand law, novel scientific evidence is evaluated under the opinion evidence rules of the Evidence Act, requiring expert evidence to be "substantially helpful" (an amalgam of relevance, reliability, and probative value).
In expert experiments conducted for the trial, TraceLex correctly identified the seven out of ten authors under realistic conditions in a single test. The Defence submission was that a 70% accuracy rate under small-sample test conditions demonstrates substantial unreliability, which cannot be cured post hoc simply because the defendant was found with the drugs in his home.
The Prosecution argued that s 51 of the Search and Surveillance Act requires only "reasonable grounds to suspect" an offence and "reasonable grounds to believe" probative evidence will be obtained. Scientific certainty is unnecessary; the threshold is lower than proof beyond reasonable doubt, and the tool's output formed part of a compelling "global picture".
3. Evidential Analogies: Informants, Calculators, or Hearsay?
Counsel tested several legal analogies to categorise AI evidence:
Anonymous Informants: Like confidential informants, AI outputs cannot be cross-examined, requiring independent verification to move beyond mere suspicion. However, unlike human informants, AI lacks human motives like malice or financial reward.
Scientific Instruments: The prosecution analogised TraceLex to calculators, breathalysers, or radar guns, where courts routinely accept technical outputs without inspecting underlying source code or electronics. The defence countered that breathalysers and other devices enjoy statutory presumptions of accuracy under the Land Transport Act 1998 and s 137 of the Evidence Act in part because they operate deterministically, whereas AI outputs are non-explainable and stochastic.
Hearsay: Academic commentary suggests treating AI outputs as hearsay. However, case law suggests that automated machine generation should be treated as documentary evidence, as hearsay requires a human declarant.
Expert Witness: Experts in criminal cases are generally expected to abide by the Code of Conduct for Expert Witnesses (Schedule 4 to the High Court Rules 2016). These require expert witnesses to give their sources, to state their reasoning, and generally demonstrate that their opinion or evidence is reliable. As a demonstration of how quickly the technology is developing, these requirements would have been difficult for early genAI models to meet in 2022/2023, but now in 2026 many products can provide their sources and explain their chain of thought. Where an AI is replacing the thinking of a human, could it be treated as an expert?
Section 30 Balancing Act & The Shadow of Tamiefuna
The secondary battleground across all three moot hearings was s 30 of the Evidence Act. If the warrant was invalid due to insufficient grounds or lack of candour, should the resulting physical drug evidence be excluded?
The prosecution and defence offered contrasting arguments across four key factors under the s 30(5) balancing test. Regarding probative value and reliability, the prosecution emphasised that the seized MDMA matched the festival pills, confirming TraceLex's accuracy; the defence countered that an opaque, unverified AI process should not be retroactively justified by post-search results. Regarding the seriousness of the offence and the importance of any rights breached, the prosecution highlighted the public harm of commercial Class B drug supply, while the defence contended that the fundamental privacy rights under s 21 of the NZBORA must be upheld regardless of the charge's severity.
The prosecution argued that there was a lack of other investigatory techniques for the untraceable dark web account, whereas the defence submitted that a lack of options does not excuse deploying unvalidated mass surveillance tools. Finally, the prosecution claimed Police acted in good faith based on international practices, while the defence argued that the use of TraceLex was reckless due to a failure to disclose the tool's known technical limitations to the issuing officer.
A central focus in the s 30 analysis was the Supreme Court’s decision in Tamiefuna v R [2025] NZSC 40. In Tamiefuna, Police took and indefinitely retained a photograph of a passenger during a routine, lawful traffic stop, later using it to link him to an unrelated robbery. The Supreme Court excluded the highly probative photograph under s 30, holding that the unlawful retention and database collation infringed s 21 of the NZBORA.
Tamiefuna is a reminder that improperly obtained evidence should not necessarily be admitted simply because it is reliable or because Police acted in good faith. Applied to AI, Tamiefuna could signal that, without careful attention to privacy obligations, long-term data scraping, intelligence retention, and reliance on black-box tools without clear statutory frameworks may constitute systemic impropriety. Such impropriety may be sufficient to warrant exclusion, even when the tool "gets the right person".
However, the s 30 balancing test must be applied in a context-specific way for every case, which can lead to difficulties in translating the outcomes to other cases. The Supreme Court has signalled that a recalibration of the balancing test is best left to Parliament to help address inconsistencies in applying the test.
The Verdicts: A Split Bench
The three High Court mock trials produced distinct rulings that reflect the current state of judicial uncertainty:
Christchurch High Court (Associate Judge Paulsen): Held the Surveillance Device Warrant was invalidly issued. TraceLex did not provide a sufficiently reliable foundation, and the warrant application was deficient and incomplete. On the s 30 balancing exercise, the Court tentatively leaned towards admitting the evidence due to the seriousness of the offending, though noting severe reservations.
Auckland High Court (Blanchard J): Held the warrant was invalidly issued. The probabilistic match did not satisfy the "reasonable belief" threshold required by statute, and Police lacked foundational reliability data. However, under s 30, the Court provisionally admitted the evidence, holding that Police acted in good faith and that the tool ultimately proved accurate.
Wellington High Court (Radich J): Held the warrant was validly issued. The Court analogised TraceLex to traditional technical equipment, ruling that an issuing officer needs to understand the nature and corroboration of the output, not the underlying software code. The Court added that even if invalid, the evidence would be admitted under s 30.
Practical Takeaways for Legal Practitioners
The R v Popper mock trials offer critical lessons for criminal barristers, prosecutors, and legal advisors navigating the emergence of AI-derived evidence:
For Prosecutors & Law Enforcement
While the moot hearings in R v Popper demonstrate that New Zealand courts may currently lean toward admitting highly probative evidence under s 30, law enforcement cannot rely on s 30 as a permanent safety net. As judicial technical literacy matures, opaque "black box" algorithms that bypass statutory safeguards will face increasing skepticism. To maintain an effective and credible justice system, the courts must balance the use of new innovative technologies against fundamental constitutional rights.
Watch the recording of the Auckland Mock Trial on YouTube: (https://www.youtube.com/live/0ofLQSC85hU).
The Mock Trial Document pack, including the intelligence profile, warrant application, and technical expert statement, is available here: https://tinyurl.com/ai-in-court-pack-2025.
Many hands make light work, and the success of the AI in Court events relied upon significant contributions from the many. We acknowledge:
We also acknowledge the AI in Court event hosted by Monash University’s Artificial Intelligence for Law Enforcement and Community Safety (AiLECS) laboratory, which served as inspiration for the mock trials.
This article was drafted with the support of Google Gemini, and the author takes responsibility for the writing and analysis.
The secondary battleground across all three moot hearings was s 30 of the Evidence Act. If the warrant was invalid due to insufficient grounds or lack of candour, should the resulting physical drug evidence be excluded?
The prosecution and defence offered contrasting arguments across four key factors under the s 30(5) balancing test. Regarding probative value and reliability, the prosecution emphasised that the seized MDMA matched the festival pills, confirming TraceLex's accuracy; the defence countered that an opaque, unverified AI process should not be retroactively justified by post-search results. Regarding the seriousness of the offence and the importance of any rights breached, the prosecution highlighted the public harm of commercial Class B drug supply, while the defence contended that the fundamental privacy rights under s 21 of the NZBORA must be upheld regardless of the charge's severity.
The prosecution argued that there was a lack of other investigatory techniques for the untraceable dark web account, whereas the defence submitted that a lack of options does not excuse deploying unvalidated mass surveillance tools. Finally, the prosecution claimed Police acted in good faith based on international practices, while the defence argued that the use of TraceLex was reckless due to a failure to disclose the tool's known technical limitations to the issuing officer.
A central focus in the s 30 analysis was the Supreme Court’s decision in Tamiefuna v R [2025] NZSC 40. In Tamiefuna, Police took and indefinitely retained a photograph of a passenger during a routine, lawful traffic stop, later using it to link him to an unrelated robbery. The Supreme Court excluded the highly probative photograph under s 30, holding that the unlawful retention and database collation infringed s 21 of the NZBORA.
Tamiefuna is a reminder that improperly obtained evidence should not necessarily be admitted simply because it is reliable or because Police acted in good faith. Applied to AI, Tamiefuna could signal that, without careful attention to privacy obligations, long-term data scraping, intelligence retention, and reliance on black-box tools without clear statutory frameworks may constitute systemic impropriety. Such impropriety may be sufficient to warrant exclusion, even when the tool "gets the right person".
However, the s 30 balancing test must be applied in a context-specific way for every case, which can lead to difficulties in translating the outcomes to other cases. The Supreme Court has signalled that a recalibration of the balancing test is best left to Parliament to help address inconsistencies in applying the test.
The Verdicts: A Split Bench
The three High Court mock trials produced distinct rulings that reflect the current state of judicial uncertainty:
Christchurch High Court (Associate Judge Paulsen): Held the Surveillance Device Warrant was invalidly issued. TraceLex did not provide a sufficiently reliable foundation, and the warrant application was deficient and incomplete. On the s 30 balancing exercise, the Court tentatively leaned towards admitting the evidence due to the seriousness of the offending, though noting severe reservations.
Auckland High Court (Blanchard J): Held the warrant was invalidly issued. The probabilistic match did not satisfy the "reasonable belief" threshold required by statute, and Police lacked foundational reliability data. However, under s 30, the Court provisionally admitted the evidence, holding that Police acted in good faith and that the tool ultimately proved accurate.
Wellington High Court (Radich J): Held the warrant was validly issued. The Court analogised TraceLex to traditional technical equipment, ruling that an issuing officer needs to understand the nature and corroboration of the output, not the underlying software code. The Court added that even if invalid, the evidence would be admitted under s 30.
Practical Takeaways for Legal Practitioners
The R v Popper mock trials offer critical lessons for criminal barristers, prosecutors, and legal advisors navigating the emergence of AI-derived evidence:
For Prosecutors & Law Enforcement
- Pre-Deployment Validation is Necessary: Police should not rely on post hoc expert validations commissioned after a legal challenge has been lodged. AI tools must undergo rigorous, independent baseline testing prior to operational deployment.
- Heightened Duty of Candour: Warrant applications relying on AI outputs should explicitly state the tool's error rates, non-linear confidence scores, and limitations. Suppressing technical uncertainty risks invalidating the resulting warrant.
- Meaningful Human Oversight: "Human-in-the-loop" processes must involve trained analysts capable of critically questioning AI outputs, rather than executing a box-ticking exercise.
- Challenge the "Machine Accuracy" Presumption: AI tools are not speed cameras or calculators. Highlighting their non-deterministic, probabilistic nature, and challenging any failure by Police to provide error rates or repeat-query testing, may undermine the apparent reliability of their outputs.
- Interrogate Confidence Scores: Unpack numerical outputs to expose their non-linearity. The court may need education to understand that a "75% confidence score" does not equate to a 75% mathematical probability of guilt.
- Apply Tamiefuna towards open source data collection: Mass data ingestion and retention practices may be challenged under s 21 of the NZBORA 1990. Arguably the State does not have unfettered rights to aggregate and profile personal data even if it is publicly available on the Internet.
While the moot hearings in R v Popper demonstrate that New Zealand courts may currently lean toward admitting highly probative evidence under s 30, law enforcement cannot rely on s 30 as a permanent safety net. As judicial technical literacy matures, opaque "black box" algorithms that bypass statutory safeguards will face increasing skepticism. To maintain an effective and credible justice system, the courts must balance the use of new innovative technologies against fundamental constitutional rights.
Watch the recording of the Auckland Mock Trial on YouTube: (https://www.youtube.com/live/0ofLQSC85hU).
The Mock Trial Document pack, including the intelligence profile, warrant application, and technical expert statement, is available here: https://tinyurl.com/ai-in-court-pack-2025.
Many hands make light work, and the success of the AI in Court events relied upon significant contributions from the many. We acknowledge:
- The judges and the Courts of New Zealand for hosting the events;
- Counsel: Claire Boshier, Kerry Cook, Helen Brown, Jacob Parry, Anselm Williams, and Chris Wilkinson-Smith;
- Student counsel: Olivia Pearce, Helen Houghton, Nikita Gundawar, Yichen (Alex) Yang, Claire Graham, and Lara Hopkinson;
- The Law Association and Meredith Connell for sponsoring parts of the events;
- New Zealand Police for supporting the development of the moot problem;
- Organising committee: James Foubister, Rebecca Thomson, Lloyd Gallagher, Jacob Parry, Matthew Mortimer-Wang, Prof Kris Gledhill, Dr Zoë Prebble, Dr Olivia Erdelyi, Dr Briony Blackmore, and Lewis Mills.
We also acknowledge the AI in Court event hosted by Monash University’s Artificial Intelligence for Law Enforcement and Community Safety (AiLECS) laboratory, which served as inspiration for the mock trials.
This article was drafted with the support of Google Gemini, and the author takes responsibility for the writing and analysis.
No comments:
Post a Comment