Chain-of-custody for AI-assisted analyses

Rerun is not reconstruct

Traditional reproducibility asks one question: can the analysis be rerun? AI-assisted analysis adds a second: can it be reconstructed? Which code did the model write, and from what prompt? Why was that variable transformed? Was the sensitivity analysis your idea or a suggestion you accepted? Does Table 1 still come from the current script?

It is not carelessness. Nothing in an ordinary workflow records any of this — the prompts sit in one tool, the code in another, the decisions nowhere. So reconstruction falls to memory, months later, under a reviewer deadline. Everyone who works this way hits it.

The method: chain of custody, not AI detection

Detecting AI-written code after the fact is weak evidence and answers the wrong question. What a reader needs is the same thing a lab notebook gives a bench scientist — a contemporaneous record: declare who or what wrote each piece of code when it enters the project, write down each analytic decision and its reason at the moment it is made, fingerprint every execution, and tie each manuscript number to the run that produced it.

Kept as an append-only log where every entry carries the hash of the previous one, the record becomes tamper-evident: anything edited, deleted, or reordered after the fact breaks the chain visibly from that point forward. You can adopt this discipline with any text file. AuditLite just makes it one command per event.

What the trail records

• Provenance — which code was human-written, AI-generated, or AI-modified; the model, version, prompt, the source files it saw, and any external tools or retrieval. Declared, never detected.
• Decision trace — why a variable, model, exclusion, transformation, or sensitivity analysis was chosen; which AI suggestions were accepted, edited, or rejected; and whether the AI introduced a new analytic decision rather than implementing yours.
• Execution trace — the exact script hash, command, interpreter and package versions, declared seeds, and input and output file hashes for every recorded run.
• Manuscript claims — each table or figure tied to the run and file it came from, so "does the manuscript still match the analysis" is a check, not a memory exercise.
• Deviations — departures from the analysis plan, recorded with the reason.

The checks Seven checks; every answer is three-way, never a coin flip

auditlite check compares the trail against the working tree: is the chain intact, were scripts edited after their last recorded run, does the data still match what the runs consumed, were outputs modified outside a recorded run, do manuscript tables still match their files, did any script run without a declared origin, does every decision carry a rationale?

AuditLite is not an AI detector, and an intact chain is not a truth claim. Provenance entries are declarations; missing declarations are flagged, not guessed. The chain check shows the trail has not been rewritten after the fact — not that the recorded events are accurate, and not that the analysis is correct. The checks record support for reconstruction, nothing more.

Install

# Python (≥ 3.10, no dependencies):
pip install auditlite

# or the protocol-identical R package:
remotes::install_github("heidihelena/auditlite", subdir = "r")

Trails are cross-compatible, hash for hash: a trail written in Python checks cleanly in R and vice versa, so a mixed Python/R project keeps one trail. The trail is plain JSON Lines in audit/trail.jsonl — local-first, no server, no account; commit it next to the analysis and it travels with the project.

The workflow in one block

auditlite init --project lung-registry-survival

# declare provenance when code enters the project
auditlite provenance analysis.py --origin ai_generated \
    --model claude-sonnet-5 --prompt "primary survival analysis per SAP v2"

# trace analytic decisions as they happen
auditlite decision --what exclusion --choice "drop ECOG 4" \
    --why "n=3 in stratum, prespecified minimum is 10" \
    --source ai_suggested --disposition edited

# run through auditlite so the execution is fingerprinted
auditlite run --input data.csv --output table1.csv \
    -- python analysis.py

# tie manuscript numbers to the run that produced them
auditlite claim table1 table1.csv

# before submission (or in CI)
auditlite check
auditlite report   # writes audit/AUDIT.md — the reconstruction report

The same API is available from Python (auditlite.init(), trail.decision(), …) and from R (audit_init(), audit_decision(), …).

Three questions, three tools

• RecoverLite checks whether a planned method can recover its target — before data collection.
• AssessLite checks whether a conclusion survives plausible assumptions.
• AuditLite checks whether the entire analytic chain can be reconstructed and defended.

Want the audit set up on your project?

AuditLite is open source — adopt it yourself in an afternoon. If you'd rather have the trail designed around your analysis plan — what to declare, which decisions to trace, how to wire the checks into your workflow before submission — ForskAI, independent research-methods support from Vahtian, can do it with you. You keep the decisions; the audit trail comes with it.

Set it up with ForskAI → Or adopt it yourself →

Questions