Pre-data recovery tests for planned study designs

Why power analysis isn't enough

Sampling, measurement, missingness, assignment, and analysis can yield estimates that are biased, poorly calibrated, unstable, or exaggerated conditional on detection. Attrition and exclusions can change the estimand itself: a study may retain precision while quietly answering a different question. Conventional power analysis rarely diagnoses these failures.

The recovery test does. You declare the estimand, data-generating assumptions, data strategy, missing-data process, and analysis strategy before simulation. Data are generated under a crossed scenario grid — null and target effects, each under declared and pessimistically perturbed assumptions — the planned analysis is applied to every simulated dataset, and the diagnosands become a verdict.

The verdict

The report converts the diagnosands into a PASS / RISK / FAIL verdict under a pre-specified, versioned threshold profile — a decision convention, not a validity classification — and recomputes it under the shipped strict and lenient profiles. A verdict that flips across profiles is itself a finding.

A PASS is evidence about the instrument, not about the world. RecoverLite checks whether your planned design and analysis can recover a declared estimand under stated assumptions. It does not validate the assumptions, and it is a prototype implementation of a working-draft protocol (Andersén, 2026).

Install

# from the r-universe (recommended):
install.packages("recoverlite",
                 repos = c("https://heidihelena.r-universe.dev",
                           "https://cloud.r-project.org"))

# or the development version from GitHub:
remotes::install_github("heidihelena/recoverlite")

Not yet on CRAN. Design declaration and data generation follow the DeclareDesign grammar; mixed-model answer strategies use lme4 with lmerTest (Satterthwaite) and pbkrtest (Kenward–Roger).

The workflow in one block

library(recoverlite)

design <- declare_recovery(
  target = target_estimand(
    estimand = "ITT mean difference at 12 weeks",
    scale    = "latent-outcome standardized mean difference",
    sesoi    = 0.40
  ),
  data_strategy   = two_arm_trial(n_per_arm = 115, allocation = 0.5),
  measurement     = measured_outcome(reliability = 0.70),
  missingness     = attrition_model(rate = 0.15, mechanism = "differential"),
  answer_strategy = planned_analysis(
    estimator = "linear_model",
    formula   = y_observed ~ treatment
  )
)

result <- recovery_test(design, sims = 2000,
                        scenarios = "confirmatory_grid", seed = 1)
verdict(result)   # PASS / RISK / FAIL under the selected profile
report(result)    # standalone recovery report; always travels with the verdict

Cluster-randomized designs use cluster_trial() with a mixed-model or cluster-level answer strategy and an explicit inference method — with few clusters, the inference method is not a detail, it is the design.

What the protocol fixes before results are known

• A crossed scenario grid — null-declared, null-pessimistic, target-declared, target-pessimistic. The null rows are required verdict rows, so an analysis that misbehaves under realistic nuisance shows up in false-positive behaviour, not only in power.
• Estimand drift as a diagnosand — target bias decomposes into estimator bias plus estimand drift (what the data strategy does to the question). An unbiased estimator aimed at a displaced contrast is a design problem, and resources repair precision, not drift.
• Pessimistic values by an evidence hierarchy — empirical > prior-study > elicited > package defaults, each labelled with its tier. The target effect is never shrunk automatically; effect-size and nuisance fragility are separate curves, outside the verdict.
• A classified failure taxonomy — fatal errors, nonconvergence, degenerate fits, and diagnostic warnings reported separately; whether degenerate fits count is pre-specified.
• Monte Carlo uncertainty in the verdict — every diagnosand carries an MCSE; a margin within 2 MCSE of its threshold caps the verdict at RISK.
• Threshold profiles, not thresholds — shipped lenient / default / strict profiles; the report shows the signed margin to every threshold and recomputes the verdict under strict and lenient.

Scope of the current prototype

Supported: two-arm parallel trials with an observed baseline, classical additive measurement error, MCAR or baseline-dependent (MAR) differential attrition, optional one-sided noncompliance; complete-case linear model and baseline-adjusted multiple imputation; cluster-randomized parallel trials with random-intercept mixed models (Wald z / Satterthwaite / Kenward–Roger) and the cluster-level t-test.

Not yet supported: crossed and longitudinal random-effects structures, Bayesian answer strategies, prediction models, and latent-variable measurement models.

For preregistrations and grants

A reusable methods sentence ships with the package:

"Design feasibility was evaluated using a pre-data recovery test, in which the planned design and analysis were simulated under declared assumptions and pessimistic perturbations to assess power, bias, coverage, precision, and model stability."

Cite via citation("recoverlite"). Andersén, H. H. (2026). Recovery before data: pre-data simulation diagnosis of planned study designs. Working paper; preprint forthcoming. Versioned releases are archived on Zenodo (doi:10.5281/zenodo.21195920).

Questions