What happens when a data analyst is handed a qualitative study

I am comfortable with math. Give me a model, a logic, a set of rules, and I can follow it anywhere. So when I was asked to supervise a qualitative study, in a field that was not mine, my first instinct was to look for the logic. Where is the structure here?

It was there the whole time. Qualitative research has a methodology you can follow, an argument you can trace, rules that hold. That surprised me, and it should not have. The logic was never the problem.

The words were.

Qualitative papers are often written in what I think of as grade-25 English: exclusive terms, non-intuitive, assumed rather than explained. The reasoning underneath is clear once you have the vocabulary. Without it you stall on the first sentence, not because the idea is hard, but because the word is a gate.

The word was the gate

Here is a small example that shows the whole problem.

Look at what happened. The idea was ordinary: a patient did not trust their doctor. But the field already had a precise term for it, with a definition and a literature behind it. Knowing the words "medical mistrust" is the difference between a vague note and a coded, defensible theme. The concept was never hard. The word was the gate.

This is the part that surprised me most, and the part I keep coming back to. A capable person can stall in qualitative work for a reason that has nothing to do with their thinking. They stall on vocabulary they were never handed. That is not a knowledge problem. It is an access problem, and access problems can be fixed.

How I found my way in

Because the field was not mine, I did what I always do with unfamiliar territory. I researched the topic, the existing tools, the methodologies people used, the keywords they returned to. I built comparison tables and timelines so I could see the scientific argumentation laid out in front of me, and I coded new tools for the team.

Tables and timelines are how an outsider makes an unfamiliar logic followable. You put the competing positions side by side, you line up when each idea arrived and what it answered, and the structure that was hidden in the prose becomes something you can point at. If you have used the Vahtian Learn timelines, you have seen the same instinct: the argument, drawn out where you can check it.

Then I asked my real question: would it survive stress testing?

Once I could follow the methodology, I looked at it the way I look at any model. Would it survive stress testing? Would it hold up when someone tried to break it?

An unstructured qualitative workflow with a single train/test split and a headline accuracy would not. It has to anticipate what a sceptical reviewer, or an audit tool like EpiNet, will probe, and it has to be designed for that from the start. The framework below is what I arrived at. It combines transparent data handling, honest evaluation, and reflexivity about my own position.

1. Specify the epistemic stance

1. Define the knowledge claim. Are you building a theory (inductive), testing one (deductive), or inferring the most plausible explanation (abductive)? Say which in the protocol, and make sure your sampling, coding, and analysis all match it.
2. Document your position. Record your relationship to the participants and the field, how it might shape your reading, and how you plan to keep that in check.
3. Align the words with the claim. Do not borrow "code saturation" from inductive coding when your analysis is reflexive thematic analysis. The term has to belong to the method.

2. Collect and format the data

1. Sample on purpose. Use purposeful or theoretical sampling that fits your stance, and keep a sampling matrix and recruitment log: who was approached, why they fit, and how you judged saturation or information power.
2. Represent each case as a node. One interview or document per node, with an identifier, the theme labels you intend to classify, and any attributes such as demographics or word-frequency vectors.
3. Annotate any edges. If cases are related, for example participants from the same site or a similarity above a threshold, record that in an edge table and say what the edge means. This matters because EpiNet applies a graph-semantics gate: a similarity graph can support clustering, but it cannot justify a causal claim about who influenced whom.

3. Pre-process and build features

1. Text features. Compute bag-of-words or TF-IDF vectors, reduce them by term filtering or embedding averaging, and combine them with any domain attributes.
2. One pipeline, versioned. Apply the same preprocessing to every case, and write the version down.
3. Record feature provenance. Keep a file describing how each feature was made, so the result can be reproduced.

4. Evaluate the model honestly

This is the step a reviewer will push on hardest, so it is the one to build most carefully.

1. Many splits, not one. A single train/test split can swing wildly depending on which cases land in the test set. Run multiple random splits (EpiNet defaults to ten) and report the mean, standard deviation, minimum, and maximum of each metric.
2. Tune once. Pick logistic regression for interpretability or a random forest for a flexible baseline, use imbalance-aware settings, and select on balanced accuracy, not overall accuracy.
3. Calibrate, and show your working. Tree models are often over-confident. Re-fit in a calibration wrapper, keep the calibrated probabilities only if they improve the held-out Brier score, and report both the raw and calibrated slope and intercept so the decision is auditable.
4. Test against chance. Shuffle the outcome labels many times and re-run the whole evaluation. If the real scores sit inside that null distribution, the features carry little signal, and you need to say so.
5. Split by community. If your data have edges, random splits leak information between train and test. Keep connected cases in the same fold. Performance usually drops, and that lower number is the honest estimate of how well the model travels to a new community.
6. Record provenance. Software versions, seeds, hyperparameters, and splits, stored in a provenance file inside the model bundle.

5. Take contestability seriously

1. Find the borderline cases. Measure each case's distance to the class centroids and flag the ones near a boundary. These are the interviews where the classification is genuinely uncertain.
2. Leave the grey zones grey. For contested cases, do not force a strong theme. Present them as ambiguous, and note what would resolve them, such as more context or a follow-up interview. More algorithms or hand-set rules will not rescue a case where the features simply do not carry the information.

6. Report with uncertainty, and reflect

1. Metrics with intervals. Report discrimination, classification, and calibration with bootstrap confidence intervals and the spread across splits. Do not over-read a single point estimate.
2. Explain null results. If the permutation test is not significant, state plainly that the model does not detect a reliable signal.
3. Provide a model card. Summarise the model, data, evaluation, assumptions, limits, and appropriate use. Say clearly that it is a demonstrator, not a clinical or policy decision tool.
4. Reflect on your role. Bring the epistemic stance from step 1 into the discussion, and explain how your position and your choices shaped the coding and the evaluation.
5. Share data responsibly. De-identify transcripts and graph structures, get consent before sharing anything raw, and think hard about the ethics of revealing edges drawn from participant interaction.

7. Plan the next iteration

1. Make it reproducible. Publish the dataset, code, and evaluation scripts, with an environment description so others can re-run the stress test.
2. Change the representation, not the score. If the model fails under permutation or community splits, revisit the features or the theme structure. Do not chase a higher number.
3. Keep a human in the loop. Treat the model as a junior assistant, not an oracle. Review its suggestions, decide what to accept or reject, and write down why.

What the framework buys you

Follow this and you can show a reviewer, or an auditor, that both the model and the qualitative coding underneath it have been stress tested. Multiple splits, calibration, permutation nulls, and community-aware evaluation expose overfitting and chance performance. Contestability analysis names the borderline cases instead of hiding them. Transparent sampling and reflexive reporting keep your own hand visible. Together they position the findings as auditable, credible, and open to contestation, rather than presented as infallible.

The logic of rigour is learnable. The vocabulary should not gatekeep it.

That is the whole reason these tools exist. MethodVahti keeps your stated method, your terminology, and your written claims aligned from protocol to publication, and flags the mismatches that a reviewer would catch. EpiNet runs the honest evaluation in step 4 and writes the model card. Neither one decides what your data mean. That was always going to be the human's job, whether the human trained as a data analyst or grew up inside qualitative research.

I came to this field from the outside, through tables and timelines and a stubborn need to find the logic. The logic was there. Once I could read the words, I could see it. My hope is that these tools hand the words over sooner, so the next person who is handed a study they did not train for can get straight to the thinking.

Heidi