Learn · Methods in the open

A minimal model for qualitative text, and how to judge it honestly

You do not need a giant transformer to get signal from interview text. A small, interpretable model can sort responses by theme, and you can read exactly which words drive each code. But a model that classifies is not the same as a model you can rely on. Here is the small model, and how EpiNet tells you whether it holds up.

Why start small

A big model hides its reasoning. A small one shows it. If you represent each document as word counts and train a single linear layer, the model learns one weight per word per class. Inspecting those weights tells you which words push a response toward each code. For a lot of qualitative coding tasks, that is enough signal, and the transparency is worth more than a few points of accuracy you cannot explain.

The pipeline

1. Preprocess. Lowercase the text, strip punctuation and stop-words, and tokenise. Build a small vocabulary from your own corpus.
2. Vectorise. Turn each document into a bag-of-words or TF-IDF vector. TF-IDF (term frequency–inverse document frequency) weights a word up when it is frequent in one document but rare across the rest, so distinctive words count for more.
3. Model. For labelled data, one fully connected layer with a softmax or sigmoid output is enough:

import torch
import torch.nn as nn

class MinimalQualModel(nn.Module):
    def __init__(self, vocab_size, num_classes):
        super().__init__()
        self.linear = nn.Linear(vocab_size, num_classes)
    def forward(self, x):
        return self.linear(x)

4. Train. Fit with cross-entropy loss and stochastic gradient descent. With TF-IDF inputs and a little regularisation, this often competes well on high-level coding tasks.
5. Interpret. Read the weight matrix. Because the model is linear, you can map each weight straight back to a word, which is what makes it transparent. For a bit more nuance without much cost, swap bag-of-words for averaged word embeddings and use a one-hidden-layer network. The principle holds: simplicity buys interpretability, and it is enough for many tasks.

The catch: classifying is not the same as holding up

Your model reports 85% accuracy. Is that real? A small cohort makes it easy to get an impressive number that hides three problems: the data leaked between train and test, the result is no better than chance, or the scores rank documents well but are numerically off. You only find out by evaluating the model honestly, not by trusting the single number.

Judge it with EpiNet

EpiNet does not change your model. It wraps any node-level model in a reproducible evaluation harness and reports calibration, permutation-null tests, and bootstrap confidence intervals instead of one accuracy score. To judge the minimal model:

1. Format the data as a graph. Treat each document or participant as a node with a unique id, the theme label as the outcome, and the word-frequency vector as attributes. If you can define links between documents, such as lexical similarity above a threshold, record them as edges. EpiNet computes graph features like degree and component size from that.
2. Run it. Install vahtian-epinet and point the command line at your CSV files:

epinet \
  --nodes my_nodes.csv --edges my_edges.csv \
  --outcome-column Theme --target-outcome 1 \
  --model logistic_regression \
  --split-strategy community --permutation-test 100 \
  --output-dir model_report

Use --model logistic_regression to mirror your minimal linear model. EpiNet runs the same evaluation either way.

3. Read the outputs. A JSON file reports discrimination (AUROC, AUPRC, balanced accuracy, MCC, F1) and calibration (Brier score, calibration slope and intercept) with bootstrap intervals. A permutation-null test checks whether your score beats chance. The model_card.md summarises it in plain language and flags data-quality warnings. Contestability analysis reports how far each prediction sits from flipping, and which features drive that.
4. Compare against baselines. Because EpiNet runs several estimators under one harness, you can see whether your model's interpretability costs you any discrimination. Community-aware splitting checks that the result is not driven by leakage between linked documents.

Related guides

See also make your analysis code citable for publishing the model with a DOI, and AI qualitative coding, human-reviewed for producing the labels in the first place. More guides are on the Learn page.