Preprocessing before you send data to AI: 5 steps, with numbers

In a classification pipeline that actually ran — 24,216 raw items, 11,922 candidates sent to the model, 18,529 result rows, 6,672 usable rows — most failures came from the data going in and from never validating the data coming out, not from the model. Five steps, each with a measured number.

1. Cut with code before spending tokens

A keyword filter dropped 50.8% of raw items (24,216 → 11,922 candidates) without a single model call. It screens on form, not meaning, so plenty gets through — and that is correct, see step 4.
The payload truncated each item at 1,100 characters, but the stored record kept no original length, so the effect of that cut cannot be verified. If you cut, record what you cut.

2. Normalise before matching

The first resume key was the first 150 characters of the text: 1,903 colliding rows. The key has to be the item's id, not its prose.
518 groups shared a (source, first 160 characters) key while differing in quote — the same item classified repeatedly, one group seven times, with no majority label to select automatically. Normalising and deduplicating is code's job, not the model's.

3. Keep the vocabulary in code, not in the model's imagination

The taxonomy is fixed and lives in code, and every label outside it is rejected: 60% of rows carried a label outside the taxonomy or none at all. Filtering on “has a label” alone is enough to corrupt the ranking from the start.

4. Let the model return null, and accept it

59.7% of items carried no problem label, and most of those nulls are genuine: how-do-I questions, achievements, ads, five-star reviews.
A null is not a bug to fix. Forcing a label on every item manufactures wrong labels, and a wrong label costs far more than a null.

5. Validate the output before anything uses it

29 times the workaround field came back as an object instead of a string. The JSON parsed, the code kept running, the data was lost. Type-check every field, not just whether it parsed.
18,529 result rows, only 6,672 usable: the rest did not vanish, it sits in the genuine-null group and the off-taxonomy group.
Real token counts were never logged at the gateway, so every cost figure is an estimate — and with an estimate you cannot tell which step is the expensive one.

The full eight-point checklist and the classification context are in An 8-point checklist for LLM-powered data pipelines and Classifying 6,672 complaints with an LLM.

1. Cut with code before spending tokens

A keyword filter dropped 50.8% of raw items (24,216 → 11,922 candidates) without a single model call. It screens on form, not meaning, so plenty gets through — and that is correct, see step 4.

The payload truncated each item at 1,100 characters, but the stored record kept no original length, so the effect of that cut cannot be verified. If you cut, record what you cut.

2. Normalise before matching

The first resume key was the first 150 characters of the text: 1,903 colliding rows. The key has to be the item's id, not its prose.

518 groups shared a (source, first 160 characters) key while differing in quote — the same item classified repeatedly, one group seven times, with no majority label to select automatically. Normalising and deduplicating is code's job, not the model's.

5. Validate the output before anything uses it

29 times the workaround field came back as an object instead of a string. The JSON parsed, the code kept running, the data was lost. Type-check every field, not just whether it parsed.

18,529 result rows, only 6,672 usable: the rest did not vanish, it sits in the genuine-null group and the off-taxonomy group.

Real token counts were never logged at the gateway, so every cost figure is an estimate — and with an estimate you cannot tell which step is the expensive one.