An 8-point checklist for LLM-powered data pipelines

Drawn from a pipeline that actually ran: 24,216 raw items, 11,922 candidates, 13 classification runs, 18,529 result rows, 6,672 usable rows. Each item below maps to a failure that happened, with how to catch it.

Against double counting

1. The resume key must be stable and unique — the item's id, not its text. The first version keyed on the first 150 characters of the text and collided on 1,903 rows.
2. Check for duplicates at both row and content level. The corpus holds 518 groups sharing a (source, first 160 characters) key but differing in quote — the same item classified repeatedly with different labels, 11,595 rows in total, with no majority label to select automatically.
3. Store the source item id in the result. This version only stored a content key, so a conflicting label cannot be traced back to the raw item to adjudicate.

Against silent loss

4. Type-check every field in the response, not just whether the JSON parsed. 29 times the workaround field came back as an object instead of a string; the code kept running and the data was still lost.
5. Treat every label as untrusted until it matches the taxonomy. 60% of rows carried a label outside the taxonomy or none at all; filtering on “has a label” alone would have corrupted the ranking.

Keeping it interpretable

6. Record what was truncated. The payload cut each item at 1,100 characters, but the stored record kept no original length — so the effect of that cut cannot be verified.
7. Store a collection timestamp per item. Without it, every question about trend direction is unanswerable even though the data is in hand.

Measurable

8. Log the real input and output token counts at the gateway. Skip this and every cost figure is an estimate, with no way to know which step is the expensive one.

Three things that need no check

Label distribution behaves: 22 labels, the least used 27 times, the most used 1,162. No extra check needed.
Severity: mean 3.31, standard deviation 0.89, range 1–5. Directly usable.
Retry on broken JSON: split the batch in half and try again, drop it if that fails too. Simple, and it does not need a heavier mechanism.

Applying this to a real pipeline: collecting the data and classifying it with an LLM.

Against double counting

1. The resume key must be stable and unique — the item's id, not its text. The first version keyed on the first 150 characters of the text and collided on 1,903 rows.

2. Check for duplicates at both row and content level. The corpus holds 518 groups sharing a (source, first 160 characters) key but differing in quote — the same item classified repeatedly with different labels, 11,595 rows in total, with no majority label to select automatically.

3. Store the source item id in the result. This version only stored a content key, so a conflicting label cannot be traced back to the raw item to adjudicate.

Against silent loss

4. Type-check every field in the response, not just whether the JSON parsed. 29 times the workaround field came back as an object instead of a string; the code kept running and the data was still lost.

5. Treat every label as untrusted until it matches the taxonomy. 60% of rows carried a label outside the taxonomy or none at all; filtering on “has a label” alone would have corrupted the ranking.

Keeping it interpretable

6. Record what was truncated. The payload cut each item at 1,100 characters, but the stored record kept no original length — so the effect of that cut cannot be verified.

7. Store a collection timestamp per item. Without it, every question about trend direction is unanswerable even though the data is in hand.

Three things that need no check

Label distribution behaves: 22 labels, the least used 27 times, the most used 1,162. No extra check needed.

Severity: mean 3.31, standard deviation 0.89, range 1–5. Directly usable.

Retry on broken JSON: split the batch in half and try again, drop it if that fails too. Simple, and it does not need a heavier mechanism.