With 24,216 raw items collected, the next step was labelling. deepseek-v4.1-flash behind a gateway, one fixed taxonomy of 22 problem labels and 15 workaround labels, each item returning: one problem label, up to three workaround labels, a tool name, a speaker role, severity 1–5, and a verbatim quote of at most 180 characters. Batches of 15 items, temperature 0.1.
Real numbers per step
- Filter before spending tokens: drop items under 110 characters (Stack Exchange titles 40), always keep 1–2 star reviews, and require every other item to match a signal-word list. 24,216 → 11,922 candidates, and 9,318 after dedup on the first 160 characters.
- 11,922 candidates at 15 per batch is 795 calls. The log shows a single run processing all 960 items, and 13 overlapping runs in total.
- 18,529 result rows, of which 7,450 carry a label inside the taxonomy. The other 60%: the model returned null (not a work complaint) or returned a label that does not exist.
- 1,903 rows are byte-identical to another row. After dedup on (source, first 130 characters of the quote): 6,672 rows.
- 778 of those are the same item classified several times with different labels — no majority label available to pick.
- Important note: resuming worked by skipping rows whose quote was already seen, and the model returned an empty quote about a quarter of the time (4,745 of 18,529). So the dedup threshold depends on whether the model produced a quote, not on the data.
Four silent failures, not one exception raised
- The ways field came back as an object instead of a string: 29 times across the corpus. 26 of those objects contained no code at all, and once a problem code landed in the workaround field. The reader handles both shapes, so nothing crashed — data just quietly disappeared.
- Workaround labels outside the taxonomy: 5 rows, including one still carrying a leading “>”.
- Valid spread: all 22 problem labels were used at least 39 times, the top one 1,162 times. No sign of the model collapsing onto a single label — the scale is usable.
- Severity behaves: mean 3.31, standard deviation 0.89, range 1–5, no out-of-range values.
The most valuable part turned out not to be the labels
59.7% of items carried no problem label. At first glance that is waste: 11,922 candidates paid for, only 7,450 labelled. But most of those nulls are genuine — how-do-I questions, achievements, ads, five-star reviews. The keyword filter screens on form only, and the LLM was the only step that could answer “is this a complaint at all”. Swap it for a tighter filter and you lose more real items than you save in tokens.
One caveat: token spend was never measured. The script counts characters of what it stored, but the payload it sent (system prompt plus items truncated at 1,100 characters each) is much larger, and the gateway in use does not log usage — so any figure like “1,500 in plus 800 out per call” is an inference, not a measurement.
The full pipeline, with self-checks for the failures above, is in the 8-point checklist; the lessons that only appear once the whole corpus is assembled are in Four things you only see with 20k items.