TL;DR — Key Takeaways
- AI models do not determine “correctness” for themselves; that standard is built through thousands of human annotation decisions, particularly when real-world data exposes gaps in existing labeling guidelines.
- Edge cases drive the evolution of training data: Annotators flag ambiguous examples, rules are clarified, affected data is relabeled and quality-checked, and those decisions ultimately influence what models learn.
- Automation works best where the standard is already clear. When the definition of a correct answer is still being negotiated, human judgment remains essential — and unresolved annotation decisions can accumulate into a form of data debt.
A model doesn’t have an innate sense of right or wrong. It has a dataset, which is simply a collection of human decisions gathered during years of training: What belongs inside a bounding box, what a label actually implies, and which edge cases even count as edge cases. Annotation gets treated like a boring step before the real machine learning starts. It’s actually the process that decides what the model treats as correct in the first place.
I spent time as an annotator on a large sports video project: Labeling player actions frame by frame, football and volleyball both, alongside 200+ other annotators on the same site. Most of the job was sitting right where the guidelines met actual broadcast footage and figuring out, case by case, what didn’t fit yet. The modeling side of things, architecture, thresholds, all of it, sat with the data science team, not with us.
That’s what this is about. Not architecture. Just what it actually feels like to sit inside a labeling pipeline, and how the small, unglamorous calls annotators make quietly end up deciding what a model learns to treat as correct.
The Problem With Messy Data
Most of the attention on a project like this goes to model architecture and tuning hyperparameters. Annotation gets waved through as repetitive labeling work — something to get past on the way to the interesting part. That assumption holds up fine right up until real data shows up.
The aim, eventually, was a model that could recognize player actions without help. Getting there meant training it on annotated data, about 20,000 labeled points a month per site, consistent enough that the pattern could actually stick.
Broadcast footage doesn’t cooperate. Cameras cut mid-action. Replays get spliced into live footage without warning. Two annotators can watch the same three-second clip and land on completely different answers for where an action starts and ends — the exact kind of disagreement inter-annotator agreement scores exist to catch. The guidelines covered the obvious cases fine. They didn’t cover everything. Nothing does on a first pass. That gap is where the job actually lived.
What the Work Actually Looked Like
Labeling sports plays, according to whatever schema I’d been trained on that cycle: Mark where an action starts and ends, tag the context, apply the rules the same way across clip after clip, day after day. Not a great dinner-table story. Somebody has to watch a replay 11 times just to agree on when a tackle starts.
Once the data left my hands, I lost the thread. Which model it went to? How confidence got scored? None of that was visible from where I sat. What I had instead was proximity. I was looking at the raw footage at the exact moment it stopped being clean, before anyone’s model ever saw it. A play, a replay transition, a camera angle that didn’t match anything in the guidelines. Guessing wasn’t an option there. I flagged it.
When the Rules Run Out
Edge cases were the important part, not because they were technically hard — they weren’t — but because each one was a small decision about what ‘correct’ would mean from then on.
Roughly, it went like this. Someone annotates a new batch against the current guidelines, and hits something those guidelines don’t cover. Instead of guessing, they raise it. The client looks it over and decides what to do, the guidelines get updated to match and annotators reapply the new rule wherever it’s needed. Quality checks the re-annotated work, and only then does the approved dataset ship to data science.
Multiply that across hundreds of annotators and thousands of clips a month and you start to see how ‘ground truth’ on a project like this actually gets made.
Not a spec written once at the start.
An accumulation, one resolved judgment call at a time, each one narrowing the schema a little further.
The figure below defines the eight-step loop that turns one flagged case into an updated rule for every annotator on the project. The client and quality team steps are the only ones outside the author’s direct role.
The Bigger Loop This Fed Into
Zoomed out, my part was one piece of a bigger system: Initial labeling from AI, then human annotation, then periodic retraining, the kind of thing usually called model-assisted labeling (MAL). Spend long enough inside it and the shape of it becomes obvious.
Roughly: The model predicts on new data, and depending on how confident it is, some of it goes straight to a quick check while the rest gets kicked up to a closer human review. That’s confidence-based routing, more or less, the same mechanism most active learning setups run on.
Senior annotators picked up whatever got flagged as uncertain. If one of those turned out to be a real edge case rather than a routine correction, that’s when the escalation process above kicked in: Flag it, get it resolved with the client, update the guidelines, re-annotate.
Once a batch of edge cases got sorted and the affected data relabeled, it went to the data science team as part of the next retraining round.
The specific points landed wherever the rotation put them, meaning I could easily relabel the same clip months later without ever clocking it. What I could see was the flow. Guidelines were never really finished. This month’s dataset wasn’t last month’s.
The following figure illustrates the wider model-assisted labeling workflow the author’s annotation work fed into. The green lane is the author’s direct experience; the blue and coral lanes reflect work handled by the AI system and the data science/ML engineering team.
What’s Actually Worth Automating
Across a bunch of different annotation projects- audio, image, video, text, details withheld for client confidentiality- the same pattern kept surfacing. Automation held up fine wherever the answer was already clear. It fell apart the moment anything needed real interpretation.
A first-pass transcription, a rough bounding box — a model can attempt those fine, because humans already agreed on what ‘correct’ looks like going in. Anything that needed a judgment call, weighing subjective quality, comparing outputs against several criteria at once, stayed human.
Not because the tech couldn’t be pointed at the problem, but because the standard itself hadn’t been nailed down yet. Ask a model for a judgment call and it’ll hand one back instantly, stated with total confidence and absolutely no reason to trust it. You can’t automate consistency around a rule that doesn’t exist yet.
Data Debt is Real, and it Isn’t Abstract
Annotation guidelines are never finished on day one. That has a cost. Good luck putting a number on it, though.
Every new camera transition, every unfamiliar play type, meant a new rule to write down, a round of re-annotation for whatever it touched and a quality check to make sure the fix actually held. What that cost in hours or accuracy, I genuinely can’t say. What I can say is that it was constant, not occasional. Staying current with shifting guidelines meant relearning parts of the rule set on a regular basis, and knowing which guideline version applied to which batch mattered more than it probably should have.
It’s basically the annotation version of tech debt: Unresolved edge cases pile up as gaps in the schema documentation, and eventually someone has to pay that down by re-annotating, by reconciling. Nobody schedules time for it on a roadmap. It just becomes next month’s problem, quietly.
Final Thoughts and Where This Leaves Things
None of the models downstream of this ever learned ‘correctness’ on their own. They learned it from a long chain of small decisions: An annotator flagging something the guidelines missed, a client deciding how to handle it, a rule getting rewritten, a batch getting relabeled. None of that is glamorous. Most of it is invisible in whatever the finished model eventually outputs. Nobody is making a documentary about the annotator who caught a weird replay-transition bug back in March. But it’s the actual mechanism, underneath everything else.
The architecture matters. The accuracy numbers matter. But the standard those numbers get measured against- what even counts as a correct label to begin with- that gets built earlier, by people sitting closer to the raw data than any model ever gets to.
