M3: The Ball Gets a Vote
That clip up top: a single point charting itself live. A box tracks the ball, a second box flashes onto whichever player the software thinks just struck it, and a line of notation builds along the bottom, shot by shot. Late in the clip, watch that second box drift off the court and land on a spectator — hang onto that, it matters later.
Freeze a rally at the instant of contact, and sometimes you genuinely can't tell who just hit the ball — it's caught in the air, equally far from both players.
This project is teaching software to watch broadcast tennis and write out the same shot-by-shot notation a human volunteer hand-writes for the Match Charting Project (cv-01). The last entry solved the pipeline's final manual step — the serve toss rises alone above the server's head, so the software now finds and boxes the game ball itself — and produced the first complete points, start to finish. But the column recording who struck each shot broke tennis's own rule that hitters alternate, whenever the ball was caught mid-air at contact, equally far from both players.
A vote, not a guess
Tennis already has a model for resolving exactly this kind of disagreement: a chair umpire weighing a disputed call. No single witness gets trusted blindly — an electronic call, a line judge's shout, the mark on the court can each say something different — and the umpire decides which to believe, not just which spoke first.
That's the idea that replaced "whoever's closest." It leans on the sport's strongest fact: hitters alternate. A rally has exactly two possible stories — near player struck first, or far player did — so instead of deciding shot by shot, the software decides once per rally, scoring both stories against independent evidence and keeping whichever one wins. That's a weighted vote: several independent signals, each trusted by a different amount, added together into one decision.
The alternating pattern itself is allowed to bend in exactly one place: if the ball vanishes from the tracker for a stretch of frames, a hit could be hiding inside that gap, undetected — and only there can the near-far-near-far pattern skip a beat.
Proximity fails, twice
The first fix tried was the obvious one: whoever's closest to the ball at contact hit it. It failed immediately — right at the instant a hit is detected, the ball is often out in open court, closer to neither player than the other.
The next idea reached for physics instead of position: after a real hit, the ball should move away from whoever struck it. Converted into court coordinates, that direction came back backwards after hits from both ends. The culprit is the homography — the recipe that turns a spot on the TV screen into a spot on the court diagram, more on it in Court Vision in Plain English — which only tells the truth for things touching the ground. It's been lying about anything airborne since the very first post that built it; bounces happen on the ground, so they were always honest, and nobody had noticed the lie until this rule leaned on it.
The evidence, weighted
Three witnesses get called per rally, in order of trust. Touch counts most: did the ball reach exactly one player's own patch of court during the contact window? The serve call counts less — a starting hunch, not gospel. Which half the ball landed in counts least: a bounce detector — the "speed suddenly collapses" signature from Court Vision in Plain English — only watches the far half of the court right now, so it can vote "the near player hit it" but not the other way round yet. Half a witness, but an honest one. Every vote that loses stays on the record instead of getting erased — a built-in honesty check, so a witness wrong more often than it should be can't quietly hide its bad calls.
Making the serve call outvotable paid off the first time it mattered. In one point, the serve detector was confident the point started at the near end; touch and landing evidence together pointed far; the vote won. A frame-by-frame check of the footage sided with the vote.
A letter only counts if the ball got there
Once a rally's pattern is settled, proximity becomes trustworthy again for a narrower job: pinning down the exact contact frame, now that it's known which player to look near. But the stroke-type letter — forehand or backhand — only gets written down if the ball demonstrably reached that player's own patch of court, not just "somewhere nearby." That patch scales with how big the player looks on screen: the far player is only about 55 pixels tall, so one fixed-size box for both would reject nearly every far-end letter before it had a chance.
Tracking boxes also go rogue at the worst moments — one wandered onto a spectator standing by a Perrier cooler, another landed on a court shadow instead of a person. That's the box that drifted off-court in the clip up top. Whenever that happens, the letter stays a question mark — the same "declined to guess" mark explained in the notation legend — rather than a wrong one.
Read with that legend, the clip's string is: a serve down the T (s6);
a forehand toward one side (f1); a forehand where the stroke is
certain but the side isn't (f?); a backhand toward the other side
(b3); a backhand with the side unresolved (b?); a shot nobody's sure
who hit, though the ball's own path still says the other side (?3); a
forehand back up the middle (f2); and an ending not written yet, so
it's ? too. Spot-checked against video, the committed letters held up.
Winning the reel before spending a cent
Before any paid tracking ran, the plan got a free review: every candidate starting box went into one grid of still images, eyeballed by hand. That alone killed most of the backup ball-finder's picks — the method that kicks in with no confident serve toss, chasing "the fastest small thing moving," fooled by lens glare, shadow edges, court-line junctions, the net band, a FedEx cooler, even a ballkid. Of 29 backup candidates reviewed, only 3 survived. Three points moving in a straight line isn't proof of a flying ball, especially mid-camera-pan — a background object can fake that chain too.
The same pass fixed the main method for free. The search box for the toss above a server's head used to be a fixed size, so on far-end serves it was searching well above a player barely 55 pixels tall — effectively searching the crowd. Scaling that box to the player's own size, plus a filter dropping racket-shaped blobs, recovered seven far-end tosses the old search had thrown into the stands.
23 clips went to the paid tracker; 18 came back usable. Five were lost outright — SAM, the promptable tracker from Court Vision in Plain English, locking onto a stuck box instead of the ball, including every borderline case that had looked "good enough" on the free review. The eyeball review, not any automatic score, was the real ground truth.
The scoreboard
| What got measured | This session | What "good" looks like |
|---|---|---|
| Full points charted | 21, up from 5 before this session | Every point in the reel, eventually |
| Serve-toss ball-finder | 18 of 25 tries, 72% | This is the primary method — most tries should land |
| Backup ball-finder, after free review | 3 of 29 candidates, about 1 in 10 | A rare fallback, not a real plan |
| Clips sent to the paid tracker | 23 sent, 18 came back usable | Every clip returns a clean track |
Both serve codes finally show up correctly on far-end serves — six of
them — s5 (to the body) and s6 (down the T), the zone
codes that had existed in the code for a while but
were never switched on; serves had been getting a cruder guess instead,
the same rough "which third of the court" scale every other shot uses.
One string came back fully committed: s5f3b3? — a body serve, a
forehand toward one side, a backhand toward the same side, and the same
unwritten ending every string gets right now.
Put it the way a tennis fan would: don't trust whoever's standing closest, trust the best evidence — and let it overrule your own serve call if it has to.
For the technical reader
- Striker-assignment history: v1 (nearest player at the event frame) died on the cusp problem. v2 (post-hit court-frame ball velocity) died on the homography's ground-plane assumption — airborne "court velocity" is dominated by vertical image-space motion and reads negative after hits from both ends; true since M1, invisible until v2 leaned on it. v3 (current): two candidate assignments per rally (near-first / far-first) plus one permitted parity flip at a ball-track hole — confirmed on SAM losing the ball for frames 54–78 of point_53, exactly where a far-end hit had gone missing; the flip is frame-verified.
- Vote weights: touch = 3, serve call = 2, landing half = 1 (the collapse/bounce detector, far-half-only by construction — one-sided, honest). point_59: SERVER-OVERRIDE — the gated serve detector said near, touch + landing said far, frame check confirms far was right.
- Letter committal: refines to nearest approach to the assigned
striker. f/b commits only past a height-scaled box gate (a fixed
pixel gate previously rejected every far-end letter, since the far
player is ~55px tall).
?cases: a "far" box on a spectator by the Perrier cooler, a "near" box on a court shadow. Frame-verified commits: 3/3 on point_16, plus a point_24 spot-check. - Dry-run montage: all 54 prompt boxes reviewed pre-spend. Killed 25/29 mover-fallback bootstraps (glare, shadow edges, line junctions, net band, a FedEx cooler, a ballkid). Toss-bootstrap fixes: search window now scales with apparent height (was a fixed 130px above a 55px far server); ball-shape filter drops elongated racquet-arc blobs. Recovered 7 far-end tosses.
- API run: 23 clips sent, 18 tracks returned, 5 lost outright (giant stuck boxes, every "weak accept"). Two lessons on the record: an all-dropped result used to crash the batch (now a graceful skip); fal chunks long videos — past ~490 frames it throws "No prompts available for this video chunk." point_55 worked trimmed to 450 frames; point_58 (893 frames, the 49-shot point) is parked until split-and-stitch exists.
serve_zone()is actually called now — v1 had defined the function and then coded serves by full-court thirds anyway. Conflicts column (votes that lost) kept on record as the honesty metric.
point_36: s6f1f?b3b??3f2? (loop v2 demo clip — far box drifts onto a spectator)
point_24: s5f3b3? (complete string, spot-checked)
Session cost: about $1.50. Project total: about $2.80 of the $9 budget — this low because the free eyeball review caught problems before they became paid API calls. The toss bootstrap carried this session; the backup ball-finder still needs a rethink, not a tune-up.