The Benchmark Grows Up

That clip up top: watch the test set grow, tile by tile, from four matches to seven — the point counter climbing from 135 to 491 human-charted points. Then its self-reported trust score recalculates live on screen — down, then back up — as the reason for the dip gets found and fixed. It closes on a real point from the Sabalenka–Pegula U.S. Open final: a serve into the body, a backhand up the middle, a forehand rally ending wide and deep — flagged correctly trustworthy at a 0.902 score by a version of the tool that had never watched that match before.

Would you trust a player's world ranking more if it came from one dazzling week at one tournament, or from a whole season played across different surfaces and different opponents?

Two posts back, cv-14 gave the project its scorecard: count how many fixes — changes, insertions, deletions — a human editor would need to turn the machine's chart into the real one — an acceptance at one fix or fewer. cv-15 then built a trust flag on top of that scorecard: every drafted point gets stamped HIGH or LOW, and HIGH means "start correcting from this draft." But that flag had only ever been tested against 135 points from four matches — barely 1 in 9 of which passed the one-fix bar to begin with — and cv-15 signed off with HIGH at 93% reliable, plus a warning: 135 points wasn't enough to trust yet, and three more matches were already on the way.

Here's why that matters. A hot week can flatter a ranking — a small field, a favorable draw, a home crowd — and it isn't wrong exactly, it's just thin. A full season is the harder test: more matches, tougher and more varied opponents, and the ranking is allowed to go down afterward even though the player didn't get worse. It's finally being asked to hold up over more evidence, and a ranking that survives a full season is worth more than a flashier one built on a single great week, precisely because it was harder to earn.

That's recalibration: redoing the trust-score math against a bigger, tougher set of examples, and trusting the number wherever it lands — even down. (More on how these trust scores get built.) The confidence flag lives or dies on two numbers: how often it's even willing to say HIGH at all — called coverage — and how often HIGH is actually right when it does — called precision. It's the same trade a line judge faces: call everything close and you're helpful but wrong a lot; call only the obvious ones and you're always right but useless. The cv-15 flag had only faced one hot tournament's worth of evidence. This session gave it a full season.

Three more finals, same machine

The pool of candidates was every match the Match Charting Project's volunteers have hand-charted — this project's answer key, explained here — cross-checked against what's findable on YouTube. The winning format was the "condensed match" upload: a 20-to-40-minute re-cut that keeps nearly every point and skips only the between-point filler, which lines up with the human chart far better than a highlights reel. Three finals made the cut: the Australian Open final at night, the U.S. Open final, and the ATP Finals indoors in Turin — 575 more charted points to draw from.

All three ran through the pipeline's existing machinery untouched — no new code, just settings. Each broadcast had one quirk: a court's blue paint too bright for the usual color threshold, a green sideline apron too close to the court's own color, a hazy light-blue apron needing a brighter ceiling — all one-line settings fixes. One old lesson got relearned, too: the recipe that converts a screen position into a court position — the homography — has to be built from footage where the camera is actually holding still. The first attempt smeared because the camera was still panning; a static five-second window during the serve toss fixed it.

New broadcasts also surfaced a new headache and caught an old bug. The headache: all three hide the scoreboard graphic at 0-0, briefly confusing the tool that uses it to tell points apart — solved by cross-checking the human charter's own serve notes. The bug: the method for tracking which end of the court a player serves from had been quietly miscounting tiebreak games as a full 13th game instead of the shorter, differently-scored game it actually is — a one-line fix that flipped a whole match's worth of serve-end guesses from mostly wrong to mostly right. And on the record: an overnight background job died silently, and two matches' worth of finished work sat untouched until someone checked by hand.

None of that dented the results. Scored against the pipeline's unchanged rules, the three new matches set new highs: the U.S. Open match got the serving end right 121 times out of 128 (95%), and Turin got rally length right, give or take one shot, 131 times out of 157 (83%). On the strict pass-fail test, the bigger set held essentially flat: 28 of 491 points needed no correction at all (5.7%), close to the earlier 7 of 135 (5.2%). The pipeline didn't get worse by testing it harder.

The honest number goes down, on purpose

On the small set — 135 points, four matches — the confidence flag's HIGH rating was right 93% of the time, willing to call HIGH on 32.6% of points. Recalculated on the full season — 491 points, seven matches, five broadcasts, same math, no changes — the honest numbers came in lower: 88% right, called on 21.2% of points. That's not the tool getting worse. That's small-sample flattery washing out.

One match makes the point almost too neatly. In the small test, the Wimbledon match looked flawless: 11 for 11, 100% right, the best result in the table. In the big test, with more than double the Wimbledon points to check, that same match fell to 69% right while claiming to flag more than half the match. The small test wasn't rigged — it just didn't have enough Wimbledon points yet to catch what was wrong. More data didn't break Wimbledon's score. It let the project finally see it, and go looking for why.

The Wimbledon blind spot

The why turned out to be a single broken check.

cv-15's best idea was a serve check: a real toss lands the ball's launch point more than 20 meters beyond the baseline, so a "serve" that starts from inside the court is really a broadcast cutting into the middle of an already-running point — and the draft gets marked down for it. On its own, it had been the confidence project's single biggest improvement.

But it only works when the system identified the serve by watching the ball's own flight. When the ball-tracking data is too spotty to trust, it falls back to guessing the serve from the players' stance and position instead — and on the Wimbledon broadcast, every single serve was identified that second way. So the serve check had zero real serves of its kind to examine, for an entire tournament's worth of points. It wasn't passing every point because it checked them and they held up; it had nothing to check, and defaulted to "trust it" silently, every time.

Under that free pass, three failures went unnoticed. One point's drafted chart recorded 8 shots, evenly spaced about a second and a half apart — but the real footage runs twice as fast, a shot every 0.9 seconds, meaning the shot-detector had missed every other stroke. It still looked trustworthy because the who's-hitting-next check happened to add up anyway — the same ball-tracking blind spots hit both checks the same way, so the errors canceled out instead of standing out. A second clip opens on a slow crossfade into a rally already underway — five real seconds of tennis before the stroke the stance method called "the serve." cv-15 had already caught this trick once, on the clay-court match, where the broadcast cut hard into the middle of a point; Wimbledon's editor does the same thing with a fade instead of a cut — exactly what the (silently broken) serve check exists to catch. And a third case: a four-shot rally in the human's chart where the ball-tracking data never shows a single net crossing — the ball's path across the net is a rally's skeleton, and this rally had none.

The fix wasn't to patch the broken check. It was to add two checks that look at the whole point instead of just the serve. First: does the ball cross the net, at rally speed, before the moment the chart claims the serve happened? If so, the clip started mid-rally already — catching both the hard-cut and the fade. Second: does the charted shot sequence show any net crossings in the tracking data at all? No crossings, no trust, however many shots the chart claims. Swapping the one broken check for these two took Wimbledon's held-out score from 69% to 85%, and the project's overall number from 88% to 94%, while cutting the worst-case disaster flags roughly in half.

The project already learned this lesson once, in a different costume. Back in cv-14, the machine's shot-direction calls came out backwards for a large share of points, because the code assumed a fixed side of the court instead of checking which end the receiver was standing at — nobody noticed until the scorecard itself gave the bug away. The Wimbledon check is the same mistake in a different jersey: a trust score is only as honest as its least-examined assumption. The project's best idea had quietly been a rubber stamp on one entire broadcast. The audit question was never "is this check smart" — it's "can this check even fire, here, on this footage."

Where the honest number landed

Test round	Willing to say HIGH (coverage)	Right when it did (precision)
Small test — 135 points, 4 matches (cv-15)	32.6%	93%
Big test — 491 points, 7 matches, before the Wimbledon fix	21.2%	88%
Big test — 491 points, 7 matches, after the Wimbledon fix	19.6%	94%

The trade for that last jump is real: 1.6 fewer points of coverage, for a genuinely more honest number — the right trade every time. See Court Vision in Plain English for why coverage and precision have to be read together.

A let-cord sensor that never buzzes isn't proof of a clean serve. It might just be unplugged — and the only way anyone finds out is by checking whether it could have buzzed at all.

For the technical reader

The three matches. t5 = Sinner–Zverev, Australian Open final 2025 (hard, night, AO feed). t6 = Sabalenka–Pegula, US Open final 2024 (hard, WTA, USO feed). t7 = Djokovic–Sinner, ATP Finals 2023 (indoor hard, Tennis TV feed). 575 MCP points on the table, 372 clips extracted, 356 scored.

Config-only integration. Every match ran through the existing fitcourt / probe / extract / boundaries / align stages plus one YAML per match — no script forks. Fixes landed in court_detect config: t5's bright coating starved the tophat threshold default (raised); t6's green apron shares an HSV band with its blue court (a second hull band, OR'd in); t7's light-blue apron rides V≈252 (value ceiling opened). Homography fit window: t5's first window carried 26.5 px of camera pan smear, fixed via a static 5-second serve-setup window (≤0.2 px residuals).

Zero-valued columns. All three broadcasters hide the score-bug at 0-0, which splits game-start points into duplicate-score fragments. Adjudication: MCP's own serve columns arbitrate most groups (a human row reading 4w plus a second-serve rally means the fragment pair is one point, serve in the later piece); the rest fell to changeover parity and eyeballed frame strips. Losers get blanked so the eval skips rather than double-charges.

Tiebreak parity bug. The staged prior counted Turin's set 2 as 12 games + a tiebreak = 13 swap-units (odd). Wrong: a 7-4 breaker is 11 points with one internal end-change at 6, and the set-end change cancels it — even parity. The set-3 server-end vote had come back 17 agree / 37 disagree, systematically flipped; correcting the prior flipped it to 37/17, verified on video.

Process failure. The run stalled overnight: the background-job watcher died silently between t6 ball tracking and t7 players jobs — both finished fine, nothing was listening. The rest of the run polled jobs directly instead of trusting completion callbacks.

Per-match scorecard, same eval, same constants:

metric              t5 AO night    t6 USO WTA      t7 Turin indoor
server end          53/71  (75%)   121/128 (95%)   133/157 (85%)
rally length ±1     47/71  (66%)    99/128 (77%)   131/157 (83%)
letters (aligned)   93/117 (79%)   120/148 (81%)   112/157 (71%)
acceptance ≤1 edit   2/71          10/128            9/157

Pooled acceptance: 28/491 (5.7%), up from 7/135 (5.2%).

Full leave-one-match-out table, before the Wimbledon fix:

LOMO (held out)   high prec (≤5 edits)   coverage
t1 night              10/10 (100%)         45.5%
t2 ctrl                 3/3 (100%)         60.0%
t3 clay               11/12  (92%)         20.3%
t4 grass              18/26  (69%)         53.1%
t5 AO                   3/4  (75%)          5.6%
t6 USO                34/35  (97%)         27.3%
t7 Turin              13/14  (93%)          8.9%
pooled                92/104 (88%)         21.2%

The strict ≤2-edit tier was re-attempted at n=491 and still dies in leave-one-match-out (0% coverage at a 19% base rate) — not shipped.

The t4 (Wimbledon) autopsy. Eight false-highs, each pulled apart against its chart CSV, the human's string, and frame strips. The launch-plausibility gate (cv-15's best find) only inspects ball-called serves; every t4 serve is stance-called; the gate was vacuous on the entire feed, contributing +0.59 logit of unearned trust per point.

Three named mechanisms: (1) t4_point_02, the half-cadence chart — 8 shots at 1.5s spacing against a real ~0.9s cadence, striker chain alternating cleanly because the ball track is blind in the same places the hit detector is; (2) the dissolve-cut mid-rally join — a crossfade into a live rally, 5 seconds of tennis before the stroke the stance detector blessed as "the serve"; (3) the spineless rally — a 4-shot chart with zero net crossings even at the weak gates.

Two dead ends, kept: weak-gated crossings don't recover the missing strokes (point_02's 17-stroke rally yields 7 weak crossings); charted cadence is real but soft (AUC 0.62, marginal flags 50/50) — a sidecar column, not a model feature.

What shipped: xr_pre_serve (rally-speed net crossings ending before the charted serve) and rally_spineless (a 3+-shot chart with no spine at all).

LOMO (held out)      before                after
t4 grass         18/26 (69%) @ 53.1%   17/20  (85%) @ 40.8%
pooled           92/104 (88%) @ 21.2%  90/96  (94%) @ 19.6%

No other fold fell below 92%. High-tier disasters halved, 12 → 6. The trade was 1.6 points of coverage. Residual: t4_point_11 stays flagged at 10 edits — its dissolve-cut leaves no pre-serve crossings because the track is blind there too.

Final refit. The shipped scorer refit on all 491 flags 97 points high at 96% in-sample; exports regenerated for all seven matches — 508 draft points, 99 flagged high.

Where things stand now: seven matches, five broadcasts, 491 human-charted points on the books, and a HIGH flag that means "start from this draft" and is right about 94% of the time on the honest, held-out test — for roughly 1 in 5 points confident enough to earn the flag at all. The benchmark finally has the size its own warnings had been asking for.

Session cost: $0.00 — three tournament finals' worth of brand-new answer-key data, added for zero marginal dollars. Project total so far: about $16.