T1: A Test Set With Answers in the Back

That clip up top: two court outlines on the same night-match frame. Red is the camera math borrowed as-is from the practice match; green is a fresh fit built from this match's own lines. Red drifts visibly off the real court — and its own accuracy numbers still said it looked fine.

Every setting this project has tuned so far was checked by the one person who tuned it. That's a chair umpire overruling his own call and then congratulating himself for getting it right — the call might be correct, but nobody should trust it just because he says so.

Previously: the plan is to feed broadcast video in and get the same shot-by-shot notation out that Match Charting Project volunteers hand-code by watching real matches (cv-01). Every dial since then — how confidently motion counts as a swing, how the serve toss gets spotted, whose shot each ball is credited to — was tuned on one practice match, Zverev–Gasquet, and checked with my own eyes (cv-09). So was the homography — the rule, built from the court's painted lines, that converts a screen position into a court position (cv-03).

Somebody else has to hold the answer key

A line judge who grades his own overrules isn't cheating, exactly — he's just human, and human eyes trust their own first read. Tennis's fix is a second set of eyes that never made the call. This project needs the same fix: an answer key built by someone who never saw its output. This devlog calls that ground truth (more: Court Vision in Plain English) — and Jeff Sackmann's Match Charting Project supplies it: volunteers who hand-code real matches, shot by shot, in the exact notation this project is trying to produce.

There's a second half to the rule, the half that makes a test count instead of a rehearsal: the moment a match gets used to fix a bug, it becomes practice, not proof. So every setting gets locked first — written down, untouched — before the software meets a match it never influenced. Freeze first, grade second: that's a benchmark.

Finding a stranger's homework

The Match Charting Project has hand-charted more than 7,500 men's matches; the practice match isn't one of them. A match from the same tournament, court, and broadcast package is: Canada Masters 2017, Nadal vs. Shapovalov, round of 16 — 225 points, charted point by point, at night under floodlights, both players left-handed. Neither fact was tested on the practice match: its shot calls assume a right-hander, its motion settings were tuned in daylight. Same court and graphics package, so any failure here traces to what actually changed.

Held in reserve: Federer vs. Haase, the semifinal — day, both right-handed, the closest match here to practice conditions. If the software struggles there too, lefties and floodlights aren't why.

The test's rules were locked before any of it ran: no setting changes once grading starts (one part stayed adjustable — appendix). For every clip that lines up with a human record, the check compares who served, where the serve landed, how long the rally ran, and how often the shot-by-shot calls match — counting only what the software committed to, not its shrugs. That 225-point answer key is public, free with attribution, and cost nothing to get.

Matching clips to the human chart

Each clip first has to be found, then matched to a point. Finding them gave the practice match the most trouble; here it barely put up a fight — point the software at the file paths and the new camera math, and 25 clips of real play fell out, no setting touched. Matching them means reading a score bug — the little scoreboard graphic in the corner, frozen pixel-for-pixel during a point, changing only when the point ends (more: Court Vision in Plain English) — and finding the line in the human's chart with that same score.

One convention nearly wrecked it: the human record lists the server's score first; this match's bug always lists Nadal's first, serving or not. Every Shapovalov service game disagreed on which number came first, costing six wrongly-matched points before I caught the pattern. Before the fix, 17 of 25 clips lined up; after it, 23 of 25 did.

The remaining two aren't errors, they're genuine ties: a 40–Advantage score that happened twice in the same game, and a deuce from a game that reached deuce ten times — indistinguishable by score alone. The process even caught a mistake of mine: I'd misread one clip's serve indicator by eye, and the human chart showed Shapovalov had actually served.

The camera math doesn't carry over

The camera math built on the practice match missed by about 20 pixels on this new footage — different filming day, camera remounted. Refitting it took three failed attempts before a fourth worked.

Attempt one reused the first homography experiment's approach, unchanged. Under floodlights, the paint boundary where the court's blue surface meets its green surround reflects light almost exactly like a real white line. The fit grabbed that paint edge and drew a court 15 to 20 pixels too big.

Attempt two made the software pickier about what counts as a white line — which killed the paint problem, and the far baseline and both sidelines along with it. Not enough real lines left to build a court from.

Attempt three searched narrow bands near where the practice match's lines had been. It failed more quietly: one band locked onto ground paint 70 pixels off; another mixed in doubles pixels while hunting for a singles line. The fit then reported a great accuracy number for itself — but that number only measures agreement with the pixels the fit chose to trust, and some of those were the wrong paint. Drawn onto the actual frame and eyeballed, the center line was obviously off, by 32 pixels: ten times what the fit had reported.

What worked: a human — me — zoomed the frame in six times and picked the court's four corners by eye, once. The same kind of one-time nudge the very first tracking experiment needed, when someone had to draw a box around the ball before it would track (cv-02). Even that failed twice at first: the corners read near the baseline were the singles corners, not the doubles corners the court is actually measured to, further out and blurred into paint at the frame's edge. Two pieces of hand-worked geometry confirmed which was which. The final fit mixed far doubles corners with near singles corners, holding attempt two's lines back as an independent check.

Where things landed

What got checked	Result	What "good" looks like
Clips found in the highlights reel	25 of 25	Every real point located, none missed
Clips matched to a specific human-charted point	23 of 25 (2 are genuine ties, not errors)	As close to all of them as honest ambiguity allows
Final camera-math accuracy	Service line 0.8 px · Baseline 1.4 px · Center line 6.1 px	Single-digit pixels, not tens
Cost of both answer keys	$0.00	Free — public data, no paid tools

This is the third time this project has hit the same wall, so it's a standing rule now: a number can vouch for a wrong answer, but a drawn line on the actual picture can't lie the same way. Every camera-math fit gets rendered onto the video and eyeballed before it's trusted — which is exactly what the two colored outlines in the clip up top are for.

Twenty-five clips found, twenty-three tied to a line from a human's chart, a court that measures accurately, every setting still frozen. Cost for both sessions: $0.00 — public data and an afternoon staring at pixels. Next: turn this into an actual scorecard, machine against human, point by point.

For the technical reader

Footage: the 8:21 TennisTV highlights reel. Ground truth lives in the repo under data/mcp/ (CC BY-NC-SA, attributed). Eval protocol, exact terms: constants frozen; segmentation could still be re-tuned for the new reel's cuts (finding points isn't the hypothesis — charting them is); metrics per aligned point are server end, serve zone, rally length, and letter accuracy over committed letters only; coverage reported beside accuracy; no silent cherry-picking.

Score-bug join: MCP's Pts column is server-first, the on-screen bug is Nadal-first. point_06's serve marker was misread by eye; the join corrected it.

Homography attempt 1 was M1's recipe, verbatim; under floodlights the blue-to-green paint boundary reads V≈155, S≈55, the same signature as a real line at grazing angle. Attempt 3's guided bands were seeded from the dev prior; its self-reported residual was a beautiful 2.9 px, against the 32 px actually on the ground — plausible-but-wrong, caught only by rendering the reprojection and looking at it.

The fix echoes the M0 ball-click precedent: one human input per match. The two cross-checks that settled which corners were real — far-corner midpoint lands at x=603 against a measured center line of 606, near-corner reads implied 643; pixels-per-meter is linear in image y (48 at the far baseline through 80 at the near service line), extrapolating to ~100 at the near baseline and predicting a doubles span of ~1095 pixels, exactly the white band's raw extent. The final fit held attempt 2's tight-mask lines back as untouched validators rather than fitting to them.