Nobody Sees the Ball Land

That clip up top: two real points from the US Open final. In each one, the yellow dots are the machine watching the ball fly — until the dots stop, mid-air, because the tracker loses it. Then the freeze-frame: a white line extends the flight the machine did see, racing toward two finish lines — the sideline and the baseline — and whichever it crosses first names the error. Both calls match what the human charter wrote.

Today the software learns to answer a question it has been failing, completely, since grading began: when a ball goes out, did it go out wide or deep?

Previously. This machine watches broadcast tennis and drafts a shot-by-shot chart of every point, which then gets graded, correction by correction, against charts written by human volunteers (how the grading works). Most parts of the draft have been inching upward for weeks. One part has not moved at all: when a rally ends with a ball landing out, the machine scored zero percent at saying which way it went out. Not "poor." Zero — across 159 tries, on eight matches, twice confirmed by different methods. The reason is simple and physical: the ball tracker loses sight of the ball a split second before an out-ball lands. Fuzzy yellow blur, busy background, and the one moment that decides the call is exactly the moment the machine goes blind.

The obvious fix — make the tracker hold on longer — is the one I'd been planning. Then a research deep-dive turned up something better, hiding in plain sight.

Hawk-Eye never sees the ball land either

Think about what Hawk-Eye actually shows you. That smooth animated arc, the ball flattening onto the paint, the "OUT" verdict by two millimeters — none of that is a photograph. No camera on Centre Court captures the ball touching the ground; at that instant the ball is a smear of pixels, half-hidden, moving at highway speed. What the system photographs is the flight. The landing is calculated — physics, fitted to the part of the flight the cameras saw clearly, finishes the sentence.

A veteran line judge does the same thing. She doesn't wait to study the skid mark. She reads the arc coming off the racket and knows the ball is sailing long before it lands. The information was in the flight the whole time.

That's the whole idea I stole this week: the boundary race — stop trying to see the landing, and instead extend the flight the machine did see until it crosses a line. If the extended path crosses the sideline first, the ball was going wide. If it crosses the baseline first, it was going deep. Whichever finish line the flight reaches first names the error. (One more entry in the plain-English primer, where the recurring ideas of this series live.)

No new AI. No new camera. Straight-line physics, fitted to the last stretch of dots the tracker recorded before it blinked — and two papers at this year's top computer-vision conference validate the family of idea, one of them placing landings within about 0.6 meters from a single broadcast camera.

What happened when I raced it

I ran the race on every point in the benchmark — the eight matches where human charters already wrote down the truth — and looked only at the points that genuinely ended with a ball landing out: 169 of them, of which 159 were plainly wide or deep (the rest were charted as both at once).

Question	Before	Now
Wide and deep errors named exactly right	0 of 159	75 of 159 — about half
Deep errors caught	0 of 90	55 of 90 — about 6 in 10
Wide errors caught	0 of 69	20 of 69 — about 3 in 10
When the race commits to wide-or-deep, how often is it right?	—	75 of 90 — about 5 in 6

From nothing to half, with arithmetic. And when the race actually picks a side — wide or deep, rather than hedging "both" or declining — it separates the two at about five in six, on flights the tracker gave up on.

The misses are honest and mostly one story: the TV camera looks down the court, so a flying ball always appears deeper than it really is — the same distortion that makes every approach shot look like it's sailing long from your couch. That bias makes the baseline finish line fire early, so some wide balls get called deep or "both." Fixing that lean is a real project for later, not a knob to fudge today.

What the video caught that the numbers didn't

Rendering that clip up top wasn't decoration — it caught two real bugs the scoreboard had quietly absorbed.

First: in one point, the machine's "last sighting" marker drew itself at the top of the screen while the ball's trail exited bottom-right. Impossible. The track data showed why: the ball supposedly moved 37 meters in three frames. Balls don't teleport — the tracker had lost the real ball and latched onto something else, and those junk points were poisoning the physics fit. The fix is a rule of nature, not a tuned setting: end the flight at the first physically impossible jump.

Second: the race originally asked an earlier stage of the pipeline which end of the court the final shot was heading toward — and that stage was sometimes wrong. One point claimed the far player hit the last shot, while the ball demonstrably died two meters beyond the far baseline. Now the race reads the direction from the flight itself and trusts nobody.

Both bugs were invisible in the accuracy tables and obvious in one rendered frame. This series keeps re-learning the same lesson, so it's earned a bolding: eyes beat derivatives — when a number and a picture disagree, believe the picture.

The dead end, kept

One more result, because this devlog keeps its failures on the record: I also tried the lazy version of "integration" — feeding the race's outputs as extra hints into the existing ending classifier, the one that learned from 11,646 human-charted matches. It did nothing. Zero percent stayed zero.

The reason is instructive. Winners — balls that land in and end the point — are nearly half of all endings, and thanks to that same looks-deeper-than-it-is distortion, the race happily extends many in-balls out too. A single flat classifier drowns the new signal in the old base rates. The race's real gift is conditional: given that a ball went out, it names the flavor. Deciding whether the ball went out at all needs different witnesses — a net-death, an observed landing, and soon (next on the roadmap) the sound of the ball itself. The architecture lesson: don't ask one judge to answer two different questions.

The takeaway: the machine stopped trying to see the ball land, read the flight instead — the way Hawk-Eye and every good line judge always have — and half of a stone-blind category came back at the cost of zero dollars and some algebra.

For the technical reader

The module is experiments/landing_spot.py in the court-vision repo (report: outputs/diag/landing_spot_report.txt; run with PYTHONPATH=.:experiments uv run python experiments/landing_spot.py).

Method (v1, deterministic, no constants fitted to truth): final flight segment = contiguous ball-track run after the last charted contact, capped at 2.0 s, cut at the first >6-frame hole and at the first teleport (>3.0 m/frame court displacement — ~90 m/s, physics bound; the same no-teleport gate boxes.py applies to players). Straight-line fit cx(t), cy(t) over the last ≤8 samples in court coordinates; target half inferred from the flight (already-out position wins, else the y-slope sign, else which half the ball occupies). Race: time-to-sideline (singles line ± the 0.25 m OUT_MARGIN from endings.py) vs time-to-baseline, extrapolation capped at 0.7 s past track death; crossings within 2 frames of each other → x (wide-and-deep); neither within cap → abstain ?.

Population: evaluate.evaluate records, truth = mcp_ending_type (w/d/x), same denominators as experiments/learn_components.py plus match #8 (g1). Full confusion (rows = truth):

        w     d     x     ?
  w    20    11    27    11   n=69
  d     4    55    18    13   n=90
  x     1     0     8     1   n=10

Loose (called any out flavor): wide 58/69 (84%), deep 77/90 (86%). Committed w-vs-d discrimination: 75/90 (83%). Per-match strict correct: t7 27/58, t6 28/44, t5 10/29, t4 9/14, t3 4/10, g1 3/11, t1 1/2, t2 1/1 — clay (t3, g1) collapses because thin tracks leave no segment to fit, which the planned clay fine-tune of the ball detector multiplies.

Version history within the session, all on the record: striker-derived target + no teleport gate → wide 30/69, deep 35/90 (overall 41%); +teleport gate → 31/69, 33/90 (41%); +flight-inferred target → 20/69, 55/90, overall 83/169 (49%), with the wide→x leak (27) being the depth-inflation bias concentrating inside the 2-frame "both" window.

The refuted integration: appending racer features (call one-hot, t_side, t_base, n_seg) to the 4-class leave-one-match-out random forest (learn_components.lomo, seed 0) moved wide/deep recall 0%→1% and cost overall accuracy (59.0%→57.4%). False out-calls on true winners/net: 204/317 (64%) — the racer is a conditional classifier P(wide vs deep | out), not an out-detector; it will enter the pipeline as a per-slot likelihood in the planned grammar-constrained fusion decoder, gated by independent out-ness evidence.

Literature anchors: TT3D (arXiv 2504.10035, CVSports @ CVPR 2025) and "Where Is The Ball" (arXiv 2506.05763, same workshop; 0.63 m mean landing error on broadcast tennis, 81 clips/10 matches). The hero video renderer is experiments/render_landing_hero.py (points t6_point_110, truth wide; t6_point_117, truth deep).

Session cost: $0.00. Project total: ~$16.

Next question: the race can name an out-ball's flavor, but something still has to say "that ball went out" in the first place — and the tracker's eyes have proven they blink at exactly the wrong moment. The next witness doesn't have eyes at all. Every broadcast microphone courtside hears the same thing you do: the pock of strings on ball. Time to give the machine ears.