M2: The Court Doesn't Return Serve
That clip up top: the pipeline calling out HIT and BOUNCE in real time, pinning every one onto a court diagram the instant it happens.
The last post ended with a warning that turned into today's homework: the system can only trust the ball's screen position at the exact moments it's touching the court — everything else is guesswork. So today's question: watching nothing but the ball's path across the screen, can software find every hit, and every bounce?
The detector I built today ended up calling all thirteen of those moments correctly. It got there by being wrong twice first, and both mistakes taught me more about how a tennis ball moves than the successes did.
Previously: last time, the software learned to convert "where the ball is on your TV screen" into "where the ball is on the actual court" — the same trick a fan does by eye, guessing where a shot landed. The catch, carried straight into today: that conversion only tells the truth for things touching the ground. A ball in mid-air converts to the position of its shadow, not the ball itself. So the software needs to know exactly when the ball is on the ground before it can trust anything else it says.
Reading the rally from the shape of the path
A ball in a rally does exactly two things, over and over, and they look nothing alike in its path.
When a player hits the ball, it reverses direction — back the way it came, sharp and immediate. When the ball bounces, nothing reverses. It keeps traveling the same way, just lower and slower. A racket sends the ball backward; the court just takes some steam off it and lets it keep going. That's the whole idea in one line: the court doesn't return serve. (More on reading events out of a flight path: Court Vision in Plain English.)
What the ball's path actually looks like
First move: turn the tracked dots into a graph of position and speed over time, and look at it.
The ball's up-down screen position, how fast that's changing, and how fast its shadow moves lengthwise down the court — across the rally's 290 usable frames (each a single still picture — more in Pixels and frames), with 10 short gaps where the tracker briefly lost the ball.
Two traps, both from camera angle. First: a ball's up-down position on screen isn't the same as its real height — a ball at the far baseline still looks "high" simply because it's small and distant on screen — the pretty arcs you'd expect a bounce to draw aren't real; they're height and distance tangled together. Second, and stranger: the shadow's sideways speed spikes to the equivalent of 200 meters per second when the ball is high on the far side. Nothing on court moves that fast — it's the camera exaggerating airborne motion.
First attempt: eight hits, one bounce
My first version found every cusp in the ball's up-down speed — where it stops falling — then called hit or bounce by whether direction flipped. It found nine such moments and called eight of them hits.
That's not how tennis works — almost every stroke is preceded by a bounce. Something was badly wrong, and the fix wasn't better math. It was doing what a chair umpire does with a disputed call: stop arguing and go look at the replay. I pulled the actual video frames around every flagged moment.
Frame 44, flagged as a hit. The ball is at Zverev's feet; his racket is nowhere near it. That's a bounce.
Watching those frame strips built a hand-verified answer key for the rally, and exposed two ways the detector was fooling itself.
Two failures, two fixes
Near-side bounces looked like hits. Right after a real bounce, the shadow speed collapses into a small, noisy number whose sign flips often enough to trick the direction test. Fix: stop asking whether direction flipped; ask how fast the ball leaves instead. A real return leaves fast; a bounce's outgoing speed collapses toward zero.
Far-side bounces were invisible, not just hard. After a far bounce, the ball rises and recedes at once — both push its screen position the same way, so there's no cusp to find at all. What does exist: the same shadow-speed exaggeration from before. Airborne, the far shadow races at wildly inflated speed; the instant the ball lands, that exaggeration vanishes and the shadow slows near the truth. That collapse marks the bounce, even where nothing in the picture looks like one.
The false alarm that taught the most
The first version of that collapse-detector immediately flagged a moment mid-rally as a bounce. The video showed the ball sailing over the net, a meter and a half up — nowhere near bouncing.
It took a minute to understand: the collapse isn't instant at ground contact — it unwinds gradually through the descent, fading as the ball loses height. An early "slowing down" is just the ball still falling. The bounce is the last collapse of a descent, not the first.
Re-checking frames after that fix surfaced one more find, and it wasn't about the ball: my own hand-built answer key had quietly stopped early, before the rally actually ended. The tracked data ran further — one more far bounce, then Gasquet's final swing, both missed on my first pass. Even a hand-checked answer key needs its own second check.
The finished detector: seven hits, six bounces, all thirteen matching the video.
The payoff
Thirteen events found, checked against video, correctly told apart as hit or bounce — zero wrong calls. Because the screen-to-court conversion already exists, each one drops onto an actual scale drawing of the court:
Thirteen dots and X's on a court diagram. Solid yellow (near-side bounces) are trustworthy to within centimeters. Hollow dots (far-side) are only good to within a few meters — the same camera-angle problem as above.
The near-side bounces landed at 19.4, 19.6, and 20.4 meters down the court — about three and a half meters inside the baseline, exactly where pro groundstrokes land. Nobody told the software that; it just fell out of the math.
Read the sequence out loud: hit, bounce, hit, bounce, hit — a rally, described shot by shot. No shot types or directions yet, but for the first time this reads like the start of a match chart, not a vision demo.
The scoreboard
| What was checked | Result |
|---|---|
| Events found in the rally | 13 (7 hits, 6 bounces) |
| Correctly told apart, hit vs. bounce | 13 of 13 |
| Wrong calls | 0 |
| Near-side bounce landing spots | within centimeters of real groundstroke depth |
| Far-side bounce landing spots | good only to within a few meters |
The takeaway
A hit throws the ball back where it came from; a bounce just lets it keep going, a little slower. That one difference — reversed or not — is enough to read a whole rally's worth of events out of a wobbly, camera-distorted flight path. The court doesn't return serve.
For the technical reader
Differentiated the M0 track: 290 usable frames, 10 small gaps. Two traps: image-y mixes height with court depth (M0's "parabolas" are projection mixtures); along-court shadow velocity spikes to ±200 m/s when the ball is high on the far side.
Detector v1: cusps in vertical velocity, classified by direction flip. Found 9 events, called 8 hits. Root-caused by pulling video frames at every event rather than debugging further — frame 44 flagged as a hit showed the ball at Zverev's feet, racket nowhere near it.
Failure 1, near-side bounces classified as hits: post-bounce shadow velocity collapses into noise (-2 to -4 m/s) and its sign flips, fooling the direction test. Fix: classify on outgoing speed, not sign — real returns leave at 5+ m/s (often 40); bounces collapse toward zero.
Failure 2, far-side bounces structurally invisible: post-bounce the ball rises and recedes simultaneously, both pushing image position the same way — no cusp exists in that coordinate. Fix: shadow-speed collapse — airborne shadow runs -30 to -50 m/s of amplified nonsense, on-ground shadow runs true speed (~-5).
False positive at frame 71: collapse detector v1 flagged it as a bounce; video shows the ball crossing the net, 1.5m up. Root cause: collapse unwinds gradually through the descent — early collapse candidates are descent-in-progress, not landing. Fix: bounce = last collapse of a descent, not first.
While re-verifying, found my own ground truth had stopped at frame 243 while the track runs to ~298 — one more far bounce, then Gasquet's final swing, found by the detector, missed by the manual labels.
Detector v4 (final): 7 hits, 6 bounces, 13/13 matched against frame-verified ground truth, 13/13 classified correctly, 0 false positives. Near-side bounce depths: 19.4, 19.6, 20.4m — ~3.5m inside the baseline, consistent with pro groundstroke depth.
Caveats: thresholds tuned to one clip, generalization untested. Far-side positions good only to a few meters — at the far baseline, one frame of timing error is meters of shadow, and the ball is already moving again by the next frame. Near-side positions look genuinely good.
Running total across the first three builds — an object tracker, a screen-to-court conversion, a frame-verified event detector — is still 60 cents, all spent on the first one. The other two were just math run over data already collected.
Next: turning this sequence of events into an actual chart — splitting footage into rallies, figuring out shot direction, and writing it all out in the Match Charting Project's own notation. That's the point where this stops being a computer-vision project and starts being a tennis project.