M3: The Toss Is the Bootstrap

That clip up top: a whole point, serve included, charting itself — the tracking box locks onto the ball at the toss on its own, no clicking, and the notation string builds along the bottom. First time this pipeline has gone start to finish, and it's still a little rough.

Every rally tracked so far needed a person to draw a box around the ball by hand first. Fine once. Absurd sixty times. The fix: a fact about tennis itself. Every point begins the same way, and the software can now spot that beginning on its own.

Two posts back, the first experiment taught a promptable tracking AI called SAM 3 to hold a tennis ball through a rally — but only after a human circled that exact ball with a box, by hand: it could follow once pointed at, never find one alone. Last time out ended on a debt: its notation string was a rally's middle, no beginning, because nothing yet knew where a point starts.

Every point starts the same way

No rally begins in the middle. Every point opens with one unmistakable action: a player stands still, tosses one ball straight up, and swings. Nothing else moves like that — the receiver stands still, every spare ball sits still in a ball kid's hands, and the one thing rising alone is the game ball.

The bootstrap: instead of a person pointing at the ball every clip, the pipeline uses something tennis always does anyway — the toss — to point at the ball for itself. It already compares each frame to a photo of the empty court and flags what's different, the same trick that finds the players for free. The only new blob near a still server, right before contact, is the rising toss — the software boxes it itself (finding, not yet following), then hands the box to the tracking software. Detect once, then track. The click is gone.

The impostors, once more

The pipeline has been fooled before, by ball kids holding spare balls and a line judge who erased himself standing still. This time it was broadcast graphics. Without a confident serve, the software falls back to chasing the fastest small mover on court — and its first two guesses were the scoreboard graphic (more on that box later) flipping on every score change, and a small net sign vibrating in the breeze.

The fix: a ball in flight keeps one direction and one rough speed, frame after frame — flying ballistically. A flickering graphic doesn't fly anywhere, so requiring three straight consistent frames knocked both out. Two clips still beat it, fooled by lens glare off the court's own lines during a camera pan — unsolved, noted rather than hidden.

The dribble charts itself

Six clips ran through the paid tracking service for 55 cents total, less than a candy bar. Those tracks met the hit-and-bounce reader on a complete point for the first time, and found something no mid-rally clip could show: servers dribble before the toss. Bouncing to yourself is a real hit and a real bounce, over and over, so the detector — correctly, by its own rules — charted the warm-up as rally. Fix: the software already knows which frame the point starts on. Discard anything before it.

The first point, start to finish

Two clips produced a full serve-to-final-shot line with nobody typing mid-point. One reads, shot by shot: serve toward the middle (s2), a backhand the software couldn't place (b?), backhand wide (b1), forehand same side (f1), forehand back to the middle (f2), backhand wide the other way (b3), two more backhands, and an ending it couldn't call — eight shots, serve confirmed right. The other was two shots and out: an unclear serve, one unplaced forehand return, an ending it couldn't call.

Checking both by eye: serve call right, shot count rally-shaped — but the column saying who hit each shot broke the serve-return pattern whenever the guess for contact caught the ball mid-air, equally far from both players. Nearest-player-to-the-ball means nothing once the ball is closest to nobody. The letters aren't trustworthy yet; the shape under them is.

What got checked	What came back	Good enough?
Serve call	Right, both points	Yes
Shot count	Rally-shaped	Yes
Who hit each shot	Breaks on mid-air contact	Not yet
Broadcast impostors	Score bug + net sign caught; 2 clips still fooled	Mostly
Cost	55 cents, six clips	Well inside budget

Say it the way a tennis fan would: you don't need to point at the ball if you already know every point starts with a toss.

For the technical reader

• Click, closed. M0's flagged manual box-prompt: closed. Serve toss → auto box prompt for SAM 3, from the frame-minus-plate difference image at the serve detector's known toss location.
• Ballistic filter, exact. 3-point chain: consistent direction, consistent per-frame step. Line-junction glare during camera pans still passes it on 2 clips — unsolved, parked.
• Pilot run. 6 clips through SAM 3, $0.55 total. First attempt hit a request validation error — free lesson that the API wants integer pixel coordinates, not floats.
• Detector. M2's v4, fps now a parameter instead of baked-in 30, first run over full point clips (previously mid-rally excerpts only).
• Two charted points, raw:

point_16:  s2b?b1f1f2b3b?b3?     (serve ✓ — 8 shots, rally-shaped)
point_31:  s?f??                 (short point: serve, return, over)

• Deferred, on purpose. 48 of 54 clips not run this session — known fixes (alternation-aware striker assignment, contact-frame refinement, sparse-track handling) are free to iterate on the six already paid for. Running cost: pilot $0.55, project total ~$1.30 of $9.

This session's six clips cost 55 cents; project total is about $1.30 of a $9 budget. 48 clips stay untouched on purpose — every fix still ahead is free to try on the six already paid for.

The point string that opened this stretch read ??2?1?1?2?2?2???, almost all question marks, no s anywhere: a rally with a middle and no beginning. It now starts with an s.