Teaching a Computer to Watch Tennis
1. Teaching a Computer to Watch Tennis
Can a computer watch a tennis broadcast and write down every shot, the way a courtside volunteer does by hand?
How is this thing doing?
Court Vision teaches a computer to chart tennis matches from broadcast video. Every number below comes from grading the machine against 511 points hand-charted by humans, across 8 matches. Dots on the trend lines are grading runs. How the numbers got here →
In the pipeline
Who served, from which end?
Trust83%
How many shots in the rally?
Verify35%–83%
Which way did the shot go?
Verify71%
Forehand or backhand?
Verify60%
Where did the serve land?
Re-key40%
How did the point end?
Re-key20%
nothing to show yet
Faults and second serves
Missingnot attempted — every draft assumes a first serve
Proven, not plugged in yet
Devlog
Working notes, written as the work happens.
2. M0: Can SAM 3 Hold Onto a Tennis Ball?
An AI that finds objects by description can't tell one tennis ball from six on court — until you point at the one in play.
3. M1: From Pixels to Meters
How four painted corners taught the software to turn a TV picture of the court into a real, to-scale court diagram — for free.
4. M2: The Court Doesn't Return Serve
Teaching software to spot every hit and bounce in a rally, just from watching the ball's path bend.
5. M3: Who Hit That?
The software found the players by noticing who never moved, then called every forehand and backhand from where the ball crossed their body.
6. M3: The Cutting Room
Most of a tennis broadcast isn't the match — the software has to spot the one camera angle that actually shows it.
7. M3: The Static Camera Isn't
The 'still' broadcast camera actually drifts mid-point — and that breaks a free trick for spotting players in video.
8. M3: The Toss Is the Bootstrap
The serve toss lets the software find the ball itself, and it charts its first full point without a human drawing a single box.
9. M3: The Ball Gets a Vote
When the software can't tell who hit the ball, it holds a vote instead of guessing — and the ball outvotes the serve call, on camera.
10. T1: A Test Set With Answers in the Back
The software faces its first honest test against a match a human already scored, and the camera math needs fixing first.
11. T1: Right Hardware, Wrong Sign
Graded against a real human chart for the first time, the system's biggest miss was a lefty problem, not a broken tool.
12. The $0 Specialist and the Load-Bearing Bug
A free specialist replaces an expensive all-purpose tracker — and exposes that its rival's biggest flaw was secretly marking the end of every point.
13. The Score Bug Is the Point ID
A TV scoreboard graphic nobody notices turns out to be the only reliable way to tell where one point ends and the next begins.
14. Within One Token of a Human
The project finally gets an honest report card: how many fixes would turn the machine's chart into a human's? Today's first answer: 2.2%.
15. The Draft Grades Itself
A confidence flag tells a volunteer chart-corrector which draft points to trust — and this post checks whether the flag itself can be trusted.
16. The Benchmark Grows Up
A bigger test made the trust score honestly drop — then a check that could never fail got caught, and the score climbed back up.
17. The 33rd Charter
Court Vision introduces itself to the real volunteers who chart tennis matches by hand, and asks whether its drafts actually save them time.
18. Nobody Sees the Ball Land
The machine couldn't tell wide errors from deep ones. The fix: stop trying to watch the bounce and calculate it, like Hawk-Eye does.
19. The Machine Grows Ears
The ball tracker blinks at the worst moments. Broadcast microphones don't. The software learns to listen for the pock of strings on ball.
20. The Court That Fits Itself
A human has clicked four corners for every match this project ever charted. A free pretrained network just took the job — and knew when to refuse it.
21. Stick Figures Beat Smudges
The forehand-or-backhand call has been stuck for two weeks. The fix: draw a skeleton on every player and read the actual swing.
The road to auto-charting
Each milestone is a devlog entry. Here's where things stand.
- ✓M0 — Track ball and players through one rally with SAM 3
- ✓M1 — Court keypoints → homography → real court coordinates
- ✓M2 — Detect hits and bounces from ball trajectory
- ●M3 — Rally segmentation → shot sequences → MCP notation
- ○M4 — Validate against a human-charted Match Charting Project match