Court Vision in Plain English

The devlog assumes you know tennis cold and computer vision not at all. That's the right assumption, but it means the posts lean on a handful of ideas that deserve to be explained properly once instead of badly fifteen times. This page is that once. Every entry in the series links back here; nothing below needs more math than a chair umpire does in their head.

Pixels and frames​

Video is a flipbook. A broadcast is 25 to 30 still pictures per second — each picture is a frame — and each frame is a grid of colored dots called pixels. A 720p frame is 1280 dots wide by 720 tall, about 900,000 pixels, and at broadcast distance the game ball occupies maybe eight of them. That's the raw material: not "a rally," just a very fast stack of photographs.

Two consequences run through the whole series. First, when a post says "frame 240," it means the 240th photograph — eight seconds into a 30fps clip. Frames are the pipeline's clock. Second, there is no motion anywhere in the data. Motion is something you infer by comparing one photograph to the next — like a line judge who never gets to watch the ball fly, only to study a strip of stills and work out what must have happened between them. Everything the pipeline does — tracking, hit detection, serve calls — is built on top of that inference. When a fast-moving ball travels a long way between two consecutive photographs, or smears into a blur inside one, the inference gets hard. That's most of why broadcast tennis is a hostile environment for this stuff.

Homography: the TV picture vs the court diagram​

The broadcast camera looks at the court from a raised angle behind the baseline, so the far half of the court appears tiny and squashed and the near half looms. A charting diagram is the opposite: top-down, to scale, every meter equal. A homography is the conversion rule between the two — a recipe that turns "the ball is at this spot on your TV screen" into "the ball is two feet inside the baseline."

You already run this conversion by instinct: watch a match on TV and you can mark, on a paper court diagram, roughly where a ball landed. The software needs the recipe written down, and it gets it from the court's own painted lines — match the lines on screen to the official dimensions (10.97 by 23.77 meters) and the conversion falls out. Four corners turn out to be enough.

The catch that echoes through the whole series: the recipe only works for things on the ground. It maps the court's surface, not the air above it. An airborne ball's screen position converts to where its shadow would be, not where the ball is — which is how a perfectly fit conversion can report a ball "18 meters past the baseline" mid-flight. Feet are on the ground, so player positions convert truthfully. The ball only tells the truth at the moment it bounces.

Background subtraction: compare everything to an empty court​

Take a photo of the court with nobody on it. Now compare each broadcast frame against that photo, pixel by pixel: whatever's different is something that moved in — a player, the ball, a ballkid. That's background subtraction, and it's the series' zero-dollar player detector.

The empty-court photo comes free, via a trick worth admiring: for each pixel, take the median value across all the frames of a clip. Anything that moves — players, ball — is only ever briefly at any given pixel, so the median votes it out of existence. The court lines, which never move, stay razor sharp. The result is a photograph of a court where nobody is playing tennis, distilled from footage where two people very much are.

The failure modes are exactly what you'd guess. Anyone who stands still long enough becomes part of the "empty court" — a line judge at parade rest literally erases himself from the data. Anyone who moves near the court reads as a player — ballkids, umpires, a flapping banner. And the whole comparison assumes the camera holds still; a mid-point pan smears the empty-court photo like a shaken Etch A Sketch, so frames have to be aligned to each other before anything else happens.

Detection vs tracking​

Two different ways to follow a ball, with opposite personalities.

Detection looks at every frame fresh: "is there a ball in this picture, and where?" Thirty times a second, no memory, no idea what happened in the previous frame. Tracking is the opposite: lock onto one specific object and follow it from frame to frame, carrying memory forward — the way you watch a rally without blinking.

Tracking gives you continuity: it knows the ball in frame 100 is the same ball as in frame 99. But memory is a liability. Blink at the wrong moment — occlusion, motion blur, a camera cut — and a tracker can lose its object or latch onto the wrong one, and everything after that is confidently wrong. A detector can't lose the plot because it never had a plot; it just answers the same question again next frame. The price is that it doesn't know which ball is which, or that the "ball" it found this frame is the one from last frame.

The series uses both. The early milestones tracked (SAM following one prompted ball); the later ones detect (a specialist finding the ball fresh every frame) and stitch the per-frame answers into a trajectory afterward. The switch — and why it happened — is cv-12.

Ball-finders: tiny specialists vs promptable generalists​

Two very different kinds of model appear in this series, and the contrast matters.

SAM 3 is a generalist — a huge model trained on everything, that you prompt. Give it video plus the words "tennis ball," or a box drawn around one specific ball, and it finds, outlines, and follows the object. Enormous range: the same model segments players, courts, umpires, anything you can name or point at. The costs: it needs to be told what to follow, it bills by the frame on a hosted API, and as a tracker it can lose the object and keep going as if it hadn't.

WASB is a specialist — a 6-megabyte network that has only ever looked at tennis balls. For each frame it outputs a heatmap: a picture the same shape as the frame where brightness means "probability the ball is here." Take the hottest spot, done. It answers exactly one question — "where's the ball in this picture?" — about thirty times a second, fresh each time. It never gets lost, because it never has to remember anything. It can't be asked anything else, and it doesn't need to be.

The tennis version: the generalist is a brilliant all-court player you have to coach before every point; the specialist is a line judge who has watched one line their entire career and is nearly unbeatable on it.

Reading events out of a trajectory​

Once the ball's positions are strung into a path over time, the charting events live in the path's shape. A hit reverses the ball's direction of travel — the ball goes back where it came from, as sharp a signature as a signal gets. A bounce reverses nothing: the ball keeps traveling the same way, just lower and slower. The series' shorthand for the distinction: the court doesn't return serve. After a hit the ball leaves fast; after a bounce, its speed collapses.

The camera angle complicates this (see the homography note — the software really places the ball's ground shadow, and an airborne ball's shadow moves at nonsense speeds), and the later posts rebuild the detector around a blunter piece of tennis sense: a live rally sends the ball across the net every shot. Count the net crossings and you have the rally's skeleton — each crossing-to-crossing slice contains exactly one hit, and the delicate cusp-hunting is demoted to locating the hit inside its slice rather than deciding whether it exists. Hits are inferred from geometry; nobody ever sees a racket touch a ball.

The score bug is metadata​

The score bug — the little scoreboard graphic in the corner — is not part of the scene. It's an overlay drawn on top of the picture, so a camera pan never moves it, lighting never changes it, and it behaves with a discipline nothing on the court can match: it is pixel-for-pixel constant for the duration of a point, and it changes at exactly one kind of moment — when a point ends.

That discipline makes it two tools. First, a point ID: if two stretches of footage show the same unchanged score bug, they are fragments of the same point, even if the director cut away in between; if the bug changes mid-stretch, that stretch holds two points. Second, a join key: read the score off the bug and you can look up the exact same point in a human-charted record of the match, which is what makes machine-vs-human grading possible point by point.

It's the least glamorous pixels on the screen and the most reliable witness in the building — the one thing every broadcaster in every country agrees to keep on camera at all times. The pipeline learning to read it is cv-13.

Ground truth and benchmarks​

Ground truth is an answer key created independently of the system being graded. For this project it's the Match Charting Project: volunteers who watched real matches and hand-coded every shot. When a post says a match "has answers in the back," it means a human already charted it, point by point, in the exact notation the pipeline is trying to produce.

Why it matters: for the first stretch of the series, the person tuning the pipeline was also the person grading it — checking his own frame calls with his own eyes. That's calling your own lines. It's fine for practice and worthless as evidence, because the same eyes that tuned the thresholds will forgive them. A benchmark is the discipline that fixes it: freeze the pipeline (no constants change, written down before the run), point it at matches somebody else charted, count the agreement, publish the number even when it's embarrassing.

One more rule rides along: anything tuned on a match can't be graded on it. The dev match is a practice court; the test matches are tournaments. The moment a test match gets used to fix a bug, it quietly becomes practice too — which is why the series keeps having to go find fresh matches.

Precision, coverage, calibration​

The finished tool doesn't just draft a chart — it flags each drafted point HIGH or LOW confidence. Three words do the work of describing whether those flags mean anything.

Coverage is how often the system is willing to say HIGH — the share of points it stamps. Precision is how often it's right when it says HIGH. The two trade off exactly like line-calling does: a judge who only calls the balls that land a foot out is nearly always right and nearly useless; one who calls everything close is helpful and often wrong. A confidence flag is only worth something if you know both numbers.

Calibration is how the flags get honest: take points that humans have already graded, look at the signals the pipeline computes about itself — did the serve call commit? does the ball track have holes? did the striker votes conflict? — and fit those signals to the actual outcomes. Then the crucial hygiene, called leave-one-match-out: each match gets flagged by a model that never saw that match during fitting. The system never grades its own homework. When cv-15 says HIGH means "start from the draft" with 93% reliability, that 93% survived exactly this test — and the fancier tier that didn't survive it is reported dead rather than quietly deleted.

Token edit distance​

The acceptance metric — the number the whole project is climbing toward.

A finished chart of a point is a string of tokens: the serve and its zone, then each shot's letter and direction, then the ending. Both charts — the machine's and the human's — get written as token lists, and the question asked is brutally practical: how many tokens would an editor have to fix — change, insert, or delete — to turn the machine's chart into the human's? That count is the token edit distance. Zero means the machine matched the volunteer exactly. A point is accepted when the distance is at most one — the chart a charter would sign after fixing at most one thing.

Two properties make it the honest metric. It grades the deliverable, not the components — server calls, rally lengths, and letters can all inch upward while no single point is actually right, and this number refuses to be fooled by that. And a refusal (?) costs exactly what an error costs, because the editor has to fix both: a shrug is an edit too. The pipeline doesn't get credit for hedging. When cv-14 reports 2.2%, that's the fraction of points a human would sign — the honest distance to the mountain top, measured in the only unit a volunteer charter cares about: corrections.

Finding the court: color masks and line votes​

A chair umpire doesn't scan the whole stadium looking for the baseline — they already know the court is the blue, or green, or clay-red rectangle in the middle of everything else, and the lines are the white paint on top of it. The software finds the court the same way, in two passes.

Pass one is color. Pixels are usually stored as red-green-blue amounts, which is a clumsy way to ask "is this court-colored?" HSV — hue, saturation, value — describes a pixel closer to how you'd say it out loud: hue is which color it is, saturation is how vivid or washed-out, value is how bright or dark. A color band is a range along those three dials that counts as "court" — bluish, fairly vivid, medium brightness, say. Paint every pixel that falls in the band and everything else disappears. The catch: every court and every broadcast needs its own band. A hard court's blue isn't a grass court's green isn't a clay court's orange, and two blue hard courts can still shoot different bands under different lights or camera color grading. There's no universal "court color" — it gets found fresh, feed by feed.

Pass two is lines. Inside that color silhouette, look for pixels that are bright and colorless — white paint — and hand them to line detection, usually a technique called the Hough transform. The plain-English version is voting: every white pixel "votes" for every straight line that could possibly pass through it, thousands of candidates, one lonely vote apiece. A stray white pixel — a sock, a stray reflection — votes for lines nobody else agrees with, so those votes go nowhere. A real court line is dozens of white pixels sitting in a row, and every one of them happens to vote for the same line, so that line's tally spikes far above the noise. Keep the high-tally lines and the court's paint is found without the software ever being told where a single line actually is — it just counts raised hands. (One line reliably out-votes honestly: the net tape, white and straight from most angles but not on the ground — which is exactly why the homography fit has to be told to ignore it.)

Stabilization: nudging frames back into line​

The background subtraction entry already flagged the problem: a broadcast camera that looks "static" to a viewer is actually being nudged by its operator, following play, and that invisible drift smears any frame-to-frame comparison the way a shaky hand smears a photograph. Stabilization is the fix, and it's worth knowing what it actually does.

ECC — Enhanced Correlation Coefficient, a mouthful whose short version is "how well do these two pictures line up" — is the algorithm usually doing the fixing. Give it two frames and it slides one of them a few pixels this way or that, checking after each nudge how well the two now agree, until it finds the shift that lines them up best — the way you'd slide two nearly-identical photos over each other on a lightbox until the court lines land exactly on top of each other. "Translation only" means it's allowed to slide but not rotate or zoom, which is plenty: a camera operator following a rally pans mostly sideways, not in circles.

Why it runs before anything else touches a frame: measured directly, a broadcast camera can pan over 30 pixels in the middle of a single point — invisible watching the footage, but enough to throw a court-coordinate calculation off by meters at the far baseline. Skip stabilization and every downstream step — the clean plate, the player detector, the homography — ends up quietly comparing frames that don't actually agree on where the court is.

Residuals and reprojection: checking a fit by drawing it​

Fitting a court model is like a practice serve: you don't just trust that the toss felt right, you watch where the ball actually lands and measure the miss. Reprojection is the watching step — take the model's answer (say, the four-corner homography fit) and draw it back onto the photo it came from, line for line. A residual is the miss, measured in pixels: the gap between where the model says a line should be and where the line actually, visibly, sits in the picture.

Small residuals sound like proof the fit is good, and usually they are. But this project has learned, more than once, that they can lie. One homography attempt reported a beautiful 2.9-pixel residual — while its center line sat 32 actual pixels off the real one. The fit had locked onto the wrong paint edge entirely, but every downstream number it produced was consistent with that wrong edge, so the residual math looked flawless. A wrong answer that agrees with itself all the way down still passes every internal check.

The only thing that caught it: rendering the fit onto the actual frame and looking, where a 32-pixel miss on the center line is obvious to anyone with eyes. That's house policy now — every geometric fit gets drawn onto its frame and eyeballed, because a self-consistent number can be fooled by its own bad assumptions, and a picture checked against the real paint cannot.

The notation legend​

Reading a Match Charting Project point string is like learning a stenographer's shorthand for tennis: learn the marks once, and a whole point compresses into a single line of text that anyone else who knows the same marks can read straight back.

A string runs left to right, in the order things happened. It opens with the serve zone — the leading digit is 4 for a serve out wide, 5 for a serve to the body, 6 for a serve down the T (the center line). Every shot after that gets a letter for its type: f for a forehand, b for a backhand, the two you'll see in almost every rally, plus rarer letters standing in for volleys, overheads, slices, drop shots, and lobs. A digit riding right after a letter is direction — roughly which third of the court, right, middle, or left, the shot crossed into. It isn't measured from whichever wing hit the ball; it's pinned to a fixed side of the court that flips depending on which end is receiving that point (see changeover parity) — the exact convention a coordinate bug in this project once got backwards.

The string closes with an ending. * is a winner — nobody got a racquet on it. n, w, d, and x mark an error and where it missed: netted, wide, deep, or both at once. One more mark rides along on any error — @ if it was unforced (the player's own doing) or # if forced (the opponent earned it), a judgment call this project still leaves to the human charter rather than the machine. And ?, anywhere in the string, means the pipeline looked and declined to guess — a shrug, not a claim.

The very first point string this series published, fully decoded: s5b2f1f2f2x@. Serve into the body. Backhand return up the middle. Forehand into an outer third. Forehand up the middle. Another forehand up the middle. Then the ball misses wide and deep at once, the hitter's own unforced error, point over. Six characters, one full point, nothing left ambiguous — the target this pipeline is graded against every time a token edit gets counted.

Changeover parity: deriving who stands where from the score​

Tennis players swap ends after every odd game — 1st, 3rd, 5th, and so on — so nobody keeps the sun or the wind the whole match; anyone who's watched a live match has sat through the changeover chair break. It's a rule, which means it's also completely predictable from the score alone, with no need to ever look at which player is standing where.

Give the software who served game one and the running score, and it can work out — by arithmetic, not by watching — which end each player stands at for any later point: count games, apply the swap-after-odd-games rule, and remember the tiebreak wrinkle, where players swap every six points instead. That's how this project maps "who is receiving this point" onto "which side of the court, in pixels, is the receiving side" without ever recognizing a face or a shirt number — in one match, knowing only that the near-side player was Nadal explained 11 of 14 confident serve calls on its own.

The reason it earns its own entry: an off-by-one here doesn't cause one small error, it silently flips which player the software believes is on which end for every point after the mistake — and since direction digits are defined relative to which end is receiving, a single wrong changeover call can invert every server call and every direction digit downstream of it at once, with nothing crashing to flag it.

The boundary race: calling an out ball nobody photographed​

No camera — not even Hawk-Eye's full multi-camera rig at a Grand Slam — actually photographs the instant a ball touches the ground on a line call. The dramatic replay on a stadium screen isn't a picture of contact; it's math, drawn to look like one.

Here's the shape of that math. The camera sees the ball flying, in a smooth, predictable arc, and then loses it — the ball lands out of frame, behind a sponsor board, or simply too far from any camera to register a pixel there. What every system does have is the flight leading up to that point: real, seen positions, enough of them to know the arc's shape. Extend that arc past the last position anyone actually saw, the same way you'd extend a ruler along a line to find where it runs off the edge of the page.

Then it's a race between two lines: watch which court boundary the extended arc crosses first. Cross the sideline before it would cross the baseline, and the ball missed wide. Cross the baseline first, and it missed long. The landing spot itself is never seen directly — the same blind spot the homography entry already flagged, where an airborne ball's screen position reads as its shadow rather than the ball, and the same trick hits and bounces already leans on to find contact points no camera really captures. The call is calculated from the flight, not read off a photograph of the ground.

Scoring a detector: F1​

The confidence entry already named the tension: a judge who only calls the balls that land a foot out is nearly always right and nearly useless; one who calls everything close is helpful and often wrong. The same tension shows up one level down, scoring a detector itself — like the specialist that finds the ball in every frame — not just a confidence flag.

Two numbers capture it. Precision asks: of everything the detector claimed was a ball, how much of that actually was? Recall (this project mostly calls it coverage) asks the opposite: of every ball that was actually there to find, how much did the detector catch? A detector can cheat either number alone — call nothing, and precision looks flawless while recall collapses to zero; call everything in sight, and recall looks flawless while precision collapses.

F1 is the single score that won't let it cheat either way. It averages precision and recall in a way that punishes a bad number in either column much harder than a normal average would, so a detector can't fatten its F1 by starving one side of the ledger to feed the other — it only climbs when both numbers are healthy at once. When this series reports the ball-finder scoring F1 94–96 on its benchmark, that's both halves of the line judge's job holding up together, not one propped up by neglecting the other.

Base rates, logits, and the scoring formula​

Grading a confidence model needs one more piece of context: how rare the thing it's hunting for is to begin with. If only 1 point in 10 would ever actually get signed off by a charter — a 10% base rate — a model that flags nothing at all is already "right" 90% of the time by doing nothing, so any real number has to be read against that, not treated as if hunting for a 10%-common thing were as easy as hunting for a 50%-common one. It also means fewer real examples exist to learn the pattern from in the first place — the way there's less film of second-serve double faults than of first serves landing in.

The model underneath a confidence flag is a logistic model, and in plain terms it does one thing: it looks at several pieces of evidence about a point — did the serve call commit cleanly? does the ball track have gaps? did the shot-callers agree? — gives each piece a weight learned from past graded points, and adds the weighted evidence into a single running score called a logit. A logit isn't a probability yet; think of it as a tally that can run positive or negative without limit as evidence piles up in either direction, which a last step then squashes into a clean 0-to-1 confidence number. One broken check in this project quietly passed every point without ever actually checking anything — it contributed +0.59 logit of trust it hadn't earned, nudging the tally up anyway.

AUC — area under the curve — measures something narrower and useful on its own: how well one signal, by itself, sorts good points from bad ones before anything gets combined. Take a random accepted point and a random rejected point and ask how often the signal correctly ranks the accepted one higher. A coin flip gets that right half the time — AUC 0.5. A perfect signal always gets it right — AUC 1.0. A signal that scored AUC 0.62 in this project, barely better than a coin flip, got demoted to a footnote for a human to glance at rather than trusted with a vote in the model.

Counterfactual headroom​

It's the same question a player asks a coach: if my second serve never broke down, how many more points would I actually win? You wouldn't guess — you'd want to replay the same matches with that one stroke perfected and count the real difference, which tells you whether drilling the second serve is worth more than drilling the backhand instead.

Counterfactual headroom is that question run on the pipeline. Take every graded point, fix exactly one component to a perfect answer — say, every direction digit exactly right, nothing else touched — and re-run the token edit distance grading across the whole set. The jump, or the lack of one, is a direct answer to "how much is this component worth," priced in the only currency this project cares about: points a charter would actually sign off on.

It turns "what should I build next" from a hunch into arithmetic, and the arithmetic can be humbling: fixing a component that's rarely even attempted can be worth surprisingly little on its own, while several components fixed together can be worth far more than any of them alone — the difference between sharpening one stroke and rebuilding a whole service motion.