AimShot is a light-gun duck hunt for an ordinary laptop. Point a toy gun at the screen, pull the trigger, and the built-in webcam works out where you are aiming. There's no sensor bar or special hardware, just a printed ArUco marker on the gun.
The Idea
The question was whether a webcam alone could turn "where is the marker in the camera image" into "where on the screen is the gun pointing", accurately enough to play a game on. The setup itself makes this hard. The gun points at the screen and the webcam sits in the screen bezel, so the marker is seen almost head-on. For 6-DoF pose estimation that's the worst case: two different poses reproject almost identically, and a naive solver flips between them from frame to frame.
So the project was as much about measurement as about the game. Every aiming decision is backed by a number.
What We Built
-
✦
Pose from foreshortening Yaw and pitch are read from the marker's perspective foreshortening: as the marker turns, its near edge grows and its far edge shrinks. These features change in step with the real angles, need no camera-intrinsics calibration, and can't flip sign.
-
✦
Three solvers, ranked automatically Calibration uses nine aim points. Three solvers are fitted to that same capture: a least-squares mapping (five unknowns per axis, so the fit is overdetermined), a centroid homography and a physical ray-cast model. The one with the best leave-one-out error is kept, and the player sees a grade from EXCELLENT to POOR.
-
✦
A ray-cast barrel model This model traces the gun barrel to the screen plane. It picks between the two possible poses for all calibration points together, because picking per point produced over 1000 px of error on held-out points. The mount offset is capped at 200 mm so it can't soak up noise.
-
✦
Measured smoothing A One-Euro filter adapts to how fast the gun is moving. Its shipped defaults were tuned on measurements, not guesswork: at 30 Hz they beat a plain exponential filter on both jitter and lag, and a test fails if that ever stops being true.
-
✦
Aim that tunes itself Shots that clearly targeted one duck count as measurements. After 8 confident shots the game learns a constant offset, and after 40 well-spread shots a full affine correction. Median statistics, a hard cap and a held-out self-check stop bad samples from taking over, and the base calibration is never modified.
-
✦
Low-latency capture Capture and detection run on a worker thread. Each new frame overwrites a single-slot mailbox instead of joining a queue, so stale frames never pile up as lag.
-
✦
A markerless follow-up A drop-in tracker uses MediaPipe hand landmarks instead of the marker. It can aim from an extended index finger, from the palm while holding a toy gun, or from the forearm. A gun-colour tracker measures the barrel axis directly.
The Stack
Python 3.12 with OpenCV contrib for ArUco detection and pygame for the game and UI. The solvers are written in NumPy with no optimiser dependency: a coarse sphere scan, then a shrinking pattern search. MediaPipe hand and pose models power the markerless path. A serial trigger for a microcontroller is written, but isn't enabled yet. The pytest suite of 299 tests needs no camera or display: it runs on fake capture devices, synthetic marker renders and SDL's dummy drivers.
What We Learned
The game is playable end to end: menu, nine-point calibration, a 60-second duck hunt, results and a live diagnostics screen. It is also fully playable with mouse aim. An end-to-end test renders a real ArUco marker into synthetic frames and runs the whole pipeline: it calibrates to EXCELLENT and predicts held-out aim points to about 1 px. That validates the maths and the wiring, not real-world accuracy.
With a simulated gun, marker size mattered more than anything else. Calibration error was 73 px with a 70 mm marker and 20 px with a 150 mm one. When the player moved out of the calibrated volume, the ray-cast model held at around 250 px while the linear solvers ran into the thousands.
On real captures, the markerless held-gun tracker measured about 170 px of error on a 1536×793 screen. The marker capture it replaced measured about 446 px. Even 170 px is still short of a standard duck's hit radius, so the ducks and their hitboxes were enlarged to match the precision the input can actually deliver. Adding calibration points didn't help, because the remaining error comes mostly from how consistently a person holds their hand.
Need something similar? Reach out. We build computer-vision prototypes that come with honest accuracy numbers.