HoloVoxel Studio is a voxel builder you control with your bare hands. A webcam feed fills the screen, a 3D scene floats on top of it, and you place, delete, recolour and rotate blocks using hand gestures. There's no controller or headset, and no install beyond a browser.
The Idea
Hand tracking now runs in real time in a web page. We wanted to find out whether it is reliable enough to build with: whether a pinch can mean "place a block", a pointed finger can mean "delete", and two hands can turn the whole structure, without false triggers making it unusable.
We built it twice. The first version showed the interaction model and grew a set of mini-games on top of it. The second rewrote the pipeline around a stricter, maths-first gesture layer, smoothed landmarks and instanced rendering, and added an optional learned classifier for custom gestures.
What We Built
-
✦
Two-hand tracking MediaPipe tracks up to two hands at 21 landmarks each, drawn as an overlay on the live camera feed. The second version converts the raw landmarks to world coordinates and smooths them with an exponential moving average (α = 0.4), so the cursor holds steady.
-
✦
A gesture vocabulary In the first version, one hand can pinch to place, point to delete, show two fingers to cycle block type, hold an open palm to move everything, make a rock sign for a 3×3 burst, and make an L-shape to clear. Two hands rotate the scene: both palms turn it about Y, both fists about X, and a palm with a fist about Z. The rewrite adds an aim state, a fist grab and a two-hand delete.
-
✦
A debounced state machine Gestures go through a state machine that requires a pose to be held before it fires and adds a short cooldown after each action. Fixed rules decide which gesture wins when two overlap. This layer is what makes a jittery hand behave like a deliberate input.
-
✦
An integer voxel world Blocks live on an integer grid stored in a map keyed by coordinate, and the world can be exported as JSON. The first version groups blocks into structures. The second renders through a single Three.js InstancedMesh with room for 20,000 voxels.
-
✦
Holographic presentation Six block materials include glowing, glass and gold, with number-key shortcuts. The camera feed is layered with particle bursts, trails, a charge ring and a hex-grid backdrop, alongside live status for the camera, hand and gesture, plus an FPS counter.
-
✦
Mini-games The first version ships five gesture games built on the same engine: Target Pop, Speed Challenge, Memory Match, Tower Build and Catch Blocks.
-
✦
A trainable gesture classifier The second version adds a TensorFlow.js network as a secondary recogniser for gestures that are hard to express as rules. It learns from landmark coordinates plus fingertip distances, needs 85% confidence held for 200 ms, and saves the trained model to the browser's IndexedDB.
The Stack
Both versions are plain browser JavaScript with no build step. Three.js renders a transparent 3D canvas over the getUserMedia webcam feed. The first version uses ES modules and the MediaPipe Tasks Vision hand landmarker. The second uses the classic MediaPipe Hands solution with camera utilities, plus TensorFlow.js for the optional classifier. Distance, angle and coordinate transforms live in a small maths module. All tracking and inference run on the device, so no video leaves the browser.
What We Learned
The rewrite settled on a layered design. A deterministic detector, built on thresholds for finger extension, curl and pinch distance, is the primary system for the core vocabulary. Machine learning is kept for gestures that are hard to express as thresholds. Between the landmark model and the voxel engine sit two filters: smoothing on every landmark, and a state machine that requires each pose to stay stable for 100 ms and enforces a 200 ms cooldown after an action. The landmarks drive the build only after both. It is still a prototype. The TensorFlow.js classifier can be trained and is loaded at startup, but it isn't yet wired into the main gesture loop, and the mini-games exist only in the first version.
Need something similar? Reach out. We prototype camera-based and gesture interfaces that run entirely in the browser.