ML × DESIGN MASTERCLASS · MODULE 02 SPATIAL COMPUTATION

YOUR HAND IS
THE MOUSE

What if interfaces understood movement instead of clicks?

Client-side MediaPipe landmark inference · No image leaves your machine.

PROJECTOR OPTIMIZED (1920×1080) PRESS [H] FOR PEDAGOGICAL DECONSTRUCTION

MEDIAPIPE / HAND LANDMARKS WAITING FOR HAND

SHOW ME YOUR HAND

Hold your hand upright towards the camera sensor

PINCH → SCALE

EXPAND

BIMANUAL SPATIAL DISTANCE

NORMALIZED TELEMETRY D

INDEX X 0.50

INDEX Y 0.50

PINCH 0.12

VELOCITY 0.05

IDLE DRIFT

Values are unitless coordinates bounded in [0.00, 1.00]

WEBCAM SENSOR

PEDAGOGICAL DECONSTRUCTION · SPATIAL INPUT

How Vision Becomes Interface

CAMERA Optical image stream capturing hand in physical space

↓

LANDMARK DETECTION MediaPipe calculates 21 3D geometric joint keypoints

↓

X / Y COORDINATES Normalized spatial vectors [0.0 - 1.0], velocity & euclidean distances

↓

            
BEHAVIOURParticle emitter flow, pinch scaling, and bimanual choreography

“Computer vision turns movement into numbers.

Design turns those numbers into behaviour.”

INDEX FINGERTIP [X, Y]

Direct spatial pointer mapped to screen coordinates with organic physical damping.

PINCH DISTANCE [4 ↔ 8]

Euclidean distance between thumb and index tip. Replaces mouse clicking and drag scaling.

INDEX VELOCITY [Δv / Δt]

Differential speed of the fingertip. Expands particle turbulence and energetic trails.

BIMANUAL SPAN [L ↔ R]

When two hands appear, their Euclidean separation triggers typographic CONDENSE vs. EXPAND transitions.

Press H or ESC to return to live experience

SYSTEM DIAGNOSTIC

The application requires access to a camera device to perform MediaPipe landmark tracking.

HTTPS / Localhost Requirement: Browsers require a secure origin to unlock webcam sensors.
Browser Permission: Allow video access when prompted in the browser URL bar.
Simulator Available: You can practice and demo all interactions using the Virtual Mouse Simulator.