{"$schema":"https://ejfox.com/schema/page-twin@1.json","kind":"project","url":"https://ejfox.com/projects/creative-interfaces","json_url":"https://ejfox.com/projects/creative-interfaces.json","generator":"ejfox.com/json-twin@1","data":{"cacheVersion":"2026-10-05-source-hash","sourceHash":"e4e3e5a9206fda139803f6cf1d9ef5c7a4f1f76a46fb44955abf004ac635b517","html":"

These are my attempts to make computers that respond to the way humans actually move and gesture.

\n

The other day I had this idea to try and control a modular synth using computer vision. I loaded YOLOv4 into a python script and had it detect objects in an episode of Atlanta I had lying around, sending the X and Y positions of every object as control voltage to the oscillators. It makes this weird, haunting, sort of robotic sound — but I kind of really like it. Then I had a fun idea to run it through the webcam, and I couldn't figure out why it wasn't working — I kept seeing the chair as an object. Then I figured it out: the chair is the instrument. Why did I make a chair an instrument? I don't know. I ended up running dance videos from Sherrie Silver, the This Is America choreographer, through it instead — not just the X and Y positions but the distances between the objects — and it comes up with some really cool, interesting generative effects. I don't really know how I use this right now, or what it's for exactly. That's kind of how I know it's art.

\n

That session: Using Computer Vision As A MIDI Controller.

\n

The other end of the spectrum: no camera, no screen at all. I've been using Talon to control my computer, so I tried to do the dishes and code at the same time — talking to Claude while my hands were in the sink, having it answer everything out loud. There are some sharp edges — a couple times it didn't pick up everything I said — but the fact that you can detach from looking at a screen and still engage in the mechanical task of writing code and making things with a computer is completely radical. It's unlike anything I've ever experienced in my life.

\n

That one's on video too: Coding While I Do The Dishes.

\n

See also Flipper Zero Tools and Terminal Tools.

\n

Hand-tracking & gesture

\n

Hand MIDI Controller

\n
\n\"Hand\n
Hand MIDI Controller — MediaPipe hand landmarks mapped to MIDI CC
\n
\n

Hand MIDI Controller is a real-time hand-tracking MIDI controller built on MediaPipe and OpenCV, turning hand movements into MIDI control signals for music production, VJing, and performance. It includes TouchDesigner integration.

\n

Handtrack WebSockets

\n

Handtrack WebSockets does real-time 3D hand tracking with MediaPipe, OpenCV, and FastAPI, streaming hand position and rotation over WebSockets. It includes a cyberpunk visualization layer that renders the tracked hand.

\n

OpenCV MIDI Controller

\n
\n\"OpenCV\n
OpenCV MIDI Controller — OpenCV object tracking driving MIDI CC
\n
\n

OpenCV MIDI Controller is an experiment in sending MIDI signals based on OpenCV object detection.

\n

Speech Echoes

\n
\n\"Live\n
Live speech rendered as decaying typographic echoes
\n
\n

Speech Echoes is a Vue app that captures live speech and renders it as decaying typographic \"echoes,\" with a font and weight captured per utterance. It's built with p5, MediaPipe, and d3-force.

\n

Glasses HUD

\n
\n\"The\n
The two lenses you compose a HUD across
\n
\n

Glasses HUD is a tool for quickly prototyping AR lens visualizations and HUDs. Drag data widgets onto the left and right lenses to compose a heads-up display from live bio, motion, location, environment, and device signals.

","title":"Hand-Tracking & Generative Art","metadata":{"title":"Hand-Tracking & Generative Art","date":"2024-07-01T04:00:00.000Z","category":"Art","featured":false,"modified":"2025-08-13T15:46:54.000Z","tech":["Computer Vision","MIDI","WebGL","Hand Tracking","Generative Art","Interactive Media"],"state":"deployed","ai-involvement":"ai-assisted","tags":["creativity","interfaces"],"words":540,"images":4,"imageDetails":{"total":4,"cloudinary":4,"withDimensions":0},"links":8,"codeBlocks":0,"headers":{"h2":1},"toc":[{"text":"Hand-tracking & gesture","slug":"hand-tracking--gesture","level":"h2","children":[{"text":"Hand MIDI Controller","slug":"hand-midi-controller","level":"h3","children":[]},{"text":"Handtrack WebSockets","slug":"handtrack-websockets","level":"h3","children":[]},{"text":"OpenCV MIDI Controller","slug":"opencv-midi-controller","level":"h3","children":[]},{"text":"Speech Echoes","slug":"speech-echoes","level":"h3","children":[]},{"text":"Glasses HUD","slug":"glasses-hud","level":"h3","children":[]}]}],"type":"post"}},"_links":{"self":"https://ejfox.com/projects/creative-interfaces.json","html":"https://ejfox.com/projects/creative-interfaces","index":"/projects.json"}}