we made jarvis, yo
theme: navigation
team HAASHtag's (Hendry, Ariana, Arthur, Song Han)'s submission for BigRedHacks 2026.
navigating HCI, reimagined.
We have all seen attempts at making spatial control for computers (handy, mitsu, and gesturecontrol). However, they all suck.
Why do they suck?
- literally just mapping your hand to cursor position
- boring! no one wants a bastardization of spatial HCI by forcing your hand to become a mouse cursor.
- unintuitive gestures w/ low gesture count
- no usage of 3D space and other non-manual controls
All previous solutions are just an attempt to stuff spatial HCI on top of an outdated keyboard-mouse interaction framework.
JARVIS!!!!
HoloTouch is designed to be...
- intuitive with easy gestures for human hands
- extensible with custom gesture creation and fine-tuning
- 3D space-first by using all the space around the user and using facial features to locate gestures
- quick, responsive, and consistent, making it an actually viable replacement for keyboard-mouse control
HoloTouch is the closest you will get to tony stark tossing windows around.
HoloTouch runs on XFCE under X11, and on GNOME under Wayland.
uv sync
uv run holotouch doctor # what this machine has, and what it lacks
uv run holotouch # or `holotouch panel` for the control panelOn GNOME, Wayland lets no outside program move windows or press keys, so HoloTouch does both through a small GNOME Shell extension of its own. Install it once, then log out and back in (GNOME Shell only finds a new extension when it starts):
uv run holotouch gnomemarkdown-docs/overview.md has the gestures and how it all works.
HoloTouch uses the built-in webcam to compute a total of >500 landmark/feature points across the user's face, hands, and forearms. We use Mediapipe for most of this computation, although we augment with a small, custom-trained neural net.
We had to solve a longstanding problem in CV called occlusion. With complex hand motions, Mediapipe-generated landmark points often get too cluttered and overlap with each other, causing gesture detection to be unreliable. We solve this through a small custom neural net (fully connected MLP using a subset of the landmark features) that takes into account for a few hundred ms of previous gestures and can look-ahead predict what gesture might come next.
HoloTouch is consistent and intuitive enough to not be a gimmick anymore. In fact, I have been using HoloTouch to skip songs when I'm working and minimize some windows. The fact that Holotouch can actually be used in my workflow and improve my quality of life shows how well-executed it is.
Accurate, quick, intuitive, and consistent spatial computing is possible. With perhaps a higher resolution camera and a little better hardware, Tony Stark's JARVIS is just a few years away.