Gestura

An accessibility-focused desktop application that enables hands-free OS navigation by mapping facial landmarks to system cursor movements and translating "mouth-open" gestures into native mouse clicks.

View GitHub Repository

Overview

Gestura bridges complex technical implementation with deep, empathy-driven product design. Designed as assistive technology for motor-impaired users, it replaces traditional mouse input with intuitive facial tracking. It processes real-time webcam feeds using computer vision to provide zero-touch, hands-free computer control.

Live Demo: Hands-free OS navigation via facial tracking in Gestura

Product Vision & Ownership

Conceptualized, researched, and initiated the core product concept by identifying a critical gap in assistive technologies for upper-body mobility-impaired individuals. Led the end-to-end vision by formulating the comprehensive technical roadmap, software specifications, and architectural project plans.

Gestura User Flow Diagram — User flow diagram

Software Engineering Highlights

Electron IPC & Security Architecture

Architected a secure Inter-Process Communication (IPC) bridge that successfully isolates the Node.js environment from the web renderer. Utilized preload scripts (contextBridge) and ipcMain/ipcRenderer to safely pass browser-based video tracking coordinates to the native OS without exposing the main process.

Gestura System Architecture — System architecture diagram

Native OS Integration & Real-time Computer Vision

Interfaced directly with low-level OS windowing and cursor APIs using robotjs, effectively handling macOS deep system constraints such as native Accessibility and Camera permissions. Integrated the clmtrackr machine learning model for real-time facial tracking.

UI/UX Design Highlights

Assistive Tech

Designed a hands-free interaction paradigm tailored specifically for motor-impaired users. Physical head movements were mapped symmetrically by purposefully mirroring the video feed, granting users a natural and intuitive spatial awareness.

Real-Time Contextual Feedback & Visual Affordances

Crafted a real-time visual feedback loop crucial for zero-touch interaction. Deployed a clear status indicator system (idle, calibrating, tracking, face-lost), along with a clear feedback for success commands (CSS ripple animations). This visually proved to the user exactly what the system was tracking and immediately confirmed gesture-based system clicks.

Gestura Interface - Calibration — Intuitive calibration interface

Gestura Interface - Live Tracking — Active tracking state with visual feedback

Technologies

ElectronReactTypeScriptrobotjsclmtrackrComputer Vision