Sign Language Translator (Gestura.ai)
A browser-accessible computer vision web application that captures webcam video, extracts 21 hand landmarks in real time via MediaPipe, classifies American Sign Language (ASL) finger-spelled alphabet gestures using a trained CNN model and geometric heuristics, and outputs real-time transcriptions with English and Hindi audio synthesis.
Attribution & Scope: Web application adaptation built during a hackathon. The underlying CNN model and landmark drawing methodology derive from academic research on sign-to-speech conversion. Recognizes finger-spelled American Sign Language (ASL) static alphabet letters and control gestures; does not claim support for all global sign languages or continuous full-body syntactic translation.
Problem & Objective
The Problem
Millions of deaf and hard-of-hearing individuals rely on sign language, but standard web communication tools offer no native interpretation for gesture inputs. Deploying real-time computer vision models in web browsers often suffers from webcam concurrency conflicts, heavy computational overhead, and gesture ambiguity between visually similar hand configurations.
Project Objective
Adapt an existing desktop-based sign language research project into a responsive, thread-safe web application featuring live video streaming, hand skeleton visualization, sentence building with spelling correction, and dual-language text-to-speech output.
Role & Contribution
Lead Developer (Hackathon Project) — Adapted and re-architected the upstream desktop computer vision scripts into a web-ready Flask application. Engineered the thread-safe background video capture loop, designed the glassmorphism web interface, and integrated multilingual text-to-speech services. Upstream model architecture originally based on academic sign-language conversion research; collaborated with Devansh Git on threading logic.
Analytical Approach
- 01
Pipeline Architecture: Engineered an end-to-end pipeline: Webcam Capture → MediaPipe 21-Landmark Extraction → 400x400 White Canvas Normalization → CNN Cluster Classification → Geometric Heuristic Disambiguation → Sentence Buffer → Audio Synthesis.
- 02
Thread-Safe Video Streaming: Implemented a background capture loop in Flask using the Double-Check Locking pattern and thread locks to eliminate camera frame race conditions and deliver smooth ~25 FPS MJPEG streams.
- 03
Dual-Stage Gesture Disambiguation: Hand landmarks were first classified into 8 gesture clusters via a pre-trained Keras CNN (`cnn8grps_rad1_model.h5`). A second-stage geometric ruleset evaluated knuckle distance ratios and coordinate thresholds to disambiguate visually adjacent signs (e.g., distinguishing A from S/E/T/M/N, C from O, and G from H).
- 04
NLP & Accessibility Integration: Connected recognized character sequences to PyEnchant dictionary suggestion engines for spelling prediction, with text-to-speech output via gTTS (English) and Google Translate API (Hindi).
Verified Technologies
Verified Findings
Hybrid CNN + Geometric Rules Solve Gesture Ambiguity
Direct 26-class single-shot classification on raw webcam images frequently confounded visually similar letters (such as A, E, M, N, S, T). Grouping letters into 8 structural classes via CNN and applying secondary finger-joint distance thresholds yielded far higher practical stability under variable lighting.
Background Thread Isolation Prevents Server Deadlocks
Decoupling webcam read operations into a dedicated background daemon thread with explicit locking prevented Flask HTTP worker processes from blocking on video I/O, maintaining responsive API and dual-stream delivery.
Clear Distinction Between Finger-Spelling and Full Sign Language
The implementation demonstrates static finger-spelling (A–Z alphabet) plus control gestures (Next, Space, Backspace). It accurately highlights the boundary between static letter recognition and continuous multi-sign syntactic interpretation.
Dashboard & UI

The live Gestura.ai interface displaying real-time webcam video feed, extracted MediaPipe hand landmarks, predicted sign letter, sentence construction buffer, and audio playback controls.
What I Learned
- Refactoring standalone desktop research scripts into a production-patterned, thread-safe web service.
- Designing hybrid machine learning pipelines that combine deep learning with geometric domain rules.
- Managing webcam hardware concurrency across multi-threaded HTTP servers.
- Honest technical communication regarding model limitations, dataset boundaries, and project attribution.