Technology & AI/Web Application / Project Contribution

Sign Language Translator (Gestura.ai)

A browser-accessible computer vision web application that captures webcam video, extracts 21 hand landmarks in real time via MediaPipe, classifies American Sign Language (ASL) finger-spelled alphabet gestures using a trained CNN model and geometric heuristics, and outputs real-time transcriptions with English and Hindi audio synthesis.

Attribution & Scope: Web application adaptation built during a hackathon. The underlying CNN model and landmark drawing methodology derive from academic research on sign-to-speech conversion. Recognizes finger-spelled American Sign Language (ASL) static alphabet letters and control gestures; does not claim support for all global sign languages or continuous full-body syntactic translation.

Landmark Tracking21 PointsMediaPipe hand skeleton extraction per frame
Model ArchitectureCNN + Heuristics8-group CNN classification + geometric distance rules
Stream Throughput~25 FPSThread-isolated MJPEG camera and skeleton visualization streams
Language SupportEnglish & HindiMultilingual TTS synthesis via gTTS and translation APIs
01 / CONTEXT

Problem & Objective

The Problem

Millions of deaf and hard-of-hearing individuals rely on sign language, but standard web communication tools offer no native interpretation for gesture inputs. Deploying real-time computer vision models in web browsers often suffers from webcam concurrency conflicts, heavy computational overhead, and gesture ambiguity between visually similar hand configurations.

Project Objective

Adapt an existing desktop-based sign language research project into a responsive, thread-safe web application featuring live video streaming, hand skeleton visualization, sentence building with spelling correction, and dual-language text-to-speech output.

02 / OWNERSHIP

Role & Contribution

Lead Developer (Hackathon Project) — Adapted and re-architected the upstream desktop computer vision scripts into a web-ready Flask application. Engineered the thread-safe background video capture loop, designed the glassmorphism web interface, and integrated multilingual text-to-speech services. Upstream model architecture originally based on academic sign-language conversion research; collaborated with Devansh Git on threading logic.

03 / METHODOLOGY

Analytical Approach

  1. 01

    Pipeline Architecture: Engineered an end-to-end pipeline: Webcam Capture → MediaPipe 21-Landmark Extraction → 400x400 White Canvas Normalization → CNN Cluster Classification → Geometric Heuristic Disambiguation → Sentence Buffer → Audio Synthesis.

  2. 02

    Thread-Safe Video Streaming: Implemented a background capture loop in Flask using the Double-Check Locking pattern and thread locks to eliminate camera frame race conditions and deliver smooth ~25 FPS MJPEG streams.

  3. 03

    Dual-Stage Gesture Disambiguation: Hand landmarks were first classified into 8 gesture clusters via a pre-trained Keras CNN (`cnn8grps_rad1_model.h5`). A second-stage geometric ruleset evaluated knuckle distance ratios and coordinate thresholds to disambiguate visually adjacent signs (e.g., distinguishing A from S/E/T/M/N, C from O, and G from H).

  4. 04

    NLP & Accessibility Integration: Connected recognized character sequences to PyEnchant dictionary suggestion engines for spelling prediction, with text-to-speech output via gTTS (English) and Google Translate API (Hindi).

04 / STACK

Verified Technologies

Backend & Web Framework
PythonFlaskMulti-ThreadingDouble-Check Locking
Computer Vision & AI
OpenCVMediaPipecvzoneTensorFlowKeras CNN Model
Frontend & Audio Services
HTML5CSS3 GlassmorphismgTTSGoogle Translate APIPyEnchant
05 / INSIGHTS

Verified Findings

Hybrid CNN + Geometric Rules Solve Gesture Ambiguity

Direct 26-class single-shot classification on raw webcam images frequently confounded visually similar letters (such as A, E, M, N, S, T). Grouping letters into 8 structural classes via CNN and applying secondary finger-joint distance thresholds yielded far higher practical stability under variable lighting.

Background Thread Isolation Prevents Server Deadlocks

Decoupling webcam read operations into a dedicated background daemon thread with explicit locking prevented Flask HTTP worker processes from blocking on video I/O, maintaining responsive API and dual-stream delivery.

Clear Distinction Between Finger-Spelling and Full Sign Language

The implementation demonstrates static finger-spelling (A–Z alphabet) plus control gestures (Next, Space, Backspace). It accurately highlights the boundary between static letter recognition and continuous multi-sign syntactic interpretation.

06 / VISUALS

Dashboard & UI

Gestura.ai Sign Language Translator Web Interface

The live Gestura.ai interface displaying real-time webcam video feed, extracted MediaPipe hand landmarks, predicted sign letter, sentence construction buffer, and audio playback controls.

07 / REFLECTION

What I Learned

  • Refactoring standalone desktop research scripts into a production-patterned, thread-safe web service.
  • Designing hybrid machine learning pipelines that combine deep learning with geometric domain rules.
  • Managing webcam hardware concurrency across multi-threaded HTTP servers.
  • Honest technical communication regarding model limitations, dataset boundaries, and project attribution.