Academic thesis AI / Machine Learning · 2026
PothPradarshak
A Real-Time Vision-Based Framework for Object Detection and Navigation Assistance for Visually Impaired Individuals
A YOLO-based vision assistant that turns Dhaka footpath scenes into spoken Bengali guidance
- Type
- Academic thesis
- Status
- Built
- Year
- 2026
- My role
- Thesis author — dataset, training, evaluation and the real-time application
- mAP@50
- 0.419 overall, held-out test split
- mAP@50 · person
- 0.837 878 test boxes
- mAP@50 · parked vehicle
- 0.781 1,360 test boxes
- Throughput
- ~25 FPS laptop RTX 3050 Ti
- Verified images
- 2,782 hand-checked labels
01Overview
PothPradarshak (পথপ্রদর্শক, “the one who shows the way”) is my undergraduate thesis prototype: a real-time system that watches a camera feed, detects people, vehicles and footpath hazards, works out where they are relative to the walker, and speaks short guidance in Bengali or English. It is a research prototype, not a certified mobility aid.
- YOLO11n detector trained on 2,782 hand-verified Dhaka footpath images
- Turns each detection into left / centre / right and near / far, then ranks by risk
- Spoken guidance in Bengali and English, with cooldowns so it never chatters
- Live camera, image upload and video modes served through FastAPI
02The problem
Off-the-shelf detectors are trained on datasets that barely represent a Dhaka footpath — parked vehicles on the pavement, broken curbs, street obstacles — and their output is a list of boxes, not something a person can act on while walking. A useful assistant has to detect the right things, decide which one matters most right now, and say it briefly without talking over itself.
03Architecture
I built both halves: a data pipeline that turns an unlabelled image folder into a verified, trainable dataset, and a runtime that goes from camera frame to spoken instruction. A YOLO11n detector is trained on footpath classes; a spatial layer converts each box into LEFT / CENTER / RIGHT and NEAR / MEDIUM / FAR; a navigation engine scores every detection and only speaks when something new or more urgent happens.
- 01CameraThreaded capture, newest frame wins
- 02DetectionYOLO11n, footpath classes
- 03Spatial analysisLeft / centre / right · near / medium / far
- 04Navigation enginePriority score, events, cooldown
- 05VoiceBengali (edge-tts) · English (pyttsx3)
- 06Web UIFastAPI · MJPEG stream · JSON API
Runtime: camera → frame → YOLO detector → spatial analysis → navigation engine → voice engine → web UI. Three threads keep it responsive — camera capture (newest frame wins, auto-reconnect), the vision pipeline, and text-to-speech synthesis — so a slow voice call never stalls detection.
The detector is isolated in a single module, the only one that imports Ultralytics, so the model can be swapped without touching the rest. A FastAPI backend serves the annotated video as an MJPEG stream and exposes detections and uploads (image or video analysis) over a JSON API.
Training side: build annotation set → auto-label → human review → prepare dataset (70/20/10 split) → train → validate. Each step is its own script so a stage can be re-run without redoing the others.
04Technologies
05Implementation
- Dataset: started from 21,359 raw, unannotated images; selected the Dhaka footpath group and removed near-duplicates with perceptual hashing, leaving 2,787 images.
- Labelling: YOLO-World (open-vocabulary) proposed 24,010 candidate boxes in 5.6 minutes; 3,454 mechanical errors were filtered automatically (cross-prompt duplicates, whole-image boxes, specks) and the rest were verified by hand — 2,782 images exported.
- Training: YOLO11n, 100 epochs, 640 px, batch 8 on a laptop RTX 3050 Ti with the CUDA build of PyTorch (about 11× faster than CPU on the same machine).
- Priority scoring: danger class +100, centre of path +50, near +30, medium +12, with confidence as the tie-break — so a nearby hazard in the walker's path always outranks a distant person off to the side.
- Speech control: the system speaks on a new object, worsening proximity, or movement into the path, with a per-object cooldown (3 s) and a global minimum gap (1.2 s) so it never chatters.
- Voice: Bengali through edge-tts (bn-BD-NabanitaNeural), English offline through pyttsx3, with the browser's Web Speech API as a fallback.
- A second, COCO-pretrained model covers twelve indoor classes (chair, bed, sink…). Those are reported separately and carry no thesis-measured accuracy.
06Challenges
- No labels at all. The image folder came with no annotations, so the labelling pipeline had to be built before any training could happen.
- Severe class imbalance. Parked vehicles and people make up over 92% of the candidate boxes; the rarest classes are well under 1% each. The model learned what it was shown.
- Being honest about tiny test sets. `stairs` scores mAP@50 0.995 — on two test boxes. That is noise, not a result, and the write-up says so.
- Speech that helps rather than distracts. Announcing every detection is useless; the priority and cooldown logic took more iteration than the detector did.
- Fitting training on a 4 GB laptop GPU, which constrained model size and batch.
07Results
The full pipeline works end to end: live camera, uploaded image and uploaded video all run through detection, spatial reasoning and Bengali/English voice guidance at about 25 FPS on a laptop RTX 3050 Ti. On the held-out test split the trained model reaches precision 0.645, recall 0.404, mAP@50 0.419 and mAP@50-95 0.297 overall. It is genuinely good at the two well-represented classes — person (mAP@50 0.837) and parked vehicle (0.781) — and weaker on the rare classes, which have few examples.
- person 0.837 n = 878
- parked_vehicle 0.781 n = 1360
- curb 0.359 n = 20
- street_obstacle 0.295 n = 99
- pole 0.268 n = 9
- tree 0.237 n = 65
- stairs 0.995 n = 2
Hatched bars have fewer than 10 test boxes. Stairs (two test boxes) and pole (nine) are too small a sample to read as evidence — a high score there is noise, not a result.
| Class | Precision | Recall | mAP@50 | mAP@50-95 | Test boxes |
|---|---|---|---|---|---|
| person | 0.811 | 0.749 | 0.837 | 0.6 | 878 |
| parked_vehicle | 0.758 | 0.688 | 0.781 | 0.528 | 1360 |
| curb | 0.413 | 0.45 | 0.359 | 0.292 | 20 |
| street_obstacle | 0.404 | 0.267 | 0.295 | 0.197 | 99 |
| pole | 0.263 | 0.222 | 0.268 | 0.221 | 9 |
| tree | 0.372 | 0.262 | 0.237 | 0.138 | 65 |
| stairs | 0.785 | 1.0 | 0.995 | 0.697 | 2 |
08Limitations
- Rare classes are weak. Accuracy drops sharply for classes with few training examples; fixing that needs more verified data per class, not more epochs.
- Proximity is estimated from bounding-box area, not measured distance.
- No object tracking between frames.
- Single camera and viewpoint — no stereo, IMU or GPS.
- Daylight data only; night performance is untested.
- Bengali speech currently needs an internet connection.
- Runs on a laptop GPU. A real aid needs an embedded target and an end-to-end latency budget.
- Research prototype: it must not be relied on as someone's only navigation aid.
09Screenshots
Taken from the project's own repository (view on GitHub).