MD Nahiduzzaman Dipu Python · Django · Machine Learning

Academic thesis AI / Machine Learning · 2026

PothPradarshak

A Real-Time Vision-Based Framework for Object Detection and Navigation Assistance for Visually Impaired Individuals

A YOLO-based vision assistant that turns Dhaka footpath scenes into spoken Bengali guidance

Type
Academic thesis
Status
Built
Year
2026
My role
Thesis author — dataset, training, evaluation and the real-time application
mAP@50
0.419
overall, held-out test split
mAP@50 · person
0.837
878 test boxes
mAP@50 · parked vehicle
0.781
1,360 test boxes
Throughput
~25 FPS
laptop RTX 3050 Ti
Verified images
2,782
hand-checked labels

01Overview

PothPradarshak (পথপ্রদর্শক, “the one who shows the way”) is my undergraduate thesis prototype: a real-time system that watches a camera feed, detects people, vehicles and footpath hazards, works out where they are relative to the walker, and speaks short guidance in Bengali or English. It is a research prototype, not a certified mobility aid.

  • YOLO11n detector trained on 2,782 hand-verified Dhaka footpath images
  • Turns each detection into left / centre / right and near / far, then ranks by risk
  • Spoken guidance in Bengali and English, with cooldowns so it never chatters
  • Live camera, image upload and video modes served through FastAPI

02The problem

Off-the-shelf detectors are trained on datasets that barely represent a Dhaka footpath — parked vehicles on the pavement, broken curbs, street obstacles — and their output is a list of boxes, not something a person can act on while walking. A useful assistant has to detect the right things, decide which one matters most right now, and say it briefly without talking over itself.

03Architecture

I built both halves: a data pipeline that turns an unlabelled image folder into a verified, trainable dataset, and a runtime that goes from camera frame to spoken instruction. A YOLO11n detector is trained on footpath classes; a spatial layer converts each box into LEFT / CENTER / RIGHT and NEAR / MEDIUM / FAR; a navigation engine scores every detection and only speaks when something new or more urgent happens.

  1. 01CameraThreaded capture, newest frame wins
  2. 02DetectionYOLO11n, footpath classes
  3. 03Spatial analysisLeft / centre / right · near / medium / far
  4. 04Navigation enginePriority score, events, cooldown
  5. 05VoiceBengali (edge-tts) · English (pyttsx3)
  6. 06Web UIFastAPI · MJPEG stream · JSON API

Runtime: camera → frame → YOLO detector → spatial analysis → navigation engine → voice engine → web UI. Three threads keep it responsive — camera capture (newest frame wins, auto-reconnect), the vision pipeline, and text-to-speech synthesis — so a slow voice call never stalls detection.

The detector is isolated in a single module, the only one that imports Ultralytics, so the model can be swapped without touching the rest. A FastAPI backend serves the annotated video as an MJPEG stream and exposes detections and uploads (image or video analysis) over a JSON API.

Training side: build annotation set → auto-label → human review → prepare dataset (70/20/10 split) → train → validate. Each step is its own script so a stage can be re-run without redoing the others.

04Technologies

  • Python
  • PyTorch
  • Ultralytics YOLO11
  • YOLO-World
  • FastAPI
  • edge-tts
  • pyttsx3
  • CUDA

05Implementation

  • Dataset: started from 21,359 raw, unannotated images; selected the Dhaka footpath group and removed near-duplicates with perceptual hashing, leaving 2,787 images.
  • Labelling: YOLO-World (open-vocabulary) proposed 24,010 candidate boxes in 5.6 minutes; 3,454 mechanical errors were filtered automatically (cross-prompt duplicates, whole-image boxes, specks) and the rest were verified by hand — 2,782 images exported.
  • Training: YOLO11n, 100 epochs, 640 px, batch 8 on a laptop RTX 3050 Ti with the CUDA build of PyTorch (about 11× faster than CPU on the same machine).
  • Priority scoring: danger class +100, centre of path +50, near +30, medium +12, with confidence as the tie-break — so a nearby hazard in the walker's path always outranks a distant person off to the side.
  • Speech control: the system speaks on a new object, worsening proximity, or movement into the path, with a per-object cooldown (3 s) and a global minimum gap (1.2 s) so it never chatters.
  • Voice: Bengali through edge-tts (bn-BD-NabanitaNeural), English offline through pyttsx3, with the browser's Web Speech API as a fallback.
  • A second, COCO-pretrained model covers twelve indoor classes (chair, bed, sink…). Those are reported separately and carry no thesis-measured accuracy.

06Challenges

  • No labels at all. The image folder came with no annotations, so the labelling pipeline had to be built before any training could happen.
  • Severe class imbalance. Parked vehicles and people make up over 92% of the candidate boxes; the rarest classes are well under 1% each. The model learned what it was shown.
  • Being honest about tiny test sets. `stairs` scores mAP@50 0.995 — on two test boxes. That is noise, not a result, and the write-up says so.
  • Speech that helps rather than distracts. Announcing every detection is useless; the priority and cooldown logic took more iteration than the detector did.
  • Fitting training on a 4 GB laptop GPU, which constrained model size and batch.

07Results

The full pipeline works end to end: live camera, uploaded image and uploaded video all run through detection, spatial reasoning and Bengali/English voice guidance at about 25 FPS on a laptop RTX 3050 Ti. On the held-out test split the trained model reaches precision 0.645, recall 0.404, mAP@50 0.419 and mAP@50-95 0.297 overall. It is genuinely good at the two well-represented classes — person (mAP@50 0.837) and parked vehicle (0.781) — and weaker on the rare classes, which have few examples.

mAP@50 by class Results for seven classes on the held-out test split — YOLO11n, 100 epochs, 640 px.
  • person 0.837 n = 878
  • parked_vehicle 0.781 n = 1360
  • curb 0.359 n = 20
  • street_obstacle 0.295 n = 99
  • pole 0.268 n = 9
  • tree 0.237 n = 65
  • stairs 0.995 n = 2

Hatched bars have fewer than 10 test boxes. Stairs (two test boxes) and pole (nine) are too small a sample to read as evidence — a high score there is noise, not a result.

Results for seven classes on the held-out test split — YOLO11n, 100 epochs, 640 px.
ClassPrecisionRecallmAP@50mAP@50-95Test boxes
person0.8110.7490.8370.6878
parked_vehicle0.7580.6880.7810.5281360
curb0.4130.450.3590.29220
street_obstacle0.4040.2670.2950.19799
pole0.2630.2220.2680.2219
tree0.3720.2620.2370.13865
stairs0.7851.00.9950.6972

08Limitations

  • Rare classes are weak. Accuracy drops sharply for classes with few training examples; fixing that needs more verified data per class, not more epochs.
  • Proximity is estimated from bounding-box area, not measured distance.
  • No object tracking between frames.
  • Single camera and viewpoint — no stereo, IMU or GPS.
  • Daylight data only; night performance is untested.
  • Bengali speech currently needs an internet connection.
  • Runs on a laptop GPU. A real aid needs an embedded target and an end-to-end latency budget.
  • Research prototype: it must not be relied on as someone's only navigation aid.

09Screenshots

Taken from the project's own repository (view on GitHub).

↑↓ to move · Enter to run · Esc to close