metadata
theme: default
title: Week 11 — Incremental Progress Update
info: |
Converted from “Week 6 Template (1).pptx”.
Original slide order, wording, and images are preserved.
author: Hamim Mahmud
canvasWidth: 1920
aspectRatio: 16/9
transition: slide-left
fonts:
sans: Ruda
weights: 400,500,600,700,800,900
provider: google
mdc: true
CE 400
Hamim Mahmud
2104065
Incremental Progress
Aug 23, 2026
Week progress
- Acquired Drone
- Captured preliminary drone footage for trajectory
analysis pipeline
- Took preliminary drone footage for photogrammetry
- Application for acquiring CCTV footage permission from
DMP
- Software selection for photogrammetry - (ongoing)
Week progress
- Vehicle Trajectory Extraction - (ongoing)
- Video Stabilization
- Vehicle segmentation
- Orientation Extraction
- Infrastructure Development
- Tunneling pipeline
- SAM 3.1 MCP server
- Cotracker MCP server
- Integration with kaggle for Agentic Workflow
- Model: DJI mini 2 (used)
- Provider: DJI
- Total cost: 32600 taka
Brief
The science of extracting 3D geometric information from 2D
images
Photogrammetry
- Subject: Polashi Intersection
- Captured amount: 151
This images are only for selecting pipeline for photogrammetry
Existing Software
Paid:
- Agisoft Metashape
- RealityCapture
- 3DF Zephyr
- Bentley iTwin Capture
Modeler
Current preference: COLMAP
Point Cloud
Trajectory Analysis & Surrogate Safety
Location: Bakshi Bazaar
Keypoint Tracking — CoTracker3
Estimating camera motion from the raw drone footage
- Dense 16×16 grid — 32,400 tracked points per frame
- 7,856 frames (≈ 5.5 min) at 3840×2160, 23.976 fps
- Point tracks are the input for both stabilization and motion analysis
Camera Motion Analysis
Derived from the CoTracker3 tracks
Path length
3,280 px
Mean speed
0.418 px/frame
Total yaw
+0.74°
Net scale
×1.0017
Drift is small but continuous — hover is not stable enough for direct trajectory extraction
Video Stabilization
- Per-frame homography fitted from the tracked points
- Camera drift, yaw and zoom compensated back to the frame-0 reference
- Result: a fixed world frame, so pixel coordinates map consistently across the clip
- Prerequisite for metric trajectory extraction
Preview: Polashi Intersection, stabilized output
Vehicle Segmentation — Meta SAM
Instance masks on the stabilized frames
- Prompt-driven masks for vehicles, road surface and vegetation
- Mask centroid and principal axis give position and orientation per vehicle
- Next step: linking masks across frames into continuous trajectories
Infrastructure — Tunneling Pipeline
TunnelMate: exposing GPU-side model servers to the agent
- SAM 3.1 and CoTracker run as MCP servers on Kaggle GPU sessions
- Tunnels publish those endpoints; the broker tracks state, traffic and expiry
- Removes manual re-hosting every time a session restarts
Agentic Workflow
- OpenCode drives the MCP servers end to end — no manual step per clip
- Health checks, job queueing and restart handled by the agent
- Batch in progress: 125 images × 7 prompts
- Frees the pipeline to run unattended between sessions
Thank You
class: week11-light
CE 400
Hamim Mahmud
2104065
Incremental Progress
Week 11
Sep 6, 2026
class: week11-light
Week 11 progress · Research focus
Week 11 — Research Focus
3D vehicle detection and world-location extraction from roadside cameras
Benchmarking BEV models for Dhaka City
Local road reconstruction supports the experiments. The research outcome is evidence about model performance in Dhaka.
Own stabilized drone footage — local traffic scene
class: week11-light
Week 11 Progress
- Captured and stabilized two new drone videos
- Fixed PLY point cloud color display in Blender
for the Gaussian splatting workflow
- Conducted preliminary tests on previously acquired
photogrammetry data under different weather conditions
- Identified the need to recapture data and improve
point cloud density
class: week11-light
Week 11 progress · Research motivation
How the earlier safety topic connects
Earlier question
Can CCTV footage reveal vehicle conflicts?
→
Required measurements
Where are road users, and how do their paths change?
→
Current thesis
Which BEV models estimate these spatial relationships reliably in Dhaka?
Existing work: road-user segmentation on aerial footage
Perception comes before safety analysis
A pretrained BEV model could reduce part of the manual perception effort. Tracking, calibration and conflict analysis still need their own validation.
The long-term motivation remains road safety; the immediate contribution is a local benchmark.
class: week11-light
Week 11 progress · Research background
What does “BEV” mean?
Bird’s-eye view
A common top-down coordinate system for describing roads, vehicle positions and movement.
- Input: camera images and calibration
- Thesis output: 3D vehicle boxes and world-coordinate locations
- Over time: positions can support tracking
A drone photo is an observation from above. A BEV model estimates a structured representation from its camera inputs.
Illustrative BEV output; not a prediction from this thesis
class: week11-light
Week 11 progress · Research question
What exactly will the benchmark answer?
Proposed research question
How accurately does BEVHeight detect vehicles in 3D and locate them in Dhaka road coordinates?
- Compare on the same held-out scenes
- Report where each model succeeds or fails
- Measure accuracy and computation cost
Primary model: BEVHeight
A vision-based roadside 3D detector. Evaluate local vehicle classes, occlusion, camera geometry, day/night and fog.
Scope still to finalize
Additional comparison models, class mapping, annotation protocol and acceptance criteria remain to be finalized.
A model ranking and documented limitations are useful outcomes—even if no model is reliable in every condition.
class: week11-light
Week 11 progress · BEVHeight
What is BEVHeight?
Roadside image
Observe vehicles from an elevated camera.
→
Height estimation
Learn image features and height above the ground.
→
BEV representation
Use calibrated geometry to place features in a top-down grid.
→
3D detection
Predict vehicle boxes, classes and confidence scores.
Why it fits this thesis
BEVHeight is designed for roadside 3D object detection. Its height-based view transformation is relevant to elevated CCTV viewpoints.
The benchmark asks whether a pretrained model transfers reliably to Dhaka scenes and local vehicle classes.
Concept illustration, not BEVHeight predictions
class: week11-light
Week 11 progress · BEVHeight inputs
What inputs does BEVHeight need?
| Input | Plain-language meaning | Source in this thesis |
|---|
| RGB image | A roadside camera frame | CCTV or a rendered virtual CCTV view |
| Camera intrinsics | Focal lengths and image centre | Camera calibration or Blender camera settings |
| Camera pose | Camera orientation and location in the reference frame | Measured/calibrated pose or virtual camera transform |
| Ground reference | Ground alignment and reference camera height | Calibrated road plane or known scene geometry |
| Image transforms | How resizing, cropping or other preprocessing changed the image | Dataset preparation pipeline |
LiDAR is not required at inference. The adapter must convert Blender axes, units and camera conventions to those expected by BEVHeight.
class: week11-light
Week 11 progress · Camera calibration
Intrinsics: how the camera forms an image
Parameters inside the camera model
- fₓ, fᵧ: focal lengths in pixels
- cₓ, cᵧ: principal point
- Lens distortion: bending introduced by the lens
K = [ fₓ 0 cₓ ]
[ 0 fᵧ cᵧ ]
[ 0 0 1 ]
Resizing changes pixel-based intrinsics; cropping shifts the principal point. Keep calibration consistent with the image supplied to BEVHeight.
class: week11-light
Week 11 progress · Camera calibration
Extrinsics: where the camera is and where it points
Transform between coordinate frames
- R: rotation from world to camera axes
- t: translation in that transformation
Xcamera = R Xworld + t
Camera centre in world coordinates:
C = −Rᵀt
Moving or tilting a camera changes its extrinsics. Translation t is not the camera’s world position.
class: week11-light
Week 11 progress · Camera calibration
Calibration connects Blender, CCTV and BEVHeight
World point
Vehicle location in metres
→ R, t →
Camera point
Location relative to the camera
→ K →
Image pixel
Projected location after perspective division
Real CCTV — proposed preparation
Estimate lens calibration and pose against measured road references. Check projected landmarks on the image.
Blender — proposed export
Derive intrinsics from camera/render settings. Convert camera transforms to the model’s axes and units; export calibration with every view.
A pixel defines a viewing ray, not a unique 3D location. BEVHeight supplies learned height information; calibrated geometry connects it to the road frame.
---
class: week11-light
---
Week 11 progress · World locations
From a camera image to a location on the road
CCTV image
Vehicle appears in perspective; pixels alone are not metres.
→
BEVHeight
Predict a 3D vehicle box in the model’s coordinate convention.
→
Calibrated transform
Convert the predicted location to one shared road coordinate frame.
→
World location
Vehicle position in metres, ready to compare with the reference.
Use one location definition
For example, the ground projection of the 3D box centre. Use the same point for predictions and reference labels.
Local road coordinates are sufficient for evaluation. Latitude/longitude requires additional georeferencing.
class: week11-light
Week 11 progress · Inputs and labels
Model inputs and reference labels have different roles
Prediction branch
RGB image + calibration
↓
Pretrained BEVHeight
↓
Predicted 3D box and confidence
↓
Location in the shared road frame
Reference branch
Known simulated vehicle geometry and pose
or reviewed real-data annotations
↓
Reference boxes and locations
↓
Compare with the model’s predictions
Labels are targets during supervised training and references during evaluation. They are not supplied to the model as answers at inference.
Nadir imagery supports real-world reference construction; the primary detection input is the roadside view. Map Dhaka vehicle categories to supported model classes explicitly.
class: week11-light
Week 11 progress · Reconstruction method
Why reconstruct roads without LiDAR?
Own Blender point-cloud scene of Polashi Intersection
Available resource: camera imagery
Overlapping drone images can recover camera poses and scene structure. A reconstructed local scene can support repeatable viewpoint experiments.
- Capture local geometry and appearance
- Reuse the scene for controlled rendering
- Investigate camera placement and visibility
Metric distances need scale control, such as measured reference distances. Reconstruction accuracy must be checked independently.
class: week11-light
Week 11 progress · Reconstruction method
Where COLMAP and Gaussian splatting fit
1 · Capture
Overlapping images of an intersection or road
→
2 · COLMAP
Estimate camera poses and 3D points
→
3 · 3D Gaussian splatting
Optimize a scene representation from images and poses
→
4 · Render
Generate views for controlled experiments
Earlier reconstruction: point cloud before the color fix
Current evidence and next milestone
Current slides demonstrate a colored Blender point cloud and scene edits. Record the actual Gaussian-splat training and rendering outputs as a separate milestone.
COLMAP reconstructs geometry. Gaussian splatting adds learned appearance for novel views. Neither automatically creates BEV labels or moving traffic.
class: week11-light
Week 11 progress · Evaluation plan
Two complementary evaluation routes
A · Real Dhaka data
Synchronized CCTV + nadir drone views
↓
Calibrated and reviewed reference labels
↓
Model predictions
↓
Errors on unseen local scenes
Real-world cross-check of detection and location
B · Reconstructed scenes
Gaussian-splat background + labeled vehicles
↓
Known virtual cameras and object transforms
↓
Model predictions
↓
Stress-test sensitivity
Exact simulation labels; real-world validity still tested
Split by intersection or capture session before reconstruction or model tuning. Nearby frames from the same scene must not leak across train and test sets.
class: week11-light
Week 11 progress · Ground truth
How the proposed ground truth is constructed
Simulation: known 3D labels
- Scale and align the reconstructed road.
- Add vehicles with known dimensions, pose and object IDs.
- Render CCTV-like views with known camera calibration.
- Export boxes and world positions directly from scene transforms.
The scene’s object metadata supplies labels. Gaussian-splat appearance alone does not.
Real footage: nadir + CCTV
- Synchronize captures of the same traffic event.
- Calibrate the nadir view to measured road coordinates.
- Match and annotate vehicles across both views.
- Compare CCTV predictions with the aligned reference positions.
Nadir views support planar position labels. Full 3D boxes need dimensions, height and orientation evidence.
Keep simulation scores and real-data scores separate. Report reference uncertainty and exclude unmatched or unobservable cases using a documented rule.
class: week11-light
Week 11 progress · Benchmark protocol
Make the comparison fair and measurable
| Decision | Proposed evaluation rule |
|---|
| Camera input | Match each model’s required camera count, calibration and viewpoint. |
| Primary task | 3D vehicle detection and world-location extraction; tracking is downstream. |
| Reference labels | Use independent annotations and measured scale; document uncertainty. |
| Metrics | 3D/BEV average precision; matched-vehicle location error in metres; recall. |
| Conditions and cost | Report results by scene/condition, plus runtime on the same hardware. |
BEVHeight targets roadside cameras. Match its calibration, ground-plane convention and class definitions; use nadir views as a reference source.
class: week11-light
Week 11 progress · Synthetic dataset
Blender as a synthetic dataset generator
Build the scene
Scaled local road background + separate vehicle assets
→
Set the scenario
Choose cameras, vehicle poses, lighting and fog
→
Render and export
Save RGB views, camera data and object labels
→
Check and package
Validate labels and convert to the BEVHeight dataset format
Existing scene foundation; automatic dataset export is proposed
One scene, paired observations
Render a roadside view and a nadir view of the same simulation state. Both share the same vehicle IDs and world coordinates.
Use explicit vehicle objects for labels. Remove or mask vehicles already captured in the static splat background.
class: week11-light
Week 11 progress · Dataset variations
Generate many variations systematically
| Variation | Examples to control | What the benchmark can test |
|---|
| Camera | Height, tilt, position and focal length | Sensitivity to viewpoint and calibration |
| Vehicles | Type, dimensions, orientation, count and placement | Class coverage, distance and occlusion |
| Lighting | HDRI, sun direction and night lights | Performance under appearance changes |
| Visibility | No fog, low fog and high fog | Detection loss as visibility declines |
| Image quality | Exposure, blur and compression | Sensitivity to CCTV image degradation |
Illustrative plan: 5 camera poses × 3 lighting settings × 3 fog levels × 20 traffic layouts = 900 roadside images per scene.
Save random seeds and settings. Keep all variants of a scene in the same dataset split; 900 renders are not 900 independent real scenes.
class: week11-light
Week 11 progress · Dataset validation
Every rendered sample needs matching labels
Export a complete sample
- RGB frame, scene ID and scenario settings
- Camera intrinsics, pose and ground reference
- Vehicle ID, class, 3D dimensions and orientation
- World position using a consistent anchor point
- Optional depth and object masks for visibility checks
Validate before benchmarking
- Project 3D box corners onto the RGB image.
- Check metres, axes and camera alignment.
- Flag truncation and heavy occlusion.
- Keep held-out real CCTV/nadir data for transfer testing.
Ordinary Gaussian splats contain captured appearance. Physically consistent relighting, shadows and fog may require a compatible renderer or a hybrid scene with conventional geometry; verify the integration.
Start with a small, manually checked batch. Scale generation only after images, calibration and labels agree.
Blender capabilities:
Render passes and object masks. Dataset workflow and variation counts below are proposed for this thesis. 3D boxes require a custom metadata exporter; RGB rendering alone does not produce them.
class: week11-light
Revisiting the Color Problem
Revisiting slide 8 from the earlier progress update: the reconstructed point cloud appeared without its original colors.
Week 11: restored color display with fix_ply_colors.py, then refined the materials and scene manually.
class: week11-light
Point Cloud Color Fix & Scene Refinement
Initial script setup
fix_ply_colors.py
- Converted imported PLY geometry to a Blender point cloud
- Connected the Col color attribute to an emission material
- Added adjustable point radius through Geometry Nodes
- Configured the camera and saved the Blender scene
Manual changes after the script
- Replaced the emission shader with Principled BSDF
- Added a sky HDRI to the world environment
- Added fog using Principled Volume
- Removed some reconstruction artifacts
class: week11-light
Preliminary Weather & Lighting Tests
- Input: previously acquired photogrammetry data
- Lighting conditions: day and night
- Weather conditions: fog and no fog
- Status: preliminary tests conducted
class: week11-light
Daytime — No Fog / Intense Fog
No fog
Intense fog
Preliminary Blender tests using previously acquired photogrammetry data
class: week11-light
Night — Fog Comparison
Preliminary renders under simulated night lighting and fog
class: week11-light
Daytime Fog — Visual Reference
Blender — intense fog
Real photograph — intense fog
Qualitative appearance reference; different scenes and viewpoints
class: week11-light
Week 11 progress · Interpreting the tests
What the weather experiments contribute
Own Blender render · night / no fog
Own Blender render · night / low fog
Own Blender render · night / high fog
Question to test
How much does model accuracy change as visibility worsens in the same scene?
What still needs validation
Document fog and lighting settings, and keep labeled vehicle positions fixed. These images demonstrate scene setup; detection accuracy is not yet measured.
The splat supplies the road background. Add independently labeled 3D vehicles to evaluate detection and world location; avoid unlabeled vehicles baked into the background.
class: week11-light
New Drone Capture — DJI_0403
Newly captured footage, stabilized for trajectory analysis
Stabilization preview · First 20 seconds
class: week11-light
New Drone Capture — DJI_0406
Newly captured footage, stabilized for trajectory analysis
Stabilization preview · First 20 seconds
class: week11-light
Week 11 progress · Future safety application
After validation: return to safety analysis
BEV perception
Road-user locations in a metric frame
→
Tracking
Paths, speed and timing
→
Conflict measures
Time-to-collision / post-encroachment time
→
Engineering review
Find recurring conflict locations
Illustration of paths in a shared coordinate system
Example: crossing movements
Estimate when two road users approach or pass through a shared conflict area. Review recurring interactions by location and time.
Requires validated trajectories and timestamps. Conflict indicators are surrogate measures; they do not directly prove a reduction in crashes.
class: week11-light
Week 11 progress · Bangladesh applications
Where this could be used in Bangladesh
| Potential setting / user | Possible use after validation | Decision supported |
|---|
| Dhaka intersections · traffic police | Turning counts, queues and recurring conflicts | Review signal timing and traffic control |
| Urban streets · city corporations | Pedestrian–vehicle interaction maps | Prioritize crossing and junction audits |
| Highway junctions · RHD | Site-level movement and visibility studies | Prioritize locations for engineering inspection |
| BRTA / university research teams | Reusable local datasets and model comparisons | Choose models based on local evidence |
| Transport planning teams | Reconstructed scenes and observed traffic overlays | Communicate design options and test scenarios |
These are proposed applications, not current deployments or agency partnerships.
class: week11-light
Week 11 progress · Digital twin potential
A path toward a traffic digital twin
Scene foundation
Reconstructed road geometry and appearance
→
Traffic layer
Validated positions and trajectories
→
Updated system
Connect observations over time
→
Planning tools
Add and validate scenario simulation
Current scene foundation — Blender reconstruction
Example future use
Overlay observed turning movements on a local intersection, then study a proposed crossing or signal plan using a validated traffic simulation.
The current 3D scene is a foundation. A live digital twin also needs updated data, traffic behavior and validation of its predictions.
class: week11-light
Week 11 progress · Thesis deliverables
What a successful thesis will deliver
Core research deliverables
- A documented Dhaka evaluation dataset and labeling protocol
- A reproducible comparison of selected compatible BEV models
- Results by scene and condition, with failure examples
- Evidence on the usefulness and limits of reconstructed scenes
Current work → next evidence
Completed groundwork
Drone capture and stabilization; point-cloud color fix; manual materials, HDRI, fog and artifact cleanup.
Next
Set up BEVHeight; align CCTV and nadir references; add labeled simulation vehicles; verify reconstruction; run the held-out benchmark.
My thesis asks which BEV models are reliable for Dhaka. The reconstruction is the experimental support; safety analysis and digital twins are downstream opportunities.
class: week11-light
Next Steps
- Recapture imagery and improve the COLMAP reconstruction
- Verify Gaussian-splat rendering and labeled Blender scenarios
- Calibrate and align CCTV and nadir reference data
- Prepare BEVHeight and run the held-out detection benchmark
class: week11-light
Week 11 progress · Discussion
Thank You
Week 11 Progress — Discussion
BEVHeight baseline · simulation labels · CCTV–nadir alignment
