Coop-SimLingo

Coop-SimLingo is the hint-enabled SimLingo baseline released with CoDrive, a closed-loop benchmark for temporally grounded cooperative vision-language driving.

Files

• coop_simlingo_fp32.pt: model-only FP32 PyTorch state dictionary.
• coop_simlingo_inference.yaml: frozen inference architecture.
• model_metadata.json: format and SHA256 metadata.
• SHA256SUMS: integrity checksums.

Usage

Clone CoDrive, create the pinned environment, and download this model through its helper:

git clone https://github.com/YongshuoLiu/CoDrive.git
cd CoDrive
conda env create -f environment-simlingo.yaml
conda activate simlingo
python -m pip install -e . --no-deps
bash codrive.sh setup

For one-image inference without CARLA:

bash codrive.sh predict \
  --image front_1024x512.jpg \
  --speed 6.2 \
  --target 15.0,0.2 \
  --next-target 30.0,0.4 \
  --scenario S1 \
  --front-car-pos 18.0,-1.2

A complete, runnable S1 record with the front image, numeric inputs, inference script, and best-005 reference output is available in the CoDrive example directory:

bash codrive.sh example --device cuda:0

The public Python API is codrive.CoopSimLingoRuntime. S1-S5 use the <FRONT_CAR_POS> token and its numeric ego-frame position. The canonical S6 cross-traffic warning is non-positional.

Inputs and cooperative hints

The model requires a rectified 1024x512 front image with 110-degree horizontal FoV, current speed in m/s, and two ego-frame navigation target points. The optional canonical hints are:

ID	Exact hint
S1	The pedestrian at position <FRONT_CAR_POS> is about to enter the ego lane.
S2	The vehicle at position <FRONT_CAR_POS> will cut into the ego lane soon.
S3	The static object at position <FRONT_CAR_POS> is blocking the ego lane.
S4	A wrong-way vehicle at position <FRONT_CAR_POS> will cross the ego left-turn path. Yield before turning left.
S5	A wrong-way vehicle at position <FRONT_CAR_POS> will cross the ego right-turn path. Yield before turning right.
S6	A vehicle approaching from the left will cross the ego right-turn path. Yield before turning right.

S1-S5 additionally require the referenced [forward, right] ego-frame position in meters. Positive values mean forward/right; negative lateral values mean left. For example, [18.0, -3.5] is 18 m ahead and 3.5 m left. The value is not a pixel, GPS/world coordinate, or distance-bearing pair. <FRONT_CAR_POS> is a hazard-anchor token and may refer to an actor or expected conflict point, not necessarily a front car. The special token must not be replaced with coordinate text; the numeric value is supplied separately through the CoDrive API.

Current speed is a non-negative scalar in meters per second, not km/h. The camera, speed, navigation targets, warning, and hazard position must describe the same timestamp. Full details and command examples are provided in docs/inference_interface.md in the CoDrive repository.

Scope

The model predicts future waypoints for recorded data, vehicle integration, and CARLA evaluation. It is not a physical-vehicle controller. Deployment requires independent localization, camera calibration, V2X message handling, waypoint control, collision monitoring, watchdogs, and emergency stopping. Follow docs/real_vehicle.md in the CoDrive repository.

Base model

The architecture uses SimLingo with OpenGVLab/InternVL2-1B. Users must comply with the licenses and terms of the upstream SimLingo and InternVL2 components.