Spaces:
Running on Zero
Running on Zero
| { | |
| "among": { | |
| "id": "among_group582_q0_5_1", | |
| "question": "Based on these four images (image 1, 2, 3, and 4) showing the cup from different viewpoints (front, left, back, and right), with each camera aligned with room walls and partially capturing the surroundings: From the viewpoint presented in image 1, what is to the left of the cup? A. White board B. TV C. Metal lockers D. Wooden podium", | |
| "gt_answer": "A", | |
| "cogmap_instruction": "[Task]\nYour task is to analyze the spatial arrangement of objects in the scene by examining the provided images, which show the scene from different viewpoints. You will then create a detailed cognitive map representing the scene using a **10x10 grid coordinate system**. \n[Rules]\n1. Focus ONLY on these categories of objects in the scene: {cup, white board, metal lockers, wooden podium, TV}\n2. Create a cognitive map with the following structure in the bird's view:\n - A 10x10 grid where [0, 0] is at the top-left corner and [9, 9] is at the bottom-right corner\n - up = towards the top of the grid (decreasing y)\n - right = towards the right of the grid (increasing x)\n - down = towards the bottom of the grid (increasing y)\n - left = towards the left of the grid (decreasing x)\n - Include positions of all objects from the specified categories\n - Estimate the center location (coordinates [x, y]) of each instance within provided categories\n - If a category contains multiple instances, include all of them\n - Object positions must maintain accurate relative spatial relationships\n - Combine and merge information from the images since they are pointing to the same scene, calibrating the object locations with grid coordinates accordingly\n3. Carefully integrate information from all views to create a single coherent spatial representation.\n", | |
| "images": [ | |
| "examples/among/view1.jpg", | |
| "examples/among/view2.jpg", | |
| "examples/among/view3.jpg", | |
| "examples/among/view4.jpg" | |
| ], | |
| "prompt": "[Task]\nGiven multiple images showing a 3D scene from different viewpoints, reason carefully about the spatial layout and answer the question.\n\n[Answer Instruction]\nOutput 4 blocks in order:\n 1. <cogmap>: Bird's-eye map with all object positions and view positions/facings.\n 2. <egomap>: Multi-view ego-centric maps \u2014 a JSON array with one ego map per view.\n Each ego map includes \"view\", \"position\", \"facing\", \"is_primary\", per-object \"ego_direction\".\n The primary view (question-referenced, after any actions) also includes \"answer_object\" and \"answer_visible_in_view\".\n 3. <think>: 4-part reasoning with multi-view ego maps:\n Part 1 \u2014 Scene layout: list all objects with allocentric positions, then all views.\n Scene: obj1 [x1,y1], obj2 [x2,y2], ...\n Views: Image 1 [x,y] facing=dir, Image 2 [x,y] facing=dir, ...\n Part 2 \u2014 Primary view selection: identify the question-relevant viewpoint.\n If actions exist: Selected: Image X [pos] facing=dir \u2192 action \u2192 [eff_pos] facing=eff_dir\n Then axis mapping: Axes: forward=..., right=...\n If no actions: Selected: Image X [pos] facing=dir\n Part 3 \u2014 Multi-view ego maps: compute ego-centric layout from EACH view.\n For the primary view (after any actions):\n Ego from [eff_pos] facing=eff_dir (primary, after action):\n obj \u0394[dx,dy]=ego_dir [Image X]\n For each secondary view:\n Ego from Image Y [pos] facing=dir:\n obj \u0394[dx,dy]=ego_dir [Image X]\n This provides complete spatial context from all perspectives.\n Part 4 \u2014 Elimination: using the PRIMARY ego map directions, check each option A/B/C/D.\n target=dir valid={...}\n A. name=ego_dir [Image X] \u2713/\u2717\n 4. <answer>: Final answer as \"X. option_text\".\nFormat: '<cogmap>...</cogmap><egomap>...</egomap><think>...</think><answer>X. option_text</answer>'\nMake sure `X. option_text` matches exactly one of the options in the question.\n\n[Task]\nYour task is to analyze the spatial arrangement of objects in the scene by examining the provided images, which show the scene from different viewpoints. You will then create a detailed cognitive map representing the scene using a **10x10 grid coordinate system**. \n[Rules]\n1. Focus ONLY on these categories of objects in the scene: {cup, white board, metal lockers, wooden podium, TV}\n2. Create a cognitive map with the following structure in the bird's view:\n - A 10x10 grid where [0, 0] is at the top-left corner and [9, 9] is at the bottom-right corner\n - up = towards the top of the grid (decreasing y)\n - right = towards the right of the grid (increasing x)\n - down = towards the bottom of the grid (increasing y)\n - left = towards the left of the grid (decreasing x)\n - Include positions of all objects from the specified categories\n - Estimate the center location (coordinates [x, y]) of each instance within provided categories\n - If a category contains multiple instances, include all of them\n - Object positions must maintain accurate relative spatial relationships\n - Combine and merge information from the images since they are pointing to the same scene, calibrating the object locations with grid coordinates accordingly\n3. Carefully integrate information from all views to create a single coherent spatial representation.\n\n\n[Multi-View Ego Map Instruction]\nGenerate ego-centric maps from the cognitive map for ALL views in the scene.\nThe primary view (question-referenced) uses the effective pose (after any rotation/translation actions).\nSecondary views use their original poses from the cognitive map.\nEach ego map computes ego_direction for all objects using \u0394=[obj_x\u2212cam_x, obj_y\u2212cam_y] projected onto forward/right axes.\n\n[View Arrangement Reference]\nThe question preamble describes the camera arrangement:\n- 4-view scenes: \"front, left, back, and right\" \u2192 Image 1=front, Image 2=left, Image 3=back, Image 4=right\n- 3-view scenes: view labels are stated in the question (e.g., \"back, left, and right\")\n- Rotation scenes: all images are from the same position with different facing directions.\n After rotation, the effective facing may or may not match an existing image.\n Use the position and effective facing for ego-centric computation \u2014 do NOT assume it matches any image.\n\n[Rotation Axis Rules]\nWhen the camera facing changes (via turn/rotation), the coordinate axes rotate:\n- facing=up: forward=[-y] (north), right=[+x] (east)\n- facing=right: forward=[+x] (east), right=[+y] (south)\n- facing=down: forward=[+y] (south), right=[-x] (west)\n- facing=left: forward=[-x] (west), right=[-y] (north)\nAlways state \"Axes: forward=..., right=...\" after rotation to make the mapping explicit.\n\n[Rules]\n1. Grid: [0,0]=top-left, [9,9]=bottom-right.\n2. Parse the viewpoint from the question; apply actions to get the effective camera pose.\n3. Compute \"ego_direction\" using \u0394=[obj_x\u2212cam_x, obj_y\u2212cam_y], project onto forward/right axes.\n4. Build ego maps for ALL views, not just the primary view.\n5. Use the PRIMARY ego map for final answer elimination.\n6. The final answer must match one of the provided multiple choice options.\n\n[JSON Schema \u2014 Multi-View EgoMap Array]\n[\n {\n \"view\": \"Image k\",\n \"position\": [x, y],\n \"facing\": \"up|right|down|left\",\n \"is_primary\": true,\n \"actions\": [\"turn right 90\"],\n \"answer_object\": \"ObjectName\",\n \"answer_visible_in_view\": \"Image k\",\n \"objects\": [\n {\"name\": \"...\", \"position\": [x, y], \"ego_direction\": \"forward|back|left|right|forward-left|...\"},\n ...\n ]\n },\n {\n \"view\": \"Image m\",\n \"position\": [x, y],\n \"facing\": \"up|right|down|left\",\n \"is_primary\": false,\n \"objects\": [...]\n },\n ...\n]\n\n[Question]\nBased on these four images (image 1, 2, 3, and 4) showing the cup from different viewpoints (front, left, back, and right), with each camera aligned with room walls and partially capturing the surroundings: From the viewpoint presented in image 1, what is to the left of the cup? A. White board B. TV C. Metal lockers D. Wooden podium\n\n[Answer]" | |
| }, | |
| "around": { | |
| "id": "around_fac184210decc5fb0df8afd4962543bbce7f6e461ee413c3c949d2e783b4fa21_q2_5", | |
| "question": "Based on these three images (image 1, 2, and 3) showing the same scene from different viewpoints (front, left, and right): From the viewpoint presented in image 2, is there a lamp behind the books? A. Yes B. No.", | |
| "gt_answer": "A", | |
| "cogmap_instruction": "[Task]\nYour task is to analyze the spatial arrangement of objects in the scene by examining the provided images, which show the scene from different viewpoints. You will then create a detailed cognitive map representing the scene using a **10x10 grid coordinate system**. \n[Rules]\n1. Focus ONLY on these categories of objects in the scene: {sculpture of horse's head, lamp, books}\n2. Create a cognitive map with the following structure in the bird's view:\n - A 10x10 grid where [0, 0] is at the top-left corner and [9, 9] is at the bottom-right corner\n - up = towards the top of the grid (decreasing y)\n - right = towards the right of the grid (increasing x)\n - down = towards the bottom of the grid (increasing y)\n - left = towards the left of the grid (decreasing x)\n - Include positions of all objects from the specified categories\n - Estimate the center location (coordinates [x, y]) of each instance within provided categories\n - If a category contains multiple instances, include all of them\n - Object positions must maintain accurate relative spatial relationships\n - Combine and merge information from the images since they are pointing to the same scene, calibrating the object locations with grid coordinates accordingly\n3. Carefully integrate information from all views to create a single coherent spatial representation.\n4. For objects [sculpture of horse's head], determine their facing direction as up, right, down, or left. For other objects, omit the facing direction.", | |
| "images": [ | |
| "examples/around/view1.png", | |
| "examples/around/view2.png", | |
| "examples/around/view3.png" | |
| ], | |
| "prompt": "[Task]\nGiven multiple images showing a 3D scene from different viewpoints, reason carefully about the spatial layout and answer the question.\n\n[Answer Instruction]\nOutput 4 blocks in order:\n 1. <cogmap>: Bird's-eye map with all object positions and view positions/facings.\n 2. <egomap>: Multi-view ego-centric maps \u2014 a JSON array with one ego map per view.\n Each ego map includes \"view\", \"position\", \"facing\", \"is_primary\", per-object \"ego_direction\".\n The primary view (question-referenced, after any actions) also includes \"answer_object\" and \"answer_visible_in_view\".\n 3. <think>: 4-part reasoning with multi-view ego maps:\n Part 1 \u2014 Scene layout: list all objects with allocentric positions, then all views.\n Scene: obj1 [x1,y1], obj2 [x2,y2], ...\n Views: Image 1 [x,y] facing=dir, Image 2 [x,y] facing=dir, ...\n Part 2 \u2014 Primary view selection: identify the question-relevant viewpoint.\n If actions exist: Selected: Image X [pos] facing=dir \u2192 action \u2192 [eff_pos] facing=eff_dir\n Then axis mapping: Axes: forward=..., right=...\n If no actions: Selected: Image X [pos] facing=dir\n Part 3 \u2014 Multi-view ego maps: compute ego-centric layout from EACH view.\n For the primary view (after any actions):\n Ego from [eff_pos] facing=eff_dir (primary, after action):\n obj \u0394[dx,dy]=ego_dir [Image X]\n For each secondary view:\n Ego from Image Y [pos] facing=dir:\n obj \u0394[dx,dy]=ego_dir [Image X]\n This provides complete spatial context from all perspectives.\n Part 4 \u2014 Elimination: using the PRIMARY ego map directions, check each option A/B/C/D.\n target=dir valid={...}\n A. name=ego_dir [Image X] \u2713/\u2717\n 4. <answer>: Final answer as \"X. option_text\".\nFormat: '<cogmap>...</cogmap><egomap>...</egomap><think>...</think><answer>X. option_text</answer>'\nMake sure `X. option_text` matches exactly one of the options in the question.\n\n[Task]\nYour task is to analyze the spatial arrangement of objects in the scene by examining the provided images, which show the scene from different viewpoints. You will then create a detailed cognitive map representing the scene using a **10x10 grid coordinate system**. \n[Rules]\n1. Focus ONLY on these categories of objects in the scene: {sculpture of horse's head, lamp, books}\n2. Create a cognitive map with the following structure in the bird's view:\n - A 10x10 grid where [0, 0] is at the top-left corner and [9, 9] is at the bottom-right corner\n - up = towards the top of the grid (decreasing y)\n - right = towards the right of the grid (increasing x)\n - down = towards the bottom of the grid (increasing y)\n - left = towards the left of the grid (decreasing x)\n - Include positions of all objects from the specified categories\n - Estimate the center location (coordinates [x, y]) of each instance within provided categories\n - If a category contains multiple instances, include all of them\n - Object positions must maintain accurate relative spatial relationships\n - Combine and merge information from the images since they are pointing to the same scene, calibrating the object locations with grid coordinates accordingly\n3. Carefully integrate information from all views to create a single coherent spatial representation.\n4. For objects [sculpture of horse's head], determine their facing direction as up, right, down, or left. For other objects, omit the facing direction.\n\n[Multi-View Ego Map Instruction]\nGenerate ego-centric maps from the cognitive map for ALL views in the scene.\nThe primary view (question-referenced) uses the effective pose (after any rotation/translation actions).\nSecondary views use their original poses from the cognitive map.\nEach ego map computes ego_direction for all objects using \u0394=[obj_x\u2212cam_x, obj_y\u2212cam_y] projected onto forward/right axes.\n\n[View Arrangement Reference]\nThe question preamble describes the camera arrangement:\n- 4-view scenes: \"front, left, back, and right\" \u2192 Image 1=front, Image 2=left, Image 3=back, Image 4=right\n- 3-view scenes: view labels are stated in the question (e.g., \"back, left, and right\")\n- Rotation scenes: all images are from the same position with different facing directions.\n After rotation, the effective facing may or may not match an existing image.\n Use the position and effective facing for ego-centric computation \u2014 do NOT assume it matches any image.\n\n[Rotation Axis Rules]\nWhen the camera facing changes (via turn/rotation), the coordinate axes rotate:\n- facing=up: forward=[-y] (north), right=[+x] (east)\n- facing=right: forward=[+x] (east), right=[+y] (south)\n- facing=down: forward=[+y] (south), right=[-x] (west)\n- facing=left: forward=[-x] (west), right=[-y] (north)\nAlways state \"Axes: forward=..., right=...\" after rotation to make the mapping explicit.\n\n[Rules]\n1. Grid: [0,0]=top-left, [9,9]=bottom-right.\n2. Parse the viewpoint from the question; apply actions to get the effective camera pose.\n3. Compute \"ego_direction\" using \u0394=[obj_x\u2212cam_x, obj_y\u2212cam_y], project onto forward/right axes.\n4. Build ego maps for ALL views, not just the primary view.\n5. Use the PRIMARY ego map for final answer elimination.\n6. The final answer must match one of the provided multiple choice options.\n\n[JSON Schema \u2014 Multi-View EgoMap Array]\n[\n {\n \"view\": \"Image k\",\n \"position\": [x, y],\n \"facing\": \"up|right|down|left\",\n \"is_primary\": true,\n \"actions\": [\"turn right 90\"],\n \"answer_object\": \"ObjectName\",\n \"answer_visible_in_view\": \"Image k\",\n \"objects\": [\n {\"name\": \"...\", \"position\": [x, y], \"ego_direction\": \"forward|back|left|right|forward-left|...\"},\n ...\n ]\n },\n {\n \"view\": \"Image m\",\n \"position\": [x, y],\n \"facing\": \"up|right|down|left\",\n \"is_primary\": false,\n \"objects\": [...]\n },\n ...\n]\n\n[Question]\nBased on these three images (image 1, 2, and 3) showing the same scene from different viewpoints (front, left, and right): From the viewpoint presented in image 2, is there a lamp behind the books? A. Yes B. No.\n\n[Answer]" | |
| }, | |
| "rotation": { | |
| "id": "rotation_group013_q3_1", | |
| "question": "These three images (image 1, 2, and 3) show the same scene from three different viewpoints. The image 2 was taken after turning the camera 90 degrees to the right (clockwise) from the position of image 1. For image 3, the camera was turned another 90 degrees right, so it's basically facing the opposite direction of image 1. Based on these three images: If I am standing at the same spot and facing the same direction as shown in image 3, what is to my left? A. Wardrobes B. Door C. Window", | |
| "gt_answer": "A", | |
| "cogmap_instruction": "[Task]\nYour task is to analyze the spatial arrangement of objects in the scene by examining the provided images, which show the scene from different viewpoints. You will then create a detailed cognitive map representing the scene using a **10x10 grid coordinate system**. \n[Rules]\n1. Focus ONLY on these categories of objects in the scene: {wardrobes, door, window}\n2. Create a cognitive map with the following structure in the bird's view:\n - A 10x10 grid where [0, 0] is at the top-left corner and [9, 9] is at the bottom-right corner\n - up = towards the top of the grid (decreasing y)\n - right = towards the right of the grid (increasing x)\n - down = towards the bottom of the grid (increasing y)\n - left = towards the left of the grid (decreasing x)\n - Include positions of all objects from the specified categories\n - Estimate the center location (coordinates [x, y]) of each instance within provided categories\n - If a category contains multiple instances, include all of them\n - Object positions must maintain accurate relative spatial relationships\n - Combine and merge information from the images since they are pointing to the same scene, calibrating the object locations with grid coordinates accordingly\n3. Carefully integrate information from all views to create a single coherent spatial representation.\n", | |
| "images": [ | |
| "examples/rotation/view1.jpg", | |
| "examples/rotation/view2.jpg", | |
| "examples/rotation/view3.jpg" | |
| ], | |
| "prompt": "[Task]\nGiven multiple images showing a 3D scene from different viewpoints, reason carefully about the spatial layout and answer the question.\n\n[Answer Instruction]\nOutput 4 blocks in order:\n 1. <cogmap>: Bird's-eye map with all object positions and view positions/facings.\n 2. <egomap>: Multi-view ego-centric maps \u2014 a JSON array with one ego map per view.\n Each ego map includes \"view\", \"position\", \"facing\", \"is_primary\", per-object \"ego_direction\".\n The primary view (question-referenced, after any actions) also includes \"answer_object\" and \"answer_visible_in_view\".\n 3. <think>: 4-part reasoning with multi-view ego maps:\n Part 1 \u2014 Scene layout: list all objects with allocentric positions, then all views.\n Scene: obj1 [x1,y1], obj2 [x2,y2], ...\n Views: Image 1 [x,y] facing=dir, Image 2 [x,y] facing=dir, ...\n Part 2 \u2014 Primary view selection: identify the question-relevant viewpoint.\n If actions exist: Selected: Image X [pos] facing=dir \u2192 action \u2192 [eff_pos] facing=eff_dir\n Then axis mapping: Axes: forward=..., right=...\n If no actions: Selected: Image X [pos] facing=dir\n Part 3 \u2014 Multi-view ego maps: compute ego-centric layout from EACH view.\n For the primary view (after any actions):\n Ego from [eff_pos] facing=eff_dir (primary, after action):\n obj \u0394[dx,dy]=ego_dir [Image X]\n For each secondary view:\n Ego from Image Y [pos] facing=dir:\n obj \u0394[dx,dy]=ego_dir [Image X]\n This provides complete spatial context from all perspectives.\n Part 4 \u2014 Elimination: using the PRIMARY ego map directions, check each option A/B/C/D.\n target=dir valid={...}\n A. name=ego_dir [Image X] \u2713/\u2717\n 4. <answer>: Final answer as \"X. option_text\".\nFormat: '<cogmap>...</cogmap><egomap>...</egomap><think>...</think><answer>X. option_text</answer>'\nMake sure `X. option_text` matches exactly one of the options in the question.\n\n[Task]\nYour task is to analyze the spatial arrangement of objects in the scene by examining the provided images, which show the scene from different viewpoints. You will then create a detailed cognitive map representing the scene using a **10x10 grid coordinate system**. \n[Rules]\n1. Focus ONLY on these categories of objects in the scene: {wardrobes, door, window}\n2. Create a cognitive map with the following structure in the bird's view:\n - A 10x10 grid where [0, 0] is at the top-left corner and [9, 9] is at the bottom-right corner\n - up = towards the top of the grid (decreasing y)\n - right = towards the right of the grid (increasing x)\n - down = towards the bottom of the grid (increasing y)\n - left = towards the left of the grid (decreasing x)\n - Include positions of all objects from the specified categories\n - Estimate the center location (coordinates [x, y]) of each instance within provided categories\n - If a category contains multiple instances, include all of them\n - Object positions must maintain accurate relative spatial relationships\n - Combine and merge information from the images since they are pointing to the same scene, calibrating the object locations with grid coordinates accordingly\n3. Carefully integrate information from all views to create a single coherent spatial representation.\n\n\n[Multi-View Ego Map Instruction]\nGenerate ego-centric maps from the cognitive map for ALL views in the scene.\nThe primary view (question-referenced) uses the effective pose (after any rotation/translation actions).\nSecondary views use their original poses from the cognitive map.\nEach ego map computes ego_direction for all objects using \u0394=[obj_x\u2212cam_x, obj_y\u2212cam_y] projected onto forward/right axes.\n\n[View Arrangement Reference]\nThe question preamble describes the camera arrangement:\n- 4-view scenes: \"front, left, back, and right\" \u2192 Image 1=front, Image 2=left, Image 3=back, Image 4=right\n- 3-view scenes: view labels are stated in the question (e.g., \"back, left, and right\")\n- Rotation scenes: all images are from the same position with different facing directions.\n After rotation, the effective facing may or may not match an existing image.\n Use the position and effective facing for ego-centric computation \u2014 do NOT assume it matches any image.\n\n[Rotation Axis Rules]\nWhen the camera facing changes (via turn/rotation), the coordinate axes rotate:\n- facing=up: forward=[-y] (north), right=[+x] (east)\n- facing=right: forward=[+x] (east), right=[+y] (south)\n- facing=down: forward=[+y] (south), right=[-x] (west)\n- facing=left: forward=[-x] (west), right=[-y] (north)\nAlways state \"Axes: forward=..., right=...\" after rotation to make the mapping explicit.\n\n[Rules]\n1. Grid: [0,0]=top-left, [9,9]=bottom-right.\n2. Parse the viewpoint from the question; apply actions to get the effective camera pose.\n3. Compute \"ego_direction\" using \u0394=[obj_x\u2212cam_x, obj_y\u2212cam_y], project onto forward/right axes.\n4. Build ego maps for ALL views, not just the primary view.\n5. Use the PRIMARY ego map for final answer elimination.\n6. The final answer must match one of the provided multiple choice options.\n\n[JSON Schema \u2014 Multi-View EgoMap Array]\n[\n {\n \"view\": \"Image k\",\n \"position\": [x, y],\n \"facing\": \"up|right|down|left\",\n \"is_primary\": true,\n \"actions\": [\"turn right 90\"],\n \"answer_object\": \"ObjectName\",\n \"answer_visible_in_view\": \"Image k\",\n \"objects\": [\n {\"name\": \"...\", \"position\": [x, y], \"ego_direction\": \"forward|back|left|right|forward-left|...\"},\n ...\n ]\n },\n {\n \"view\": \"Image m\",\n \"position\": [x, y],\n \"facing\": \"up|right|down|left\",\n \"is_primary\": false,\n \"objects\": [...]\n },\n ...\n]\n\n[Question]\nThese three images (image 1, 2, and 3) show the same scene from three different viewpoints. The image 2 was taken after turning the camera 90 degrees to the right (clockwise) from the position of image 1. For image 3, the camera was turned another 90 degrees right, so it's basically facing the opposite direction of image 1. Based on these three images: If I am standing at the same spot and facing the same direction as shown in image 3, what is to my left? A. Wardrobes B. Door C. Window\n\n[Answer]" | |
| } | |
| } |