zli12321 commited on
Commit
1ad6e66
Β·
verified Β·
1 Parent(s): 608468f

Add sample trajectories section and chart to model card

Browse files
Files changed (2) hide show
  1. README.md +162 -0
  2. assets/sample_trajectories.svg +57 -0
README.md CHANGED
@@ -12,3 +12,165 @@ tags:
12
  # RSR-27B: Recursive Self-Rewrite (RSR) Model
13
 
14
  RSR-27B is Qwen-3.8-27B finetuned with Recursive Self-Rewrite (RSR) trajectories.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
12
  # RSR-27B: Recursive Self-Rewrite (RSR) Model
13
 
14
  RSR-27B is Qwen-3.8-27B finetuned with Recursive Self-Rewrite (RSR) trajectories.
15
+
16
+ ## Sample trajectories
17
+
18
+ 3 SWR tasks that base Qwen3.8-27B passed under continue-until-timeout. For each: the expert
19
+ trajectory, the runbook the base model wrote from it, and a passing rewrite (base Qwen3.8-27B as plain
20
+ Terminus-2, runbook kept private). Browse them side by side in the **[interactive trajectory viewer](https://huggingface.co/spaces/IntelligenceLab/RSR-27B-trajectories)**.
21
+
22
+ ![Expert vs. passing rewrite per task: commands by kind, steps, and agent time](assets/sample_trajectories.svg)
23
+
24
+ | Task | Domain | Steps, expert β†’ rewrite | Agent time, expert β†’ rewrite | Runbook replays passed |
25
+ |---|---|---|---|---|
26
+ | [tree_sitter_markdown_inline__05](https://intelligencelab-rsr-27b-trajectories.static.hf.space/tree_sitter_markdown_inline__05.html) | Data, query & formal (tree-sitter) | 64 β†’ 34 | 2.0 h β†’ 22 min | 4 of 4 |
27
+ | [cmake_generated_header_config__06](https://intelligencelab-rsr-27b-trajectories.static.hf.space/cmake_generated_header_config__06.html) | Systems & infrastructure (CMake) | 59 β†’ 27 | 2.0 h β†’ 21 min | 4 of 4 |
28
+ | [openfoam_scalar_transport_boundary_probe__02](https://intelligencelab-rsr-27b-trajectories.static.hf.space/openfoam_scalar_transport_boundary_probe__02.html) | Physical science & simulation (OpenFOAM) | 56 β†’ 23 | 1.9 h β†’ 28 min | 4 of 4 |
29
+
30
+ <details>
31
+ <summary><b>tree_sitter_markdown_inline__05</b>: runbook and step lists</summary>
32
+
33
+ [Open this task side by side](https://intelligencelab-rsr-27b-trajectories.static.hf.space/tree_sitter_markdown_inline__05.html)
34
+
35
+ Data, query & formal (tree-sitter). The expert never marked the task complete: it kept working until the 2-hour limit ended the run, and the work it left in place passed. Runbook: `runbook_01`, checked by the leak judge; 4 of 4 replays passed, in 13, 15, 15, 34 steps, and the one shown took 34.
36
+
37
+ **Runbook summary.** Create an editable project in /app/workspace and a /app/workspace/reconstruct.py that reads a scenario JSON from stdin and prints a JSON summary matching the oracle reference_summary for the tree_sitter_markdown_inline demo. The reference_summary is a fingerprint of three output files (generate.log, parse.txt, parse_metrics.json) produced by a tree-sitter markdown parse of a fixed input.
38
+
39
+ **Runbook milestones**
40
+
41
+ 1. Understand the reference output structure and signature computation
42
+ 2. Understand the parameter-to-input mapping and the reference_summary computation rules
43
+ 3. Write reconstruct.py that produces the correct output for public scenarios
44
+ 4. Create project.json and ensure the workspace is a complete editable project
45
+ 5. Ensure robustness for hidden scenarios centered on 'table'
46
+
47
+ **Expert** β€” Base Qwen3.8-27B Β· continue-until-timeout: 64 steps, 153 commands
48
+
49
+ - Step 1: First, let me explore the directory structure to understand what's available - observations, tools, and any existing files. β€” `ls -la /app/`
50
+ - Step 2: Read the manifest.json, extract the reference outputs, read the feedback tool, and the README to understand what's expected. β€” `cat /app/observations/manifest.json`
51
+ - Step 3: Read all the extracted files to understand the expected output format, parameters, and behavior. β€” `cat /tmp/obs/public_manifest.json`
52
+ - Step 4: Read the native files (generate.log, parse.txt, parse_metrics.json) and the oracle pretty output for both public scenarios to understand the full pict β€” `cat /tmp/obs/public_1/native/generate.log`
53
+ - Step 5: Read public_2's native files to compare with public_1 and understand how parameters affect the output. β€” `cat /tmp/obs/public_2/native/generate.log`
54
+ - Step 6: Let me write a quick Python script to verify the signature computation, then create the initial reconstruct.py and test it with the feedback tool. β€” `python3 -c "`
55
+ - Step 7: Let me check if tree-sitter is available, look for any input files, and understand the mechanism better. Then I'll create the initial reconstruct.py. β€” `which tree-sitter 2>/dev/null; tree-sitter --version 2>/dev/null; echo "exit: $?"`
56
+ - Step 8: Let me explore the tree-sitter CLI capabilities, check for markdown grammar, and understand the parse tree structure better. Then I'll create the init β€” `tree-sitter --help 2>&1 | head -30`
57
+ - … 53 more steps …
58
+ - Step 62: Let me look at the tree-sitter-markdown-inline grammar to understand how it works, and try to find the input bundle or any other clues about the param β€” `cat /root/github/ts-md3/package/tree-sitter-markdown-inline/grammar.js | head -100`
59
+ - Step 63: Let me create a comprehensive test to understand the parameter-to-input mapping, and then build a robust reconstruct.py that can handle hidden scenari β€” `python3 << 'PYEOF'`
60
+ - Step 64: Let me create a comprehensive test to understand the parameter-to-input mapping, and then build a robust reconstruct.py that can handle hidden scenari β€” `python3 << 'PYEOF'`
61
+
62
+ **Rewrite** β€” Base Qwen3.8-27B Β· plain Terminus-2, runbook kept private: 34 steps, 78 commands
63
+
64
+ - Step 1: Step 1: Explore the directory structure to understand what's available - observations, tools, and any existing files. Step 2: Extract and examine the β€” `ls -la /app/`
65
+ - Step 2: 1. Read the manifest.json and README.md 2. Extract the reference_outputs.tar.gz to examine oracle outputs 3. Read the feedback tool to understand how β€” `cat /app/observations/manifest.json`
66
+ - Step 3: Read the oracle outputs, native files, and manifest to understand the exact structure and expected behavior. β€” `cat /tmp/obs/public_1/oracle_output.json`
67
+ - Step 4: 1. Read the native output files (generate.log, parse.txt, parse_metrics.json) 2. Read the feedback tool 3. Read the scenario.json and signature.json f β€” `cat /tmp/obs/public_1/native/generate.log`
68
+ - Step 5: 1. Read the feedback tool to understand signature computation 2. Verify the signature computation by computing sha256 of the summary object 3. Check i β€” `cat /app/tools/feedback`
69
+ - Step 6: 1. Analyze the native files to understand how numbers and keywords are extracted 2. Check the artifact_manifest.json for additional context 3. Check t β€” `python3 -c "`
70
+ - Step 7: 1. Analyze keyword extraction from native files 2. Check if tree-sitter markdown grammar is available 3. Check what markdown input produces the observ β€” `python3 -c "`
71
+ - Step 8: 1. Check for tree-sitter markdown grammar availability 2. Test tree-sitter parse with a sample markdown 3. Analyze the keyword extraction rules more c β€” `find / -name '*.node' 2>/dev/null | head -20`
72
+ - … 23 more steps …
73
+ - Step 32: Run final checks: verify files exist, run feedback one more time, and test the reconstruct.py with both public scenarios. β€” `ls -la /app/workspace/`
74
+ - Step 33: Task is complete. All deliverables are in place and validated. **(marks done)**
75
+ - Step 34: Confirm task completion. **(marks done)**
76
+
77
+ </details>
78
+
79
+ <details>
80
+ <summary><b>cmake_generated_header_config__06</b>: runbook and step lists</summary>
81
+
82
+ [Open this task side by side](https://intelligencelab-rsr-27b-trajectories.static.hf.space/cmake_generated_header_config__06.html)
83
+
84
+ Systems & infrastructure (CMake). The expert never marked the task complete: it kept working until the 2-hour limit ended the run, and the work it left in place passed. Runbook: `runbook_02`, checked by the leak judge; 4 of 4 replays passed, in 24, 27, 27, 29 steps, and the one shown took 27.
85
+
86
+ **Runbook summary.** Create /app/workspace/reconstruct.py that reads a scenario JSON from stdin (with parameters build_type, configured_version, consumer_project) and prints JSON to stdout matching the oracle output shape. The output is a summary of a CMake build with parameter-dependent values embedded in CMakeCache.txt (SILR_PARAM_* entries, CMAKE_BUILD_TYPE) and CMakeConfigureLog.yaml (random build IDs, number counts). The solution must work for hidden scenarios, not just the two public ones.
87
+
88
+ **Runbook milestones**
89
+
90
+ 1. Understand the oracle output structure and how parameters affect it
91
+ 2. Understand the exact number extraction pattern for CMakeCache.txt
92
+ 3. Understand the CMakeConfigureLog.yaml differences (number_count and cmtc_XXXXX IDs)
93
+ 4. Build reconstruct.py that handles both public and hidden scenarios
94
+ 5. Write reconstruct.py with the correct mechanism
95
+ 6. Ensure hidden scenarios work correctly
96
+
97
+ **Expert** β€” Base Qwen3.8-27B Β· continue-until-timeout: 59 steps, 94 commands
98
+
99
+ - Step 1: First, explore the directory structure to understand what's available - observations, tools, and any existing files. β€” `ls -la /app/`
100
+ - Step 2: Read the manifest.json, README.md, and extract the reference outputs to understand the expected behavior. β€” `cat /app/observations/manifest.json`
101
+ - Step 3: Extract the reference_outputs.tar.gz to examine the public manifest and oracle outputs, understanding the expected output format. β€” `mkdir -p /tmp/ref && cd /tmp/ref && tar xzf /app/observations/reference_outputs.tar.gz && `
102
+ - Step 4: Examine the extracted public_1 and public_2 directories, and get the full JSON structure of the public manifest to understand all fields. β€” `find /tmp/ref/public_1 -type f | head -30`
103
+ - Step 5: Let me examine the oracle_output.json files directly and compare the two public scenarios to understand what changes with parameters. β€” `cat /tmp/ref/public_1/oracle_output.json | python3 -m json.tool | head -100`
104
+ - Step 6: Compare the two oracle outputs, look at the native CMakeCache.txt files to see how parameters are embedded, and understand the full output structure. β€” `python3 -c "`
105
+ - Step 7: Compare the two oracle outputs in detail to understand what changes with parameters, and examine the native files more closely. β€” `python3 << 'EOF'`
106
+ - Step 8: Examine the native CMakeCache.txt and CMakeConfigureLog.yaml files in detail to understand the exact format and how parameters affect them. β€” `cat /tmp/ref/public_1/native/build/CMakeCache.txt`
107
+ - … 48 more steps …
108
+ - Step 57: First, let me look at the current reconstruct.py, the workspace files, observations, and the feedback tool to understand the full picture. β€” `cat /app/workspace/reconstruct.py`
109
+ - Step 58: *(reply could not be parsed; the harness asked for valid JSON)*
110
+ - Step 59: First, let me look at the current reconstruct.py, the workspace files, observations, and the feedback tool to understand the full picture. β€” `cat /app/workspace/reconstruct.py`
111
+
112
+ **Rewrite** β€” Base Qwen3.8-27B Β· plain Terminus-2, runbook kept private: 27 steps, 58 commands
113
+
114
+ - Step 1: 1. List contents of /app/observations and /app/tools 2. Extract the reference outputs tarball 3. Examine the public manifest and oracle outputs to und β€” `ls -la /app/observations/`
115
+ - Step 2: 1. Look at the public manifest structure 2. Examine the oracle_output.json for both public scenarios 3. Compare the two to understand what changes bet β€” `cat /tmp/ref/public_manifest.json | python3 -m json.tool | head -80`
116
+ - Step 3: 1. Look at the scenario.json files for both public scenarios 2. Compare the oracle_output.json structures 3. Look at the native files that differ betw β€” `cat /tmp/ref/public_1/scenario.json`
117
+ - Step 4: 1. Compare the two oracle outputs to find which files differ 2. Examine the CMakeCache.txt differences (likely contains parameter values) 3. Look at t β€” `python3 -c "`
118
+ - Step 5: 1. Compare the CMakeCache.txt entries between the two oracle outputs 2. Compare the CMakeConfigureLog.yaml entries 3. Look at the native CMakeCache.tx β€” `python3 -c "`
119
+ - Step 6: 1. Look at the native CMakeCache.txt to see the parameter entries 2. Look at CMakeConfigureLog.yaml to understand the cmtc_ IDs 3. Understand the full β€” `grep -n 'SILR_PARAM\|CMAKE_BUILD_TYPE' /tmp/ref/public_1/native/build/CMakeCache.txt`
120
+ - Step 7: 1. Search for cmtc_ in the yaml file more broadly 2. Look at the full entry structure for CMakeCache.txt and CMakeConfigureLog.yaml 3. Understand how β€” `grep -i 'cmtc' /tmp/ref/public_1/native/build/CMakeFiles/CMakeConfigureLog.yaml | head -10`
121
+ - Step 8: 1. See the full CMakeCache.txt entry structure (all fields) 2. See the full CMakeConfigureLog.yaml entry structure 3. Understand the CMAKE_BUILD_TYPE β€” `python3 -c "`
122
+ - … 16 more steps …
123
+ - Step 25: 1. Run one more comprehensive test with a hidden scenario 2. Verify the full output structure matches expectations 3. Confirm the feedback tool passes β€” `echo '{"parameters":{"build_type":5.5,"configured_version":4.4,"consumer_project":3.3},"sc`
124
+ - Step 26: The task is complete. The solution correctly handles both public and hidden scenarios. **(marks done)**
125
+ - Step 27: Task is complete. Marking as done. **(marks done)**
126
+
127
+ </details>
128
+
129
+ <details>
130
+ <summary><b>openfoam_scalar_transport_boundary_probe__02</b>: runbook and step lists</summary>
131
+
132
+ [Open this task side by side](https://intelligencelab-rsr-27b-trajectories.static.hf.space/openfoam_scalar_transport_boundary_probe__02.html)
133
+
134
+ Physical science & simulation (OpenFOAM). The expert never marked the task complete: it kept working until the 2-hour limit ended the run, and the work it left in place passed. Runbook: `runbook_01`, checked by the leak judge; 4 of 4 replays passed, in 23, 23, 33, 51 steps, and the one shown took 23.
135
+
136
+ **Runbook summary.** Reconstruct a parametric OpenFOAM scalar-transport boundary probe. Build /app/workspace/reconstruct.py that reads a scenario JSON from stdin and prints a JSON object matching the oracle output structure (containing 'summary' and 'parameter_effect'). The output must generalize to hidden parameter combinations (inlet_scalar, mesh, velocity, steps, probe_location), not just the two public cases.
137
+
138
+ **Runbook milestones**
139
+
140
+ 1. Understand the reference output structure and identify parameter-dependent fields
141
+ 2. Determine the parameter_effect digest formula
142
+ 3. Determine the formulas for parameter-dependent values in blockMesh.log
143
+ 4. Build reconstruct.py that generalizes to any parameter combination
144
+ 5. Validate against public feedback
145
+
146
+ **Expert** β€” Base Qwen3.8-27B Β· continue-until-timeout: 56 steps, 72 commands
147
+
148
+ - Step 1: First, explore the directory structure to understand what's available - observations, tools, and any existing files. β€” `ls -la /app/`
149
+ - Step 2: Read the manifest.json, README.md, feedback tool, and extract the reference outputs to understand the expected behavior. β€” `cat /app/observations/manifest.json`
150
+ - Step 3: Extract the reference_outputs.tar.gz to examine the expected output format and understand the oracle behavior. β€” `mkdir -p /tmp/ref && cd /tmp/ref && tar xzf /app/observations/reference_outputs.tar.gz && `
151
+ - Step 4: View the full public_manifest.json content, and also check what other files are in the extracted tar. β€” `ls -la /tmp/ref/`
152
+ - Step 5: Save the full JSON to a file and examine its structure programmatically. β€” `python3 -c "`
153
+ - Step 6: Examine the structure of parameter_effect and summary for both scenarios. β€” `python3 -c "`
154
+ - Step 7: Examine the summary structure for both scenarios in detail. β€” `python3 -c "`
155
+ - Step 8: Save the full oracle output to a file and examine the top-level structure of the summary. β€” `python3 -c "`
156
+ - … 45 more steps …
157
+ - Step 54: Create a basic reconstruct.py that uses the oracle data directly to see if the feedback tool accepts it. Then I'll work on understanding the formulas. β€” `python3 << 'PYEOF'`
158
+ - Step 55: Create a basic reconstruct.py that uses the oracle data directly to see if the feedback tool accepts it. Then I'll work on understanding the formulas. β€” `python3 << 'PYEOF'`
159
+ - Step 56: Create a basic reconstruct.py that uses the oracle data directly to see if the feedback tool accepts it. Then I'll work on understanding the formulas. β€” `python3 << 'PYEOF'`
160
+
161
+ **Rewrite** β€” Base Qwen3.8-27B Β· plain Terminus-2, runbook kept private: 23 steps, 51 commands
162
+
163
+ - Step 1: 1. List files in /app/observations and /app/tools 2. Extract the reference_outputs.tar.gz to understand the output structure 3. Examine the public man β€” `ls -la /app/observations/`
164
+ - Step 2: 1. Read the manifest.json to understand the scenario structure 2. Extract reference_outputs.tar.gz to see the oracle outputs 3. Read the README.md in β€” `cat /app/observations/manifest.json`
165
+ - Step 3: 1. Look at the public_manifest.json 2. Examine the oracle_output.json structure for public_1 3. Compare the two scenarios to understand what changes β€” `python3 -c "import json; d=json.load(open('/tmp/ref/public_manifest.json')); print(json.du`
166
+ - Step 4: 1. Get the full structure of oracle_output.json including all keys 2. Compare the two public scenarios to identify what changes 3. Look at the signatu β€” `python3 -c "`
167
+ - Step 5: 1. Read the feedback tool to understand how matching works 2. Compare all 24 output files between the two scenarios to find what changes 3. Look at sc β€” `cat /app/tools/feedback`
168
+ - Step 6: 1. Compare all 24 output files between the two scenarios to identify which ones change 2. For files that change, identify the pattern of change 3. Bui β€” `python3 -c "`
169
+ - Step 7: 1. Compare the blockMesh.log entries between the two scenarios in detail 2. Look at the actual native blockMesh.log files to understand the content 3. β€” `python3 -c "`
170
+ - Step 8: 1. Look at the blockMeshDict to understand the mesh parameters 2. Compare the two blockMesh.log native files 3. Analyze the formula for the changing v β€” `cat /tmp/ref/public_1/native/case/system/blockMeshDict`
171
+ - … 12 more steps …
172
+ - Step 21: 1. Re-examine the blockMesh.log more carefully for any other parameter-dependent values 2. Check if the mesh statistics in the log should change with β€” `python3 -c "`
173
+ - Step 22: The solution is complete. Let me do one final verification and mark the task as complete. β€” `python3 /app/tools/feedback` **(marks done)**
174
+ - Step 23: Task is complete. Both public scenarios pass and the solution generalizes to hidden scenarios. **(marks done)**
175
+
176
+ </details>
assets/sample_trajectories.svg ADDED