Instructions to use keepsloading/icml_repro_scratch with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use keepsloading/icml_repro_scratch with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="keepsloading/icml_repro_scratch")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("keepsloading/icml_repro_scratch")
model = AutoModelForCausalLM.from_pretrained("keepsloading/icml_repro_scratch", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use keepsloading/icml_repro_scratch with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "keepsloading/icml_repro_scratch"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "keepsloading/icml_repro_scratch",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/keepsloading/icml_repro_scratch

SGLang

How to use keepsloading/icml_repro_scratch with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "keepsloading/icml_repro_scratch" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "keepsloading/icml_repro_scratch",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "keepsloading/icml_repro_scratch" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "keepsloading/icml_repro_scratch",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use keepsloading/icml_repro_scratch with Docker Model Runner:
```
docker model run hf.co/keepsloading/icml_repro_scratch
```

keepsloading commited on Jul 18

Commit

bc3f47c

verified ·

1 Parent(s): 3b031b5

Upload folder using huggingface_hub

Browse files

Files changed (2) hide show

.gitattributes +3 -0
outputs/eval/logs/vanilla/ruler_a.log +211 -47

.gitattributes CHANGED Viewed

@@ -2,3 +2,6 @@
 *.arrow filter=lfs diff=lfs merge=lfs -text
 *.jsonl filter=lfs diff=lfs merge=lfs -text
 tokenizer.json filter=lfs diff=lfs merge=lfs -text

 *.arrow filter=lfs diff=lfs merge=lfs -text
 *.jsonl filter=lfs diff=lfs merge=lfs -text
 tokenizer.json filter=lfs diff=lfs merge=lfs -text
+outputs/l2a_style/stage2/checkpoint-25/optimizer.pt filter=lfs diff=lfs merge=lfs -text
+outputs/l2a_style/stage2/checkpoint-50/optimizer.pt filter=lfs diff=lfs merge=lfs -text
+outputs/l2a_style/stage2/checkpoint-75/optimizer.pt filter=lfs diff=lfs merge=lfs -text

outputs/eval/logs/vanilla/ruler_a.log CHANGED Viewed

@@ -1,13 +1,177 @@
-2026-07-18:10:07:41 WARNING  [config.evaluate_config:281] --limit SHOULD ONLY BE USED FOR TESTING. REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT.
-2026-07-18:10:07:45 INFO     [_cli.run:376] Selected Tasks: ['niah_single_1', 'niah_single_3', 'niah_multikey_2', 'niah_multiquery', 'ruler_vt', 'ruler_fwe', 'ruler_qa_hotpot']
-2026-07-18:10:07:46 INFO     [evaluator:211] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
-2026-07-18:10:07:46 INFO     [evaluator:236] Initializing hf model, with arguments: {'pretrained': '/workspace', 'trust_remote_code': True, 'dtype': 'bfloat16', 'max_length': 16384, 'attn_implementation': 'sdpa'}
-2026-07-18:10:07:49 INFO     [models.huggingface:161] Using device 'cuda:0'
-2026-07-18:10:07:49 INFO     [models.huggingface:423] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
  0%|          | 0/1 [00:00<?, ?it/s][A
  0%|          | 0/500 [00:00<?, ?it/s][A
 12%|█▏        | 60/500 [00:01<00:07, 59.90it/s][A
 24%|██▍       | 120/500 [00:02<00:06, 59.31it/s][A
 36%|███▌      | 181/500 [00:03<00:05, 59.99it/s][A
 49%|████▊     | 243/500 [00:04<00:04, 60.56it/s][A
 61%|██████    | 304/500 [00:05<00:03, 59.89it/s][A
 73%|███████▎  | 364/500 [00:06<00:02, 59.63it/s][A
 85%|████████▍ | 424/500 [00:07<00:01, 59.16it/s][A
 97%|█████████▋| 485/500 [00:08<00:00, 59.51it/s][A
 Traceback (most recent call last):
-  File "/opt/conda/lib/python3.11/site-packages/lm_eval/tasks/ruler/prepare_niah.py", line 31, in <module>
-    import wonderwords
-ModuleNotFoundError: No module named 'wonderwords'
 During handling of the above exception, another exception occurred:
@@ -37,41 +201,41 @@ Traceback (most recent call last):
   File "/opt/conda/lib/python3.11/site-packages/lm_eval/tasks/__init__.py", line 430, in <lambda>
     lambda task: self._load_individual_task_or_group(task),
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-  File "/opt/conda/lib/python3.11/site-packages/lm_eval/tasks/__init__.py", line 327, in _load_individual_task_or_group
-    task_config = self._get_config(name_or_config)
-                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-  File "/opt/conda/lib/python3.11/site-packages/lm_eval/tasks/__init__.py", line 235, in _get_config
-    return utils.load_yaml_config(yaml_path, mode="full")
-           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-  File "/opt/conda/lib/python3.11/site-packages/lm_eval/utils.py", line 543, in load_yaml_config
-    yaml_config = yaml.load(file, Loader=loader)
-                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-  File "/opt/conda/lib/python3.11/site-packages/yaml/__init__.py", line 81, in load
-    return loader.get_single_data()
-           ^^^^^^^^^^^^^^^^^^^^^^^^
-  File "/opt/conda/lib/python3.11/site-packages/yaml/constructor.py", line 51, in get_single_data
-    return self.construct_document(node)
-           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-  File "/opt/conda/lib/python3.11/site-packages/yaml/constructor.py", line 60, in construct_document
-    for dummy in generator:
-  File "/opt/conda/lib/python3.11/site-packages/yaml/constructor.py", line 413, in construct_yaml_map
-    value = self.construct_mapping(node)
-            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-  File "/opt/conda/lib/python3.11/site-packages/yaml/constructor.py", line 218, in construct_mapping
-    return super().construct_mapping(node, deep=deep)
-           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-  File "/opt/conda/lib/python3.11/site-packages/yaml/constructor.py", line 143, in construct_mapping
-    value = self.construct_object(value_node, deep=deep)
-            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-  File "/opt/conda/lib/python3.11/site-packages/yaml/constructor.py", line 100, in construct_object
-    data = constructor(self, node)
-           ^^^^^^^^^^^^^^^^^^^^^^^
-  File "/opt/conda/lib/python3.11/site-packages/lm_eval/utils.py", line 523, in import_function
-    spec.loader.exec_module(module)
-  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
-  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
-  File "/opt/conda/lib/python3.11/site-packages/lm_eval/tasks/ruler/niah_utils.py", line 8, in <module>
-    from lm_eval.tasks.ruler.prepare_niah import generate_samples, get_haystack
-  File "/opt/conda/lib/python3.11/site-packages/lm_eval/tasks/ruler/prepare_niah.py", line 35, in <module>
-    raise ImportError(
-ImportError: Please install the `wonderwords` and `nltk` packages to run this script. You can install them with `pip install lm_eval["ruler"]` or`pip install wonderwords nltk`.

+2026-07-18:10:14:08 WARNING  [config.evaluate_config:281] --limit SHOULD ONLY BE USED FOR TESTING. REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT.
+2026-07-18:10:14:12 INFO     [_cli.run:376] Selected Tasks: ['niah_single_1', 'niah_single_3', 'niah_multikey_2', 'niah_multiquery', 'ruler_vt', 'ruler_fwe', 'ruler_qa_hotpot']
+2026-07-18:10:14:13 INFO     [evaluator:211] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
+2026-07-18:10:14:13 INFO     [evaluator:236] Initializing hf model, with arguments: {'pretrained': '/workspace', 'trust_remote_code': True, 'dtype': 'bfloat16', 'max_length': 16384, 'attn_implementation': 'sdpa'}
+2026-07-18:10:14:16 INFO     [models.huggingface:161] Using device 'cuda:0'
+2026-07-18:10:14:16 INFO     [models.huggingface:423] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
+2026-07-18:10:14:26 WARNING  [api.task:856] niah_single_1: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:10:14:26 INFO     [tasks.ruler.common_utils:26] Using tokenizer /workspace for synthetic tasks.
+2026-07-18:10:14:35 WARNING  [api.task:856] niah_single_3: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:10:14:46 WARNING  [api.task:856] niah_multikey_2: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:10:14:53 WARNING  [api.task:856] niah_multiquery: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:10:15:02 WARNING  [api.task:856] ruler_vt: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:10:15:02 INFO     [tasks.ruler.common_utils:26] Using tokenizer /workspace for synthetic tasks.
+Max length 500 | Current length 307 | Noises: 5
+Max length 500 | Current length 410 | Noises: 10
+Max length 500 | Current length 531 | Noises: 15
+Num noises: 10
  0%|          | 0/1 [00:00<?, ?it/s][A
+Max length 8192 | Current length 791 | Noises: 10
+Max length 8192 | Current length 1025 | Noises: 20
+Max length 8192 | Current length 1273 | Noises: 30
+Max length 8192 | Current length 1510 | Noises: 40
+Max length 8192 | Current length 1749 | Noises: 50
+Max length 8192 | Current length 1996 | Noises: 60
+Max length 8192 | Current length 2231 | Noises: 70
+Max length 8192 | Current length 2473 | Noises: 80
+Max length 8192 | Current length 2705 | Noises: 90
+Max length 8192 | Current length 2953 | Noises: 100
+Max length 8192 | Current length 3186 | Noises: 110
+Max length 8192 | Current length 3431 | Noises: 120
+Max length 8192 | Current length 3671 | Noises: 130
+Max length 8192 | Current length 3916 | Noises: 140
+Max length 8192 | Current length 4149 | Noises: 150
+Max length 8192 | Current length 4386 | Noises: 160
+Max length 8192 | Current length 4630 | Noises: 170
+Max length 8192 | Current length 4871 | Noises: 180
+Max length 8192 | Current length 5110 | Noises: 190
+Max length 8192 | Current length 5349 | Noises: 200
+Max length 8192 | Current length 5590 | Noises: 210
+Max length 8192 | Current length 5832 | Noises: 220
+Max length 8192 | Current length 6070 | Noises: 230
+Max length 8192 | Current length 6311 | Noises: 240
+Max length 8192 | Current length 6544 | Noises: 250
+Max length 8192 | Current length 6791 | Noises: 260
+Max length 8192 | Current length 7031 | Noises: 270
+Max length 8192 | Current length 7271 | Noises: 280
+Max length 8192 | Current length 7513 | Noises: 290
+Max length 8192 | Current length 7751 | Noises: 300
+Max length 8192 | Current length 7986 | Noises: 310
+Max length 8192 | Current length 8230 | Noises: 320
+Num noises: 310
  0%|          | 0/500 [00:00<?, ?it/s][A
 12%|█▏        | 60/500 [00:01<00:07, 59.90it/s][A
 24%|██▍       | 120/500 [00:02<00:06, 59.31it/s][A
 36%|███▌      | 181/500 [00:03<00:05, 59.99it/s][A
 49%|████▊     | 243/500 [00:04<00:04, 60.56it/s][A
 61%|██████    | 304/500 [00:05<00:03, 59.89it/s][A
 73%|███████▎  | 364/500 [00:06<00:02, 59.63it/s][A
 85%|████████▍ | 424/500 [00:07<00:01, 59.16it/s][A
 97%|█████████▋| 485/500 [00:08<00:00, 59.51it/s][A
+2026-07-18:10:15:12 WARNING  [api.task:856] ruler_fwe: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
+2026-07-18:10:15:25 WARNING  [api.task:856] ruler_qa_hotpot: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
+For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
 Traceback (most recent call last):
+  File "/opt/conda/lib/python3.11/site-packages/urllib3/connection.py", line 198, in _new_conn
+    sock = connection.create_connection(
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/opt/conda/lib/python3.11/site-packages/urllib3/util/connection.py", line 85, in create_connection
+    raise err
+  File "/opt/conda/lib/python3.11/site-packages/urllib3/util/connection.py", line 73, in create_connection
+    sock.connect(sa)
+TimeoutError: [Errno 110] Connection timed out
+The above exception was the direct cause of the following exception:
+Traceback (most recent call last):
+  File "/opt/conda/lib/python3.11/site-packages/urllib3/connectionpool.py", line 787, in urlopen
+    response = self._make_request(
+               ^^^^^^^^^^^^^^^^^^^
+  File "/opt/conda/lib/python3.11/site-packages/urllib3/connectionpool.py", line 493, in _make_request
+    conn.request(
+  File "/opt/conda/lib/python3.11/site-packages/urllib3/connection.py", line 494, in request
+    self.endheaders()
+  File "/opt/conda/lib/python3.11/http/client.py", line 1298, in endheaders
+    self._send_output(message_body, encode_chunked=encode_chunked)
+  File "/opt/conda/lib/python3.11/http/client.py", line 1058, in _send_output
+    self.send(msg)
+  File "/opt/conda/lib/python3.11/http/client.py", line 996, in send
+    self.connect()
+  File "/opt/conda/lib/python3.11/site-packages/urllib3/connection.py", line 325, in connect
+    self.sock = self._new_conn()
+                ^^^^^^^^^^^^^^^^
+  File "/opt/conda/lib/python3.11/site-packages/urllib3/connection.py", line 207, in _new_conn
+    raise ConnectTimeoutError(
+urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x7f91c0eaee90>, 'Connection to curtis.ml.cmu.edu timed out. (connect timeout=None)')
+The above exception was the direct cause of the following exception:
+Traceback (most recent call last):
+  File "/opt/conda/lib/python3.11/site-packages/requests/adapters.py", line 644, in send
+    resp = conn.urlopen(
+           ^^^^^^^^^^^^^
+  File "/opt/conda/lib/python3.11/site-packages/urllib3/connectionpool.py", line 841, in urlopen
+    retries = retries.increment(
+              ^^^^^^^^^^^^^^^^^^
+  File "/opt/conda/lib/python3.11/site-packages/urllib3/util/retry.py", line 519, in increment
+    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
+    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='curtis.ml.cmu.edu', port=80): Max retries exceeded with url: /datasets/hotpot/hotpot_dev_distractor_v1.json (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f91c0eaee90>, 'Connection to curtis.ml.cmu.edu timed out. (connect timeout=None)'))
 During handling of the above exception, another exception occurred:
   File "/opt/conda/lib/python3.11/site-packages/lm_eval/tasks/__init__.py", line 430, in <lambda>
     lambda task: self._load_individual_task_or_group(task),
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/opt/conda/lib/python3.11/site-packages/lm_eval/tasks/__init__.py", line 328, in _load_individual_task_or_group
+    return _load_task(task_config, task=name_or_config)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/opt/conda/lib/python3.11/site-packages/lm_eval/tasks/__init__.py", line 288, in _load_task
+    task_object = ConfigurableTask(config=config)
+                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/opt/conda/lib/python3.11/site-packages/lm_eval/api/task.py", line 748, in __init__
+    self.download(self.config.dataset_kwargs)
+  File "/opt/conda/lib/python3.11/site-packages/lm_eval/api/task.py", line 860, in download
+    self.dataset = self.config.custom_dataset(
+                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/opt/conda/lib/python3.11/site-packages/lm_eval/tasks/ruler/qa_utils.py", line 240, in get_hotpotqa
+    return get_qa_dataset("hotpotqa", **kwargs)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/opt/conda/lib/python3.11/site-packages/lm_eval/tasks/ruler/qa_utils.py", line 222, in get_qa_dataset
+    qas, docs = read_hotpotqa()
+                ^^^^^^^^^^^^^^^
+  File "/opt/conda/lib/python3.11/site-packages/lm_eval/tasks/ruler/qa_utils.py", line 79, in read_hotpotqa
+    data = download_json(url)
+           ^^^^^^^^^^^^^^^^^^
+  File "/opt/conda/lib/python3.11/site-packages/lm_eval/tasks/ruler/qa_utils.py", line 38, in download_json
+    response = requests.get(url)
+               ^^^^^^^^^^^^^^^^^
+  File "/opt/conda/lib/python3.11/site-packages/requests/api.py", line 73, in get
+    return request("get", url, params=params, **kwargs)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/opt/conda/lib/python3.11/site-packages/requests/api.py", line 59, in request
+    return session.request(method=method, url=url, **kwargs)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/opt/conda/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
+    resp = self.send(prep, **send_kwargs)
+           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/opt/conda/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
+    r = adapter.send(request, **kwargs)
+        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+  File "/opt/conda/lib/python3.11/site-packages/requests/adapters.py", line 665, in send
+    raise ConnectTimeout(e, request=request)
+requests.exceptions.ConnectTimeout: HTTPConnectionPool(host='curtis.ml.cmu.edu', port=80): Max retries exceeded with url: /datasets/hotpot/hotpot_dev_distractor_v1.json (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f91c0eaee90>, 'Connection to curtis.ml.cmu.edu timed out. (connect timeout=None)'))