| # generate-offline-data |
|
|
| Generates training data for speculator models by extracting hidden states from a running vLLM server. Connects to a vLLM endpoint via the OpenAI-compatible API and saves output as individual `.safetensors` files for offline training. |
|
|
| ## Features |
|
|
| - **Automatic resumption** — Detects existing `.safetensors` files in the output directory and skips already-completed samples, so interrupted runs can be resumed without reprocessing. |
| - **Error handling with auto-retries** — Failed requests are automatically retried up to `--max-retries` times. Samples that still fail are skipped by default, allowing the rest of the dataset to complete. |
| - **Consecutive failure detection** — Aborts early after `--max-consecutive-errors` consecutive failures to avoid silently churning through the dataset when the server is unreachable. |
| - **Async concurrency** — Sends multiple requests to the vLLM server in parallel, controlled by `--concurrency`, for high throughput. |
| - **Output validation** — Optional `--validate-outputs` flag verifies that saved hidden states match expected token IDs and sequence lengths. |
|
|
| ## Basic Usage |
|
|
| ```bash |
| speculators generate-offline-data \ |
| --preprocessed-data ./preprocessed_dataset \ |
| --output ./training_data \ |
| --max-samples 5000 |
| ``` |
|
|
| ## Arguments |
|
|
| ### Model Arguments |
|
|
| - **`--endpoint`** (str, default: `http://localhost:8000/v1`) The address of the vLLM instance to use for hidden states generation. The vLLM instance must be configured for hidden states extraction (see [launch_vllm.py](launch_vllm.md)). |
|
|
| - **`--model`** (str, default: `None`) HuggingFace model ID or local path for the target model. Used for verification only - the model is auto-detected from the vLLM endpoint. |
|
|
| ### Data Arguments |
|
|
| - **`--preprocessed-data`** (str, required) Path to preprocessed dataset (produced by [prepare-data](prepare_data.md)). |
|
|
| - **`--max-samples`** (int, default: `None`) Maximum number of samples to process. If `None`, processes all samples. |
|
|
| ### Output Arguments |
|
|
| - **`--output`** (str, default: `None`) Directory to save generated `.safetensors` files. Defaults to `<preprocessed-data>/hidden_states`. |
|
|
| ### Hidden States Generation Arguments |
|
|
| - **`--concurrency`** (int, default: `32`) Number of active vLLM requests at a time. The number of async workers is set to `2 * concurrency`. |
|
|
| - **`--validate-outputs`** (flag) Load generated safetensor files and verify that output token IDs match prompt tokens and hidden states sequence length matches the number of tokens. |
|
|
| - **`--request-timeout`** (float) Timeout in seconds for each individual vLLM request. |
|
|
| - **`--max-retries`** (int) Maximum number of retry attempts per request on failure. |
|
|
| - **`--fail-on-error`** (flag) Abort when a request fails after all retries. By default, failed samples are skipped. |
|
|
| - **`--max-consecutive-errors`** (int, default: value of `--concurrency`) Abort after this many consecutive sample failures (each sample already retried `--max-retries` times). Prevents silently churning through the entire dataset when the server is down. Ignored when `--fail-on-error` is set. |
|
|
| ### Multi-Node Arguments |
|
|
| - **`--world-size`** (int, default: `1`) Number of nodes participating in data generation. Each node is assigned a contiguous, non-overlapping chunk of the dataset. This is the number of nodes, not the number of GPUs. |
|
|
| - **`--rank`** (int, default: `0`) Zero-based index of the current node. Must be in the range `[0, world-size)`. |
|
|
| ## Full Example |
|
|
| ```bash |
| speculators generate-offline-data \ |
| --endpoint http://localhost:8000/v1 \ |
| --preprocessed-data ./preprocessed_dataset \ |
| --output ./hidden_states \ |
| --concurrency 64 \ |
| --validate-outputs |
| ``` |
|
|
