File size: 3,706 Bytes
a181ec9 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 | # generate-offline-data
Generates training data for speculator models by extracting hidden states from a running vLLM server. Connects to a vLLM endpoint via the OpenAI-compatible API and saves output as individual `.safetensors` files for offline training.
## Features
- **Automatic resumption** — Detects existing `.safetensors` files in the output directory and skips already-completed samples, so interrupted runs can be resumed without reprocessing.
- **Error handling with auto-retries** — Failed requests are automatically retried up to `--max-retries` times. Samples that still fail are skipped by default, allowing the rest of the dataset to complete.
- **Consecutive failure detection** — Aborts early after `--max-consecutive-errors` consecutive failures to avoid silently churning through the dataset when the server is unreachable.
- **Async concurrency** — Sends multiple requests to the vLLM server in parallel, controlled by `--concurrency`, for high throughput.
- **Output validation** — Optional `--validate-outputs` flag verifies that saved hidden states match expected token IDs and sequence lengths.
## Basic Usage
```bash
speculators generate-offline-data \
--preprocessed-data ./preprocessed_dataset \
--output ./training_data \
--max-samples 5000
```
## Arguments
### Model Arguments
- **`--endpoint`** (str, default: `http://localhost:8000/v1`) The address of the vLLM instance to use for hidden states generation. The vLLM instance must be configured for hidden states extraction (see [launch_vllm.py](launch_vllm.md)).
- **`--model`** (str, default: `None`) HuggingFace model ID or local path for the target model. Used for verification only - the model is auto-detected from the vLLM endpoint.
### Data Arguments
- **`--preprocessed-data`** (str, required) Path to preprocessed dataset (produced by [prepare-data](prepare_data.md)).
- **`--max-samples`** (int, default: `None`) Maximum number of samples to process. If `None`, processes all samples.
### Output Arguments
- **`--output`** (str, default: `None`) Directory to save generated `.safetensors` files. Defaults to `<preprocessed-data>/hidden_states`.
### Hidden States Generation Arguments
- **`--concurrency`** (int, default: `32`) Number of active vLLM requests at a time. The number of async workers is set to `2 * concurrency`.
- **`--validate-outputs`** (flag) Load generated safetensor files and verify that output token IDs match prompt tokens and hidden states sequence length matches the number of tokens.
- **`--request-timeout`** (float) Timeout in seconds for each individual vLLM request.
- **`--max-retries`** (int) Maximum number of retry attempts per request on failure.
- **`--fail-on-error`** (flag) Abort when a request fails after all retries. By default, failed samples are skipped.
- **`--max-consecutive-errors`** (int, default: value of `--concurrency`) Abort after this many consecutive sample failures (each sample already retried `--max-retries` times). Prevents silently churning through the entire dataset when the server is down. Ignored when `--fail-on-error` is set.
### Multi-Node Arguments
- **`--world-size`** (int, default: `1`) Number of nodes participating in data generation. Each node is assigned a contiguous, non-overlapping chunk of the dataset. This is the number of nodes, not the number of GPUs.
- **`--rank`** (int, default: `0`) Zero-based index of the current node. Must be in the range `[0, world-size)`.
## Full Example
```bash
speculators generate-offline-data \
--endpoint http://localhost:8000/v1 \
--preprocessed-data ./preprocessed_dataset \
--output ./hidden_states \
--concurrency 64 \
--validate-outputs
```
|