| # Serve Nemotron 3.5 Parse |
|
|
| ## Native DFlash2 |
|
|
| Download both models together. Use vLLM with native Nemotron Parse and |
| `DFlash2DraftModel` support; no model-specific plugin is needed. |
|
|
| ```bash |
| hf download nvidia/Nemotron-3.5-Parse --local-dir ./parse35 |
| vllm serve ./parse35 \ |
| --served-model-name nemotron-parse \ |
| --trust-remote-code \ |
| --speculative-config '{"method":"dflash","model":"./parse35/dflash2","num_speculative_tokens":32}' |
| ``` |
|
|
| The speculator path is a local subdirectory, not a second Hub model ID. |
| Use 8, 16, or 32 speculative tokens to tune for your workload; the length stays |
| fixed for that server. See [performance](../dflash2/README.md#performance). |
| For base inference, omit `--speculative-config`. |
|
|
| Use the [model card's HTTP or Transformers examples](../README.md#run-parse). |
| Transformers does not need the serving plugin and uses normal decoding. |
| For overlapping document/crop requests, use the downloaded |
| [`DocumentParser` helpers](postprocessing.md). |
|
|
| ## Prompts |
|
|
| ```python |
| TASK_PROMPT = "</s><s><predict_bbox><predict_classes><output_markdown><predict_text_in_pic>" |
| ``` |
|
|
| Use this task string in Transformers and HTTP requests. Preserve the class and |
| coordinate tokens in generated text for postprocessing. For offline vLLM, |
| prepend the image placeholder directly: `"<pad>" + TASK_PROMPT`, with |
| `add_special_tokens=False`. Do not insert a newline between them. |
|
|
| The model takes RGB page images; render PDFs to pages first. Coordinates refer |
| to the evaluated image content. The [prompt controls](../README.md#task-prompts) |
| request boxes, classes, or plain text; table export formats are chosen after inference. |
|
|
| ## Throughput tuning |
|
|
| Concurrency is the number of outstanding requests across all stages, not the |
| number of documents. Increase it while observing memory, latency and completed |
| page throughput. Double and triple pass reuse page/crop images, so retain the |
| multimodal processor cache. Prefix caching, batching budgets and draft length |
| can affect different workloads differently. |
|
|
| Use [the measured profiles and protocols](../reference/dflash2-passes/README.md) |
| when reproducing a number. The basic launch command above prioritizes usability; |
| it does not set every benchmarking option. A page that hits its token limit is |
| partial output, even if the benchmark itself finishes successfully. |
|
|
| ## One-token MTP |
|
|
| MTP weights are included in the target weights. The established MTP serving path |
| uses the repository's serving plugin. Follow the |
| [reference installation](serving-reference.md#installation-reference-mtp-plugin-environment) |
| and [MTP command](serving-reference.md#one-token-mtp) in a separate environment. |
| That guide preserves its pinned reproduction setup. Do not install its serving |
| extras into the native DFlash2 environment. Choose one speculator per server. |
|
|
