Any-to-Any
Transformers
Safetensors
nlduni
feature-extraction
custom_code
image-text-to-text
text-to-image
image-to-image
Instructions to use nvidia/NLD-Uni with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use nvidia/NLD-Uni with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("nvidia/NLD-Uni", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
Shufan Li commited on
Commit ·
a5d1066
1
Parent(s): 300c478
fix readme
Browse files
README.md
CHANGED
|
@@ -18,6 +18,52 @@ For image understanding, NLD-Uni encodes images into continuous visual features
|
|
| 18 |
|
| 19 |
This card describes the NLD-Uni architecture generally.
|
| 20 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 21 |
### License/Terms of Use
|
| 22 |
|
| 23 |
GOVERNING TERMS: Use of this model is governed by the [NVIDIA Open Source License Agreement](https://huggingface.co/nvidia/NL-Diffusion-Image/raw/main/LICENSE).
|
|
@@ -181,47 +227,3 @@ Users are responsible for model inputs and outputs and for implementing appropri
|
|
| 181 |
|
| 182 |
Report model quality concerns, security vulnerabilities, or NVIDIA AI concerns through [NVIDIA’s reporting channel](https://www.nvidia.com/en-us/support/submit-security-vulnerability/).
|
| 183 |
|
| 184 |
-
## Environment
|
| 185 |
-
|
| 186 |
-
Install the dependencies in `requirements.txt` with Python 3.13 and CUDA 12.6. Install PyTorch before building FlashAttention:
|
| 187 |
-
|
| 188 |
-
```bash
|
| 189 |
-
pip install -r requirements.txt
|
| 190 |
-
pip install --no-build-isolation flash-attn==2.8.0.post2
|
| 191 |
-
```
|
| 192 |
-
|
| 193 |
-
Validated on an NVIDIA H100 80 GB. The eight BF16 weight shards contain all 1,557 active model tensors. `weights_manifest.json` records their hashes.
|
| 194 |
-
|
| 195 |
-
## Python inference
|
| 196 |
-
|
| 197 |
-
```python
|
| 198 |
-
import torch
|
| 199 |
-
from transformers import AutoModelForCausalLM
|
| 200 |
-
|
| 201 |
-
model = AutoModelForCausalLM.from_pretrained(
|
| 202 |
-
'/path/to/hf_checkpoint', # or 'organization/model-name'
|
| 203 |
-
trust_remote_code=True,
|
| 204 |
-
torch_dtype=torch.bfloat16,
|
| 205 |
-
).to('cuda').eval()
|
| 206 |
-
|
| 207 |
-
image = model.text_to_image('A red ceramic mug on a wooden table.', seed=42)
|
| 208 |
-
image.save('mug.png')
|
| 209 |
-
|
| 210 |
-
edited = model.image_edit(image, 'Change the mug to blue.', seed=42)
|
| 211 |
-
edited.save('blue_mug.png')
|
| 212 |
-
|
| 213 |
-
answer = model.generate('What color is the mug?', image=edited)
|
| 214 |
-
print(answer)
|
| 215 |
-
```
|
| 216 |
-
|
| 217 |
-
Image inputs accept Pillow images or local file paths. Image methods return Pillow images; `generate` returns text by default. Token IDs can be supplied through `input_ids` for text prompts, and `return_token_ids=True` returns the generated token tensor. The `inference.py` command exposes each task.
|
| 218 |
-
|
| 219 |
-
## Minimal command-line inference
|
| 220 |
-
|
| 221 |
-
```bash
|
| 222 |
-
python inference.py /path/to/hf_checkpoint --task t2i --prompt 'A red mug.' --output mug.png
|
| 223 |
-
python inference.py /path/to/hf_checkpoint --task edit --image mug.png --prompt 'Make the mug blue.' --output blue.png
|
| 224 |
-
python inference.py /path/to/hf_checkpoint --task understand --image blue.png --prompt 'What color is the mug?'
|
| 225 |
-
```
|
| 226 |
-
|
| 227 |
-
Replace the model argument with the published Hugging Face model ID for remote loading. The standalone script only uses Transformers and this repository's model code.
|
|
|
|
| 18 |
|
| 19 |
This card describes the NLD-Uni architecture generally.
|
| 20 |
|
| 21 |
+
## Environment
|
| 22 |
+
|
| 23 |
+
Install the dependencies in `requirements.txt` with Python 3.13 and CUDA 12.6. Install PyTorch before building FlashAttention:
|
| 24 |
+
|
| 25 |
+
```bash
|
| 26 |
+
pip install -r requirements.txt
|
| 27 |
+
pip install --no-build-isolation flash-attn==2.8.0.post2
|
| 28 |
+
```
|
| 29 |
+
|
| 30 |
+
Validated on an NVIDIA H100 80 GB. The eight BF16 weight shards contain all 1,557 active model tensors. `weights_manifest.json` records their hashes.
|
| 31 |
+
|
| 32 |
+
## Python inference
|
| 33 |
+
|
| 34 |
+
```python
|
| 35 |
+
import torch
|
| 36 |
+
from transformers import AutoModelForCausalLM
|
| 37 |
+
|
| 38 |
+
model = AutoModelForCausalLM.from_pretrained(
|
| 39 |
+
'/path/to/hf_checkpoint', # or 'organization/model-name'
|
| 40 |
+
trust_remote_code=True,
|
| 41 |
+
torch_dtype=torch.bfloat16,
|
| 42 |
+
).to('cuda').eval()
|
| 43 |
+
|
| 44 |
+
image = model.text_to_image('A red ceramic mug on a wooden table.', seed=42)
|
| 45 |
+
image.save('mug.png')
|
| 46 |
+
|
| 47 |
+
edited = model.image_edit(image, 'Change the mug to blue.', seed=42)
|
| 48 |
+
edited.save('blue_mug.png')
|
| 49 |
+
|
| 50 |
+
answer = model.generate('What color is the mug?', image=edited)
|
| 51 |
+
print(answer)
|
| 52 |
+
```
|
| 53 |
+
|
| 54 |
+
Image inputs accept Pillow images or local file paths. Image methods return Pillow images; `generate` returns text by default. Token IDs can be supplied through `input_ids` for text prompts, and `return_token_ids=True` returns the generated token tensor. The `inference.py` command exposes each task.
|
| 55 |
+
|
| 56 |
+
## Minimal command-line inference
|
| 57 |
+
|
| 58 |
+
```bash
|
| 59 |
+
python inference.py /path/to/hf_checkpoint --task t2i --prompt 'A red mug.' --output mug.png
|
| 60 |
+
python inference.py /path/to/hf_checkpoint --task edit --image mug.png --prompt 'Make the mug blue.' --output blue.png
|
| 61 |
+
python inference.py /path/to/hf_checkpoint --task understand --image blue.png --prompt 'What color is the mug?'
|
| 62 |
+
```
|
| 63 |
+
|
| 64 |
+
Replace the model argument with the published Hugging Face model ID for remote loading. The standalone script only uses Transformers and this repository's model code.
|
| 65 |
+
|
| 66 |
+
|
| 67 |
### License/Terms of Use
|
| 68 |
|
| 69 |
GOVERNING TERMS: Use of this model is governed by the [NVIDIA Open Source License Agreement](https://huggingface.co/nvidia/NL-Diffusion-Image/raw/main/LICENSE).
|
|
|
|
| 227 |
|
| 228 |
Report model quality concerns, security vulnerabilities, or NVIDIA AI concerns through [NVIDIA’s reporting channel](https://www.nvidia.com/en-us/support/submit-security-vulnerability/).
|
| 229 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|