API.md

# Canter API and inference parameters

[Model card](README.md) · [Example gallery](GALLERY.md) ·
[Technical report](TECHNICAL_REPORT.md)

Canter exposes a high-level image pipeline and a lower-level latent inference
engine. Configuration uses frozen dataclasses and enums.

## Minimal use

```python
from canter import CanterPipeline

pipe = CanterPipeline.from_pretrained("data-archetype/canter")
output = pipe("A red rally car driving around a wet forest road")
image = output.image
```

### Generation arguments

| Parameter | Default | Description |
| --- | --- | --- |
| `prompts` | required | One prompt string or a sequence containing one prompt per image. |
| `negative_prompts` | `None` | Optional CFG negative prompt string or sequence. The count must match `prompts`. `None` uses the learned unconditional token. |
| `config` | `CanterPipelineConfig()` | Inference and output settings. |
| `initial_noise` | `None` | Optional float32 latent noise tensor with the configured batch and spatial shape. |
| `progress` | `None` | Optional callback receiving completed and total solver updates. |

`CanterPipeline` loads the flow-matching denoiser and text tokenizer with the
bundled
[`SmolLM2-360M`](https://huggingface.co/HuggingFaceTB/SmolLM2-360M) weights. It
compiles the selected text backend, downloads DINAC-AE-D2, generates latents,
and decodes them into images.

## Loading the pipeline

```python
from canter import CanterPipeline, TextAttentionBackend, WeightDType

pipe = CanterPipeline.from_pretrained(
    "data-archetype/canter",
    dtype=WeightDType.BFLOAT16,
    text_backend=TextAttentionBackend.JAGGED,
    device="cuda",
    revision=None,
    cache_dir=None,
    compile_model=True,
)
```

### `CanterPipeline.from_pretrained`

| Parameter | Default | Description |
| --- | --- | --- |
| `path_or_repo_id` | required | A Hugging Face repository ID or downloaded model repository directory. |
| `dtype` | `WeightDType.BFLOAT16` | Weight storage dtype. Compute still uses bfloat16 AMP with explicit float32 operations. |
| `text_backend` | `TextAttentionBackend.JAGGED` | Text refinement and cross-attention layout. |
| `device` | `"cuda"` | CUDA device string or `torch.device`. |
| `revision` | `None` | Hugging Face branch, commit, or immutable release tag. `None` uses the package's pinned default release. Pass `"main"` explicitly to follow the moving repository head. |
| `cache_dir` | `None` | Optional Hugging Face cache directory. |
| `compile_model` | `True` | Compile the selected inference kernels. Compilation errors are reported directly. |
| `vae` | `None` | Optional compatible `CanterVae` instance. `None` downloads DINAC-AE-D2 automatically. |

### Weight dtypes

| Enum | Value | Description |
| --- | --- | --- |
| `WeightDType.BFLOAT16` | `"bfloat16"` | Default compact release with required float32 parameter islands retained. |
| `WeightDType.FLOAT32` | `"float32"` | Full-float32 weight storage release. Model compute remains bfloat16 AMP. |

### Text backends

| Enum | Value | Description |
| --- | --- | --- |
| `TextAttentionBackend.JAGGED` | `"jagged"` | Packed variable-length text using CUDA FlashAttention and jagged NestedTensor kernels. |
| `TextAttentionBackend.DENSE` | `"dense"` | Dense text tensors. Useful for batch size one and uniform prompt lengths. |

Only the selected backend is compiled. The backend cannot be changed after
compilation.

## Pipeline configuration

```python
from canter import (
    CanterInferenceConfig,
    CanterOutputType,
    CanterPipelineConfig,
)

config = CanterPipelineConfig(
    inference=CanterInferenceConfig(),
    output_type=CanterOutputType.PIL,
)
output = pipe("A portrait lit by a large north-facing window", config=config)
```

### `CanterPipelineConfig`

| Field | Default | Description |
| --- | --- | --- |
| `inference` | `CanterInferenceConfig()` | Latent generation settings. |
| `output_type` | `CanterOutputType.PIL` | Selected decoded or latent output representation. |

### Output types

| Enum | Result |
| --- | --- |
| `CanterOutputType.PIL` | `output.images` contains one PIL image per prompt. `output.image` returns the sole image for batch size one. |
| `CanterOutputType.TENSOR` | `output.image_tensor` contains clamped float32 RGB pixels in `[-1, 1]`. |
| `CanterOutputType.LATENT` | VAE decoding is skipped. `output.latents` contains whitened float32 model latents. |

Every output also contains the descending float32 solver schedule in
`output.schedule`.

## Inference configuration

### Defaults

```python
from canter import (
    CanterInferenceConfig,
    CfgGuidance,
    PdgCurve,
    PdgGuidance,
    PdgMode,
    Schedule,
    Solver,
)

config = CanterInferenceConfig(
    height=1216,
    width=832,
    steps=50,
    solver=Solver.ABM2,
    schedule=Schedule.BETA,
    log_snr_shift=0.0,
    cfg=CfgGuidance(
        enabled=False,
        scale=None,
        start_step=0,
        stop_step=None,
    ),
    pdg=PdgGuidance(
        enabled=True,
        mode=PdgMode.FULL,
        curve=PdgCurve.CONSTANT,
        noisy_scale=2.5,
        clean_scale=2.5,
        power=3.0,
        start_step=0,
        stop_step=None,
    ),
    self_attention_gain=-0.03,
    euler_maruyama_multiplier=1.0,
    seed=42,
    generator=None,
)
```

### `CanterInferenceConfig`

| Field | Default | Description |
| --- | --- | --- |
| `height` | `1216` | Output height in pixels. Must be positive and divisible by 16. |
| `width` | `832` | Output width in pixels. Must be positive and divisible by 16. |
| `steps` | `50` | Number of solver state updates. A run with 50 updates uses 51 schedule points. |
| `solver` | `Solver.ABM2` | Numerical solver. |
| `schedule` | `Schedule.BETA` | Timestep spacing. |
| `log_snr_shift` | `0.0` | Additive log-SNR schedule shift. Zero leaves the selected schedule unchanged. |
| `cfg` | disabled | Classifier-free guidance settings. |
| `pdg` | full-path PDG 2.5 | Path-drop guidance settings. |
| `self_attention_gain` | `-0.03` | Gain applied exclusively to image self-attention on the main denoiser path. |
| `euler_maruyama_multiplier` | `1.0` | Non-negative stochastic noise multiplier used only by Euler-Maruyama. |
| `seed` | `42` | Random seed. Set to `None` when supplying `generator`. |
| `generator` | `None` | Optional CUDA `torch.Generator` on the same device as the model. Exactly one of `seed` and `generator` is required. |

The seed controls latent noise, SPRINT routing, and stochastic solver
operations. The VAE decoder retains its published seed behavior.

### PNG metadata

Images downloaded from the Gradio interface contain a `canter` PNG text field
with compact JSON. The object begins with the prompt, effective per-image seed,
width, height, steps, solver, and schedule. It then records PDG, CFG,
self-attention gain, logSNR shift, Euler-Maruyama multiplier, numbered release,
and weight dtype.

### Solvers

| Enum | Value | Description |
| --- | --- | --- |
| `Solver.EULER` | `"euler"` | First-order deterministic Euler updates. |
| `Solver.EULER_MARUYAMA` | `"euler_maruyama"` | Stochastic reverse-SDE updates. Uses `euler_maruyama_multiplier`. |
| `Solver.DPMPP_2M` | `"dpmpp_2m"` | Flow-matching DPM++ 2M updates with finite start handling. |
| `Solver.ABM2` | `"abm2"` | Variable-step Adams-Bashforth-Moulton updates with corrected-state reevaluation. |

ABM2 is the default and performs additional denoiser evaluations for its
corrected states.

### Schedules

| Enum | Value | Description |
| --- | --- | --- |
| `Schedule.LINEAR` | `"linear"` | Uniform spacing from noisy to clean. |
| `Schedule.BETA` | `"beta"` | Beta(0.6, 0.6) quantile spacing with more schedule density near the endpoints. |

Schedules and solver state remain float32.

## CFG

```python
from canter import (
    CanterInferenceConfig,
    CanterPipelineConfig,
    CfgGuidance,
    PdgGuidance,
    PdgMode,
)

cfg = CfgGuidance(
    enabled=True,
    scale=3.0,
    start_step=0,
    stop_step=29,
)

output = pipe(
    "A studio portrait with soft natural light",
    negative_prompts="oversaturated, harsh contrast",
    config=CanterPipelineConfig(
        inference=CanterInferenceConfig(
            cfg=cfg,
            pdg=PdgGuidance(mode=PdgMode.COMBINED_CFG_PDG),
        ),
    ),
)
```

### `CfgGuidance`

| Field | Default | Description |
| --- | --- | --- |
| `enabled` | `False` | Enable classifier-free guidance. |
| `scale` | `None` | Non-negative guidance scale. Required when CFG is enabled or when the selected PDG mode uses CFG. |
| `start_step` | `0` | First active solver update, inclusive. |
| `stop_step` | `None` | Last active solver update, inclusive. `None` selects the final update. |

Step indices run from `0` through `steps - 1`.

An explicit negative prompt replaces the learned unconditional token on CFG
branches. Negative prompts require enabled CFG or a CFG-dependent PDG mode.
For prompt batches, supply one negative prompt per positive prompt.

## PDG

```python
from canter import PdgCurve, PdgGuidance, PdgMode

pdg = PdgGuidance(
    enabled=True,
    mode=PdgMode.FULL,
    curve=PdgCurve.POWER,
    noisy_scale=2.0,
    clean_scale=2.5,
    power=3.0,
    start_step=0,
    stop_step=None,
)
```

### `PdgGuidance`

| Field | Default | Description |
| --- | --- | --- |
| `enabled` | `True` | Enable path-drop guidance. |
| `mode` | `PdgMode.FULL` | Path and CFG interaction policy. |
| `curve` | `PdgCurve.CONSTANT` | Scale interpolation from noisy to clean. |
| `noisy_scale` | `2.5` | PDG scale at the first noisy schedule point. |
| `clean_scale` | `2.5` | PDG scale at the final clean schedule point. |
| `power` | `3.0` | Positive exponent used by the power curve. |
| `start_step` | `0` | First active solver update, inclusive. |
| `stop_step` | `None` | Last active solver update, inclusive. `None` selects the final update. |

Constant PDG requires equal `noisy_scale` and `clean_scale`. Disabled PDG
requires `enabled=False` and `mode=PdgMode.NONE`.

Increasing the PDG scale sharpens the generated distribution. Moderate values
improve image structure and fine detail at the cost of reduced variety.
Pushing the scale too far can introduce structural defects, excessive
contrast, and oversaturation.

### PDG curves

| Enum | Value | Scale behavior |
| --- | --- | --- |
| `PdgCurve.CONSTANT` | `"constant"` | Uses one scale throughout inference. |
| `PdgCurve.LINEAR` | `"linear"` | Interpolates linearly from `noisy_scale` to `clean_scale`. |
| `PdgCurve.POWER` | `"power"` | Interpolates with `position ** power`. |

### PDG modes

| Enum | Value | Behavior |
| --- | --- | --- |
| `PdgMode.NONE` | `"none"` | No PDG branch. Required when PDG is disabled. |
| `PdgMode.FULL` | `"full"` | Guides from the middle-skipped path toward the full main path. |
| `PdgMode.THREE_QUARTER` | `"three_quarter"` | Guides from the 75 percent SPRINT path toward the full main path. |
| `PdgMode.ALTERNATE_PDG_FIRST` | `"alternate_pdg_first"` | Alternates PDG on even updates and CFG on odd updates within the PDG window. |
| `PdgMode.ALTERNATE_CFG_FIRST` | `"alternate_cfg_first"` | Alternates CFG on even updates and PDG on odd updates within the PDG window. |
| `PdgMode.COMBINED_CFG_PDG` | `"combined_cfg_pdg"` | Evaluates CFG and PDG together and averages their guidance deltas. |
| `PdgMode.PDG_WITH_ALTERNATING_CFG` | `"pdg_with_alternating_cfg"` | Applies PDG on every active update and adds CFG on odd updates. |
| `PdgMode.CFG_TO_PDG` | `"cfg_to_pdg"` | Uses CFG before the PDG window, then uses full-path PDG inside the window. |

The five compound modes require `cfg.scale`. `start_step=0` with
`PdgMode.CFG_TO_PDG` begins directly with PDG.

## Self-attention gain

`self_attention_gain` changes image self-attention only on the full main path.
The model multiplies main-path attention queries by
`exp(self_attention_gain)`. Negative values reduce the attention-logit scale
and therefore increase the effective softmax temperature. This softens the
main prediction and helps moderate PDG oversaturation. Text self-attention,
cross-attention, and weak guidance paths retain their trained scales.

The default is `-0.03`. A value of `0.0` uses the trained main-path
self-attention scale without adjustment. Smaller images generally benefit from
more negative values. As image size increases, the gain should move closer to
zero.

## Custom configuration example

```python
from canter import (
    CanterInferenceConfig,
    CanterOutputType,
    CanterPipelineConfig,
    CfgGuidance,
    PdgCurve,
    PdgGuidance,
    PdgMode,
    Schedule,
    Solver,
)

inference = CanterInferenceConfig(
    height=1024,
    width=1024,
    steps=40,
    solver=Solver.DPMPP_2M,
    schedule=Schedule.LINEAR,
    log_snr_shift=0.5,
    cfg=CfgGuidance(
        enabled=True,
        scale=3.0,
        start_step=0,
        stop_step=11,
    ),
    pdg=PdgGuidance(
        enabled=True,
        mode=PdgMode.CFG_TO_PDG,
        curve=PdgCurve.POWER,
        noisy_scale=2.0,
        clean_scale=2.5,
        power=3.0,
        start_step=12,
        stop_step=39,
    ),
    self_attention_gain=-0.03,
    euler_maruyama_multiplier=1.0,
    seed=123,
    generator=None,
)

config = CanterPipelineConfig(
    inference=inference,
    output_type=CanterOutputType.PIL,
)
image = pipe("A glass greenhouse during heavy rain", config=config).image
```

## Prompt batches

A sequence of prompts generates one image per prompt:

```python
output = pipe(
    [
        "A windswept beach under dark clouds",
        "A sunlit kitchen with white tiled walls",
    ]
)

first, second = output.images
```

Batch size one is the primary inference path. The jagged backend packs active
prompt tokens without padding them through text refinement and
cross-attention. Prompts longer than 512 tokens are truncated with a warning.

## Latent generation

Use `CanterInferenceEngine` to generate latents without loading or calling the
VAE:

```python
from canter import CanterComponents, CanterInferenceEngine

components = CanterComponents.from_pretrained("data-archetype/canter")
engine = CanterInferenceEngine(components)
output = engine.generate("A mountain road in winter")

latents = output.latents
schedule = output.schedule
```

`CanterInferenceEngine.generate` accepts:

| Parameter | Default | Description |
| --- | --- | --- |
| `prompts` | required | One string or a sequence of strings. |
| `negative_prompts` | `None` | Optional negative prompt string or sequence for CFG branches. The count must match `prompts`. |
| `config` | `CanterInferenceConfig()` | Latent inference settings. |
| `initial_noise` | `None` | Optional float32 noise with shape `[batch, 128, height / 16, width / 16]`. |
| `progress` | `None` | Optional callback receiving `(completed_updates, total_updates)`. |

An empty or whitespace-only prompt uses the learned unconditional token without
running the text encoder. Every prompt in a batch must be either blank or
nonblank.

The returned latents use float32 and channels-last memory format.

## Pipeline metadata

`pipe.metadata` records the resolved release, weight dtype, source
digests, text-encoder revision, VAE repository, and resolved immutable VAE
revision. Applications that require reproducibility should store this metadata
with their outputs and pin a release tag.