canter / API.md

Publish Canter v0001

47cf0df verified about 4 hours ago

preview code

Raw

History Blame Contribute Delete

15.3 kB

Canter API and inference parameters

Model card · Example gallery · Technical report

Canter exposes a high-level image pipeline and a lower-level latent inference engine. Configuration uses frozen dataclasses and enums.

Minimal use

from canter import CanterPipeline

pipe = CanterPipeline.from_pretrained("data-archetype/canter")
output = pipe("A red rally car driving around a wet forest road")
image = output.image

Generation arguments

Parameter	Default	Description
`prompts`	required	One prompt string or a sequence containing one prompt per image.
`negative_prompts`	`None`	Optional CFG negative prompt string or sequence. The count must match `prompts`. `None` uses the learned unconditional token.
`config`	`CanterPipelineConfig()`	Inference and output settings.
`initial_noise`	`None`	Optional float32 latent noise tensor with the configured batch and spatial shape.
`progress`	`None`	Optional callback receiving completed and total solver updates.

CanterPipeline loads the flow-matching denoiser and text tokenizer with the bundled SmolLM2-360M weights. It compiles the selected text backend, downloads DINAC-AE-D2, generates latents, and decodes them into images.

Loading the pipeline

from canter import CanterPipeline, TextAttentionBackend, WeightDType

pipe = CanterPipeline.from_pretrained(
    "data-archetype/canter",
    dtype=WeightDType.BFLOAT16,
    text_backend=TextAttentionBackend.JAGGED,
    device="cuda",
    revision=None,
    cache_dir=None,
    compile_model=True,
)

`CanterPipeline.from_pretrained`

Parameter	Default	Description
`path_or_repo_id`	required	A Hugging Face repository ID or downloaded model repository directory.
`dtype`	`WeightDType.BFLOAT16`	Weight storage dtype. Compute still uses bfloat16 AMP with explicit float32 operations.
`text_backend`	`TextAttentionBackend.JAGGED`	Text refinement and cross-attention layout.
`device`	`"cuda"`	CUDA device string or `torch.device`.
`revision`	`None`	Hugging Face branch, commit, or immutable release tag. `None` uses the package's pinned default release. Pass `"main"` explicitly to follow the moving repository head.
`cache_dir`	`None`	Optional Hugging Face cache directory.
`compile_model`	`True`	Compile the selected inference kernels. Compilation errors are reported directly.
`vae`	`None`	Optional compatible `CanterVae` instance. `None` downloads DINAC-AE-D2 automatically.

Weight dtypes

Enum	Value	Description
`WeightDType.BFLOAT16`	`"bfloat16"`	Default compact release with required float32 parameter islands retained.
`WeightDType.FLOAT32`	`"float32"`	Full-float32 weight storage release. Model compute remains bfloat16 AMP.

Text backends

Enum	Value	Description
`TextAttentionBackend.JAGGED`	`"jagged"`	Packed variable-length text using CUDA FlashAttention and jagged NestedTensor kernels.
`TextAttentionBackend.DENSE`	`"dense"`	Dense text tensors. Useful for batch size one and uniform prompt lengths.

Only the selected backend is compiled. The backend cannot be changed after compilation.

Pipeline configuration

from canter import (
    CanterInferenceConfig,
    CanterOutputType,
    CanterPipelineConfig,
)

config = CanterPipelineConfig(
    inference=CanterInferenceConfig(),
    output_type=CanterOutputType.PIL,
)
output = pipe("A portrait lit by a large north-facing window", config=config)

`CanterPipelineConfig`

Field	Default	Description
`inference`	`CanterInferenceConfig()`	Latent generation settings.
`output_type`	`CanterOutputType.PIL`	Selected decoded or latent output representation.

Output types

Enum	Result
`CanterOutputType.PIL`	`output.images` contains one PIL image per prompt. `output.image` returns the sole image for batch size one.
`CanterOutputType.TENSOR`	`output.image_tensor` contains clamped float32 RGB pixels in `[-1, 1]`.
`CanterOutputType.LATENT`	VAE decoding is skipped. `output.latents` contains whitened float32 model latents.

Every output also contains the descending float32 solver schedule in output.schedule.

Inference configuration

Defaults

from canter import (
    CanterInferenceConfig,
    CfgGuidance,
    PdgCurve,
    PdgGuidance,
    PdgMode,
    Schedule,
    Solver,
)

config = CanterInferenceConfig(
    height=1216,
    width=832,
    steps=50,
    solver=Solver.ABM2,
    schedule=Schedule.BETA,
    log_snr_shift=0.0,
    cfg=CfgGuidance(
        enabled=False,
        scale=None,
        start_step=0,
        stop_step=None,
    ),
    pdg=PdgGuidance(
        enabled=True,
        mode=PdgMode.FULL,
        curve=PdgCurve.CONSTANT,
        noisy_scale=2.5,
        clean_scale=2.5,
        power=3.0,
        start_step=0,
        stop_step=None,
    ),
    self_attention_gain=-0.03,
    euler_maruyama_multiplier=1.0,
    seed=42,
    generator=None,
)

`CanterInferenceConfig`

Field	Default	Description
`height`	`1216`	Output height in pixels. Must be positive and divisible by 16.
`width`	`832`	Output width in pixels. Must be positive and divisible by 16.
`steps`	`50`	Number of solver state updates. A run with 50 updates uses 51 schedule points.
`solver`	`Solver.ABM2`	Numerical solver.
`schedule`	`Schedule.BETA`	Timestep spacing.
`log_snr_shift`	`0.0`	Additive log-SNR schedule shift. Zero leaves the selected schedule unchanged.
`cfg`	disabled	Classifier-free guidance settings.
`pdg`	full-path PDG 2.5	Path-drop guidance settings.
`self_attention_gain`	`-0.03`	Gain applied exclusively to image self-attention on the main denoiser path.
`euler_maruyama_multiplier`	`1.0`	Non-negative stochastic noise multiplier used only by Euler-Maruyama.
`seed`	`42`	Random seed. Set to `None` when supplying `generator`.
`generator`	`None`	Optional CUDA `torch.Generator` on the same device as the model. Exactly one of `seed` and `generator` is required.

The seed controls latent noise, SPRINT routing, and stochastic solver operations. The VAE decoder retains its published seed behavior.

PNG metadata

Images downloaded from the Gradio interface contain a canter PNG text field with compact JSON. The object begins with the prompt, effective per-image seed, width, height, steps, solver, and schedule. It then records PDG, CFG, self-attention gain, logSNR shift, Euler-Maruyama multiplier, numbered release, and weight dtype.

Solvers

Enum	Value	Description
`Solver.EULER`	`"euler"`	First-order deterministic Euler updates.
`Solver.EULER_MARUYAMA`	`"euler_maruyama"`	Stochastic reverse-SDE updates. Uses `euler_maruyama_multiplier`.
`Solver.DPMPP_2M`	`"dpmpp_2m"`	Flow-matching DPM++ 2M updates with finite start handling.
`Solver.ABM2`	`"abm2"`	Variable-step Adams-Bashforth-Moulton updates with corrected-state reevaluation.

ABM2 is the default and performs additional denoiser evaluations for its corrected states.

Schedules

Enum	Value	Description
`Schedule.LINEAR`	`"linear"`	Uniform spacing from noisy to clean.
`Schedule.BETA`	`"beta"`	Beta(0.6, 0.6) quantile spacing with more schedule density near the endpoints.

Schedules and solver state remain float32.

CFG

from canter import (
    CanterInferenceConfig,
    CanterPipelineConfig,
    CfgGuidance,
    PdgGuidance,
    PdgMode,
)

cfg = CfgGuidance(
    enabled=True,
    scale=3.0,
    start_step=0,
    stop_step=29,
)

output = pipe(
    "A studio portrait with soft natural light",
    negative_prompts="oversaturated, harsh contrast",
    config=CanterPipelineConfig(
        inference=CanterInferenceConfig(
            cfg=cfg,
            pdg=PdgGuidance(mode=PdgMode.COMBINED_CFG_PDG),
        ),
    ),
)

`CfgGuidance`

Field	Default	Description
`enabled`	`False`	Enable classifier-free guidance.
`scale`	`None`	Non-negative guidance scale. Required when CFG is enabled or when the selected PDG mode uses CFG.
`start_step`	`0`	First active solver update, inclusive.
`stop_step`	`None`	Last active solver update, inclusive. `None` selects the final update.

Step indices run from 0 through steps - 1.

An explicit negative prompt replaces the learned unconditional token on CFG branches. Negative prompts require enabled CFG or a CFG-dependent PDG mode. For prompt batches, supply one negative prompt per positive prompt.

PDG

from canter import PdgCurve, PdgGuidance, PdgMode

pdg = PdgGuidance(
    enabled=True,
    mode=PdgMode.FULL,
    curve=PdgCurve.POWER,
    noisy_scale=2.0,
    clean_scale=2.5,
    power=3.0,
    start_step=0,
    stop_step=None,
)

`PdgGuidance`

Field	Default	Description
`enabled`	`True`	Enable path-drop guidance.
`mode`	`PdgMode.FULL`	Path and CFG interaction policy.
`curve`	`PdgCurve.CONSTANT`	Scale interpolation from noisy to clean.
`noisy_scale`	`2.5`	PDG scale at the first noisy schedule point.
`clean_scale`	`2.5`	PDG scale at the final clean schedule point.
`power`	`3.0`	Positive exponent used by the power curve.
`start_step`	`0`	First active solver update, inclusive.
`stop_step`	`None`	Last active solver update, inclusive. `None` selects the final update.

Constant PDG requires equal noisy_scale and clean_scale. Disabled PDG requires enabled=False and mode=PdgMode.NONE.

Increasing the PDG scale sharpens the generated distribution. Moderate values improve image structure and fine detail at the cost of reduced variety. Pushing the scale too far can introduce structural defects, excessive contrast, and oversaturation.

PDG curves

Enum	Value	Scale behavior
`PdgCurve.CONSTANT`	`"constant"`	Uses one scale throughout inference.
`PdgCurve.LINEAR`	`"linear"`	Interpolates linearly from `noisy_scale` to `clean_scale`.
`PdgCurve.POWER`	`"power"`	Interpolates with `position ** power`.

PDG modes

Enum	Value	Behavior
`PdgMode.NONE`	`"none"`	No PDG branch. Required when PDG is disabled.
`PdgMode.FULL`	`"full"`	Guides from the middle-skipped path toward the full main path.
`PdgMode.THREE_QUARTER`	`"three_quarter"`	Guides from the 75 percent SPRINT path toward the full main path.
`PdgMode.ALTERNATE_PDG_FIRST`	`"alternate_pdg_first"`	Alternates PDG on even updates and CFG on odd updates within the PDG window.
`PdgMode.ALTERNATE_CFG_FIRST`	`"alternate_cfg_first"`	Alternates CFG on even updates and PDG on odd updates within the PDG window.
`PdgMode.COMBINED_CFG_PDG`	`"combined_cfg_pdg"`	Evaluates CFG and PDG together and averages their guidance deltas.
`PdgMode.PDG_WITH_ALTERNATING_CFG`	`"pdg_with_alternating_cfg"`	Applies PDG on every active update and adds CFG on odd updates.
`PdgMode.CFG_TO_PDG`	`"cfg_to_pdg"`	Uses CFG before the PDG window, then uses full-path PDG inside the window.

The five compound modes require cfg.scale. start_step=0 with PdgMode.CFG_TO_PDG begins directly with PDG.

Self-attention gain

self_attention_gain changes image self-attention only on the full main path. The model multiplies main-path attention queries by exp(self_attention_gain). Negative values reduce the attention-logit scale and therefore increase the effective softmax temperature. This softens the main prediction and helps moderate PDG oversaturation. Text self-attention, cross-attention, and weak guidance paths retain their trained scales.

The default is -0.03. A value of 0.0 uses the trained main-path self-attention scale without adjustment. Smaller images generally benefit from more negative values. As image size increases, the gain should move closer to zero.

Custom configuration example

from canter import (
    CanterInferenceConfig,
    CanterOutputType,
    CanterPipelineConfig,
    CfgGuidance,
    PdgCurve,
    PdgGuidance,
    PdgMode,
    Schedule,
    Solver,
)

inference = CanterInferenceConfig(
    height=1024,
    width=1024,
    steps=40,
    solver=Solver.DPMPP_2M,
    schedule=Schedule.LINEAR,
    log_snr_shift=0.5,
    cfg=CfgGuidance(
        enabled=True,
        scale=3.0,
        start_step=0,
        stop_step=11,
    ),
    pdg=PdgGuidance(
        enabled=True,
        mode=PdgMode.CFG_TO_PDG,
        curve=PdgCurve.POWER,
        noisy_scale=2.0,
        clean_scale=2.5,
        power=3.0,
        start_step=12,
        stop_step=39,
    ),
    self_attention_gain=-0.03,
    euler_maruyama_multiplier=1.0,
    seed=123,
    generator=None,
)

config = CanterPipelineConfig(
    inference=inference,
    output_type=CanterOutputType.PIL,
)
image = pipe("A glass greenhouse during heavy rain", config=config).image

Prompt batches

A sequence of prompts generates one image per prompt:

output = pipe(
    [
        "A windswept beach under dark clouds",
        "A sunlit kitchen with white tiled walls",
    ]
)

first, second = output.images

Batch size one is the primary inference path. The jagged backend packs active prompt tokens without padding them through text refinement and cross-attention. Prompts longer than 512 tokens are truncated with a warning.

Latent generation

Use CanterInferenceEngine to generate latents without loading or calling the VAE:

from canter import CanterComponents, CanterInferenceEngine

components = CanterComponents.from_pretrained("data-archetype/canter")
engine = CanterInferenceEngine(components)
output = engine.generate("A mountain road in winter")

latents = output.latents
schedule = output.schedule

CanterInferenceEngine.generate accepts:

Parameter	Default	Description
`prompts`	required	One string or a sequence of strings.
`negative_prompts`	`None`	Optional negative prompt string or sequence for CFG branches. The count must match `prompts`.
`config`	`CanterInferenceConfig()`	Latent inference settings.
`initial_noise`	`None`	Optional float32 noise with shape `[batch, 128, height / 16, width / 16]`.
`progress`	`None`	Optional callback receiving `(completed_updates, total_updates)`.

An empty or whitespace-only prompt uses the learned unconditional token without running the text encoder. Every prompt in a batch must be either blank or nonblank.

The returned latents use float32 and channels-last memory format.

Pipeline metadata

pipe.metadata records the resolved release, weight dtype, source digests, text-encoder revision, VAE repository, and resolved immutable VAE revision. Applications that require reproducibility should store this metadata with their outputs and pin a release tag.