jiffy / docs /release.md
kidzik's picture
Release Jiffy v0.1.0 experimental inference adapter
90d3e4b verified
|
Raw History Blame Contribute Delete
1.95 kB

Release Status

Version 0.1.0 is an experimental prerelease, not a calibrated or production equivalent of TypeSafe Jev. Code is Apache-2.0; model weights are downloaded separately under Google's own terms.

Verified

  • Standalone DiffusionGemma package, CLI, and authenticated local HTTP adapter.
  • Pinned model revision and runtime dependencies.
  • CPU contract, validation, cache-isolation, and HTTP tests.
  • Official TypeSafe Python SDK interoperability tests.
  • H100 checks covering images, 32K inputs, and 255-option inference.
  • Public JevBench evaluation: 195/231 correct, all responses valid.
  • Fresh standalone dependency installation; pip check passes.

The compatibility report documents the workloads and numerical checks. Characterization tests do not imply model equivalence: a mixed Score fixture shifted by 0.0428 versus the sequential reference, above the stricter 0.03 equivalence threshold. This is an acknowledged prerelease limitation.

Before a Production Release

  • Broader held-out document/image accuracy and calibration evaluation.
  • Resolve or explicitly accept shared-prefix numerical drift against those data.
  • Prompt-injection, missing evidence, unreadable image, and abstention evaluation.
  • Maximum-size and repeated mixed-length memory/load testing.
  • Deployment-specific TLS, access control, observability, and request scheduling.
  • Validation beyond H100; quantized deployment is not supported.

The backend uses private pinned Transformers interfaces. Runtime upgrades need numerical regression testing. Local SDK callers must serialize GPU access; the HTTP server enforces one active GPU request and rejects overlap.

Optional GPU tests require the checkpoint and a suitable CUDA environment. CPU CI does not download weights or establish GPU accuracy or performance. The workflow remains an inactive template in ci/github-actions.yml until a credential with workflow-write permission activates it.