Gemma 4 Text Decoder — Karume

This is an unofficial conversion of google/gemma-4-E2B-it, google/gemma-4-E4B-it and google/gemma-4-E2B-it-assistant for Karume. It is not affiliated with or endorsed by Google.

What is this

A chat distribution: the text decoder of google/gemma-4-E2B-it / google/gemma-4-E4B-it, converted into the WebGPU inference runtime Karume's container format (a .krm part sequence whose first part carries the graph and model descriptors). Runs as-is in the browser and in Deno — string in, string out.

Tokenizer and chat format ship with the weights. chat() renders the turns, encodes them, samples, and decodes incrementally, so callers hand over messages and read back text fragments as they are decided.
Per-layer embeddings are container assets, not graph weights. They are gathered on the host and handed to the graph as an input, which keeps them out of the GPU's resident set; the loader reads only the vocabulary ranges a conversation actually touches.
Context: the default capacity is 4096 tokens per conversation, and the loader accepts any capacity between the chunk length in use and 131072 (the model's declared position limit); rotary cos/sin are generated on the host per chunk from the declared rope parameters, so no position table ships. Prefill runs in chunks of 768 rows by default, and the chunk length itself can be raised up to 768 — the traced upper bound of the chunk symbol, which the graph does not carry.
Not multimodal here. The published checkpoint also carries vision and audio towers; this distribution contains neither, and cannot take images or audio.
Not readable by transformers (it's a different container with an embedded graph); the reader is a pipeline that implements gemma4/1.
Exporter used for the conversion: karume/0.16.0. The distribution manifest is karume.json (karume/5).

Base weights and attribution

Converted from the upstream checkpoint, its weights shipped unchanged (bf16) — the original checkpoint files are not distributed here.

e2b: google/gemma-4-E2B-it, licensed Apache 2.0 (license / full text; a verbatim copy is in LICENSE.md).
e4b: google/gemma-4-E4B-it, licensed Apache 2.0 (license / full text; a verbatim copy is in LICENSE.md).
assistant: google/gemma-4-E2B-it-assistant, the multi-token-prediction drafter head, licensed Apache 2.0 under the same terms. It is redistributed here with its clustered sparse output head replaced by a dense projection over the full vocabulary, and it reads the key/value states and embedding table of the main model rather than carrying its own. Its own weights are quantized to int8 throughout, linear layers included.
Changes made here (also listed in NOTICE.md, per Apache 2.0 §4(b)): the text decoder was extracted and re-expressed in the Karume container format; the weights keep the checkpoint's bf16 values unchanged; the per-layer embeddings were moved out of the graph into container assets that the host gathers; the exit was narrowed to the selected rows' logits and hidden states; rotary cos/sin arrive as host-generated inputs. No retraining and no fine-tuning.
What the model can and cannot do — capabilities, limitations, intended use, evaluations and safety guidance — is documented on the upstream model card, and this repository does not restate it: read google/gemma-4-E2B-it before using it.

Models

Model	Pipeline	Quants	Default quant
`e2b` (default)	`gemma4/1`	`bf16`	`bf16`
`e4b`	`gemma4/1`	`bf16`	`bf16`

model selects one of these; omitted, it is e2b. quant defaults to that model's own default quant.

Usage

import { Pipeline } from "jsr:@karume/models/gemma";

await using pipeline = await Pipeline.fromPretrained({
  repo: "hdae/karume-gemma4",
  // Pin a commit for reproducible builds — without it you track `main`, and a future
  // repo update (renamed files, new manifest format) may break your app.
  // Copy the full hash from this repo's "Files and versions" tab:
  // revision: "<full commit sha>",
}, {
  // model: "e2b", // default — available: e2b / e4b
  // quant: "bf16", // default — available: bf16
});

const stream = pipeline.chat([
  { role: "user", content: "What is the capital of France?" },
], {
  maxNewTokens: 128,
  // sampler: { temperature: 0 }, // greedy — overrides the recommended default below
});

// Fragments arrive as the decoder settles them (multi-byte characters are held back
// until they are complete).
for await (const text of stream) Deno.stdout.write(new TextEncoder().encode(text));
console.log(await stream.done); // { reason: "eos" | "max-tokens" | "capacity" | … }

Messages are plain system / developer / user / assistant turns with text content; content parts such as images or audio are rejected rather than silently dropped. For function calling and thinking, use new ChatSession(pipeline, { tools, enableThinking }): a turn can end on a tool call, and sendToolResults continues it. With no sampler in the request, generation uses this repository's recommended default: temperature 1.0, top-k 64, top-p 0.95. Weights are fetched once and cached (verified against karume.json's size / sha256).

Model: e2b

Quants

Quant	What it is	Download	Weights	Compute
`bf16` (default)	Original weights (bf16) — The checkpoint's own bf16 weights, the per-layer embeddings included, with f32 compute; GEMV and attention use the reference summation order. The drafter head is int8 throughout.	8.71 GiB (4.38 GiB of assets, read on the host)	`model` = `bf16` / `drafter` = `i8`	—

If no quant is given, it runs as bf16 (this model's recommended default). Per-file size and sha256 live in karume.json — verify against that at the fetch layer. Dtype labels use the runtime's storage dtype vocabulary (f16 / bf16 / i8 / i4 / i2), not the fp16 spelling common elsewhere in the ecosystem. Weights ship as Karume container files (.krm), split into numbered parts; a part is fetched and verified on its own.

Generation

Derived from the exported graph and the checkpoint's own generation_config.json, and checked against each other when this repository was assembled.

context: 4096 tokens per conversation (prompt + generated) by default; any capacity from the chunk length in use up to 131072 can be chosen at load time
prefill chunk: 768 rows per step by default, up to 768 (the traced upper bound of the chunk symbol)
recommended sampler: temperature 1.0, top-k 64, top-p 0.95 — used when a request omits sampler; pass { temperature: 0 } for greedy decoding
stop tokens come from the tokenizer asset in this repository, not from the caller
device limits: maxBufferSize ≥ 805,306,368 B / maxStorageBufferBindingSize ≥ 805,306,368 B (the largest single tensor must bind)

Model: e4b

Quants

Quant	What it is	Download	Weights	Compute
`bf16` (default)	Original weights (bf16) — The checkpoint's own bf16 weights, the per-layer embeddings included, with f32 compute; GEMV and attention use the reference summation order.	13.9 GiB (5.26 GiB of assets, read on the host)	`model` = `bf16`	—

Generation

Derived from the exported graph and the checkpoint's own generation_config.json, and checked against each other when this repository was assembled.

context: 4096 tokens per conversation (prompt + generated) by default; any capacity from the chunk length in use up to 131072 can be chosen at load time
prefill chunk: 768 rows per step by default, up to 768 (the traced upper bound of the chunk symbol)
recommended sampler: temperature 1.0, top-k 64, top-p 0.95 — used when a request omits sampler; pass { temperature: 0 } for greedy decoding
stop tokens come from the tokenizer asset in this repository, not from the caller
device limits: maxBufferSize ≥ 1,342,177,280 B / maxStorageBufferBindingSize ≥ 1,342,177,280 B (the largest single tensor must bind)

Downloads last month: -; Downloads are not tracked for this model. How to track

Model tree for hdae/karume-gemma4

Base model

google/gemma-4-E2B

Finetuned

google/gemma-4-E2B-it

Quantized

(452)

this model