FilBona's picture
EmbeddingGemma 2 (text) in Core ML, 8-bit, fixed 512 tokens, for the Apple Neural Engine
cecc10e verified
Raw History Blame Contribute Delete
409 Bytes
# Pesi a 8 bit (lineari simmetrici per canale): 518 MB → 262 MB, coseno con l'originale ≥ 0,998.
import coremltools as ct
import coremltools.optimize.coreml as cto
m = ct.models.MLModel("EG2fixed512.mlpackage")
q8 = cto.linear_quantize_weights(m, cto.OptimizationConfig(
global_config=cto.OpLinearQuantizerConfig(mode="linear_symmetric", granularity="per_channel")))
q8.save("EG2-512-int8.mlpackage")