ankahi / backend_developmentndtesting.md
Add files using upload-large-folder tool
dc7e099 verified 6 months ago
34.2 kB
# Ankahi (अनकही) — Backend, Core System & Testing Specification
## Model Deployment · Pipeline Conversion · Robustness Testing · Benchmarking

---

> **AGENT INSTRUCTION:** This document contains NO finished code. It contains precise, actionable instructions, resource pointers, search queries, and architectural decisions that you must execute step by step. For every section marked **[SEARCH REQUIRED]**, you must query the listed resources and documentation before proceeding — do not rely on memory or assumptions, as MediaPipe, LiteRT, and Unsloth APIs change frequently. For every benchmark, you must produce actual measured numbers — no fabricated figures. This document also defines the exact format for the results report that must accompany the Kaggle submission.

---

## PART 1: THE CRITICAL PATH — .litertlm CONVERSION

This is the single biggest blocker to a working demo. Without the on-device binary, the Flutter app has nothing to run. Do this before anything else.

---

### 1.1 Overview: What the conversion pipeline does

```
Unsloth fine-tuned Gemma 4 E4B
  (stage1 + stage2 adapter, merged)
         ↓
[bitsandbytes INT8 quantization]
  (merge_and_quantize.py)
         ↓
Hugging Face safetensors checkpoint
  (float32 or int8, standard format)
         ↓
[MediaPipe Model Maker conversion]
  (convert_to_litertlm.py)
         ↓
.task or .litertlm flatbuffer file
  (~2.3 GB for E4B INT8)
         ↓
Sideloaded to Android via adb push
  (or bundled in APK assets for small adapters)
         ↓
MediaPipe GenAI / LiteRT-LM runtime
  running on Android CPU/GPU/NPU
```

---

### 1.2 Environment Setup

**[SEARCH REQUIRED]** Before running anything, read these resources:
- Search: `"mediapipe model maker" "LLM inference" "gemma" site:ai.google.dev`
- URL to read: `https://ai.google.dev/edge/mediapipe/solutions/genai/llm_inference`
- URL to read: `https://ai.google.dev/edge/mediapipe/solutions/genai/llm_inference/android`
- Search: `"mediapipe-model-maker" pip install python version compatibility 2025 2026`
- Search: `"LiteRT" "LiteRT-LM" gemma conversion site:github.com/google`

**Why this matters:** `mediapipe-model-maker` has strict Python and CUDA version dependencies that change between releases. You must verify the current requirements before installing. Installing the wrong version wastes hours.

**Recommended environment (verify against current docs):**
```
Python:      3.10 or 3.11 (verify with mediapipe docs — NOT 3.12)
CUDA:        11.8 or 12.1 (check current compatibility matrix)
PyTorch:     The version compatible with your CUDA
mediapipe:   Latest stable (search: "mediapipe pypi latest version")
mediapipe-model-maker: Must match mediapipe version exactly
Recommended: Use a fresh conda environment
conda create -n ankahi_convert python=3.10
conda activate ankahi_convert
# Then follow exact installation from current mediapipe docs
```

**[SEARCH REQUIRED]** For the H100 server specifically:
- Search: `mediapipe-model-maker H100 installation cuSPARSELt compatibility`
- Cross-reference with the environment cascade issue documented in the Ankahi handover — the H100 server had pyannote/CUDA/cuSPARSELt conflicts. The conversion environment should be kept SEPARATE from the training environment.
- Run conversion in its own conda environment. Do not mix with the Unsloth training environment.

---

### 1.3 Step-by-Step Conversion Pipeline

**[SEARCH REQUIRED]** For each step below, search for the current API before running:

#### Step A: Verify the merged checkpoint

The `merge_and_quantize.py` script should have already produced a merged safetensors checkpoint. Verify:

```python
# Check the output directory — you need to confirm these files exist:
# - config.json
# - tokenizer.json / tokenizer_config.json / tokenizer.model
# - model-00001-of-XXXXX.safetensors (and remaining shards)
# - generation_config.json
# Verification check:
from transformers import AutoModelForCausalLM, AutoTokenizer
# Load the checkpoint to verify it's valid before conversion
# If this fails, the merge step needs to be rerun
```

**[SEARCH REQUIRED]** Search: `"Gemma 4" "safetensors" "merge lora" "unsloth" verification checklist`

#### Step B: INT8 Quantization verification

The handover doc specifies bitsandbytes INT8. Verify the quantization is correctly applied:

```python
# Check quantization was applied:
# model.config should show quantization_config with load_in_8bit=True
# Run a quick forward pass to confirm no NaN outputs:
# inputs = tokenizer("WANT WATER COLD", return_tensors="pt")
# outputs = model.generate(**inputs, max_new_tokens=20)
# print(tokenizer.decode(outputs[0]))
# Expected: Hindi sentence, no garbage tokens, no infinite repetition
```

**Known issue from handover:** INT8 quantization causes repetition loops on short inputs. Confirm `repetition_penalty=1.2` is in `generation_config.json`. If not, add it before conversion — it must be baked into the model config for on-device inference.

#### Step C: Convert to LiteRT-LM format

**[SEARCH REQUIRED]** The exact API for this changes between MediaPipe releases. Before running `convert_to_litertlm.py`, search:
- `"mediapipe model maker" "llm" "gemma" "convert" "python" site:ai.google.dev 2025 OR 2026`
- `"LiteRT" "flatbuffer" ".task" gemma 4 conversion tutorial`
- Check: `https://github.com/google-ai-edge/mediapipe-samples` for the most recent LLM inference examples

The conversion call will look approximately like this (verify against current docs — do not run this verbatim):

```python
# APPROXIMATE — verify exact API from current mediapipe docs before running
import mediapipe as mp
# OR: from mediapipe.tasks.python.genai import converter as genai_converter
# Key parameters to understand and configure:
# - model_path: path to your merged safetensors checkpoint directory
# - output_path: where to write the .litertlm or .task file  
# - output_type: "TFLITE" or "LITERTLM" — check current options
# - quantization_type: "w8a8" (weights 8-bit, activations 8-bit) or "w4a8" — w8a8 matches your bitsandbytes INT8
# - backend: "CPU" or "GPU" — verify Android GPU support for your target devices
# - lora_rank: 8 (must match Stage 2 adapter rank)
```

**[SEARCH REQUIRED]** Critical constraint documented in handover: LoRA rank must be 4 or 8 for MediaPipe GenAI compatibility. This is already set. But verify the conversion tool's current LoRA rank support:
- Search: `"mediapipe" "lora" "rank 8" android "llm inference" compatibility 2025 OR 2026`

#### Step D: Adapter conversion (separate files)

The base model and persona adapters are separate files. The base model is the large .litertlm file. Each 30MB adapter must also be converted to a format MediaPipe can load at runtime.

**[SEARCH REQUIRED]**:
- Search: `mediapipe genai "lora adapter" android "load at runtime" ".task" OR ".bin" format`
- Search: `"mediapipe-model-maker" "lora" "convert adapter" python example`
- Search: `LiteRT-LM "hot-swappable" adapter loading android kotlin`

Expected output structure:
```
ankahi_bundle/
├── model/
│   ├── ankahi_base_e4b_int8.litertlm   (~2.3 GB)
│   └── symlinks → point to above (use cp -L when moving across filesystems!)
├── adapters/
│   ├── arjun_v1.ankahi                 (~30 MB, converted LoRA)
│   ├── ananya_v1.ankahi
│   ├── priya_v1.ankahi
│   ├── rohan_v1.ankahi
│   └── zara_v1.ankahi
└── voices/
    └── (AI4Bharat TTS voice clone files, one per persona)
```

#### Step E: Quick validation on-device

After conversion, validate before spending time on the Flutter integration:

```bash
# Use adb to push and test directly
adb push ankahi_base_e4b_int8.litertlm /sdcard/ankahi/model/
adb push arjun_v1.ankahi /sdcard/ankahi/adapters/
# [SEARCH REQUIRED] Search for:
# "mediapipe" "llm inference" "android" "command line test" OR "standalone test"
# There may be a MediaPipe demo APK you can sideload for quick validation
# before building the full Flutter app
```

**[SEARCH REQUIRED]**: 
- Search: `"mediapipe" "llm_inference" android demo app github`
- Search: `google-ai-edge mediapipe-samples llm inference android kotlin example`
- URL: `https://github.com/google-ai-edge/mediapipe-samples/tree/main/examples/llm_inference/android`

---

### 1.4 AI4Bharat svara-TTS Integration

**[SEARCH REQUIRED]** This is listed as "Next Step #3" in the handover but should be treated as highest priority after model conversion.

Resources to read:
- Search: `"AI4Bharat" "Indic TTS" OR "IndicTTS" OR "svara" python pip install 2025 2026`
- URL: `https://github.com/AI4Bharat/Indic-TTS` — check current state, branches, releases
- Search: `AI4Bharat TTS "voice cloning" python inference minimal example`
- Search: `AI4Bharat TTS android deployment OR "mobile" OR "on-device"`

**Key questions to answer from the search:**
1. Does AI4Bharat's current TTS support on-device Android inference, or does it require a server?
2. If server-required: Is there a small model variant suitable for on-device use?
3. What format does the voice cloning model accept? (wav file? sampling rate? duration?)
4. What languages are reliably supported right now?

**Fallback plan if AI4Bharat TTS cannot run on-device:**
- **Option A:** Run TTS on a lightweight server (FastAPI) that the device calls over local WiFi only (not cloud). This still works for the demo.
- **Option B:** Use a pre-synthesized voice library with the cloned voice (record 50–100 phoneme combinations, stitch on-device). Cruder, but zero-latency.
- **Option C:** Use Android's built-in TTS engine (`TextToSpeech`) as a fallback only — it won't sound like the parent, but it ensures the app works.

**[SEARCH REQUIRED]** For Option A (if needed):
- Search: `AI4Bharat TTS "FastAPI" OR "flask" serve REST API example`
- Search: `AI4Bharat TTS model size "hindi" inference latency`

---

## PART 2: TESTING STRATEGY & ROBUSTNESS

---

### 2.1 Testing Philosophy

The Ankahi system has three distinct testing domains:

```
Domain 1: ML Pipeline Tests (pytest)
  → Model output quality, adapter isolation, safety layer
  
Domain 2: Inference Pipeline Tests (device benchmarking)
  → Latency, memory, battery, hardware compatibility
  
Domain 3: Application Tests (Flutter)
  → Widget tests, integration tests, accessibility tests
```

The handover reports 36/36 unit tests passing. This section expands that into a full test strategy for Kaggle submission evidence.

---

### 2.2 ML Pipeline Tests (src/ankahi/eval/)

The existing test suite covers: BLEU-4 4-gram potential issue (fixed), BLEU-4 smoothing, chrF++ calculation, IndicSBERT heatmap generation. Expand it as follows:

#### Test Suite A: Output Quality Tests
```python
# File: tests/test_output_quality.py
# Test 1: Language accuracy per language
# For each of 5 supported languages, run 20 test inputs from test.jsonl
# Assert chrF++ score > threshold (set threshold based on Stage 1 training result)
# Test 2: Code-switching fidelity  
# Input: Hinglish pictogram sequences (e.g., "WANT + MOVIE + NIGHT + WITH + FRIENDS")
# Expected output: natural Hinglish ("Yaar, raat ko movie dekhni hai")
# Assert: output contains both Hindi and English words (simple heuristic check)
# Test 3: Persona isolation (the 5×5 heatmap test)
# For each of 5 adapters: run 10 persona-specific inputs
# Compute IndicSBERT similarity between:
#   - Arjun adapter output for Arjun-specific input
#   - Ananya adapter output for same input
# Assert: diagonal scores > 0.85, off-diagonal < 0.65
# This is the "no leakage" test — adapters must not sound like each other
# Test 4: Repetition loop detection
# Run 50 short inputs (1–3 pictograms)
# Assert: no output repeats any phrase more than 2 times
# Assert: all outputs are < 25 tokens (AAC sentences should be short)
# Test 5: chrF++ vs BLEU-4 comparison
# For morphologically rich languages (Hindi, Tamil, Bengali):
# Assert: chrF++ score > BLEU-4 score (validates the metric choice)
# Test 6: Safety layer completeness
# 50 self-harm trigger inputs (from the safety test set)
# Assert: 100% refusal rate (0 completions on forbidden content)
# 20 legitimate anger/pain inputs
# Assert: 100% completion rate (safety layer must NOT block these)
```

#### Test Suite B: Adapter Loading Tests
```python
# File: tests/test_adapter_switching.py
# Test 1: Cold load time
# Measure time to load each adapter from disk to GPU memory
# Assert: load time < 5 seconds per adapter on H100
# Test 2: Hot swap correctness  
# Load Arjun adapter → generate sentence
# Swap to Ananya adapter → generate same sentence
# Assert: sentences are different (adapter is actually being swapped)
# Test 3: Memory after swap
# After 5 consecutive adapter swaps:
# Assert: GPU memory usage returns to baseline (gc.collect + empty_cache working)
# Test 4: Rank constraint validation
# Programmatically verify each Stage 2 adapter has rank=8
# Assert: all adapters fail to load if rank > 8 (simulated)
```

#### Test Suite C: Audio Tower Tests
```python
# File: tests/test_audio_tower.py
# Test 1: Dysarthric speech disambiguation
# Scenario: pictogram input says "DOG", audio signal suggests "DRINK" (fricative sounds)
# Assert: output sentence is drink-related, not dog-related
# Use pre-recorded test audio clips from the TORGO corpus augmentation
# Test 2: No audio input baseline
# Assert: system works correctly with audio input disabled
# The audio tower is an enhancement, not a requirement
# Test 3: Ambient noise robustness
# Inject synthetic noise (café noise, TV audio) into audio input
# Assert: output quality does not degrade by more than 10% chrF++ vs clean audio
```

---

### 2.3 Device Benchmarking Tests

This is the most important section for the Kaggle submission. You need real numbers.

#### Hardware Setup
**[SEARCH REQUIRED]** Before benchmarking:
- Search: `"MediaPipe" "LLM inference" Android benchmark "prefill speed" "decode speed" measurement`
- Search: `LiteRT android profiling tool GPU CPU NPU benchmark script`
- URL: `https://ai.google.dev/edge/mediapipe/solutions/genai/llm_inference#performance_benchmarks`

**Target devices for benchmarking (use what you have, document what you used):**
```
Tier 1 (target market): Any device with Snapdragon 680 or similar (e.g., Realme Narzo 50, ~₹12,000)
Tier 2 (mid-range):     Any device with Snapdragon 778G or similar (e.g., Poco X5 Pro, ~₹25,000)
Tier 3 (development):   Any available Android tablet/phone you have access to
Note: Document the exact device model, Android version, and RAM in results.
```

**If no physical device is available:**
- Use Android emulator with GPU acceleration (note this in results as "emulated, not representative")
- Or use MediaPipe's published benchmarks and cite them as reference

#### Benchmark Test Suite D: Latency
```python
# File: benchmarks/latency_benchmark.py
# Metric 1: Time to First Token (TTFT)
# = Time from input submission to first output token
# Measure across 3 input lengths: 1 tile (short), 3 tiles (medium), 6 tiles (long)
# Run 20 trials each, report mean ± std
# Metric 2: Total generation time
# = Time from input to complete sentence output
# Target: < 200ms (from handover spec)
# If > 200ms: document and explain (model size, quantization level)
# Metric 3: TTS synthesis time
# = Time from text ready to audio playing
# Measured separately from inference
# Target: < 300ms
# Metric 4: End-to-end latency
# = Tap "Speak" → audio begins playing
# Target: < 500ms total (inference + TTS)
```

#### Benchmark Test Suite E: Memory & Battery
```python
# File: benchmarks/resource_benchmark.py
# Metric 1: RAM usage during inference
# Measure with adb shell dumpsys meminfo package.name
# Record: base app RAM, + model loaded RAM, + during inference peak RAM
# Assert: total does not exceed 4GB (leaves headroom for Android OS on 6GB devices)
# Metric 2: Battery consumption
# Run continuous inference for 30 minutes (simulating active communication session)
# Measure battery % delta
# Target: < 5% per hour
# Tool: adb shell dumpsys battery OR Android Battery Historian
# Metric 3: Thermal performance
# Monitor device temperature during 30-minute sustained use
# Note: thermal throttling will increase latency — document the curve
# adb shell cat /sys/class/thermal/thermal_zone*/temp
# Metric 4: Cold start time  
# Time from app icon tap to communication screen ready (model loaded)
# Target: < 5 seconds (model should stay resident after first load)
```

#### Benchmark Test Suite F: Hardware Compatibility
```python
# File: benchmarks/compatibility_check.py
# Check which inference backends are available on the device:
# - CPU (always available)
# - GPU (via MediaPipe GPU delegate)
# - NPU/DSP (via NNAPI delegate)
# [SEARCH REQUIRED]: 
# Search: mediapipe genai "gpu delegate" OR "nnapi delegate" android compatibility
# Search: LiteRT android "backend fallback" cpu gpu npu strategy
# Test each backend:
# 1. Run 10 inference trials on each available backend
# 2. Record latency and memory for each
# 3. Report which backend is recommended for the target device class
```

---

### 2.4 Flutter Application Tests

#### Widget Tests
```dart
// test/widget_test.dart
// Test 1: PictogramTile renders correctly
// - Check tile displays Hindi label
// - Check tile displays English label  
// - Check tile is accessible (Semantics widget present with correct label)
// - Check tile responds to tap
// Test 2: PhraseBar state management
// - Start empty → assert SPEAK button is disabled
// - Add 1 tile → assert SPEAK button is enabled
// - Clear → assert returns to empty state
// Test 3: Result card appearance
// - Mock inference result
// - Assert Hindi text is displayed
// - Assert "Play again" button is present
// - Assert "Save to history" button is present
// Test 4: Category tab switching  
// - Tap second category tab
// - Assert grid content changes
// - Assert animation completes without error
```

#### Integration Tests
```dart
// integration_test/communication_flow_test.dart
// [SEARCH REQUIRED]: Search: flutter integration_test mediapipe method channel mock
// Test 1: Full communication flow (with mocked inference)
// Mock InferenceService to return "मुझे पानी चाहिए" for any input
// 1. Tap three pictogram tiles
// 2. Assert tiles appear in phrase bar
// 3. Tap SPEAK button
// 4. Assert result card appears with correct text
// 5. Assert TTS is called (mock TTS service)
// Test 2: Persona switching
// 1. Start with Arjun persona active
// 2. Navigate to Personas screen
// 3. Tap "Set Active" on Ananya
// 4. Assert adapter switch is triggered
// 5. Navigate back to communication screen
// 6. Assert persona indicator shows Ananya
// Test 3: Offline resilience
// Disable network (can be done in integration test setup)
// Assert: app works identically — no network errors, no degraded functionality
```

#### Accessibility Tests
```dart
// test/accessibility_test.dart
// Test 1: All tiles meet contrast requirements
// Programmatically check color contrast ratio for each category color + text
// Assert: all ratios >= 4.5:1
// Test 2: Touch target sizes
// Assert: all interactive widgets have minimum 72×72 logical pixels
// Test 3: TalkBack labels present
// Assert: every PictogramTile has non-empty Semantics.label
// Assert: SPEAK button has Semantics with state description
// Test 4: Animation disable respect
// Simulate AccessibilityFeatures.disableAnimations = true
// Assert: no AnimationController is started when flag is set
```

---

## PART 3: RESULTS REPORT FORMAT

---

### 3.1 The Results Document Structure

This section defines the exact format for the benchmarking results document that will accompany the Kaggle submission. This document should be a single, well-formatted Markdown file named `RESULTS.md` in the repository root, AND should be mirrored as a well-designed section of the Kaggle notebook.

**The document must contain:**

---

### Template: RESULTS.md

```markdown
# Ankahi — System Evaluation Results
**Version:** 1.0.0  
**Date:** [Run date]  
**Hardware (training):** NVIDIA H100 (40GB VRAM)  
**Hardware (inference benchmark):** [Device model, Android version, RAM]  
---
## Section 1: Training Metrics
### Stage 1: Base AAC SFT
| Metric | Value |
|--------|-------|
| Final training loss | 0.6028 |
| Training pairs | 16,500 |
| Languages | Hindi, Punjabi, Tamil, Bengali, Hinglish |
| LoRA rank | 16 |
| Training time | [X hours on H100] |
| Tokens packed (MemoryPackCollator) | 2048-token blocks |
| Training speedup (vs. non-packed) | ~3× |
### Stage 2: Persona Adapters
| Adapter | Language | Rank | Size (MB) | Vocab coverage |
|---------|----------|------|-----------|----------------|
| Arjun | Hindi+Punjabi | 8 | [X] | [N] items |
| Ananya | Tamil+English | 8 | [X] | [N] items |
| Priya | Bengali+Hindi | 8 | [X] | [N] items |
| Rohan | Hindi+Marathi+English | 8 | [X] | [N] items |
| Zara | Urdu+Hindi+Telugu | 8 | [X] | [N] items |
---
## Section 2: Language Quality Evaluation
### BLEU-4 by Language (Stage 1 outputs)
[INSERT BAR CHART: horizontal bars, one per language, showing BLEU-4 score 0.0–1.0]
| Language | BLEU-4 | chrF++ |
|----------|--------|--------|
| Hindi | [X] | [X] |
| Punjabi | [X] | [X] |
| Tamil | [X] | [X] |
| Bengali | [X] | [X] |
| Hinglish | [X] | [X] |
| **Average** | **[X]** | **[X]** |
**Note:** chrF++ is the primary metric — it is better suited to morphologically rich Indian languages than BLEU-4. BLEU-4 is included for comparison.
### BLEU-4 vs. Zero-Shot GPT-4 Baseline
[INSERT GROUPED BAR CHART: Ankahi vs GPT-4 zero-shot on same test set, per language]
Context: GPT-4 zero-shot with a "translate this pictogram sequence to natural Hindi" prompt was used as a baseline. Ankahi's fine-tuned system should outperform it on code-switched and persona-specific outputs where GPT-4 has no grounding.
---
## Section 3: Adapter Isolation (No-Leakage Verification)
### IndicSBERT Cross-Adapter Similarity Heatmap
[INSERT 5×5 HEATMAP: rows = query adapter, cols = reference adapter]
[Color scale: low similarity = white/cream, high similarity = forest green]
[Diagonal = 1.0 (perfect self-similarity)]
```
          Arjun  Ananya  Priya  Rohan  Zara
Arjun   [ 1.00   X.XX    X.XX   X.XX  X.XX ]
Ananya  [ X.XX   1.00    X.XX   X.XX  X.XX ]
Priya   [ X.XX   X.XX    1.00   X.XX  X.XX ]
Rohan   [ X.XX   X.XX    X.XX   1.00  X.XX ]
Zara    [ X.XX   X.XX    X.XX   X.XX  1.00 ]
```
**Interpretation:** Off-diagonal values should be < 0.65. Values above this threshold would indicate persona "bleed" — one adapter sounding too much like another. A well-isolated adapter system protects each child's communication identity.

---

## Section 4: Safety Layer Verification

| Test Category | Inputs Tested | Refusals | Refusal Rate |
|---------------|---------------|----------|--------------|
| Self-harm triggers | 50 | [X] | [X]% |
| Inappropriate content | 50 | [X] | [X]% |
| Legitimate anger | 20 | 0 (correct) | 0% |
| Pain expression | 20 | 0 (correct) | 0% |
| Frustration | 20 | 0 (correct) | 0% |

**Target:** 100% refusal on harmful content, 0% refusal on legitimate emotional expression.

---

## Section 5: On-Device Inference Benchmarks

### Test Device
| Property | Value |
|----------|-------|
| Device | [Model name] |
| Processor | [SoC model] |
| RAM | [X] GB |
| Android version | [X] |
| Backend used | [CPU / GPU / NPU] |
| Backend selection rationale | [Why this backend was chosen] |

### Latency Breakdown
[INSERT BAR CHART: Stacked horizontal bars showing time components]

| Component | Mean (ms) | Std Dev (ms) | P95 (ms) |
|-----------|-----------|--------------|----------|
| Pictogram → tokens | [X] | [X] | [X] |
| LLM prefill (TTFT) | [X] | [X] | [X] |
| LLM decode (full sentence) | [X] | [X] | [X] |
| Adapter swap overhead | [X] | [X] | [X] |
| TTS synthesis | [X] | [X] | [X] |
| **End-to-end (tap → audio)** | **[X]** | **[X]** | **[X]** |

Target: end-to-end < 500ms.

[INSERT LINE CHART: Latency over 100 consecutive inferences — shows warmup curve and steady state]

### Latency by Input Length
[INSERT LINE CHART: x-axis = number of pictograms selected (1–8), y-axis = inference time ms]

Expected: roughly linear scaling. Deviations should be noted and explained.

---

## Section 6: Memory & Battery

### Memory Profile
[INSERT ANNOTATED SCREENSHOT or BAR CHART of memory usage at key states]

| State | RSS Memory (MB) |
|-------|----------------|
| App launched, no model | [X] |
| Base model loaded | [X] |
| Base model + Arjun adapter | [X] |
| During active inference | [X] (peak) |
| After inference, idle | [X] |

### Battery Consumption
| Session duration | Battery consumed | Implied drain/hour |
|-----------------|------------------|--------------------|
| 10 minutes active | [X]% | [X]% / hour |
| 30 minutes active | [X]% | [X]% / hour |

Target: < 5% per hour sustained.

### Thermal Behavior
[INSERT LINE CHART: Device temperature (°C) over 30-minute sustained use session]
Note: At what temperature does thermal throttling begin? What is the latency impact?

---

## Section 7: Unit Test Coverage

| Test Suite | Tests | Passed | Failed | Coverage |
|------------|-------|--------|--------|----------|
| Output quality | [X] | [X] | [X] | [X]% |
| Adapter switching | [X] | [X] | [X] | [X]% |
| Audio tower | [X] | [X] | [X] | [X]% |
| Safety layer | [X] | [X] | [X] | [X]% |
| Flutter widget tests | [X] | [X] | [X] | [X]% |
| Flutter integration tests | [X] | [X] | [X] | [X]% |
| **Total** | **[X]** | **[X]** | **[X]** | **[X]%** |

Previously confirmed: 36/36 passing in the original test suite. All new tests must achieve ≥ 95% pass rate.

---

## Section 8: Comparison vs. Commercial AAC Baseline

[INSERT RADAR CHART: 6 axes, comparing "Commercial AAC (best available)" vs "Ankahi"]

Axes:
1. Hindi / North Indian language support (0–5 scale)
2. South Indian language support (0–5)  
3. Code-switching / Hinglish support (0–5)
4. Offline capability (0–5)
5. Dysarthric speech tolerance (0–5)
6. Affordability under ₹20,000 (0–5)

Expected result: Ankahi fills or approaches the full radar on all 6 axes. Commercial AAC (designed for English-speaking markets) scores poorly on axes 1–4 and 6.

---

## Section 9: Open Research Artifacts

| Artifact | Location | Description |
|----------|----------|-------------|
| Training dataset | HuggingFace: ankahi/Ankahi-AAC-SFT | 16,500 code-switched AAC pairs |
| Base model weights | HuggingFace: ankahi/ankahi-gemma4-e4b-stage1 | Stage 1 fine-tuned Gemma 4 E4B |
| Persona adapters | HuggingFace: ankahi/ankahi-personas | 5 Rank-8 LoRA adapters |
| Eval benchmark | GitHub: ankahi/ankahi/benchmarks/ | chrF++, BLEU-4, IndicSBERT evaluation code |
| Source code | GitHub: [repo link] | Apache 2.0 |

---

## Appendix A: Chart Generation Code

All charts in this document were generated with the code in `benchmarks/generate_charts.py`. To reproduce:

```bash
pip install matplotlib seaborn pandas numpy
python benchmarks/generate_charts.py --results benchmarks/results.json --output charts/
```

Input format for results.json:
```json
{
  "bleu4": { "hindi": X.XX, "punjabi": X.XX, "tamil": X.XX, "bengali": X.XX, "hinglish": X.XX },
  "chrf": { "hindi": X.XX, ... },
  "adapter_similarity": [[1.0, X.XX, ...], [X.XX, 1.0, ...], ...],
  "latency_ms": { "prefill_mean": X, "prefill_std": X, "decode_mean": X, ... },
  "memory_mb": { "idle": X, "model_loaded": X, "peak_inference": X },
  "battery_pct_per_hour": X.X,
  "safety": { "harmful_refusal_rate": 1.0, "legitimate_expression_rate": 1.0 }
}
```
```
---
### 3.2 Chart Generation Scripts
**[SEARCH REQUIRED]** For the benchmark visualization:
- Search: `seaborn heatmap "IndicSBERT" similarity matrix annotation python`
- Search: `matplotlib radar chart python "fill" multiple series comparison`
Implement the following in `benchmarks/generate_charts.py`:
**Chart 1: BLEU-4 by Language (horizontal bar chart)**
```python
# Use matplotlib, horizontal bars
# Colors: forest green (#2D6A4F) for Ankahi, gray for baseline
# Sort bars by score (highest at top)
# Add data labels on bars
# Save as: charts/bleu4_by_language.png (300 DPI)
```
**Chart 2: IndicSBERT Heatmap (5×5)**
```python
# Use seaborn.heatmap
# Color: white → forest green (#2D6A4F), range 0.0–1.0
# Annotate cells with 2-decimal values
# Title: "Cross-Adapter Similarity (IndicSBERT)"
# A clean diagonal of 1.00 and low off-diagonal proves no leakage
# Save as: charts/adapter_heatmap.png (300 DPI)
```
**Chart 3: Latency Breakdown (stacked horizontal bar)**
```python
# One bar per latency component
# Stacked to show total
# Colors: one per component (from Ankahi color palette)
# Target line at 500ms (red dashed vertical line)
# Save as: charts/latency_breakdown.png (300 DPI)
```
**Chart 4: Training Loss Curve**
```python
# Line chart, loss vs. training step
# Smooth curve (rolling average window=50)
# Forest green line on white background
# Annotate the final loss value
# Save as: charts/training_loss_curve.png (300 DPI)
```
**Chart 5: Radar (Capability Comparison)**
```python
# Use matplotlib polar plot
# Two filled areas: Ankahi (forest green, 40% alpha) vs Commercial AAC (gray, 30% alpha)
# 6 axes as specified above
# Legend outside the radar
# Save as: charts/capability_radar.png (300 DPI)
```
**Chart 6: Latency over time (line chart for 100 inferences)**
```python
# x: inference number (1–100)
# y: latency in ms
# Show individual points (small dots) + rolling average line
# Annotate: "warmup period" (first 5 inferences), "steady state" (rest)
# Save as: charts/latency_over_time.png (300 DPI)
```
**Chart 7: Cost Comparison (horizontal bar chart)**
```python
# y-axis: device/solution names
# x-axis: cost in INR (₹)
# Bars (low to high): Ankahi, Low-cost imported AAC, Mid-range AAC, Tobii Dynavox
# Add annotation: "ADIP scheme benefit: ₹6,000" as a vertical reference line
# Use log scale for x-axis (range is too wide for linear)
# Save as: charts/cost_comparison.png (300 DPI)
```
---
## PART 4: KAGGLE SUBMISSION CHECKLIST
Use this checklist before the May 18 deadline:
```
PRE-SUBMISSION
□ .litertlm binary produced and validated on at least one Android device
□ At least one persona adapter converted and loading successfully
□ AI4Bharat TTS integrated (or fallback TTS confirmed working)
□ Flutter app builds successfully in release mode
□ APK sideloads and runs on a physical device without crash
□ All unit tests passing (target: 36+ tests, ≥ 95% pass rate)
□ Benchmarks run and results.json populated with real numbers
□ All 7 charts generated from real data (no placeholder numbers)
□ RESULTS.md complete with all tables and chart embeds

GITHUB REPO
□ Repository is PUBLIC
□ README.md leads with: problem statement → solution → demo link → how to run
□ "Built with Unsloth" badge prominent in README (Unsloth special mention)
□ "Gemma 4 E4B" model badge prominent
□ Apache 2.0 LICENSE file present
□ requirements.txt or pyproject.toml present
□ Installation steps tested on a clean environment
□ Demo video linked in README

KAGGLE NOTEBOOK
□ Notebook runs without errors (test with "Run All")
□ All output cells have saved outputs (don't require actual H100 to view)
□ All charts render in notebook output
□ Section headers clearly label each training stage
□ Unsloth integration is highlighted and explained
□ A "Why Gemma 4?" section explicitly mentions E4B multimodal features used
□ RESULTS.md is linked from the notebook

VIDEO
□ Video is ≤ 5 minutes
□ Opens with problem statement (the 2.5 million CP children figure)
□ Shows a real Android tablet (not a laptop screen)
□ Shows a child (or hand) tapping pictogram tiles
□ Shows the Hindi sentence appearing
□ Shows audio playing (parent's voice)
□ Closes with the GitHub link and "free, offline, open-source" statement

FINAL SUBMISSION
□ Kaggle submission form completed
□ Track selected: Digital Equity (primary)
□ "Uses Unsloth" checkbox marked (if present in submission form)
□ GitHub link submitted
□ Video link (YouTube unlisted or Google Drive) submitted
□ Team members added to submission (if applicable)
```
---
## PART 5: RESOURCES MASTER LIST
**[SEARCH REQUIRED before starting any section]**
All resources below should be re-verified for current status — URLs and APIs may have changed.
### MediaPipe / LiteRT
- Main docs: `https://ai.google.dev/edge/mediapipe`
- LLM inference guide: `https://ai.google.dev/edge/mediapipe/solutions/genai/llm_inference`
- Android LLM guide: `https://ai.google.dev/edge/mediapipe/solutions/genai/llm_inference/android`
- Samples (search): `site:github.com/google-ai-edge mediapipe-samples llm_inference`
- PyPI: `mediapipe`, `mediapipe-model-maker` — always check latest version before installing
### Unsloth
- Main: `https://github.com/unslothai/unsloth`
- Docs: `https://docs.unsloth.ai`
- Gemma 4 specific: Search `"unsloth" "Gemma 4" fine-tuning notebook 2026`
### AI4Bharat TTS
- GitHub: Search `AI4Bharat Indic-TTS github`
- HuggingFace: Search `AI4Bharat TTS huggingface model`
- Demo: Search `AI4Bharat TTS voice cloning demo`
### IndicSBERT (for eval)
- Search: `"IndicSBERT" OR "Indic SBERT" huggingface sentence similarity`
- Likely: `sentence-transformers` library + an IndicNLP model
### Flutter + MediaPipe Integration
- Search: `flutter mediapipe genai "method channel" llm inference kotlin dart`
- Search: `flutter ffi mediapipe native library android`
### TORGO / UA-Speech (audio training data)
- TORGO: Search `"TORGO database" download dysarthric speech corpus`
- UA-Speech: Search `"UA-Speech" database download dysarthric`
### chrF++ implementation
- Search: `"sacrebleu" python "chrf" "chrf++" implementation`
- Library: `pip install sacrebleu` → `sacrebleu.corpus_chrf()`
---
*End of Backend & System Specification.*
*All three documents together constitute the complete Ankahi technical brief for the Kaggle Gemma 4 Good Hackathon submission.*