A 3.1 GiB fine-tune that runs offline on a phone, reported honestly.

The findings

Base model versus base + adapter on 50 held-out prompts, identical system prompt, identical greedy decoding. The strongest result is machine-validated; the headline accuracy figure is not.

Protocol

Contamination control R

The precondition for any held-out claim: the evaluation prompts must not appear in training. If they do, every number downstream measures memorisation instead of capability.

Relay-packet compliance R

The app cannot route a distress message over the mesh without a SOS|WHO:|LOC:|NEED: packet. Because a format checker decides this, it is the one result here that requires no human judgement at all.

Response length R

A radio operator reading an answer aloud under pressure needs it short. Length is a crude proxy for that, but it is the one style claim on this page that is fully machine-checkable, so it gets reported properly rather than as a single mean.

Team-graded accuracy H

These are the numbers most likely to be quoted, and the weakest evidence on the page. They are reproduced here with their problems attached rather than cleaned up.

Safety behaviour H

This is the result that actually matters, and the one a throughput chart cannot show. On three adversarial prompts the stock model did the dangerous thing.

What fine-tuning did not fix

Published deliberately. A fine-tune report that lists only wins is not evidence, and every one of these would be found by a reviewer within an hour.

1. Anaphylaxis fails in both models H

   Neither the base model nor the fine-tune recognises throat-tightening plus wheezing after stings as anaphylaxis, and neither mentions an adrenaline auto-injector. This is a potentially life-threatening gap in the model's headline domain, and fine-tuning did not touch it. It is the reason the deployment guidance says human-supervised only.

2. Multilingual barely moved: 38% → 43% H

   The stated differentiator is the weakest result. The base model answers in the wrong language or refuses; the fine-tune answers in-language but sometimes degenerates into repetition, and on one Marathi prompt it emitted a garbled packet with hallucinated fields and scored worse than base. Root cause is data volume — roughly three training examples per non-English language.

3. Numeric reasoning regressed H

   On one resource-allocation prompt the fine-tune assigned 36 of 18 available volunteers — arithmetically impossible. The base model was vaguer but not wrong.

4. General capability was never re-tested

   No MMLU, MedQA, or IFEval run exists for either model. Narrow supervised fine-tuning on 1,628 examples commonly causes measurable regression on general reasoning, so catastrophic forgetting cannot currently be ruled out. This is an absence of evidence, not evidence of absence — and it is cheap to fix.

5. No few-shot baseline

   Both models got the same system prompt, which is correct. But a 3-shot prompted base model was never tried. If three in-context examples get the base model to 4/4 packets, the value proposition of fine-tuning changes substantially. This is the first question a reviewer asks and it is currently unanswered.

6. A withdrawn claim: “best quality-per-watt”

   An earlier write-up used this phrase. No power, current, or energy measurement was ever taken. A watt-denominated claim without a watt measurement is unsupportable, so it is withdrawn everywhere. Measuring millijoules per token would be genuinely differentiating — few on-device LLM reports publish energy, and for grid-down use it is arguably the most decision-relevant metric of all.

On-device measurements M

One run per model, 12 July 2026, on a OnePlus 15 — Snapdragon 8 Elite Gen 5, Hexagon v81, 15.5 GB RAM — through llama.cpp's ggml-hexagon backend at temperature 0.

What this report does and does not establish

Stated plainly, because the difference is the whole value of the document.

The experiments that would close the gaps

Ordered by credibility gained per unit of effort. The first three need no new data and no GPU — only grading discipline.

Reproduce the R claims

Every reproducible number on this page is recomputed from the released artefacts, every figure is generated from that output, and a fourth script checks the page did not drift from it. No GPU, no network, no model download — it reads the CSV and JSONL files in the repo.

git clone https://github.com/Kesav2k04/Sankat-Mochan.git
cd Sankat-Mochan

python docs/benchmarks/verify_benchmarks.py
# Asserts the headline claims against the artefacts.        22/22 pass

python docs/benchmarks/compute_chart_data.py
# Derives the distributions behind them: the per-prompt Jaccard
# spread, the paired length pairs, exact binomial intervals and a
# 20,000-rep bootstrap at a fixed seed.  -> chart_data.json

python docs/benchmarks/make_figures.py
# Redraws every figure on this page from that JSON.

python docs/benchmarks/verify_page_numbers.py
# Checks this page quotes those numbers correctly.          50/50 pass