Project Charlie
A voice assistant that never talks to a server. Speech recognition and language understanding both run on the phone, with no API keys anywhere in the codebase.
The budget is smaller than the phone suggests
Android gives each process a fraction of physical RAM, and the system will kill an app that crosses the line rather than negotiate with it. The working ceiling was 2GB peak including native allocations. A speech model and a language model each want several hundred megabytes resident, so holding both at once spends the entire budget before the interface has drawn a frame. The first build did exactly that and was killed within seconds of the first utterance — and not deterministically, because whether it died depended on what else the phone happened to be doing.
the calls that mattered
Decisions
- 01
Run the models sequentially, not concurrently
Transcription always finishes before generation begins — that is what the data dependency says, not an optimisation. Keeping both models resident therefore pays full price for an overlap that never happens. Loading each engine, using it, and freeing it before the next one loads makes peak memory the larger of the two rather than their sum. It costs a model load per turn, which is a real latency trade, but the alternative is not a slower app — it is an app that gets killed.
- 02
Make teardown explicit, not garbage-collected
The sequential design is only safe if memory is actually released before the next load. Tying teardown to finalizers turns 'freed before the next load' into 'freed eventually', which silently breaks the whole strategy while the code still looks correct. Release is an explicit call on a known line.
- 03
One-directional ownership across the JNI bridge
Two C++ inference engines sit behind a native bridge into Jetpack Compose. Kotlin holds an opaque handle; C++ owns the allocation. Neither side assumes anything about the other's lifetime, because the moment those two stories disagree the result is a use-after-free that surfaces somewhere unrelated, minutes later.
what came out
Outcome
- Peak resident memory held at 650–800MB, inside the 2GB ceiling with room for the system
- Speech in and an answer out with the network off entirely — no cloud calls, no API keys
- Idle footprint near zero between turns, because nothing stays resident
Built with
- Kotlin
- C++ (NDK/JNI)
- whisper.cpp
- llama.cpp
- Jetpack Compose
- Room/SQLite