Two models, one phone, 2GB

Every voice assistant I have ever used is a microphone with a network connection. You speak, it uploads, a datacentre thinks, and something comes back. It works beautifully right up to the moment you are on a train with no signal, and then it is a very expensive way to display the time.

I wanted to know whether the whole loop could live on the phone. Not the cloud-with-a-cache version — actually on the phone, airplane mode on, no API keys anywhere in the codebase. That became Project Charlie: speech in, speech understood, answer out, all of it local.

The interesting part turned out to have nothing to do with machine learning. It was memory.

The budget is smaller than it looks

The phone I targeted has 4GB of RAM. You would think that gives you plenty of room for a couple of small models.

It does not. Android hands each app a per-process heap limit that is a fraction of physical RAM, and the rest of the system is not politely waiting for you to finish. The launcher, the system UI, whatever the user had open before you — they all want to still exist when the user comes back. Cross the line and lowmemorykiller does not negotiate. Your process is simply gone, and from the user's side the app "crashed" with no stack trace worth reading.

The working budget I set myself was 2GB peak, native allocations included. That is the number that mattered, and it is a lot tighter than 4GB sounds.

Here is what wants to fit inside it:

Add the two models together and hold both resident and you are already at the ceiling before the UI has drawn a single frame.

The obvious architecture is the one that kills you

The first version did the obvious thing. Load both models at startup, keep them warm, respond fast.

It died. Consistently, within seconds of the first utterance.

And the failure mode is worse than a plain crash, because it is not deterministic in a way you can chase. It depends on what else the phone was doing. It works on your desk with everything else swiped away, and it dies in someone's pocket. That is the kind of bug that makes you distrust your own test results.

Sequential, not concurrent

The fix was to stop treating the two models as a pipeline that exists, and start treating them as a pipeline that happens.

You never actually need both at once. Transcription finishes before generation starts — that is not an optimization, it is just what the data dependency says. The second model cannot begin until the first has produced its output. So keeping both resident is paying full price for an overlap that never occurs.

The loop became:

suspend fun respondTo(audio: AudioBuffer): String {
    val transcript = whisper.use { engine ->
        engine.load()
        engine.transcribe(audio)
    }   // freed here, before the LLM exists at all
 
    return llm.use { engine ->
        engine.load()
        engine.generate(transcript, history = memory.recent())
    }
}

use is doing the load work here, but the important word is freed. The speech model's memory is returned before the language model asks for a single byte. Peak resident set is now the larger of the two, not the sum.

That change alone brought the ceiling down to 650–800MB peak, comfortably inside the 2GB budget, with room left for the system to breathe.

What it costs

It is not free. Loading a model from storage takes real time, and doing it on every turn means the user waits twice per exchange instead of zero times.

That trade was worth making, and I would make it again, because the alternative is not "slightly slower" — the alternative is an app that does not run. A response that arrives in a couple of seconds is a product. A process that gets killed is not.

There is also a nice second-order effect: because nothing stays resident between turns, the app's idle footprint is tiny. Leave it open in the background and it is not quietly holding a gigabyte hostage.

The JNI bridge is where the real bugs live

Both engines are C++. The app is Kotlin and Compose. Everything crosses through JNI, and JNI is where the comfortable guarantees stop.

Two things I would tell anyone starting this:

Ownership has to be explicit and one-directional. The Kotlin side holds an opaque handle. The C++ side owns the allocation. Nothing on the Kotlin side ever assumes it can touch that memory, and nothing on the C++ side assumes the Kotlin object is still alive. The moment those two stories disagree you get a use-after-free that surfaces as a crash somewhere completely unrelated, minutes later.

Free deterministically, not when the GC feels like it. This is the one that matters for the whole design above. If model teardown is tied to finalizers or garbage collection, "freed before the next load" becomes "freed eventually, probably, maybe after the next load" — and the entire sequential strategy quietly stops working while still appearing correct in the code. Release is an explicit call, on a known line, before the next load().

What I actually learned

I went in expecting the hard part to be model quality — whether a 0.5B model quantized down to fit on a phone would produce anything worth reading. That was a real question, and the honest answer is that it is capable within a narrow band and you design the product around that band rather than pretending otherwise.

But it was not the hard part. The hard part was that on-device is a systems problem wearing an ML costume. The constraint was never intelligence. It was a memory ceiling, a process lifecycle that can end your app without asking, and a language boundary with no safety net across it.

The models are the easy bit. Someone else trained them. What is left for you is the engineering, and the engineering is where it actually gets won or lost.

It runs. Airplane mode, no network, speech in and an answer out, inside 800MB. That is the whole point.

The source is on GitHub. It is a personal systems project rather than a polished product, and reads like one — but the memory work described here is all in there.

• On-device AI
• Android
• Systems