Why I'm rebuilding my journaling app to run entirely on-device.

The moment it clicked was around the tenth entry I typed into my prototype. I'd just written something I'd never say out loud, and I watched the request go out to a Groq endpoint. Even if the company promised “we won't look,” the entry was sitting on someone else's server, and I was the one who put it there.

Cloud AI for journaling is a policy claim. Architecture beats policy. I'm rebuilding it to not have to make that claim.

The stack I'm testing

Everything runs on-device on iOS and Android:

• Llama 3.2 3B Instruct (Q4_K_M, ~2.0 GB) for reflection, via llama.rn (which wraps llama.cpp). Metal GPU offload on iOS (all 99 layers). CPU-only on Android - the current Android build doesn't link Vulkan.
• Whisper small.en (q5_1, ~190 MB) for voice transcription, via whisper.rn. English-only in v1 - the multilingual weights waste capacity that could go to English accuracy, and I don't have signal for multilingual demand yet.
• MiniLM sentence embeddings for semantic search, on-device.
• SQLite with a local vector extension for the journal + embeddings. Nothing syncs.

Model choice isn't locked. I ran the same reflection prompts through Qwen 2.5 3B and Gemma 2 2B first. Llama 3.2 3B was the least-bad on the specific job of “notice a pattern and reflect it back without sounding like a fortune cookie.” That could change before v1.

Two decisions I'll get asked about:

Why not CoreML / Neural Engine on iOS? Metal via llama.cpp is portable across Android too, so I can swap models with one build. Neural Engine has fixed-model constraints; llama.cpp lets me iterate on model choice without re-converting.

Why not native STT (SFSpeechRecognizer / Android SpeechRecognizer)? SFSpeechRecognizer routes long-form audio through Apple servers by default; opting out is per-request and easy to miss. Android's SpeechRecognizer behaviour varies by OEM and can silently phone home. Whisper on-device is the only path where I can verify audio never leaves the phone.

What's working (measured)

W5 device tests, means across 20 sequential entries per device:

• iPhone 17 (A19 Pro, 8 GB RAM, Metal GPU): save-tap → complete reflection = 11.6 s mean (first token at 7.6 s).
• OnePlus 9R (Snapdragon 870, 8 GB RAM, CPU-only): 59 s mean (first token at 46 s).
• Reliability: 100% pipeline success on both devices. Structured JSON extraction (emotions, triggers) parses cleanly every run.
• Semantic search: vector query across all past entries, returned inline with the reflection. Sub-second.
• Airplane mode: everything above still works. Zero network calls after the one-time model download.
• Storage: ~2.2 GB on first launch (Wi-Fi gated). Grows ~4 KB per entry after that - a year of daily journaling adds ~1.5 MB.

The Android number is slow. That's the honest current state - CPU-only inference on a 2020 mid-tier SoC. Newer Snapdragons (8 Gen 2+) should land 30-50% faster, but I don't have hardware to confirm.

What isn't yet

I'm not shipped. This is a bet, not a product. Real issues:

Reflection quality gap. A 3B model on-device is not a 70B model in a data center. Reflections are competent, sometimes surprising, sometimes flat. Prompt engineering for a 3B is a different craft - more explicit, less trust that the model will infer. Some of my drafts read like a well-intentioned but slightly dim therapist. Working on it.

Android is slow. 5-10× the iOS latency because llama.rn's Android build doesn't link Vulkan yet. Some of the gap is also the SoC itself. Newer Snapdragons should help; a proper Vulkan delegate would help more. Not something I can fix at the app layer.

Model coresidence on 4 GB devices. iPhone 12 sits at the memory edge with both models loaded. My plan is dual-model swap: unload Whisper before Llama runs (voice transcription happens first, then LLM), release LLM after reflection completes. Works in prototype; not battle-tested at scale.

Thermals + battery. Five entries in a row noticeably warms the device, especially on Android. Each reflection costs roughly the CPU cycles of 30 seconds of 1080p playback. Daily use adds <5% drain; a power session (10+ entries) can hit 15%.

Device floor. iPhone 11 and older, or Android below 6 GB RAM / Snapdragon 855, aren't in the shipping band. Reflections can exceed two minutes on those; memory pressure risks jetsam during voice recording.

Model updates. v1 doesn't have an in-app model update path. If I release an improved model in v1.5, users re-download via a new app version. Fixing this in v1.5.

Why this shape of app matters

A cloud journal asks you to trust the company. That's a policy claim. If the company changes hands, changes leadership, changes its business model - the policy can move too. Meanwhile the entries still sit on the servers.

An on-device journal asks you to trust your phone. That's an architectural claim. The entries can't be sold, subpoenaed, breached, or fed into a training pipeline, because they aren't anywhere except your device. Different in kind, not degree.

The 5-10× latency vs cloud is the price of that guarantee. It's a real trade. I'm not pretending it isn't.

If you're doing similar work and want to compare notes - happy to. Beta link is in the footer.