Putting LLMs in Production Mobile Apps

Shipping an AI feature is easy in a demo and hard in production. Here's the architecture that held up.

Never call the model directly from the client

Put a proxy in front:

func SuggestHandler(w http.ResponseWriter, r *http.Request) {
    if cached, ok := cache.Get(key); ok {
        writeJSON(w, cached)
        return
    }
    out := llm.Complete(ctx, prompt)
    cache.Set(key, out, 24*time.Hour)
    writeJSON(w, out)
}

The proxy handles caching, rate limits, retries, and cost control — none of which belong on a phone.

Design prompts like APIs

Versioned, tested, with structured output. Treat them as contracts.

Personalize on device

Lightweight on-device signals make suggestions feel like a coach without shipping private data to a server.