What On-Prem Deployment Taught Me About LLM Serving

How a privacy constraint walked me through three serving architectures, and a proof of concept that took the same four GPUs from 5 concurrent users to 30.

When I started building an LLM-assisted annotation platform for a research lab, I assumed the interesting problems would be in the AI. The model turned out to be the easy part. Most of what I learned came from the question of where the model had to run, and from everything that one constraint forced on the rest of the design.

The pitch that got shut down in one meeting

My first proposal was the obvious one: use a hosted API like OpenAI or Anthropic. We weren’t retraining any models (adapting the system to a task happens at inference time, with no fine-tuning), so a hosted API seemed like a perfect fit. No GPUs to manage, no serving infrastructure, best-in-class models, and I could spend my time on the product instead of the plumbing.

It was shut down immediately, and for good reason: the data this tool would handle includes PII and patient-critical information. Sending that to a third-party API was a non-starter.

There was a second, subtler requirement behind the first. The platform is open source, and we couldn’t know where it would ultimately run: a researcher’s laptop, a hospital’s on-prem server, a university HPC cluster, or someone’s cloud account. Avoiding hosted APIs wasn’t enough; the architecture had to work anywhere. That pushed me toward an adapter layer for inference providers early on, which ended up paying for itself in ways I didn’t expect.

In hindsight, the privacy requirement made more design decisions for me than any whiteboard session did.

Starting with Ollama

With hosted APIs off the table, I reached for what I knew: Ollama. It’s excellent at what it’s designed for. One command pulls a model, the API is clean and local, and it runs on anything from a laptop to a GPU node. I stood it up on our university’s HPC cluster, wired it into the platform, and it got us through the early pilots without trouble.

Then, in a planning discussion, someone asked a question I should have asked myself months earlier:

“Would more than five people be able to annotate on this platform at the same time, using a medium-sized model?”

The honest answer was no, and I couldn’t properly explain why until I went looking.

Ollama is built for local, mostly single-user use. Its concurrency is a fixed parallelism setting with a simple request queue behind it. It has none of the machinery that high-concurrency serving depends on: no load balancing across replicas, no throughput-oriented scheduling, and no continuous batching, where the server merges many in-flight requests into shared GPU forward passes instead of handling them in fixed slots. Without continuous batching in particular, concurrent users end up waiting in line for the GPU rather than sharing its throughput. Our practical ceiling was around five simultaneous annotators, and past that latency degraded fast.

The uncomfortable part is that the pilots had validated the product without ever stressing the serving path. I had been the only kind of user I tested for.

The proof of concept: vLLM + LiteLLM

I spent some time reading up on how production LLM serving works, then built a proof of concept that replaced Ollama with two components:

• vLLM as the inference engine. Its two core ideas, continuous batching and PagedAttention (which manages the GPU’s KV cache in pages, the way an OS manages virtual memory), exist to serve many concurrent requests efficiently on fixed hardware. This is the layer that turns one GPU allocation into a shared resource.
• LiteLLM as the gateway in front of it. It exposes an OpenAI-compatible API and handles routing, so the application talks to one stable interface no matter what’s serving behind it. Because of the provider adapter layer from the start of the project, swapping the backend needed almost no application changes; the abstraction built for deployment portability is what made the migration cheap.

The results, on the same four V100s with the same medium-sized model:

The numbers come from load tests against each stack on the same GPU allocation, with scripted users issuing realistic annotation requests concurrently.

That’s a 6× improvement in concurrency without touching the model or the hardware. What mattered more than the number was what it implied: the ceiling was no longer an artifact of the serving software. If we needed more users, the answer became “smaller model or more GPUs”, a resourcing question, instead of “the platform can’t do that”, an architecture problem.

That was the real lesson for me: in an LLM application, the serving stack can matter as much as the model. Scheduling and memory management decide how much of the hardware’s throughput users actually see. A GPU running an unbatched server isn’t slow, it’s mostly idle.

Where it ended up

The vLLM + LiteLLM setup stayed a proof of concept. The shipped product still runs on the simpler stack the pilots validated, and that was a deliberate scope call: our deployments never put more than a handful of simultaneous annotators on the system, so the simple stack was enough for what we were shipping, and the remaining project time was better spent elsewhere. What the POC bought us was a measured answer to “what would it take to scale this?”, instead of a guess. That answer, along with a working understanding of continuous batching, KV cache management, and gateway/engine separation, is what I take with me to the next system.

What I’d do differently

I’d ask the concurrency question on day one: not “does it work?” but “does it work when N people show up at once?”, with N taken from the customer’s world rather than from my test setup. I’d also pay closer attention to what the real bottleneck is. A well-architected system should be limited by resources you can buy more of, not by software choices baked in early.

And I’d remember that a prototype that never ships still counts, as long as you measure it. The 5→30 result changed how the team thought about the platform’s future even though the code behind it never merged.