How models run

You don't need this page to call peach-models. But if you are sending a lot of requests, or wondering why a call is slow, it explains what happens on the GPU and how to make your calls fast.

A short GPU primer

A GPU is a processor built to do the same simple calculation on a lot of data at once. AI models are mostly huge grids of numbers (the model's weights) that get multiplied with your input, which is exactly the kind of work a GPU is good at.

Three facts explain most of the behaviour you will see:

The model has to fit in GPU memory. A GPU has a fixed amount of memory (48 GB on the L40S cards we use). A translation model takes 18 to 31 GB, an embedding model a few GB. So only a handful of models fit on one GPU, and adding another copy of a model (a replica) is only possible where memory is free.
Reading the weights is the slow part. For every step of work, the GPU has to read the model's weights from its memory. Doing that for one request or for 20 requests at the same time costs almost the same. That is why processing requests together is so much more efficient than one by one.
Text models write their output one piece at a time. Models that generate text (translation, chat) produce their answer one token at a time. A token is roughly a short word or part of a word, about 3 to 4 characters in European languages. Each token is one full step through the model. A 500-token translation takes 500 steps, whatever else happens.

Two kinds of models

Because of those facts, peach-models runs models in one of two ways.

One request at a time

Most models (embeddings, speech to text, classification, text to speech, images and the chat LLM) process one request at a time per replica. The libraries they use can't usefully share the GPU between requests: running two at once would just slow both down, or run out of memory.

For these models:

Capacity grows with the number of replicas. Three replicas means three requests running at once.
Waiting requests sit in a queue in front of the model. The queue is ordered by priority, and codops take turns within the same priority.
When the queue is full, new requests are rejected quickly with 503 and a Retry-After header, instead of waiting a long time.
The number of replicas is set by the Peach team, based on measured demand. If you plan a large new workload, tell us beforehand so we can add capacity.

Replicas kept free for live traffic

A batch job can easily keep every replica of a model busy. A live request arriving at that moment would then have to wait behind it. To prevent this, models with two or more replicas keep half of their replicas free for interactive requests (priority realtime or high) whenever there has been interactive traffic recently:

When it applies: while the model has seen an interactive request in the last 20 minutes. After 20 minutes without one, batch work may use every replica again.
How much is kept free: half the replicas, rounded up. With 4 replicas, 2 are kept for interactive requests and batch work uses the other 2. Interactive requests can use all 4.
Models with one replica are never reserved, so batch work is not locked out.
Translation models don't do this, since they process many requests at once without a queue.

What this means for you:

Mark bulk work as bulk. Send it with priority low or batch. Then it uses the free capacity without slowing down live pages.
Set priority explicitly in batch jobs. A request without a priority that contains one item counts as high, which is interactive. A batch job that sends one text per call without setting priority is treated as live traffic: it competes for the reserved replicas, and is limited by the smaller interactive queue limit (see Limits and errors).
Reserve realtime and high for requests a person is waiting for.
Large embedding lists are split up. A list of more than 6 texts is cut into chunks of up to 16, which run on several replicas in parallel. Sending one list of 500 texts is still much better than 500 separate calls.

Many requests at once (vLLM)

The translation models run on vLLM, an engine built for text-generation models. vLLM uses continuous batching: it keeps a group of requests on the GPU and moves all of them forward one token per step. When one request finishes, a new one takes its place straight away, without waiting for the others.

Since reading the weights is the expensive part, and every step reads them once for the whole group, 20 requests together take only a little longer per step than one request alone. In our load tests, one replica of translategemma-4b handled about 14 to 19 times more requests per second with 24 requests in parallel than with one at a time.

For these models:

You get the most out of them by sending requests in parallel. Up to a point: each model has a sweet spot. Beyond it, more parallel requests don't increase throughput; each one just takes longer.
Priority has no effect. Requests are processed in arrival order.
Requests beyond the model's limit wait in a queue on our side. See Limits and errors.

Which models work which way

Model type	How it runs	How to send requests
Embeddings	one at a time per replica, several replicas	lists of texts in one call, a few calls in parallel
Speech to text	one at a time per replica	a few files in parallel, or a webhook for long audio
Chat LLM	one at a time per replica	one or two calls at a time
Classification, text to speech, images	one at a time per replica	a few calls in parallel
Translation (`translategemma-4b`, `translategemma-12b`)	many at once (vLLM)	many separate calls in parallel

The web UI shows each model's current replica count and queue.

Translation in practice

Each translation model currently runs as one replica. These numbers come from our load tests, with English to Swedish:

	Max in progress at once	Sweet spot	Approx. throughput, short text (~24 words)	Approx. throughput, paragraph (~140 words)
`translategemma-4b`	32	16 to 24 parallel requests	25 to 30 per second	about 6 per second
`translategemma-12b`	16	about 8 parallel requests	about 6 per second	about 1 per second

These numbers are shared by everyone using the model. They depend on text length and language, and will change as we tune the service. Treat them as a guide, not a guarantee.

What this means for your code:

Split long texts and send the parts in parallel. One request per paragraph, with 8 (12b) to 16-20 (4b) requests in flight, then join the results.
Don't join several texts into one request. One long request runs as one item in the group, so you lose the speed-up from parallel processing. It also takes longer per character (see below) and can hit the output limit.
Expect longer texts to be slower per character. Each new token costs a little more as the text grows, so a long paragraph translates at fewer characters per second than a short one.
Keep each request under the output limit. The output is capped at 1024 tokens per request (about 3,000 to 4,000 characters, depending on the language). Longer output is cut off.
Choose 4b if its quality is good enough for you. It handles about 5 times as many requests as 12b.

There is a ready-to-use example in Code examples.