Limits and errors

peach-models is shared by every member, so it protects itself from overload. This page explains the limits, what each HTTP status code means, and how your code should react.

The limits below are the current production values. We adjust them as demand changes, so build your code to react to the status codes, not to the exact numbers.

Queue limits

Models that take one request at a time (embeddings, speech to text, the chat LLM, classification and others; see How models run) have a queue in front of them. A new request is rejected with 503 when any of these is reached:

Limit	Current value	What happens
Requests waiting for one model, all codops together	64	everyone's new requests to that model get `503`
Requests waiting for one model from your codops	48	your new requests get `503`; other codops are not affected
Interactive requests (`realtime` or `high`) waiting from your codops	16	your new interactive requests get `503`, so they fail fast instead of waiting behind a long queue

Translation models (many requests at once) have no such queue limit today. Up to 32 requests (translategemma-4b) or 16 requests (translategemma-12b) are processed at once, and the rest wait. If you send far more than that, you will see responses slow down until requests reach their time limit (504). We may add a 503 limit to these models too, so handle it now.

Time limits

Every request has a maximum processing time. When it is reached, the request is stopped and you get 504.

Model	Time limit
Most models	60 seconds
Translation, `whisper-large-v3`	300 seconds
Chat LLM	900 seconds

Webhooks remove the time limit for your client. On the native routes, pass a webhook URL. If the result is not ready in time, you get 200 with "status": "pending" right away, and the result is sent as a POST to your URL when it is done. The URL must be reachable from the Peach cluster.

Your HTTP client probably has its own timeout as well. Set it a little higher than the model's time limit, or you will cut off requests that would have succeeded.

Status codes

Code	Meaning	What to do
`200`	Success. With a webhook, the body may have `"status": "pending"`; the result follows by `POST`.	Nothing.
`400`	Invalid input, for example an unknown priority value or more than 2048 texts in one embeddings call.	Fix the request. Don't retry it unchanged.
`401`	Missing or wrong credentials.	Check the codops and token. OpenAI-style routes need `Authorization: Bearer <codops>:<token>`.
`404`	Unknown model name.	Check the name with `GET /v1/models`.
`422`	The request body is missing a field or has the wrong type.	Fix the request. The body says which field is wrong.
`429`	The Peach team has temporarily paused some traffic, for example during an incident. `Retry-After` says how many seconds until the pause ends.	Wait for `Retry-After` (or your backoff, if longer), then retry. If the pause is longer than you can wait, stop.
`500`	The model failed on this request.	Retry once. If it keeps failing, contact us with the request time.
`503`	The model is overloaded: one of the queue limits above was reached. `Retry-After` is set to 5 seconds.	Wait for `Retry-After` or your backoff, whichever is longer, then retry. Lower how many requests you send in parallel.
`504`	The request took longer than the model's time limit.	Retry, ideally with smaller input. For long jobs, use a webhook.

How to retry

A good client:

Retries 429, 503 and 504, plus network errors. Don't retry 400, 401, 404 or 422; the same request will fail the same way.
Waits before each retry, for whichever is longer: the Retry-After value when the response has one, or its own exponential backoff (1, 2, 4, 8 seconds), plus a little randomness. See Why exponential backoff.
Gives up after a few attempts (4 or 5) and reports the error.
Limits how many requests it has in flight, for example with a semaphore. Retrying without a limit makes overload worse.

Code examples has a complete Python version.

Why exponential backoff

When a model is overloaded, retrying straight away only adds more requests to a queue that is already full. If many clients do that at once, the model never gets the chance to catch up.

Exponential backoff means doubling the wait after each failed attempt: 1 second, then 2, then 4, then 8. A short blip costs you a second, while a longer overload is met with longer and longer pauses that give the model room to recover.

Jitter means adding a small random amount to each wait, for example up to one second. Without it, clients that failed at the same moment all retry at the same moment too, and hit the model in a burst again. With it, their retries spread out.

Retry-After and backoff together

Use both. Retry-After is the server telling you the minimum time to wait; your backoff makes sure you wait longer each time an overload goes on. So wait for whichever of the two is longer.

Our 503 responses always say Retry-After: 5. A client that only follows the header retries every 5 seconds for as long as the overload lasts, and never slows down. Taking the longer of the two fixes that. A worked example, with 5 attempts:

Attempt	Result	Your backoff	Wait before the next attempt
1	`503`, `Retry-After: 5`	1 s	5 s + jitter (server's value is longer)
2	`503`, `Retry-After: 5`	2 s	5 s + jitter
3	`503`, `Retry-After: 5`	4 s	5 s + jitter
4	`503`, `Retry-After: 5`	8 s	8 s + jitter (your backoff is now longer)
5	`200`		done

A 429 can have a much longer Retry-After, up to the length of the traffic pause. If that is longer than your job can wait, stop and report it rather than retrying. When there is no Retry-After at all (a network error, or a 504), use your backoff alone.

If all 5 attempts fail, stop and report the error rather than retrying forever. A cap on the wait (for example 30 seconds) also helps if you allow more attempts.

Seeing where the time goes: Server-Timing

Every model response includes a Server-Timing HTTP header. You don't need to ask for it:

Server-Timing: total;dur=1834.2, queue;dur=412.7

Part	Meaning
`total`	milliseconds from when our API received the request until it answered
`queue`	milliseconds spent waiting in the queue before a model started on it. Only on models that take one request at a time; translation responses have `total` only

Large embedding lists that were split into chunks also get an X-Peach-Split-Count header with the number of chunks.

How to read it:

Your own measured time is much higher than total: the extra time is the network or your client, not the model.
queue is high: the model is busy. Lower your parallel requests or use a lower priority for bulk work.
Translation: total grows as you add parallel requests, but the number of texts per second does not: you are past the sweet spot. Go back down.
total is high even with few requests: the work itself is slow. Send smaller inputs, or use a smaller model.

To see the header with curl, add -D -:

curl -s -D - -o /dev/null https://api.models.peach.ebu.io/v1/embeddings \
  -H "Authorization: Bearer ${CODOPS}:${TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"model": "labse-embedding", "input": "hello world"}' | grep -i server-timing

Fair use

Send bulk work with batch or low priority, so live traffic stays fast.
Stay around the sweet spots in How models run rather than sending as many requests as possible.
Planning a large new workload, such as backfilling a whole archive? Tell the Peach team first, so we can make sure there is capacity.