Skip to content

Large models (peach-models)

peach-models is a shared service that runs large AI models on GPUs for every member on the platform. You don't deploy, size or maintain a model yourself. You call an API and get a result.

That matters because GPUs are scarce and expensive. If every codops loaded its own embedding model, each would hold a GPU that sits idle most of the day. Running each model once, behind one service, keeps the hardware busy and gives every codops access to models none of them would run alone.
API https://api.models.peach.ebu.io
Web UI models.peach.ebu.io
Interactive API reference https://api.models.peach.ebu.io/docs

This section has four pages:

  • Overview (this page): what the service does, how it runs, and how to call it.
  • How models run: a short primer on GPUs, vLLM, and why some models handle many requests at once while others take one at a time. Read this if you want to make your calls fast.
  • Limits and errors: queues, timeouts, HTTP status codes, and how to retry.
  • Code examples: Python examples you can copy, including parallel requests with retries.

What is available

The service covers the usual jobs a recommendation or content platform needs:

  • Embeddings: turn text into vectors for similarity search and related content.
  • Speech to text: transcribe audio, with a streaming variant.
  • Translation: between 55 languages.
  • Text generation: a chat-style LLM.
  • Classification: purpose-built models such as constructiveness scoring.
  • Text to speech and image generation, when those models are enabled.

The set of models changes over time, so rather than keep a list here, ask the service:

GET https://api.models.peach.ebu.io/v1/models

Or open the web UI. It shows every model with its status, replica count and queue, and has a playground for trying a model without writing any code.

How it runs

Where the GPUs are

Most models run on 8x NVIDIA L40S GPUs in our own bare-metal server, rather than on rented cloud GPUs. The GPUs are passed through to virtual machines that join our Kubernetes cluster as hybrid nodes, so our own hardware and cloud machines form one cluster. When demand is high, extra cloud GPU machines can be added. You call the same API either way.

A request goes through three steps:

  1. The API front door checks your codops and token, works out the request's priority, and records usage for your codops.
  2. A queue per model holds requests until the model has room. How this queue behaves depends on the kind of model (see How models run).
  3. A model replica runs the request on a GPU. A replica is one loaded copy of a model. Popular models have several replicas, others have one.

Each model is loaded into GPU memory once and stays there, so requests do not pay a loading cost. After a restart or a scale-up, a model can take from 30 seconds to a few minutes to load. During that time requests wait or are rejected with a retryable error.

Calling it from a task or endpoint

The easiest route is pipe_algorithms_lib.deep_serve, which wraps the service and picks up your credentials from the environment:

from pipe_algorithms_lib.deep_serve import calculate_embeddings

vectors = await calculate_embeddings(
    ["some text", "some other text"],
    model="labse",
)

It offers three helpers:

Helper Does
calculate_embeddings(texts, model=..., dimensions=..., normalize_embeddings=...) embeds one string or a list of them
transcribe(audio_file, model=..., language=...) speech to text
model_inference(model, query) any other model's inference endpoint

All three are async, so call them from an async def task or endpoint.

Credentials resolve automatically: the codops from CODOPS and the token from CODOPS_{CODOPS}_SECRET, both already set for you on the cluster and in PEACH Lab. PEACH_MODELS_HOST points the client at a different environment.

Calling it over HTTP

There are two API styles, both authenticated per codops.

OpenAI-compatible

The /v1/... routes follow the OpenAI API, so an existing OpenAI client library works if you point its base URL here. Authenticate with a bearer token of {codops}:{token}:

curl https://api.models.peach.ebu.io/v1/embeddings \
  -H "Authorization: Bearer ${CODOPS}:${TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"model": "labse-embedding", "input": "hello world"}'

Available routes: /v1/chat/completions, /v1/embeddings, /v1/audio/transcriptions, /v1/audio/speech, /v1/images/generations, /v1/translations and /v1/models. Translation is not part of the OpenAI API, so /v1/translations has its own simple shape (see Code examples).

Native

The native routes take codops and token in the request body and offer options the OpenAI shape has no room for, such as priority and webhook:

curl https://api.models.peach.ebu.io/models/labse-embedding/embeddings \
  -H "Content-Type: application/json" \
  -d '{"codops": "<codops>", "token": "<token>", "input": ["hello", "world"],
       "dimensions": 1024, "priority": "high"}'

Per model type: /models/{model}/embeddings, /transcribe, /transcribe/stream, /translate, /tts, /images and /inference. The full parameter list for each is in the interactive reference.

Priority

Every request has a priority. On models that take one request at a time, the service serves higher priorities first when the queue is long, and keeps part of the capacity free for interactive work. Use it honestly: it is what stops a large overnight batch from delaying someone's live page.

Preset Value Use for
realtime 0 live streaming, a user waiting for the response
high 2 time-sensitive work
normal 5 the default for batches
low 7 background work in a task
batch 9 large batch jobs with no deadline

A number from 0 (highest) to 10 works too. If you leave it out, a single-item request (one text, one audio file, one translation) counts as high, and an embeddings request with a list counts as normal.

Within the same priority, codops take turns, so one member sending a lot of requests cannot block another member's requests.

Note

The translation models ignore priority. They process many requests at the same time in arrival order; see How models run.

Good habits

  • Retry the retryable errors. 503, 429 and 504 are normal under load. Wait for Retry-After or your own backoff, whichever is longer, and try again. See Limits and errors.
  • Send embeddings as lists, up to 2048 texts per call, rather than one call per text.
  • Send translations as separate, parallel requests, one paragraph each, rather than one long text. See How models run.
  • Use batch or low priority and a webhook for bulk work, so live traffic stays fast.
  • Access is per codops. Your token identifies you, and usage is shown per codops on the UI's stats page.