Skip to content

Large models (peach-models)

peach-models is a shared service that hosts large models on GPUs for the whole platform. You do not deploy, size or maintain a model yourself — you call an API and get a result.

That matters because GPUs are scarce. One embedding model loaded per codops would occupy a GPU per codops, mostly idle. Hosting them once, behind a service that scales replicas with demand, keeps that hardware busy and gives every codops access to models none of them would run alone.

Where the GPUs actually are

Most of this runs on 8× NVIDIA L40S in our own bare-metal server, rather than on rented cloud GPUs.

Getting there took some engineering. The GPUs are passed through to KVM virtual machines (vfio-pci), and those VMs join our EKS cluster as hybrid nodes, reaching the AWS-managed control plane over a Tailscale WireGuard mesh — which is what lets a machine behind NAT, with no public IP, act as a first-class Kubernetes worker.

The upshot for you is that it is invisible: one cluster, one scheduler, and a pod lands on our own hardware or in the cloud according to what it asked for. You call the same API either way.
API https://api.models.peach.ebu.io
Web UI models.peach.ebu.io
Interactive API reference https://api.models.peach.ebu.io/docs

What is available

The service covers the usual jobs a recommendation or content platform needs:

  • Embeddings — turn text into vectors for similarity search and related content.
  • Speech to text — transcribe audio, with a streaming variant.
  • Translation — between language pairs.
  • Classification — purpose-built models such as constructiveness scoring.
  • Text to speech and image generation.

Which models are loaded changes as demand does — one that nobody has called for a week gets unloaded to free VRAM, and comes back when it is needed again. So rather than reproduce a list here that would go stale, ask the service:

GET https://api.models.peach.ebu.io/models

or open the web UI, which shows every model with its status, replica count and queue depth, and has a playground for trying one without writing any code.

Calling it from a task or endpoint

The easiest route is pipe_algorithms_lib.deep_serve, which wraps the service and picks your credentials up from the environment:

from pipe_algorithms_lib.deep_serve import calculate_embeddings

vectors = await calculate_embeddings(
    ["some text", "some other text"],
    model="labse",
)

It offers three helpers:

Helper Does
calculate_embeddings(texts, model=..., dimensions=..., normalize_embeddings=...) embeds one string or a list of them
transcribe(audio_file, model=..., language=...) speech to text
model_inference(model, query) any other model's inference endpoint

All three are async, so call them from an async def task or endpoint.

Credentials resolve automatically: the codops from CODOPS and the token from CODOPS_{CODOPS}_SECRET, both already set for you on the cluster and in PEACH Lab. PEACH_MODELS_HOST points the client at a different environment.

Calling it over HTTP

Two API surfaces, both authenticated per codops.

OpenAI-compatible

The /v1/… routes speak the OpenAI API, so an existing OpenAI client library works by pointing its base URL here. Authenticate with a bearer token of {codops}:{token}:

curl https://api.models.peach.ebu.io/v1/embeddings \
  -H "Authorization: Bearer ${CODOPS}:${TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"model": "labse-embedding", "input": "hello world"}'

/v1/chat/completions, /v1/embeddings, /v1/audio/transcriptions, /v1/audio/speech, /v1/images/generations, /v1/translations and /v1/models are all available.

Native

The native routes take codops and token in the request body and expose options the OpenAI shape has no room for:

curl https://api.models.peach.ebu.io/models/labse-embedding/embeddings \
  -H "Content-Type: application/json" \
  -d '{"codops": "sesr", "token": "…", "input": ["hello", "world"],
       "dimensions": 1024, "priority": "high"}'

Per model type: /models/{model}/embeddings, /transcribe, /transcribe/stream, /translate, /tts, /images and /inference. The full parameter list for each is in the interactive reference.

Priority

Every request carries a priority, and the service serves higher priorities first when the queue is deep. Use it honestly — it is what keeps a large overnight batch from delaying somebody's live page load.

Preset Use for
realtime live streaming, a user waiting on the response
high time-sensitive work
normal the default
low background work in a task
batch large batch jobs with no deadline

A number from 0 (highest) to 10 works too. If you omit it on an embeddings call, a single item is treated as high and a batch as normal.

A webhook URL can be passed instead of waiting on the response, which suits long batch jobs.

Things worth knowing

  • Requests queue. A cold or busy model makes you wait rather than fail. Set a timeout you are happy with, and prefer batch priority plus a webhook for bulk work.
  • A model that has not been called in a while may be unloaded, so the first request after that waits for it to load again. This is normal, not an error.
  • Embeddings are batched. Passing a list in one call is much faster than a call per item, up to 2048 items.
  • Access is per codops. Your token identifies you, and usage is attributed to your codops — visible in the UI's stats page.