Skip to content

GPU & other resources

Every task and every endpoint component runs as its own Ray worker with a resource request — CPU, memory and, when you need one, a GPU. Ray uses that request to place the work on a suitable node, so it is both a performance setting and a scheduling one.

Most of the time you should not set it at all: the CI sizes tasks and endpoints from their own historical usage. You set it explicitly when you need a GPU, or when the automatic sizing is wrong for a workload it has not seen yet.

Where resources are set
Tasks and endpoints in peach.conf a resource: block — the supported route
@peach.task / @peach.endpoint decorator arguments — experimental

Tasks

Add a resource block to the task:

tasks:
  train_model:
    notebook: notebooks/train.ipynb
    method: train_model
    cron: "0 3 * * *"
    resource:
      num_cpus: 4
      memory: 8000000000     # bytes — 8 GB
      num_gpus: 1

num_cpus, memory and num_gpus are passed straight through to Ray. Memory is in bytes.

How the final numbers are chosen:

  1. You set any of num_cpus, memory or num_gpus → your block is used as given, and automatic sizing is skipped entirely. The build reports this as manual.
  2. Otherwise, if the task has run before → CPU and memory come from its own observed usage (metrics). A num_gpus you set is still honoured.
  3. Otherwise → a conservative default of 1 CPU and 3 GB (cold-start), until there is history to size against.

Because setting one key opts out of automatic sizing for all of them, a task that only needs num_gpus: 1 should usually set num_cpus and memory too — otherwise it keeps whatever it had before the GPU was added.

Endpoint components

The resource block goes on the component, not the endpoint:

endpoints:
  summarize:
    url: /summarize
    py_env: llm
    components:
      main:
        notebook: notebooks/Endpoints.ipynb
        init: init_model
        method: summarize
        resource:
          num_gpus: 1
          num_cpus: 2
          memory: 12000000000
          min_replicas: 1
          max_replicas: 2

Besides CPU, memory and GPU, the same block tunes how the component scales:

Key Meaning
memory bytes; a 200 MB floor is applied underneath whatever you ask for
num_cpus (or cpu) CPU cores, fractional allowed
num_gpus (or gpu) GPUs, fractional allowed
min_replicas replicas kept running. 0 lets the component scale to zero
initial_replicas replicas started on deploy
max_replicas ceiling for autoscaling (default 8)
target_ongoing_requests in-flight requests per replica before scaling up (default 5)
max_ongoing_requests hard in-flight ceiling per replica (default 10)
max_replicas_per_node replicas allowed on one node (default 1)

As with tasks, CPU and memory are sized from the component's own traffic unless you set them. Components that are actually being called are kept warm automatically; idle ones are allowed to scale to zero.

Where these can be overridden

A few endpoints have operational overrides applied by the platform on top of your block, and the Serve configuration in peach-k8s is the final word at deploy time. If a component is not running with the numbers you set, that is where to look — or ask the PEACH core team.

Asking for a GPU

Set num_gpus on the task or component. Two things follow automatically:

  • The work is scheduled onto the GPU node pool. Nothing else is needed to get it there.
  • Health-check timings are relaxed, because loading a model into a GPU takes longer than a normal replica start.

num_gpus is fractional — num_gpus: 0.3 puts three components on one GPU, which is worth doing for models that do not need a whole one. Whole-GPU requests are limited by how many GPUs the pool actually has, so a component asking for a GPU it cannot get simply waits, unscheduled.

Keep GPU components on a small max_replicas. Autoscaling a GPU deployment to eight replicas asks for eight GPUs.

With the decorators (experimental)

Experimental

The decorator route is newer than peach.conf and is still being built out. Resources work for tasks; for endpoints they are not applied yet. Use peach.conf for anything that needs a specific resource shape in production — in particular for anything using a GPU.

@peach.task

Resource arguments work, and are the same ones Ray takes. Any keyword that is not a decorator argument of its own is passed through to @ray.remote:

@peach.task(num_cpus=4, num_gpus=1, memory=8 * 1024**3, py_env="ml")
def train_model():
    ...

This applies whether you call the function yourself or it runs on a cron schedule. Note there is no automatic sizing here: a decorator task gets what you ask for, and Ray's own defaults otherwise.

@peach.endpoint

@peach.endpoint accepts a ray_actor_options argument, but a production decorator endpoint does not use it yet. Deployments generated from the peach/ folder are given fixed cold-start CPU and memory, kept warm at one replica, with no metrics-driven sizing — the resource pipeline that backs peach.conf components has not been extended to them.

So ray_actor_options today only affects an ephemeral app run with peach_serve() from a notebook:

@peach.endpoint(name="demo", py_env="llm", ray_actor_options={"num_gpus": 1})
def demo(query: str):
    ...

peach_serve(demo)   # the GPU request applies here

If a decorator endpoint needs a GPU or a large memory allocation in production, define it in peach.conf instead for now.