Experiment tracking and model registry (MLflow)
MLflow keeps a record of your machine learning work. Each time you train a
model you log a run: the parameters you used, the metrics you got, and the
files you produced (the model itself, plots, sample predictions). Runs are
grouped into experiments, so you can compare them side by side in a web UI.
When a model is good enough you add it to the model registry, give it an
alias such as champion, and load it by that name from your tasks and
endpoints.
Peach hosts an MLflow server per codops. Your codops' runs, models and files stay inside your codops: nobody from another codops can see them.
| Web UI and API (external) | https://mlflow.<codops>.peach.ebu.io |
| API (in-cluster) | http://peach-codops-<codops>-mlflow:5000 |
| MLflow version | 3.18 |
Replace <codops> with your codops name.
MLflow is enabled per codops. If it is not yet available on yours, ask the Peach team.
Where your data is stored
- Runs, parameters, metrics and the model registry are stored in your codops' own Postgres database, which is backed up daily (kept for 7 days).
- Files (models, plots, anything you log with
log_artifactorlog_model) go to your codops' S3 bucket, undermlflow/. They pass through the MLflow server, so you never need AWS credentials to log or download them. Deleting a run's files is permanent. - No usage data leaves Peach. MLflow's built-in usage telemetry is switched off on the server and in the web UI.
Usage rule: no raw event data or client identifiers
MLflow is for experiment parameters, metrics and trained models. Do not store raw event data or client identifiers in it, in any form: not as parameters, metrics or tags, and not inside logged files such as datasets, sample predictions or evaluation tables.
Peach's GDPR erasure process removes a person's data by their client identifier, and it does not cover MLflow. Anything identifying a person that ends up in MLflow could not be erased. Log aggregates instead.
Connect from inside Peach (tasks, endpoints, Ray jobs)
Code running in your codops (tasks, endpoints, Prefect flows, Ray jobs) connects to the in-cluster address. No login and no token are needed.
import mlflow
mlflow.set_tracking_uri("http://peach-codops-<codops>-mlflow:5000")
mlflow.set_experiment("my-experiment")
with mlflow.start_run():
mlflow.log_param("n_estimators", 50)
mlflow.log_metric("r2", 0.47)
You can also set the address as an environment variable instead of calling
set_tracking_uri:
MLFLOW_TRACKING_URI=http://peach-codops-<codops>-mlflow:5000
Note
The in-cluster address only works from your own codops' workloads. PEACH Lab notebooks cannot use it: they use the external address with a login, as described below.
Connect from your laptop or PEACH Lab
The external address uses the same GitLab login as the rest of Peach. The MLflow Python client sends it as a token:
-
Sign in with the Peach command line (peach-cli). In PEACH Lab no browser is needed: it signs you in as the lab's owner.
bash peach auth login -
Point MLflow at the external address and give it your token:
```python import json, os, pathlib import mlflow
credentials = json.loads((pathlib.Path.home() / ".peach" / "credentials.json").read_text()) os.environ["MLFLOW_TRACKING_TOKEN"] = credentials["token"]
mlflow.set_tracking_uri("https://mlflow.
.peach.ebu.io") ```
From then on, every MLflow call (start_run, log_model, load_model,
search_runs, ...) works as in the in-cluster example.
Your token expires after 12 hours
When it expires, MLflow calls fail with 401 Unauthorized. Run
peach auth login again and reload the token. A training run that takes
longer than that should run inside Peach (as a task or Ray job), where no
token is needed.
Keep the token out of notebooks, code and git. Load it from
~/.peach/credentials.json as shown above; never paste it into a cell.
Web UI
Open https://mlflow.<codops>.peach.ebu.io in your browser and sign in with
GitLab. You can compare runs, look at metrics charts, download files and manage
registered models and their aliases.
Full example: train, register, load
This example trains a scikit-learn model, logs it, registers it, sets the
champion alias and loads it back by that alias.
import mlflow
from mlflow import MlflowClient
from sklearn.datasets import load_diabetes
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import train_test_split
mlflow.set_tracking_uri("http://peach-codops-<codops>-mlflow:5000")
mlflow.set_experiment("diabetes")
# Log parameters and training metrics automatically. The model itself is
# logged explicitly below.
mlflow.sklearn.autolog(log_models=False)
X, y = load_diabetes(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)
with mlflow.start_run(run_name="random-forest"):
model = RandomForestRegressor(n_estimators=50, max_depth=6).fit(X_train, y_train)
mlflow.log_metric("test_r2", model.score(X_test, y_test))
mlflow.sklearn.log_model(
model,
name="model",
input_example=X_test[:3],
registered_model_name="diabetes-rf",
# Needed for tree-based scikit-learn models, see "Things worth knowing".
skops_trusted_types=["sklearn.tree._tree.Tree"],
)
# Mark the newest version as the one to use.
client = MlflowClient()
version = client.get_registered_model("diabetes-rf").latest_versions[-1].version
client.set_registered_model_alias("diabetes-rf", "champion", version)
# Anywhere else (a task, an endpoint): load the model by its alias.
champion = mlflow.pyfunc.load_model("models:/diabetes-rf@champion")
print(champion.predict(X_test[:2]))
Install the client in your Python environment:
# pyproject.toml
dependencies = ["mlflow>=3.18", "scikit-learn"]
Using a registered model in an endpoint
Load the model once, when the endpoint starts, not on every request. To roll
out a new model, move the champion alias to the new version in the web UI or
with set_registered_model_alias, then restart or redeploy the endpoint. To
roll back, move the alias back.
Evaluating LLM apps (GenAI)
If your codops builds something on top of a language model (a chatbot, a summariser, a tagging pipeline), MLflow can record what it did and have another model grade the results:
- Traces record each call to your app: the input, the output and the steps in between.
- Judges (also called scorers) are a language model asked a question about a trace, such as "is this answer factually correct?". You can run them on demand or let the server run them on every new trace.
- Issue detection reads a set of traces and groups the problems it finds (wrong answers, irrelevant answers, errors) into issues you can track.
Judges and issue detection need a language model. You connect one once per codops, in MLflow's AI Gateway, and then refer to it by name.
1. Connect a model in the AI Gateway
The simplest choice is peach-models: the model runs on Peach GPUs and your traces never leave Peach. It speaks the OpenAI API, so you add it as an OpenAI provider with a different address.
In the web UI, open AI Gateway and choose Create endpoint:
| Field | Value |
|---|---|
| Endpoint Name | for example peach-llm; you use this name in your code |
| Provider | OpenAI |
| Model | a text generation model from https://api.models.peach.ebu.io/v1/models, for example qwen3.8-27b-llm |
| API key | create a new one, named for example peach-models, with the value <codops>:<token>: your codops name and its peach-models token (the value of CODOPS_<CODOPS>_SECRET) |
| API Base URL | https://api.models.peach.ebu.io/v1 |
The API key is stored encrypted on your codops' MLflow server and can be reused for more endpoints. Anyone in your codops can use the endpoints, but nobody can read the key back.
You can also connect an external provider such as OpenAI or Anthropic with your own API key. Your traces are then sent to that provider, so check with your organisation that this is allowed for the content in them before you do.
To check the endpoint, send it a message:
import requests
r = requests.post(
"http://peach-codops-<codops>-mlflow:5000/gateway/peach-llm/mlflow/invocations",
json={"messages": [{"role": "user", "content": "Reply with one word: pong"}]},
)
print(r.json()["choices"][0]["message"]["content"])
2. Record traces from your app
Add @mlflow.trace to the function that answers a request:
import mlflow
mlflow.set_tracking_uri("http://peach-codops-<codops>-mlflow:5000")
mlflow.set_experiment("my-assistant")
@mlflow.trace
def answer(question: str) -> str:
... # call your model, search, etc.
return reply
A trace stores the full input and output, so if your app handles text written by your audience, the traces contain it too: trace a test or sample set rather than live audience traffic unless your organisation has agreed to that.
Each call now appears under Traces in the experiment. For LangChain,
LlamaIndex, the OpenAI client and many other libraries,
mlflow.<library>.autolog() records the inner steps as well (see the upstream
tracing docs linked in the sidebar).
3. Write a judge
A judge is a question, written in plain language, and the endpoint that answers
it. {{ inputs }} and {{ outputs }} are replaced with the trace's input and
output:
from mlflow.genai.judges import make_judge
correct = make_judge(
name="factually_correct",
instructions=(
"Is the answer in {{ outputs }} a factually correct answer to the "
"question in {{ inputs }}? Answer 'yes' or 'no'."
),
feedback_value_type=str,
model="gateway:/peach-llm", # your AI Gateway endpoint
)
Run it on traces you already have:
traces = mlflow.search_traces(max_results=50)
result = mlflow.genai.evaluate(data=traces, scorers=[correct])
The verdicts, with the judge's reasoning, are attached to each trace and shown in the web UI.
A judge is a language model and makes mistakes of its own. Check a sample of its verdicts by hand before you rely on them, and reword the instructions if it gets things wrong. A more specific question ("does the answer name the right city?") usually works better than a general one ("is this answer good?").
4. Score new traces automatically
Register the judge and start it. The MLflow server then grades new traces in the background without your code running. It checks once a minute, but leaves each trace alone for 5 minutes first so the trace can finish, so expect a verdict 5 to 6 minutes after the call:
from mlflow.genai.scorers import ScorerSamplingConfig
registered = correct.register()
registered.start(sampling_config=ScorerSamplingConfig(sample_rate=0.1))
sample_rate=0.1 grades one trace in ten; every graded trace is a call to the
model, so start low. Stop it with registered.stop(), or in the web UI under
the experiment's Judges tab.
5. Find issues
In the experiment's traces, select some traces and choose Detect issues, pick your AI Gateway endpoint and the kinds of problem to look for (correctness, relevance, safety, ...). The server analyses them in the background and lists what it found under Issues. With Schedule issue detection it runs hourly, daily or weekly on new traces.
Note
Judges and issue detection call the model once or more per trace. On peach-models that load is shared with every other member, so prefer a sample of your traces over all of them.
Things worth knowing
- Everyone in your codops has full access. There are no per-user permissions inside MLflow: anyone who can sign in for your codops can read, change and delete every experiment, run and model. Agree on naming conventions for experiments and models within your team.
- Tree-based scikit-learn models need one extra argument. MLflow 3.18 saves
scikit-learn models in a safer format and refuses tree models
(
RandomForest*,GradientBoosting*,ExtraTrees*,DecisionTree*) with the errorThe saved sklearn model references untrusted types. If you trained the model yourself, passskops_trusted_types=["sklearn.tree._tree.Tree"]tolog_model, as in the example above. - Use an MLflow 3 client, ideally the same version as the server (3.18). Much older clients may not support every feature.
- GenAI features are switched on per codops. If the AI Gateway or scheduled scoring does not work on yours, ask the Peach team. Experiment tracking, the model registry and model files do not depend on them.
- MLflow does not serve models. To put a model behind an API, load it in a Peach endpoint (see above) or use peach-models for large models.
Related
- Command line (peach-cli): signing in with
peach auth login. - Tasks and Recommendation API (endpoints): where in-cluster training and serving code runs.
- PEACH Lab: notebooks, connecting with the external address.
- Python environments: adding
mlflowto your dependencies.