Skip to content

Data Science workflow

In very simple words to build recommendation service it is required to get some data about user and items, build model from it based on some algorithm and then serve recommendations back to user based on this model.

In PEACH there are several components which are supposed to simplify and automate the process of building recommendations:

  • Collect API and proposed data format for broadcasters to integrate in their products to send data to PEACH
  • PEACH Lab - web-based interactive development environment ready to be used by data scientists based on Jupyter Lab
  • Git repositories in EBU GitLab to share the code and ideas between engineers from different members
  • Schedulers for tasks and api endpoints to rollout the algorithms to production
  • Recommendation API to serve results back to the users by broadcasters products
  • Dashboards and A/B tests to iterate on algorithms and monitor its performance

PEACH Lab

PEACH Lab is our packaging of Jupyter Lab with additional extensions and libraries.

It provides:

  • ready to use Python 3 environment with common libraries preinstalled
  • integration with EBU GitLab and an extension to perform common git operations
  • Configured with access to the data stored in S3, Druid, Milvus, Codex (MongoDB), influxDB, Redis, Kafka, etc...
  • GitLab repo with example and tutorials to help users learn the platform
  • pipe-algorithms-lib with useful utilities and common functions
  • Useful notebooks to visualize state of tasks/endpoints in production
  • Template for new projects to start with

Tasks

Many recommendation algorithms contain a part which needs to be recomputed as new data (usually user-item interaction events or new items) becomes available. Having implemented such part, one can create a task which will run periodically recomputing the model. Such periodic automatic execution could also be useful for time series metrics computation for the dashboard.

Tasks typically load data from databases (Druid, Milvus, InfluxDB, MongoDB) and/or from S3 / Redis / Kafka, make calculations and store the result of computations (for instance matrix representing user etc) into Redis.

Recommendation API (also called endpoints)

After recommendation model is built by the task it can be used to serve recommendations based on some set of input parameters, for example, user ID or item ID.

Recommendation API is HTTP REST API where it is possible to define custom function and attach it to specific url.

In a typical use case inside this function recommendation model will be loaded from Redis and will be transformed to a set of recommended items for corresponding values of input parameters. Afterwards some Business Rules can be applied to the results and it is served back as JSON.

Performance is a critical requirement for the vast majority of endpoints. An endpoint which talks to S3 / MongoDB / Milvus / InfluxDB doesn't meet this performance requirement, this is why endpoints should rely on Redis and/or Druid instead.

Going to Production

Both Tasks and Endpoints are defined declaratively using PEACH configuration files.

This configuration together with notebooks and code should be pushed to GitLab repositories.

From a push to production: CI builds a tagged image, it lands in dev, and you promote it to prod

Wiring your repository to the build

Any GitLab repository can be built by the PEACH pipeline. Add a .gitlab-ci.yml that includes the shared CI file:

include:
  project: pipe/peach-jobs-ci
  file: deploy-on-peach.yml

For most repositories that is the entire file — see sr-notebooks for a live example. Everything else (the runner, the image, the configuration validation and the deploy itself) comes from the included file.

Two job token permissions have to be granted once, because the pipeline and your repository authenticate to each other:

  • In your repository, under Settings → CI/CD → Job token permissions, allow pipe/peach-jobs-ci to authenticate against your project.
  • In peach-jobs-ci, under its own Settings → CI/CD → Token Access, your project has to be allowlisted so that the validation job can clone the engine using your repository's own CI_JOB_TOKEN. Ask the PEACH core team for this one.

No tokens need to be declared as CI/CD variables in your repository.

What a push builds

A push to your default branch (main or master) validates your configuration against the peach-jobs-ci schema, then builds an image and tags it <codops>_vX.Y.Z, incremented from the last tag for your codops.

That tag is deployed to your codops' dev environment. A push never deploys to production: you promote a tag there deliberately, once you are happy with it in dev. (A codops that has no dev environment is deployed straight to prod.)

Promoting to production, and rolling back

Two interfaces do the same job — use whichever suits you.

The ops dashboard, at https://{codops}.peach.ebu.io, shows the tag currently running, the deploy log and who deployed what. Deploy new tag lists recent tags with their commit messages: pick the one you have been running in dev to promote it, or an earlier one to roll back. The dev environment has its own dashboard at https://{codops}-dev.peach.ebu.io. Deploying requires admin rights.

Slack: run /ops in your ops-{codops} channel. It shows Dev and Prod side by side — ArgoCD status, the running tag, the tag in the repository and how long it has been up — with a Promote button to lift the dev tag into prod, and Choose tag to deploy to pick any recent tag for either environment.

Rolling back is not a special operation: it is a deploy of an older tag.

As soon as Git is used to track algorihms and configurations users can benefit from well-known features such as:

  • Git Feature Branches workflow - proceed with your work in separate branch and merge it to master when you are ready
  • Versioning - Algorithms are versioned by git commit ids
  • Collaboration - Give access to GitLab repositries to colleagues to work together
  • Code reviews - Ask a colleague or memeber of PEACH core team to review the changes before going to production

Where to go next