Ray libraries

Here is the stack of Ray libraries:

There are four layers to Ray's unified compute framework:

1. Ray cluster
2. Ray Core
3. Ray AI Runtime (native libraries)
4. Integrations and ecosystem

Ray cluster

Starting at the bottom with a cluster (opens in a new tab), Ray sets up and manages cloud-agnostic clusters of computers so that you can run distributed applications on them. You can deploy a Ray cluster on AWS, GCP or on Kubernetes via the officially supported KubeRay (opens in a new tab) project.

Ray Core

Ray Core is the foundation that Ray's ML libraries (Ray AIR) and third-party integrations (Ray ecosystem) are built on. This library enables Python developers to easily build scalable, distributed systems that can run on a laptop, cluster, cloud or Kubernetes.

Ray Core provides a small number of core primitives (i.e., tasks, actors, objects) for building and scaling distributed applications. Expanding on the key abstractions mentioned before:

1. Tasks (opens in a new tab): Remote, stateless Python functions
   • Ray tasks are arbitrary Python functions that are executed asychronously on separate Python workers on a Ray cluster nodes. Users can specify their resource requirements in terms of CPUs, GPUs, and custom resources which are used by the cluster scheduler to distribute tasks for parallelized execution.
2. Actors (opens in a new tab): Remote stateful Python classes.
   • What tasks are to functions, actors are to classes; Actors extend the Ray API from functions (tasks) to classes. An actor is a stateful worker (or a service), and the methods of an actor are scheduled on that specific worker and can access and mutate the state of that worker. Like tasks, actors support CPU, GPU, and custom resource requirements.
3. Objects (opens in a new tab): In-memory, immutable objects or values that can be accessed anywhere in the computing cluster.
   • In Ray, tasks and actors create and compute on objects. These remote objects can be stored anywhere in a Ray cluster. Object References are used to refer to them, and they are cached in Ray's distributed shared memory object store.
4. Placement Groups (opens in a new tab)
   • Placement groups allow users to atomically reserve groups of resources across multiple nodes (i.e., gang scheduling). They can be then used to schedule Ray tasks and actors packed as close as possible for locality (PACK), or spread apart (SPREAD).
   • Placement groups are generally used for gang-scheduling actors, but also support tasks.
5. Environment Dependencies (opens in a new tab)
   • When Ray executes tasks and actors on remote machines, their environment dependencies (e.g., Python packages, local files, environment variables) must be available for the code to run. To address this problem, you can (1) prepare your dependencies on the cluster in advance using the Ray Cluster Launcher (opens in a new tab), or (2) use Ray's runtime environments (opens in a new tab) to install them on the fly.

Ray AI Runtime

Ray AIR is built on top of Ray Core and focuses on distributed both individual and end-to-end machine learning workflows.

Ray AIR enables end-to-end ML development and provides multiple options to integrate with other tools and libraries from the MLOps ecosystem. Each of the five native libraries that Ray AIR wraps distributes a specific ML task:

1. Ray Data (opens in a new tab): Scalable, framework-agnostic data loading and transformation across training, tuning, and prediction.
2. Ray Train (opens in a new tab): Distributed multi-node and multi-core model training with fault tolerance that integrates with popular training libraries.
3. Ray Tune (opens in a new tab): Scales hyperparameter tuning to optimize model performance.
4. Ray Serve (opens in a new tab): Deploys models for online inference, with optional microbatching to improve performance.
5. Ray RLlib (opens in a new tab): Distributed reinforcement learning workloads that integrate with the other Ray AIR libraries.

For more detail and a walkthrough, see this page.

Integrations and ecosystem libraries

Ray integrates with a rich ecosystem (opens in a new tab) of the most popular distributed computation libraries, machine learning libraries and frameworks. Instead of trying to impose a new standards, Ray allows you to scale existing workloads by unifying tools in a common interface. This interface enables you to run ML tasks in a distributed way, a property most of the respective backends don't have, or not to the same extent. Here are a handful of integrations to highlight:

• Ray Datasets
  • Dask (opens in a new tab)
  • Pandas/Modin (opens in a new tab)
• Ray Train and RLlib
  • Tensorflow (opens in a new tab)
  • PyTorch (opens in a new tab)
• Ray Tune
  • HyperOpt (opens in a new tab)
  • Optuna (opens in a new tab)
• Ray Serve
  • FastAPI (opens in a new tab)
  • gradio (opens in a new tab)

Ray also provides an easy way to "glue" and integrate distributed libraries, with code that retains the look and feel of standard Python.

Shifting the focus towards libraries means users can ignore the complexities of distributed computation and, instead, zero in on the algorithmic aspect. Drawing parallels, this reminds us of the third generation GPU architectures when tools like Unity or Unreal Engine 4 became popular. In these situations, the intricacies of the GPU were de-emphasized in favor of these engines' programmability and versatility.