Reve says its training infrastructure has reached ten-thousand-GPU scale while supporting Reve 2.0, Reve 2.1 and earlier multimodal models. Its new technical report explains the machinery behind those runs: Google Kubernetes Engine, KubeRay, a researcher-facing command-line tool, cross-region training and several layers of failure recovery.
The primary source is Training Reve 2.0 at scale, written by Joe Wang with contributions from Kyra Mo and Ben Cuan. The page carries no visible publication date, so the supplied October 28, 2026 announcement date should be treated as an approximate release window. Reve has not published a complete reference implementation alongside the report.
The most reusable lesson is how the pieces divide responsibility. Kubernetes manages infrastructure resources; Ray coordinates distributed work; training code manages synchronization and checkpoints; separate recovery systems handle failed replicas, unhealthy nodes and diagnosis. Replacing a container, repairing a GPU host and preserving a valid training state are different operations, and the report keeps them distinct.
GKE and KubeRay Divide the Orchestration Work
Reve runs its platform on Google Cloud, using Kubernetes as the compute platform and Ray for distributed training. Its internal command-line tool, GKR, stands for Google Kubernetes Engine Runner.
The report’s architecture assigns distinct jobs to the main components:
| Component | Responsibility |
|---|---|
| GKE and Kubernetes | Place and run the underlying compute workloads |
| KubeRay | Manage the lifecycle of Ray jobs and clusters |
| Ray Train and PyTorch | Coordinate and execute distributed training |
| GKR | Provide researchers with submission, inspection and management commands |
| ArgoCD, Terraform and Kustomize | Configure the underlying infrastructure and platform |
| Warden | Detect and resolve node faults |
A submitted RayJob creates a temporary RayCluster. Kubernetes creates its head and worker pods, while the Ray head coordinates distributed work. Researchers can also request a persistent RayCluster, which requires explicit deletion, or an ordinary Kubernetes Job for tasks that do not need Ray.
This boundary is worth preserving when adapting the design. Kubernetes can successfully keep pods running while a distributed training application is stuck. Application-aware coordination still has to sit above the container scheduler.
GKR Makes Resource Requests Explicit
GKR accepts command-line arguments or YAML, resolves defaults and generates Kubernetes resources through Jinja templates. It translates researcher requests into resource requests, selectors, tolerations and affinity rules.
Those Kubernetes controls determine which machines are eligible, how workloads fit and which scheduling restrictions apply. Researchers get a smaller interface without having to maintain those manifests themselves.
One detail in Reve’s example deserves attention: GPU, CPU and memory requests are per pod. Its example specifies two pods, each requesting eight H200 GPUs, 80 CPUs and 512 GiB of memory.
The resulting job requests are therefore:
- 16 GPUs: two pods multiplied by eight GPUs.
- 160 CPU units: two multiplied by 80.
- 1,024 GiB of memory: two multiplied by 512 GiB, equivalent to 1 TiB.
These are requested resources, not measured utilization. The arithmetic is simple, but making the scope explicit prevents an expensive class of configuration mistakes.
Reve also separates infrastructure configuration from application configuration. A standardized training driver uses Hydra to combine reusable defaults, YAML settings and command-line overrides. The report gives this example:
python train.py path/to/config.yaml run.name=example
The command selects a training configuration and overrides the run name. It illustrates the configuration interface; it is not a standalone reproduction recipe for Reve’s models.
Preflight Checks Run Against the Allocated Machines
GKR validates names, argument combinations and rendered manifests before applying resources. After Kubernetes places the pods, it runs checks against the actual allocation.
Those checks inspect kernel logs and command output for hardware, memory and driver errors. They also test computation and collective operations, then verify shared mounts, libraries and transport settings, according to the report.
The timing matters. Manifest validation can catch a malformed request, but it cannot establish that the assigned machines can communicate correctly. A collective-operation test exercises part of the path that distributed training will depend on.
I’d copy this boundary before copying the entire CLI. A consistent launch interface helps researchers, but testing the allocated hardware before a long run protects the compute budget.
Cross-Region Training Changes the Synchronization Boundary
Reve describes cross-region local learners using DiLoCo-style synchronization. The basic idea is to perform training work locally between synchronization events, reducing the frequency of communication across regions.
That changes where network performance becomes a constraint:
| Training pattern | Communication behavior | Practical consequence |
|---|---|---|
| Conventional synchronous data parallelism | Replicas synchronize gradients each step | Frequent synchronization makes network latency and bandwidth important |
| DiLoCo-style local learners | Learners perform local updates between synchronization events | Less frequent cross-region communication, with additional optimization and state-management considerations |
This is a conceptual comparison, not a published head-to-head benchmark of Reve’s implementation. The supplied evidence does not provide a matched throughput or model-quality comparison.
The engineering tradeoff is consequential. Regions can offer separate pools of available GPUs, but treating distant machines as one tightly synchronized cluster can make every training step wait on a long-distance network path. Local learners move more work inside each local group and reduce how often groups must coordinate.
That still leaves algorithmic decisions: synchronization frequency, handling diverging local states and recovery when a learner disappears. The public Reve torchft fork explicitly labels LocalSGD and DiLoCo experimental. A team adopting the pattern should budget for training-algorithm validation alongside infrastructure integration.
My view is that the report’s cross-region design is most useful as a way to rethink the synchronization boundary. It is not evidence that any existing synchronous training job can be spread across regions unchanged.
Staged Checkpoints Separate Progress From Durability
Reve’s report describes staged checkpoints using PyTorch Distributed Checkpoint. Checkpoints preserve the state needed to resume training, making them the fallback when live continuation is unavailable.
The useful engineering distinction is between capturing a consistent training state and making that state durably recoverable. A checkpoint pipeline can have work remaining after its initial staging step. Engineers adopting this pattern need to identify which completion signal means the checkpoint is safe to resume from.
That distinction affects recovery policy:
- A recently staged snapshot may offer less protection than a completed durable checkpoint.
- A restart must select a consistent recovery point.
- Checkpoint frequency must balance saving work against the overhead of capturing and storing state.
Reve’s report places checkpointing alongside live fault tolerance. That combination is more informative than either feature alone: live continuation can avoid some restarts, while durable checkpoints protect progress when continuation fails.
A checkpoint system and a fault-tolerant training loop solve overlapping problems, but neither makes the other unnecessary.
torchft Keeps Healthy Replicas Working
Reve uses torchft for live fault tolerance, including auto-shrinking after failures. The goal is to let healthy training replicas continue while failed capacity drops out of the active group.
The torchft repository describes three relevant building blocks:
- Per-step heartbeating to determine which workers are healthy.
- Fault-tolerant process groups that report errors and can be reinitialized.
- Checkpoint transports that support recovery from a healthy peer when scaling up.
A lighthouse server coordinates replica groups, with a manager associated with each group. Membership changes can occur at training-step granularity.
Frequently Asked Questions
3 questions
1What Is Reve’s Training Infrastructure Built On?
Reve uses GKE, KubeRay, Ray Train and PyTorch, with an internal CLI called GKR. ArgoCD, Terraform and Kustomize manage platform configuration.
