Arctic RL Overview

Arctic RL Overview

What is Arctic RL?

Arctic RL is a compute engine designed to integrate into existing RL frameworks rather than replace them. The framework retains control over the training loop, rollouts, rewards, and advantage estimation, while Arctic RL provides the heavy compute engines behind a clean client interface.

Arctic RL addresses key challenges in current post-training frameworks by offering:

• Unified GPU Backends — consistent building blocks for on-prem or remote deployments
• Unified System Optimizations — high performance portability across different frameworks
• Modular Components — training, sampling, and log-prob engines that can colocate on shared GPUs or split across separate ones

Core Compute Engines

Arctic RL provides three specialized compute engines orchestrated over Ray:

Training Engine

A DeepSpeed engine that handles forward and backward passes, plus optimizer updates. This is the primary compute backbone for gradient-based learning.

Log-prob / Reference Engine

A forward-only DeepSpeed engine for computing log probabilities and reference model evaluations. Can use either vLLM or DeepSpeed as the backend.

Sampling Engine

A vLLM engine with ArcticInference for fast rollouts and token generation. Includes multi-replica scheduling, load-balancing, weight synchronization, and router-replay capabilities.

Quick Start Example

Initialize Arctic RL in your training loop with a simple configuration:

from arctic_platform.rl import ArcticRLClientConfig, create_arctic_rl_client

config = ArcticRLClientConfig(
    model_name="Qwen/Qwen3-4B",
    comm_protocol="ray",        # or "http"
    training_gpus=8,
    sampling_gpus=8,
    log_prob_gpus=0,
    colocate=True,
)
client = create_arctic_rl_client(config)

The RL framework integrates this module by constructing a client and driving standard operations:

• generate — rollout token generation
• forward/backward — gradient computation
• optimizer_step — weight updates
• sync_weights — synchronize weights across engines
• wake/sleep — memory management

Key Configuration Options

Core configuration is managed through ArcticRLClientConfig:

Advanced Configuration

For fine-grained control, pass additional configuration dictionaries:

config = ArcticRLClientConfig(
    model_name="Qwen/Qwen3-4B",
    training_gpus=8,
    sampling_gpus=8,
    training_config={
        "optimizer": {"lr": 1e-5, "weight_decay": 0.01},
        "dtype": "bfloat16",
        "gradient_checkpointing": True,
    },
    vllm_config={
        "max_seq_len": 4096,
        "tensor_parallel_size": 4,
    },
    arctic_inference_config={
        "use_fca": True,
    },
)

Framework Integrations

Arctic RL integrates with existing RL frameworks to power their training and inference:

Currently Integrated

• SkyRL — Integration merged with documentation

Integration in Progress

• Verl — Reference integration driving Arctic RL from PPO/GRPO trainers, with end-to-end recipes available under recipes/rl/verl/
  • Simple single-GPU GSM8K starter
  • Txt2SQL recipe
  • Long-context QA recipe

Upcoming Integrations

Multiple additional framework integrations are in development, including TRL, Axolotl, unsloth, PrimeRL, and others.

Performance Optimizations

ZoRRo Train (Prompt Deduplication)

During RL training (PPO/GRPO), the same prompt is sampled many times to explore different responses. For long-sequence tasks, 80–95% of tokens in a batch are redundant prompt tokens. With transformer attention's O(n²) cost, recomputing shared prompts dominates compute time and memory usage.

ZoRRo Train eliminates this waste through automatic prompt deduplication at all levels:

1. Detects sequences sharing a prompt
2. Packs each unique prompt once
3. Runs the model once over deduplicated sequences
4. Transparently reconstructs per-response logprobs and entropy

The result is mathematically equivalent to naive forward/backward (gradients match baseline within numerical precision) while substantially reducing memory usage and increasing throughput. The longer and more-shared the prompts, the larger the benefit.

Enable via the RL config:

config = ArcticRLClientConfig(
    model_name="Qwen/Qwen3-4B",
    training_gpus=8,
    training_config={"zorro_train": {"enable": True}},
)

Supported model families include both dense and MoE models. See arctic_platform/rl/zorro_train/README.md for the full list.

ZoRRo Inference (Forest Cascade Attention)

During RL rollouts, many sequences are generated from the same prompt. In the decode step, standard attention re-reads the KV cache of shared prefixes once per request, causing the sampler to spend most memory bandwidth fetching identical keys and values repeatedly.

ZoRRo Inference removes this waste with Forest Cascade Attention (FCA), which deduplicates shared KV reads at the attention layer:

1. Discovers groups of requests sharing a KV-cache prefix
2. Splits each attention call into a grouped pass over shared prefix blocks plus per-request pass over unique suffix blocks
3. Reduces two partial results with rigorous weighting

FCA reads each shared prefix block once per group instead of once per request, cutting redundant memory accesses while remaining mathematically equivalent to standard attention.

FCA is implemented in the vLLM sampling engine and activates transparently for decode-heavy batches with shared prefixes.

For more details, see the Forest Cascade Attention README in Arctic Inference.

Communication Protocols

Arctic RL supports two communication protocols:

HTTP Protocol

• Default option
• Binds the RL server on the primary node's routable IP at port 7000
• Allows off-node Ray workers to reach the driver node by IP
• Use with comm_protocol="http"

Ray Protocol

• Native Ray communication
• Ideal for fully distributed Ray clusters
• Use with comm_protocol="ray"

The host and port are automatically derived from the communication protocol unless explicitly overridden.

Deployment Backends

Local Backend

Launches a fresh server instance on the current node with Ray cluster orchestration. Requires at least one of training_gpus, sampling_gpus, or log_prob_gpus to be greater than zero.

DSS Platform Backend

Connects to a remote dss-platform deployment for managed distributed training.

Health and Initialization

Arctic RL includes built-in health checks and timeout management:

Reconnection

Arctic RL supports reconnecting to pre-existing jobs without reinitializing. Populate the training_job_id, sampling_job_id, and log_prob_job_id fields in the configuration to reconnect:

config = ArcticRLClientConfig.reconnect_config()
client = create_arctic_rl_client(config)

Next Steps

To get started with Arctic RL training, see the recipes for hands-on examples with supported frameworks.