Snowflake Research

Framework Integrations

Arctic RLAgent View (Markdown)

Overview

Arctic RL integrates into existing RL frameworks rather than replacing them, with completed integrations into SkyRL and an in-progress integration into Verl, plus additional framework integrations planned for TRL, Axolotl, unsloth, and PrimeRL.

Framework Integrations

Arctic RL is designed to integrate into existing RL frameworks rather than replace them. The RL framework retains ownership of the training loop, rollouts, rewards, and advantage estimation, while Arctic Platform provides the heavy compute engines through a thin client interface.

Integration Architecture

Arctic Platform provides three core compute engines orchestrated over Ray:

  • Training engine — a DeepSpeed engine that runs forward/backward passes and optimizer steps
  • Log-prob / reference engine — a forward-only DeepSpeed engine for reference and old log-prob computation
  • Sampling engine — a vLLM engine with ArcticInference for fast rollouts

These engines can be colocated on shared GPUs (via fractional Ray resources) or split across separate GPUs. They keep the sampler in sync with the trainer through NCCL or CUDA-IPC weight transfer. The RL framework interacts with the compute engines over Ray or HTTP.

An RL framework integrates this module by constructing a client and driving standard operations: generate, forward/backward, optimizer step, sync_weights, and wake/sleep for memory management.

Client Configuration Example

from arctic_platform.rl import ArcticRLClientConfig, create_arctic_rl_client

config = ArcticRLClientConfig(
    model_name="Qwen/Qwen3-4B",
    comm_protocol="ray",        # or "http"
    training_gpus=8,
    sampling_gpus=8,
    log_prob_gpus=0,
    colocate=True,
)
client = create_arctic_rl_client(config)

Current Integrations

SkyRL (Integrated)

Arctic RL is fully integrated into SkyRL.

Verl (In Progress)

Arctic RL integration into Verl is complete and drives the Verl PPO/GRPO trainer through Arctic RL.

Upcoming Integrations

Multiple framework integrations are in active development and will be documented here as they become available:

  • TRL
  • Axolotl
  • unsloth
  • PrimeRL
  • Additional frameworks under consideration

Integration Benefits

By integrating Arctic RL into existing frameworks, you gain:

  • Unified GPU backends for both on-prem and remote execution
  • Unified system optimizations providing high-performance portability across frameworks
  • Composable, modular architecture that keeps your training loop intact
  • Transparent optimizations like ZoRRo Train (prompt deduplication) and ZoRRo Inference (Forest Cascade Attention)
  • Flexible resource configuration with per-engine GPU allocation and optional colocation