Simple Single-GPU GRPO (GSM8K)

Simple Single-GPU GRPO (GSM8K)

Overview

This recipe provides the smallest end-to-end Arctic RL loop: GRPO training for Qwen3-1.7B on GSM8K on a single GPU, powered by Arctic RL with the ZoRRo trainer.

Key features:

• Pure GRPO without a frozen reference model (no KL anchoring)
• Single node, single GPU — no Ray cluster or hostfile required
• Deepspeed ZeRO stage-2, vLLM rollout (TP=1)
• Built-in GSM8K reward (exact match on #### <number> final answer)
• Colocated training + sampling on one GPU

Architecture

Topology:

• 1 node, 1 GPU
• colocate=True (training and sampling share the GPU)
• Deepspeed ZeRO stage-2
• vLLM rollout with tensor parallelism (TP=1)
• No frozen reference model, so log-prob pool is disabled (log_prob_gpus=0)
• Log-probs are recomputed through the training engine via ZoRRo

Reward: GSM8K is scored by verl's built-in reward function using data_source="openai/gsm8k" for exact match on the gold #### <number> final answer.

Installation

1. Create and activate a fresh conda environment

conda create -y -n simple python=3.12
conda activate simple
pip install uv

2. Clone repositories

git clone https://github.com/Snowflake-AI-Research/Arctic-Platform
git clone -b arctic_rl_share_v0.7.1 --single-branch https://github.com/Snowflake-AI-Research/verl
cd Arctic-Platform/recipes/rl/verl/simple

3. Install pinned dependencies

The recipe assumes CUDA 12.9. If you use a different version, update the torch index URL and the cuda-bindings pin in requirements.txt.

# Install torch (CUDA 12.9) first
uv pip install torch==2.10.0 --index-url https://download.pytorch.org/whl/cu129 -U

# Install remaining dependencies with overrides to lock transitive deps
uv pip install -r requirements.txt --override overrides.txt

# Prepare to build flash-attn
uv pip install -U pip wheel packaging setuptools

4. Install Flash Attention

Option A: Prebuilt wheel (recommended)

uv pip install flash-attn --no-build-isolation

Option B: Download prebuilt FA2 wheel Download from flash-attention releases.

Option C: Use Flash Attention 3 (faster on Hopper)

• Download a prebuilt FA3 wheel from flash-attention3-wheels
• Install it: uv pip install <fa3_wheel>
• In the launcher, comment out flash_attention_v=flash_attention_2 and uncomment the GPU-type auto-selection block

5. Install verl (Snowflake fork)

cd ../../../../../verl
grep -v flash-attn requirements.txt > requirements-no-fa.txt
uv pip install -r requirements-no-fa.txt
uv pip install -e .
cd -

Data Preparation

Run the data download script from the recipe directory:

python download_data.py --output_dir ~/data/gsm8k

This pulls GSM8K from HuggingFace and writes verl-compatible parquets with gold #### <number> answers as reward ground truth.

Output structure:

~/data/gsm8k/
├── train.parquet      # ~7.5k rows
└── test.parquet       # ~1.3k rows

Training

The launcher requires no Ray cluster or hostfile—just execute it. It defaults to a single GPU and reads the parquets from step 2.

Basic launch

bash run_qwen3_1.7b_gsm8k_grpo_arl.sh \
    data.train_files=~/data/gsm8k/train.parquet \
    data.val_files=~/data/gsm8k/test.parquet

Or, if using the default data path (~/data/gsm8k):

bash run_qwen3_1.7b_gsm8k_grpo_arl.sh

Environment variables

Edit these at the top of run_qwen3_1.7b_gsm8k_grpo_arl.sh:

Key recipe knobs

Set these inside run_qwen3_1.7b_gsm8k_grpo_arl.sh: