What question did this study set out to answer?

The aim is to evaluate how a critic-free reinforcement learning pipeline performs in a memory-constrained environment.

July 4, 2026Open Access

Memory-Constrained Reasoning: A native QLoRA engine for entropy-driven GRPO

EBEndika BlancoUniversity of the Basque Country

Key Points

The aim is to evaluate how a critic-free reinforcement learning pipeline performs in a memory-constrained environment.
Utilized a PyTorch engine for deep learning with a 4-bit frozen backbone and manual LoRA adapters.
Implemented explicit phase boundaries, VRAM-aware checkpointing, and phase-local out-of-memory recovery.
Conducted testing on an RTX 3060 Ti with a focus on 768-token profiling.
Demonstrated positive but statistically uncertain macro pass@1 deltas with LoRA adapters.
Paired bootstrap analysis indicated improved average sampled correctness with shorter responses.
Achieved successful operational functionality within the confines of an 8 GB GPU without out-of-memory errors.

Abstract

Reasoning post-training is usually reported as an objective plus a final score, but on small GPUs the training loop can decide what is possible. We test whether a critic-free Reinforcement Learning from Verifiable Rewards (RLVR) pipeline for compact mathematical reasoning can remain stable, observable, and recoverable inside a fixed 8 GB profile. We present a PyTorch-orchestrated engine for DeepSeek-R1-Distill-Qwen-1.5B with a 4-bit NF4 frozen backbone, manual LoRA adapters, explicit phase boundaries, VRAM-aware checkpointing, and phase-local OOM recovery. Here, direct orchestration means control of the training loop, not a pure-PyTorch kernel stack. The runtime exposes rollout generation, frozen-policy scoring, reward computation, adapter updates, cache release, checkpointing, and benchmarking. The learning path uses verifier-rewarded GRPO-family optimization with response caps, truncation handling, sequence-length diagnostics, token-level entropy masking, and semantic-entropy curriculum ordering. The systems work supports the ML question: how verifier-backed GRPO behaves when sparse reward, response length, and entropy-based selectivity interact in a compact SLM. The selected 768-token profile completes a 40-step diagnostic on an RTX 3060 Ti without OOM backoff. In prompt-matched evaluation, the LoRA adapters show positive but statistically uncertain macro pass@1 deltas. A paired bootstrap gives tighter evidence for average sampled correctness, shorter responses, and lower truncation. The result is an 8 GB GRPO artifact with measured prompt-matched behavioral change under a short local response budget.

Demander à l'IA

Bookmark

View Full Paper