Reinforcement learning (RL) post-training for large language models (LLMs) coordinates multiple models across generation, inference, and training on GPU clusters. Several factors may change during a run, including resource availability, sequence length, memory pressure, and stage bottlenecks. As a consequence, an execution plan that was initially suitable can then become slow or even infeasible over time. However, adapting a job whose models share GPUs entails significant challenges: deciding whether a new plan is worth the transition cost, reusing the job’s distributed state, and coordinating GPU transfers across models and stages. Nereus targets these challenges as a cost-aware runtime that adapts RL post-training jobs into efficient execution plans. Its low-overhead controller selects a memory-feasible global plan and admits the transition using a cost model calibrated against the running job. To estimate and execute a transition, Nereus represents the distributed state of each replica of a model-stage (one model in one stage) as an Elastic Model Unit. It then employs a global transition graph to order the transformations and GPU transfers of these units. In a trace built from real data, online TP/PP adaptation reduces average step latency by 27.7% relative to the initial fixed TP/PP layout with DP scaling. In a 1,000-step run reaching 1,024 GPUs, six transitions consume 0.079% of total run time. Nereus improves end-to-end 8B PPO throughput by 2.14-7.27× over OpenRLHF and by 1.10-1.47× over Verl across diverse clusters.

Long-context training hits a wall: standard attention costs O(S²), so doubling the sequence length quadruples the attention work. DeepSeek Sparse Attention (DSA), used by GLM-5.2, breaks that curve by using a lightweight indexer that selects the top-k most relevant key/value tokens per query, and the main attention runs only over that top-k set, turning the dominant term near-linear. But a sparse-attention layer is only as good as its kernels: a naive implementation materializes several S × S intermediates that are memory-prohibitive past 32K tokens. NVIDIA closes that gap with fused DSA kernels in NVIDIA cuDNN, wired through cuDNN frontend, Megatron Core, and NeMo Megatron Bridge with support for context parallelism (CP), packed sequences (THD), and GLM-5.2’s new IndexShare. On GLM-5.2 743B training, the cuDNN backend delivers ~3.4× higher throughput than the TileLang reference on NVIDIA GB200 (~2× on NVIDIA H100), at an equivalent memory footprint.

Training-inference mismatch (TIM) is the silent failure mode for MoE: the training path and the inference path diverge, gradients stop matching the policy you deploy, and optimization drifts. In this blog post, we walk through a real case where TIM is much worse on a DeepSeekV3-architecture model (Moonlight-16B-A3B) than on Qwen3 MoE models (Qwen3-30B-A3B). We investigate why common mitigations failed, and how we fixed Router Replay R3 across vLLM and veRL to remove the mismatch without discarding samples.

Macaron-V1 is an open agent-model family for experiential intelligence: learning from experience in real environments and continuing to learn after deployment. It is organized around two system goals. Adaptation is pursued through recursive improvement of versioned model-harness pairs, where experience from one configuration is evaluated under an external contract and used to construct its successor. Collaboration is pursued via the Mixture-of-LoRA (MoL) architecture that freezes a base model, composes specialist LoRA adapters, and selects one LoRA per user turn. The flagship Macaron-V1-Venti (748B) combines a 744B GLM-5.2 base with four LoRAs for chat, agent, coding, and GenUI; the Qwen3.6-35B-based Macaron-V1-Tall (50B) uses the same design for local deployment. This report presents Macaron-V1 as a co-designed system spanning architecture, algorithms, and infrastructure. The MoL architecture supports continual learning through extensible LoRA specialists. The algorithm combines Model-Harness Co-design and recursive self-improvement loop, including the UI4A component-native GenUI harness, a stateful action substrate, versioned Harness Context Protocol contract, and the agentic RL framework MindForge. The supporting infrastructure includes the post-training platform MinT, the long-context RL method LongStraw, and stability techniques for sparse MoE and DSA base models. We evaluate Macaron-V1 on Personal Intelligence, GenUI, and general capability benchmarks against frontier baselines. Our results validate the current system, while compounding gains from continual learning and collective intelligence remain open questions.

Long-context RL post-training is constrained by the lifetime of state and gradients, not attention cost alone. In GRPO, one multi-million-token prompt must serve old-policy and reference scoring plus multiple policy responses, while conventional autograd keeps the prompt graph and all response graphs live alongside model weights, caches, and distributed communication buffers. We present LongStraw, an objective-aware, architecture-aware system for resident-state virtualization, response replay, and distributed-gradient execution. Its transaction captures the shared prompt without autograd, retains only the architecture-required state on explicitly owned pages, restores that state for each group member, scores old/reference branches without a graph, replays one policy response at a time with autograd, and accumulates the resulting gradients before one distributed finalization and optimizer step. This schedule bounds the live training graph by the response suffix while reusing the expensive prompt computation across the complete GRPO group. We instantiate this design for two incompatible model structures. Qwen3.6-27B combines 48 recurrent GDN layers with 16 full-attention layers; LongStraw keeps the compact recurrent state and physically CP8-sharded KV pages, composes global attention through cross-rank LSE/output merging, and performs blockwise response replay. GLM-5.2 combines a 78-layer MLA/DSA attention stack with a 256-expert, top-8 MoE tail. Its implementation keeps CP-sharded MLA latent pages and DSA indexer-key pages in CPU memory, stages one layer at a time, reconstructs IndexShare-aware global sparse selection over CP32, and dispatches routed response tokens over EP32. The two paths share one transaction contract while specializing the retained state, replay operator, and collective communication to the architecture.

Parameter-efficient fine-tuning (PEFT) is usually treated as a cheaper alternative to full fine-tuning. We study a broader role: small trainable adapters as persistent local state on top of strong shared foundation models. In this framing, the base model provides shared competence while adapters carry instance-specific behavior such as preferences, skills, tool habits, and memory-like updates. We organize the problem around three scaling axes: Scale Up, where stronger shared priors make small local updates more useful; Scale Down, where we study how small adapters can be while remaining reliable; and Scale Out, where many persistent adapted instances coexist. MinT provides one infrastructure example for managing adapter identity, revision, provenance, evaluation, and serving residency. Together, the results suggest that PEFT can be a compact substrate for persistent personal models rather than only a budget substitute for full fine-tuning.

We present MindLab Toolkit (MinT), a managed infrastructure system for Low-Rank Adaptation (LoRA) post-training and online serving. MinT targets a setting where many trained policies are produced over a small number of expensive base-model deployments. Instead of materializing each policy as a merged full checkpoint, MinT keeps the base model resident and moves exported LoRA adapter revisions through rollout, update, export, evaluation, serving, and rollback, hiding distributed training, serving, scheduling, and data movement behind a service interface. MinT scales this path along three axes. Scale Up extends LoRA RL to frontier-scale dense and MoE architectures, including MLA and DSA attention paths, with training and serving validated beyond 1T total parameters. Scale Down moves only the exported LoRA adapter, which can be under 1% of base-model size in rank-1 settings; adapter-only handoff reduces the measured step by 18.3x on a 4B dense model and 2.85x on a 30B MoE, while concurrent multi-policy GRPO shortens wall time by 1.77x and 1.45x without raising peak memory. Scale Out separates durable policy addressability from CPU/GPU working sets: a tensor-parallel deployment supports 10^6-scale addressable catalogs (measured single-engine sweeps through 100K) and thousand-adapter active waves at cluster scale, with cold loading treated as scheduled service work and packed MoE LoRA tensors improving live engine loading by 8.5-8.7x. MinT thus manages million-scale LoRA policy catalogs while training and serving selected adapter revisions over shared 1T-class base models.

Large Language Models (LLMs) fine-tuned via Reinforcement Learning from Human Feedback (RLHF) and Reinforcement Learning with Verifiable Rewards (RLVR) significantly improve the alignment of human-AI values and further raise the upper bound of AI capabilities, particularly in reasoning-intensive, long-context Chain-of-Thought (long-CoT) tasks. However, existing RLHF (or RLVR) frameworks commonly face challenges such as inference bottlenecks and complexity barriers, restricting their accessibility for newcomers. To bridge this gap, we introduce OpenRLHF, a user-friendly, scalable, and easy-to-learn open-source RLHF framework built upon Ray, vLLM, DeepSpeed, and HuggingFace Transformers, featuring a simplified design, clear code structure, and comprehensive documentation to facilitate entry for researchers and practitioners. Experimental results show that OpenRLHF achieves superior training efficiency with speedups ranging from 1.22texttimes to 1.68texttimes across different model sizes compared to state-of-the-art frameworks, while requiring significantly fewer lines of code for implementation. OpenRLHF is publicly available at https://github.com/OpenRLHF/OpenRLHF, and has already been adopted by leading institutions to accelerate RLHF research and learning.

As demand grows for training reasoning-capable large language models (LLMs), Reinforcement Learning from Human Feedback (RLHF) has emerged as a cornerstone technique. However, conventional RLHF pipelines—especially those using Proximal Policy Optimization (PPO)—are often hindered by substantial computational overhead. This challenge is particularly pronounced with models that excel at complex reasoning tasks (such as OpenAI-o1 and DeepSeek-R1), where generating long chain-of-thought (CoT) outputs can account for up to 90% of total training time. These models must produce detailed, step-by-step reasoning that can span thousands of tokens, making inference significantly more time-consuming than the training phase itself. As a pioneering inference framework, vLLM provides a user-friendly interface for generating RLHF samples and updating model weights.

We present what we believe is the first end-to-end Reinforcement Learning (RL) with Low-Rank Adaptor (LoRA) on a trillion-parameter reasoning model. Our system runs on large Mixture-of-Experts (MoE) models with 10% GPUs compared to conventional full-parameter RL. Our solutions have also been contributed to major open-source projects: NVIDIA’s Megatron-Bridge and Volcengine’s verl.

Designing and optimizing task-specific quantum circuits are crucial to leverage the advantage of quantum computing. Recent large language model (LLM)-based quantum circuit generation has emerged as a promising automatic solution. However, the fundamental challenges remain unaddressed: (i) parameterized quantum gates require precise numerical values for optimal performance, which also depend on multiple aspects, including the number of quantum gates, their parameters, and the layout/depth of the circuits. (ii) LLMs often generate low-quality or incorrect quantum circuits due to the lack of quantum domain-specific knowledge. We propose QUASAR, an agentic reinforcement learning (RL) framework for quantum circuits generation and optimization based on tool-augmented LLMs. To align the LLM with quantum-specific knowledge and improve the generated quantum circuits, QUASAR designs (i) a quantum circuit verification approach with external quantum simulators and (ii) a sophisticated hierarchical reward mechanism in RL training. Extensive evaluation shows improvements in both syntax and semantic performance of the generated quantum circuits. When augmenting a 4B LLM, QUASAR has achieved the validity of 99.31% in Pass@1 and 100% in Pass@10, outperforming industrial LLMs of GPT-4o, GPT-5 and DeepSeek-V3 and several supervised-fine-tuning (SFT)-only and RL-only baselines.

Hi! My name is Songlin Jiang, a junior undergraduate from Lanzhou University, China, majoring in Computer Science and Technology. It’s my first time participating in the Google Summer of Code with openSUSE Project mentors. In this blog, I’m going to introduce you to my work and experience so far.

In contemporary society, there are problems existed in the online voting systems such as repeated fraudulent voting, data tampering, loss of voting data, and failure to verify the outcome of the voting data, and these problems causes inconvenience to the users. Meanwhile, the mobile pay systems represented by Wechat pay are distinguished due to its safety and convenience. If we combine the voting system with Wechat pay, a huge difference will be made. My system is based on Wechat pay. When you want to vote, you just need to pay a little, then a refund will give to you to ensure that the voting is real. The system can double check the voting record by checking the trading record saved on the Wechat Sever. As a result, the system provides users with a safe and fair environment to vote and it also reduces the developing cost.

2024 - 2025

Travel Funding for Google Blockly Summit

2023 - 2024

Aalto University School of Science Dean's Incentive Scholarship

2022 - 2024

SECCLO Erasmus Mundus Scholarship

(26/767) Full-ride scholarship, includes full tuition fee waiver, covers all living costs and travel expenses

2024

Delegate of OpenSUSE Project for Google Summer of Code 2024 Mentor Summit

Joint funded by Google and openSUSE Travel Support Program

2024

Travel Fund for The Linux Foundation Open Source Summit Europe 2024

To speak at the LFX Mentorship Showcase

2024

Certified Kubernetes Security Specialist (CKS), CNCF

2023

Black Hat Europe 2023 Student Scholarship

2022

Cloud Captains in Linux Foundation Training (LiFT) Scholarship (China)

2022

Huawei Cloud Certified Developer Associate in Internet of Things (HCCDA - IoT)

2022

Certified Kubernetes Administrator (CKA), CNCF

2022

Recognized Tencent Open Source Contributor

Only 30+ were issued globally by the end of 2022

2022

Intern of Excellence Award at Tencent

Only 2/4000+ received all the 3 kinds of intern awards

2022

Intern of Excellence & Star of Open Source Award in openEuler Open Source Internship

Only 3/150+ interns received both awards

2021

China Ministry of Education & Huawei "Intelligent Base" Future Star Scholarship

2021

Software Design Engineer, China Qualification of Computer and Software Professional

2020

Academic Aces in Linux Foundation Training (LiFT) Scholarship (China)

2020

CATTI Level 3 (English & Chinese) Translator

2020

Best Paper Award in Lanzhou University Summer Social Practice Report

Paper: Research on the Functionality and Optimization of Urban Communities in China During COVID-19 in 2020

2020

Award in openEuler University Developer Competition

(24/547 Teams) I participated alone, ranked 1st in final defense

2019

Lanzhou University Outstanding Undergraduate International Exchange Scholarship

(Only 1) Full covering of visa fees, traveling fees, and living costs, plus the host university exchange program tuition fee waiver agreement

2019

Google InnoCamp Hangzhou BoostMakers - TensorFlow Boost

Only 40+/2000+ got this certificate

2019

Excellent Project Award in Lanzhou University Innovation and Entrepreneurship Plan

Only 122/1268 got this award

2018 - 2022

Nine Lanzhou University Undergraduate Scholarships

3 x scholarship for academic excellence, 5 x scholarship for innovation and entrepreneurship projects, 2020 scholarship for IELTS test fee