参考

• CS336: Language Modeling from Scratch, Spring 2026

• GPU 高性能编程

• How to Scale Your Model：JAX Scaling Book

• Transformer training memory：Transformer memory usage overview

• Transformer FLOPs：Transformer FLOPs accounting overview

• What shapes do matrix multiplications like?

• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning

• Triton fused softmax tutorial

• PyTorch benchmarking recipe

• PyTorch profiler recipe

• PyTorch Distributed documentation

• NCCL tests: collective performance metrics

• NVIDIA NCCL technical talk

• UvA DL Notebooks: Profiling and Scaling Single-GPU Transformer Models

• JAX Profiling 文档

• JAX checkpoint/remat 文档

• UvA DL Notebooks: Part 2.2: (Fully-Sharded) Data Parallelism

• ZeRO: Memory Optimizations Toward Training Trillion Parameter Models

• JAX shard_map 文档

• Flax nn.Partitioned 文档

• UvA DL Notebooks: Tensor Parallelism

• Megatron-LM

• GSPMD

• JAX shard_map 文档

• UVA Deep Learning Course：Part 3.1 Pipeline Parallelism

• 完整 JAX/Flax Notebook

• GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism

• UVA Deep Learning Course：Part 3.2 Looping Pipelines

• Megatron-LM：Interleaved Pipeline Parallelism

• Breadth-First Pipeline Parallelism

• Megatron Core：Mixture of Experts

• DeepEP

• MegaBlocks: Efficient Sparse Training with Mixture-of-Experts

• Switch Transformers

• DeepSeek-V3 Technical Report

• How to Scale Your Model：Inference

• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints

• DeepSeek-V2：Multi-head Latent Attention

• Reducing Transformer Key-Value Cache Size with Cross-Layer Attention

• AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration

• Minitron：Pruning and Knowledge Distillation

• Fast Inference from Transformers via Speculative Decoding

• Accelerating Large Language Model Decoding with Speculative Sampling

• Orca：Iteration-level Scheduling 与 Selective Batching

• Efficient Memory Management for Large Language Model Serving with PagedAttention

• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning

• Online normalizer calculation for softmax

• Making Deep Learning Go Brrrr From First Principles

• ZeRO: Memory Optimizations Toward Training Trillion Parameter Models

• How to Scale Your Model

• The Ultra-Scale Playbook

• Lightning: Distributed Communication

• Stanford CS336 Spring 2026, Lecture 12: Evaluation

• MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark

• GPQA: A Graduate-Level Google-Proof Q&A Benchmark

• Humanity’s Last Exam

• Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference

• Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators

• WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild

• SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

• Establishing Best Practices for Building Rigorous Agentic Benchmarks

• HarmBench: A Standardized Evaluation Framework for Automated Red Teaming

• AIR-Bench 2024: A Safety Benchmark Based on Risk Categories from Regulations and Policies

• GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks

• MedHELM: Holistic Evaluation of Large Language Models for Medical Tasks

• Clio: Privacy-Preserving Insights into Real-World AI Use

• Proving Test Set Contamination in Black Box Language Models

• Do Large Language Model Benchmarks Test Reliability?

• Bootstrap Methods: Another Look at the Jackknife

• HELM

• Consent in Crisis: The Rapid Decline of the AI Data Commons

• The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text

• A Survey on Data Selection for Language Models

• DataComp-LM: In Search of the Next Generation of Training Sets for Language Models

• OpenWebMath: An Open Dataset of High-Quality Mathematical Web Text

• Deduplicating Training Data Makes Language Models Better

• Mining of Massive Datasets: Finding Similar Items

• UniMax: Fairer and More Effective Language Sampling for Large-Scale Multilingual Pretraining

• RegMix: Data Mixture as Regression for Language Model Pre-training

• OpenThoughts: Data Recipes for Reasoning Models

• SWE-smith: Scaling Data for Software Engineering Agents

• From SWE-ZERO to SWE-HERO: Execution-free to Execution-based Fine-tuning for Software Engineering Agents

• SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents

• Training Language Models to Follow Instructions with Human Feedback

• Finetuned Language Models Are Zero-Shot Learners (FLAN)

• Self-Instruct: Aligning Language Models with Self-Generated Instructions

• Alpaca: A Strong, Replicable Instruction-Following Model

• LIMA: Less Is More for Alignment

• How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources

• Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?

• Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions

• MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies

• JetMoE: Reaching Llama2 Performance with 0.1M Dollars

• Stanford CS336 Spring 2026, Lecture 15: After Pretraining

• Training Language Models to Follow Instructions with Human Feedback

• Learning to Summarize from Human Feedback

• Benchmarking Large Language Models for News Summarization

• Human Feedback is not Gold Standard

• Whose Opinions Do Language Models Reflect?

• UltraFeedback: Boosting Language Models with Scaled AI Feedback

• Constitutional AI: Harmlessness from AI Feedback

• Proximal Policy Optimization Algorithms

• Direct Preference Optimization: Your Language Model is Secretly a Reward Model

• Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback

• SimPO: Simple Preference Optimization with a Reference-Free Reward

• Tülu 3: Pushing Frontiers in Open Language Model Post-Training

• Scaling Laws for Reward Model Overoptimization

• AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback

• A Long Way to Go: Investigating Length Correlations in RLHF

• GPT-4 Technical Report

• Stanford CS336 Spring 2026, Lecture 16: Reinforcement Learning from Verifiable Rewards

• Proximal Policy Optimization Algorithms

• High-Dimensional Continuous Control Using Generalized Advantage Estimation

• DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

• Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs

• Understanding R1-Zero-Like Training: A Critical Perspective

• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

• Kimi k1.5: Scaling Reinforcement Learning with LLMs

• Qwen3 Technical Report

• Qwen3-Coder-Next Technical Report

• Learning Transferable Visual Models From Natural Language Supervision（CLIP）

• Sigmoid Loss for Language Image Pre-Training（SigLIP）

• An Image is Worth 16×16 Words（Vision Transformer）

• Visual Instruction Tuning（LLaVA）

• LLaVA-OneVision: Easy Visual Task Transfer

• Qwen-VL: A Frontier Large Vision-Language Model with Versatile Abilities

• Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution

• Qwen3-VL Technical Report

• Chameleon: Mixed-Modal Early-Fusion Foundation Models

• Neural Discrete Representation Learning（VQ-VAE）