LLM from Scratch
中文 GitHub
Repository Map

Docs

Module-based project documentation rendered inside the site from mirrored English and Chinese Markdown chapters.

Tokenizer / Transformer / Training / Scale / Alignment

This docs section is a system map for the repository rather than a beginner tutorial.

Each chapter is written from the code outward:

  • ownership boundaries between modules
  • the concrete control flow through functions and classes
  • the tensors or records that move across those boundaries
  • the current implementation limits that matter when you run or extend the code

The chapters are grouped into three layers:

  • the core stack that turns text into a decoder-only model
  • the scale-and-performance layer that covers kernels, distributed execution, and preprocessing
  • the alignment workflows that fine-tune and evaluate a math model on gsm8k

Two reading paths work well:

  • start with Project Overview if you want the whole repository topology first
  • start with Tokenizer and Vocabulary if you want to follow the actual data path from raw text to training batches

Use the overview cards below to jump directly to the subsystem you want to inspect.

Core Stack

Core Stack

The baseline path from raw text to a trainable decoder-only model.

Core Stack

Project Overview

A system map of the repository, from tokenizer and transformer internals to kernels, distributed training, and alignment workflows.

README.mdllm/transformer.pyllm/bpe_tokenizer.py
Core Stack

Tokenizer and Vocabulary

How the repository trains and applies a byte-pair tokenizer, from regex pre-tokenization to merge learning and byte-level encode/decode.

llm/bpe_tokenizer.pyllm/args.py
Core Stack

Transformer Core

Decoder-only transformer internals implemented in llm/transformer.py, including RMSNorm, RoPE, SwiGLU, custom attention, and custom loss.

llm/transformer.py
Core Stack

Training Loop and Checkpointing

How llm/training.py slices data, initializes distributed execution, runs validation, applies schedules, and saves checkpoints.

llm/training.pyllm/checkpoint.pyllm/generating.py
Scale & Performance

Scale & Performance

The layers that make larger runs practical: kernels, distributed execution, and data preparation.

Scale & Performance

Flash Attention and Kernel Optimization

How the Triton Flash Attention path in kernel/ reduces attention memory pressure and how it is validated against a reference implementation.

kernel/flash_attention_triton.pykernel/flash_attention_mock.pybench_mark/bench_mark_flash_attention.py
Scale & Performance

Distributed Training and Sharded Optimizer

How parallel/ddp.py and parallel/sharded_optimizer.py synchronize gradients, bucket communication, and reduce optimizer-state memory.

parallel/ddp.pyparallel/sharded_optimizer.pyllm/training.py
Scale & Performance

Data Processing Pipeline

The preprocessing stack in data_processing/, from HTML extraction and language filtering to deduplication, PII masking, harmful-content checks, and quality classification.

data_processing/html_process.pydata_processing/language_identification.pydata_processing/quality_filter.py
Alignment Workflows

Alignment Workflows

Concrete SFT and RLFT implementations built around gsm8k and Qwen2.5-Math-1.5B.

Alignment Workflows

Supervised Fine-Tuning on gsm8k

How alignment/sft.py turns gsm8k examples into prompt-completion training data and measures accuracy plus format compliance.

alignment/sft.pyalignment/dataset.pyalignment/r1_prompt.py
Alignment Workflows

Reinforcement Learning Fine-Tuning on gsm8k

How the repo implements reward shaping, grouped normalization, GRPO-style clipping, and multi-GPU role separation for RLFT.

alignment/train_rl.pyalignment/grpo.pyalignment/drgrpo_grader.py