Tokenizer
A from-scratch BPE tokenizer that learns merges and special tokens from raw text.
Implemented System
Architecture
See how tokenizer, transformer blocks, training, kernels, and distributed execution fit together.
Transformer Core
Decoder-only blocks with RMSNorm, RoPE, SwiGLU, custom attention, and custom loss.
Training Loop
Checkpointing, validation, generation, cosine scheduling, and optimizer utilities.
Parallel + Kernel
Custom DDP, sharded optimizer, and Triton Flash Attention 2 support.
Data Pipeline
HTML extraction, filtering, deduplication, masking, and quality tooling.