LLM Introductory Courses

Introductory Courses

Covers NLP fundamentals and includes a miniGPT project.

Official site: CS224N Stanford
Video: (2025, bilingual) Stanford CS224N — Deep Learning for Natural Language Processing
Slides and assignments: Quark Cloud Drive

Course homepage: CMU Advanced NLP Spring 2025
Code: GitHub Repo
Assignments:
- minLLaMA: HW1 (implement LLaMA from scratch — extremely high quality)
- RAG: HW2

Code: NanoGPT GitHub
Learning materials: [LLM Training Series] NanoGPT source code walkthrough and Chinese GPT training practice

Assignment 1: Implement a BPE tokenizer, Transformer architecture, and Adam optimizer; train on TinyStories and OpenWebText (PyTorch primitives only).
Assignment 2: Implement Flash Attention 2 in Triton; distributed data parallelism + optimizer sharding.
Assignment 3: Scaling Laws. Fit scaling laws using IsoFLOP, simulating experiments under a fixed compute budget.
Assignment 4: Data pipeline. Convert Common Crawl HTML to text, filter (quality, harmful content, PII), deduplicate.
Assignment 5: Alignment. Implement supervised fine-tuning, expert iteration, GRPO and variants; run RL on Qwen 2.5 Math 1.5B to improve MATH benchmark performance.