Folder contents:
• Dockerfile & docker‑compose.yml → isolate your environment
• run.sh → convenience wrapper (sh run.sh step2_tokenizer.py)
• step1_env_check.py … step12_mini‑decoder.py → twelve incremental Python scripts
• mini_gpt.pth → trained weights from step7
• notes.txt → your personal plan
Prerequisites:
• Git‑clone this folder
• Have Docker (optional) or a local PyTorch install
• (Optionally) run docker compose up step1 --build to verify PyTorch & MPS/CPU 1. Step 1: Environment Check (step1_env_check.py)Goal → make sure PyTorch is installed and pick the right device (CPU / GPU / Apple MPS)
Key concepts:
• torch.version
• torch.device(...)
• tensors live on “device” (CPU vs GPU vs MPS)
Running it prints your PyTorch version, host architecture, and creates a toy tensor to confirm everything works.1. Step 2: Tiny Corpus & Tokenizer (step2_tokenizer.py)Goal → turn text into numbers (“tokenization”)
Code highlights:
• corpus = “I am Sree from Schogini.com”
• vocab = sorted(set(corpus)) ➔ all unique characters
• char_to_idx, idx_to_char dicts
• encoded = torch.tensor([char_to_idx[ch] for ch in corpus])
• decoded back again with idx_to_char
Key concepts for beginners:
• Every NLP model works on numbers, not raw strings.
• Token = smallest unit (here, a character).
• Mapping str ↔ int lets us embed them in a neural net. 1. Step 3: Tiny Transformer (step3_tiny_transformer.py)Goal → define the simplest “transformer‐style” network: embed → linear → logits
Network:
• nn.Embedding(vocab_size, embedding_dim)
• nn.Linear(embedding_dim, vocab_size)
Forward pass:
1. take a batch of token IDs
2. embed into 16‑dim vectors
3. project back to “vocab_size” scores (logits)
Logits are raw scores you’ll convert with softmax later. 1. Step 4: Forward Pass with Debugging (step4_forward_pass.py)Goal → walk through shapes & values
Adds:
• builds tiny vocab from “hello world”
• prints out shapes at each stage, e.g.
– Input IDs: [5]
– After Embedding: [5×16]
– After Linear: [5×vocab_size]
Why it matters:
• Verifying dimensions is the single most important debugging trick in deep learning. 1. Step 5: Tiny Training Loop (step5_train_simple.py)Goal → teach the model to predict next character in “hello”
Pipeline:
1. encode “hello” → [h, e, l, l, o]
2. x = first 4 tokens, y = last 4 tokens shifted by one
3. forward(x) ➔ logits of shape [4×vocab_size]
4. CrossEntropyLoss(logits, y)
5. backprop + optimizer.step()
6. print loss every 10 epochs
Key concepts:
• CrossEntropyLoss combines softmax + negative‑log‑likelihood
• optimizer.zero_grad(), loss.backward(), optimizer.step()
• predict next‑token is the core of autoregressive language models 1. Step 6: Simple Inference & Greedy Decoding (step6_simple_inference.py)Goal → after training, generate a sequence character‑by‑character
Greedy decoding:
1. start with a single token (e.g. “h”)
2. model(input_ids) ➔ logits
3. softmax + argmax ➔ index of most probable next char
4. append & repeat
You’ll see the model output “hellllllllll…”, a sign it learned the pattern. 1. Step 7: Build a Mini‑GPT (step7_mini_gpt.py)Goal → a minimal GPT‑style block with:
• token + position embeddings
• causal self‑attention (nn.MultiheadAttention)
• feed‑forward (MLP)
• layer norms & residual connections
Details:
• block_size = context window length
• position_embedding = nn.Embedding(2×block_size, dim) (avoid overflow)
• attn_mask = upper‐triangular inf mask for causality
• train on random 8‑token chunks of “hello world” for 500 epochs
• save model.state_dict() → mini_gpt.pth
Key concepts:
• Positional encodings let the model know “where” tokens sit.
• Multi‑head self‑attention builds contextualized representations. 1. Step 8: Bonus—Inspecting One‑Head Attention (step8_bonus_multihead_mini_gpt.py)Goal → step through one‑head attention + residuals by printing intermediate shapes
Adds:
• print after embedding+positional, after attention, after feed‑forward
• single head (num_heads=1) for simplicity
• slide a 1‑batch training loop
This reinforces your mental model of the Transformer decoder block. 1. Step 9: Load & Infer from Your Saved GPT (step9_load_and_infer.py)Goal → re‑instantiate the same MiniGPT class, load mini_gpt.pth, and sample
Key points:
• Your class definition must match exactly
• model.load_state_dict(torch.load("mini_gpt.pth"))
• Use multinomial sampling for randomness
• Maintain a sliding context window to avoid infinite growth 1. Step 10: Top‑k Sampling & Long Context (step10_upgraded_top-k-sampling-long-context.py)Goal → replace greedy decoding with top‑k sampling and enforce a fixed context window
Top‑k sampling:
1. take logits for last token
2. pick the top k highest scores
3. renormalize with softmax
4. sample from that smaller distribution
Long context:
• always keep only the last block_size tokens as input
This mimics GPT‑style generation more closely. 1. Step 11: Temperature Control (step11_temperature.py)Goal → add temperature to control randomness
Temperature scaling:
• logits ← logits / T
– T > 1 → flattens (more random)
– T < 1 → sharpens (more deterministic)
Combined with top‑k, this gives you fine control over creativity vs coherence. 1. Step 12: Building a Mini Decoder Block (step12_mini‑decoder.py)Goal → pull everything together into a reusable MiniDecoderBlock:
Layers:
1. token + positional embeddings
2. causal multi‑head attention (mask via a helper)
3. add & layer‑norm
4. feed‑forward MLP
5. add & layer‑norm
You run it on dummy input (batch_size=2, seq_len=5) to confirm you get a (2×5×dim) output.## Wrapping Up
• You now have a character‑level mini GPT from scratch.
• You’ve seen how tokenization, embeddings, attention, training loops, and sampling strategies all fit together.
• Try extending it: bigger vocab, different temperature schedules, beam search, or full word‑level tokenization.
Happy hacking! 🚀