Mini-GPT is a small language model built from scratch in PyTorch, covering every layer from tokenizer to attention to sampling. It sits behind a streaming chat interface and fine-tunes itself on a schedule using the feedback people give it.
The Idea
Calling a hosted model through an API teaches you very little about what happens inside one. We wanted a GPT we could read end to end. That meant writing the tokenizer, the attention, the transformer blocks, the training loop and the generation code ourselves, small enough to train on a workstation and clear enough to document line by line.
The second goal was to close the loop. A chat UI collects thumbs-up and thumbs-down ratings. A scheduler turns the good answers into training data, fine-tunes the model and hot-reloads it, with no one restarting anything.
What We Built
-
✦
A GPT from first principles Token and position embeddings feed a stack of pre-norm transformer blocks, ending in a next-token head trained with cross-entropy. Five size presets run from a CPU-friendly 50K-parameter model up to roughly 50M parameters.
-
✦
Modern attention options Rotary position embeddings, a KV cache, a flash-attention path and grouped-query attention can each be switched on in the larger presets.
-
✦
Three tokenizers There's a character-level tokenizer, a byte-pair encoder written from scratch, and a SentencePiece wrapper, so the same model can be trained at different granularities.
-
✦
A real training loop The training loop supports mixed precision, gradient accumulation, learning-rate warmup into a cosine schedule, TensorBoard logging and periodic checkpoints. Separate scripts clean dialog data and combine it into a chatbot fine-tuning set.
-
✦
Streaming chat server A Flask API streams tokens over Server-Sent Events. It keeps short per-session memory and offers four personas, each with its own sampling temperature. Responses can be regenerated, and the API also exposes model info, stats and downloads. Generation supports temperature, top-k, top-p and repetition penalty.
-
✦
Learning from feedback Every rating is stored. An APScheduler job checks hourly and fine-tunes once at least five new positive examples have built up. Before each run it backs up the current model, keeping the last five versions, and it hot-reloads the server when training finishes.
-
✦
Built-in documentation A documentation site of nine pages walks through the tokenizer, embeddings, attention, transformer blocks, the GPT model, training, inference and the server API. It includes an interactive embedding simulator.
The Stack
The model and training code are pure PyTorch, and nothing is borrowed from a model library. The server uses Flask with Flask-CORS, and Gunicorn is in the production requirements. APScheduler handles background fine-tuning. The chat UI and the docs site are plain HTML, CSS and JavaScript served by the same app. A pytest suite covers attention, the model and the tokenizers. Alongside it we wrote a companion beginner series that rebuilds tokenization in plain Python with no libraries at all, as the first step toward explaining how a model like this reads text.
What We Learned
The full loop works. Base training saved checkpoints past 230,000 steps. The chatbot model serves through the streaming UI, and the logged automatic fine-tune trained on five feedback examples in under six seconds, reaching a final loss of 0.43, with a backup of the previous model saved first. The test suite has 37 tests, covering attention shapes, causal masking and the KV cache, model forward passes, generation and gradient flow, and tokenizer round-trips. One limit is deliberate: the deployed chatbot uses a character-level checkpoint, so it shows the mechanics clearly rather than competing with production models on fluency. The BPE and SentencePiece paths are there for larger runs.
Need something similar? Reach out. We build and fine-tune models when an off-the-shelf API isn't the right fit.