Build the Model Before You Build the Startup

Imagine being 17, closing a game, and opening a terminal with one ambitious thought: build a language model from scratch. The sensible version of that goal is not to recreate a frontier model in a bedroom. It is to make each layer small enough to inspect, break, measure, and rebuild.

A large language model, or LLM, is a neural network trained to predict the next token. A token is a small piece of text: perhaps a character, part of a word, or a common whole word. The task sounds modest until the model must repeat it across millions or billions of examples. Learning how that machinery works gives you something more valuable than a flashy demo: a reliable instinct for which ideas are real and which are marketing fog.

Start with a model that can fail in public

Your first model should be small enough to disappoint you quickly. That is a feature. The Transformer architecture introduced in 2017 made modern language models practical by replacing recurrent processing with attention, a mechanism that lets each token assign different levels of importance to other tokens in the same sequence. (arxiv.org)

A character-level model is a good first step. Instead of using words or subwords, it reads one character at a time. Feed it a small collection of plays, train it to guess the next character, and you will soon see outputs that look almost like English before collapsing into nonsense. That strange moment is useful: you can watch a model learn spelling patterns, names, punctuation, and the rhythm of dialogue without hiding the process behind a large library.

Andrej Karpathy’s nanoGPT repository documents this progression with a Shakespeare example built from a 1 MB text file. Its teaching configuration uses a six-layer Transformer with a 256-character context, and the README reports a roughly three-minute run on one A100 GPU. A November 2025 update now calls nanoGPT old and deprecated, pointing readers toward nanochat, but the older code remains a remarkably clear map of the basic training loop. (github.com)

The training loop is the whole trick

Training is a repeated guessing game. Give the model a sequence of tokens, ask for the next one at every position, measure how wrong the guesses were, then adjust the model’s parameters. Parameters are the learned numbers inside the network. A tensor, the multidimensional array used by frameworks such as PyTorch, carries those numbers through the calculation.

A stripped-down language-model step looks like this:

x = tokens[:,:-1] # everything except the final token
y = tokens[:, 1:] # the answer is the next token

logits = model(x) # raw scores for every possible next token
loss = F.cross_entropy(
 logits.reshape(-1, logits.size(-1)),
 y.reshape(-1)
)

optimizer.zero_grad
loss.backward
optimizer.step

The loss is a number that summarizes how badly the predictions missed. Calling backward computes gradients, which indicate how each parameter contributed to that error. The optimizer uses those gradients to nudge the parameters toward better predictions. Seeing this happen in code is more educational than memorizing a diagram of a Transformer.

Move from characters to tokens

Once the character model works, build a tokenizer. A tokenizer converts raw text into integer IDs that a neural network can process. Byte-pair encoding, or BPE, starts with small units and repeatedly merges pairs that appear frequently, allowing common words to become single tokens while rare words can be split into several pieces. (huggingface.co)

This change affects nearly everything. A different tokenizer changes sequence length, vocabulary size, memory use, and the kinds of patterns the model sees. You will also need a decoder-only Transformer, meaning a model that predicts text from left to right without seeing future tokens. Its causal mask blocks information from later positions, while self-attention connects each current position to the earlier context.

Implement one block yourself before reaching for a high-level library. Include an embedding table, which turns token IDs into learned vectors; a signal for token order; masked self-attention; a small feed-forward network; residual connections that add an earlier signal back into the computation; normalization to keep values well behaved; and an output layer that turns the final vectors into scores over the vocabulary. The list looks intimidating on paper. In code, each part is a small function with shapes you can print and inspect.

A text completer is not automatically a chat assistant. Pretraining means broad next-token learning over a large text collection. Fine-tuning means continuing from those learned weights with a narrower dataset, such as instructions, conversations, or technical documents. The from-scratch GPT-2 walkthrough explicitly separates this language-model stage from later chat fine-tuning. (github.com)

Data decides what the model becomes

The data pipeline is the repeatable path from raw files to training batches, and it deserves as much attention as the model code. Keep a validation set: held-out text that the model never trains on and that you use to check whether it is learning general patterns rather than memorizing the examples. Record where the text came from, remove duplicates, and inspect samples instead of trusting a single loss number.

Data contamination is a particularly nasty trap. It happens when evaluation questions or their answers accidentally appear in the training data, allowing a model to score well by remembering material it was supposed to encounter only during testing. Research on contamination shows why clean evaluation matters: a benchmark score can look impressive while saying less about genuine generalization. (arxiv.org)

Model size is not the only knob that matters. The Chinchilla scaling study found that, under a fixed compute budget, model parameters and training tokens should grow together rather than putting all the budget into a larger network. The practical lesson is encouraging for independent learners: a smaller model trained on enough clean data can teach you more—and sometimes perform better—than a larger model that was stopped too early.

Let hardware teach you

How do you build an LLM from scratch without a data center? Change the question from What is the biggest model I can train? to What is the largest experiment that will teach me one thing?

A laptop or CPU can handle a tiny character model. A single GPU opens the door to larger experiments. Multiple GPUs introduce distributed data parallel training, where several copies of the model calculate updates and synchronize their gradients. That is valuable systems knowledge, but it is a poor place to begin if you have not yet verified the model on one device.

The scale gap is real. The nanoGPT documentation puts a GPT-2-sized 124-million-parameter reproduction at roughly four days on a node with eight A100 40 GB GPUs. That is very different from the small Shakespeare experiment, and it is a useful warning against treating a frontier model as the natural first project.

Optimization comes after correctness. FlashAttention-2, for example, keeps the attention calculation exact while reorganizing how GPU memory and parallel work are handled; its paper reports roughly a two-times speedup over the earlier FlashAttention implementation in its benchmarks. Studying improvements like this teaches an important engineering lesson: moving data through memory can matter as much as doing arithmetic on it.

Why the startup can wait

The startup advice is easy to misread. A company is a vehicle; understanding is the engine. Building an LLM from scratch exposes the real friction points: messy data, unstable training, GPU memory limits, misleading evaluations, slow inference, and the gap between a convincing sample and a dependable product.

There is also a useful counterpoint. Language models can produce polished essays while remaining helpless at ordinary physical tasks. Text prediction gives a model patterns in language, not a body, persistent goals, or feedback from the physical world. Learning how an LLM works helps you see that boundary clearly—and perhaps notice where a different architecture is needed.

At 17, or at any other age, the durable advantage is not owning the biggest checkpoint. It is knowing how text becomes numbers, how numbers become predictions, and how predictions become behavior. Start with a model small enough to follow line by line, then let ambition grow with evidence.