LoRA Speedrun: making fine-tuning speed measurable (and fair)

You know the feeling: two LoRA fine-tuning experiments both claim “60%+ accuracy,” yet the winners are hard to believe. One run might be fast because it used a better data loader, another might be slow because it re-tokenized everything from scratch, and neither poster can prove the leaderboard clock was actually running on the same kind of machine.

That’s exactly the problem the LoRA Speedrun repo tries to solve. It turns parameter-efficient fine-tuning (LoRA-style adapters) into a “speedrun” format: fixed task, fixed hardware, and a public wall-clock leaderboard where records are independently re-run with fresh seeds before they count. (github.com)

LoRA speedrun in one picture: adapters, not full retrains

Before talking tricks, let’s pin down the meaning of “LoRA fine-tuning” for beginners.

LoRA (Low-Rank Adaptation) is a method for adapting a pre-trained language model without updating every weight. Instead, it injects small trainable matrices into specific layers (for many transformer models, common targets are attention projection layers and/or MLP projections). The rest of the base model weights stay frozen (unchanged), so training updates are lightweight.

In this speedrun track, the base model is frozen and the only trainable artifact you submit is a PEFT adapter (a package containing the small learned parameters). The task enforces an upper bound on how many trainable parameters your adapter can contain, so “make it faster” doesn’t become “update the whole network.” (raw.githubusercontent.com)

The frozen task: Qwen2.5-1.5B + GSM8K, judged by accuracy and wall-clock

What makes this repo different from a typical GitHub experiment folder is that the rules are explicit and frozen.

This track (v1) is named gsm8k-qwen2.5-1.5b-l40s, and it is frozen as of 2026-07-18—meaning base model, dataset, evaluation protocol, and hardware are treated as immutable for that leaderboard. (raw.githubusercontent.com)

Here are the key “physics of the contest” constraints:

• Base model: Qwen/Qwen2.5-1.5B (base, not instruct) (raw.githubusercontent.com)
• Training data: GSM8K train split only (7,473 examples) (raw.githubusercontent.com)
• Evaluation data: GSM8K test split, but only for scoring after training (raw.githubusercontent.com)
• Goal: hit ≥ 57.0% exact-match accuracy on the evaluation protocol (raw.githubusercontent.com)
• Hardware: 1× NVIDIA L40S (48 GB) in a Modal Sandbox with network blocked (raw.githubusercontent.com)
• What’s timed: the harness measures the wall-clock time from launching training until it exits (tokenization and adapter saving are included; model/dataset downloading is pre-cached and not timed) (raw.githubusercontent.com)

Why does that matter? Because “training time” isn’t one thing. It’s tokenization, dataloader overhead, loss computation cost, batch sizing behavior, and even the time it takes your code to write adapter files. If you don’t standardize the clock, you end up optimizing the benchmark you accidentally created, not the one you intended.

And yes, the leaderboard explicitly answers the underlying question: Why do two LoRA runs that both reach similar accuracy still feel incomparable? In this format, they’re comparable because the clock and the rules are shared. (github.com)

The current record: sequence packing + completion-only loss masking

As of the repo’s current leaderboard state, the record is held at 6m 05s with 61.1% GSM8K accuracy, attributed to sequence packing + completion-only loss masking, trained for 2 epochs. (github.com)

The baseline (plain LoRA, no tricks) is listed at 11m 57s with 59.4% accuracy. The record is roughly a 2× speedup while also improving accuracy—an unusually clean outcome for fine-tuning “speed tricks.” (raw.githubusercontent.com)

Now let’s unpack what those two techniques usually mean in practice.

Sequence packing: fewer pads, fuller GPU utilization

Sequence packing is a data batching trick. Instead of padding each training example up to a max sequence length, we concatenate multiple examples into one longer sequence (up to the context limit) and create an attention mask that prevents the model from mixing tokens across example boundaries in ways that would break supervision.

In plain language: packing tries to make the GPU work on “useful tokens” rather than wasting computation on padding.

In transformer training, padding tokens still travel through many parts of the computation graph unless the attention mask and implementation are smart. Packing reduces the percentage of padding, so throughput (tokens/sec) rises, and wall-clock time drops—especially when the dataset has variable-length problems (GSM8K often does).

Completion-only loss masking: don’t pay loss for the prompt

With causal language modeling (next-token prediction), training typically computes a cross-entropy loss for many or all positions in the input.

But supervised fine-tuning often has a template structure like:

• Prompt: Question: ...\nAnswer:
• Completion: #### 12345

In that setup, the prompt is meant to condition the model, while the completion is what should be learned. Completion-only loss masking means: compute loss only on the tokens that belong to the completion, and mask out the prompt tokens.

The common PyTorch mechanism is to set masked label positions to -100, because many loss functions (including torch.nn.CrossEntropyLoss) treat -100 as “ignore this position.”

Here’s the core idea in code:

# input_ids: token ids for a packed training example
# labels: next-token targets (often input_ids shifted)
# prompt_mask: True for prompt tokens, False for completion tokens

labels_masked = labels.clone()
labels_masked[prompt_mask] = -100  # ignore prompt positions in loss

loss = torch.nn.functional.cross_entropy(
    logits.view(-1, logits.size(-1)),
    labels_masked.view(-1),
    ignore_index=-100,
)

When packing is involved, masking becomes even more crucial. Each packed segment contains its own prompt and completion, so the “ignore prompt loss” rule needs to apply at the right boundaries.

This is the kind of trick that feels minor until you connect it to performance: reducing which tokens participate in loss can cut compute and memory pressure, which can compound with packing.

How the leaderboard stays honest: timed reruns, anti-tampering, and sandboxing

A public leaderboard only works if someone can’t cheat by accidentally (or intentionally) changing the rules.

This repo’s verification pipeline looks like an audit trail:

1. Static validation (CI): required files and config shape are checked, and obvious issues are rejected. (raw.githubusercontent.com)
2. AI security screen (advisory): an automated reviewer checks for things like network usage or suspicious harness changes; it can influence human review but doesn’t auto-block. (raw.githubusercontent.com)
3. Human code review: a maintainer reads the training script and config before running anything. (raw.githubusercontent.com)
4. Timed reruns in a network-blocked Modal Sandbox: your submission is executed 3× with fresh seeds selected at verification time, on the spec L40S hardware. The harness owns the timer. (raw.githubusercontent.com)
5. Adapter audit and hash re-verification: the harness audits adapter parameter counts and verifies base model/dataset content hashes in isolation to prevent cache-poisoning. (raw.githubusercontent.com)

Two details are especially important for beginners:

• Wall-clock time is the score, not “steps”. Many training setups can hide work in setup/teardown, background threads, or tokenization pipelines. The harness measures the full process runtime. (raw.githubusercontent.com)
• Evaluation is offline and deterministic by protocol. Model/dataset downloads aren’t timed, and the evaluation procedure is fixed: prompt format, greedy decoding settings, and how answers are extracted. (raw.githubusercontent.com)

What you can borrow from the speedrun mindset (even without chasing the record)

This repo isn’t only about LoRA. It’s about measurement.

Once the task is frozen, speed optimizations become discoverable. The leaderboard rewards techniques that improve real runtime—things like sequence packing and completion-only loss masking that reduce wasted compute.

And once verification is public and repeatable, “my run was faster” stops being hand-wavy. Any record can be independently re-verified under the same sandbox rules and timer ownership. (github.com)

The best takeaway is almost boring in a good way: when ML improvements are measured on inconsistent clocks, the leaderboard becomes a story generator. When the rules are frozen and the runtime is audited, optimization turns back into engineering.

Closing thought

LoRA Speedrun treats fine-tuning like a benchmark game: same hardware, same task, same scoring rules, and evidence that the speed claim survives reruns. Techniques like sequence packing and completion-only loss masking matter more in that world, because they move the only number that counts here: wall-clock time—while still meeting the GSM8K accuracy bar. (raw.githubusercontent.com)