Build the whole stack.
ForgeLM is a GPT-style language-model implementation in C#. The project includes the byte-level BPE tokenizer, transformer, forward and backward numerical kernels, AdamW optimizer, dataset preparation, checkpoints, and text generation.
Writing those pieces makes their contracts visible. What is a token? Which tensors survive until the backward pass? What does it mean to resume a training run? A working model needs answers to all three.
One model, three compute backends.
The model calls an ITensorBackend interface. The CPU path uses SIMD and parallel matrix multiplication. CUDA uses ILGPU, with cuBLAS paths for matrix multiplication. The Windows Direct3D 12 backend uses ComputeSharp. These are separate implementations of the same numerical operations.
public interface ITensorBackend
{
void MatMulNN(
Tensor a, Tensor b, Tensor y,
int M, int K, int N,
bool accumulate = false);
// Other forward and backward kernels…
}Shortened from ITensorBackend.cs. Matrix shapes and gradient accumulation are explicit.
The backend comparison matters for correctness as well as speed. A fast matrix multiplication that changes the gradient, or an allocation pattern that slowly exhausts device memory, is not a successful optimization.
The GPU wasn’t the whole bottleneck.
During the Forge-220M run, the attention path called cuBLAS separately for each batch/head slot. At the production shape, seven attention matrix operations produced 57,344 individual GEMM submissions per optimizer update. One host thread was saturated while GPU utilization swung between 7% and 82%.
A narrow binding to cublasSgemmStridedBatched let one call submit the independent attention matrices together. The change reused the existing cuBLAS handle, stream, and device allocations. Ordinary projection matrices kept their existing path.
The displayed losses matched in those measurements. The change also passed the then-current 121-test suite, including batched matrix operations, full-model gradients, overfit, allocation, checkpoint resume, and CPU comparisons. Only then was it promoted into training.
A second experiment cached attention activations instead of reconstructing them during backward. It improved the same-host synthetic benchmark by 17.6%, using 19.0 GB instead of 15.4 GB. The first normal production logs showed a smaller 9.6% improvement. Keeping both numbers matters: a short benchmark and a real training loop answer different questions.
The performance work also rejected a larger physical batch: it consumed more VRAM and ran slower. The useful result is the measured tradeoff, including the changes that did not help.
A checkpoint should actually resume.
Forge checkpoints retain the weights, optimizer moments, global step, learning-rate schedule, and data-sampler position. Restarting from that state continues the same trajectory instead of resetting warmup or revisiting the start of the corpus.
Checkpoint files include a SHA-256 trailer and are bound to their tokenizer and data identity. Saves rotate the previous generation. The CLI distinguishes exact resume from loading weights into a fresh training stage; instruction tuning needs the latter.
From a base model to instruction tuning.
The first released model, Forge-98M, has 97,934,592 parameters. It was trained on an RTX 4090 with a 1.024-billion-token FineWeb-Edu budget. Its model card records the following result:
The validation loss is the mean of 50 deterministic held-out batches. It measures next-token prediction on that split; it is not a chat, coding, or reasoning score.
The larger Forge-220M experiment uses a prepared 80/20 mixture of FineWeb-Edu and general FineWeb. Instruction tuning adds conversations with system and user tokens visible as context, while only assistant responses and end-of-sequence tokens contribute to the training loss.
What it actually says.
These are recorded outputs from Forge-220M-Instruct, trimmed only for length. They are static samples, generated locally before this redesign. Open a prompt to read the response.
Temperature 0.7 · top-k 30 · repetition penalty 1.1 · no-repeat 4-gram.
Prompt wrapper: User: {prompt}\nAssistant:
What is the meaning of life?
Life is a complex and multifaceted concept that encompasses various aspects of human experience, relationships, and responsibilities. On one hand, life is often described as a state of complete relaxation, joyful connection, or contentment with our surroundings. This can be achieved through various means, such as exercise, meditation, hobbies, or spending time in nature.
On the other hand, life can include a wide range of activities and experiences that bring joy, fulfillment, and a sense of purpose. Life is often marked by moments of personal growth, such as learning new skills, developing new skills, or pursuing personal interests.
While life may seem like an all-consuming pursuit, it can actually be a blessing in disguise. By nurturing and connecting with others, individuals can cultivate meaningful relationships, gain a deeper understanding of themselves, and develop the skills and mindset necessary to live a fulfilling life.
Tell me a short story about a robot learning to paint.
One day, a mysterious robot named "Aurora" was discovered by a group of rebels at a high-society gathering in Chicago. Aurora was equipped with an advanced brush, allowing it to create beautiful and intricate paintings.
Aurora's brushes began to blend seamlessly into the background, revealing hidden patterns and colors that seemed to be shifting and shimmering. She began by painting in small pieces, using them to convey emotions and emotions. As the days passed, she became more confident with her paintings, able to bring joy and vibrancy to the scenes.
Explain photosynthesis in simple terms.
Photosynthesis is a process in which plants, algae, and some bacteria convert sunlight into energy. This process is the primary source of energy for most living organisms on Earth.
Photosynthesis also involves the conversion of carbon dioxide into glucose, which is a key component of organic matter and serves as a simple carbohydrate for the plant.
How do I bake a loaf of bread?
Baking a loaf is a delicate artful process that requires attention to the ingredients, techniques, and time constraints. Here's a simple recipe to get you started:
1. Prepare the Bread and Whisks: Before the bread is rolled out, you want to ensure it's perfectly chopped and coated. Preheat your oven to 375°F (145°C). 2. Preheat the Oven: place the pot over the oven rack, ensuring the oven is fully heated (about 225°F). The key is to prevent the oil from boiling.
What is an algorithm?
An algorithm is a set of instructions that a computer can execute, using logic and mathematical principles to solve problems or achieve a specific goal. It is a way to solve a problem by breaking it down into smaller steps and using them to make the algorithm work more efficiently.
An algorithm typically consists of a series of steps, known as steps, which are used to perform the necessary actions.
The bread answer is confidently wrong: 375°F is not 145°C, and the instructions quickly become incoherent. Even the better answers can repeat ideas. The model is an experiment in building and inspecting a training stack, and these samples show its limits as well as its progress.
Source and experiment notes.
- ForgeLM source repository ↗
- Forge-98M model card ↗
- Recorded instruction samples and settings · JSON
- Attention benchmark measurements · JSON
Numbers here come from the saved model card and August 2026 performance record. The training and GPU benchmarks were not rerun for this write-up.