This is the first in a series about building a small language model that writes Magic: The Gathering cards into a compiler I built by hand. This one is the origin story: how a joke turned into a problem I couldn’t put down, and the wall that made everything after it necessary. You don’t need to know anything about Magic to follow along: where the game matters, I’ll explain it.
The joke Link to heading
It started on a hot summer afternoon at a friend’s place, playing Commander. If you don’t play: Commander is the kitchen-table format of Magic. Four people, big decks, games that sprawl for hours. There’s a lot of downtime while other people take their turns, and downtime is when the table talk happens.
We were talking about how Wizards of the Coast (the company that makes Magic) had started leaning on AI-generated artwork in their marketing. Banners, promotional images, that sort of thing. It had already caused a stir in the community; even the marketing art got backlash. Everything except the cards themselves. And the joke more or less wrote itself: imagine the meltdown if they did it on actual cards.
We didn’t particularly care about the controversy. But somewhere in there the joke turned into an idea, the way jokes do when you’re the kind of person who can’t leave a “wouldn’t it be funny if” alone. What if I just… showed them? Generated actual cards? Custom commanders based on each of us, cards representing the outcome of a game or a location that is dear to us, that sort of thing. One concrete example of that is turning my dog Stella into a husky commander for a tribal deck.
(A flash-forward, because she earned it: here’s what that request would eventually become. If you play Magic, something on this card is quietly wrong. Hold that thought; it’s the whole story.)
That was the whole ambition. Make a few funny cards, get a laugh, move on.
However, as you may have already guessed, I did not move on.
“It’s just a file” Link to heading
Here’s the thing that turned a joke into a project: I already knew it was possible, because I knew how the cards get made.
I’d proxied cards before. In Commander, proxies (custom-printed stand-ins) are common for casual play, and I’d made my own to give a deck a consistent visual identity. The tool I settled on for this project was Magic Set Editor (MSE). While not the fanciest or the easiest to build around, it had qualities that the others fundamentally didn’t. Decades of development, a community of users, and a command-line interface.
This is a fan project for making custom cards, I figured. The whole point is people sharing their creations, same as I wanted to. So they have no incentive to make the format hostile or obfuscated: it’s built to be produced and passed around. And I’d parsed genuinely nasty formats before; a fan-made editor’s save file is probably something civilized, like a zip full of XML. (A .docx file, underneath, is exactly that, a zip of XML, and I’d had to crack one open by hand at work to generate Word reports for a client who needed features the usual Python libraries like docxtpl couldn’t produce.) And indeed the MSE export format is a zip of XML and images.
So now I could write cards as files. I could generate them in code. There was just one piece missing, and it was the piece the whole joke depended on: I didn’t want to write code for each card. I wanted to describe a card in plain English (“Stella, a husky commander for a dog tribal deck”) and have something turn that description into a real, mechanically-valid Magic card.
That’s a job for a language model. Understanding a human request phrased in natural language and producing structured output is the thing LLMs are supposed to be good at.
Boy, was I wrong.
The first real version Link to heading
I’d built Discord bots before, so a Discord bot was the obvious interface: you type a command, you get a card back. It wasn’t the perfect choice in the long run, but it let my friends actually use the thing at close to no cost, and it came with a feature I didn’t appreciate at the time: every card the bot produced was sitting in a channel/thread, with the people who saw it able to react to it, plus basic logs of every step the pipeline took and what the LLM produced as step outputs. I knew that if I wanted to improve the generation in the future, collecting feedback would be useful, so it was implemented from the start while having 0 concrete use case, but in the end it proved to be useful. So if you’re doing your own project, I’d heavily recommend you to start with a strong logging system that lasts over time. It may seem not worth the investment initially, but trust me, that saved me so much time: it let me break my assumptions when they were false but not obvious, see patterns of failure emerge over time, and finally it allows me to share traces of the project at its initial stage while I’m writing this, multiple months later.
The actual generation, underneath the bot, started simple. A user types a description. The bot hands it to an LLM, asks for the card’s fields (name, cost, type, rules text), drops them into the MSE format, renders the image, sends it back.
And one constraint shaped everything that followed: I wanted it all running locally. I build my own infrastructure, I have my own GPU, and I didn’t want this depending on someone’s API. Everything on my own hardware.
At the time, that hardware was an RTX 2080 Ti. And this is where I hit the first wall.
The wall Link to heading
An RTX 2080 Ti can hold, at most, a 14-billion-parameter model, and only with offloading, where part of the model spills over to slower memory because it won’t fit on the GPU. Offloading works, but is slow.
I want to be precise about this, because it matters for everything downstream: the problem was never that large language models can’t do this. A frontier model, the kind you reach through an API, would very likely have handled it fine. The problem was that I’d chosen to run locally, and the biggest model I could physically fit on a 2080 Ti was nowhere near frontier scale. Every limitation I’m about to describe is a limitation of the models I could run.
But raw speed wasn’t even the first problem. The first problem was that the output was wrong, and it got wronger as the requests got harder.
The smallest models (I’m talking 1.5B parameters) could handle “a goblin” and little else. Push past trivial and they produced genuine garbage. By around 4B, the output became consistent: recognizably card-shaped, structurally plausible, but still frequently wrong, just wrong in more interesting and less obvious ways. And “wrong” had a shape, which is the whole reason the rest of this series exists.
Here’s what that looks like in practice: the same request, “a golden knife”, handled by three sizes of the same model family, smallest to largest.
The smallest gets the vibe and fumbles the game: “Gold Metallic” isn’t a subtype that exists, and worse than that, the card basically does nothing; it costs 3 mana to generate one temporary mana. Everything about it is a downside. While we can joke that this is a good representation of a golden knife, it costs a lot for nothing. I wanted a real card, not a pure joke.
The middle one has real bones (a genuine activated ability, an effect with multiple costs and a correct separation of the cost elements), but the second part of the cost contains a small but critical mistake. “Sacrifice a golden artifact”: the “golden” is invalid in Magic syntax; there would be no way to know if a given artifact is golden or not. This small error makes the card invalid, and any Magic player will quickly spot it has been AI generated.
The biggest gets remarkably close: a proper Equipment subtype, legal templating, a correct balance of cost and effect. The only issue is what is missing this time! The card is valid but not legal. It lacks the Equipment’s distinctive aspect, being able to be equipped: to do so in MTG you have a special ability called “equip” that allows you to attach the equipment to a creature. And it is missing, making it once more unclear to a player whether the card is real or an AI-generated one.
So let’s see examples now of what it started with and how it evolved rapidly.
Here’s the oldest card I still have a recording of:
It looks badly broken: art shoved to the top, an empty text box, no type where there should have been. But the interesting part is not the rendering that is wrong (this is my attempt at a custom renderer before MSE); what was happening underneath is what is interesting, because even this early there was already a staged pipeline behind every card, with logs that let you move from ’the POC is not viable’ to ‘maybe there is potential here’. The actual trace for this one:
1. Parsing your request...
Input: "a goblin with two heads and a giant mace"
2. Generating card data...
Concept: A goblin creature with two heads and a giant mace...
3. Card data created:
Type: Creature — Goblin Brute Mana: 2R P/T: 3/2
Text: Whenever this creature attacks, it gets +1/+0 until end of turn.
4. Balance validation:
Balanced: No (adjusted) Power rating: 3.8/5.0
Issues: Mana efficiency too high for a 3/2 with a repeatable combat boost...
5. Card aesthetics: ...
6. Artwork generated: ...
7. Final quality check:
Intent match: Partial Confidence: 70%
Regenerating...
The logic worked. It parsed the request, produced a coherent card (a 3/2 Goblin Brute for 2R with a sensible attack trigger), ran it through a balance check that correctly flagged it as a little overpowered and decided regeneration was needed at 70% confidence (which was not implemented yet). What you’re looking at isn’t the model failing; it’s the seam, the half-second before the system finished assembling the card. The scaffolding was there from card one. The output was just still rough. That kind of broken, I could fix with engineering.
Now let’s fast-forward a bit to a place where things start to be interesting, with the POC already showing signs of progress but at the same time showing a totally different kind of issue than before.
Here’s the pipeline, asked for something as simple as it gets, “an angel”:
It looks great at first glance. Clean art, proper frame, a flavor line about a solemn vow to shield the innocent, with just a minor _ prefix and suffix that could be solved by a simple s/_//. The issue lies in what’s above, in the place where the oracle text, what the card does, should be.
Instead we get the card name repeated there.
And the rules text (the part that says what the card actually does) is just the card’s own name typed back at you. “Petrified Heartwood Halberd.” That’s the whole ability. The card does nothing.
I have the full trace, and it shows where this went wrong, and it wasn’t at the end. It was three stages in:
[21:47:53] parsing
Concept: Divine Intervention (Sorcery, W, B): At beginning's end, create a 2/2 white Angel...
Types: Sorcery, Creature Forced type: Creature — Angel
...
[21:48:05] lore
World: Innistrad
Backstory: Guarding the realm with sacred vigilance, this angel adorns halberd and crown, symbolizing the unity
Flavor text: For justice, I will wield my blade and shield, until the realm is secure.
[21:48:14] card_ability_text
1 keywords, 24 chars text
[21:48:17] balance
Status: Balanced
Power rating: 3.0/5.0
Issues: None
[21:48:24] validation
Intent match: Yes
Confidence: 80%
Notes: While capturing the essence, the card could better represent 'Pious' and 'Resolute' traits through additional text or a more impactful ability related to Divine Intervention or mentorship.
...
There are a few interesting things going on here, so let’s walk the trace stage by stage.
Parsing. The problem shows up immediately: it got the intent wrong.
Lore. Then things take a turn. It clearly picked an appropriate plane from Magic for the card requested, and the backstory even corrected the parsing’s intent mistake, landing on better flavor text than the one that made it onto the card.
Ability text. Here the contradiction returns: it chose 1 keyword, which seems fine. But a keyword with 24 characters? That’s oddly long, suspiciously close to the length of the card name itself. So that’s the step where the failure lives.
Balance. One step later, another crack: the balance check calls the card balanced with a 3.0/5.0 score, even though we just saw the ability text was wrong. This is the first trace of a bigger issue we’ll see coming later in this story.
Validation. The final step is the strange one. It failed at its actual job (serving as a gate, checking whether the shipped card matched the original request) and yet its notes still show traces of a thought that something was off.
It got worse before it got better Link to heading
The obvious move, when a single LLM call won’t get it right, is to decompose the problem. So I did, and kept doing it. Split the request into understanding intent, then generating from that cleaned-up intent. Then more stages, more structure, more validation between them. I kept going until I had a full DAG: a directed graph of LLM calls, thirteen of them by the end, plus a retrieval system (RAG) holding both the official rules and a database of every Magic card ever printed.
More stages caused intent drift. Each LLM call is a small game of telephone. By the time a request had passed through enough hands, the thing that came out the far end had quietly drifted from what was asked for. You can watch it happen in the Halberd trace: the request was “an angel,” and somewhere in the chain it became a malformed “Divine Intervention” fragment that the later stages then faithfully preserved. The decomposition that was supposed to improve fidelity was, past a certain point, eroding it.
The retries were brutal on local hardware. Every rejection re-ran expensive work. And there was a cruel irony baked in: the bigger models I could run, which produced better cards, were also stricter validators. They rejected more, which triggered more retries. Better quality and more retries pulled against each other on a machine that was already slow.
The numbers tell the story. The smallest workable model, on a request it could handle with no retries, took about a minute and a half per card. A bigger one, on a hard request with multiple retries, took forty-five minutes or more. For one card.
I tried to fix the speed by routing by complexity: each step of the pipeline gets assigned a difficulty tier and I preconfigure the model size depending on that tier. We saw that even a 1.5B model can create a good lore step and costs 1.5 seconds, so if it can improve speed by trading some capacity where we don’t need it, this should fix the issue.
Well sadly, while the idea seemed sound, the hardware reality broke it: loading even a 1.5B model next to the larger one like a 14B would cause the unloading of the larger model, and the cold start dropped performance to the ground, or CPU offloading did the same. So if I wanted to use the larger model my GPU could run, and which proved to be good, I had to run only the largest and tolerate the speed.
Every door I opened to fix one problem led to a room with a new problem in it.
Why I didn’t quit Link to heading
Reading this back, the obvious question is: why keep going? Forty-five minutes for one card is not a good system. I wanted to draft those custom cards with my friends and even implemented a generate-set command that used an orchestrator LLM to generate a coherent set and run the pipeline for each intent. But with 45 min per card that’s impossible to do consistently, and I let you imagine the electric bill for just a one-off joke…
Well, the answer is that, sometimes, it worked. And when it worked, it was genuinely delightful.
Here’s a card that came out of a simple request, “an old treefolk”:
Unlike all the failures I keep showing you, this one is flawless. While it doesn’t respect the initial idea at 100%, my friends and I agreed that this was indeed impressive.
There’s a real ability. It parses, it’s playable, it’s sensibly costed for a 2/2 common, and it would not look out of place in an actual Magic set. The art fits, the flavor text is evocative, the whole thing coheres. If I slipped this card into a stack of real ones, you might not catch it. Even the slightly underwhelming effect has an in-world excuse: it’s a common, and commons are supposed to be modest. This is the pipeline doing exactly what I’d dreamed it would do back at the kitchen table.
This card did not come from the biggest model I could run. It came from a medium-sized one, which proved that bigger is not always better (which we will come back to in the last article of the series).
It wasn’t a one-off, either:
Correct creature type, a coherent little Food engine, and the art, flavor and mechanics all telling the same biscuit-motivated story. When the pipeline landed one, it landed it.
Then there were cards like this one. I fed the pipeline the lyrics of a fan-made song about an enemy faction in a video game, and we can see the same pattern from the smallest model but replicated through larger ones:
If Verdant Sentinel was a quiet miracle, this one felt outright bad; however, under the hood there are still signs of good behavior that could be integrated in a system. It read the lyrics, understood they were about technological control and forced unity, chose a fitting three-color identity, generated matching dystopian art, and pulled the name, J.O.E.L, as an acronym straight out of the song’s themes. The flavor text, “Unity is the only constant. Individual thought is a flaw,” nails the tone. I gave it song lyrics and it handed me back a legendary creature that got it.
Now, any Magic player looking at J.O.E.L for more than a millisecond will notice it’s a vanilla 4/4, overcosted, with the name in place of some rule or nothing. This is not what is interesting in this one; what is interesting is the fact that it correctly understood the concept of a fan song that is made to be vague and not cite the original material directly, and produced a card that has the name of the enemy faction as the card name, a fitting art, and even a fitting color identity for Magic. For those unaware of the game, the other two colors of the game, green and white, represent both nature and life, and they weren’t chosen. This proved that an LLM could understand abstract things that a program could never realistically do alone.
Verdant Sentinel and J.O.E.L came out of the same pipeline, ran through the same stages, and earned the same confident sign-off. One was a card you could shuffle into a real deck. The other was a beautiful shell. And nothing in the system could tell them apart: from the inside, the triumph and the failure looked identical. When it worked, it produced good Magic cards. When it didn’t, I waited forty-five minutes for a robot whose rules box just said its own name. The problem was never that the pipeline couldn’t make a good card. It was that it couldn’t reliably know when it had.
The prompt that broke it Link to heading
While J.O.E.L was the one that broke intermediate models, larger models could still arrive at something coherent, and this is interesting as the next issue is one that scale couldn’t beat on my hardware, and it made the next step of the story worthwhile.
So here is the kind of request that broke the larger model. There are other types that broke it in different ways, but this is the most classic example I could think of:
A sorcery that represents the butterfly effect: a minor effect now, and an overwhelming effect three turns later.
This is a hard ask. It requires the model to invent a mechanic: a way to express “small now, enormous later” in Magic’s actual rules vocabulary. And the way it failed pointed me at where the wall stood.
The best model I could run got the easy half right every time: the minor immediate effect. But the delayed escalation was always nonsense. It would do things like put three “wind counters” on a target with no rule anywhere defining what those counters did or how anything interacted with them. Sometimes it gestured at “next turn” with no valid mechanic behind it. Every attempt at the escalating effect was mechanically invalid filler.
There it was again, the same failure as the Halberd, just dressed up: the model does the part it can do, and fakes the part it can’t. It couldn’t invent a valid novel mechanic, so it papered over the gap with official-sounding words that meant nothing. And no amount of more stages, more retries, or bigger-models-I-could-run was going to fix that, because it wasn’t a quantity problem. The model didn’t have a reliable internal sense of what makes Magic text valid. It had a reliable sense of what makes Magic text look valid. Those are different, and the gap between them was costing me forty-five minutes a card to not close.
The realization Link to heading
So here’s where I’d backed myself, and the one door left.
The good part of the LLM was real: it understood natural-language requests from humans. Stella the husky commander, song lyrics, the butterfly effect: it could take a vague human vibe and have genuine ideas about it. I didn’t want to lose that. That was the entire point.
The bad part was specific and stubborn: it could not reliably produce mechanically-valid, correctly-structured Magic text. It faked structure it didn’t understand, and, worse, it couldn’t tell when it had.
Those two facts sitting next to each other suggested the move. What if the LLM didn’t have to output finished Magic text at all? What if it produced some intermediate representation: a structured form I could then render into a real card deterministically, the way the MSE file got rendered into an image?
It may sound to you like a step back: going from “the model writes English” to “the model writes an abstraction that another program turns back into English” seems like adding a wheel, not removing one. And it’s even more roundabout than it sounds: you have to parse the output, ensure what the LLM produces deserializes and is valid, but once you have a structured representation, you can render it into any language, not just English, something I’d only fully appreciate later. But that indirection is exactly the win, for one reason: it hands me control over the wording. Magic’s text isn’t free prose. It’s tightly governed by the game’s rules. A card’s text has to follow specific patterns and a specific internal order, or it’s simply invalid. And here’s the part that makes Magic such a brutal test case: players can spot an invalid card instantly. They’ve been trained on the real thing for years, absorbing the exact phrasing and ordering Magic follows until a wrong card feels wrong the way a grammatical error feels wrong in your mother tongue.
You can’t fake your way past that audience. So if I controlled the representation, I could guarantee the structure was valid before it ever reached a player. The same way I’d already guaranteed the visual quality through the MSE renderer.
Underneath, I realized, I’d been doing something much weaker: calling different LLMs in a loop and hoping, then asking the same kind of model whether the result was any good. What I actually wanted was a target the model could aim at, where hitting the target was something I could mechanically check. Not “does this look like Magic,” but “does this parse.” Correctness I could measure, instead of correctness a model felt confident about.
That meant building the representation. A formal language for Magic cards, with a grammar, that a model could be trained to produce and that software could deterministically turn into a real card.
That meant building a compiler.
I had never built a compiler. I figured it would be easy.
Next: I’d reverse-engineered binary game formats, I knew Python’s AST internals, I’d written assembly and built ROP chains as a pentester. None of it prepared me for what happened when I actually tried to build a parser that worked at scale. Article 2: “I’d never built a compiler. Here’s where my instincts were wrong.”