Hi! The goal of this project was to see whether an agent can learn a skill without changing its weights, just by keeping notes on what worked, and how that compares to actually fine-tuning on the same experience.
I picked capture-the-flag (CTF) challenges as the domain for one reason: a flag is either right or wrong. That matters a lot here. A memory only gets written after a verified solve, never because the agent thinks it did well, so the agent can't fool itself into "learning" from a wrong answer.
The Setup
One ReAct agent (think, run a shell command, read the output, repeat) is frozen, and only the learning part changes between arms:
- A: no memory, the baseline
- B: paste raw past solves into the prompt
- C: retrieve the most similar past episodes
- D: an ACE-style playbook, where a reflector turns solves into short strategy notes and a curator keeps the playbook tidy (the one I built the study around)
- E: QLoRA fine-tuning on the verified solves
- F: E's weights plus D's playbook
The challenges are 12 small, self-contained puzzles across 5 families (encodings, classic ciphers, XOR, forensics, reversing), all local with synthetic flags. Each family has a "sibling" set: same trick, different surface (base64 → base32, Caesar → Atbash), which is the real test of whether a lesson transfers. Flags are redacted from every memory and from the training data, and the training data is decontaminated against the test challenges.
First Run: Too Easy
I started on the 27B model I already had loaded, and it solved everything. Every arm, every seed, every challenge, in about 4 turns. Success can't go up from 100%, so the only thing left to measure is how fast each arm solves:
The result was the opposite of what I expected. Raw examples (B) and episodic retrieval (C) each save a full turn on same-family challenges (CI [+0.20, +1.80]), while my playbook (D) saves only +0.40, and its interval touches zero. Flip to the sibling families and almost everything washes out.
My best guess at why: the heuristic reflector writes notes like this real one from the playbook:
"For crypto/encoding: identify the encoding and decode it. Typical tool path: read -> shell -> submit."
That's true, but not very actionable. A raw example hands the agent the exact command that worked. A concrete recipe beats an abstract lesson, at least for a model this strong.
Second Run, a Model That Fails
So I needed a model that actually fails some tasks. All six arms went on a matched Qwen2.5 7B base, including fine-tuning E on that same base (0.44 GPU-hours of QLoRA on one 3090):
| Arm | Same family | Sibling family |
|---|---|---|
| A no memory | 40% (n = 10) | 60% (n = 10) |
| B raw examples | 30% (n = 10) | 50% (n = 10) |
| C episodic retrieval | 50% (n = 10) | 30% (n = 10) |
| D playbook | 50% (n = 10) | 60% (n = 10) |
| E QLoRA weights | 80% (n = 5) | 60% (n = 5) |
| F weights + playbook | 60% (n = 5) | 60% (n = 5) |
The baseline solves 40% of same-family challenges. Fine-tuned weights (E) hit 80%, the biggest jump by far. Flip the first chart to the 7B base, though, and E's interval is [0, +0.8], so it's not clearly better than doing nothing. The memory arms land between −0.10 and +0.10, and on sibling challenges every arm ties or trails the baseline's 60%.
And E's 80% needs a few asterisks:
- E and F got 20 turns and longer outputs per step, while A–D got 15 (the serve jobs weren't equalized)
- E's training data came from a single source challenge, 64 examples total
- E and F have one seed, A–D have two
So the honest headline is: at this scale, memory and weights aren't distinguishable on success, and my structured memory lost to plain examples on efficiency. Memory is free to learn (zero training GPU-hours), weights cost about half a GPU-hour, and neither clearly wins.
Things That Went Wrong
The fine-tuned model was trained, then evaluated, and every request returned a 404. The model server read the relative path to the adapter as a registry URL, the eval script treated the failures as "best-effort," and the job reported success anyway. That's why every long job in my loop now has to name the files it's supposed to produce and fails if they aren't there.
The other lesson was retries. The 7B job got retried about 8 times while I fixed memory limits and exit codes, and the analysis pooled all of them, so some arms looked like they had 8 copies of each seed. Those runs are now kept aside (with a README saying why) and only the last complete set counts.
Conclusions and Future Improvement
A negative result for the arm I built, but a clear pointer to the next experiment:
- An LLM reflector that writes concrete command recipes instead of abstract bullets
- Equalize the turn budget and run E/F on more seeds
- Build E's training set from more than one challenge
- Run the bigger InterCode benchmark (the Docker adapter is written, but those runs never happened)
Code, the full report, and every run log: github.com/michael951021/mem-vs-ft-ctf.