I wanted to test a simple idea: can a small coding model become more useful by learning from its own attempts between sessions? I used tasks from a real project rather than a synthetic library.
Overnight, five variants of the same model produced 235 fresh responses. None solved any of the seven held-out coding tasks. This is a preliminary negative result. The useful part was finding out how much the headline score hides.
What counted as experience
The model was Qwen3.5-4B. The project was Goose Mini Browser, written in Goose, a language distinct from Go. Ten tasks were used for training; seven were reserved for evaluation: three primary tasks and four transfer tasks within the same repository.
- Baseline: the original model without fine-tuning or persistent memory.
- SFT: training on reference corrections.
- Own experience: attempts from its own run, feedback, and a final correction.
- Other experience: comparable material from a different run.
- RAG: the original model with external memory drawn from permitted experience.
The SFT control matters. Beating the baseline after seeing reference answers would not, by itself, show that learning from mistakes helped. In the experience conditions, the earlier failed attempt was masked context; the loss supervised only the final correction.
The overnight comparison
Every method answered the same seven coding tasks and 40 general-skill questions afresh: 47 responses each, 235 overall. I used seed 42, greedy decoding, and the final cycle-three adapters from fixed experience banks. The output budgets were 4096 new tokens for a correction and 384 for a control answer.
The larger study planned three seeds and a separate adaptive phase. I reduced the correction budget from 9216 after observing the original baseline and a slow partial SFT run. This overnight branch is therefore not a new independent confirmatory study. Its conditions and results are kept separate.
The model received pre-localized code and returned strict JSON describing exact replacements. This tested patching in a supplied context, not autonomous repository navigation.
Zero solved tasks, different failure paths
All five methods scored 0/3 on the primary tasks and 0/4 on transfer. Seven tasks and a shared floor do not establish equivalence.
Before execution, the baseline had six accepted edits out of seven. SFT had two, own experience one, and other experience and RAG none. Rejection included malformed JSON and invalid exact replacements: an empty old fragment, a no-op, or a duplicate key.
Three accepted baseline edits failed compilation or execution. Three ran but failed the task checks. The accepted SFT and own-experience edits also failed the task checks. SFT reached the generation cap twice; each experience variant did so three times. A cap hit alone does not prove truncation before the answer ended. RAG had no cap hits and still produced no accepted edits.
The control score improved, with some forgetting
On the 40 control questions, baseline scored 17, SFT 21, own experience 24, other experience 24, and RAG 18: 42.5%, 52.5%, 60%, 60%, and 45%.
These questions came from an earlier development set. They are neither an independent broad benchmark nor evidence of newly acquired Goose knowledge.
Compared with baseline, own experience gained nine correct answers and lost two; other experience gained eight and lost one; RAG gained five and lost four. SFT gained four without losses.
In the direct own-experience versus SFT comparison, five questions favored experience and two favored SFT. A net three answers out of forty is a reason to repeat the comparison, not an established method advantage. Matching totals for own and other experience do not establish equivalence either.
Did training actually work?
After the run, I audited the training data and masks without retuning on the held-out tasks. The 27 training runs contained 90 records: ten distinct tasks repeated across three conditions and three seeds, not ninety independent tasks.
All 90 target answers parse as strict JSON and apply exactly to their training snapshots. Re-encoding reproduced the original hashes of input tokens, labels, and attention masks for every record. Supervision covered the final correction, not the preceding failed attempt. All 27 logs show finite loss and 256 changed tensors. This supports that training happened; it does not establish generalization.
Two hypotheses remain. Experience branches trained on five-message conversations in which a correction follows failure and feedback; evaluation asked for a first answer. That context mismatch could matter, but it cannot alone explain SFT degradation because SFT did not have the same mismatch.
Two training targets were also longer than the overnight output budget: 8604 and 5303 tokens versus 4096. Every held-out reference fits within the budget. Length deserves a controlled development test, not a causal claim from these observations.
Runtime, and Dota
Successful worker processes took about 4 hours 59 minutes in total. Generation was about 1 hour 49 minutes within that total. These figures must not be added.
Each task used a fresh worker. The remainder includes hashing, imports, loading, teardown, and monitoring, not just model loading. Preparation, training, waiting for a game, and some failed work are excluded. This is not total research cost.
When Dota started, the active worker stopped; after the game exited, the same task was retried. The interrupted response was excluded. I did not measure FPS, game latency, or memory-reclamation delay, so this does not establish that the game was unaffected.
A later diagnostic probe failed before model loading because Windows Application Control blocked a pandas module. It produced no new responses and does not explain the completed overnight results. The CPU audit succeeded without disabling Windows protection. GPU compatibility for continuation remains untested.
What I can claim
In these conditions, I did not observe useful transfer to held-out coding tasks. Trained branches more often produced unusable edits. Aggregate control-question scores increased, while some previously correct answers were lost.
I cannot conclude that models cannot learn from experience, or that weights are better or worse than external memory. One seed, seven tasks, and one repository are insufficient. Exposure to the project's public history during pretraining is unknown.
Data separation was checked through input packets and saved hashes. The solver had no tools, but the trusted Python host had filesystem access. This was not complete technical isolation of knowledge.
The next step is to test response format and the training-context mismatch on development tasks, freeze the changes in advance, and then repeat without tuning on these seven tasks. The full study is paused. When experience belongs in weights versus external memory remains a possible conclusion of future work, not a question this run settled.
Results and reproducibility
Aggregate overnight results contain the comparison conditions and chart counts. This is a public summary, not a self-contained training reproduction. Full reports, input hashes, and CPU-audit records are preserved in the local research archive and are not part of this publication.
Model revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a. This is a completed preliminary overnight comparison of five methods, not the completion of the full research program.