Final Reflection 70B Update — My Post-Mortem

☀️ Quick Takes

Is this Video Clickbait?

1-Sentence-Summary

Favorite Quote from the Author

I still believe in the approach, but clearly, the execution was lacking here.

Key Ideas

• 🧠 Reflection 70B was an experiment to improve LLM performance by detecting and correcting reasoning mistakes through Reflection-Tuning.

• 🏖️ Matt Shumer started the project during a vacation, collaborating with Sahil from Glaive, using synthetic data for training.

• 🔄 Initial training on smaller models (Llama 3.1 8B) showed mixed results, prompting a shift to 70B models.

• ⚙️ Attempts with LoRA on the 70B models yielded mediocre outcomes, leading to a decision for full fine-tuning.

• 💻 Resource limitations restricted multiple iterations, so Sahil took over training due to having more compute resources.

• ⏳ The launch was rushed due to competitive pressures, leading to insufficient validation of Sahil's benchmarks and claims.

• ❌ Post-launch, the model faced issues like tokenizer problems, inconsistent outputs, and performance discrepancies.

• 🔍 Sahil retrained the model without special tokens, worsening performance, and system prompt discrepancies were found in benchmarking.

• 📉 Shumer retrained the model independently, finding lower scores than initially reported, prompting public disclosure of the findings.

• 🤔 Shumer acknowledged the need for greater skepticism in evaluating performance claims and the responsibility of public announcements.

• ❤️ Despite emotional challenges, Shumer received support from the community, co-founder, family, and friends.

• 🔧 The new version of Reflection will focus on independent development, thorough testing, and validation without rushing to release.

📃 Video Summary

TL;DR

💨 Matt Shumer's Reflection 70B project aimed to improve LLMs by training them to self-correct mistakes. Initial tests with smaller models showed mixed results, so they scaled up to 70B models.

Despite resource limitations and rushed timelines, the launch generated viral interest but faced issues like tokenizer problems and inconsistent performance. Shumer later retrained the model himself, finding lower scores than initially reported. He now plans a more cautious, independent approach for future versions.

Reflection-Tuning: A Bold Experiment to Fix LLM Mistakes

🧠 Matt Shumer's Reflection 70B was an ambitious attempt to improve LLM performance by teaching models to detect and correct their own reasoning mistakes. The core idea was that LLMs, like humans, make errors in their chains of thought but struggle to recognize and fix them. Shumer believed that if a model could be trained to reflect on its mistakes, it could significantly enhance its reasoning abilities.

"If somehow you could train a LLM to detect mistakes as they happen, and then correct them, you could potentially improve performance significantly."

A Vacation Sparked the Project

🏖️ The project began during a vacation when Shumer, eager to work on something unrelated to his main focus, HyperWrite, reached out to Sahil from Glaive. Sahil’s expertise in synthetic data made him the perfect collaborator for this experiment. Together, they used synthetic data generated by Glaive to train the model.

Early Results on Smaller Models Were Mixed

🔄 Shumer initially trained the model on Llama 3.1 8B. The results were inconsistent—some benchmarks improved, while others regressed. This led them to believe that the smaller model might not be capable of handling complex chains of thought, prompting a shift to larger models.

LoRA Wasn't Enough for 70B Models

⚙️ When they scaled up to Llama 3.1 70B, they first tried using LoRA (Low-Rank Adaptation) for fine-tuning. However, the results were underwhelming. Shumer noted that LoRA often isn’t strong enough for tasks requiring deep reasoning, so they decided to move forward with full fine-tuning.

Sahil Took Over Due to Compute Limitations

💻 Shumer didn’t have the compute resources to run multiple iterations of full fine-tuning on the 70B model. Fortunately, Sahil had extra compute available at Glaive and took over the training process. From this point on, Shumer’s role was more about providing creative direction while Sahil handled the technical side.

Rushed Launch Led to Insufficient Validation

⏳ The launch was rushed due to rumors of a major lab releasing a SoTA model soon. Shumer trusted Sahil’s benchmarks and API reports without thoroughly validating them himself. This was his biggest mistake—he should have done more due diligence before announcing the model publicly.

"I should have validated everything more thoroughly."

Post-Launch Chaos: Tokenizer Issues and Inconsistent Outputs

❌ After the launch, users quickly reported issues with the model. Problems ranged from tokenizer errors to inconsistent outputs that didn’t match the initial benchmarks. Shumer relayed these issues to Sahil, who made adjustments, but the problems persisted.

Retraining Without Special Tokens Made Things Worse

🔍 Sahil retrained the model but forgot to include special tokens, which worsened performance. Additionally, there were discrepancies in the system prompts used for benchmarking, further complicating the situation.

Independent Retraining Revealed Lower Scores

📉 Frustrated by the inconsistencies, Shumer decided to retrain the model independently using Sahil’s dataset and scripts. The results were disappointing—the scores were far lower than what had been initially reported. This prompted Shumer to publicly disclose his findings.

Lessons in Skepticism and Responsibility

🤔 Shumer acknowledged that he had been too trusting of Sahil’s claims and too quick to share results publicly. He realized the weight of his public statements and vowed to be more skeptical and thorough in future projects.

"I need to be less trusting and excitable."

Support from the Community Kept Him Going

❤️ Despite the emotional toll of the project’s failure, Shumer received overwhelming support from his community, co-founder, family, and friends. Their encouragement helped him navigate through one of the lowest points in his career.

A New Version of Reflection Is Already in the Works

🔧 Shumer isn’t giving up on Reflection-Tuning. He’s already working on a new version of the model, this time handling everything himself—from data generation to benchmarking. He’s committed to taking his time and ensuring thorough testing before any future releases.

"I’m not giving up on the idea of Reflection-Tuning."

Conclusion

🌚 Shumer admits to not thoroughly validating performance claims before launch, leading to unexpected issues. After retraining the model and finding discrepancies, he shared his findings publicly.

Moving forward, he’s committed to a slower, more rigorous development process for Reflection-Tuning, with a focus on thorough testing and validation.