Today's AI/ML headlines are brought to you by ThreatPerspective

Is Reflection 70B the most powerful open-source LLM or a scam?

HyperWrite founder and CEO Matt Shumer announced that his new model, Reflection 70B, uses a simple trick to solve LLM hallucinations and delivers impressive benchmark results that beat larger and even closed models like GPT-4o. Shumer collaborated with synthetic data provider, Glaive, to create the new model which is based on Meta’s Llama 3.1-70B Instruct model. In the launch announcement on Hugging Face, Shumer said. “Reflection Llama-3.1 70B is (currently) the world’s top open-source LLM, trained with a new technique called Reflection-Tuning that teaches a LLM to detect mistakes in its reasoning and correct course.” If Shumer found a way

The post Is Reflection 70B the most powerful open-source LLM or a scam? appeared first on DailyAI.
Reflection playground is down for now. Source: Reflection Playground
Some users questioned the impressive benchmarks. The GSM8K of over 99% looked especially suspect.

Hey Matt! This is super interesting, but I’m quite surprised to see a GSM8k score of over 99%. My understanding is that it’s likely that more than 1% of GSM8k is mislabeled (the correct answer is actually wrong)!

Hugh Zhang (@hughbzhang) September 5, 2024