Researchers have been boasting about grabbing headlines but ultimately achieving nothing.
Every day, we see a new chatbot, either developed by large tech companies or recently funded startups, or from the open-source community. Developers have taken many shortcuts in replicating OpenAI's ChatGPT. The most common approach now is to train chatbots using data generated by ChatGPT.
The latest chatbot claims to outperform ChatGPT. OpenChat is a decentralized open-source chat tool that recently scored 105.7% compared to ChatGPT in the Vicuna GPT-4 benchmark test. This is a significant achievement, but upon closer inspection, it is not as it seems.
This is the second model claiming to perform better than ChatGPT in the same Vicuna benchmark test. Previously, Orca, a 13 billion parameter model trained on GPT-4 data, claimed to outperform OpenAI's model.
Same Old Story
Firstly, OpenChat is built on top of LLaMA-13B, which means the model cannot be used for commercial purposes as Meta's LLaMA is only for research purposes and not for commercial use. Additionally, before boasting about the model's benchmark test, another thing to consider is the dataset used for fine-tuning. This LLaMA-based model was trained on 6,000 dialogues out of 90,000 dialogues provided on ShareGPT, a platform for sharing ChatGPT and GPT-4 generations and outputs on the internet.
When it comes to evaluating and benchmarking on the Vicuna GPT-4 benchmark, it only tests style and not the information generated by the model. Furthermore, it is a GPT-based evaluation metric, which means any model trained on ChatGPT or GPT-4 data will receive higher ratings when subjected to GPT testing, making the benchmark test unreliable.
Recently, Hugging Face also found similar issues with other open-source models. The founder of Hugging Face claimed that there is a significant difference between the evaluation benchmarks in the model papers and the evaluation benchmarks on the "Hugging Face" benchmark. David Hinkle, VP of Software Engineering at Securly, pointed out that many models claiming better performance than LLaMA or GPT-4 are nowhere to be found on the Open LLM leaderboard.
All Talk, Little Results
In short, models trained on ChatGPT data perform better when benchmarked against the same model-based metrics. It's like a student taking an exam, rewriting the answers to match the provided correct answers, and then the teacher matching the answers. Obviously, the performance will be better.
Andriy Mulyar from Atlas Nomic AI pointed out that this is all false hype. It is a wrong path to use outputs generated by ChatGPT to imitate ChatGPT. Furthermore, these models only imitate the style of ChatGPT, making the quality of chatbots sound better on individual tasks. If considering a comprehensive evaluation across all types of general tasks, ChatGPT is much better than any other assistant.
Interestingly, researchers have realized the issues with evaluating models on the Vicuna GPT-4 benchmark after these criticisms. Therefore, they have transitioned to MT-bench to test the performance of OpenChat. In this case, the model's performance is significantly lower than ChatGPT based on GPT-3.5, which describes the differences between evaluation benchmarks.
Users on Twitter pointed out that the model is more prone to hallucinations compared to ChatGPT, and not just in terms of evaluation metrics for the model. "I just tried this model, and it's not good at all. Have you tried this model before publishing this article?" one Twitter user said.
Thanks GPT
Regardless of the metrics and benchmarks, high-quality data works wonders for chatbots based on large language models. Therefore, the only model to be thankful for is ChatGPT, as every model today is trained on synthetic data generated by chatbots. No one has the secret weapon developed by OpenAI for GPT. Recently, OpenAI was asked if open-source could replicate what the company has built through Vicuna or LLaMA, and Ilya Sutskever gave a negative answer.
"The trend of this new model outperforming all other models in benchmark tests has been going on for a while, but when evaluated with the same metrics as other models, the 'new model' performs poorly. Additionally, while the open-source community has been trying to replicate ChatGPT, training on ChatGPT data may not be the best approach as OpenAI has faced several lawsuits for training on internet data.
Alphabet Q1 paid subscribers reached 350M, fueled by YouTube & Google One. YouTube ad revenue slightly missed at $9.88B amid Premium growth. Total Q1 revenue hit $109.9B, beating forecasts.
Anthropic releases Claude connectors for Adobe, Blender, and Ableton, enabling AI workflow automation and app control. It also sponsors the Blender Development Foundation.
Apple introduces an App Store model splitting 12-month plans into monthly installments. On iOS 26.4+, it lowers upfront costs without reducing total price, adding billing tracking.
Google launches Aadhaar digital IDs in Wallet for India, enabling secure QR/NFC verification. A phased partner rollout targets key sectors, while passport-based credentials expand globally.
YouTube tests "Ask YouTube," an AI tool blending text and video for complex queries. Initially for US Premium users, it supports interactive follow-ups, expanding Google's AI search.
Google grants US DoD classified AI access, joining OpenAI and xAI in defense deals amid Anthropic's dispute. Contractual safeguards against surveillance and lethal weapons remain unenforceable.
Mistral AI launches Workflows, a Temporal-powered orchestration platform for secure, Python-driven enterprise AI automation featuring human-in-the-loop approvals.