A few days ago, Wizard LM and Phind released a fine-tuned model called WizardCoder 34B based on Code Llama. Currently, they are fiercely debating whether Phind used Wizard LM's wizardcoder-style dataset to train their V1 model. However, Phind has refuted this claim, but the debate continues.
Everyone is rigorously evaluating OpenAI's HumanEval, trying to outperform GPT-4 on various tasks. Two days after Code Llama was released, Wizard LM introduced WizardCoder 34B, a fine-tuned version based on Code Llama. The company proudly claims that WizardCoder 34B performs even better than GPT-4, ChatGPT-3.5, and Claude-2 on HumanEval, with a pass rate of 73.2% on the first attempt.
Wizard LM seems to be attempting to deceive developers by cleverly ignoring the fact that it compared the 73.2% score with the HumanEval rating of GPT-4's March version, rather than the August version calculated by Wizard LM. In the August version, GPT-4 achieved an 82% rating in HumanEval. It is worth noting that according to the GPT-4 Technical Report (2023/03/15), the HumanEval results for GPT-4 and ChatGPT-3.5 were 67.0 and 48.1, respectively.
However, Wizard LM is not the only player in this competition. Another startup, Phind, also claims that their optimized versions, CodeLlama-34B and CodeLlama-34B-python, have pass rates of 67.6% and 69.5%, respectively, on HumanEval using their own Phind dataset. These numbers are almost on par with GPT-4.
The obsession with GPT-4 is evident in the open-source community, which considers it the ultimate benchmark. In any Meta research paper based on LLM, they compare the results with models based on GPT, especially OpenAI's HumanEvals.
Ironically, Meta needs OpenAI, and vice versa. In the paper "Code Llama: Open Foundation Models for Code," the term "GPT" is used 37 times. On the other hand, OpenAI does not use the words "Meta" or "Llama" in their "GPT-4 Technical Report." What would happen if the open-source community no longer compared itself to closed-source models? Clearly, the evaluation metrics created by OpenAI give meaning to the existence of open-source models; otherwise, it would be difficult to assess their performance and status.
In Meta's research paper on Code Llama, they did not use any self-defined evaluation metrics. Apart from HumanEval, the only other metric used was MBPP (Mostly Basic Python Programming) created by Google. Another important thing to note is that GPT-4 does more than just coding tasks. Meta is creating models for specific tasks and trying to surpass GPT-4 in these specific tasks.
If a model is specifically designed for coding, it is likely to outperform GPT-4. Phind's performance is also comparable to GPT-4 on HumanEval. Furthermore, it is highly likely that Code Llama was trained using datasets generated by GPT-4. Otherwise, it would be quite challenging for open-source models to compete with GPT-4.
Is HumanEval sufficient? There has been ongoing discussion on Reddit about whether HumanEval is an appropriate parameter for measuring the efficiency of large language models in coding. This discussion points out that HumanEval, which can solve 160 Python programming problems, is still not sufficient to meet people's requirements, and a single number based on 160 programs cannot fully capture the usage of code models in practical applications.
The discussion further states that factors such as code explanation, docstring generation, code completion, Stack Overflow questions, and writing tests are not captured by HumanEval. A user named X expresses the same view, saying, "Unfortunately, the actual performance of Python code still far exceeds that of GPT-4." "I tried different real examples to create the smallest Flask microservice, and GPT-4 still outperforms all open-source LLMs," he adds, praising GPT-4's ability to be used in the real world.
Interestingly, Can Xu, a senior researcher at Wizard LM, responded that he would look into it and try to improve the model. Xu said, "Thank you for pointing out areas for potential improvement; we will soon provide examples in real-life scenarios."
In another conversation, a user named X mentions that these model benchmarks often do not measure the performance of models in actual workflows. Michael Royzen, co-founder of Phind, responds to this by saying that this is an early experiment aimed at reproducing (and surpassing) the results of the "Unnatural CodeLlama" mentioned in the paper. He says that more work needs to be done in the future to make these models production-ready. Royzen optimistically adds, "In the future, we will have a mix of expert Code Llama models that I believe will be competitive in real-world workflows."
While open-source models may not have reached the standards of GPT-4 yet and are striving to catch up, it is encouraging that they are openly discussing and acknowledging their shortcomings with the community. The discussions between Wizard LM and Phind on X are a good sign that the open-source community is dedicated.
This transparency in the open-source community is a positive step towards "responsible AI." In contrast, OpenAI keeps its trade secrets hidden, leaving everyone to speculate about their upcoming plans.
Alphabet Q1 paid subscribers reached 350M, fueled by YouTube & Google One. YouTube ad revenue slightly missed at $9.88B amid Premium growth. Total Q1 revenue hit $109.9B, beating forecasts.
Anthropic releases Claude connectors for Adobe, Blender, and Ableton, enabling AI workflow automation and app control. It also sponsors the Blender Development Foundation.
Apple introduces an App Store model splitting 12-month plans into monthly installments. On iOS 26.4+, it lowers upfront costs without reducing total price, adding billing tracking.
Google launches Aadhaar digital IDs in Wallet for India, enabling secure QR/NFC verification. A phased partner rollout targets key sectors, while passport-based credentials expand globally.
YouTube tests "Ask YouTube," an AI tool blending text and video for complex queries. Initially for US Premium users, it supports interactive follow-ups, expanding Google's AI search.
Google grants US DoD classified AI access, joining OpenAI and xAI in defense deals amid Anthropic's dispute. Contractual safeguards against surveillance and lethal weapons remain unenforceable.
Mistral AI launches Workflows, a Temporal-powered orchestration platform for secure, Python-driven enterprise AI automation featuring human-in-the-loop approvals.