Mistral AI Releases First Model Mistral 7B, Surpassing Llama 2 13B

The unique Word Art logo and record-breaking $118 million seed funding of French startup Mistral AI have attracted widespread attention. The company has released its first large-scale language AI model, Mistral 7B.

This 7.3 billion parameter model surpasses larger competitors, including Meta's Llama 2 13B (one of Meta's newer and smaller models), and is claimed to be the most powerful language model of its size to date.

It can handle English tasks while also providing natural encoding capabilities, offering a variety of use cases for businesses.

Mistral has announced that it will open-source this new model under the Apache 2.0 license, allowing anyone to fine-tune and use it without restrictions, including for enterprise cases.

Introducing Mistral 7B

Mistral AI was founded earlier this year by individuals from Google DeepMind and Meta, with a mission to provide useful AI for businesses by leveraging publicly available data and customer-contributed data.

Now, with the release of Mistral 7B, the company has embarked on this mission, providing teams with a small-scale model for low-latency text summarization, classification, text completion, and code completion.

Although the model has just been released, Mistral AI claims that it has already surpassed open-source competitors. In benchmark tests covering various tasks, the model easily outperformed Llama 2 7B and 13B.

For example, in a large-scale multi-task language understanding (MMLU) test covering 57 subjects such as mathematics, American history, computer science, and law, the new model achieved an accuracy rate of 60.1%, compared to slightly above 44% and 55% for Llama 2 7B and 13B, respectively.

Similarly, in tests covering common-sense reasoning and reading comprehension, Mistral 7B achieved accuracy rates of 69% and 64%, surpassing both Llama models. The only model that matched Mistral 7B's performance was Llama 2 13B in the world knowledge test, which Mistral claims may be due to the limited number of parameters in the model, restricting the amount of knowledge it can compress.

In a blog post, the company wrote, "To make accurate comparisons, we re-evaluated all metrics of all models using our evaluation process. Mistral 7B clearly outperforms Llama 2 13B on all metrics and is on par with Llama 34B in many benchmark tests."

As for coding tasks, although Mistral claims that this new model is "far superior" to other models, benchmark test results show that it still falls short of the fine-tuned CodeLlama 7B. In the 0-shot Humaneval and 3-shot MBPP (manual validation subset) tests, the Meta model achieved accuracy rates of 31.1% and 52.5%, respectively, while Mistral 7B's accuracy rates were close, at 30.5% and 47.5%.

High-performance small-scale models could benefit businesses

While this is just the beginning, Mistral has demonstrated the ability of a small-scale model to deliver high performance across multiple tasks, which could bring significant benefits to businesses.

For example, in MMLU, Mistral 7B exhibits three times the performance (23 billion parameters) of Llama 2. This directly saves memory and provides cost-effectiveness without compromising the final output.

The company claims to achieve faster inference by using Group Query Attention (GQA) and handles longer sequences at a lower cost with Sliding Window Attention (SWA).

The company plans to further refine this work by releasing a larger model that can perform better inference and support multiple languages, expected to debut in 2024.

Currently, Mistral 7B can be deployed anywhere, from on-premises to AWS, GCP, or Azure clouds, using the company's reference implementation and vLLM inference server and Skypilot.