AWS Launches New Code Generation Model, Joins Amazon SageMaker JumpStart Platform

AWS recently announced that there are two new base models available on Amazon SageMaker JumpStart: Code Llama and Mistral 7B. Users can deploy these models with just a click, providing AWS users with private inference endpoints for code generation tasks. Code Llama is an optimized version of Meta's Llama 2 base model, using the same license. It has three variants: Basic, Python, and Instruct, with each variant having three model sizes: 7B, 13B, and 34B parameters, totaling nine options. In addition to code generation, it can also perform code completion, while the Instruct model can operate based on chat-like natural language instructions. Mistral 7B is a large language model (LLM) with 7 billion parameters, released under the Apache 2.0 license. Mistral 7B has two variants: Basic and Instruct. In addition to code generation, Mistral 7B is a general-purpose text generation model, performing similarly to Code Llama 7B and outperforming the larger Llama 2 13B base model in all natural language processing benchmarks. According to AWS: "Today, we are excited to announce the availability of the Code Llama base model developed by Meta through Amazon SageMaker JumpStart, enabling customers to perform inference with a one-click deployment. Code Llama is an advanced large language model (LLM) capable of generating code and natural language from code and natural language prompts. Code Llama is free for research and commercial use. You can try running this model using SageMaker JumpStart, a machine learning (ML) hub that provides access to algorithms, models, and ML solutions, allowing you to quickly get started with ML." The Mistral 7B model supports a maximum context length of 8k tokens. This longer context can be used for "few-shot" context learning in tasks such as question answering or retaining chat history. The Instruct variant supports a special multi-turn prompt format: <s>[INST] {user_prompt_0} [/INST] {assistant_response_0} </s><s>[INST] {user_prompt_1} [/INST] Different sizes of the Code Llama model support different context lengths: 10k, 32k, and 48k respectively. However, the 7B model only supports 10k tokens on the ml.g5.2xlarge instance type. All models can perform code generation, but only the 7B and 13B models can perform code completion. This task guides the model to generate code to be placed between provided code prefixes and code suffixes. In the prompt, there are special input tokens <PRE>, <SUF>, and <MID> to mark their positions in the prompt. The models can accept these parts in two different orders: suffix-prefix-middle (SPM) and prefix-suffix-middle (PSM). Meta's paper on Code Llama suggests using the PSM format when "the prefix does not end with a space or token boundary". The PSM format is as follows: <PRE> {prefix_code} <SUF>{suffix_code} <MID> The Instruct version of Code Llama is designed for conversational-like interactions and, according to Meta, "significantly improves performance" on multiple NLP benchmarks, despite the moderate cost of code generation. One example application of this model is generating and explaining code-based solutions to questions posed in natural language, such as how to accomplish specific tasks using Bash commands. Code Llama Instruct adopts a similar special prompt format as Mistral 7B and offers options for "system" prompts: <s>[INST] <<SYS>> {system_prompt} <</SYS>> The announcement for Code Llama mentions that these models are available in the US East (Northern Virginia), US West (Oregon), and Europe (Ireland) regions. AWS has not yet announced the availability of Mistral in any regions.