More Gemini Features Coming to Google TV
Google TV integrates Gemini AI for image/video generation, enhances Google Photos, and adds a dedicated YouTube Shorts feed to the homepage.
Read moreWhat is this about and why is it important?
Large language models (LLMs) such as Anthropic's Claude or Meta's LLaMA2 have demonstrated impressive capabilities across various natural language tasks. However, their knowledge and task-specific skills are relatively general—if you want to perform more specialized domain-specific tasks that require explicit knowledge, you need to find ways to inject knowledge into the models and teach them task-specific behaviors. LLM-driven applications need to function properly within the target domain, provide accurate answers instead of generating illusions, and ensure safety, privacy, and appropriate content.
These challenges are often referred to as the "three Hs"—helpfulness, honesty, and harmlessness. Overcoming these challenges is particularly important when designing high-quality FM-driven applications.
There are several options for imparting domain knowledge to the base models:
1. Source knowledge: You can dynamically provide knowledge through prompt engineering, generation with context integration, or example-based approaches. This option presents in a relatively straightforward manner, such as zero-shot or few-shot prompts. However, there are also more complex methods for integrating evolving external knowledge sources. The approach of injecting knowledge through dynamic prompt composition, obtaining information from various external "tools" such as retrieval-augmented generation, database retrievers, data lake retrievers, real-time systems via RESTful APIs, etc., is becoming increasingly popular.
2. Parameterized knowledge: You can directly update the model parameters through fine-tuning. By fine-tuning the pretrained weights on a smaller (compared to pretraining) domain-specific dataset, a domain-specific knowledge base or task can be formed.
As explained by Heiko Hotz in a blog post, there is a trade-off in choosing the right method (or combination of both). In this article, we will focus on the parameterized approach and demonstrate how to fine-tune the LLaMA2 model using PEFT (Parameter-Efficient Fine-Tuning) on Amazon SageMaker.
Our goal is to adapt LLaMA2 to a specific domain, acquire the latest knowledge to address the "knowledge truncation" problem where the models lack awareness of recent information that is not part of their training data. This is a more achievable task for many practitioners compared to fine-tuning for specific tasks, as they can simply use a text corpus containing domain-specific information as the training dataset instead of manually creating or collecting datasets specific to a particular task, such as conversational or instructional datasets.
Since having a model fine-tuned for specific tasks is beneficial for many relevant LLM-driven use cases, we will also demonstrate that the proposed setup is equally applicable to models that have already undergone fine-tuning for specific tasks, such as LLaMA2-chat, without losing their task-specific characteristics (such as following instructions, conversational behavior, etc.).
By demonstrating this end-to-end workflow of knowledge injection, we provide a practical guide for adapting the base model to meet specific requirements.
Model
The LLaMA2 model was released in July 2023, accompanied by a research paper. In the paper, Touvron et al. state that LLaMA2 is "a collection of large language models (LLMs) pretrained and fine-tuned with scales ranging from 7 billion to 700 billion parameters. Our fine-tuned LLM models are referred to as Llama 2-Chat, optimized for conversational use cases. Our models outperform most open-source chat models we tested and are a suitable alternative to closed-source models according to our human evaluations of their helpfulness and safety."
As mentioned above, the LLaMA2 model comes in sizes of 7B, 13B, and 70B. They can be used both as pure completion models and optimized for conversational use cases. After pretraining with approximately 20 trillion tokens, they support a context length of up to 4096 tokens. Fine-tuning for conversational use cases was performed with over 100,000 examples and further optimized using over 1 million training samples representing human preferences.
In AWS, LLaMA2 models can be easily deployed using SageMaker JumpStart or obtained from the HuggingFace model hub using AWSxHuggingFace LLM DLC.
Fine-tuning
Fine-tuning effectively injects domain-specific knowledge into base models like LLaMA2 through transfer learning. The process updates the model's weights by continuously pretraining on domain-specific data while keeping the overall network structure unchanged. Compared to full pretraining, which requires large-scale datasets and computational resources, fine-tuning offers higher sample and computational efficiency. Parameter-Efficient Fine-Tuning (PEFT) techniques, such as (q)LoRA methods, can lightweightly inject domain-specific knowledge into general language models like LLaMA2 with minimal overhead.
When discussing fine-tuning, there are two different approaches:
1. Fine-tuning the decoder architecture of LLaMA2 with domain-specific corpora using a locally autoregressive causal language modeling (CLM) approach, which allows injecting domain-specific knowledge or specific language/vocabulary (e.g., medical research terms/speech) into LLM while preserving its general language capabilities.
2. On the other hand, fine-tuning the model with task-specific corpora allows fine-tuning the model for specific tasks such as following instructions or conversational behavior.
These models leverage self-supervised training methods by optimizing loss functions specific to language modeling (LM). Only-decoder models like LLaMA2 adopt the causal language modeling (CLM) approach with unidirectional context. Simply put, they are trained by self-regressively predicting subsequent tokens based on preceding tokens as semantic context.
As mentioned above, PEFT techniques can lightweightly inject domain-specific knowledge by updating a portion of the model's parameters with minimal overhead. Methods like Low-Rank Adaptation (LoRA) or Quantized Low-Rank Adaptation (QLoRA) freeze the weights of the pretrained model and inject trainable rank-decomposed matrices into each layer of the Transformer architecture, significantly reducing the number of trainable parameters for downstream tasks.
Scenario
On July 26, 2023, AWS announced various features in the field of generative AI at the AWS Summit in New York.
Given the knowledge cutoff of LLaMA2, unfortunately, the pure LLaMA2 models cannot provide any information about these announcements. We hope to inject this knowledge into the base models (FM) and elevate their knowledge level to enable them to do so.
As we do not intend to fine-tune the models for specific tasks but rather absorb domain-specific knowledge, we can adopt the classic CLM-based approach.
To be responsible with resources, we will fine-tune the LLaMA2-13b and LLaMA2-13b-chat models. Fine-tuning for the 7b and 70b versions will also be operated with adjusted training cluster configurations. After fine-tuning, we will deploy the base models as well as the fine-tuned models and perform advanced performance comparisons.
Subsequently, we will walk through the detailed operations of each fine-tuning model step.
Data Loading
We use LangChain's WebBaseLoader to load data from an array of websites identified by URLs.
loader = WebBaseLoader(["https://aws.amazon.com/blogs/aws/preview-enable-foundation-models-to-complete-tasks-with-agents-for-amazon-bedrock/", "https://aws.amazon.com/blogs/aws/aws-entity-resolution-match-and-link-related-records-from-multiple-applications-and-data-stores/", "https://aws.amazon.com/blogs/database/the-role-of-vector-datastores-in-generative-ai-applications/", "https://aws.amazon.com/blogs/big-data/introducing-the-vector-engine-for-amazon-opensearch-serverless-now-in-preview/", "https://aws.amazon.com/blogs/big-data/build-data-integration-jobs-with-ai-companion-on-aws-glue-studio-notebook-powered-by-amazon-codewhisperer/", "https://aws.amazon.com/blogs/aws/new-amazon-ec2-p5-instances-powered-by-nvidia-h100-tensor-core-gpus-for-accelerating-generative-ai-and-hpc-applications/"])
data = loader.load()
License Agreement and HuggingFace Login
Before starting data preparation and training, we need to accept the license agreement for the LLaMA2 model. This involves registering an email address on the Meta website matching your HuggingFace account.
Then, we authenticate with the HuggingFace hub at runtime.
huggingface-cli login --token ***enter huggingface token***
Data Preparation
First, we perform some preprocessing on the raw websites. In a real-world application, more emphasis can be placed on this stage. For demonstration purposes, we simply remove all larger space connections to obtain a coherent and fluent text corpus. Then, we load the list of documents into a HuggingFace dataset.
from datasets import Dataset
def strip_spaces(doc):
return {"text": doc.page_content.replace(" ", "")}
stripped_data = list(map(strip_spaces, data))
dataset = Dataset.from_list(stripped_data)
In the next step, we tokenize the text corpus to make it suitable for LLM processing. For this, we use the LLaMA2 tokenizer loaded from the HuggingFace Hub. The loaded tokenizer, based on the model's context window (2048 tokens), can be used to tokenize the created batches.
from itertools import chain
from functools import partial
from transformers import AutoTokenizer
model_id = "meta-llama/Llama-2–13b-chat-hf" # alternatively "meta-llama/Llama-2–13b-hf"
tokenizer = AutoTokenizer.from_pretrained(model_id,use_auth_token=True) # previous authentication with HuggingFace needed, see here https://huggingface.co/docs/huggingface_hub/main/en/package_reference/login
tokenizer.pad_token = tokenizer.eos_token
# empty list to save remainder from batches to use in next batch
remainder = {"input_ids": [], "attention_mask": [], "token_type_ids": []}
def chunk(sample, chunk_length=2048):
# define global remainder variable to save remainder from batches to use in next batch
global remainder
# Concatenate all texts and add remainder from previous batch
concatenated_examples = {k: list(chain(*sample[k])) for k in sample.keys()}
concatenated_examples = {k: remainder[k] + concatenated_examples[k] for k in concatenated_examples.keys()}
# get total number of tokens for batch
batch_total_length = len(concatenated_examples[list(sample.keys())[0]])
# get max number of chunks for batch
if batch_total_length >= chunk_length:
batch_chunk_length = (batch_total_length // chunk_length) * chunk_length
# Split by chunks of max_len.
result = {
k: [t[i : i + chunk_length] for i in range(0, batch_chunk_length, chunk_length)]
for k, t in concatenated_examples.items()
}
# add remainder to global variable for next batch
remainder = {k: concatenated_examples[k][batch_chunk_length:] for k in concatenated_examples.keys()}
# prepare labels
result["labels"] = result["input_ids"].copy()
return result
# tokenize and chunk dataset
lm_dataset = dataset.map(
lambda sample: tokenizer(sample["text"]), batched=True, remove_columns=list(dataset.features)
).map(
partial(chunk, chunk_length=2048),
batched=True,
)
# Print total number of samples
print(f"Total number of samples: {len(lm_dataset)}")
Then, we save the dataset on S3 for use in the training job.
# save train_dataset to s3
training_input_path = f's3://{sess.default_bucket()}/processed/llama/genai-nyc-summit/train'
lm_dataset.save_to_disk(training_input_path)
print("uploaded data to:")
print(f"training dataset to: {training_input_path}")
Training
Now, we can trigger an Amazon SageMaker training job to perform CLM-tied QLoRA fine-tuning based on the preprocessed data.
The hyperparameters and training script are adjusted based on Philipp's blog post. The only exception is the number of training epochs performed, where we choose a relatively high number of epochs, i.e., 20. This is because our training dataset is relatively small (around 32k tokens). In a real-world application, it is recommended to use larger datasets for fine-tuning.
Then, we can execute the training job, which will perform fine-tuning and save our model files to S3.
from sagemaker.huggingface import HuggingFace, HuggingFaceModel
from huggingface_hub import HfFolder
# define Training Job Name
job_name = f'huggingface-qlora-{model_id.replace("/", "-")}-{time.strftime("%Y-%m-%d-%H-%M-%S", time.localtime())}'
# hyperparameters, which are passed into the training job
hyperparameters ={
'model_id': model_id, # pre-trained model
'dataset_path': '/opt/ml/input/data/training', # path where sagemaker will save training dataset
'epochs': 20, # number of training epochs
'per_device_train_batch_size': 2, # batch size for training
'lr': 2e-4, # learning rate used during training
'hf_token': HfFolder.get_token(), # huggingface token to access llama 2
'merge_weights': True, # wether to merge LoRA into the model (needs more memory)
}
# create the Estimator
huggingface_estimator = HuggingFace(
entry_point = 'run_clm.py', # train script
source_dir = 'scripts', # directory which includes all the files needed for training
instance_type = 'ml.g5.4xlarge', # instances type used for the training job
instance_count = 1, # the number of instances used for training
base_job_name = job_name, # the name of the training job
role = role, # Iam role used in training job to access AWS ressources, e.g. S3
volume_size = 300, # the size of the EBS volume in GB
transformers_version = '4.28', # the transformers version used in the training job
pytorch_version = '2.0', # the pytorch_version version used in the training job
py_version = 'py310', # the python version used in the training job
hyperparameters = hyperparameters, # the hyperparameters passed to the training job
environment = { "HUGGINGFACE_HUB_CACHE": "/tmp/.cache" }, # set env variable to cache models in /tmp
)
# define a data input dictonary with our uploaded s3 uris
data = {'training': training_input_path}
# starting the train job with our uploaded datasets as input
huggingface_estimator.fit(data, wait=False)
Deployment
The two base models, LLaMA2-13b and LLaMA2-13b-chat, can be conveniently deployed using SageMaker JumpStart.
The two fine-tuned models can be deployed using AWSxHuggingFace LLM DLC. For this, we set the model_data parameter of the HuggingFaceModel class to the S3 path of the model files. Then, we can set the HF_MODEL_ID environment variable in the hosted container to the default model path in the DLC ("/opt/ml/model").
from sagemaker.huggingface import get_huggingface_llm_image_uri
# retrieve the llm image uri
llm_image = get_huggingface_llm_image_uri(
"huggingface",
version="0.8.2"
)
# print ecr image uri
print(f"llm image uri: {llm_image}")
# sagemaker config
instance_type = "ml.g5.12xlarge"
number_of_gpu = 4
health_check_timeout = 300
# TGI config
config = {
'HF_MODEL_ID': "/opt/ml/model", # path to where sagemaker stores the model
'SM_NUM_GPUS': json.dumps(number_of_gpu), # Number of GPU used per replica
'MAX_INPUT_LENGTH': json.dumps(1024), # Max length of input text
'MAX_TOTAL_TOKENS': json.dumps(2048), # Max length of the generation (including input text),
# 'HF_MODEL_QUANTIZE': "bitsandbytes", # comment in to quantize
}
# create HuggingFaceModel
llm_model = HuggingFaceModel(
role=role,
image_uri=llm_image,
model_data="s3://sagemaker-us-east-1-308819823671/huggingface-qlora-llama2-13b-chat-2023--2023-08-02-08-54-16-604/output/model.tar.gz",
env=config
)
# Deploy model to an endpoint
# https://sagemaker.readthedocs.io/en/stable/api/inference/model.html#sagemaker.model.Model.deploy
llm = llm_model.deploy(
endpoint_name="llama-2-13b-chat-hf-nyc-finetuned", # alternatively "llama-2-13b-hf-nyc-finetuned"
initial_instance_count=1,
instance_type=instance_type,
# volume_size=400, # If using an instance with local SSD storage, volume_size must be None, e.g. p4 but not p3
container_startup_health_check_timeout=health_check_timeout, # 10 minutes to be able to load the model
)
Testing
After deploying all four models, we want to test their performance on a sample question. The blog post about Amazon EC2 P5 instances states, "P5 instances provide eight NVIDIA H100 Tensor Core GPUs with 640 GB of high-bandwidth GPU memory, third-generation AMD EPYC processors, 2 TB of system memory, and 30 TB of local NVMe storage. P5 instances also provide 3200 Gbps of aggregated network bandwidth, support GPUDirect RDMA for low-latency and efficient scale-out performance by bypassing CPU communication between nodes."
We pose the following question: "What are Amazon EC2 P5 instances? What type of GPUs are they equipped with?" The chat models provide the following answers using the inference configuration max_new_tokens = 200, top_p = 0.9, temperature =
Google TV integrates Gemini AI for image/video generation, enhances Google Photos, and adds a dedicated YouTube Shorts feed to the homepage.
Read moreAlphabet Q1 paid subscribers reached 350M, fueled by YouTube & Google One. YouTube ad revenue slightly missed at $9.88B amid Premium growth. Total Q1 revenue hit $109.9B, beating forecasts.
Read moreAnthropic releases Claude connectors for Adobe, Blender, and Ableton, enabling AI workflow automation and app control. It also sponsors the Blender Development Foundation.
Read moreApple introduces an App Store model splitting 12-month plans into monthly installments. On iOS 26.4+, it lowers upfront costs without reducing total price, adding billing tracking.
Read moreGoogle launches Aadhaar digital IDs in Wallet for India, enabling secure QR/NFC verification. A phased partner rollout targets key sectors, while passport-based credentials expand globally.
Read moreYouTube tests "Ask YouTube," an AI tool blending text and video for complex queries. Initially for US Premium users, it supports interactive follow-ups, expanding Google's AI search.
Read moreGoogle grants US DoD classified AI access, joining OpenAI and xAI in defense deals amid Anthropic's dispute. Contractual safeguards against surveillance and lethal weapons remain unenforceable.
Read moreMistral AI launches Workflows, a Temporal-powered orchestration platform for secure, Python-driven enterprise AI automation featuring human-in-the-loop approvals.
Read moreCustom subliminal audio maker

AI Social Media Manager

AI Search Visibility

Workout and Calorie Tracker
Stock research platform

AI interview assistant

Scientific figure generator

AI courses with real code