Update Llava GPU Example (#12311)
* update-llava-example * add warmup * small fix on llava example * remove space& extra print prompt * renew example * small fix --------- Co-authored-by: Jinhe Tang <jin.tang1337@gmail.com>
This commit is contained in:
Zijie Li
GitHub
parent
f53bb4ea0b
commit
cd5e22cee5
2 changed files with 75 additions and 351 deletions
|
|
@ -1,11 +1,11 @@
|
||||||
# LLaVA
|
# LLaVA
|
||||||
In this directory, you will find examples on how you could use IPEX-LLM `optimize_model` API on LLaVA models on [Intel GPUs](../../../README.md). For illustration purposes, we utilize the [liuhaotian/llava-v1.5-7b](https://huggingface.co/liuhaotian/llava-v1.5-7b) as a reference LLaVA model.
|
In this directory, you will find examples on how you could use IPEX-LLM `optimize_model` API to accelerate LLaVA models on [Intel GPUs](../../../README.md). For illustration purposes, we utilize the [llava-hf/llava-1.5-7b-hf](https://huggingface.co/llava-hf/llava-1.5-7b-hf) as a reference LLaVA model.
|
||||||
|
|
||||||
## 0. Requirements
|
## 0. Requirements
|
||||||
To run these examples with IPEX-LLM on Intel GPUs, we have some recommended requirements for your machine, please refer to [here](../../../README.md#requirements) for more information.
|
To run these examples with IPEX-LLM on Intel GPUs, we have some recommended requirements for your machine, please refer to [here](../../../README.md#requirements) for more information.
|
||||||
|
|
||||||
## Example: Multi-turn chat centered around an image using `generate()` API
|
## Example: Predict Tokens using `generate()` API
|
||||||
In the example [generate.py](./generate.py), we show a basic use case for a LLaVA model to start a multi-turn chat centered around an image using `generate()` API, with IPEX-LLM INT4 optimizations on Intel GPUs.
|
In the example [generate.py](./generate.py), we show a basic use case for a LLaVA model to predict the next N tokens using `generate()` API, with IPEX-LLM INT4 optimizations on Intel GPUs.
|
||||||
### 1. Install
|
### 1. Install
|
||||||
#### 1.1 Installation on Linux
|
#### 1.1 Installation on Linux
|
||||||
We suggest using conda to manage environment:
|
We suggest using conda to manage environment:
|
||||||
|
|
@ -15,12 +15,7 @@ conda activate llm
|
||||||
# below command will install intel_extension_for_pytorch==2.1.10+xpu as default
|
# below command will install intel_extension_for_pytorch==2.1.10+xpu as default
|
||||||
pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/
|
pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/
|
||||||
|
|
||||||
pip install einops # install dependencies required by llava
|
pip install transformers==4.43.0
|
||||||
|
|
||||||
git clone https://github.com/haotian-liu/LLaVA.git # clone the llava libary
|
|
||||||
cp generate.py ./LLaVA/ # copy our example to the LLaVA folder
|
|
||||||
cd LLaVA # change the working directory to the LLaVA folder
|
|
||||||
git checkout tags/v1.2.0 -b 1.2.0 # Get the branch which is compatible with transformers 4.36
|
|
||||||
```
|
```
|
||||||
|
|
||||||
#### 1.2 Installation on Windows
|
#### 1.2 Installation on Windows
|
||||||
|
|
@ -32,12 +27,7 @@ conda activate llm
|
||||||
# below command will install intel_extension_for_pytorch==2.1.10+xpu as default
|
# below command will install intel_extension_for_pytorch==2.1.10+xpu as default
|
||||||
pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/
|
pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/
|
||||||
|
|
||||||
pip install einops # install dependencies required by llava
|
pip install transformers==4.43.0
|
||||||
|
|
||||||
git clone https://github.com/haotian-liu/LLaVA.git # clone the llava libary
|
|
||||||
copy generate.py .\LLaVA\ # copy our example to the LLaVA folder
|
|
||||||
cd LLaVA # change the working directory to the LLaVA folder
|
|
||||||
git checkout tags/v1.2.0 -b 1.2.0 # Get the branch which is compatible with transformers 4.36
|
|
||||||
```
|
```
|
||||||
|
|
||||||
### 2. Configures OneAPI environment variables for Linux
|
### 2. Configures OneAPI environment variables for Linux
|
||||||
|
|
@ -116,42 +106,30 @@ set SYCL_CACHE_PERSISTENT=1
|
||||||
> For the first time that each model runs on Intel iGPU/Intel Arc™ A300-Series or Pro A60, it may take several minutes to compile.
|
> For the first time that each model runs on Intel iGPU/Intel Arc™ A300-Series or Pro A60, it may take several minutes to compile.
|
||||||
### 4. Running examples
|
### 4. Running examples
|
||||||
|
|
||||||
```bash
|
```
|
||||||
python ./generate.py --image-path-or-url 'https://llava-vl.github.io/static/images/monalisa.jpg'
|
python ./generate.py
|
||||||
```
|
```
|
||||||
|
|
||||||
In the example, several arguments can be passed to satisfy your requirements:
|
Arguments info:
|
||||||
|
- `--repo-id-or-model-path REPO_ID_OR_MODEL_PATH`: argument defining the huggingface repo id for the LLaVA model (e.g. `llava-hf/llava-1.5-7b-hf`) to be downloaded, or the path to the huggingface checkpoint folder. It is default to be `'llava-hf/llava-1.5-7b-hf'`.
|
||||||
- `--repo-id-or-model-path REPO_ID_OR_MODEL_PATH`: argument defining the huggingface repo id for the LLaVA model (e.g. `liuhaotian/llava-v1.5-7b` to be downloaded, or the path to the huggingface checkpoint folder. It is default to be `'liuhaotian/llava-v1.5-7b'`.
|
- `--image-url-or-path IMAGE_URL_OR_PATH`: argument defining the image to be infered. It is default to be `'http://farm6.staticflickr.com/5268/5602445367_3504763978_z.jpg'`.
|
||||||
- `--image-path-or-url IMAGE_PATH_OR_URL`: argument defining the input image that the chat will focus on. It is required.
|
- `--prompt PROMPT`: argument defining the prompt to be infered (with integrated prompt format for chat). It is default to be `'Describe image in detail'`.
|
||||||
- `--n-predict N_PREDICT`: argument defining the max number of tokens to predict. It is default to be `512`.
|
- `--n-predict N_PREDICT`: argument defining the max number of tokens to predict. It is default to be `32`.
|
||||||
|
|
||||||
If you encounter some network error (which means your machine is unable to access huggingface.co) when running this example, refer to [Trouble Shooting](#4-trouble-shooting) section.
|
|
||||||
|
|
||||||
|
|
||||||
#### Sample Output
|
#### Sample Output
|
||||||
#### [liuhaotian/llava-v1.5-7b](https://huggingface.co/liuhaotian/llava-v1.5-7b)
|
#### [llava-hf/llava-1.5-7b-hf](https://huggingface.co/llava-hf/llava-1.5-7b-hf)
|
||||||
|
|
||||||
```log
|
```log
|
||||||
USER: Do you know who drew this painting?
|
Inference time: xxxx s
|
||||||
ASSISTANT: Yes, the painting is a portrait of a woman by Leonardo da Vinci. It's a famous artwork known as the "Mona Lisa."
|
-------------------- Input Image --------------------
|
||||||
USER: Can you describe this painting?
|
http://farm6.staticflickr.com/5268/5602445367_3504763978_z.jpg
|
||||||
ASSISTANT: The painting features a well-detailed portrait of a woman, painted in oil on a canvas. The woman appears to be a young woman staring straight ahead in a direct gaze towards the viewer. The woman's facial features are rendered sharply in the brush strokes, giving her a lifelike, yet enigmatic expression.
|
-------------------- Prompt --------------------
|
||||||
The background of the image mainly showcases the woman's face, with some hills visible in the lower part of the painting. The artist employs a wide range of shades, evoking a sense of depth and realism in the subject matter. The technique used in this portrait sets it apart from other artworks during the Renaissance period, making it a notable piece in art history.
|
Describe image in detail
|
||||||
|
-------------------- Output --------------------
|
||||||
|
<s> USER: <image>
|
||||||
|
Describe image in detail ASSISTANT: The image features a young girl holding a white teddy bear in her hands. She is smiling and appears to be enjoying the moment. The girl is
|
||||||
```
|
```
|
||||||
|
|
||||||
The sample input image is:
|
The sample input image is (which is fetched from [COCO dataset](https://cocodataset.org/#explore?id=264959)):
|
||||||
|
|
||||||
<a href="https://llava-vl.github.io/static/images/monalisa.jpg"><img width=400px src="https://llava-vl.github.io/static/images/monalisa.jpg" ></a>
|
<a href="http://farm6.staticflickr.com/5268/5602445367_3504763978_z.jpg"><img width=400px src="http://farm6.staticflickr.com/5268/5602445367_3504763978_z.jpg" ></a>
|
||||||
|
|
||||||
### 5 Trouble shooting
|
|
||||||
|
|
||||||
#### 5.1 SSLError
|
|
||||||
If you encounter the following output, it means your machine has some trouble accessing huggingface.co.
|
|
||||||
```log
|
|
||||||
requests.exceptions.SSLError: (MaxRetryError("HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /openai/clip-vit-large-patch14-336/resolve/main/config.json (Caused by SSLError(SSLZeroReturnError(6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:1129)')))"),
|
|
||||||
```
|
|
||||||
|
|
||||||
You can resolve this problem with the following steps:
|
|
||||||
1. Download https://huggingface.co/openai/clip-vit-large-patch14-336 on some machine that can access huggingface.co, and put it in huggingface's local cache (default to be `~/.cache/huggingface/hub`) on the machine that you are going to run this example.
|
|
||||||
2. Set the environment variable (`export TRANSFORMERS_OFFLINE=1`) before you run the example.
|
|
||||||
|
|
@ -13,328 +13,74 @@
|
||||||
# See the License for the specific language governing permissions and
|
# See the License for the specific language governing permissions and
|
||||||
# limitations under the License.
|
# limitations under the License.
|
||||||
#
|
#
|
||||||
# Some parts of this file is adapted from
|
|
||||||
# https://github.com/haotian-liu/LLaVA/blob/v1.1.1/llava/model/builder.py
|
|
||||||
# and
|
|
||||||
# https://github.com/haotian-liu/LLaVA/blob/v1.1.1/llava/serve/cli.py
|
|
||||||
#
|
|
||||||
# Copyright 2023 Haotian Liu
|
|
||||||
#
|
|
||||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
|
||||||
# you may not use this file except in compliance with the License.
|
|
||||||
# You may obtain a copy of the License at
|
|
||||||
#
|
|
||||||
# http://www.apache.org/licenses/LICENSE-2.0
|
|
||||||
#
|
|
||||||
# Unless required by applicable law or agreed to in writing, software
|
|
||||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
||||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
||||||
# See the License for the specific language governing permissions and
|
|
||||||
# limitations under the License.
|
|
||||||
|
|
||||||
import argparse
|
import argparse
|
||||||
import torch
|
import os
|
||||||
|
|
||||||
|
import requests
|
||||||
import time
|
import time
|
||||||
|
import torch
|
||||||
from transformers import AutoModelForCausalLM
|
from PIL import Image
|
||||||
from llava.model.language_model.llava_llama import LlavaLlamaForCausalLM
|
from transformers import LlavaForConditionalGeneration, AutoProcessor
|
||||||
from transformers import AutoTokenizer
|
|
||||||
from transformers import TextStreamer
|
|
||||||
|
|
||||||
from llava.constants import (
|
|
||||||
DEFAULT_IMAGE_PATCH_TOKEN,
|
|
||||||
IMAGE_TOKEN_INDEX,
|
|
||||||
DEFAULT_IMAGE_TOKEN,
|
|
||||||
DEFAULT_IM_START_TOKEN,
|
|
||||||
DEFAULT_IM_END_TOKEN
|
|
||||||
)
|
|
||||||
from llava.conversation import conv_templates, SeparatorStyle
|
|
||||||
from llava.utils import disable_torch_init
|
|
||||||
from llava.mm_utils import (
|
|
||||||
process_images,
|
|
||||||
tokenizer_image_token,
|
|
||||||
get_model_name_from_path,
|
|
||||||
KeywordsStoppingCriteria
|
|
||||||
)
|
|
||||||
|
|
||||||
from ipex_llm import optimize_model
|
from ipex_llm import optimize_model
|
||||||
|
|
||||||
# Load the pretrained model.
|
|
||||||
# Adapted from llava.model.builder.load_pretrained_model.
|
|
||||||
def load_pretrained_model(model_path, model_base, model_name, load_8bit=False, load_4bit=False,
|
|
||||||
device_map="auto", device="cpu"):
|
|
||||||
kwargs = {"device_map": device_map}
|
|
||||||
|
|
||||||
if 'llava' in model_name.lower():
|
|
||||||
# Load LLaVA model
|
|
||||||
if 'lora' in model_name.lower() and model_base is None:
|
|
||||||
warnings.warn('There is `lora` in model name but no `model_base` is provided.'
|
|
||||||
'If you are loading a LoRA model, please provide the `model_base` argument'
|
|
||||||
'. Detailed instruction:'
|
|
||||||
'https://github.com/haotian-liu/LLaVA#launch-a-model-worker-lora-weights-unmerged.')
|
|
||||||
if 'lora' in model_name.lower() and model_base is not None:
|
|
||||||
lora_cfg_pretrained = AutoConfig.from_pretrained(model_path)
|
|
||||||
tokenizer = AutoTokenizer.from_pretrained(
|
|
||||||
model_base, use_fast=False)
|
|
||||||
print('Loading LLaVA from base model...')
|
|
||||||
model = LlavaLlamaForCausalLM.from_pretrained(model_base, low_cpu_mem_usage=True,
|
|
||||||
config=lora_cfg_pretrained, **kwargs)
|
|
||||||
token_num, tokem_dim = model.lm_head.out_features, model.lm_head.in_features
|
|
||||||
if model.lm_head.weight.shape[0] != token_num:
|
|
||||||
model.lm_head.weight = torch.nn.Parameter(torch.empty(
|
|
||||||
token_num, tokem_dim, device=model.device, dtype=model.dtype))
|
|
||||||
model.model.embed_tokens.weight = torch.nn.Parameter(torch.empty(
|
|
||||||
token_num, tokem_dim, device=model.device, dtype=model.dtype))
|
|
||||||
|
|
||||||
print('Loading additional LLaVA weights...')
|
|
||||||
if os.path.exists(os.path.join(model_path, 'non_lora_trainables.bin')):
|
|
||||||
non_lora_trainables = torch.load(os.path.join(model_path,
|
|
||||||
'non_lora_trainables.bin'),
|
|
||||||
map_location='cpu')
|
|
||||||
else:
|
|
||||||
# this is probably from HF Hub
|
|
||||||
from huggingface_hub import hf_hub_download
|
|
||||||
|
|
||||||
def load_from_hf(repo_id, filename, subfolder=None):
|
|
||||||
cache_file = hf_hub_download(
|
|
||||||
repo_id=repo_id,
|
|
||||||
filename=filename,
|
|
||||||
subfolder=subfolder)
|
|
||||||
return torch.load(cache_file, map_location='cpu')
|
|
||||||
non_lora_trainables = load_from_hf(
|
|
||||||
model_path, 'non_lora_trainables.bin')
|
|
||||||
non_lora_trainables = {(k[11:] if k.startswith(
|
|
||||||
'base_model.') else k): v for k, v in non_lora_trainables.items()}
|
|
||||||
if any(k.startswith('model.model.') for k in non_lora_trainables):
|
|
||||||
non_lora_trainables = {(k[6:] if k.startswith(
|
|
||||||
'model.') else k): v for k, v in non_lora_trainables.items()}
|
|
||||||
model.load_state_dict(non_lora_trainables, strict=False)
|
|
||||||
|
|
||||||
from peft import PeftModel
|
|
||||||
print('Loading LoRA weights...')
|
|
||||||
model = PeftModel.from_pretrained(model, model_path)
|
|
||||||
print('Merging LoRA weights...')
|
|
||||||
model = model.merge_and_unload()
|
|
||||||
print('Model is loaded...')
|
|
||||||
elif model_base is not None:
|
|
||||||
# this may be mm projector only
|
|
||||||
print('Loading LLaVA from base model...')
|
|
||||||
if 'mpt' in model_name.lower():
|
|
||||||
if not os.path.isfile(os.path.join(model_path, 'configuration_mpt.py')):
|
|
||||||
shutil.copyfile(os.path.join(model_base, 'configuration_mpt.py'), os.path.join(
|
|
||||||
model_path, 'configuration_mpt.py'))
|
|
||||||
tokenizer = AutoTokenizer.from_pretrained(
|
|
||||||
model_base, use_fast=True)
|
|
||||||
cfg_pretrained = AutoConfig.from_pretrained(
|
|
||||||
model_path, trust_remote_code=True)
|
|
||||||
model = LlavaMPTForCausalLM.from_pretrained(
|
|
||||||
model_base, low_cpu_mem_usage=True, config=cfg_pretrained, **kwargs)
|
|
||||||
else:
|
|
||||||
tokenizer = AutoTokenizer.from_pretrained(
|
|
||||||
model_base, use_fast=False)
|
|
||||||
cfg_pretrained = AutoConfig.from_pretrained(model_path)
|
|
||||||
model = LlavaLlamaForCausalLM.from_pretrained(
|
|
||||||
model_base, low_cpu_mem_usage=True, config=cfg_pretrained, **kwargs)
|
|
||||||
|
|
||||||
mm_projector_weights = torch.load(os.path.join(
|
|
||||||
model_path, 'mm_projector.bin'), map_location='cpu')
|
|
||||||
mm_projector_weights = {k: v.to(torch.float32)
|
|
||||||
for k, v in mm_projector_weights.items()}
|
|
||||||
model.load_state_dict(mm_projector_weights, strict=False)
|
|
||||||
else:
|
|
||||||
if 'mpt' in model_name.lower():
|
|
||||||
tokenizer = AutoTokenizer.from_pretrained(
|
|
||||||
model_path, use_fast=True)
|
|
||||||
model = LlavaMPTForCausalLM.from_pretrained(
|
|
||||||
model_path, low_cpu_mem_usage=True, **kwargs)
|
|
||||||
else:
|
|
||||||
tokenizer = AutoTokenizer.from_pretrained(
|
|
||||||
model_path, use_fast=False)
|
|
||||||
model = LlavaLlamaForCausalLM.from_pretrained(
|
|
||||||
model_path, low_cpu_mem_usage=True)
|
|
||||||
else:
|
|
||||||
# Load language model
|
|
||||||
if model_base is not None:
|
|
||||||
# PEFT model
|
|
||||||
from peft import PeftModel
|
|
||||||
tokenizer = AutoTokenizer.from_pretrained(
|
|
||||||
model_base, use_fast=False)
|
|
||||||
model = AutoModelForCausalLM.from_pretrained(
|
|
||||||
model_base, torch_dtype=torch.float32, low_cpu_mem_usage=True, device_map="auto")
|
|
||||||
print(f"Loading LoRA weights from {model_path}")
|
|
||||||
model = PeftModel.from_pretrained(model, model_path)
|
|
||||||
print(f"Merging weights")
|
|
||||||
model = model.merge_and_unload()
|
|
||||||
print('Convert to FP32...')
|
|
||||||
model.to(torch.float32)
|
|
||||||
else:
|
|
||||||
use_fast = False
|
|
||||||
if 'mpt' in model_name.lower():
|
|
||||||
tokenizer = AutoTokenizer.from_pretrained(
|
|
||||||
model_path, use_fast=True)
|
|
||||||
model = AutoModelForCausalLM.from_pretrained(
|
|
||||||
model_path, low_cpu_mem_usage=True, trust_remote_code=True, **kwargs)
|
|
||||||
else:
|
|
||||||
tokenizer = AutoTokenizer.from_pretrained(
|
|
||||||
model_path, use_fast=False)
|
|
||||||
model = AutoModelForCausalLM.from_pretrained(
|
|
||||||
model_path, low_cpu_mem_usage=True, **kwargs)
|
|
||||||
|
|
||||||
image_processor = None
|
|
||||||
|
|
||||||
if 'llava' in model_name.lower():
|
|
||||||
mm_use_im_start_end = getattr(
|
|
||||||
model.config, "mm_use_im_start_end", False)
|
|
||||||
mm_use_im_patch_token = getattr(
|
|
||||||
model.config, "mm_use_im_patch_token", True)
|
|
||||||
if mm_use_im_patch_token:
|
|
||||||
tokenizer.add_tokens(
|
|
||||||
[DEFAULT_IMAGE_PATCH_TOKEN], special_tokens=True)
|
|
||||||
if mm_use_im_start_end:
|
|
||||||
tokenizer.add_tokens(
|
|
||||||
[DEFAULT_IM_START_TOKEN, DEFAULT_IM_END_TOKEN], special_tokens=True)
|
|
||||||
model.resize_token_embeddings(len(tokenizer))
|
|
||||||
|
|
||||||
vision_tower = model.get_vision_tower()
|
|
||||||
if not vision_tower.is_loaded:
|
|
||||||
vision_tower.load_model()
|
|
||||||
vision_tower.to(device=device, dtype=torch.float32)
|
|
||||||
image_processor = vision_tower.image_processor
|
|
||||||
|
|
||||||
if hasattr(model.config, "max_sequence_length"):
|
|
||||||
context_len = model.config.max_sequence_length
|
|
||||||
else:
|
|
||||||
context_len = 2048
|
|
||||||
|
|
||||||
return tokenizer, model, image_processor, context_len
|
|
||||||
|
|
||||||
# Initialize conversation from templates and get conversation roles.
|
|
||||||
def get_conv_and_role(model_name):
|
|
||||||
if 'llama-2' in model_name.lower():
|
|
||||||
conv_mode = "llava_llama_2"
|
|
||||||
elif "v1" in model_name.lower():
|
|
||||||
conv_mode = "llava_v1"
|
|
||||||
elif "mpt" in model_name.lower():
|
|
||||||
conv_mode = "mpt"
|
|
||||||
else:
|
|
||||||
conv_mode = "llava_v0"
|
|
||||||
|
|
||||||
conv = conv_templates[conv_mode].copy()
|
|
||||||
if "mpt" in model_name.lower():
|
|
||||||
roles = ('user', 'assistant')
|
|
||||||
else:
|
|
||||||
roles = conv.roles
|
|
||||||
|
|
||||||
return conv, roles
|
|
||||||
|
|
||||||
# Load image from a url or path.
|
|
||||||
def load_image(image_file):
|
|
||||||
import requests
|
|
||||||
from PIL import Image
|
|
||||||
from io import BytesIO
|
|
||||||
|
|
||||||
if image_file.startswith('http://') or image_file.startswith('https://'):
|
|
||||||
response = requests.get(image_file)
|
|
||||||
image = Image.open(BytesIO(response.content)).convert('RGB')
|
|
||||||
else:
|
|
||||||
image = Image.open(image_file).convert('RGB')
|
|
||||||
return image
|
|
||||||
|
|
||||||
def generate_image_tensor(image_file):
|
|
||||||
image = load_image(image_file)
|
|
||||||
model_cfg = {"image_aspect_ratio": 'pad'}
|
|
||||||
image_tensor = process_images([image], image_processor, model_cfg)
|
|
||||||
return image_tensor
|
|
||||||
|
|
||||||
# Generate input prompt with user input.
|
|
||||||
def get_prompt(mm_use_im_start_end, first_round, conv, user_input):
|
|
||||||
if first_round:
|
|
||||||
# first message
|
|
||||||
if mm_use_im_start_end:
|
|
||||||
user_input = DEFAULT_IM_START_TOKEN + DEFAULT_IMAGE_TOKEN + \
|
|
||||||
DEFAULT_IM_END_TOKEN + '\n' + user_input
|
|
||||||
else:
|
|
||||||
user_input = DEFAULT_IMAGE_TOKEN + '\n' + user_input
|
|
||||||
conv.append_message(conv.roles[0], user_input)
|
|
||||||
else:
|
|
||||||
# later messages
|
|
||||||
conv.append_message(conv.roles[0], user_input)
|
|
||||||
conv.append_message(conv.roles[1], None)
|
|
||||||
return conv.get_prompt()
|
|
||||||
|
|
||||||
def get_stopping_criteria(conv, tokenizer, input_ids):
|
|
||||||
stop_str = conv.sep if conv.sep_style != SeparatorStyle.TWO else conv.sep2
|
|
||||||
keywords = [stop_str]
|
|
||||||
stopping_criteria = KeywordsStoppingCriteria(keywords, tokenizer, input_ids)
|
|
||||||
return stopping_criteria
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == '__main__':
|
if __name__ == '__main__':
|
||||||
parser = argparse.ArgumentParser(
|
parser = argparse.ArgumentParser(description='Predict Tokens using `generate()` API for LLaVA model')
|
||||||
description='Predict Tokens using `generate()` API for LLaVA model')
|
parser.add_argument('--repo-id-or-model-path', type=str, default="llava-hf/llava-1.5-7b-hf",
|
||||||
parser.add_argument('--repo-id-or-model-path', type=str, default="liuhaotian/llava-v1.5-7b",
|
|
||||||
help='The huggingface repo id for the LLaVA model to be downloaded'
|
help='The huggingface repo id for the LLaVA model to be downloaded'
|
||||||
', or the path to the huggingface checkpoint folder')
|
', or the path to the huggingface checkpoint folder')
|
||||||
parser.add_argument('--image-path-or-url', type=str,
|
parser.add_argument('--image-url-or-path', type=str,
|
||||||
required=True, help='Image path or url for the input image that the chat will focus on')
|
default='http://farm6.staticflickr.com/5268/5602445367_3504763978_z.jpg',
|
||||||
parser.add_argument('--n-predict', type=int, default=512,
|
help='The URL or path to the image to infer')
|
||||||
|
parser.add_argument('--prompt', type=str, default="Describe image in detail",
|
||||||
|
help='Prompt to infer')
|
||||||
|
parser.add_argument('--n-predict', type=int, default=32,
|
||||||
help='Max tokens to predict')
|
help='Max tokens to predict')
|
||||||
|
|
||||||
args = parser.parse_args()
|
args = parser.parse_args()
|
||||||
model_path = args.repo_id_or_model_path
|
model_path = args.repo_id_or_model_path
|
||||||
model_name = get_model_name_from_path(model_path)
|
image_path = args.image_url_or_path
|
||||||
|
prompt = args.prompt
|
||||||
|
|
||||||
# Disable the redundant torch default initialization to accelerate model creation.
|
model = LlavaForConditionalGeneration.from_pretrained(model_path)
|
||||||
disable_torch_init()
|
model = optimize_model(model, low_bit='sym_int4').eval()
|
||||||
|
model = model.half().to("xpu")
|
||||||
|
|
||||||
# Load model
|
processor = AutoProcessor.from_pretrained(model_path)
|
||||||
tokenizer, model, image_processor, _ = load_pretrained_model(model_path=model_path,
|
|
||||||
model_base=None,
|
|
||||||
model_name=model_name)
|
|
||||||
|
|
||||||
# With only one line to enable IPEX-LLM optimization on model
|
# here the prompt tuning refers to https://huggingface.co/llava-hf/llava-1.5-7b-hf#using-pure-transformers
|
||||||
# When running LLMs on Intel iGPUs for Windows users, we recommend setting `cpu_embedding=True` in the optimize_model function.
|
messages = [
|
||||||
# This will allow the memory-intensive embedding layer to utilize the CPU instead of iGPU.
|
{
|
||||||
model = optimize_model(model).to('xpu')
|
"role": "user",
|
||||||
|
"content": [
|
||||||
|
{"type": "image"},
|
||||||
|
{"type": "text", "text": prompt}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
]
|
||||||
|
text = processor.apply_chat_template(messages, add_generation_prompt=True)
|
||||||
|
|
||||||
# Generate image tensor
|
if os.path.exists(image_path):
|
||||||
image_tensor = generate_image_tensor(args.image_path_or_url)
|
image = Image.open(image_path)
|
||||||
|
else:
|
||||||
|
image = Image.open(requests.get(image_path, stream=True).raw)
|
||||||
|
|
||||||
# Get conversation template and roles
|
inputs = processor(text=text, images=image, return_tensors="pt").to('xpu')
|
||||||
conv, roles = get_conv_and_role(model_name)
|
|
||||||
|
|
||||||
first_round = True
|
with torch.inference_mode():
|
||||||
while True:
|
# warmup
|
||||||
try:
|
output = model.generate(**inputs, do_sample=False, max_new_tokens=args.n_predict)
|
||||||
user_input = input(f"{roles[0]}: ")
|
|
||||||
except EOFError:
|
|
||||||
user_input = ""
|
|
||||||
if not user_input:
|
|
||||||
print("exit...")
|
|
||||||
break
|
|
||||||
|
|
||||||
print(f"{roles[1]}: ", end="")
|
# start inference
|
||||||
|
st = time.time()
|
||||||
|
output = model.generate(**inputs, do_sample=False, max_new_tokens=args.n_predict)
|
||||||
|
et = time.time()
|
||||||
|
|
||||||
prompt = get_prompt(model.config.mm_use_im_start_end, first_round, conv, user_input)
|
output_str = processor.decode(output[0])
|
||||||
first_round = False
|
print(f'Inference time: {et-st} s')
|
||||||
input_ids = tokenizer_image_token(
|
print('-'*20, 'Input Image', '-'*20)
|
||||||
prompt, tokenizer, IMAGE_TOKEN_INDEX, return_tensors='pt').unsqueeze(0).to('xpu')
|
print(image_path)
|
||||||
stopping_criteria = get_stopping_criteria(conv, tokenizer, input_ids)
|
print('-'*20, 'Prompt', '-'*20)
|
||||||
streamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)
|
print(prompt)
|
||||||
|
print('-'*20, 'Output', '-'*20)
|
||||||
# Generate predicted tokens
|
print(output_str)
|
||||||
with torch.inference_mode():
|
|
||||||
st = time.time()
|
|
||||||
output_ids = model.generate(
|
|
||||||
input_ids,
|
|
||||||
images=image_tensor,
|
|
||||||
do_sample=True,
|
|
||||||
max_new_tokens=args.n_predict,
|
|
||||||
streamer=streamer,
|
|
||||||
use_cache=True,
|
|
||||||
stopping_criteria=[stopping_criteria])
|
|
||||||
end = time.time()
|
|
||||||
#print(f'Inference time: {end-st} s')
|
|
||||||
|
|
||||||
outputs = tokenizer.decode(output_ids[0, :].cpu(), skip_special_tokens=True).strip()
|
|
||||||
conv.messages[-1][-1] = outputs
|
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue