ayo/ipex-llm
1
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

Update MiniCPM-V-2_6 Example (#11958)

Browse source 
* Update example scripts regarding warmup, stream generate, moudles to not convert, etc.

* Update readme accordingly

* Fix based on comments

* Small fix

* Remove n_predict
This commit is contained in:
Yuwen Hu 2024-08-29 18:23:48 +08:00 committed by GitHub
parent 6fc9340d53
commit 7abe17d6f7
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
3 changed files with 134 additions and 191 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

39
python/llm/example/GPU/HuggingFace/Multimodal/MiniCPM-V-2_6/README.md
View file

@ -5,7 +5,7 @@ In this directory, you will find examples on how you could apply IPEX-LLM INT4 o
To run these examples with IPEX-LLM on Intel GPUs, we have some recommended requirements for your machine, please refer to [here](../../../README.md#requirements) for more information.
## Example: Predict Tokens using `chat()` API
In the example [generate.py](./generate.py), we show a basic use case for a MiniCPM-V-2_6 model to predict the next N tokens using `chat()` API, with IPEX-LLM INT4 optimizations on Intel GPUs.
In the example [chat.py](./chat.py), we show a basic use case for a MiniCPM-V-2_6 model to predict the next N tokens using `chat()` API, with IPEX-LLM INT4 optimizations on Intel GPUs.
### 1. Install
#### 1.1 Installation on Linux
We suggest using conda to manage environment:
@ -15,7 +15,7 @@ conda activate llm
# below command will install intel_extension_for_pytorch==2.1.10+xpu as default
pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/
pip install timm peft transformers==4.40.0 trl
pip install transformers==4.40.0 trl
```
#### 1.2 Installation on Windows
@ -27,7 +27,7 @@ conda activate llm
# below command will install intel_extension_for_pytorch==2.1.10+xpu as default
pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/
pip install timm peft transformers==4.40.0 trl
pip install transformers==4.40.0 trl
```
### 2. Configures OneAPI environment variables for Linux
@ -106,15 +106,23 @@ set SYCL_CACHE_PERSISTENT=1
> For the first time that each model runs on Intel iGPU/Intel Arc™ A300-Series or Pro A60, it may take several minutes to compile.
### 4. Running examples
```
python ./generate.py --prompt 'What is in the image?'
```
- chat without streaming mode:
```
python ./generate.py --prompt 'What is in the image?'
```
- chat in streaming mode:
```
python ./generate.py --prompt 'What is in the image?' --stream
```
> [!TIP]
> For chatting in streaming mode, it is recommended to set the environment variable `PYTHONUNBUFFERED=1`.
Arguments info:
- `--repo-id-or-model-path REPO_ID_OR_MODEL_PATH`: argument defining the huggingface repo id for the MiniCPM-V-2_6 (e.g. `openbmb/MiniCPM-V-2_6`) to be downloaded, or the path to the huggingface checkpoint folder. It is default to be `'openbmb/MiniCPM-V-2_6'`.
- `--image-url-or-path IMAGE_URL_OR_PATH`: argument defining the image to be infered. It is default to be `'http://farm6.staticflickr.com/5268/5602445367_3504763978_z.jpg'`.
- `--prompt PROMPT`: argument defining the prompt to be infered (with integrated prompt format for chat). It is default to be `'What is in the image?'`.
- `--n-predict N_PREDICT`: argument defining the max number of tokens to predict. It is default to be `32`.
- `--stream`: flag to chat in streaming mode
#### Sample Output
@ -122,21 +130,20 @@ Arguments info:
```log
Inference time: xxxx s
-------------------- Input --------------------
-------------------- Input Image --------------------
http://farm6.staticflickr.com/5268/5602445367_3504763978_z.jpg
-------------------- Prompt --------------------
-------------------- Input Prompt --------------------
What is in the image?
-------------------- Output --------------------
The image features a young child holding a white teddy bear with a pink tutu. The child is wearing a striped dress and is standing in front of a stone wall with some red flowers in the background.
-------------------- Chat Output --------------------
The image features a young child holding a white teddy bear wearing a pink dress. The background shows some red flowers and stone walls, suggesting an outdoor setting.
```
```log
Inference time: xxxx s
-------------------- Input --------------------
-------------------- Input Image --------------------
http://farm6.staticflickr.com/5268/5602445367_3504763978_z.jpg
-------------------- Prompt --------------------
-------------------- Input Prompt --------------------
图片里有什么？
-------------------- Output --------------------
这幅图片展示了一个年幼的孩子，可能是一个蹒跚学步的幼儿，手里拿着一个毛绒玩具熊。孩子穿着一件条纹连衣裙，主要颜色是粉红色和白色。毛绒熊是白色的，戴着一条粉色的蝴蝶结围裙。背景中有红色的花朵，暗示着室外的环境，可能是一个花园或公园。
-------------------- Stream Chat Output --------------------
图片中有一个穿着粉红色连衣裙的小孩，手里拿着一只穿着粉色芭蕾裙的白色泰迪熊。背景中有红色花朵和石头墙，表明照片可能是在户外拍摄的。
```
The sample input image is (which is fetched from [COCO dataset](https://cocodataset.org/#explore?id=264959)):

111
python/llm/example/GPU/HuggingFace/Multimodal/MiniCPM-V-2_6/chat.py Normal file
View file

@ -0,0 +1,111 @@
#
# Copyright 2016 The BigDL Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
import os
import time
import argparse
import requests
import torch
from PIL import Image
from ipex_llm.transformers import AutoModel
from transformers import AutoTokenizer
if __name__ == '__main__':
parser = argparse.ArgumentParser(description='Predict Tokens using `chat()` API for openbmb/MiniCPM-V-2_6 model')
parser.add_argument('--repo-id-or-model-path', type=str, default="openbmb/MiniCPM-V-2_6",
help='The huggingface repo id for the openbmb/MiniCPM-V-2_6 model to be downloaded'
', or the path to the huggingface checkpoint folder')
parser.add_argument('--image-url-or-path', type=str,
default='http://farm6.staticflickr.com/5268/5602445367_3504763978_z.jpg',
help='The URL or path to the image to infer')
parser.add_argument('--prompt', type=str, default="What is in the image?",
help='Prompt to infer')
parser.add_argument('--stream', action='store_true',
help='Whether to chat in streaming mode')
args = parser.parse_args()
model_path = args.repo_id_or_model_path
image_path = args.image_url_or_path
# Load model in 4 bit,
# which convert the relevant layers in the model into INT4 format
# When running LLMs on Intel iGPUs for Windows users, we recommend setting `cpu_embedding=True` in the from_pretrained function.
# This will allow the memory-intensive embedding layer to utilize the CPU instead of iGPU.
model = AutoModel.from_pretrained(model_path,
load_in_low_bit="sym_int4",
optimize_model=True,
trust_remote_code=True,
use_cache=True,
modules_to_not_convert=["vpm", "resampler"])
model = model.half().to('xpu')
tokenizer = AutoTokenizer.from_pretrained(model_path,
trust_remote_code=True)
model.eval()
query = args.prompt
if os.path.exists(image_path):
image = Image.open(image_path).convert('RGB')
else:
image = Image.open(requests.get(image_path, stream=True).raw).convert('RGB')
# Generate predicted tokens
# here the prompt tuning refers to https://huggingface.co/openbmb/MiniCPM-V-2_6/blob/main/README.md
msgs = [{'role': 'user', 'content': [image, args.prompt]}]
# ipex_llm model needs a warmup, then inference time can be accurate
model.chat(
image=None,
msgs=msgs,
context=None,
tokenizer=tokenizer,
)
if args.stream:
res = model.chat(
image=None,
msgs=msgs,
context=None,
tokenizer=tokenizer,
stream=True
)
print('-'*20, 'Input Image', '-'*20)
print(image_path)
print('-'*20, 'Input Prompt', '-'*20)
print(args.prompt)
print('-'*20, 'Stream Chat Output', '-'*20)
for new_text in res:
print(new_text, flush=True, end='')
else:
st = time.time()
res = model.chat(
image=None,
msgs=msgs,
context=None,
tokenizer=tokenizer,
)
torch.xpu.synchronize()
end = time.time()
print(f'Inference time: {end-st} s')
print('-'*20, 'Input Image', '-'*20)
print(image_path)
print('-'*20, 'Input Prompt', '-'*20)
print(args.prompt)
print('-'*20, 'Chat Output', '-'*20)
print(res)

175
python/llm/example/GPU/HuggingFace/Multimodal/MiniCPM-V-2_6/generate.py
View file

@ -1,175 +0,0 @@
#
# Copyright 2016 The BigDL Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
from typing import List, Tuple, Optional, Union
import math
import timm
import torch
import torch.nn.functional as F
# patched: `timm` has limited support for XPU backend, so we need to use CPU as a workaround
def resample_abs_pos_embed(
posemb: torch.Tensor,
new_size: List[int],
old_size: Optional[List[int]] = None,
num_prefix_tokens: int = 1,
interpolation: str = 'bicubic',
antialias: bool = True,
verbose: bool = False,
):
# sort out sizes, assume square if old size not provided
num_pos_tokens = posemb.shape[1]
num_new_tokens = new_size[0] * new_size[1] + num_prefix_tokens
if num_new_tokens == num_pos_tokens and new_size[0] == new_size[1]:
return posemb
if old_size is None:
hw = int(math.sqrt(num_pos_tokens - num_prefix_tokens))
old_size = hw, hw
if num_prefix_tokens:
posemb_prefix, posemb = posemb[:, :num_prefix_tokens], posemb[:, num_prefix_tokens:]
else:
posemb_prefix, posemb = None, posemb
# do the interpolation
embed_dim = posemb.shape[-1]
orig_dtype = posemb.dtype
posemb = posemb.float() # interpolate needs float32
posemb = posemb.reshape(1, old_size[0], old_size[1], -1).permute(0, 3, 1, 2)
#posemb = F.interpolate(posemb, size=new_size, mode=interpolation, antialias=antialias)
posemb = F.interpolate(posemb.to("cpu"), size=new_size, mode=interpolation, antialias=antialias).to(posemb.device)
posemb = posemb.permute(0, 2, 3, 1).reshape(1, -1, embed_dim)
posemb = posemb.to(orig_dtype)
# add back extra (class, etc) prefix tokens
if posemb_prefix is not None:
posemb = torch.cat([posemb_prefix, posemb], dim=1)
if not torch.jit.is_scripting() and verbose:
_logger.info(f'Resized position embedding: {old_size} to {new_size}.')
return posemb
def _pos_embed(self, x: torch.Tensor) -> torch.Tensor:
if self.pos_embed is None:
return x.view(x.shape[0], -1, x.shape[-1])
if self.dynamic_img_size:
B, H, W, C = x.shape
pos_embed = resample_abs_pos_embed(
self.pos_embed,
(H, W),
num_prefix_tokens=0 if self.no_embed_class else self.num_prefix_tokens,
)
x = x.view(B, -1, C)
else:
pos_embed = self.pos_embed
to_cat = []
if self.cls_token is not None:
to_cat.append(self.cls_token.expand(x.shape[0], -1, -1))
if self.reg_token is not None:
to_cat.append(self.reg_token.expand(x.shape[0], -1, -1))
if self.no_embed_class:
# deit-3, updated JAX (big vision)
# position embedding does not overlap with class token, add then concat
x = x + pos_embed
if to_cat:
x = torch.cat(to_cat + [x], dim=1)
else:
# original timm, JAX, and deit vit impl
# pos_embed has entry for class token, concat then add
if to_cat:
x = torch.cat(to_cat + [x], dim=1)
x = x + pos_embed
return self.pos_drop(x)
setattr(timm.models.VisionTransformer, "_pos_embed", _pos_embed)
import os
import time
import argparse
import requests
from PIL import Image
from ipex_llm.transformers import AutoModel
from transformers import AutoTokenizer
if __name__ == '__main__':
parser = argparse.ArgumentParser(description='Predict Tokens using `generate()` API for openbmb/MiniCPM-V-2_6 model')
parser.add_argument('--repo-id-or-model-path', type=str, default="openbmb/MiniCPM-V-2_6",
help='The huggingface repo id for the openbmb/MiniCPM-V-2_6 model to be downloaded'
', or the path to the huggingface checkpoint folder')
parser.add_argument('--image-url-or-path', type=str,
default='http://farm6.staticflickr.com/5268/5602445367_3504763978_z.jpg',
help='The URL or path to the image to infer')
parser.add_argument('--prompt', type=str, default="What is in the image?",
help='Prompt to infer')
parser.add_argument('--n-predict', type=int, default=32,
help='Max tokens to predict')
args = parser.parse_args()
model_path = args.repo_id_or_model_path
image_path = args.image_url_or_path
# Load model in 4 bit,
# which convert the relevant layers in the model into INT4 format
# When running LLMs on Intel iGPUs for Windows users, we recommend setting `cpu_embedding=True` in the from_pretrained function.
# This will allow the memory-intensive embedding layer to utilize the CPU instead of iGPU.
model = AutoModel.from_pretrained(model_path,
load_in_low_bit="sym_int4",
optimize_model=True,
trust_remote_code=True,
use_cache=True)
model = model.half().to(device='xpu')
tokenizer = AutoTokenizer.from_pretrained(model_path,
trust_remote_code=True)
model.eval()
query = args.prompt
if os.path.exists(image_path):
image = Image.open(image_path).convert('RGB')
else:
image = Image.open(requests.get(image_path, stream=True).raw).convert('RGB')
# Generate predicted tokens
# here the prompt tuning refers to https://huggingface.co/openbmb/MiniCPM-V-2_6/blob/main/README.md
msgs = [{'role': 'user', 'content': [image, args.prompt]}]
st = time.time()
res = model.chat(
image=None,
msgs=msgs,
context=None,
tokenizer=tokenizer,
sampling=False,
temperature=0.7
)
end = time.time()
print(f'Inference time: {end-st} s')
print('-'*20, 'Input', '-'*20)
print(image_path)
print('-'*20, 'Prompt', '-'*20)
print(args.prompt)
output_str = res
print('-'*20, 'Output', '-'*20)
print(output_str)