ayo/ipex-llm
1
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

[LLM] support for Yi AWQ model (#9648)

Browse source
This commit is contained in:
Heyang Sun 2023-12-11 14:07:34 +08:00 committed by GitHub
parent 82255f9726
commit 9f02f96160
3 changed files with 7 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
python/llm/example/CPU/HF-Transformers-AutoModels/Advanced-Quantizations/AWQ/README.md
View file

@ -7,6 +7,7 @@ This example shows how to directly run 4-bit AWQ models using BigDL-LLM on Intel
- [Mistral-7B-v0.1-AWQ](https://huggingface.co/TheBloke/Mistral-7B-v0.1-AWQ) - [Mistral-7B-v0.1-AWQ](https://huggingface.co/TheBloke/Mistral-7B-v0.1-AWQ)
- [vicuna-7B-v1.5-AWQ](https://huggingface.co/TheBloke/vicuna-7B-v1.5-AWQ) - [vicuna-7B-v1.5-AWQ](https://huggingface.co/TheBloke/vicuna-7B-v1.5-AWQ)
- [vicuna-13B-v1.5-AWQ](https://huggingface.co/TheBloke/vicuna-13B-v1.5-AWQ) - [vicuna-13B-v1.5-AWQ](https://huggingface.co/TheBloke/vicuna-13B-v1.5-AWQ)
- [Yi-6B-AWQ](https://huggingface.co/TheBloke/Yi-6B-AWQ)
## Requirements ## Requirements
To run these examples with BigDL-LLM, we have some recommended requirements for your machine, please refer to [here](../../../README.md#system-support) for more information. To run these examples with BigDL-LLM, we have some recommended requirements for your machine, please refer to [here](../../../README.md#system-support) for more information.
@ -23,6 +24,7 @@ pip install autoawq==0.1.6 --no-deps
pip install --pre --upgrade bigdl-llm[all] # install bigdl-llm with 'all' option pip install --pre --upgrade bigdl-llm[all] # install bigdl-llm with 'all' option
pip install transformers==4.35.0 pip install transformers==4.35.0
pip install accelerate==0.24.1 pip install accelerate==0.24.1
pip install einops
``` ```
### 2. Run ### 2. Run

2
python/llm/example/GPU/HF-Transformers-AutoModels/Advanced-Quantizations/AWQ/README.md
View file

@ -7,6 +7,7 @@ This example shows how to directly run 4-bit AWQ models using BigDL-LLM on Intel
- [Mistral-7B-v0.1-AWQ](https://huggingface.co/TheBloke/Mistral-7B-v0.1-AWQ) - [Mistral-7B-v0.1-AWQ](https://huggingface.co/TheBloke/Mistral-7B-v0.1-AWQ)
- [vicuna-7B-v1.5-AWQ](https://huggingface.co/TheBloke/vicuna-7B-v1.5-AWQ) - [vicuna-7B-v1.5-AWQ](https://huggingface.co/TheBloke/vicuna-7B-v1.5-AWQ)
- [vicuna-13B-v1.5-AWQ](https://huggingface.co/TheBloke/vicuna-13B-v1.5-AWQ) - [vicuna-13B-v1.5-AWQ](https://huggingface.co/TheBloke/vicuna-13B-v1.5-AWQ)
- [Yi-6B-AWQ](https://huggingface.co/TheBloke/Yi-6B-AWQ)
## Requirements ## Requirements
To run these examples with BigDL-LLM, we have some recommended requirements for your machine, please refer to [here](../../../README.md#requirements) for more information. To run these examples with BigDL-LLM, we have some recommended requirements for your machine, please refer to [here](../../../README.md#requirements) for more information.
@ -23,6 +24,7 @@ pip install --pre --upgrade bigdl-llm[xpu] -f https://developer.intel.com/ipex-w
pip install transformers==4.35.0 pip install transformers==4.35.0
pip install autoawq==0.1.6 --no-deps pip install autoawq==0.1.6 --no-deps
pip install accelerate==0.24.1 pip install accelerate==0.24.1
pip install einops
``` ```
### 2. Configures OneAPI environment variables ### 2. Configures OneAPI environment variables

3
python/llm/src/bigdl/llm/transformers/awq/awq.py
View file

@ -70,6 +70,7 @@ layer_type_dict = {
"mistral": "MistralDecoderLayer", "mistral": "MistralDecoderLayer",
"gpt_neox": "GPTNeoXDecoderLayer", "gpt_neox": "GPTNeoXDecoderLayer",
"aquila": "AquilaDecoderLayer", "aquila": "AquilaDecoderLayer",
"Yi": "YiDecoderLayer",
} }
@ -133,6 +134,8 @@ def get_blocks(model):
layers = model.gpt_neox.layers layers = model.gpt_neox.layers
elif "mistral" in str(model.__class__).lower(): elif "mistral" in str(model.__class__).lower():
layers = model.model.layers layers = model.model.layers
elif "yi" in str(model.__class__).lower():
layers = model.model.layers
else: else:
invalidInputError(False, f"Model type {type(model)} isn't supported.") invalidInputError(False, f"Model type {type(model)} isn't supported.")
return layers return layers