ayo/ipex-llm
1
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

update transformer version (#9631)

Browse source
This commit is contained in:
ZehuaCao 2023-12-08 09:36:00 +08:00 committed by GitHub
parent e9299adb3b
commit 6eca8a8bb5
3 changed files with 39 additions and 20 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

53
python/llm/example/CPU/QLoRA-FineTuning/alpaca-qlora/README.md
View file

@ -8,12 +8,13 @@ This example ports [Alpaca-LoRA](https://github.com/tloen/alpaca-lora/tree/main)
conda create -n llm python=3.9 conda create -n llm python=3.9
conda activate llm conda activate llm
pip install --pre --upgrade bigdl-llm[all] pip install --pre --upgrade bigdl-llm[all]
pip install datasets transformers==4.34.0 pip install datasets transformers==4.35.0
pip install fire peft==0.5.0 pip install fire peft==0.5.0
pip install accelerate==0.23.0 pip install accelerate==0.23.0
``` ```
### 2. Configures environment variables ### 2. Configures environment variables
```bash ```bash
source bigdl-llm-init -t source bigdl-llm-init -t
``` ```
@ -32,43 +33,52 @@ python ./alpaca_qlora_finetuning_cpu.py \
**Note**: You could also specify `--base_model` to the local path of the huggingface model checkpoint folder and `--data_path` to the local path of the dataset JSON file. **Note**: You could also specify `--base_model` to the local path of the huggingface model checkpoint folder and `--data_path` to the local path of the dataset JSON file.
#### Sample Output #### Sample Output
```log ```log
{'loss': 1.9231, 'learning_rate': 2.9999945367033285e-05, 'epoch': 0.0} {'loss': 1.9231, 'learning_rate': 2.9999945367033285e-05, 'epoch': 0.0}
{'loss': 1.8622, 'learning_rate': 2.9999781468531096e-05, 'epoch': 0.01} {'loss': 1.8622, 'learning_rate': 2.9999781468531096e-05, 'epoch': 0.01}
{'loss': 1.9043, 'learning_rate': 2.9999508305687345e-05, 'epoch': 0.01} {'loss': 1.9043, 'learning_rate': 2.9999508305687345e-05, 'epoch': 0.01}
{'loss': 1.8967, 'learning_rate': 2.999912588049185e-05, 'epoch': 0.01} {'loss': 1.8967, 'learning_rate': 2.999912588049185e-05, 'epoch': 0.01}
{'loss': 1.9658, 'learning_rate': 2.9998634195730358e-05, 'epoch': 0.01} {'loss': 1.9658, 'learning_rate': 2.9998634195730358e-05, 'epoch': 0.01}
{'loss': 1.8386, 'learning_rate': 2.9998033254984483e-05, 'epoch': 0.02} {'loss': 1.8386, 'learning_rate': 2.9998033254984483e-05, 'epoch': 0.02}
{'loss': 1.809, 'learning_rate': 2.999732306263172e-05, 'epoch': 0.02} {'loss': 1.809, 'learning_rate': 2.999732306263172e-05, 'epoch': 0.02}
{'loss': 1.8552, 'learning_rate': 2.9996503623845395e-05, 'epoch': 0.02} {'loss': 1.8552, 'learning_rate': 2.9996503623845395e-05, 'epoch': 0.02}
1%|█ | 8/1164 [xx:xx<xx:xx:xx, xx s/it] 1%|█ | 8/1164 [xx:xx<xx:xx:xx, xx s/it]
``` ```
### Guide to finetuning QLoRA on one node with multiple sockets ### Guide to finetuning QLoRA on one node with multiple sockets
1. install extra lib 1. install extra lib
```bash ```bash
# need to run the alpaca stand-alone version first # need to run the alpaca stand-alone version first
# for using mpirun # for using mpirun
pip install oneccl_bind_pt -f https://developer.intel.com/ipex-whl-stable pip install oneccl_bind_pt -f https://developer.intel.com/ipex-whl-stable
``` ```
2. modify conf in `finetune_one_node_two_sockets.sh` and run 2. modify conf in `finetune_one_node_two_sockets.sh` and run
``` ```
source ${conda_env}/lib/python3.9/site-packages/oneccl_bindings_for_pytorch/env/setvars.sh source ${conda_env}/lib/python3.9/site-packages/oneccl_bindings_for_pytorch/env/setvars.sh
bash finetune_one_node_two_sockets.sh bash finetune_one_node_two_sockets.sh
``` ```
### Guide to use different prompts or different datasets ### Guide to use different prompts or different datasets
Now the prompter is for the datasets with `instruction` `input`(optional) and `output`. If you want to use different datasets, Now the prompter is for the datasets with `instruction` `input`(optional) and `output`. If you want to use different datasets,
you can add template file xxx.json in templates. And then update utils.prompter.py's `generate_prompt` method and update `generate_and_tokenize_prompt` method to fix the dataset. you can add template file xxx.json in templates. And then update utils.prompter.py's `generate_prompt` method and update `generate_and_tokenize_prompt` method to fix the dataset.
For example, I want to train llama2-7b with [english_quotes](https://huggingface.co/datasets/Abirate/english_quotes) just like [this example](https://github.com/intel-analytics/BigDL/blob/main/python/llm/example/CPU/QLoRA-FineTuning/qlora_finetuning_cpu.py) For example, I want to train llama2-7b with [english_quotes](https://huggingface.co/datasets/Abirate/english_quotes) just like [this example](https://github.com/intel-analytics/BigDL/blob/main/python/llm/example/CPU/QLoRA-FineTuning/qlora_finetuning_cpu.py)
1. add template english_quotes.json 1. add template english_quotes.json
```json ```json
{ {
"prompt": "{quote} ->: {tags}" "prompt": "{quote} ->: {tags}"
} }
``` ```
2. update prompter.py and add new generate_prompt method 2. update prompter.py and add new generate_prompt method
```python ```python
def generate_quote_prompt(self, quote: str, tags: Union[None, list]=None,) -> str: def generate_quote_prompt(self, quote: str, tags: Union[None, list]=None,) -> str:
tags = str(tags) tags = str(tags)
@ -79,7 +89,9 @@ def generate_quote_prompt(self, quote: str, tags: Union[None, list]=None,) -> st
print(res) print(res)
return res return res
``` ```
3. update generate_and_tokenize_prompt method 3. update generate_and_tokenize_prompt method
```python ```python
def generate_and_tokenize_prompt(data_point): def generate_and_tokenize_prompt(data_point):
full_prompt = prompter.generate_quote_prompt( full_prompt = prompter.generate_quote_prompt(
@ -89,7 +101,9 @@ def generate_and_tokenize_prompt(data_point):
data_point["quote"], data_point["tags"] data_point["quote"], data_point["tags"]
) )
``` ```
4. choose prompt `english_quotes` to train 4. choose prompt `english_quotes` to train
```bash ```bash
python ./quotes_qlora_finetuning_cpu.py \ python ./quotes_qlora_finetuning_cpu.py \
--base_model "meta-llama/Llama-2-7b-hf" \ --base_model "meta-llama/Llama-2-7b-hf" \
@ -98,37 +112,44 @@ python ./quotes_qlora_finetuning_cpu.py \
--prompt_template_name "english_quotes" --prompt_template_name "english_quotes"
``` ```
### Guide to finetuning QLoRA using different models ### Guide to finetuning QLoRA using different models
Make sure you fully understand the entire finetune process and the model is the latest version. Make sure you fully understand the entire finetune process and the model is the latest version.
Using [Baichuan-7B](https://huggingface.co/baichuan-inc/Baichuan-7B/tree/main) as an example: Using [Baichuan-7B](https://huggingface.co/baichuan-inc/Baichuan-7B/tree/main) as an example:
1. Update the Tokenizer first. Because the base example is for llama model. 1. Update the Tokenizer first. Because the base example is for llama model.
```bash ```bash
from transformers import LlamaTokenizer from transformers import LlamaTokenizer
AutoTokenizer.from_pretrained(base_model) AutoTokenizer.from_pretrained(base_model)
``` ```
2. Maybe some models need to add `trust_remote_code=True` in from_pretrained model and tokenizer 2. Maybe some models need to add `trust_remote_code=True` in from_pretrained model and tokenizer
``` ```
tokenizer = AutoTokenizer.from_pretrained(base_model, trust_remote_code=True) tokenizer = AutoTokenizer.from_pretrained(base_model, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(base_model, xxxxx, trust_remote_code=True) model = AutoModelForCausalLM.from_pretrained(base_model, xxxxx, trust_remote_code=True)
``` ```
3. Modify the `target_modules` according to the model you need to train, you can refer to [here](https://stackoverflow.com/questions/76768226/target-modules-for-applying-peft-lora-on-different-models/76779946#76779946). 3. Modify the `target_modules` according to the model you need to train, you can refer to [here](https://stackoverflow.com/questions/76768226/target-modules-for-applying-peft-lora-on-different-models/76779946#76779946).
Or just search for the recommended training target modules. Or just search for the recommended training target modules.
```bash
```bash
lora_target_modules: List[str] = ["W_pack"] lora_target_modules: List[str] = ["W_pack"]
``` ```
4. Maybe need to change the `tokenizer.pad_token_id = tokenizer.eod_id` (Qwen) 4. Maybe need to change the `tokenizer.pad_token_id = tokenizer.eod_id` (Qwen)
5. (Only for baichuan) According to this [issue](https://github.com/baichuan-inc/Baichuan2/issues/204#issuecomment-1774372008), 5. (Only for baichuan) According to this [issue](https://github.com/baichuan-inc/Baichuan2/issues/204#issuecomment-1774372008),
need to modify the [tokenization_baichuan.py](https://huggingface.co/baichuan-inc/Baichuan-7B/blob/main/tokenization_baichuan.py#L74) to fix issue. need to modify the [tokenization_baichuan.py](https://huggingface.co/baichuan-inc/Baichuan-7B/blob/main/tokenization_baichuan.py#L74) to fix issue.
6. finetune as normal 6. finetune as normal
7. Using the [export_merged_model.py](https://github.com/intel-analytics/BigDL/blob/main/python/llm/example/GPU/QLoRA-FineTuning/export_merged_model.py) to merge. But also need to update tokenizer and model to ensure successful merge weight. 7. Using the [export_merged_model.py](https://github.com/intel-analytics/BigDL/blob/main/python/llm/example/GPU/QLoRA-FineTuning/export_merged_model.py) to merge. But also need to update tokenizer and model to ensure successful merge weight.
```bash ```bash
from transformers import AutoTokenizer # noqa: F402 from transformers import AutoTokenizer # noqa: F402
tokenizer = AutoTokenizer.from_pretrained(base_model, trust_remote_code=True) tokenizer = AutoTokenizer.from_pretrained(base_model, trust_remote_code=True)
base_model = AutoModelForCausalLM.from_pretrained(base_model,trust_remote_code=True) base_model = AutoModelForCausalLM.from_pretrained(base_model,trust_remote_code=True)
``` ```
### 4. Finetuning in docker and multiple nodes (k8s) ### 4. Finetuning in docker and multiple nodes (k8s)
If you want to run multi-process fine-tuning, or do not want to manually install the above dependencies, we provide a docker solution to quickly start a one-container finetuning. Please refer to [here](https://github.com/intel-analytics/BigDL/tree/main/docker/llm/finetune/qlora/cpu/docker#fine-tune-llm-with-bigdl-llm-container). If you want to run multi-process fine-tuning, or do not want to manually install the above dependencies, we provide a docker solution to quickly start a one-container finetuning. Please refer to [here](https://github.com/intel-analytics/BigDL/tree/main/docker/llm/finetune/qlora/cpu/docker#fine-tune-llm-with-bigdl-llm-container).

3
python/llm/example/CPU/QLoRA-FineTuning/alpaca-qlora/alpaca_qlora_finetuning_cpu.py
View file

@ -41,7 +41,6 @@ import accelerate
from transformers import LlamaTokenizer from transformers import LlamaTokenizer
from peft import ( from peft import (
LoraConfig,
get_peft_model_state_dict, get_peft_model_state_dict,
set_peft_model_state_dict, set_peft_model_state_dict,
) )
@ -50,7 +49,7 @@ from utils.prompter import Prompter
from bigdl.llm.transformers import AutoModelForCausalLM from bigdl.llm.transformers import AutoModelForCausalLM
# import them from bigdl.llm.transformers.qlora to get a BigDL-LLM compatible Peft model # import them from bigdl.llm.transformers.qlora to get a BigDL-LLM compatible Peft model
from bigdl.llm.transformers.qlora import get_peft_model, prepare_model_for_kbit_training from bigdl.llm.transformers.qlora import get_peft_model, prepare_model_for_kbit_training, LoraConfig
from bigdl.llm.utils.isa_checker import ISAChecker from bigdl.llm.utils.isa_checker import ISAChecker
def get_int_from_env(env_keys, default): def get_int_from_env(env_keys, default):

3
python/llm/example/CPU/QLoRA-FineTuning/qlora_finetuning_cpu.py
View file

@ -20,8 +20,7 @@ import os
import transformers import transformers
from transformers import LlamaTokenizer from transformers import LlamaTokenizer
from peft import LoraConfig from bigdl.llm.transformers.qlora import get_peft_model, prepare_model_for_kbit_training, LoraConfig
from bigdl.llm.transformers.qlora import get_peft_model, prepare_model_for_kbit_training
from bigdl.llm.transformers import AutoModelForCausalLM from bigdl.llm.transformers import AutoModelForCausalLM
from datasets import load_dataset from datasets import load_dataset
import argparse import argparse