ayo/ipex-llm
1
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

bugfix for qlora finetuning on GPU (#12298)

Browse source 
* bugfix for qlora 100 step error

* indent fix

* annotation fix
This commit is contained in:
Jinhe 2024-10-30 16:54:10 +08:00 committed by GitHub
parent 70037ad55f
commit 46d8300f6b
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
2 changed files with 8 additions and 2 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
python/llm/example/GPU/LLM-Finetuning/QLoRA/alpaca-qlora/README.md
View file

@ -23,6 +23,8 @@ pip install bitsandbytes scipy
source /opt/intel/oneapi/setvars.sh # necessary to run before installing deepspeed
pip install git+https://github.com/microsoft/DeepSpeed.git@78c518e
pip install git+https://github.com/intel/intel-extension-for-deepspeed.git@ec33277
# (optional) install mpirun to run multi-card finetuning
sudo apt install openmpi-bin
```
### 2. Configures OneAPI environment variables

8
python/llm/src/ipex_llm/transformers/low_bit_linear.py
View file

@ -794,8 +794,12 @@ class LowBitLinear(nn.Linear):
self.weight.qtype, input_seq_size)
result = result.to(x.dtype)
else:
result = xe_linear.forward_new(x_2d, self.weight.data,
self.weight.qtype, input_seq_size)
if self.weight.qtype == NF4:
result = xe_linear.forward_new(x_2d, self.weight.data.view(torch.uint8),
self.weight.qtype, input_seq_size)
else:
result = xe_linear.forward_new(x_2d, self.weight.data,
self.weight.qtype, input_seq_size)
if do_empty_cache:
torch.xpu.empty_cache()