ayo/ipex-llm
1
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1
ipex-llm/python/llm/example/gpu
History
Ruonan Wang 4de73f592e LLM: add gpu example of chinese-llama-2-7b (#8960) 
* add gpu example of chinese -llama2

* update model name and link

* update name
2023-09-13 10:16:51 +08:00
..
baichuan Reorganize GPU examples (#8844) 2023-08-30 08:32:08 +08:00
chatglm2 Update optimize_model=True in llama2 chatglm2 arc examples (#8878) 2023-09-05 10:35:37 +08:00
chinese-llama2 LLM: add gpu example of chinese-llama-2-7b (#8960) 2023-09-13 10:16:51 +08:00
falcon Reorganize GPU examples (#8844) 2023-08-30 08:32:08 +08:00
gpt-j Update optimize_model=True in llama2 chatglm2 arc examples (#8878) 2023-09-05 10:35:37 +08:00
internlm Reorganize GPU examples (#8844) 2023-08-30 08:32:08 +08:00
llama2 Update optimize_model=True in llama2 chatglm2 arc examples (#8878) 2023-09-05 10:35:37 +08:00
mpt Reorganize GPU examples (#8844) 2023-08-30 08:32:08 +08:00
qwen Reorganize GPU examples (#8844) 2023-08-30 08:32:08 +08:00
starcoder Reorganize GPU examples (#8844) 2023-08-30 08:32:08 +08:00
voiceassistant Add known issue in arc voice assistant example (#8902) 2023-09-07 09:28:26 +08:00
whisper Reorganize GPU examples (#8844) 2023-08-30 08:32:08 +08:00
README.md LLM: update GPU doc (#8845) 2023-08-30 09:24:19 +08:00

README.md

BigDL-LLM Transformers INT4 Optimization for Large Language Model on Intel GPUs

You can use BigDL-LLM to run almost every Huggingface Transformer models with INT4 optimizations on your laptops with Intel GPUs. This directory contains example scripts to help you quickly get started using BigDL-LLM to run some popular open-source models in the community. Each model has its own dedicated folder, where you can find detailed instructions on how to install and run it.

Verified Hardware Platforms

  • Intel Arc™ A-Series Graphics
  • Intel Data Center GPU Flex Series

Recommended Requirements

To apply Intel GPU acceleration, there’re several steps for tools installation and environment preparation.

Step 1, only Linux system is supported now, Ubuntu 22.04 is prefered.

Step 2, please refer to our driver installation for general purpose GPU capabilities.

Note

: IPEX 2.0.110+xpu requires Intel GPU Driver version is Stable 647.21.

Step 3, you also need to download and install Intel® oneAPI Base Toolkit. OneMKL and DPC++ compiler are needed, others are optional.

Note

: IPEX 2.0.110+xpu requires Intel® oneAPI Base Toolkit's version >= 2023.2.0.

Best Known Configuration on Linux

For better performance, it is recommended to set environment variables on Linux:

export USE_XETLA=OFF
export SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=1