ayo/ipex-llm
1
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

Reorganize LLM GPU installation document (#9777)

Browse source
This commit is contained in:
Yishuo Wang 2024-01-03 13:53:05 +08:00 committed by GitHub
parent 3ab3105bab
commit 5c6543e070
1 changed files with 194 additions and 173 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

367
docs/readthedocs/source/doc/LLM/Overview/install_gpu.md
View file

@ -1,22 +1,118 @@
# BigDL-LLM Installation: GPU # BigDL-LLM Installation: GPU
## Windows
## PyTorch 2.1 ### Prerequisites
### Prerequisite
BigDL-LLM on Windows supports Intel iGPU and dGPU.
```eval_rst
.. important::
BigDL-LLM on Windows only supports PyTorch 2.1.
```
To apply Intel GPU acceleration, there're several prerequisite steps for tools installation and environment preparation:
* Step 1: Install [Visual Studio 2022](https://visualstudio.microsoft.com/downloads/) Community Edition and select "Desktop development with C++" workload, like [this](https://learn.microsoft.com/en-us/cpp/build/vscpp-step-0-installation?view=msvc-170#step-4---choose-workloads)
* Step 2: Install or update to latest [GPU driver](https://www.intel.com/content/www/us/en/download/785597/intel-arc-iris-xe-graphics-windows.html)
* Step 3: Install [Intel® oneAPI Base Toolkit](https://www.intel.com/content/www/us/en/developer/tools/oneapi/base-toolkit-download.html) 2024.0
### Install BigDL-LLM From PyPI
We recommend using [miniconda](https://docs.conda.io/en/latest/miniconda.html) to create a python 3.9 enviroment:
```eval_rst
.. important::
``bigdl-llm`` is tested with Python 3.9, 3.10 and 3.11. Python 3.9 is recommended for best practices.
```
The easiest ways to install `bigdl-llm` is the following commands:
```
conda create -n llm python=3.9 libuv
conda activate llm
pip install --pre --upgrade bigdl-llm[xpu] -f https://developer.intel.com/ipex-whl-stable-xpu
```
### Install BigDL-LLM From Wheel
If you encounter network issues when installing IPEX, you can also install BigDL-LLM dependencies for Intel XPU from source achieves. First you need to download and install torch/torchvision/ipex from wheels listed below before installing `bigdl-llm`.
Download the wheels on Windows system:
```
wget https://intel-extension-for-pytorch.s3.amazonaws.com/ipex_stable/xpu/torch-2.1.0a0%2Bcxx11.abi-cp39-cp39-win_amd64.whl
wget https://intel-extension-for-pytorch.s3.amazonaws.com/ipex_stable/xpu/torchvision-0.16.0a0%2Bcxx11.abi-cp39-cp39-win_amd64.whl
wget https://intel-extension-for-pytorch.s3.amazonaws.com/ipex_stable/xpu/intel_extension_for_pytorch-2.1.10%2Bxpu-cp39-cp39-win_amd64.whl
```
You may install dependencies directly from the wheel archives and then install `bigdl-llm` using following commands:
```
pip install torch-2.1.0a0+cxx11.abi-cp39-cp39-win_amd64.whl
pip install torchvision-0.16.0a0+cxx11.abi-cp39-cp39-win_amd64.whl
pip install intel_extension_for_pytorch-2.1.10+xpu-cp39-cp39-win_amd64.whl
pip install --pre --upgrade bigdl-llm[xpu]
```
### Runtime Configuration
To use GPU acceleration on Windows, several environment variables are required before running a GPU example.
Make sure you are using CMD as PowerShell is not supported:
```
call "C:\Program Files (x86)\Intel\oneAPI\setvars.bat"
```
Please also set the following environment variable for iGPU:
```
set SYCL_CACHE_PERSISTENT=1
set BIGDL_LLM_XMX_DISABLED=1
```
```eval_rst
.. note::
For the first time that **each model** runs on **iGPU**, it may take around several minutes to compile.
```
### Troubleshooting
todo
## Linux
### Prerequisites
BigDL-LLM for GPU supports on Linux has been verified on:
* Intel Arc™ A-Series Graphics
* Intel Data Center GPU Flex Series
* Intel Data Center GPU Max Series
```eval_rst
.. important::
BigDL-LLM on Linux only supports PyTorch 2.0 and PyTorch 2.1.
```
```eval_rst
.. important::
We currently support the Ubuntu 20.04 operating system and later.
```
```eval_rst ```eval_rst
.. tabs:: .. tabs::
.. tab:: Linux .. tab:: PyTorch 2.1
BigDL-LLM for GPU supports on Linux with PyTorch 2.1 has been verified on:
* Intel Arc™ A-Series Graphics
* Intel Data Center GPU Flex Series
* Intel Data Center GPU Max Series
.. note::
We currently support the Ubuntu 20.04 operating system or later.
To enable BigDL-LLM for Intel GPUs with PyTorch 2.1, here're several prerequisite steps for tools installation and environment preparation: To enable BigDL-LLM for Intel GPUs with PyTorch 2.1, here're several prerequisite steps for tools installation and environment preparation:
@ -29,38 +125,46 @@
See `release page <https://dgpu-docs.intel.com/releases/index.html>`_ for latest version. See `release page <https://dgpu-docs.intel.com/releases/index.html>`_ for latest version.
* Step 2: Download and install `Intel® oneAPI Base Toolkit <https://www.intel.com/content/www/us/en/developer/tools/oneapi/base-toolkit-download.html>`_ with version 2024.0. OneMKL and DPC++ compiler are needed, others are optional. * Step 2: Download and install `Intel® oneAPI Base Toolkit <https://www.intel.com/content/www/us/en/developer/tools/oneapi/base-toolkit-download.html>`_ with version 2024.0. OneDNN, OneMKL and DPC++ compiler are needed, others are optional.
.. seealso:: .. seealso::
We recommend you to use `this offline package <https://registrationcenter-download.intel.com/akdlm/IRC_NAS/20f4e6a1-6b0b-4752-b8c1-e5eacba10e01/l_BaseKit_p_2024.0.0.49564_offline.sh>`_ to install oneapi. We recommend you to use `this offline package <https://registrationcenter-download.intel.com/akdlm/IRC_NAS/20f4e6a1-6b0b-4752-b8c1-e5eacba10e01/l_BaseKit_p_2024.0.0.49564_offline.sh>`_ to install oneapi.
.. tab:: Windows .. tab:: Pytorch 2.0
BigDL-LLM on Windows supports Intel iGPU and dGPU. To enable BigDL-LLM for Intel GPUs with PyTorch 2.0, here're several prerequisite steps for tools installation and environment preparation:
To apply Intel GPU acceleration, there're several prerequisite steps for tools installation and environment preparation:
* Step 1: Install `Visual Studio 2022 <https://visualstudio.microsoft.com/downloads/>`_ Community Edition and select "Desktop development with C++" workload * Step 1: Install Intel GPU Driver version >= stable_775_20_20231219. Highly recommend installing the latest version of intel-i915-dkms using apt.
* Step 2: Install or update to latest GPU driver .. seealso::
Please refer to our `driver installation <https://dgpu-docs.intel.com/driver/installation.html>`_ for general purpose GPU capabilities.
See `release page <https://dgpu-docs.intel.com/releases/index.html>`_ for latest version.
* Step 2: Download and install `Intel® oneAPI Base Toolkit <https://www.intel.com/content/www/us/en/developer/tools/oneapi/base-toolkit-download.html>`_ with version 2023.2. OneDNN, OneMKL and DPC++ compiler are needed, others are optional.
.. seealso::
We recommend you to use `this offline package <https://registrationcenter-download.intel. com/akdlm/IRC_NAS/992857b9-624c-45de-9701-f6445d845359/l_BaseKit_p_2023.2.0.49397_offline. sh>`_ to install oneapi.
* Step 3: Install `Intel® oneAPI Base Toolkit <https://www.intel.com/content/www/us/en/developer/tools/oneapi/base-toolkit-download.html>`_ 2024.0
``` ```
### Install BigDL-LLM ### Install BigDL-LLM From PyPI
We recommend using [Conda](https://docs.conda.io/en/latest/miniconda.html) to create a python 3.9 enviroment: We recommend using [miniconda](https://docs.conda.io/en/latest/miniconda.html) to create a python 3.9 enviroment:
```eval_rst ```eval_rst
.. important:: .. important::
``bigdl-llm`` is tested with Python 3.9, which is recommended for best practices. ``bigdl-llm`` is tested with Python 3.9, 3.10 and 3.11. Python 3.9 is recommended for best practices.
``` ```
```eval_rst ```eval_rst
.. tabs:: .. tabs::
.. tab:: Linux .. tab:: Pytorch 2.1
.. code-block:: bash .. code-block:: bash
@ -69,145 +173,24 @@ We recommend using [Conda](https://docs.conda.io/en/latest/miniconda.html) to cr
pip install --pre --upgrade bigdl-llm[xpu_2.1] -f https://developer.intel.com/ipex-whl-stable-xpu pip install --pre --upgrade bigdl-llm[xpu_2.1] -f https://developer.intel.com/ipex-whl-stable-xpu
.. tab:: Windows .. tab:: Pytorch 2.0
.. code-block:: cmd .. code-block:: bash
conda create -n llm python=3.9 libuv conda create -n llm python=3.9
conda activate llm conda activate llm
pip install --pre --upgrade bigdl-llm[xpu] -f https://developer.intel.com/ipex-whl-stable-xpu pip install --pre --upgrade bigdl-llm[xpu] -f https://developer.intel.com/ipex-whl-stable-xpu
``` ```
## PyTorch 2.0 ### Install BigDL-LLM From Wheel
```eval_rst
.. note::
BigDL-LLM for GPU with PyTorch 2.0 supports Ubuntu 20.04 operating system or later. If you encounter network issues when installing IPEX, you can also install BigDL-LLM dependencies for Intel XPU from source achieves. First you need to download and install torch/torchvision/ipex from wheels listed below before installing `bigdl-llm`.
```
### Prerequisite
BigDL-LLM for GPU supports on Linux with PyTorch 2.0 has been verified on:
* Intel Arc™ A-Series Graphics
* Intel Data Center GPU Flex Series
* Intel Data Center GPU Max Series
To enable BigDL-LLM for Intel GPUs with PyTorch 2.0, here're several prerequisite steps for tools installation and environment preparation:
- Step 1: Install Intel GPU Driver version >= stable_775_20_20231219. Highly recommend installing the latest version of intel-i915-dkms using apt.
```eval_rst
.. seealso::
Please refer to our `driver installation <https://dgpu-docs.intel.com/driver/installation.html>`_ for general purpose GPU capabilities.
See `release page <https://dgpu-docs.intel.com/releases/index.html>`_ for latest version.
```
- Step 2: Download and install [Intel® oneAPI Base Toolkit](https://www.intel.com/content/www/us/en/developer/tools/oneapi/base-toolkit-download.html) with version 2023.2.0. OneMKL and DPC++ compiler are needed, others are optional.
```eval_rst
.. seealso::
We recommend you to use `this offline package <https://registrationcenter-download.intel.com/akdlm/IRC_NAS/992857b9-624c-45de-9701-f6445d845359/l_BaseKit_p_2023.2.0.49397_offline.sh>`_ to install oneapi.
```
### Install BigDL-LLM
We recommend using [Conda](https://docs.conda.io/en/latest/miniconda.html) to create a python 3.9 enviroment:
```eval_rst
.. important::
``bigdl-llm`` is tested with Python 3.9, which is recommended for best practices.
```
```bash
conda create -n llm python=3.9
conda activate llm
pip install --pre --upgrade bigdl-llm[xpu] -f https://developer.intel.com/ipex-whl-stable-xpu
```
## Known issues
### 1. For Linux users, Ubuntu 22.04 and Linux kernel 6.2.0 may cause performance bad (driver version < stable_775_20_20231219).
For driver version < stable_775_20_20231219, the performance on Linux kernel 6.2.0 is worse than Linux kernel 5.19.0. You can use `sudo apt update && sudo apt install -y intel-i915-dkms intel-fw-gpu` to install the latest driver to solve this issue (need reboot OS).
Tips: You can use `sudo apt list --installed | grep intel-i915-dkms` to check your intel-i915-dkms's version, the version should be latest and >= `1.23.9.11.231003.15+i19-1`.
### 2. For Linux users, driver installation meet unmet dependencies: intel-i915-dkms
The last apt install command of the driver installation may get following error:
```
The following packages have unmet dependencies:
intel-i915-dkms : Conflicts: intel-platform-cse-dkms
Conflicts: intel-platform-vsec-dkms
```
You can use `sudo apt install -y intel-i915-dkms intel-fw-gpu` to instead. As the intel-platform-cse-dkms and intel-platform-vsec-dkms are already provided by intel-i915-dkms.
### 3. Best known configurations
For running a LLM model with BigDL-LLM optimizations, several environment variables are recommended:
```eval_rst ```eval_rst
.. tabs:: .. tabs::
.. tab:: Linux .. tab:: PyTorch 2.1
For Intel Arc™ A-Series Graphics and Intel Data Center GPU Flex Series, we recommend:
.. code-block:: bash
# configures OneAPI environment variables
source /opt/intel/oneapi/setvars.sh
export USE_XETLA=OFF
export SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=1
For Intel Data Center GPU Max Series, we recommend:
.. code-block:: bash
# configures OneAPI environment variables
source /opt/intel/oneapi/setvars.sh
export LD_PRELOAD=${LD_PRELOAD}:${CONDA_PREFIX}/lib/libtcmalloc.so
export SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=1
export ENABLE_SDP_FUSION=1
Please note that ``libtcmalloc.so`` can installed by ``conda install -c conda-forge -y gperftools=2.10``
.. tab:: Windows
Make sure you are using CMD as PowerShell is not supported:
.. code-block:: cmd
# configures OneAPI environment variables
call "C:\Program Files (x86)\Intel\oneAPI\setvars.bat"
set SYCL_CACHE_PERSISTENT=1
Please also set the following environment variable for iGPU:
.. code-block:: cmd
set BIGDL_LLM_XMX_DISABLED=1
.. note::
For the first time that **each** model runs on **a new machine**, it may take around several minutes to compile.
```
### 4. How to install from wheel
If you encounter network issues when installing IPEX, you can also install BigDL-LLM dependencies for Intel XPU from source achieves. First you need to install the target torch/torchvision/ipex versions from downloaded wheels [here](https://developer.intel.com/ipex-whl-stable-xpu) before installing `bigdl-llm`.
```eval_rst
.. tabs::
.. tab:: PyTorch 2.1 Linux
.. code-block:: bash .. code-block:: bash
@ -228,28 +211,7 @@ If you encounter network issues when installing IPEX, you can also install BigDL
# install bigdl-llm for Intel GPU # install bigdl-llm for Intel GPU
pip install --pre --upgrade bigdl-llm[xpu_2.1] pip install --pre --upgrade bigdl-llm[xpu_2.1]
.. tab:: PyTorch 2.1 Windows .. tab:: PyTorch 2.0
.. code-block:: bash
# get the wheels on Windows system
wget https://intel-extension-for-pytorch.s3.amazonaws.com/ipex_stable/xpu/torch-2.1.0a0%2Bcxx11.abi-cp39-cp39-win_amd64.whl
wget https://intel-extension-for-pytorch.s3.amazonaws.com/ipex_stable/xpu/torchvision-0.16.0a0%2Bcxx11.abi-cp39-cp39-win_amd64.whl
wget https://intel-extension-for-pytorch.s3.amazonaws.com/ipex_stable/xpu/intel_extension_for_pytorch-2.1.10%2Bxpu-cp39-cp39-win_amd64.whl
Then you may install directly from the wheel archives using following commands:
.. code-block:: cmd
# install the packages from the wheels
pip install torch-2.1.0a0+cxx11.abi-cp39-cp39-win_amd64.whl
pip install torchvision-0.16.0a0+cxx11.abi-cp39-cp39-win_amd64.whl
pip install intel_extension_for_pytorch-2.1.10+xpu-cp39-cp39-win_amd64.whl
# install bigdl-llm for Intel GPU
pip install --pre --upgrade bigdl-llm[xpu]
.. tab:: PyTorch 2.0 Linux
.. code-block:: bash .. code-block:: bash
@ -271,3 +233,62 @@ If you encounter network issues when installing IPEX, you can also install BigDL
pip install --pre --upgrade bigdl-llm[xpu] pip install --pre --upgrade bigdl-llm[xpu]
``` ```
### Runtime Configuration
To use GPU acceleration on Linux, several environment variables are required or recommended before running a GPU example.
```eval_rst
.. tabs::
.. tab:: Intel Arc™ A-Series and Intel Data Center GPU Flex
For Intel Arc™ A-Series Graphics and Intel Data Center GPU Flex Series, we recommend:
.. code-block:: bash
# configures OneAPI environment variables
source /opt/intel/oneapi/setvars.sh
export USE_XETLA=OFF
export SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=1
.. tab:: Intel Data Center GPU Max
For Intel Data Center GPU Max Series, we recommend:
.. code-block:: bash
# configures OneAPI environment variables
source /opt/intel/oneapi/setvars.sh
export LD_PRELOAD=${LD_PRELOAD}:${CONDA_PREFIX}/lib/libtcmalloc.so
export SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=1
export ENABLE_SDP_FUSION=1
Please note that ``libtcmalloc.so`` can be installed by ``conda install -c conda-forge -y gperftools=2.10``
```
### Known issues
#### 1. Ubuntu 22.04 and Linux kernel 6.2.0 may cause performance bad (driver version < stable_775_20_20231219)
For driver version < stable_775_20_20231219, the performance on Linux kernel 6.2.0 is worse than Linux kernel 5.19.0. You can use `sudo apt update && sudo apt install -y intel-i915-dkms intel-fw-gpu` to install the latest driver to solve this issue (need reboot OS).
Tips: You can use `sudo apt list --installed | grep intel-i915-dkms` to check your intel-i915-dkms's version, the version should be latest and >= `1.23.9.11.231003.15+i19-1`.
#### 2. Driver installation meet unmet dependencies: intel-i915-dkms
The last apt install command of the driver installation may get following error:
```
The following packages have unmet dependencies:
intel-i915-dkms : Conflicts: intel-platform-cse-dkms
Conflicts: intel-platform-vsec-dkms
```
You can use `sudo apt install -y intel-i915-dkms intel-fw-gpu` to instead. As the intel-platform-cse-dkms and intel-platform-vsec-dkms are already provided by intel-i915-dkms.
### Troubleshooting
todo