diff --git a/docs/readthedocs/source/doc/LLM/Overview/install_gpu.md b/docs/readthedocs/source/doc/LLM/Overview/install_gpu.md index 5bd125ff..807d2904 100644 --- a/docs/readthedocs/source/doc/LLM/Overview/install_gpu.md +++ b/docs/readthedocs/source/doc/LLM/Overview/install_gpu.md @@ -1,22 +1,118 @@ # BigDL-LLM Installation: GPU +## Windows -## PyTorch 2.1 -### Prerequisite +### Prerequisites + +BigDL-LLM on Windows supports Intel iGPU and dGPU. + +```eval_rst +.. important:: + + BigDL-LLM on Windows only supports PyTorch 2.1. +``` + +To apply Intel GPU acceleration, there're several prerequisite steps for tools installation and environment preparation: + +* Step 1: Install [Visual Studio 2022](https://visualstudio.microsoft.com/downloads/) Community Edition and select "Desktop development with C++" workload, like [this](https://learn.microsoft.com/en-us/cpp/build/vscpp-step-0-installation?view=msvc-170#step-4---choose-workloads) + +* Step 2: Install or update to latest [GPU driver](https://www.intel.com/content/www/us/en/download/785597/intel-arc-iris-xe-graphics-windows.html) + +* Step 3: Install [Intel® oneAPI Base Toolkit](https://www.intel.com/content/www/us/en/developer/tools/oneapi/base-toolkit-download.html) 2024.0 + +### Install BigDL-LLM From PyPI + +We recommend using [miniconda](https://docs.conda.io/en/latest/miniconda.html) to create a python 3.9 enviroment: + +```eval_rst +.. important:: + + ``bigdl-llm`` is tested with Python 3.9, 3.10 and 3.11. Python 3.9 is recommended for best practices. +``` + +The easiest ways to install `bigdl-llm` is the following commands: + +``` +conda create -n llm python=3.9 libuv +conda activate llm + +pip install --pre --upgrade bigdl-llm[xpu] -f https://developer.intel.com/ipex-whl-stable-xpu +``` + +### Install BigDL-LLM From Wheel + +If you encounter network issues when installing IPEX, you can also install BigDL-LLM dependencies for Intel XPU from source achieves. First you need to download and install torch/torchvision/ipex from wheels listed below before installing `bigdl-llm`. + +Download the wheels on Windows system: + +``` +wget https://intel-extension-for-pytorch.s3.amazonaws.com/ipex_stable/xpu/torch-2.1.0a0%2Bcxx11.abi-cp39-cp39-win_amd64.whl +wget https://intel-extension-for-pytorch.s3.amazonaws.com/ipex_stable/xpu/torchvision-0.16.0a0%2Bcxx11.abi-cp39-cp39-win_amd64.whl +wget https://intel-extension-for-pytorch.s3.amazonaws.com/ipex_stable/xpu/intel_extension_for_pytorch-2.1.10%2Bxpu-cp39-cp39-win_amd64.whl +``` + +You may install dependencies directly from the wheel archives and then install `bigdl-llm` using following commands: + +``` +pip install torch-2.1.0a0+cxx11.abi-cp39-cp39-win_amd64.whl +pip install torchvision-0.16.0a0+cxx11.abi-cp39-cp39-win_amd64.whl +pip install intel_extension_for_pytorch-2.1.10+xpu-cp39-cp39-win_amd64.whl + +pip install --pre --upgrade bigdl-llm[xpu] +``` + +### Runtime Configuration + +To use GPU acceleration on Windows, several environment variables are required before running a GPU example. + +Make sure you are using CMD as PowerShell is not supported: + +``` +call "C:\Program Files (x86)\Intel\oneAPI\setvars.bat" +``` + +Please also set the following environment variable for iGPU: + +``` +set SYCL_CACHE_PERSISTENT=1 +set BIGDL_LLM_XMX_DISABLED=1 +``` + +```eval_rst +.. note:: + + For the first time that **each model** runs on **iGPU**, it may take around several minutes to compile. +``` + +### Troubleshooting + +todo + +## Linux + +### Prerequisites + +BigDL-LLM for GPU supports on Linux has been verified on: + +* Intel Arc™ A-Series Graphics +* Intel Data Center GPU Flex Series +* Intel Data Center GPU Max Series + +```eval_rst +.. important:: + + BigDL-LLM on Linux only supports PyTorch 2.0 and PyTorch 2.1. +``` + +```eval_rst +.. important:: + + We currently support the Ubuntu 20.04 operating system and later. +``` ```eval_rst .. tabs:: - .. tab:: Linux - - BigDL-LLM for GPU supports on Linux with PyTorch 2.1 has been verified on: - - * Intel Arc™ A-Series Graphics - * Intel Data Center GPU Flex Series - * Intel Data Center GPU Max Series - - .. note:: - - We currently support the Ubuntu 20.04 operating system or later. + .. tab:: PyTorch 2.1 To enable BigDL-LLM for Intel GPUs with PyTorch 2.1, here're several prerequisite steps for tools installation and environment preparation: @@ -29,38 +125,46 @@ See `release page `_ for latest version. - * Step 2: Download and install `Intel® oneAPI Base Toolkit `_ with version 2024.0. OneMKL and DPC++ compiler are needed, others are optional. + * Step 2: Download and install `Intel® oneAPI Base Toolkit `_ with version 2024.0. OneDNN, OneMKL and DPC++ compiler are needed, others are optional. .. seealso:: We recommend you to use `this offline package `_ to install oneapi. - .. tab:: Windows + .. tab:: Pytorch 2.0 - BigDL-LLM on Windows supports Intel iGPU and dGPU. + To enable BigDL-LLM for Intel GPUs with PyTorch 2.0, here're several prerequisite steps for tools installation and environment preparation: - To apply Intel GPU acceleration, there're several prerequisite steps for tools installation and environment preparation: - * Step 1: Install `Visual Studio 2022 `_ Community Edition and select "Desktop development with C++" workload + * Step 1: Install Intel GPU Driver version >= stable_775_20_20231219. Highly recommend installing the latest version of intel-i915-dkms using apt. - * Step 2: Install or update to latest GPU driver + .. seealso:: + + Please refer to our `driver installation `_ for general purpose GPU capabilities. + + See `release page `_ for latest version. + + * Step 2: Download and install `Intel® oneAPI Base Toolkit `_ with version 2023.2. OneDNN, OneMKL and DPC++ compiler are needed, others are optional. + + .. seealso:: + + We recommend you to use `this offline package `_ to install oneapi. - * Step 3: Install `Intel® oneAPI Base Toolkit `_ 2024.0 ``` -### Install BigDL-LLM +### Install BigDL-LLM From PyPI -We recommend using [Conda](https://docs.conda.io/en/latest/miniconda.html) to create a python 3.9 enviroment: +We recommend using [miniconda](https://docs.conda.io/en/latest/miniconda.html) to create a python 3.9 enviroment: ```eval_rst .. important:: - ``bigdl-llm`` is tested with Python 3.9, which is recommended for best practices. + ``bigdl-llm`` is tested with Python 3.9, 3.10 and 3.11. Python 3.9 is recommended for best practices. ``` ```eval_rst .. tabs:: - .. tab:: Linux + .. tab:: Pytorch 2.1 .. code-block:: bash @@ -69,145 +173,24 @@ We recommend using [Conda](https://docs.conda.io/en/latest/miniconda.html) to cr pip install --pre --upgrade bigdl-llm[xpu_2.1] -f https://developer.intel.com/ipex-whl-stable-xpu - .. tab:: Windows + .. tab:: Pytorch 2.0 - .. code-block:: cmd + .. code-block:: bash - conda create -n llm python=3.9 libuv + conda create -n llm python=3.9 conda activate llm pip install --pre --upgrade bigdl-llm[xpu] -f https://developer.intel.com/ipex-whl-stable-xpu ``` -## PyTorch 2.0 -```eval_rst -.. note:: +### Install BigDL-LLM From Wheel - BigDL-LLM for GPU with PyTorch 2.0 supports Ubuntu 20.04 operating system or later. -``` - -### Prerequisite - -BigDL-LLM for GPU supports on Linux with PyTorch 2.0 has been verified on: - -* Intel Arc™ A-Series Graphics -* Intel Data Center GPU Flex Series -* Intel Data Center GPU Max Series - -To enable BigDL-LLM for Intel GPUs with PyTorch 2.0, here're several prerequisite steps for tools installation and environment preparation: - -- Step 1: Install Intel GPU Driver version >= stable_775_20_20231219. Highly recommend installing the latest version of intel-i915-dkms using apt. - ```eval_rst - .. seealso:: - - Please refer to our `driver installation `_ for general purpose GPU capabilities. - - See `release page `_ for latest version. - ``` -- Step 2: Download and install [Intel® oneAPI Base Toolkit](https://www.intel.com/content/www/us/en/developer/tools/oneapi/base-toolkit-download.html) with version 2023.2.0. OneMKL and DPC++ compiler are needed, others are optional. - ```eval_rst - .. seealso:: - - We recommend you to use `this offline package `_ to install oneapi. - ``` - -### Install BigDL-LLM - -We recommend using [Conda](https://docs.conda.io/en/latest/miniconda.html) to create a python 3.9 enviroment: - -```eval_rst -.. important:: - - ``bigdl-llm`` is tested with Python 3.9, which is recommended for best practices. -``` - -```bash -conda create -n llm python=3.9 -conda activate llm - -pip install --pre --upgrade bigdl-llm[xpu] -f https://developer.intel.com/ipex-whl-stable-xpu -``` - -## Known issues -### 1. For Linux users, Ubuntu 22.04 and Linux kernel 6.2.0 may cause performance bad (driver version < stable_775_20_20231219). - -For driver version < stable_775_20_20231219, the performance on Linux kernel 6.2.0 is worse than Linux kernel 5.19.0. You can use `sudo apt update && sudo apt install -y intel-i915-dkms intel-fw-gpu` to install the latest driver to solve this issue (need reboot OS). - -Tips: You can use `sudo apt list --installed | grep intel-i915-dkms` to check your intel-i915-dkms's version, the version should be latest and >= `1.23.9.11.231003.15+i19-1`. - -### 2. For Linux users, driver installation meet unmet dependencies: intel-i915-dkms -The last apt install command of the driver installation may get following error: -``` -The following packages have unmet dependencies: - intel-i915-dkms : Conflicts: intel-platform-cse-dkms - Conflicts: intel-platform-vsec-dkms -``` -You can use `sudo apt install -y intel-i915-dkms intel-fw-gpu` to instead. As the intel-platform-cse-dkms and intel-platform-vsec-dkms are already provided by intel-i915-dkms. - -### 3. Best known configurations - -For running a LLM model with BigDL-LLM optimizations, several environment variables are recommended: +If you encounter network issues when installing IPEX, you can also install BigDL-LLM dependencies for Intel XPU from source achieves. First you need to download and install torch/torchvision/ipex from wheels listed below before installing `bigdl-llm`. ```eval_rst .. tabs:: - .. tab:: Linux - - For Intel Arc™ A-Series Graphics and Intel Data Center GPU Flex Series, we recommend: - - .. code-block:: bash - - # configures OneAPI environment variables - source /opt/intel/oneapi/setvars.sh - - export USE_XETLA=OFF - export SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=1 - - For Intel Data Center GPU Max Series, we recommend: - - .. code-block:: bash - - # configures OneAPI environment variables - source /opt/intel/oneapi/setvars.sh - - export LD_PRELOAD=${LD_PRELOAD}:${CONDA_PREFIX}/lib/libtcmalloc.so - export SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=1 - export ENABLE_SDP_FUSION=1 - - Please note that ``libtcmalloc.so`` can installed by ``conda install -c conda-forge -y gperftools=2.10`` - - - - .. tab:: Windows - - Make sure you are using CMD as PowerShell is not supported: - - .. code-block:: cmd - - # configures OneAPI environment variables - call "C:\Program Files (x86)\Intel\oneAPI\setvars.bat" - - set SYCL_CACHE_PERSISTENT=1 - - Please also set the following environment variable for iGPU: - - .. code-block:: cmd - - set BIGDL_LLM_XMX_DISABLED=1 - - .. note:: - - For the first time that **each** model runs on **a new machine**, it may take around several minutes to compile. - -``` - -### 4. How to install from wheel - -If you encounter network issues when installing IPEX, you can also install BigDL-LLM dependencies for Intel XPU from source achieves. First you need to install the target torch/torchvision/ipex versions from downloaded wheels [here](https://developer.intel.com/ipex-whl-stable-xpu) before installing `bigdl-llm`. - -```eval_rst -.. tabs:: - .. tab:: PyTorch 2.1 Linux + .. tab:: PyTorch 2.1 .. code-block:: bash @@ -228,28 +211,7 @@ If you encounter network issues when installing IPEX, you can also install BigDL # install bigdl-llm for Intel GPU pip install --pre --upgrade bigdl-llm[xpu_2.1] - .. tab:: PyTorch 2.1 Windows - - .. code-block:: bash - - # get the wheels on Windows system - wget https://intel-extension-for-pytorch.s3.amazonaws.com/ipex_stable/xpu/torch-2.1.0a0%2Bcxx11.abi-cp39-cp39-win_amd64.whl - wget https://intel-extension-for-pytorch.s3.amazonaws.com/ipex_stable/xpu/torchvision-0.16.0a0%2Bcxx11.abi-cp39-cp39-win_amd64.whl - wget https://intel-extension-for-pytorch.s3.amazonaws.com/ipex_stable/xpu/intel_extension_for_pytorch-2.1.10%2Bxpu-cp39-cp39-win_amd64.whl - - Then you may install directly from the wheel archives using following commands: - - .. code-block:: cmd - - # install the packages from the wheels - pip install torch-2.1.0a0+cxx11.abi-cp39-cp39-win_amd64.whl - pip install torchvision-0.16.0a0+cxx11.abi-cp39-cp39-win_amd64.whl - pip install intel_extension_for_pytorch-2.1.10+xpu-cp39-cp39-win_amd64.whl - - # install bigdl-llm for Intel GPU - pip install --pre --upgrade bigdl-llm[xpu] - - .. tab:: PyTorch 2.0 Linux + .. tab:: PyTorch 2.0 .. code-block:: bash @@ -271,3 +233,62 @@ If you encounter network issues when installing IPEX, you can also install BigDL pip install --pre --upgrade bigdl-llm[xpu] ``` + +### Runtime Configuration + +To use GPU acceleration on Linux, several environment variables are required or recommended before running a GPU example. + +```eval_rst +.. tabs:: + .. tab:: Intel Arc™ A-Series and Intel Data Center GPU Flex + + For Intel Arc™ A-Series Graphics and Intel Data Center GPU Flex Series, we recommend: + + .. code-block:: bash + + # configures OneAPI environment variables + source /opt/intel/oneapi/setvars.sh + + export USE_XETLA=OFF + export SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=1 + + .. tab:: Intel Data Center GPU Max + + For Intel Data Center GPU Max Series, we recommend: + + .. code-block:: bash + + # configures OneAPI environment variables + source /opt/intel/oneapi/setvars.sh + + export LD_PRELOAD=${LD_PRELOAD}:${CONDA_PREFIX}/lib/libtcmalloc.so + export SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=1 + export ENABLE_SDP_FUSION=1 + + Please note that ``libtcmalloc.so`` can be installed by ``conda install -c conda-forge -y gperftools=2.10`` + +``` + +### Known issues + +#### 1. Ubuntu 22.04 and Linux kernel 6.2.0 may cause performance bad (driver version < stable_775_20_20231219) + +For driver version < stable_775_20_20231219, the performance on Linux kernel 6.2.0 is worse than Linux kernel 5.19.0. You can use `sudo apt update && sudo apt install -y intel-i915-dkms intel-fw-gpu` to install the latest driver to solve this issue (need reboot OS). + +Tips: You can use `sudo apt list --installed | grep intel-i915-dkms` to check your intel-i915-dkms's version, the version should be latest and >= `1.23.9.11.231003.15+i19-1`. + +#### 2. Driver installation meet unmet dependencies: intel-i915-dkms + +The last apt install command of the driver installation may get following error: + +``` +The following packages have unmet dependencies: + intel-i915-dkms : Conflicts: intel-platform-cse-dkms + Conflicts: intel-platform-vsec-dkms +``` + +You can use `sudo apt install -y intel-i915-dkms intel-fw-gpu` to instead. As the intel-platform-cse-dkms and intel-platform-vsec-dkms are already provided by intel-i915-dkms. + +### Troubleshooting + +todo