ipex-llm/docs/mddocs/Quickstart/install_pytorch26_gpu.md

Install IPEX-LLM on Intel GPU with PyTorch 2.6

This guide demonstrates how to install IPEX-LLM on Intel GPUs with PyTorch 2.6 support.

IPEX-LLM with PyTorch 2.6 provides a simpler prerequisites setup process, without requiring manual installation of oneAPI. Besides, it offers broader platform support with AOT (Ahead of Time) Compilation.

Tip

For details on which device IPEX-LLM PyTorch 2.6 supports with AOT compilation, you could refer to here (Windows or Linux) for more information.

Table of Contents

• Windows Quickstart
  • Install Prerequisites
  • Install ipex-llm
  • Runtime Configurations
  • Verify Installation
• Linux Quickstart
  • Install Prerequisites
  • Install ipex-llm
  • Runtime Configurations
  • Verify Installation

Windows Quickstart

Install Prerequisites

Update GPU Driver

We recommend updating your GPU driver to the latest. A system reboot is necessary to apply the changes after the installation is complete.

Setup Python Environment

Visit Miniforge installation page, download the Miniforge installer for Windows, and follow the instructions to complete the installation.

After installation, open the Miniforge Prompt, create a new python environment llm-pt26:

conda create -n llm-pt26 python=3.11

Activate the newly created environment llm-pt26:

conda activate llm-pt26

Install ipex-llm

With the llm-pt26 environment active, use pip to install ipex-llm for GPU:

• For Intel Core™ Ultra Processors (Series 2) with processor number 2xxH (code name Arrow Lake):

  Choose either US or CN website for extra-index-url:

  • For US:

    pip install --pre --upgrade ipex-llm[xpu_2.6_arl] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/arl/us/
    

  • For CN:

    pip install --pre --upgrade ipex-llm[xpu_2.6_arl] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/arl/cn/
    

Tip

For other Intel Core™ Ultra Processors, such as 2xxHX, please refer to the installation instruction below (i.e. for other Intel iGPU and dGPU).

• For other Intel iGPU and dGPU:

  pip install --pre --upgrade ipex-llm[xpu_2.6] --extra-index-url https://download.pytorch.org/whl/xpu
  

Runtime Configurations

For optimal performance, it is recommended to set several environment variables. Please check out the suggestions based on your device.

With the llm-pt26 environment active:

• For Intel Arc™ A-Series GPU (code name Alchemist)

  set SYCL_CACHE_PERSISTENT=1
  set UR_L0_USE_IMMEDIATE_COMMANDLISTS=0
  

Tip

It is recommanded to experiment with UR_L0_USE_IMMEDIATE_COMMANDLISTS=0 or 1 for best performance on Intel Arc™ A-Series GPU.

• For other Intel iGPU and dGPU:

  set SYCL_CACHE_PERSISTENT=1
  :: [optional] The following environment variable may improve performance, but in some cases, it may also lead to performance degradation
  set SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=1
  

Note

The environment variable SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS determines the usage of immediate command lists for task submission to the GPU. It is highly recommanded to experiment with SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=1 or 0 on your device for best performance.

You could refer to here regarding more information about Level Zero Immediate Command Lists.

Verify Installation

You can verify if ipex-llm is successfully installed following below steps:

• Open the Miniforge Prompt and activate the Python environment llm-pt26 you previously created:

  conda activate llm-pt26
  

• Set environment variables according to the Runtime Configurations section.

• Launch the Python interactive shell by typing python in the Miniforge Prompt window and then press Enter.

• Copy following code to Miniforge Prompt line by line and press Enter after copying each line.

  import torch
  from ipex_llm.transformers import AutoModel, AutoModelForCausalLM
  tensor_1 = torch.randn(1, 1, 40, 128).to('xpu')
  tensor_2 = torch.randn(1, 1, 128, 40).to('xpu')
  print(torch.matmul(tensor_1, tensor_2).size())
  

  It should output following content at the end:

  torch.Size([1, 1, 40, 40])
  

• To exit the Python interactive shell, simply press Ctrl+Z then press Enter (or input exit() then press Enter).

Linux Quickstart

Install Prerequisites

Install GPU Driver

We recommend following Intel client GPU driver installation guide to install your GPU driver.

Setup Python Environment

Download and install the Miniforge as follows if you don't have conda installed on your machine:

wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh
bash Miniforge3-Linux-x86_64.sh
source ~/.bashrc

You can use conda --version to verify you conda installation.

After installation, create a new python environment llm-pt26:

conda create -n llm-pt26 python=3.11

Activate the newly created environment llm-pt26:

conda activate llm-pt26

Install ipex-llm

With the llm-pt26 environment active, use pip to install ipex-llm for GPU:

pip install --pre --upgrade ipex-llm[xpu_2.6] --extra-index-url https://download.pytorch.org/whl/xpu

Runtime Configurations

For optimal performance, it is recommended to set several environment variables. Please check out the suggestions based on your device.

With the llm-pt26 environment active:

unset OCL_ICD_VENDORS
export SYCL_CACHE_PERSISTENT=1
# [optional] The following environment variable may improve performance, but in some cases, it may also lead to performance degradation
export SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=1

Note

The environment variable SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS determines the usage of immediate command lists for task submission to the GPU. It is highly recommanded to experiment with SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=1 or 0 on your device for best performance.

You could refer to here regarding more information about Level Zero Immediate Command Lists.

Verify Installation

You can verify if ipex-llm is successfully installed following below steps:

• Activate the Python environment llm-pt26 you previously created:

  conda activate llm-pt26
  

• Set environment variables according to the Runtime Configurations section.

• Launch the Python interactive shell by typing python in the terminal and then press Enter.

• Copy following code to Miniforge Prompt line by line and press Enter after copying each line.

  import torch
  from ipex_llm.transformers import AutoModel, AutoModelForCausalLM
  tensor_1 = torch.randn(1, 1, 40, 128).to('xpu')
  tensor_2 = torch.randn(1, 1, 128, 40).to('xpu')
  print(torch.matmul(tensor_1, tensor_2).size())
  

  It should output following content at the end:

  torch.Size([1, 1, 40, 40])
  

• To exit the Python interactive shell, simply press Ctrl+C then press Enter (or input exit() then press Enter).