ipex-llm/python/llm/example/GPU/LangChain/transformer_int4_gpu

Langchain examples

The examples in this folder shows how to use LangChain with bigdl-llm on Intel GPU.

1. Install bigdl-llm

Follow the instructions in GPU Install Guide to install bigdl-llm

2. Install Required Dependencies for langchain examples.

pip install langchain==0.0.184
pip install -U chromadb==0.3.25
pip install -U pandas==2.0.3

3. Configures OneAPI environment variables

3.1 Configurations for Linux

source /opt/intel/oneapi/setvars.sh

3.2 Configurations for Windows

call "C:\Program Files (x86)\Intel\oneAPI\setvars.bat"

Note: Please make sure you are using CMD (Anaconda Prompt if using conda) to run the command as PowerShell is not supported.

4. Runtime Configurations

For optimal performance, it is recommended to set several environment variables. Please check out the suggestions based on your device.

4.1 Configurations for Linux

For Intel Arc™ A-Series Graphics and Intel Data Center GPU Flex Series

export USE_XETLA=OFF
export SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=1

For Intel Data Center GPU Max Series

export LD_PRELOAD=${LD_PRELOAD}:${CONDA_PREFIX}/lib/libtcmalloc.so
export SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=1
export ENABLE_SDP_FUSION=1

Note: Please note that libtcmalloc.so can be installed by conda install -c conda-forge -y gperftools=2.10.

4.2 Configurations for Windows

For Intel iGPU

set SYCL_CACHE_PERSISTENT=1
set BIGDL_LLM_XMX_DISABLED=1

For Intel Arc™ A300-Series or Pro A60

set SYCL_CACHE_PERSISTENT=1

For other Intel dGPU Series

There is no need to set further environment variables.

Note: For the first time that each model runs on Intel iGPU/Intel Arc™ A300-Series or Pro A60, it may take several minutes to compile.

5. Run the examples

5.1. Streaming Chat

python chat.py -m MODEL_PATH -q QUESTION

arguments info:

• -m MODEL_PATH: required, path to the model
• -q QUESTION: question to ask. Default is What is AI?.

5.1. RAG (Retrival Augmented Generation)

python rag.py -m <path_to_model> [-q QUESTION] [-i INPUT_PATH]

arguments info:

• -m MODEL_PATH: required, path to the model.
• -q QUESTION: question to ask. Default is What is BigDL?.
• -i INPUT_PATH: path to the input doc.