diff --git a/docs/mddocs/Quickstart/axolotl_quickstart.md b/docs/mddocs/Quickstart/axolotl_quickstart.md index 4a2cbb3a..f0a2dfa1 100644 --- a/docs/mddocs/Quickstart/axolotl_quickstart.md +++ b/docs/mddocs/Quickstart/axolotl_quickstart.md @@ -4,7 +4,7 @@ See the demo of finetuning LLaMA2-7B on Intel Arc GPU below. - +[![Demo video](https://llm-assets.readthedocs.io/en/latest/_images/axolotl-qlora-linux-arc.png)](https://llm-assets.readthedocs.io/en/latest/_images/axolotl-qlora-linux-arc.mp4) ## Quickstart @@ -12,13 +12,13 @@ See the demo of finetuning LLaMA2-7B on Intel Arc GPU below. IPEX-LLM's support for [Axolotl v0.4.0](https://github.com/OpenAccess-AI-Collective/axolotl/tree/v0.4.0) is only available for Linux system. We recommend Ubuntu 20.04 or later (Ubuntu 22.04 is preferred). -Visit the [Install IPEX-LLM on Linux with Intel GPU](https://ipex-llm.readthedocs.io/en/latest/doc/LLM/Quickstart/install_linux_gpu.html), follow [Install Intel GPU Driver](https://ipex-llm.readthedocs.io/en/latest/doc/LLM/Quickstart/install_linux_gpu.html#install-intel-gpu-driver) and [Install oneAPI](https://ipex-llm.readthedocs.io/en/latest/doc/LLM/Quickstart/install_linux_gpu.html#install-oneapi) to install GPU driver and Intel® oneAPI Base Toolkit 2024.0. +Visit the [Install IPEX-LLM on Linux with Intel GPU](./install_linux_gpu.md), follow [Install Intel GPU Driver](./install_linux_gpu.md#install-gpu-driver) and [Install oneAPI](./install_linux_gpu.md#install-oneapi) to install GPU driver and Intel® oneAPI Base Toolkit 2024.0. ### 1. Install IPEX-LLM for Axolotl Create a new conda env, and install `ipex-llm[xpu]`. -```cmd +```bash conda create -n axolotl python=3.11 conda activate axolotl # install ipex-llm @@ -27,7 +27,7 @@ pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-exte Install [axolotl v0.4.0](https://github.com/OpenAccess-AI-Collective/axolotl/tree/v0.4.0) from git. -```cmd +```bash # install axolotl v0.4.0 git clone https://github.com/OpenAccess-AI-Collective/axolotl/tree/v0.4.0 cd axolotl @@ -62,46 +62,37 @@ For more technical details, please refer to [Llama 2](https://arxiv.org/abs/2307 By default, Axolotl will automatically download models and datasets from Huggingface. Please ensure you have login to Huggingface. -```cmd +```bash huggingface-cli login ``` If you prefer offline models and datasets, please download [Llama-2-7B](https://huggingface.co/meta-llama/Llama-2-7b) and [alpaca_2k_test](https://huggingface.co/datasets/mhenrichsen/alpaca_2k_test). Then, set `HF_HUB_OFFLINE=1` to avoid connecting to Huggingface. -```cmd +```bash export HF_HUB_OFFLINE=1 ``` #### 2.2 Set Environment Variables -```eval_rst -.. note:: - - This is a required step on for APT or offline installed oneAPI. Skip this step for PIP-installed oneAPI. -``` +> [!NOTE] +> This is a required step on for APT or offline installed oneAPI. Skip this step for PIP-installed oneAPI. Configure oneAPI variables by running the following command: -```eval_rst -.. tabs:: - .. tab:: Linux - - .. code-block:: bash - - source /opt/intel/oneapi/setvars.sh - +```bash +source /opt/intel/oneapi/setvars.sh ``` Configure accelerate to avoid training with CPU. You can download a default `default_config.yaml` with `use_cpu: false`. -```cmd +```bash mkdir -p ~/.cache/huggingface/accelerate/ wget -O ~/.cache/huggingface/accelerate/default_config.yaml https://raw.githubusercontent.com/intel-analytics/ipex-llm/main/python/llm/example/GPU/LLM-Finetuning/axolotl/default_config.yaml ``` As an alternative, you can config accelerate based on your requirements. -```cmd +```bash accelerate config ``` @@ -113,7 +104,7 @@ After finishing accelerate config, check if `use_cpu` is disabled (i.e., `use_cp Prepare `lora.yml` for Axolotl LoRA finetune. You can download a template from github. -```cmd +```bash wget https://raw.githubusercontent.com/intel-analytics/ipex-llm/main/python/llm/example/GPU/LLM-Finetuning/axolotl/lora.yml ``` @@ -143,13 +134,13 @@ lora_fan_in_fan_out: Launch LoRA training with the following command. -```cmd +```bash accelerate launch finetune.py lora.yml ``` In Axolotl v0.4.0, you can use `train.py` instead of `-m axolotl.cli.train` or `finetune.py`. -```cmd +```bash accelerate launch train.py lora.yml ``` @@ -157,7 +148,7 @@ accelerate launch train.py lora.yml Prepare `lora.yml` for QLoRA finetune. You can download a template from github. -```cmd +```bash wget https://raw.githubusercontent.com/intel-analytics/ipex-llm/main/python/llm/example/GPU/LLM-Finetuning/axolotl/qlora.yml ``` @@ -188,13 +179,13 @@ lora_fan_in_fan_out: Launch LoRA training with the following command. -```cmd +```bash accelerate launch finetune.py qlora.yml ``` In Axolotl v0.4.0, you can use `train.py` instead of `-m axolotl.cli.train` or `finetune.py`. -```cmd +```bash accelerate launch train.py qlora.yml ``` @@ -206,7 +197,7 @@ Warning: this section will install axolotl main ([796a085](https://github.com/Op Axolotl main has lots of new dependencies. Please setup a new conda env for this version. -```cmd +```bash conda create -n llm python=3.11 conda activate llm # install axolotl main @@ -229,7 +220,7 @@ Based on [axolotl Llama-3 QLoRA example](https://github.com/OpenAccess-AI-Collec Prepare `llama3-qlora.yml` for QLoRA finetune. You can download a template from github. -```cmd +```bash wget https://raw.githubusercontent.com/intel-analytics/ipex-llm/main/python/llm/example/GPU/LLM-Finetuning/axolotl/llama3-qlora.yml ``` @@ -262,19 +253,19 @@ lora_target_linear: true lora_fan_in_fan_out: ``` -```cmd +```bash accelerate launch finetune.py llama3-qlora.yml ``` You can also use `train.py` instead of `-m axolotl.cli.train` or `finetune.py`. -```cmd +```bash accelerate launch train.py llama3-qlora.yml ``` Expected output -```cmd +```bash {'loss': 0.237, 'learning_rate': 1.2254711850265387e-06, 'epoch': 3.77} {'loss': 0.6068, 'learning_rate': 1.1692453482951115e-06, 'epoch': 3.77} {'loss': 0.2926, 'learning_rate': 1.1143322458989303e-06, 'epoch': 3.78} @@ -291,24 +282,24 @@ Expected output ## Troubleshooting -#### TypeError: PosixPath +### TypeError: PosixPath Error message: `TypeError: argument of type 'PosixPath' is not iterable` This issue is related to [axolotl #1544](https://github.com/OpenAccess-AI-Collective/axolotl/issues/1544). It can be fixed by downgrading datasets to 2.15.0. -```cmd +```bash pip install datasets==2.15.0 ``` -#### RuntimeError: out of device memory +### RuntimeError: out of device memory Error message: `RuntimeError: Allocation is out of device memory on current platform.` This issue is caused by running out of GPU memory. Please reduce `lora_r` or `micro_batch_size` in `qlora.yml` or `lora.yml`, or reduce data using in training. -#### OSError: libmkl_intel_lp64.so.2 +### OSError: libmkl_intel_lp64.so.2 Error message: `OSError: libmkl_intel_lp64.so.2: cannot open shared object file: No such file or directory` -oneAPI environment is not correctly set. Please refer to [Set Environment Variables](#set-environment-variables). +oneAPI environment is not correctly set. Please refer to [Set Environment Variables](#22-set-environment-variables). diff --git a/docs/mddocs/Quickstart/benchmark_quickstart.md b/docs/mddocs/Quickstart/benchmark_quickstart.md index ba26b770..a677398e 100644 --- a/docs/mddocs/Quickstart/benchmark_quickstart.md +++ b/docs/mddocs/Quickstart/benchmark_quickstart.md @@ -4,7 +4,7 @@ We can perform benchmarking for IPEX-LLM on Intel CPUs and GPUs using the benchm ## Prepare The Environment -You can refer to [here](https://ipex-llm.readthedocs.io/en/latest/doc/LLM/Overview/install.html) to install IPEX-LLM in your environment. The following dependencies are also needed to run the benchmark scripts. +You can refer to [here](../Overview/install.md) to install IPEX-LLM in your environment. The following dependencies are also needed to run the benchmark scripts. ``` pip install pandas @@ -65,109 +65,98 @@ Some parameters in the yaml file that you can configure: - `task`: There are three tasks: `continuation`, `QA` and `summarize`. `continuation` refers to writing additional content based on prompt. `QA` refers to answering questions based on prompt. `summarize` refers to summarizing the prompt. -```eval_rst -.. note:: - - If you want to benchmark the performance without warmup, you can set ``warm_up: 0`` and ``num_trials: 1`` in ``config.yaml``, and run each single model and in_out_pair separately. -``` +> [!NOTE] +> If you want to benchmark the performance without warmup, you can set ``warm_up: 0`` and ``num_trials: 1`` in ``config.yaml``, and run each single model and in_out_pair separately. ## Run on Windows -Please refer to [here](https://ipex-llm.readthedocs.io/en/latest/doc/LLM/Overview/install_gpu.html#runtime-configuration) to configure oneAPI environment variables. +Please refer to [here](../Overview/install_gpu.md#runtime-configuration) to configure oneAPI environment variables. Choose corresponding commands base on your device. -```eval_rst -.. tabs:: - .. tab:: Intel iGPU +- For **Intel iGPU**: - .. code-block:: bash + ```bash + set SYCL_CACHE_PERSISTENT=1 + set BIGDL_LLM_XMX_DISABLED=1 + + python run.py + ``` - set SYCL_CACHE_PERSISTENT=1 - set BIGDL_LLM_XMX_DISABLED=1 +- For **Intel Arc™ A300-Series or Pro A60**: - python run.py + ```bash + set SYCL_CACHE_PERSISTENT=1 + python run.py + ``` - .. tab:: Intel Arc™ A300-Series or Pro A60 +- For **Other Intel dGPU Series**: - .. code-block:: bash - - set SYCL_CACHE_PERSISTENT=1 - python run.py - - .. tab:: Other Intel dGPU Series - - .. code-block:: bash - - # e.g. Arc™ A770 - python run.py - -``` + ```bash + # e.g. Arc™ A770 + python run.py + ``` ## Run on Linux -```eval_rst -.. tabs:: - .. tab:: Intel Arc™ A-Series and Intel Data Center GPU Flex +Please choose corresponding commands base on your device. - For Intel Arc™ A-Series Graphics and Intel Data Center GPU Flex Series, we recommend: +- For **Intel Arc™ A-Series and Intel Data Center GPU Flex**: - .. code-block:: bash + For Intel Arc™ A-Series Graphics and Intel Data Center GPU Flex Series, we recommend: + + ```bash + ./run-arc.sh + ``` - ./run-arc.sh +- For **Intel iGPU**: - .. tab:: Intel iGPU + For Intel iGPU, we recommend: + + ```bash + ./run-igpu.sh + ``` - For Intel iGPU, we recommend: +- For **Intel Data Center GPU Max**: - .. code-block:: bash + Please note that you need to run ``conda install -c conda-forge -y gperftools=2.10`` before running the benchmark script on Intel Data Center GPU Max Series. + + ```bash + ./run-max-gpu.sh + ``` - ./run-igpu.sh +- For **Intel SPR**: - .. tab:: Intel Data Center GPU Max + For Intel SPR machine, we recommend: + + ```bash + ./run-spr.sh + ``` - Please note that you need to run ``conda install -c conda-forge -y gperftools=2.10`` before running the benchmark script on Intel Data Center GPU Max Series. + The scipt uses a default numactl strategy. If you want to customize it, please use ``lscpu`` or ``numactl -H`` to check how cpu indexs are assigned to numa node, and make sure the run command is binded to only one socket. - .. code-block:: bash +- For **Intel HBM**: - ./run-max-gpu.sh + For Intel HBM machine, we recommend: + + ```bash + ./run-hbm.sh + ``` + + The scipt uses a default numactl strategy. If you want to customize it, please use ``numactl -H`` to check how the index of hbm node and cpu are assigned. - .. tab:: Intel SPR - - For Intel SPR machine, we recommend: - - .. code-block:: bash - - ./run-spr.sh - - The scipt uses a default numactl strategy. If you want to customize it, please use ``lscpu`` or ``numactl -H`` to check how cpu indexs are assigned to numa node, and make sure the run command is binded to only one socket. - - .. tab:: Intel HBM - - For Intel HBM machine, we recommend: - - .. code-block:: bash - - ./run-hbm.sh - - The scipt uses a default numactl strategy. If you want to customize it, please use ``numactl -H`` to check how the index of hbm node and cpu are assigned. - - For example: - - - .. code-block:: bash - - node 0 1 2 3 - 0: 10 21 13 23 - 1: 21 10 23 13 - 2: 13 23 10 23 - 3: 23 13 23 10 - - - here hbm node is the node whose distance from the checked node is 13, node 2 is node 0's hbm node. - - And make sure the run command is binded to only one socket. - -``` + For example: + + ```bash + node 0 1 2 3 + 0: 10 21 13 23 + 1: 21 10 23 13 + 2: 13 23 10 23 + 3: 23 13 23 10 + ``` + + here hbm node is the node whose distance from the checked node is 13, node 2 is node 0's hbm node. + + And make sure the run command is binded to only one socket. ## Result diff --git a/docs/mddocs/Quickstart/bigdl_llm_migration.md b/docs/mddocs/Quickstart/bigdl_llm_migration.md index a1ef5051..0b7643e1 100644 --- a/docs/mddocs/Quickstart/bigdl_llm_migration.md +++ b/docs/mddocs/Quickstart/bigdl_llm_migration.md @@ -4,10 +4,9 @@ This guide helps you migrate your `bigdl-llm` application to use `ipex-llm`. ## Upgrade `bigdl-llm` package to `ipex-llm` -```eval_rst -.. note:: - This step assumes you have already installed `bigdl-llm`. -``` +> [!NOTE] +> This step assumes you have already installed `bigdl-llm`. + You need to uninstall `bigdl-llm` and install `ipex-llm`With your `bigdl-llm` conda environment activated, execute the following command according to your device type and location: ### For CPU @@ -19,23 +18,20 @@ pip install --pre --upgrade ipex-llm[all] # for cpu ### For GPU Choose either US or CN website for `extra-index-url`: -```eval_rst -.. tabs:: - .. tab:: US +- For **US**: - .. code-block:: cmd + ```bash + pip uninstall -y bigdl-llm + pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/ + ``` - pip uninstall -y bigdl-llm - pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/ +- For **CN**: - .. tab:: CN - - .. code-block:: cmd - - pip uninstall -y bigdl-llm - pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/cn/ -``` + ```bash + pip uninstall -y bigdl-llm + pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/cn/ + ``` ## Migrate `bigdl-llm` code to `ipex-llm` There are two options to migrate `bigdl-llm` code to `ipex-llm`. diff --git a/docs/mddocs/Quickstart/chatchat_quickstart.md b/docs/mddocs/Quickstart/chatchat_quickstart.md index e482751a..48c09de8 100644 --- a/docs/mddocs/Quickstart/chatchat_quickstart.md +++ b/docs/mddocs/Quickstart/chatchat_quickstart.md @@ -10,12 +10,13 @@ 简体中文 - - + + ->You can change the UI language in the left-side menu. We currently support **English** and **简体中文** (see video demos below). +> [!NOTE] +> You can change the UI language in the left-side menu. We currently support **English** and **简体中文** (see video demos below). ## Langchain-Chatchat Architecture @@ -66,7 +67,7 @@ You can now click `Dialogue` on the left-side menu to return to the chat UI. The
-For more information about how to use Langchain-Chatchat, refer to Official Quickstart guide in [English](./README_en.md#), [Chinese](./README_chs.md#), or the [Wiki](https://github.com/chatchat-space/Langchain-Chatchat/wiki/). +For more information about how to use Langchain-Chatchat, refer to Official Quickstart guide in [English](https://github.com/chatchat-space/Langchain-Chatchat/blob/master/README_en.md#), [Chinese](https://github.com/chatchat-space/Langchain-Chatchat/blob/master/README.md#), or the [Wiki](https://github.com/chatchat-space/Langchain-Chatchat/wiki/). ### Trouble Shooting & Tips diff --git a/docs/mddocs/Quickstart/continue_quickstart.md b/docs/mddocs/Quickstart/continue_quickstart.md index 68623118..cd1f42cd 100644 --- a/docs/mddocs/Quickstart/continue_quickstart.md +++ b/docs/mddocs/Quickstart/continue_quickstart.md @@ -5,30 +5,25 @@ Below is a demo of using `Continue` with [CodeQWen1.5-7B](https://huggingface.co/Qwen/CodeQwen1.5-7B-Chat) running on Intel A770 GPU. This demo illustrates how a programmer used `Continue` to find a solution for the [Kaggle's _Titanic_ challenge](https://www.kaggle.com/competitions/titanic/), which involves asking `Continue` to complete the code for model fitting, evaluation, hyper parameter tuning, feature engineering, and explain generated code. - +[![Demo video](https://llm-assets.readthedocs.io/en/latest/_images/continue_demo_ollama_backend_arc.png)](https://llm-assets.readthedocs.io/en/latest/_images/continue_demo_ollama_backend_arc.mp4) ## Quickstart -This guide walks you through setting up and running **Continue** within _Visual Studio Code_, empowered by local large language models served via [Ollama](./ollama_quickstart.html) with `ipex-llm` optimizations. +This guide walks you through setting up and running **Continue** within _Visual Studio Code_, empowered by local large language models served via [Ollama](./ollama_quickstart.md) with `ipex-llm` optimizations. ### 1. Install and Run Ollama Serve -Visit [Run Ollama with IPEX-LLM on Intel GPU](./ollama_quickstart.html), and follow the steps 1) [Install IPEX-LLM for Ollama](./ollama_quickstart.html#install-ipex-llm-for-ollama), 2) [Initialize Ollama](./ollama_quickstart.html#initialize-ollama) 3) [Run Ollama Serve](./ollama_quickstart.html#run-ollama-serve) to install, init and start the Ollama Service. +Visit [Run Ollama with IPEX-LLM on Intel GPU](./ollama_quickstart.md), and follow the steps 1) [Install IPEX-LLM for Ollama](./ollama_quickstart.md#1-install-ipex-llm-for-ollama), 2) [Initialize Ollama](./ollama_quickstart.md#2-initialize-ollama) 3) [Run Ollama Serve](./ollama_quickstart.md#3-run-ollama-serve) to install, init and start the Ollama Service. +> [!IMPORTANT] +> If the `Continue` plugin is not installed on the same machine where Ollama is running (which means `Continue` needs to connect to a remote Ollama service), you must configure the Ollama service to accept connections from any IP address. To achieve this, set or export the environment variable `OLLAMA_HOST=0.0.0.0` before executing the command `ollama serve`. -```eval_rst -.. important:: - - If the `Continue` plugin is not installed on the same machine where Ollama is running (which means `Continue` needs to connect to a remote Ollama service), you must configure the Ollama service to accept connections from any IP address. To achieve this, set or export the environment variable `OLLAMA_HOST=0.0.0.0` before executing the command `ollama serve`. - -.. tip:: - - If your local LLM is running on Intel Arc™ A-Series Graphics with Linux OS (Kernel 6.2), it is recommended to additionaly set the following environment variable for optimal performance before executing `ollama serve`: - - .. code-block:: bash - - export SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=1 -``` +> [!TIP] +> If your local LLM is running on Intel Arc™ A-Series Graphics with Linux OS (Kernel 6.2), it is recommended to additionaly set the following environment variable for optimal performance before executing `ollama serve`: +> +> ```bash +> export SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=1 +> ``` ### 2. Pull and Prepare the Model @@ -36,29 +31,24 @@ Visit [Run Ollama with IPEX-LLM on Intel GPU](./ollama_quickstart.html), and fol Now we need to pull a model for coding. Here we use [CodeQWen1.5-7B](https://huggingface.co/Qwen/CodeQwen1.5-7B-Chat) model as an example. Open a new terminal window, run the following command to pull [`codeqwen:latest`](https://ollama.com/library/codeqwen). +- For **Linux users**: -```eval_rst -.. tabs:: - .. tab:: Linux + ```bash + export no_proxy=localhost,127.0.0.1 + ./ollama pull codeqwen:latest + ``` - .. code-block:: bash +- For **Windows users**: - export no_proxy=localhost,127.0.0.1 - ./ollama pull codeqwen:latest + Please run the following command in Miniforge Prompt. - .. tab:: Windows + ```cmd + set no_proxy=localhost,127.0.0.1 + ollama pull codeqwen:latest + ``` - Please run the following command in Miniforge Prompt. - - .. code-block:: cmd - - set no_proxy=localhost,127.0.0.1 - ollama pull codeqwen:latest - -.. seealso:: - - Besides CodeQWen, there are other coding models you might want to explore, such as Magicoder, Wizardcoder, Codellama, Codegemma, Starcoder, Starcoder2, and etc. You can find these models in the `Ollama model library `_. Simply search for the model, pull it in a similar manner, and give it a try. -``` +> [!NOTE] +> Besides CodeQWen, there are other coding models you might want to explore, such as Magicoder, Wizardcoder, Codellama, Codegemma, Starcoder, Starcoder2, and etc. You can find these models in the [`Ollama model library`](https://ollama.com/library). Simply search for the model, pull it in a similar manner, and give it a try. #### 2.2 Prepare the Model and Pre-load @@ -72,8 +62,8 @@ Start by creating a file named `Modelfile` with the following content: FROM codeqwen:latest PARAMETER num_ctx 4096 ``` -Next, use the following commands in the terminal (Linux) or Miniforge Prompt (Windows) to create a new model in Ollama named `codeqwen:latest-continue`: +Next, use the following commands in the terminal (Linux) or Miniforge Prompt (Windows) to create a new model in Ollama named `codeqwen:latest-continue`: ```bash ollama create codeqwen:latest-continue -f Modelfile @@ -87,8 +77,6 @@ Finally, preload the new model by executing the following command in a new termi ollama run codeqwen:latest-continue ``` - - ### 3. Install `Continue` Extension Search for `Continue` in the VSCode `Extensions Marketplace` and install it just like any other extension. diff --git a/docs/mddocs/Quickstart/deepspeed_autotp_fastapi_quickstart.md b/docs/mddocs/Quickstart/deepspeed_autotp_fastapi_quickstart.md index f99c6731..17e51dca 100644 --- a/docs/mddocs/Quickstart/deepspeed_autotp_fastapi_quickstart.md +++ b/docs/mddocs/Quickstart/deepspeed_autotp_fastapi_quickstart.md @@ -1,10 +1,10 @@ # Run IPEX-LLM serving on Multiple Intel GPUs using DeepSpeed AutoTP and FastApi -This example demonstrates how to run IPEX-LLM serving on multiple [Intel GPUs](../README.md) by leveraging DeepSpeed AutoTP. +This example demonstrates how to run IPEX-LLM serving on multiple [Intel GPUs](../../../python/llm/example/GPU/README.md) by leveraging DeepSpeed AutoTP. ## Requirements -To run this example with IPEX-LLM on Intel GPUs, we have some recommended requirements for your machine, please refer to [here](../README.md#recommended-requirements) for more information. For this particular example, you will need at least two GPUs on your machine. +To run this example with IPEX-LLM on Intel GPUs, we have some recommended requirements for your machine, please refer to [here](../../../python/llm/example/GPU/README.md#requirements) for more information. For this particular example, you will need at least two GPUs on your machine. ## Example @@ -24,7 +24,8 @@ pip install mpi4py fastapi uvicorn conda install -c conda-forge -y gperftools=2.10 # to enable tcmalloc ``` -> **Important**: IPEX 2.1.10+xpu requires Intel® oneAPI Base Toolkit's version == 2024.0. Please make sure you have installed the correct version. +> [!IMPORTANT] +> IPEX 2.1.10+xpu requires Intel® oneAPI Base Toolkit's version == 2024.0. Please make sure you have installed the correct version. ### 2. Run tensor parallel inference on multiple GPUs @@ -35,7 +36,6 @@ We provide example usage for `Llama-2-7b-chat-hf` model running on Arc A770 Run Llama-2-7b-chat-hf on two Intel Arc A770: ```bash - # Before run this script, you should adjust the YOUR_REPO_ID_OR_MODEL_PATH in last line # If you want to change server port, you can set port parameter in last line @@ -52,7 +52,8 @@ If you successfully run the serving, you can get output like this: [0] INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit) ``` -> **Note**: You could change `NUM_GPUS` to the number of GPUs you have on your machine. And you could also specify other low bit optimizations through `--low-bit`. +> [!NOTE] +> You could change `NUM_GPUS` to the number of GPUs you have on your machine. And you could also specify other low bit optimizations through `--low-bit`. ### 3. Sample Input and Output @@ -83,7 +84,8 @@ And you should get output like this: ``` -**Important**: The first token latency is much larger than rest token latency, you could use [our benchmark tool](https://github.com/intel-analytics/ipex-llm/blob/main/python/llm/dev/benchmark/README.md) to obtain more details about first and rest token latency. +> [!IMPORTANT] +> The first token latency is much larger than rest token latency, you could use [our benchmark tool](../../../python/llm/dev/benchmark/README.md) to obtain more details about first and rest token latency. ### 4. Benchmark with wrk diff --git a/docs/mddocs/Quickstart/dify_quickstart.md b/docs/mddocs/Quickstart/dify_quickstart.md index 97e4ae2d..7e0e9c03 100644 --- a/docs/mddocs/Quickstart/dify_quickstart.md +++ b/docs/mddocs/Quickstart/dify_quickstart.md @@ -6,7 +6,7 @@ *See the demo of a RAG workflow in Dify running LLaMA2-7B on Intel A770 GPU below.* - +[![Demo video](https://llm-assets.readthedocs.io/en/latest/_images/dify-rag-small.png)](https://llm-assets.readthedocs.io/en/latest/_images/dify-rag-small.mp4) ## Quickstart @@ -99,13 +99,8 @@ NEXT_PUBLIC_PUBLIC_API_PREFIX=http://localhost:5001/api NEXT_PUBLIC_SENTRY_DSN= ``` -```eval_rst - -.. note:: - - If you encounter connection problems, you may run `export no_proxy=localhost,127.0.0.1` before starting API servcie, Worker service and frontend. - -``` +> [!NOTE] +> If you encounter connection problems, you may run `export no_proxy=localhost,127.0.0.1` before starting API servcie, Worker service and frontend. ### 3. How to Use `Dify` diff --git a/docs/mddocs/Quickstart/fastchat_quickstart.md b/docs/mddocs/Quickstart/fastchat_quickstart.md index b154026d..e89c64f6 100644 --- a/docs/mddocs/Quickstart/fastchat_quickstart.md +++ b/docs/mddocs/Quickstart/fastchat_quickstart.md @@ -20,7 +20,6 @@ To add GPU support for FastChat, you may install **`ipex-llm`** as follows: ```bash pip install --pre --upgrade ipex-llm[xpu,serving] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/ - ``` ## 2. Start the service @@ -61,7 +60,7 @@ export SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=1 python3 -m ipex_llm.serving.fastchat.ipex_llm_worker --model-path REPO_ID_OR_YOUR_MODEL_PATH --low-bit "sym_int4" --trust-remote-code --device "xpu" ``` -We have also provided an option `--load-low-bit-model` to load models that have been converted and saved into disk using the `save_low_bit` interface as introduced in this [document](https://ipex-llm.readthedocs.io/en/latest/doc/LLM/Overview/KeyFeatures/hugging_face_format.html#save-load). +We have also provided an option `--load-low-bit-model` to load models that have been converted and saved into disk using the `save_low_bit` interface as introduced in this [document](../Overview/KeyFeatures/hugging_face_format.md#save--load). Check the following examples: @@ -72,7 +71,7 @@ python -m ipex_llm.serving.fastchat.ipex_llm_worker --model-path /Low/Bit/Model/ #### For self-speculative decoding example: -You can use IPEX-LLM to run `self-speculative decoding` example. Refer to [here](https://github.com/intel-analytics/ipex-llm/tree/c9fac8c26bf1e1e8f7376fa9a62b32951dd9e85d/python/llm/example/GPU/Speculative-Decoding) for more details on intel MAX GPUs. Refer to [here](https://github.com/intel-analytics/ipex-llm/tree/c9fac8c26bf1e1e8f7376fa9a62b32951dd9e85d/python/llm/example/GPU/Speculative-Decoding) for more details on intel CPUs. +You can use IPEX-LLM to run `self-speculative decoding` example. Refer to [here](../../../python/llm/example/GPU/Speculative-Decoding) for more details on intel MAX GPUs. Refer to [here](../../../python/llm/example/CPU/Speculative-Decoding) for more details on intel CPUs. ```bash # Available low_bit format only including bf16 on CPU. @@ -102,9 +101,9 @@ For a full list of accepted arguments, you can refer to the main method of the ` #### IPEX-LLM vLLM worker -We also provide the `vllm_worker` which uses the [vLLM](https://github.com/intel-analytics/ipex-llm/tree/main/python/llm/example/CPU/vLLM-Serving) engine for better hardware utilization. +We also provide the `vllm_worker` which uses the vLLM engine (on [CPU](../../../python/llm/example/CPU/vLLM-Serving) / [GPU](../../../python/llm/example/GPU/vLLM-Serving)) for better hardware utilization. -To run using the `vLLM_worker`, we don't need to change model name, just simply uses the following command: +To run using the `vLLM_worker`, we don't need to change model name, just simply uses the following command: ```bash # On CPU