diff --git a/docs/readthedocs/source/_templates/sidebar_quicklinks.html b/docs/readthedocs/source/_templates/sidebar_quicklinks.html index e91b5342..1d620add 100644 --- a/docs/readthedocs/source/_templates/sidebar_quicklinks.html +++ b/docs/readthedocs/source/_templates/sidebar_quicklinks.html @@ -69,7 +69,7 @@ using DeepSpeed AutoTP and FastApi
  • - Run RAGFlow using Ollama with IPEX_LLM + Run RAGFlow with IPEX_LLM on Intel GPU
    • diff --git a/docs/readthedocs/source/doc/LLM/Quickstart/index.rst b/docs/readthedocs/source/doc/LLM/Quickstart/index.rst index a5d9ed1f..8f7b7f2f 100644 --- a/docs/readthedocs/source/doc/LLM/Quickstart/index.rst +++ b/docs/readthedocs/source/doc/LLM/Quickstart/index.rst @@ -8,12 +8,22 @@ IPEX-LLM Quickstart This section includes efficient guide to show you how to: +================= +Install +================= + + * |bigdl_llm_migration_guide|_ * `Install IPEX-LLM on Linux with Intel GPU <./install_linux_gpu.html>`_ * `Install IPEX-LLM on Windows with Intel GPU <./install_windows_gpu.html>`_ * `Install IPEX-LLM in Docker on Windows with Intel GPU <./docker_windows_gpu.html>`_ -* `Run PyTorch Inference on Intel GPU using Docker (on Linux or WSL) <./docker_benchmark_quickstart.html>`_ + +================= +Inference +================= + * `Run Performance Benchmarking with IPEX-LLM <./benchmark_quickstart.html>`_ +* `Run PyTorch Inference on Intel GPU using Docker (on Linux or WSL) <./docker_benchmark_quickstart.html>`_ * `Run Local RAG using Langchain-Chatchat on Intel GPU <./chatchat_quickstart.html>`_ * `Run Text Generation WebUI on Intel GPU <./webui_quickstart.html>`_ * `Run Open WebUI on Intel GPU <./open_webui_with_ollama_quickstart.html>`_ @@ -23,12 +33,21 @@ This section includes efficient guide to show you how to: * `Run llama.cpp with IPEX-LLM on Intel GPU <./llama_cpp_quickstart.html>`_ * `Run Ollama with IPEX-LLM on Intel GPU <./ollama_quickstart.html>`_ * `Run Llama 3 on Intel GPU using llama.cpp and ollama with IPEX-LLM <./llama3_llamacpp_ollama_quickstart.html>`_ +* `Run RAGFlow with IPEX_LLM on Intel GPU <./ragflow_quickstart.html>`_ + +================= +Serving +================= + * `Run IPEX-LLM Serving with FastChat <./fastchat_quickstart.html>`_ * `Run IPEX-LLM Serving with vLLM on Intel GPU <./vLLM_quickstart.html>`_ -* `Finetune LLM with Axolotl on Intel GPU <./axolotl_quickstart.html>`_ * `Run IPEX-LLM serving on Multiple Intel GPUs using DeepSpeed AutoTP and FastApi <./deepspeed_autotp_fastapi_quickstart.html>`_ -* `Run RAGFlow using Ollama with IPEX_LLM <./ragflow_quickstart.html>`_ +================= +Finetune +================= + +* `Finetune LLM with Axolotl on Intel GPU <./axolotl_quickstart.html>`_ .. |bigdl_llm_migration_guide| replace:: ``bigdl-llm`` Migration Guide .. _bigdl_llm_migration_guide: bigdl_llm_migration.html diff --git a/docs/readthedocs/source/doc/LLM/Quickstart/ragflow_quickstart.md b/docs/readthedocs/source/doc/LLM/Quickstart/ragflow_quickstart.md index 384ec4e0..bb1a857f 100644 --- a/docs/readthedocs/source/doc/LLM/Quickstart/ragflow_quickstart.md +++ b/docs/readthedocs/source/doc/LLM/Quickstart/ragflow_quickstart.md @@ -1,18 +1,12 @@ -# Run RAGFlow using Ollama with IPEX_LLM +# Run RAGFlow with IPEX_LLM on Intel GPU -[ollama/ollama](https://github.com/ollama/ollama) is popular framework designed to build and run language models on a local machine; you can now use the C++ interface of [`ipex-llm`](https://github.com/intel-analytics/ipex-llm) as an accelerated backend for `ollama` running on Intel **GPU** *(e.g., local PC with iGPU, discrete GPU such as Arc, Flex and Max)*. +[RAGFlow](https://github.com/infiniflow/ragflow) is an open-source RAG (Retrieval-Augmented Generation) engine based on deep document understanding; by integrating it with [`ipex-llm`](https://github.com/intel-analytics/ipex-llm), users can now easily leverage local LLMs running on Intel GPU (e.g., local PC with iGPU, discrete GPU such as Arc, Flex and Max). -See the demo of running Qwen2-7B on Intel Arc GPU below. + +*See the demo of ragflow running Qwen2:7B on Intel Arc A770 below.* -```eval_rst -.. note:: - - `ipex-llm[cpp]==2.5.0b20240527` is consistent with `v0.1.34 `_ of ollama. - - Our current version is consistent with `v0.1.39 `_ of ollama. -``` ## Quickstart @@ -22,17 +16,18 @@ See the demo of running Qwen2-7B on Intel Arc GPU below. - RAM >= 16 GB - Disk >= 50 GB - Docker >= 24.0.0 & Docker Compose >= v2.26.1 -- Ollama service initialized -### 1. Install and Run Ollama Serve -Visit [Run Ollama with IPEX-LLM on Intel GPU](./ollama_quickstart.html), and follow the steps 1) [Install IPEX-LLM for Ollama](./ollama_quickstart.html#install-ipex-llm-for-ollama), 2) [Initialize Ollama](./ollama_quickstart.html#initialize-ollama) 3) [Run Ollama Serve](./ollama_quickstart.html#run-ollama-serve) to install, init and start the Ollama Service. +### 1. Install and Start `Ollama` Service on Intel GPU + +Follow the steps in [Run Ollama with IPEX-LLM on Intel GPU Guide](./ollama_quickstart.md) to install and run Ollama on Intel GPU. Ensure that `ollama serve` is running correctly and can be accessed through a local URL (e.g., `https://127.0.0.1:11434`) or a remote URL (e.g., `http://your_ip:11434`). + ```eval_rst .. important:: - If the `Ragflow` is not deployed on the same machine where Ollama is running (which means `Ragflow` needs to connect to a remote Ollama service), you must configure the Ollama service to accept connections from any IP address. To achieve this, set or export the environment variable `OLLAMA_HOST=0.0.0.0` before executing the command `ollama serve`. + If the `RAGFlow` is not deployed on the same machine where Ollama is running (which means `RAGFlow` needs to connect to a remote Ollama service), you must configure the Ollama service to accept connections from any IP address. To achieve this, set or export the environment variable `OLLAMA_HOST=0.0.0.0` before executing the command `ollama serve`. .. tip:: @@ -43,11 +38,9 @@ Visit [Run Ollama with IPEX-LLM on Intel GPU](./ollama_quickstart.html), and fol export SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=1 ``` -### 2. Pull and Prepare the Model +### 2. Pull Model -#### 2.1 Pull Model - -Now we need to pull a model for coding. Here we use [Qwen/Qwen2-7B](https://huggingface.co/Qwen/Qwen2-7B) model as an example. Open a new terminal window, run the following command to pull [`qwen2:latest`](https://ollama.com/library/qwen2). +Now we need to pull a model for RAG using Ollama. Here we use [Qwen/Qwen2-7B](https://huggingface.co/Qwen/Qwen2-7B) model as an example. Open a new terminal window, run the following command to pull [`qwen2:latest`](https://ollama.com/library/qwen2). ```eval_rst @@ -61,7 +54,7 @@ Now we need to pull a model for coding. Here we use [Qwen/Qwen2-7B](https://hugg .. tab:: Windows - Please run the following command in Miniforge Prompt. + Please run the following command in Miniforge or Anaconda Prompt. .. code-block:: cmd @@ -70,43 +63,51 @@ Now we need to pull a model for coding. Here we use [Qwen/Qwen2-7B](https://hugg .. seealso:: - Besides Qwen2, there are other coding models you might want to explore, such as Magicoder, Wizardcoder, Codellama, Codegemma, Starcoder, Starcoder2, and etc. You can find these models in the `Ollama model library `_. Simply search for the model, pull it in a similar manner, and give it a try. + Besides Qwen2, there are other LLM models you might want to explore, such as Llama3, Phi3, Mistral, etc. You can find all available models in the `Ollama model library `_. Simply search for the model, pull it in a similar manner, and give it a try. ``` -### 3. Initialize Ragflow +### 3. Start `RAGFlow` Service -Ensure `vm.max_map_count` >= 262144: -> To check the value of `vm.max_map_count`: -> -> ```bash -> $ sysctl vm.max_map_count -> ``` -> -> Reset `vm.max_map_count` to a value at least 262144 if it is not. -> -> ```bash -> # In this case, we set it to 262144: -> $ sudo sysctl -w vm.max_map_count=262144 -> ``` -> -> This change will be reset after a system reboot. To ensure your change remains permanent, add or update the `vm.max_map_count` value in **/etc/sysctl.conf** accordingly: -> -> ```bash -> vm.max_map_count=262144 -> ``` +#### 3.1 Download `RAGFlow` -Clone the repo: +You can either clone the repository or download the source zip from [github](https://github.com/infiniflow/ragflow/archive/refs/heads/main.zip): ```bash $ git clone https://github.com/infiniflow/ragflow.git ``` -### 4. Start up Ragflow server from Docker +#### 3.2 Environment Settings + +Ensure `vm.max_map_count` is set to at least 262144. To check the current value of `vm.max_map_count`, use: + +```bash +$ sysctl vm.max_map_count +``` + +##### Changing `vm.max_map_count` + +To set the value temporarily, use: + +```bash +$ sudo sysctl -w vm.max_map_count=262144 +``` + +To make the change permanent and ensure it persists after a reboot, add or update the following line in `/etc/sysctl.conf`: + +```bash +vm.max_map_count=262144 +``` + +### 3.3 Start the `RAGFlow` server using Docker Build the pre-built Docker images and start up the server: -> Running the following commands automatically downloads the *dev* version RAGFlow Docker image. To download and run a specified Docker version, update `RAGFLOW_VERSION` in **docker/.env** to the intended version, for example `RAGFLOW_VERSION=v0.7.0`, before running the following commands. +```eval_rst +.. note:: + + Running the following commands automatically downloads the *dev* version RAGFlow Docker image. To download and run a specified Docker version, update `RAGFLOW_VERSION` in **docker/.env** to the intended version, for example `RAGFLOW_VERSION=v0.7.0`, before running the following commands. +``` ```bash $ export no_proxy=localhost,127.0.0.1 @@ -115,8 +116,11 @@ $ chmod +x ./entrypoint.sh $ docker compose up -d ``` - -> The core image is about 9 GB in size and may take a while to load. +```eval_rst +.. note:: + + The core image is about 9 GB in size and may take a while to load. +``` Check the server status after having the server up and running: @@ -124,7 +128,7 @@ Check the server status after having the server up and running: $ docker logs -f ragflow-server ``` -_The following output confirms a successful launch of the system:_ +Upon successful deployment, you will see logs in the terminal similar to the following: ```bash ____ ______ __ @@ -139,15 +143,12 @@ _The following output confirms a successful launch of the system:_ * Running on http://x.x.x.x:9380 INFO:werkzeug:Press CTRL+C to quit ``` -> If you skip this confirmation step and directly log in to RAGFlow, your browser may prompt a `network anomaly` error because, at that moment, your RAGFlow may not be fully initialized. -In your web browser, enter the IP address of your server and log in to RAGFlow. -> With the default settings, you only need to enter `http://IP_OF_YOUR_MACHINE` (**sans** port number) as the default HTTP serving port `80` can be omitted when using the default configurations. -In [service_conf.yaml](./docker/service_conf.yaml), select the desired LLM factory in `user_default_llm` and update the `API_KEY` field with the corresponding API key. -> See [llm_api_key_setup](https://ragflow.io/docs/dev/llm_api_key_setup) for more information. +Open a browser and navigate to the URL displayed in the terminal logs. Look for messages like `Running on http://ip:port`. For local deployment, you can usually access the web portal at `http://127.0.0.1:9380`. For remote access, use `http://your_ip:9380`. -### 5. Using the Ragflow + +### 4. Using `RAGFlow` ```eval_rst .. note:: @@ -158,20 +159,21 @@ In [service_conf.yaml](./docker/service_conf.yaml), select the desired LLM facto #### Log-in -If this is your first time using it, you need to register. After registering, log in with the registered account to access the interface. +If this is your first time using RAGFlow, you will need to register. After registering, log in with your new account to access the portal. - - - +
    + + + + + + +
    - - - - #### Configure `Ollama` service URL -Access the Ollama settings through **Settings -> Model Providers** in the menu. Fill out the and **Base url**, and then hit the **OK** button at the bottom. +Access the Ollama settings through **Settings -> Model Providers** in the menu. Fill out the **Base URL**, and then click the **OK** button at the bottom. @@ -191,36 +193,40 @@ If the connection is successful, you will see the model listed down **Show more ``` #### Create Knowledge Base -Go to **Knowledge Base** after clicking **Knowledge Base** at the top bar. Hit the **+Create knowledge base** button on the right. You will be prompted to input a name for the knowledge base. + +Go to **Knowledge Base** by clicking on **Knowledge Base** in the top bar. Click the **+Create knowledge base** button on the right. You will be prompted to input a name for the knowledge base. + #### Edit Knowledge Base -After inputting a name, you will be directed to edit the knowledge base. Hit the **Dataset** on the left, and then hit **+ Add file -> Local files**. - - - +After entering a name, you will be directed to edit the knowledge base. Click on **Dataset** on the left, then click **+ Add file -> Local files**. Upload your file in the pop-up window and click **OK**. -Choose the file you want to train, and hit the green start button marked to start parsing the file. +
    + + + + + + +
    - - - +After the upload is successful, you will see a new record in the dataset. The _**Parsing Status**_ column will show `UNSTARTED`. Click the green start button in the _**Action**_ column to begin file parsing. Once parsing is finished, the _**Parsing Status**_ column will change to **SUCCESS**. - - - +
    + + + + + + +
    -It will show **SUCCESS** when the parsing is completed. - - - - -Then you can go to **Configuration** and hit **Save** at the bottom to save the changes. +Next, go to **Configuration** on the left menu and click **Save** at the bottom to save the changes. @@ -228,27 +234,36 @@ Then you can go to **Configuration** and hit **Save** at the bottom to save the #### Chat with the Model -Start new conversations with **Chat** at the top navbar. +Start new conversations by clicking **Chat** in the top navbar. + +On the left side, create a conversation by clicking **Create an Assistant**. Under **Assistant Setting**, give it a name and select your knowledge bases. -On the left-side, create a conversation by clicking **Create an Assistant**. Under **Assistant Setting**, give it a name and select your Knowledgebases. -Then go to **Model Setting**, choose your model added by Ollama. Make sure to disable the **Max Tokens** toggle and hit **OK** to start. + +Next, go to **Model Setting**, choose your model added by Ollama, and disable the **Max Tokens** toggle. Finally, click **OK** to start. + +```eval_rst +.. tip:: + + Enabling the **Max Tokens** toggle may result in very short answers. +```
    + Input your questions into the **Message Resume Assistant** textbox at the bottom, and click the button on the right to get responses. -#### Exit RAGFlow +#### Exit To shut down the RAGFlow server, use **Ctrl+C** in the terminal where the Ragflow server is runing, then close your browser tab.