ayo/ipex-llm
1
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

revise private GPT quickstart and a few fixes for other quickstart (#10967)

Browse source
This commit is contained in:
Shengsheng Huang 2024-05-08 21:18:20 +08:00 committed by GitHub
parent 37820e1d86
commit 11df5f9773
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
4 changed files with 39 additions and 34 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

1
docs/readthedocs/source/_toc.yml
View file

@ -28,6 +28,7 @@ subtrees:
- file: doc/LLM/Quickstart/open_webui_with_ollama_quickstart
- file: doc/LLM/Quickstart/privateGPT_quickstart
- file: doc/LLM/Quickstart/continue_quickstart
- file: doc/LLM/Quickstart/dify_quickstart
- file: doc/LLM/Quickstart/benchmark_quickstart
- file: doc/LLM/Quickstart/llama_cpp_quickstart
- file: doc/LLM/Quickstart/ollama_quickstart

18
docs/readthedocs/source/doc/LLM/Quickstart/dify_quickstart.md
View file

@ -19,23 +19,15 @@ Follow the steps in [Run Ollama on Intel GPU Guide](./ollama_quickstart.md) to i
### 2. Install and Start `Dify`
```eval_rst
.. note::
The following steps are verified on Linux.
```
#### Download `Dify`
#### 2.1 Download `Dify`
You can either clone the repository or download the source zip from [github](https://github.com/langgenius/dify/archive/refs/heads/main.zip):
```bash
git clone https://github.com/langgenius/dify.git
```
#### Setup Redis and PostgreSQL
#### 2.2 Setup Redis and PostgreSQL
Next, deploy PostgreSQL and Redis. You can choose to utilize Docker, following the steps in the [Local Source Code Start Guide](https://docs.dify.ai/getting-started/install-self-hosted/local-source-code#clone-dify), or proceed without Docker using the following instructions:
@ -68,7 +60,7 @@ DB_PORT=5432
DB_DATABASE=dify # change if needed
```
#### Server Deployment
#### 2.3 Server Deployment
Follow the steps in the [`Server Deployment` section in Local Source Code Start Guide](https://docs.dify.ai/getting-started/install-self-hosted/local-source-code#server-deployment) to deploy and start the Dify Server.
@ -87,7 +79,7 @@ INFO:werkzeug: * Debugger PIN: 227-697-894
```
#### Deploy the frontend page
#### 2.4 Deploy the frontend page
Refer to the instructions provided in the [`Deploy the frontend page` section in Local Source Code Start Guide](https://docs.dify.ai/getting-started/install-self-hosted/local-source-code#deploy-the-frontend-page) to deploy the frontend page.
@ -112,7 +104,7 @@ NEXT_PUBLIC_SENTRY_DSN=
```
### How to Use `Dify`
### 3. How to Use `Dify`
For comprehensive usage instructions of Dify, please refer to the [Dify Documentation](https://docs.dify.ai/). In this section, we'll only highlight a few key steps for local LLM setup.

2
docs/readthedocs/source/doc/LLM/Quickstart/open_webui_with_ollama_quickstart.md
View file

@ -2,7 +2,7 @@
[Open WebUI](https://github.com/open-webui/open-webui) is a user friendly GUI for running LLM locally; by porting it to [`ipex-llm`](https://github.com/intel-analytics/ipex-llm), users can now easily run LLM in [Open WebUI](https://github.com/open-webui/open-webui) on Intel **GPU** *(e.g., local PC with iGPU, discrete GPU such as Arc, Flex and Max)*.
See the demo of running Mistral:7B on Intel Arc A770 below.
*See the demo of running Mistral:7B on Intel Arc A770 below.*
<video src="https://llm-assets.readthedocs.io/en/latest/_images/open_webui_demo.mp4" width="100%" controls></video>

52
docs/readthedocs/source/doc/LLM/Quickstart/privateGPT_quickstart.md
View file

@ -1,31 +1,41 @@
# Run PrivateGPT with IPEX-LLM on Intel GPU
[zylon-ai/private-gpt](https://github.com/zylon-ai/private-gpt) is a production-ready AI project that allows you to ask questions about your documents using the power of Large Language Models (LLMs), even in scenarios without an Internet connection; you can easily run PrivateGPT using `Ollama` with IPEX-LLM on Intel **GPU** *(e.g., local PC with iGPU, discrete GPU such as Arc, Flex and Max)*.
[PrivateGPT](https://github.com/zylon-ai/private-gpt) is a production-ready AI project that allows users to chat over documents, etc.; by integrating it with [`ipex-llm`](https://github.com/intel-analytics/ipex-llm), users can now easily leverage local LLMs running on Intel GPU (e.g., local PC with iGPU, discrete GPU such as Arc, Flex and Max).
See the demo of running Mistral-7B on Intel iGPU below:
*See the demo of privateGPT running Mistral:7B on Intel Arc A770 below.*
<video src="https://github.com/bigdl-project/bigdl-project.github.io/raw/2ca10e374da99310f7b8f5b01bbd69242eab4d3a/assets/quickstart/privateGPT_quickstart/privateGPT-windows-MTL.mp4" width="100%" controls></video>
<video src="https://llm-assets.readthedocs.io/en/latest/_images/PrivateGPT-ARC.mp4" width="100%" controls></video>
## Quickstart
### 1 Run Ollama with Intel GPU
Follow the instructions on the [Run Ollama with IPEX-LLM on Intel GPU](https://ipex-llm.readthedocs.io/en/latest/doc/LLM/Quickstart/ollama_quickstart.html) to install and run Ollama Serve. Please ensure that the Ollama server continues to run while you're using the PrivateGPT.
### 1. Install and Start `Ollama` Service on Intel GPU
Follow the steps in [Run Ollama on Intel GPU Guide](./ollama_quickstart.md) to install and run Ollama on Intel GPU. Ensure that `ollama serve` is running correctly and can be accessed through a local URL (e.g., `https://127.0.0.1:11434`) or a remote URL (e.g., `http://your_ip:11434`).
### 2. Install PrivateGPT
#### Download PrivateGPT
Use `git` to clone the [zylon-ai/private-gpt](https://github.com/zylon-ai/private-gpt).
You can either clone the repository or download the source zip from [github](https://github.com/zylon-ai/private-gpt/archive/refs/heads/main.zip):
```bash
git clone https://github.com/zylon-ai/private-gpt
```
#### Install Dependencies
You may run below commands to install PrivateGPT dependencies:
Execute the following commands in a terminal to install the dependencies of PrivateGPT:
```cmd
cd private-gpt
pip install poetry
pip install ffmpy==0.3.1
poetry install --extras "ui llms-ollama embeddings-ollama vector-stores-qdrant"
```
For more details, refer to the [PrivateGPT installation Guide](https://docs.privategpt.dev/installation/getting-started/main-concepts).
### 3. Start PrivateGPT
@ -36,7 +46,7 @@ Change PrivateGPT settings by modifying `settings.yaml` and `settings-ollama.yam
* `settings.yaml` is always loaded and contains the default configuration. In order to run PrivateGPT locally, you need to replace the tokenizer path under the `llm` option with your local path.
* `settings-ollama.yaml` is loaded if the ollama profile is specified in the PGPT_PROFILES environment variable. It can override configuration from the default `settings.yaml`. You can modify the settings in this file according to your preference. It is worth noting that to use the options `llm_model: <Model Name>` and `embedding_model: <Embedding Model Name>`, you need to first use `ollama pull` to fetch the models locally.
To learn more about the configuration of PrivatePGT, please refer to [PrivateGPT Main Concepts](https://docs.privategpt.dev/installation/getting-started/main-concepts)
To learn more about the configuration of PrivatePGT, please refer to [PrivateGPT Main Concepts](https://docs.privategpt.dev/installation/getting-started/main-concepts).
#### Start the service
@ -74,17 +84,19 @@ Run below commands to start the service in another terminal:
#### Chat with the Model
Select the "LLM Chat" option in the upper left corner of the page to chat with LLM.
To chat with the LLM, select the "LLM Chat" option located in the upper left corner of the page. Type your messages at the bottom of the page and click the "Submit" button to receive responses from the model.
<a href="https://raw.githubusercontent.com/bigdl-project/bigdl-project.github.io/2ca10e374da99310f7b8f5b01bbd69242eab4d3a/assets/quickstart/privateGPT_quickstart/privateGPT-LLM-Chat.png" target="_blank">
<img src="https://raw.githubusercontent.com/bigdl-project/bigdl-project.github.io/2ca10e374da99310f7b8f5b01bbd69242eab4d3a/assets/quickstart/privateGPT_quickstart/privateGPT-LLM-Chat.png" width=100%; />
</a>
#### Using RAG
Select the "Query Files" option in the upper left corner of the page, then click the "Upload File(s)" button to upload documents. Once the document vectorization is completed, you can proceed with document-based QA.
<a href="https://raw.githubusercontent.com/bigdl-project/bigdl-project.github.io/2ca10e374da99310f7b8f5b01bbd69242eab4d3a/assets/quickstart/privateGPT_quickstart/privateGPT-Query-Files.png" target="_blank">
<img src="https://raw.githubusercontent.com/bigdl-project/bigdl-project.github.io/2ca10e374da99310f7b8f5b01bbd69242eab4d3a/assets/quickstart/privateGPT_quickstart/privateGPT-Query-Files.png" width=100%; />
<p align="center"><a href="https://llm-assets.readthedocs.io/en/latest/_images/privateGPT-LLM-Chat.png" target="_blank" align="center">
<img src="https://llm-assets.readthedocs.io/en/latest/_images/privateGPT-LLM-Chat.png" alt="image-p1" width=100%; />
</a></p>
#### Chat over Documents (RAG)
To interact with documents, select the "Query Files" option in the upper left corner of the page. Click the "Upload File(s)" button to upload documents. After the documents have been vectorized, you can type your messages at the bottom of the page and click the "Submit" button to receive responses from the model based on the uploaded content.
<a href="https://llm-assets.readthedocs.io/en/latest/_images/privateGPT-Query-Files.png" target="_blank">
<img src="https://llm-assets.readthedocs.io/en/latest/_images/privateGPT-Query-Files.png" width=100%; />
</a>