ayo/ipex-llm
1
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1
ipex-llm/python/llm/portable-zip
History
ZehuaCao 4127b99ed6
Fix null pointer dereferences error. (#11125) 
* delete unused function on tgi_server

* update

* update

* fix style
2024-05-30 16:16:10 +08:00
..
.gitignore LLM: add ui for portable-zip (#9262) 2023-11-01 15:36:59 +08:00
chat-ui.bat Migrate portable zip to ipex-llm (#10617) 2024-04-07 13:58:58 +08:00
chat.bat LLM: adjust portable zip content (#9054) 2023-10-09 10:51:19 +08:00
chat.py Fix null pointer dereferences error. (#11125) 2024-05-30 16:16:10 +08:00
kv_cache.py LLM: improve response speed in multi-turn chat (#9299) 2023-11-01 10:30:44 +08:00
README-ui.md Update_document by heyang (#30) 2024-03-25 10:06:02 +08:00
README.md Fix baichuan-13b issue on portable zip under transformers 4.36 (#10746) 2024-04-12 16:27:01 -07:00
setup.bat Migrate portable zip to ipex-llm (#10617) 2024-04-07 13:58:58 +08:00
setup.md Update_document by heyang (#30) 2024-03-25 10:06:02 +08:00

README.md

IPEX-LLM Portable Zip For Windows: User Guide

Introduction

This portable zip includes everything you need to run an LLM with IPEX-LLM optimizations (except models) . Please refer to How to use section to get started.

13B model running on an Intel 11-Gen Core PC (real-time screen capture)

Verified Models

  • Llama-2-7b-chat-hf
  • Yi-6B-Chat
  • Mixtral-8x7B-Instruct-v0.1
  • Mistral-7B-Instruct-v0
  • ChatGLM2-6b
  • ChatGLM3-6b
  • Baichuan-13B-Chat
  • Baichuan2-7B-Chat
  • internlm-chat-7b
  • internlm2-chat-7b
  • Qwen-7B-Chat

How to use

  1. Download the zip from link here.
  2. (Optional) You could also build the zip on your own. Run setup.bat and it will generate the zip file.
  3. Unzip ipex-llm.zip.
  4. Download the model to your computer. Please ensure there is a file named config.json in the model folder, otherwise the script won't work.

  1. Go into the unzipped folder and double click chat.bat. Input the path of the model (e.g. path\to\model, note that there's no slash at the end of the path). Press Enter and wait until model finishes loading. Then enjoy chatting with the model!

  1. If you want to stop chatting, just input stop and the model will stop running.