ipex-llm/python/llm/example/CPU/Deepspeed-AutoTP/README.md

Run Tensor-Parallel BigDL Transformers INT4 Inference with Deepspeed

1. Install Dependencies

Install necessary packages (here Python 3.9 is our test environment):

bash install.sh

2. Initialize Deepspeed Distributed Context

Like shown in example code deepspeed_autotp.py, you can construct parallel model with Python API:

# Load in HuggingFace Transformers' model
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(...)


# Parallelize model on deepspeed
import deepspeed

model = deepspeed.init_inference(
    model, # an AutoModel of Transformers
    mp_size = world_size, # instance (process) count
    dtype=torch.float16,
    replace_method="auto")

Then, returned model is converted into a deepspeed InferenceEnginee type.

3. Optimize Model with BigDL-LLM Low Bit

Distributed model managed by deepspeed can be further optimized with BigDL low-bit Python API, e.g. sym_int4:

# Apply BigDL-LLM INT4 optimizations on transformers
from bigdl.llm import optimize_model

model = optimize_model(model.module.to(f'cpu'), low_bit='sym_int4')
model = model.to(f'cpu:{local_rank}') # move partial model to local rank

Then, a bigdl-llm transformers is returned, which in the following, can serve in parallel with native APIs.

4. Start Python Code

You can try deepspeed with BigDL LLM by:

bash run.sh

If you want to run your own application, there are necessary configurations in the script which can also be ported to run your custom deepspeed application:

# run.sh
source bigdl-nano-init
unset OMP_NUM_THREADS # deepspeed will set it for each instance automatically
source /opt/intel/oneccl/env/setvars.sh
......
export FI_PROVIDER=tcp
export CCL_ATL_TRANSPORT=ofi
export CCL_PROCESS_LAUNCHER=none

Set the above configurations before running deepspeed please to ensure right parallel communication and high performance.