updated ppl README (#11807)

* edit README.md

* update the branch

* edited README.md

* updated

* updated description

---------

Co-authored-by: jenniew <jenniewang123@gmail.com>
This commit is contained in:
RyuKosei 2024-08-16 15:49:25 +08:00 committed by GitHub
parent e07a55665c
commit 3b630fb9df
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
2 changed files with 10 additions and 8 deletions

View file

@ -3,23 +3,25 @@ Perplexity (PPL) is one of the most common metrics for evaluating language model
## Run on Wikitext ## Run on Wikitext
Download the dataset from [here](https://paperswithcode.com/dataset/wikitext-2), unzip it and we will use the test dataset `wiki.test.raw` for evaluation.
```bash ```bash
python run_wikitext.py --model_path meta-llama/Meta-Llama-3-8B/ --data_path wikitext-2-raw-v1/wikitext-2-raw/wiki.test.raw --precision sym_int4 --use-cache --device xpu pip install datasets
```
An example to run perplexity on wikitext:
```bash
python run_wikitext.py --model_path meta-llama/Meta-Llama-3-8B --dataset path=wikitext,name=wikitext-2-raw-v1 --precision sym_int4 --device xpu --stride 512 --max_length 4096
# Run with stride
python run_wikitext.py --model_path meta-llama/Meta-Llama-3-8B/ --data_path wikitext-2-raw-v1/wikitext-2-raw/wiki.test.raw --precision fp16 --device xpu --stride 512
``` ```
## Run on [THUDM/LongBench](https://github.com/THUDM/LongBench) dataset ## Run on [THUDM/LongBench](https://github.com/THUDM/LongBench) dataset
```bash ```bash
python run.py --model_path <path/to/model> --precisions sym_int4 fp8 --device xpu --datasets dataset_names --dataset_path <path/to/dataset> --language en pip install datasets
``` ```
A more specific example to run perplexity on Llama2-7B using the default English datasets:
An example to run perplexity on chatglm3-6b using the default Chinese datasets("multifieldqa_zh", "dureader", "vcsum", "lsht", "passage_retrieval_zh")
```bash ```bash
python run.py --model_path meta-llama/Llama-2-7b-chat-hf --precisions float16 sym_int4 --device xpu --language en python run_longbench.py --model_path THUDM/chatglm3-6b --precisions float16 sym_int4 --device xpu --language zh
``` ```
Notes: Notes: