ayo/ipex-llm
1
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

[Orca] Add bigdl-submit mode support (#5960)

Browse source 
* add bigdl-submit mode support

* modify all submit-with-bigdl

* remove submit-with-bigdl info
This commit is contained in:
SONG Ge 2022-09-27 15:17:41 +08:00 committed by GitHub
parent bff6795199
commit 79f29244d8
1 changed files with 3 additions and 3 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

6
docs/readthedocs/source/doc/UserGuide/hadoop.md
View file

@ -103,11 +103,11 @@ Follow the steps below if you need to run BigDL with [spark-submit](https://spar
sc = init_orca_context(cluster_mode="spark-submit") sc = init_orca_context(cluster_mode="spark-submit")
``` ```
- Use `spark-submit-with-bigdl` to submit your BigDL program (e.g. script.py). You can adjust the configurations according to your cluster settings. Note that if `environment.tar.gz` is not under the same directory with `script.py`, you may need to modify its path in `--archives` in the running command below. - Use `bigdl-submit` to submit your BigDL program (e.g. script.py). You can adjust the configurations according to your cluster settings. Note that if `environment.tar.gz` is not under the same directory with `script.py`, you may need to modify its path in `--archives` in the running command below.
For `yarn-cluster` mode: For `yarn-cluster` mode:
```bash ```bash
spark-submit-with-bigdl \ bigdl-submit \
--conf spark.yarn.appMasterEnv.PYSPARK_PYTHON=environment/bin/python \ --conf spark.yarn.appMasterEnv.PYSPARK_PYTHON=environment/bin/python \
--conf spark.executorEnv.PYSPARK_PYTHON=environment/bin/python \ --conf spark.executorEnv.PYSPARK_PYTHON=environment/bin/python \
--master yarn \ --master yarn \
@ -124,7 +124,7 @@ Follow the steps below if you need to run BigDL with [spark-submit](https://spar
For `yarn-client` mode: For `yarn-client` mode:
```bash ```bash
PYSPARK_PYTHON=environment/bin/python spark-submit-with-bigdl \ PYSPARK_PYTHON=environment/bin/python bigdl-submit \
--master yarn \ --master yarn \
--deploy-mode client \ --deploy-mode client \
--executor-memory 10g \ --executor-memory 10g \