Update Python Develop Guide (#4054)
* fix docs * update * update * update * update * update
This commit is contained in:
Kai Huang
GitHub
parent
1375438de3
commit
2f39947b16
1 changed files with 56 additions and 26 deletions
|
|
@ -2,13 +2,13 @@
|
|||
|
||||
---
|
||||
|
||||
Analytics Zoo source code is available at [GitHub](https://github.com/intel-analytics/analytics-zoo):
|
||||
BigDL source code is available at [GitHub](https://github.com/intel-analytics/BigDL):
|
||||
|
||||
```bash
|
||||
git clone https://github.com/intel-analytics/analytics-zoo.git
|
||||
git clone https://github.com/intel-analytics/BigDL.git
|
||||
```
|
||||
|
||||
By default, `git clone` will download the development version of Analytics Zoo. If you want a release version, you can use the command `git checkout` to change the specified version.
|
||||
By default, `git clone` will download the development version of BigDL. If you want a release version, you can use the command `git checkout` to change the specified version.
|
||||
|
||||
|
||||
### **1. Python**
|
||||
|
|
@ -18,49 +18,79 @@ By default, `git clone` will download the development version of Analytics Zoo.
|
|||
To generate a new [whl](https://pythonwheels.com/) package for pip install, you can run the following script:
|
||||
|
||||
```bash
|
||||
bash analytics-zoo/pyzoo/dev/build.sh linux default false
|
||||
cd BigDL/python/dev
|
||||
bash release_default_linux_spark246.sh default false false false # build on Spark 2.4.6 for linux
|
||||
# Use release_default_linux_spark312.sh to build on Spark 3.1.2 for linux
|
||||
# Use release_default_mac_spark246.sh to build on Spark 2.4.6 for mac
|
||||
# Use release_default_mac_spark312.sh to build on Spark 3.1.2 for mac
|
||||
```
|
||||
|
||||
**Arguments:**
|
||||
|
||||
- The first argument is the __platform__ to build for. Either 'linux' or 'mac'.
|
||||
- The second argument is the analytics-zoo __version__ to build for. 'default' means the default version for the current branch. You can also specify a different version if you wish, e.g., '0.6.0.dev1'.
|
||||
- You can also add other profiles to build the package, especially Spark and BigDL versions.
|
||||
For example, under the situation that `pyspark==2.4.3` is a dependency, you need to add profiles `-Dspark.version=2.4.3 -Dbigdl.artifactId=bigdl-SPARK_2.4 -P spark_2.4+` to build Analytics Zoo for Spark 2.4.3.
|
||||
- The first argument is the BigDL __version__ to build for. 'default' means the default version (`BigDL/python/version.txt`) for the current branch. You can also specify a different version if you wish, e.g., '0.14.0.dev1'.
|
||||
- The second argument is whether to __quick build__ BigDL Scala dependencies. You need to set it to be 'true' for the first build. In later builds, if you don't make any changes in BigDL Scala, you can set it to be 'false' so that the Scala dependencies would not be re-built.
|
||||
- The third argument is whether to __upload__ the packages to pypi. Set it to 'false' if you are simply developing BigDL for your own usage.
|
||||
- The fourth argument is whether to add __spark suffix__ (i.e. -spark2 or -spark3) to BigDL package names. Just set this to be 'false' if you are simply developing BigDL for your own usage.
|
||||
- You can also add other Maven profiles to build the package (if any) after the fourth argument, for example '-Ddata-store-url=..', etc.
|
||||
|
||||
|
||||
After running the above command, you will find a `whl` file under the folder `analytics-zoo/pyzoo/dist/`. You can then directly pip install it to your local Python environment:
|
||||
After running the above command, you will find a `whl` file for each submodule of BigDL and you can then directly pip install them to your local Python environment:
|
||||
```bash
|
||||
pip install analytics-zoo/pyzoo/dist/analytics_zoo-VERSION-py2.py3-none-PLATFORM_x86_64.whl
|
||||
# Install bigdl-nano
|
||||
cd BigDL/python/nano/src/dist
|
||||
pip install bigdl_nano-*.whl
|
||||
|
||||
# Install bigdl-dllib
|
||||
cd BigDL/python/dllib/src/dist
|
||||
pip install bigdl_dllib-*.whl
|
||||
|
||||
# Install bigdl-orca, which depends on bigdl-dllib and you need to install bigdl-dllib first
|
||||
cd BigDL/python/orca/src/dist
|
||||
pip install bigdl_orca-*.whl
|
||||
|
||||
# Install bigdl-friesian, which depends on bigdl-orca and you need to install bigdl-dllib and bigdl-orca first
|
||||
cd BigDL/python/friesian/src/dist
|
||||
pip install bigdl_friesian-*.whl
|
||||
|
||||
# Install bigdl-chronos, which depends on bigdl-orca and bigdl-nano. You need to install bigdl-dllib, bigdl-orca and bigdl-nano first
|
||||
cd BigDL/python/chronos/src/dist
|
||||
pip install bigdl_chronos-*.whl
|
||||
|
||||
# Install bigdl-serving
|
||||
cd BigDL/python/serving/src/dist
|
||||
pip install bigdl_serving-*.whl
|
||||
```
|
||||
|
||||
See [here](./python.md) for more instructions to run analytics-zoo after pip install.
|
||||
See [here](./python.md) for more instructions to run BigDL after pip install.
|
||||
|
||||
|
||||
#### **1.2 IDE Setup**
|
||||
Any IDE that support python should be able to run Analytics Zoo. PyCharm works fine for us.
|
||||
Any IDE that support Python should be able to run BigDL. PyCharm works fine for us.
|
||||
|
||||
You need to do the following preparations before starting the IDE to successfully run an Analytics Zoo Python program in the IDE:
|
||||
You need to do the following preparations before starting the IDE to successfully run a BigDL Python program in the IDE:
|
||||
|
||||
- Build Analytics Zoo; see [here](#21-build) for more instructions.
|
||||
- Prepare Spark environment by either setting `SPARK_HOME` as the environment variable or pip install `pyspark`. Note that the Spark version should match the one you build Analytics Zoo on.
|
||||
- Set BIGDL_CLASSPATH:
|
||||
- Build BigDL; see [here](#build) for more instructions.
|
||||
- Prepare Spark environment by either setting `SPARK_HOME` as the environment variable or `pip install pyspark`. Note that the Spark version should match the one you build BigDL on.
|
||||
- Check the jars under `BigDL/dist/lib` and set the environment variable `BIGDL_CLASSPATH`. Modify SPARKVERSION and BIGDLVERSION(Scala) as appropriate:
|
||||
```bash
|
||||
export BIGDL_CLASSPATH=analytics-zoo/dist/lib/analytics-zoo-*-jar-with-dependencies.jar
|
||||
export BIGDL_CLASSPATH=BigDL/dist/lib/bigdl-dllib-spark_SPARKVERSION-BIGDLVERSION-jar-with-dependencies.jar:BigDL/dist/lib/bigdl-orca-spark_SPARKVERSION-BIGDLVERSION-jar-with-dependencies.jar:BigDL/dist/lib/bigdl-friesian-spark_SPARKVERSION-BIGDLVERSION-jar-with-dependencies.jar
|
||||
```
|
||||
- Configure BigDL source files to the Python interpreter:
|
||||
|
||||
You can easily do this after launching PyCharm by right clicking the folder `BigDL/python/dllib/src` -> __Mark Directory As__ -> __Sources Root__ (also do this for `BigDL/python/nano/src`, `BigDL/python/orca/src`, `BigDL/python/friesian/src`, `BigDL/python/chronos/src`, `BigDL/python/serving/src` if necessary).
|
||||
|
||||
Alternatively, you can add BigDL source files to `PYTHONPATH`:
|
||||
```bash
|
||||
export PYTHONPATH=BigDL/python/dllib/src:BigDL/python/nano/src:BigDL/python/orca/src:BigDL/python/friesian/src:BigDL/python/chronos/src:BigDL/python/serving/src:$PYTHONPATH
|
||||
```
|
||||
|
||||
- Prepare BigDL Python environment by either downloading BigDL source code from [GitHub](https://github.com/intel-analytics/BigDL) or pip install `bigdl`. Note that the BigDL version should match the one you build Analytics Zoo on.
|
||||
- Add `pyzoo` and `spark-analytics-zoo.conf` to `PYTHONPATH`:
|
||||
- Add `spark-bigdl.conf` to `PYTHONPATH`:
|
||||
```bash
|
||||
export PYTHONPATH=analytics-zoo/pyzoo:analytics-zoo/dist/conf/spark-analytics-zoo.conf:$PYTHONPATH
|
||||
```
|
||||
If you download BigDL from [GitHub](https://github.com/intel-analytics/BigDL), you also need to add `BigDL/pyspark` to `PYTHONPATH`:
|
||||
```bash
|
||||
export PYTHONPATH=BigDL/pyspark:$PYTHONPATH
|
||||
export PYTHONPATH=BigDL/python/dist/conf/spark-bigdl.conf:$PYTHONPATH
|
||||
```
|
||||
|
||||
The above environmental variables should be available when running or debugging code in IDE.
|
||||
* In PyCharm, go to RUN -> Edit Configurations. In the "Run/Debug Configurations" panel, you can update the above environment variables in your configuration.
|
||||
The above environment variables should be available when running or debugging code in the IDE. When running applications in PyCharm, you can add runtime environment variables by clicking __Run__ -> __Edit Configurations__; then in the __Run/Debug Configurations__ panel, you can add necessary environment variables to your applications.
|
||||
|
||||
|
||||
### **2. Scala**
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue