From 8ba635c1015e12a7b70587f55b0718ce3bc4c209 Mon Sep 17 00:00:00 2001 From: dding3 Date: Mon, 25 Oct 2021 23:07:41 -0700 Subject: [PATCH] add dllib user guide (#3212) * add dllib user guide * add spark-submit-with-bigdl script * copy scripts to zip --- .../source/doc/DLlib/Overview/dllib.md | 316 +++++++++++++++++- 1 file changed, 313 insertions(+), 3 deletions(-) diff --git a/docs/readthedocs/source/doc/DLlib/Overview/dllib.md b/docs/readthedocs/source/doc/DLlib/Overview/dllib.md index beb7fbef..e34da25a 100644 --- a/docs/readthedocs/source/doc/DLlib/Overview/dllib.md +++ b/docs/readthedocs/source/doc/DLlib/Overview/dllib.md @@ -13,14 +13,324 @@ It includes the functionalities of the [original BigDL](https://github.com/intel ### 2.1 Install -### 2.2 Run +#### 2.1.1 **Download a pre-built library** +You can download the bigdl-dllib build from the [Release Page](../release.md). -### 2.3 Get started (example) +#### 2.1.2 **Link with a release version** + +Currently, dllib releases are hosted on maven central; here's an example to add the dllib dependency to your own project: +```xml + + com.intel.analytics.bigdl + bigdl-dllib-[spark_2.4.6|spark_3.1.2] + ${BIGD_DLLIB_VERSION} + +``` +Please choose the suffix according to your Spark platform. + +SBT developers can use +```sbt +libraryDependencies += "com.intel.analytics.bigdl" % "dllib-[spark_2.4.6|spark_3.1.2]" % "${BIGDL_DLLIB_VERSION}" +``` + +### 2.2 Run +#### 2.2.1 **Set Environment Variables** +Set **BIGDL_HOME** and **SPARK_HOME**: + +* If you download bigdl-dllib from the [Release Page](../release-download.md) +```bash +export SPARK_HOME=folder path where you extract the spark package +export BIGDL_HOME=folder path where you extract the bigdl package +``` + +--- +#### 2.2.2 **Use Interactive Spark Shell** +You can try bigdl-dllib easily using the Spark interactive shell. Run below command to start spark shell with bigdl-dllib support: +```bash +${BIGDL_HOME}/bin/spark-shell-with-dllib.sh +``` +You will see a welcome message looking like below: +``` +Welcome to + ____ __ + / __/__ ___ _____/ /__ + _\ \/ _ \/ _ `/ __/ '_/ + /___/ .__/\_,_/_/ /_/\_\ version 2.4.3 + /_/ + +Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_181) +Spark context available as sc. +scala> +``` + +To use BigDL, you should first initialize the environment as below. +```scala +scala> import com.intel.analytics.bigdl.dllib.NNContext +import com.intel.analytics.bigdl.dllib.NNContext +scala> NNContext.initNNContext() +2021-10-25 10:12:36 WARN SparkContext:66 - Using an existing SparkContext; some configuration may not take effect. +2021-10-25 10:12:36 WARN SparkContext:66 - Using an existing SparkContext; some configuration may not take effect. +res0: org.apache.spark.SparkContext = org.apache.spark.SparkContext@525c0f74 +``` + +Once the environment is successfully initiated, you'll be able to play with dllib API's. +For instance, to experiment with the ````dllib.keras```` APIs in dllib, you may try below code: +```scala +scala> import com.intel.analytics.bigdl.dllib.keras.layers._ +scala> import com.intel.analytics.bigdl.numeric.NumericFloat +scala> import com.intel.analytics.bigdl.dllib.utils.Shape + +scala> val seq = Sequential() + val layer = ConvLSTM2D(32, 4, returnSequences = true, borderMode = "same", + inputShape = Shape(8, 40, 40, 32)) + seq.add(layer) +``` + +--- + +#### 2.2.3 **Run as a Spark Program** +You can run a bigdl-dllib program, e.g., the [Image Inference](https://github.com/intel-analytics/BigDL/blob/branch-2.0/scala/dllib/src/main/scala/com/intel/analytics/bigdl/dllib/example/nnframes/imageInference), as a standard Spark program (running on either a local machine or a distributed cluster) as follows: + +1. Download the pretrained caffe model and prepare the images + +2. Run the following command: +```bash +# Spark local mode +${BIGDL_HOME}/bin/spark-submit-with-dllib.sh \ + --master local[2] \ + --class com.intel.analytics.bigdl.dllib.example.languagemodel.PTBWordLM \ + ${BIGDL_HOME}/jars/bigdl-dllib-0.14.0-SNAPSHOT-jar-with-dependencies.jar \ #change to your jar file if your download is not spark_2.4.3-0.14.0 + -f DATA_PATH \ + -b 4 \ + --numLayers 2 --vocab 100 --hidden 6 \ + --numSteps 3 --learningRate 0.005 -e 1 \ + --learningRateDecay 0.001 --keepProb 0.5 + +# Spark standalone mode +## ${SPARK_HOME}/sbin/start-master.sh +## check master URL from http://localhost:8080 +${BIGDL_HOME}/bin/spark-submit-with-dllib.sh \ + --master spark://... \ + --executor-cores cores_per_executor \ + --total-executor-cores total_cores_for_the_job \ + --class com.intel.analytics.bigdl.dllib.example.languagemodel.PTBWordLM \ + ${BIGDL_HOME}/jars/bigdl-dllib-0.14.0-SNAPSHOT-jar-with-dependencies.jar \ #change to your jar file if your download is not spark_2.4.3-0.14.0 + -f DATA_PATH \ + -b 4 \ + --numLayers 2 --vocab 100 --hidden 6 \ + --numSteps 3 --learningRate 0.005 -e 1 \ + --learningRateDecay 0.001 --keepProb 0.5 + +# Spark yarn client mode +${BIGDL_HOME}/bin/spark-submit-with-dllib.sh \ + --master yarn \ + --deploy-mode client \ + --executor-cores cores_per_executor \ + --num-executors executors_number \ + --class com.intel.analytics.bigdl.dllib.example.languagemodel.PTBWordLM \ + ${BIGDL_HOME}/jars/bigdl-dllib-0.14.0-SNAPSHOT-jar-with-dependencies.jar \ #change to your jar file if your download is not spark_2.4.3-0.14.0 + -f DATA_PATH \ + -b 4 \ + --numLayers 2 --vocab 100 --hidden 6 \ + --numSteps 3 --learningRate 0.005 -e 1 \ + --learningRateDecay 0.001 --keepProb 0.5 + +# Spark yarn cluster mode +${BIGDL_HOME}/bin/spark-submit-with-dllib.sh \ + --master yarn \ + --deploy-mode cluster \ + --executor-cores cores_per_executor \ + --num-executors executors_number \ + --class com.intel.analytics.bigdl.dllib.example.languagemodel.PTBWordLM \ + ${BIGDL_HOME}/jars/bigdl-dllib-0.14.0-SNAPSHOT-jar-with-dependencies.jar \ #change to your jar file if your download is not spark_2.4.3-0.14.0 + -f DATA_PATH \ + -b 4 \ + --numLayers 2 --vocab 100 --hidden 6 \ + --numSteps 3 --learningRate 0.005 -e 1 \ + --learningRateDecay 0.001 --keepProb 0.5 +``` + + The parameters used in the above command are: + + * -f: The path where you put your PTB data. + * -b: The mini-batch size. The mini-batch size is expected to be a multiple of *total cores* used in the job. In this example, the mini-batch size is suggested to be set to *total cores * 4* + * --learningRate: learning rate for adagrad + * --learningRateDecay: learning rate decay for adagrad + * --hidden: hiddensize for lstm + * --vocabSize: vocabulary size, default 10000 + * --numLayers: numbers of lstm cell, default 2 lstm cells + * --numSteps: number of words per record in LM + * --keepProb: the probability to do dropout + +If you are to run your own program, do remember to do the initialize before call other bigdl-dllib API's, as shown below. +```scala + // Scala code example + import com.intel.analytics.bigdl.dllib.NNContext + NNContext.initNNContext() +``` +--- + +### 2.3 Get started +--- + +This section show a single example of how to use dllib to build a deep learning application on Spark, using Keras APIs + +--- +#### **LeNet Model on MNIST using Keras-Style API** + +This tutorial is an explanation of what is happening in the [lenet](https://github.com/intel-analytics/BigDL/tree/branch-2.0/scala/dllib/src/main/scala/com/intel/analytics/bigdl/dllib/example/keras) example + +A bigdl-dllib program starts with initialize as follows. +````scala + val conf = Engine.createSparkConf() + .setAppName("Train Lenet on MNIST") + .set("spark.task.maxFailures", "1") + val sc = new SparkContext(conf) + Engine.init +```` + +After the initialization, we need to: + +1. Load train and validation data by _**creating the [```DataSet```](https://github.com/intel-analytics/BigDL/blob/branch-2.0/scala/dllib/src/main/scala/com/intel/analytics/bigdl/dllib/feature/dataset/DataSet.scala)**_ (e.g., ````SampleToGreyImg````, ````GreyImgNormalizer```` and ````GreyImgToBatch````): + +````scala + val trainSet = (if (sc.isDefined) { + DataSet.array(load(trainData, trainLabel), sc.get, param.nodeNumber) + } else { + DataSet.array(load(trainData, trainLabel)) + }) -> SampleToGreyImg(28, 28) -> GreyImgNormalizer(trainMean, trainStd) -> GreyImgToBatch( + param.batchSize) + + val validationSet = DataSet.array(load(validationData, validationLabel), sc) -> + BytesToGreyImg(28, 28) -> GreyImgNormalizer(testMean, testStd) -> GreyImgToBatch( + param.batchSize) +```` + +2. We then define Lenet model using Keras-style api +````scala + val input = Input(inputShape = Shape(28, 28, 1)) + val reshape = Reshape(Array(1, 28, 28)).inputs(input) + val conv1 = Convolution2D(6, 5, 5, activation = "tanh").setName("conv1_5x5").inputs(reshape) + val pool1 = MaxPooling2D().inputs(conv1) + val conv2 = Convolution2D(12, 5, 5, activation = "tanh").setName("conv2_5x5").inputs(pool1) + val pool2 = MaxPooling2D().inputs(conv2) + val flatten = Flatten().inputs(pool2) + val fc1 = Dense(100, activation = "tanh").setName("fc1").inputs(flatten) + val fc2 = Dense(classNum, activation = "softmax").setName("fc2").inputs(fc1) + Model(input, fc2) + ```` + +3. After that, we configure the learning process. Set the ````optimization method```` and the ````Criterion```` (which, given input and target, computes gradient per given loss function): +````scala + model.compile(optimizer = optimMethod, + loss = ClassNLLCriterion[Float](logProbAsInput = false), + metrics = Array(new Top1Accuracy[Float](), new Top5Accuracy[Float](), new Loss[Float])) +```` + +Finally we _**train the model**_ by calling ````model.fit````: +````scala + model.fit(trainSet, nbEpoch = param.maxEpoch, validationData = validationSet) +```` + +--- ## 3. Python user guide ### 3.1 Install +#### 3.1.1 Official Release + +Run below command to install _bigdl-dllib_. + +```bash +conda create -n my_env python=3.7 +conda activate my_env +pip install bigdl-dllib +``` + +#### 3.1.2 Nightly build + +You can install the latest nightly build of bigdl-dllib as follows: +```bash +pip install --pre --upgrade bigdl-dllib +``` + + ### 3.2 Run -### 3.3 Get started (example) +#### **3.2.1 Interactive Shell** + +You may test if the installation is successful using the interactive Python shell as follows: + +* Type `python` in the command line to start a REPL. +* Try to run the example code below to verify the installation: + + ```python + from bigdl.dllib.utils.nncontext import * + + sc = init_nncontext() # Initiation of bigdl-dllib on the underlying cluster. + ``` + +#### **3.2.2 Jupyter Notebook** + +You can start the Jupyter notebook as you normally do using the following command and run bigdl-dllib programs directly in a Jupyter notebook: + +```bash +jupyter notebook --notebook-dir=./ --ip=* --no-browser +``` + +#### **3.2.3 Python Script** + +You can directly write bigdl-dlllib programs in a Python file (e.g. script.py) and run in the command line as a normal Python program: + +```bash +python script.py +``` +--- +### 3.3 Get started +--- + +#### **Autograd Examples using bigdl-dllb keras Python API** + +This tutorial describes the [Autograd](https://github.com/intel-analytics/BigDL/tree/branch-2.0/python/dllib/examples/autograd). + +The example first do the initializton using `init_nncontext()`: +```python + sc = init_nncontext() +``` + +It then generate the input data X_, Y_ + +```python + data_len = 1000 + X_ = np.random.uniform(0, 1, (1000, 2)) + Y_ = ((2 * X_).sum(1) + 0.4).reshape([data_len, 1]) +``` + +It then define the custom loss + +```python +def mean_absolute_error(y_true, y_pred): + result = mean(abs(y_true - y_pred), axis=1) + return result +``` + +After that, the example creates the model as follows and set the criterion as the custom loss: +```python + a = Input(shape=(2,)) + b = Dense(1)(a) + c = Lambda(function=add_one_func)(b) + model = Model(input=a, output=c) + + model.compile(optimizer=SGD(learningrate=1e-2), + loss=mean_absolute_error) +``` +Finally the example trains the model by calling `model.fit`: + +```python + model.fit(x=X_, + y=Y_, + batch_size=32, + nb_epoch=int(options.nb_epoch), + distributed=False) +``` \ No newline at end of file