diff --git a/docs/readthedocs/source/doc/DLlib/Overview/clipping.md b/docs/readthedocs/source/doc/DLlib/Overview/clipping.md
new file mode 100644
index 00000000..3e2709df
--- /dev/null
+++ b/docs/readthedocs/source/doc/DLlib/Overview/clipping.md
@@ -0,0 +1,20 @@
+## ConstantGradientClipping ##
+
+Set constant gradient clipping during the training process.
+
+```scala
+model.setConstantGradientClipping(clipNorm)
+```
+param:
+   * min: The minimum value to clip by.
+   * max: The maximum value to clip by.
+
+## GradientClippingByL2Norm ##
+
+Clip gradient to a maximum L2-Norm during the training process.
+
+```scala
+model.setGradientClippingByL2Norm(clipNorm)
+```
+param:
+   * clipNorm: Gradient L2-Norm threshold
diff --git a/docs/readthedocs/source/doc/DLlib/Overview/freeze.md b/docs/readthedocs/source/doc/DLlib/Overview/freeze.md
new file mode 100644
index 00000000..fbe1b6b3
--- /dev/null
+++ b/docs/readthedocs/source/doc/DLlib/Overview/freeze.md
@@ -0,0 +1,107 @@
+## Model Freeze
+To "freeze" a model means to exclude some layers of model from training.
+
+```scala
+model.freeze("layer1", "layer2")
+model.unFreeze("layer1", "layer2")
+```
+* The model can be "freezed" by calling ```freeze()```. If a model is freezed,
+its parameters(weight/bias, if exists) are not changed in training process.
+If model names are passed, then layers that match the given names will be freezed.
+* The whole model can be "unFreezed" by calling ```unFreeze()```.
+If model names are provided, then layers that match the given names will be unFreezed.
+* stop the input gradient of layers that match the given names. Their input gradient are not computed.
+And they will not contributed to the input gradient computation of layers that depend on them.
+
+Original model without "freeze"
+```scala
+val reshape = Reshape(Array(4)).inputs()
+val fc1 = Linear(4, 2).setName("fc1").inputs()
+val fc2 = Linear(4, 2).setName("fc2").inputs(reshape)
+val cadd_1 = CAddTable().setName("cadd").inputs(fc1, fc2)
+val output1_1 = ReLU().inputs(cadd_1)
+val output2_1 = Threshold(10.0).inputs(cadd_1)
+
+val model = Graph(Array(reshape, fc1), Array(output1_1, output2_1))
+
+val input = T(Tensor(T(0.1f, 0.2f, -0.3f, -0.4f)),
+  Tensor(T(0.5f, 0.4f, -0.2f, -0.1f)))
+val gradOutput = T(Tensor(T(1.0f, 2.0f)), Tensor(T(3.0f, 4.0f)))
+
+fc1.element.getParameters()._1.apply1(_ => 1.0f)
+fc2.element.getParameters()._1.apply1(_ => 2.0f)
+model.zeroGradParameters()
+println("output1: \n", model.forward(input))
+model.backward(input, gradOutput)
+model.updateParameters(1)
+println("fc2 weight \n", fc2.element.parameters()._1(0))
+```
+```
+(output1:
+, {
+	2: 0.0
+	   0.0
+	   [com.intel.analytics.bigdl.tensor.DenseTensor of size 2]
+	1: 2.8
+	   2.8
+	   [com.intel.analytics.bigdl.tensor.DenseTensor of size 2]
+ })
+(fc2 weight
+,1.9	1.8	2.3	2.4
+1.8	1.6	2.6	2.8
+[com.intel.analytics.bigdl.tensor.DenseTensor of size 2x4])
+```
+
+"Freeze" ```fc2```, the parameters of ```fc2``` is not changed.
+```scala
+fc1.element.getParameters()._1.apply1(_ => 1.0f)
+fc2.element.getParameters()._1.apply1(_ => 2.0f)
+model.zeroGradParameters()
+model.freeze("fc2")
+println("output2: \n", model.forward(input))
+model.backward(input, gradOutput)
+model.updateParameters(1)
+println("fc2 weight \n", fc2.element.parameters()._1(0))
+```
+
+```
+(output2:
+, {
+	2: 0.0
+	   0.0
+	   [com.intel.analytics.bigdl.tensor.DenseTensor of size 2]
+	1: 2.8
+	   2.8
+	   [com.intel.analytics.bigdl.tensor.DenseTensor of size 2]
+ })
+(fc2 weight
+,2.0	2.0	2.0	2.0
+2.0	2.0	2.0	2.0
+[com.intel.analytics.bigdl.tensor.DenseTensor of size 2x4])
+```
+"unFreeze" ```fc2```, the parameters of ```fc2``` will be updated.
+```scala
+fc1.element.getParameters()._1.apply1(_ => 1.0f)
+fc2.element.getParameters()._1.apply1(_ => 2.0f)
+model.zeroGradParameters()
+model.unFreeze()
+println("output3: \n", model.forward(input))
+model.backward(input, gradOutput)
+model.updateParameters(1)
+println("fc2 weight \n", fc2.element.parameters()._1(0))
+```
+```
+(output3:
+, {
+	2: 0.0
+	   0.0
+	   [com.intel.analytics.bigdl.tensor.DenseTensor of size 2]
+	1: 2.8
+	   2.8
+	   [com.intel.analytics.bigdl.tensor.DenseTensor of size 2]
+ })
+(fc2 weight
+,1.9	1.8	2.3	2.4
+1.8	1.6	2.6	2.8
+[com.intel.analytics.bigdl.tensor.DenseTensor of size 2x4])
+```
diff --git a/docs/readthedocs/source/doc/DLlib/Overview/getting-started.md b/docs/readthedocs/source/doc/DLlib/Overview/getting-started.md
new file mode 100644
index 00000000..1aea760c
--- /dev/null
+++ b/docs/readthedocs/source/doc/DLlib/Overview/getting-started.md
@@ -0,0 +1,281 @@
+# DLLib Getting Start Guide
+
+## 1. Creating dev environment
+
+#### Scala project (maven & sbt)
+
+- **Maven**
+
+To use BigDL DLLib to build your own deep learning application, you can use maven to create your project and add bigdl-dllib to your dependency. Please add below code to your pom.xml to add BigDL DLLib as your dependency:
+```
+<dependency>
+    <groupId>com.intel.analytics.bigdl</groupId>
+    <artifactId>bigdl-dllib-spark_2.4.6</artifactId>
+    <version>0.14.0</version>
+</dependency>
+```
+
+- **SBT**
+```
+libraryDependencies += "com.intel.analytics.bigdl" % "bigdl-dllib-spark_2.4.6" % "0.14.0"
+```
+For more information about how to add BigDL dependency, please refer https://bigdl.readthedocs.io/en/latest/doc/UserGuide/scala.html#build-a-scala-project
+
+#### IDE (Intelij)
+Open up IntelliJ and click File => Open
+
+Navigate to your project. If you have add BigDL DLLib as dependency in your pom.xml.
+The IDE will automatically download it from maven and you are able to run your application.
+
+For more details about how to setup IDE for BigDL project, please refer https://bigdl-project.github.io/master/#ScalaUserGuide/install-build-src/#setup-ide
+
+
+## 2. Code initialization
+```NNContext``` is the main entry for provisioning the dllib program on the underlying cluster (such as K8s or Hadoop cluster), or just on a single laptop.
+
+It is recommended to initialize `NNContext` at the beginning of your program:
+```
+import com.intel.analytics.bigdl.dllib.NNContext
+import com.intel.analytics.bigdl.dllib.keras.Model
+import com.intel.analytics.bigdl.dllib.keras.models.Models
+import com.intel.analytics.bigdl.dllib.keras.optimizers.Adam
+import com.intel.analytics.bigdl.dllib.nn.ClassNLLCriterion
+import com.intel.analytics.bigdl.dllib.utils.Shape
+import com.intel.analytics.bigdl.dllib.keras.layers._
+import com.intel.analytics.bigdl.numeric.NumericFloat
+import org.apache.spark.ml.feature.VectorAssembler
+import org.apache.spark.sql.SQLContext
+import org.apache.spark.sql.functions._
+import org.apache.spark.sql.types.DoubleType
+
+val sc = NNContext.initNNContext("dllib_demo")
+```
+For more information about ```NNContext```, please refer to [NNContext](https://bigdl.readthedocs.io/en/latest/doc/DLlib/Overview/dllib.html#nn-context)
+
+## 3. Distributed Data Loading
+
+#### Using Spark Dataframe APIs
+DLlib supports Spark Dataframes as the input to the distributed training, and as
+the input/output of the distributed inference. Consequently, the user can easily
+process large-scale dataset using Apache Spark, and directly apply AI models on
+the distributed (and possibly in-memory) Dataframes without data conversion or serialization
+
+We used [Pima Indians onset of diabetes](https://raw.githubusercontent.com/jbrownlee/Datasets/master/pima-indians-diabetes.data.csv) as dataset for the demo. It's a standard machine learning dataset from the UCI Machine Learning repository. It describes patient medical record data for Pima Indians and whether they had an onset of diabetes within five years.
+The dataset can be download with:
+```
+wget https://raw.githubusercontent.com/jbrownlee/Datasets/master/pima-indians-diabetes.data.csv
+```
+
+We create Spark session so we can use Spark API to load and process the data
+```
+val spark = new SQLContext(sc)
+```
+
+Load the data into Spark DataFrame
+```
+val path = "pima-indians-diabetes.data.csv"
+val df = spark.read.options(Map("inferSchema"->"true","delimiter"->",")).csv(path)
+      .toDF("num_times_pregrant", "plasma_glucose", "blood_pressure", "skin_fold_thickness", "2-hour_insulin", "body_mass_index", "diabetes_pedigree_function", "age", "class")
+```
+
+## 4. Model Definition
+
+#### Using Keras-like APIs
+
+To define a model, you can use the [Keras Style API](https://bigdl.readthedocs.io/en/latest/doc/DLlib/Overview/keras-api.html).
+```
+val x1 = Input(Shape(8))
+val dense1 = Dense(12, activation="relu").inputs(x1)
+val dense2 = Dense(8, activation="relu").inputs(dense1)
+val dense3 = Dense(2).inputs(dense2)
+val dmodel = Model(x1, dense3)
+```
+
+After creating the model, you will have to decide which loss function to use in training.
+
+Now you can use `compile` function of the model to set the loss function, optimization method.
+```
+dmodel.compile(optimizer = new Adam(),
+  loss = ClassNLLCriterion())
+```
+
+Now the model is built and ready to train.
+
+## 5. Distributed Model Training
+Now you can use 'fit' begin the training, please set the feature columns and label columns. Model Evaluation can be performed periodically during a training.
+If the model accepts single input(eg. column `feature1`) and single output(eg. column `label`), please set the feature columns of the model as :
+```
+model.fit(x=dataframe, batchSize=4, nbEpoch = 2,
+  featureCols = Array("feature1"), labelCols = Array("label"))
+```
+
+If the feature column for the model is a Spark ML Vector. Please assemble related columns into a Vector and pass it to the model. eg.
+```
+val assembler = new VectorAssembler()
+  .setInputCols(Array("num_times_pregrant", "plasma_glucose", "blood_pressure", "skin_fold_thickness", "2-hour_insulin", "body_mass_index", "diabetes_pedigree_function", "age"))
+  .setOutputCol("features")
+val assembleredDF = assembler.transform(df)
+val df2 = assembleredDF.withColumn("label",col("class").cast(DoubleType) + lit(1))
+```
+
+If your model accepts multiple inputs(eg. column `f1`, `f2`, `f3`), please set the features as below:
+```
+model.fit(x=dataframe, batchSize=4, nbEpoch = 2,
+  featureCols = Array("f1", "f2", "f3"), labelCols = Array("label"))
+```
+If one of the inputs is a Spark ML Vector, please assemble it before pass the data to the model.
+
+Similarly, if the model accepts multiple outputs(eg. column `label1`, `label2`), please set the label columns as below:
+```
+model.fit(x=dataframe, batchSize=4, nbEpoch = 2,
+  featureCols = Array("f1", "f2", "f3"), labelCols = Array("label1", "label2"))
+```
+
+Then split it into traing part and validation part
+```
+val Array(trainDF, valDF) = df2.randomSplit(Array(0.8, 0.2))
+```
+
+The model is ready to train.
+```
+dmodel.fit(x=trainDF, batchSize=4, nbEpoch = 2,
+  featureCols = Array("features"), labelCols = Array("label"), valX = valDF
+)
+```
+
+## 6. Model saving and loading
+When training is finished, you may need to save the final model for later use.
+
+BigDL allows you to save your BigDL model on local filesystem, HDFS, or Amazon s3.
+- **save**
+```
+val modelPath = "/tmp/demo/keras.model"
+dmodel.saveModel(modelPath)
+```
+
+- **load**
+```
+val loadModel = Models.loadModel(modelPath)
+
+val preDF2 = loadModel.predict(valDF, featureCols = Array("features"), predictionCol = "predict")
+```
+
+You may want to refer [Save/Load](https://bigdl.readthedocs.io/en/latest/doc/DLlib/Overview/keras-api.html#save)
+
+## 7. Distributed evaluation and inference
+After training finishes, you can then use the trained model for prediction or evaluation.
+
+- **inference**
+```
+dmodel.predict(trainDF, featureCols = Array("features"), predictionCol = "predict")
+```
+
+- **evaluation**
+```
+dmodel.evaluate(trainDF, batchSize = 4, featureCols = Array("features"),
+  labelCols = Array("label"))
+```
+
+## 8. Checkpointing and resuming training
+You can configure periodically taking snapshots of the model.
+```
+val cpPath = "/tmp/demo/cp"
+dmodel.setCheckpoint(cpPath, overWrite=false)
+```
+You can also set ```overWrite``` to ```true``` to enable overwriting any existing snapshot files
+
+After training stops, you can resume from any saved point. Choose one of the model snapshots to resume (saved in checkpoint path, details see Checkpointing). Use Models.loadModel to load the model snapshot into an model object.
+```
+val loadModel = Models.loadModel(path)
+```
+
+## 9. Monitor your training
+
+- **Tensorboard**
+
+BigDL provides a convenient way to monitor/visualize your training progress. It writes the statistics collected during training/validation. Saved summary can be viewed via TensorBoard.
+
+In order to take effect, it needs to be called before fit.
+```
+dmodel.setTensorBoard("./", "dllib_demo")
+```
+For more details, please refer [visulization](visualization.md)
+
+## 10. Transfer learning and finetuning
+
+- **freeze and trainable**
+BigDL DLLib supports exclude some layers of model from training.
+```
+dmodel.freeze(layer_names)
+```
+Layers that match the given names will be freezed. If a layer is freezed, its parameters(weight/bias, if exists) are not changed in training process.
+
+BigDL DLLib also support unFreeze operations. The parameters for the layers that match the given names will be trained(updated) in training process
+```
+dmodel.unFreeze(layer_names)
+```
+For more information, you may refer [freeze](freeze.md)
+
+## 11. Hyperparameter tuning
+- **optimizer**
+
+DLLib supports a list of optimization methods.
+For more details, please refer [optimization](optim-Methods.md)
+
+- **learning rate scheduler**
+
+DLLib supports a list of learning rate scheduler.
+For more details, please refer [lr_scheduler](learningrate-Scheduler.md)
+
+- **batch size**
+
+DLLib supports set batch size during training and prediction. We can adjust the batch size to tune the model's accuracy.
+
+- **regularizer**
+
+DLLib supports a list of regularizers.
+For more details, please refer [regularizer](regularizers.md)
+
+- **clipping**
+
+DLLib supports gradient clipping operations.
+For more details, please refer [gradient_clip](clipping.md)
+
+## 12. Running program
+You can run a bigdl-dllib program as a standard Spark program (running on either a local machine or a distributed cluster) as follows:
+```
+# Spark local mode
+${BIGDL_HOME}/bin/spark-submit-with-dllib.sh \
+  --master local[2] \
+  --class class_name \
+  jar_path
+
+# Spark standalone mode
+## ${SPARK_HOME}/sbin/start-master.sh
+## check master URL from http://localhost:8080
+${BIGDL_HOME}/bin/spark-submit-with-dllib.sh \
+  --master spark://... \
+  --executor-cores cores_per_executor \
+  --total-executor-cores total_cores_for_the_job \
+  --class class_name \
+  jar_path
+
+# Spark yarn client mode
+${BIGDL_HOME}/bin/spark-submit-with-dllib.sh \
+ --master yarn \
+ --deploy-mode client \
+ --executor-cores cores_per_executor \
+ --num-executors executors_number \
+ --class class_name \
+ jar_path
+
+# Spark yarn cluster mode
+${BIGDL_HOME}/bin/spark-submit-with-dllib.sh \
+ --master yarn \
+ --deploy-mode cluster \
+ --executor-cores cores_per_executor \
+ --num-executors executors_number \
+ --class class_name
+ jar_path
+```
+For more detail about how to run BigDL scala application, please refer https://bigdl.readthedocs.io/en/latest/doc/UserGuide/scala.html
diff --git a/docs/readthedocs/source/doc/DLlib/Overview/learningrate-Scheduler.md b/docs/readthedocs/source/doc/DLlib/Overview/learningrate-Scheduler.md
new file mode 100644
index 00000000..c714b42e
--- /dev/null
+++ b/docs/readthedocs/source/doc/DLlib/Overview/learningrate-Scheduler.md
@@ -0,0 +1,436 @@
+## Poly ##
+
+**Scala:**
+```scala
+val lrScheduler = Poly(power=0.5, maxIteration=1000)
+```
+**Python:**
+```python
+lr_scheduler = Poly(power=0.5, max_iteration=1000, bigdl_type="float")
+```
+
+A learning rate decay policy, where the effective learning rate follows a polynomial decay, to be zero by the max_iteration. Calculation: base_lr (1 - iter/maxIteration) `^` (power)
+
+ `power` coeffient of decay, refer to calculation formula
+
+ `maxIteration` max iteration when lr becomes zero
+
+**Scala example:**
+```scala
+import com.intel.analytics.bigdl.dllib.optim.SGD._
+import com.intel.analytics.bigdl.dllib.optim._
+import com.intel.analytics.bigdl.dllib.tensor.{Storage, Tensor}
+import com.intel.analytics.bigdl.dllib.tensor.TensorNumericMath.TensorNumeric.NumericFloat
+import com.intel.analytics.bigdl.dllib.utils.T
+
+val optimMethod = new SGD[Double](0.1)
+optimMethod.learningRateSchedule = Poly(3, 100)
+def feval(x: Tensor[Double]): (Double, Tensor[Double]) = {
+  return (0.1, Tensor[Double](Storage(Array(1.0, 1.0))))
+}
+val x = Tensor[Double](Storage(Array(10.0, 10.0)))
+optimMethod.optimize(feval, x)
+> print(optimMethod.learningRateSchedule.currentRate)
+-0.1
+optimMethod.optimize(feval, x)
+> print(optimMethod.learningRateSchedule.currentRate)
+-0.0970299
+```
+**Python example:**
+```python
+optim_method = SGD(0.1)
+optimMethod.learningRateSchedule = Poly(3, 100)
+```
+
+## Default ##
+
+It is the default learning rate schedule. For each iteration, the learning rate would update with the following formula:
+ l_{n + 1} = l / (1 + n * learning_rate_decay) where `l` is the initial learning rate
+
+**Scala:**
+```scala
+val lrScheduler = Default()
+```
+**Python:**
+```python
+lr_scheduler = Default()
+```
+
+**Scala example:**
+```scala
+val optimMethod = new SGD[Double](0.1, 0.1)
+def feval(x: Tensor[Double]): (Double, Tensor[Double]) = {
+  return (0.1, Tensor[Double](Storage(Array(1.0, 1.0))))
+}
+val x = Tensor[Double](Storage(Array(10.0, 10.0)))
+optimMethod.optimize(feval, x)
+> print(optimMethod.learningRateSchedule.currentRate)
+-0.1
+optimMethod.optimize(feval, x)
+> print(optimMethod.learningRateSchedule.currentRate)
+-0.09090909090909091
+optimMethod.optimize(feval, x)
+> print(optimMethod.learningRateSchedule.currentRate)
+-0.08333333333333334
+```
+
+**Python example:**
+```python
+optimMethod = SGD(leaningrate_schedule=Default())
+```
+
+## NaturalExp ##
+
+A learning rate schedule, which rescale the learning rate by exp ( -decay_rate * iter / decay_step ) referring to tensorflow's learning rate decay # natural_exp_decay
+
+ `decay_step` how often to apply decay
+
+ `gamma` the decay rate. e.g. 0.96
+
+**Scala:**
+```scala
+val learningRateScheduler = NaturalExp(1, 1)
+```
+
+**Scala example:**
+```scala
+val optimMethod = new SGD[Double](0.1)
+optimMethod.learningRateSchedule = NaturalExp(1, 1)
+def feval(x: Tensor[Double]): (Double, Tensor[Double]) = {
+  (0.1, Tensor[Double](Storage(Array(1.0, 1.0))))
+}
+val x = Tensor[Double](Storage(Array(10.0, 10.0)))
+val state = T("epoch" -> 0, "evalCounter" -> 0)
+optimMethod.state = state
+optimMethod.optimize(feval, x)
+> print(optimMethod.learningRateSchedule.currentRate)
+-0.1
+
+optimMethod.optimize(feval, x)
+> print(optimMethod.learningRateSchedule.currentRate)
+-0.036787944117144235
+
+optimMethod.optimize(feval, x)
+> print(optimMethod.learningRateSchedule.currentRate)
+-0.013533528323661271
+```
+
+## Exponential ##
+
+A learning rate schedule, which rescale the learning rate by lr_{n + 1} = lr * decayRate `^` (iter / decayStep)
+
+ `decayStep` the inteval for lr decay
+
+ `decayRate` decay rate
+
+ `stairCase` if true, iter / decayStep is an integer division and the decayed learning rate follows a staircase function.
+
+**Scala:**
+```scala
+val learningRateSchedule = Exponential(10, 0.96)
+```
+
+**Python:**
+```python
+exponential = Exponential(100, 0.1)
+```
+
+**Scala example:**
+```scala
+val optimMethod = new SGD[Double](0.05)
+optimMethod.learningRateSchedule = Exponential(10, 0.96)
+def feval(x: Tensor[Double]): (Double, Tensor[Double]) = {
+  (0.1, Tensor[Double](Storage(Array(1.0, 1.0))))
+}
+val x = Tensor[Double](Storage(Array(10.0, 10.0)))
+val state = T("epoch" -> 0, "evalCounter" -> 0)
+optimMethod.state = state
+optimMethod.optimize(feval, x)
+> print(optimMethod.learningRateSchedule.currentRate)
+-0.05
+
+optimMethod.optimize(feval, x)
+> print(optimMethod.learningRateSchedule.currentRate)
+-0.049796306069892535
+```
+
+**Python example:**
+```python
+optimMethod = SGD(leaningrate_schedule=Exponential(100, 0.1))
+```
+
+## Plateau ##
+
+Plateau is the learning rate schedule when a metric has stopped improving. Models often benefit from reducing the learning rate by a factor of 2-10 once learning stagnates. It monitors a quantity and if no improvement is seen for a 'patience' number of epochs, the learning rate is reduced.
+
+ `monitor` quantity to be monitored, can be Loss or score
+
+ `factor` factor by which the learning rate will be reduced. new_lr = lr * factor
+
+ `patience` number of epochs with no improvement after which learning rate will be reduced.
+
+ `mode` one of {min, max}. In min mode, lr will be reduced when the quantity monitored has stopped decreasing;
+ in max mode it will be reduced when the quantity monitored has stopped increasing
+
+ `epsilon` threshold for measuring the new optimum, to only focus on significant changes.
+
+ `cooldown` number of epochs to wait before resuming normal operation after lr has been reduced.
+
+ `minLr` lower bound on the learning rate.
+
+**Scala:**
+```scala
+val learningRateSchedule = Plateau(monitor="score", factor=0.1, patience=10, mode="min", epsilon=1e-4f, cooldown=0, minLr=0)
+```
+
+**Python:**
+```python
+plateau = Plateau("score", factor=0.1, patience=10, mode="min", epsilon=1e-4, cooldown=0, minLr=0)
+```
+
+**Scala example:**
+```scala
+val optimMethod = new SGD[Double](0.05)
+optimMethod.learningRateSchedule = Plateau(monitor="score", factor=0.1, patience=10, mode="min", epsilon=1e-4f, cooldown=0, minLr=0)
+def feval(x: Tensor[Double]): (Double, Tensor[Double]) = {
+  (0.1, Tensor[Double](Storage(Array(1.0, 1.0))))
+}
+val x = Tensor[Double](Storage(Array(10.0, 10.0)))
+val state = T("epoch" -> 0, "evalCounter" -> 0)
+optimMethod.state = state
+optimMethod.optimize(feval, x)
+> print(optimMethod.learningRateSchedule.currentRate)
+
+
+optimMethod.optimize(feval, x)
+> print(optimMethod.learningRateSchedule.currentRate)
+
+```
+
+**Python example:**
+```python
+optimMethod = SGD(leaningrate_schedule=Plateau("score"))
+```
+
+## Warmup ##
+
+A learning rate gradual increase policy, where the effective learning rate increase delta after each iteration. Calculation: base_lr + delta * iteration
+
+ `delta` increase amount after each iteration
+
+**Scala:**
+```scala
+val learningRateSchedule = Warmup(delta = 0.05)
+```
+
+**Python:**
+```python
+warmup = Warmup(delta=0.05)
+```
+
+**Scala example:**
+```scala
+val lrSchedules = new SequentialSchedule(100)
+lrSchedules.add(Warmup(0.3), 3).add(Poly(3, 100), 100)
+val optimMethod = new SGD[Double](learningRate = 0.1, learningRateSchedule = lrSchedules)
+
+def feval(x: Tensor[Double]): (Double, Tensor[Double]) = {
+  return (0.1, Tensor[Double](Storage(Array(1.0, 1.0))))
+}
+val x = Tensor[Double](Storage(Array(10.0, 10.0)))
+optimMethod.optimize(feval, x)
+> print(optimMethod.learningRateSchedule.currentRate)
+-0.1
+
+optimMethod.optimize(feval, x)
+> print(optimMethod.learningRateSchedule.currentRate)
+-0.4
+```
+
+**Python example:**
+```python
+optimMethod = SGD(leaningrate_schedule=Warmup(0.05))
+```
+
+## SequentialSchedule ##
+
+A learning rate scheduler which can stack several learning rate schedulers.
+
+ `iterationPerEpoch` iteration numbers per epoch
+
+**Scala:**
+```scala
+val learningRateSchedule = SequentialSchedule(iterationPerEpoch=100)
+```
+
+**Python:**
+```python
+sequentialSchedule = SequentialSchedule(iteration_per_epoch=5)
+```
+
+**Scala example:**
+```scala
+val lrSchedules = new SequentialSchedule(100)
+lrSchedules.add(Warmup(0.3), 3).add(Poly(3, 100), 100)
+val optimMethod = new SGD[Double](learningRate = 0.1, learningRateSchedule = lrSchedules)
+
+def feval(x: Tensor[Double]): (Double, Tensor[Double]) = {
+  return (0.1, Tensor[Double](Storage(Array(1.0, 1.0))))
+}
+val x = Tensor[Double](Storage(Array(10.0, 10.0)))
+optimMethod.optimize(feval, x)
+> print(optimMethod.learningRateSchedule.currentRate)
+-0.1
+
+optimMethod.optimize(feval, x)
+> print(optimMethod.learningRateSchedule.currentRate)
+-0.4
+
+optimMethod.optimize(feval, x)
+> print(optimMethod.learningRateSchedule.currentRate)
+-0.7
+
+optimMethod.optimize(feval, x)
+> print(optimMethod.learningRateSchedule.currentRate)
+-1.0
+
+optimMethod.optimize(feval, x)
+> print(optimMethod.learningRateSchedule.currentRate)
+-0.9702989999999999
+```
+
+**Python example:**
+```python
+sequentialSchedule = SequentialSchedule(5)
+poly = Poly(0.5, 2)
+sequentialSchedule.add(poly, 5)
+```
+
+## EpochDecay ##
+
+**Scala:**
+```scala
+def decay(epoch: Int): Double =
+  if (epoch >= 1) 2.0 else if (epoch >= 2) 1.0 else 0.0
+
+val learningRateSchedule = EpochDecay(decay)
+```
+
+It is an epoch decay learning rate schedule. The learning rate decays through a function argument on number of run epochs l_{n + 1} = l_{n} * 0.1 `^` decayType(epoch)
+
+ `decayType` is a function with number of run epochs as the argument
+
+**Scala example:**
+```scala
+def decay(epoch: Int): Double =
+  if (epoch == 1) 2.0 else if (epoch == 2) 1.0 else 0.0
+
+val optimMethod = new SGD[Double](1000)
+optimMethod.learningRateSchedule = EpochDecay(decay)
+def feval(x: Tensor[Double]): (Double, Tensor[Double]) = {
+  return (0.1, Tensor[Double](Storage(Array(1.0, 1.0))))
+}
+val x = Tensor[Double](Storage(Array(10.0, 10.0)))
+val state = T("epoch" -> 0)
+for(e <- 1 to 3) {
+  state("epoch") = e
+  optimMethod.state = state
+  optimMethod.optimize(feval, x)
+  if(e <= 1) {
+    assert(optimMethod.learningRateSchedule.currentRate==10)
+  } else if (e <= 2) {
+    assert(optimMethod.learningRateSchedule.currentRate==100)
+  } else {
+    assert(optimMethod.learningRateSchedule.currentRate==1000)
+  }
+}
+```
+
+## Regime ##
+
+A structure to specify hyper parameters by start epoch and end epoch. Usually work with [[EpochSchedule]].
+
+ `startEpoch` start epoch
+
+ `endEpoch` end epoch
+
+ `config` config table contains hyper parameters
+
+## EpochSchedule ##
+
+A learning rate schedule which configure the learning rate according to some pre-defined [[Regime]]. If the running epoch is within the interval of a regime `r` [r.startEpoch, r.endEpoch], then the learning
+ rate will take the "learningRate" in r.config.
+
+ `regimes` an array of pre-defined [[Regime]].
+
+**Scala:**
+```scala
+val regimes: Array[Regime] = Array(
+  Regime(1, 3, T("learningRate" -> 1e-2, "weightDecay" -> 2e-4)),
+  Regime(4, 7, T("learningRate" -> 5e-3, "weightDecay" -> 2e-4)),
+  Regime(8, 10, T("learningRate" -> 1e-3, "weightDecay" -> 0.0))
+)
+val learningRateScheduler = EpochSchedule(regimes)
+```
+
+**Scala example:**
+```scala
+val regimes: Array[Regime] = Array(
+  Regime(1, 3, T("learningRate" -> 1e-2, "weightDecay" -> 2e-4)),
+  Regime(4, 7, T("learningRate" -> 5e-3, "weightDecay" -> 2e-4)),
+  Regime(8, 10, T("learningRate" -> 1e-3, "weightDecay" -> 0.0))
+)
+
+val state = T("epoch" -> 0)
+val optimMethod = new SGD[Double](0.1)
+optimMethod.learningRateSchedule = EpochSchedule(regimes)
+def feval(x: Tensor[Double]): (Double, Tensor[Double]) = {
+  return (0.1, Tensor[Double](Storage(Array(1.0, 1.0))))
+}
+val x = Tensor[Double](Storage(Array(10.0, 10.0)))
+for(e <- 1 to 10) {
+  state("epoch") = e
+  optimMethod.state = state
+  optimMethod.optimize(feval, x)
+  if(e <= 3) {
+    assert(optimMethod.learningRateSchedule.currentRate==-1e-2)
+    assert(optimMethod.weightDecay==2e-4)
+  } else if (e <= 7) {
+    assert(optimMethod.learningRateSchedule.currentRate==-5e-3)
+    assert(optimMethod.weightDecay==2e-4)
+  } else if (e <= 10) {
+    assert(optimMethod.learningRateSchedule.currentRate==-1e-3)
+    assert(optimMethod.weightDecay==0.0)
+  }
+}
+```
+
+## EpochStep ##
+
+A learning rate schedule which rescale the learning rate by `gamma` for each `stepSize` epochs.
+
+ `stepSize` For how many epochs to update the learning rate once
+
+ `gamma` the rescale factor
+
+ **Scala:**
+ ```scala
+ val learningRateScheduler = EpochStep(1, 0.5)
+ ```
+
+ **Scala example:**
+ ```scala
+ val optimMethod = new SGD[Double](0.1)
+ optimMethod.learningRateSchedule = EpochStep(1, 0.5)
+ def feval(x: Tensor[Double]): (Double, Tensor[Double]) = {
+   (0.1, Tensor[Double](Storage(Array(1.0, 1.0))))
+ }
+ val x = Tensor[Double](Storage(Array(10.0, 10.0)))
+ val state = T("epoch" -> 0)
+ for(e <- 1 to 10) {
+   state("epoch") = e
+   optimMethod.state = state
+   optimMethod.optimize(feval, x)
+   assert(optimMethod.learningRateSchedule.currentRate==(-0.1 * Math.pow(0.5, e)))
+ }
+ ```
diff --git a/docs/readthedocs/source/doc/DLlib/Overview/optim-Methods.md b/docs/readthedocs/source/doc/DLlib/Overview/optim-Methods.md
new file mode 100644
index 00000000..0330e5fb
--- /dev/null
+++ b/docs/readthedocs/source/doc/DLlib/Overview/optim-Methods.md
@@ -0,0 +1,376 @@
+## Adam ##
+
+**Scala:**
+```scala
+val optim = new Adam(learningRate=1e-3, learningRateDecay=0.0, beta1=0.9, beta2=0.999, Epsilon=1e-8)
+```
+**Python:**
+```python
+optim = Adam(learningrate=1e-3, learningrate_decay=0.0, beta1=0.9, beta2=0.999, epsilon=1e-8, bigdl_type="float")
+```
+
+An implementation of Adam optimization, first-order gradient-based optimization of stochastic  objective  functions. http://arxiv.org/pdf/1412.6980.pdf
+
+ `learningRate` learning rate. Default value is 1e-3. 
+ 
+ `learningRateDecay` learning rate decay. Default value is 0.0.
+ 
+ `beta1` first moment coefficient. Default value is 0.9.
+ 
+ `beta2` second moment coefficient. Default value is 0.999.
+ 
+ `Epsilon` for numerical stability. Default value is 1e-8.
+ 
+
+**Scala example:**
+```scala
+import com.intel.analytics.bigdl.dllib.optim._
+import com.intel.analytics.bigdl.dllib.tensor.Tensor
+import com.intel.analytics.bigdl.dllib.tensor.TensorNumericMath.TensorNumeric.NumericFloat
+import com.intel.analytics.bigdl.dllib.utils.T
+
+val optm = new Adam(learningRate=0.002)
+def rosenBrock(x: Tensor[Float]): (Float, Tensor[Float]) = {
+    // (1) compute f(x)
+    val d = x.size(1)
+
+    // x1 = x(i)
+    val x1 = Tensor[Float](d - 1).copy(x.narrow(1, 1, d - 1))
+    // x(i + 1) - x(i)^2
+    x1.cmul(x1).mul(-1).add(x.narrow(1, 2, d - 1))
+    // 100 * (x(i + 1) - x(i)^2)^2
+    x1.cmul(x1).mul(100)
+
+    // x0 = x(i)
+    val x0 = Tensor[Float](d - 1).copy(x.narrow(1, 1, d - 1))
+    // 1-x(i)
+    x0.mul(-1).add(1)
+    x0.cmul(x0)
+    // 100*(x(i+1) - x(i)^2)^2 + (1-x(i))^2
+    x1.add(x0)
+
+    val fout = x1.sum()
+
+    // (2) compute f(x)/dx
+    val dxout = Tensor[Float]().resizeAs(x).zero()
+    // df(1:D-1) = - 400*x(1:D-1).*(x(2:D)-x(1:D-1).^2) - 2*(1-x(1:D-1));
+    x1.copy(x.narrow(1, 1, d - 1))
+    x1.cmul(x1).mul(-1).add(x.narrow(1, 2, d - 1)).cmul(x.narrow(1, 1, d - 1)).mul(-400)
+    x0.copy(x.narrow(1, 1, d - 1)).mul(-1).add(1).mul(-2)
+    x1.add(x0)
+    dxout.narrow(1, 1, d - 1).copy(x1)
+
+    // df(2:D) = df(2:D) + 200*(x(2:D)-x(1:D-1).^2);
+    x0.copy(x.narrow(1, 1, d - 1))
+    x0.cmul(x0).mul(-1).add(x.narrow(1, 2, d - 1)).mul(200)
+    dxout.narrow(1, 2, d - 1).add(x0)
+
+    (fout, dxout)
+  }  
+val x = Tensor(2).fill(0)
+> print(optm.optimize(rosenBrock, x))
+(0.0019999996
+0.0
+[com.intel.analytics.bigdl.tensor.DenseTensor$mcD$sp of size 2],[D@302d88d8)
+```
+**Python example:**
+```python
+optim_method = Adam(learningrate=0.002)
+                  
+optimizer = Optimizer(
+    model=mlp_model,
+    training_rdd=train_data,
+    criterion=ClassNLLCriterion(),
+    optim_method=optim_method,
+    end_trigger=MaxEpoch(20),
+    batch_size=32)
+
+```
+## SGD ##
+
+**Scala:**
+```scala
+val optimMethod = new SGD(learningRate= 1e-3,learningRateDecay=0.0,
+                      weightDecay=0.0,momentum=0.0,dampening=Double.MaxValue,
+                      nesterov=false,learningRateSchedule=Default(),
+                      learningRates=null,weightDecays=null)
+```
+
+**Python:**
+```python
+optim_method = SGD(learningrate=1e-3,learningrate_decay=0.0,weightdecay=0.0,
+                   momentum=0.0,dampening=DOUBLEMAX,nesterov=False,
+                   leaningrate_schedule=None,learningrates=None,
+                   weightdecays=None,bigdl_type="float")
+```
+
+A plain implementation of SGD which provides optimize method. After setting 
+optimization method when create Optimize, Optimize will call optimization method at the end of 
+each iteration.
+ 
+**Scala example:**
+```scala
+val optimMethod = new SGD[Float](learningRate= 1e-3,learningRateDecay=0.0,
+                               weightDecay=0.0,momentum=0.0,dampening=Double.MaxValue,
+                               nesterov=false,learningRateSchedule=Default(),
+                               learningRates=null,weightDecays=null)
+optimizer.setOptimMethod(optimMethod)
+```
+
+**Python example:**
+```python
+optim_method = SGD(learningrate=1e-3,learningrate_decay=0.0,weightdecay=0.0,
+                  momentum=0.0,dampening=DOUBLEMAX,nesterov=False,
+                  leaningrate_schedule=None,learningrates=None,
+                  weightdecays=None,bigdl_type="float")
+                  
+optimizer = Optimizer(
+    model=mlp_model,
+    training_rdd=train_data,
+    criterion=ClassNLLCriterion(),
+    optim_method=optim_method,
+    end_trigger=MaxEpoch(20),
+    batch_size=32)
+```
+
+## Adadelta ##
+
+
+*AdaDelta* implementation for *SGD* 
+It has been proposed in `ADADELTA: An Adaptive Learning Rate Method`.
+http://arxiv.org/abs/1212.5701.
+
+**Scala:**
+```scala
+val optimMethod = new Adadelta(decayRate = 0.9, Epsilon = 1e-10)
+```
+**Python:**
+```python
+optim_method = AdaDelta(decayrate = 0.9, epsilon = 1e-10)
+```
+
+
+**Scala example:**
+```scala
+optimizer.setOptimMethod(new Adadelta(0.9, 1e-10))
+
+```
+
+
+**Python example:**
+```python
+optimizer = Optimizer(
+    model=mlp_model,
+    training_rdd=train_data,
+    criterion=ClassNLLCriterion(),
+    optim_method=Adadelta(0.9, 0.00001),
+    end_trigger=MaxEpoch(20),
+    batch_size=32)
+```
+
+## RMSprop ##
+
+An implementation of RMSprop (Reference: http://arxiv.org/pdf/1308.0850v5.pdf, Sec 4.2)
+
+* learningRate : learning rate
+* learningRateDecay : learning rate decay
+* decayRate : decayRate, also called rho
+* Epsilon : for numerical stability
+
+## Adamax ##
+
+An implementation of Adamax http://arxiv.org/pdf/1412.6980.pdf
+
+Arguments:
+
+* learningRate : learning rate
+* beta1 : first moment coefficient
+* beta2 : second moment coefficient
+* Epsilon : for numerical stability
+
+Returns:
+
+the new x vector and the function list {fx}, evaluated before the update
+
+## Adagrad ##
+
+**Scala:**
+```scala
+val adagrad = new Adagrad(learningRate = 1e-3,
+                          learningRateDecay = 0.0,
+                          weightDecay = 0.0)
+
+```
+
+ An implementation of Adagrad. See the original paper:
+ <http://jmlr.org/papers/volume12/duchi11a/duchi11a.pdf>
+
+**Scala example:**
+```scala
+import com.intel.analytics.bigdl.dllib.tensor.TensorNumericMath.TensorNumeric.NumericFloat
+import com.intel.analytics.bigdl.dllib.optim._
+import com.intel.analytics.bigdl.dllib.tensor._
+import com.intel.analytics.bigdl.dllib.utils.T
+
+val adagrad = new Adagrad(0.01, 0.0, 0.0)
+    def feval(x: Tensor[Float]): (Float, Tensor[Float]) = {
+      // (1) compute f(x)
+      val d = x.size(1)
+      // x1 = x(i)
+      val x1 = Tensor[Float](d - 1).copy(x.narrow(1, 1, d - 1))
+      // x(i + 1) - x(i)^2
+      x1.cmul(x1).mul(-1).add(x.narrow(1, 2, d - 1))
+      // 100 * (x(i + 1) - x(i)^2)^2
+      x1.cmul(x1).mul(100)
+      // x0 = x(i)
+      val x0 = Tensor[Float](d - 1).copy(x.narrow(1, 1, d - 1))
+      // 1-x(i)
+      x0.mul(-1).add(1)
+      x0.cmul(x0)
+      // 100*(x(i+1) - x(i)^2)^2 + (1-x(i))^2
+      x1.add(x0)
+      val fout = x1.sum()
+      // (2) compute f(x)/dx
+      val dxout = Tensor[Float]().resizeAs(x).zero()
+      // df(1:D-1) = - 400*x(1:D-1).*(x(2:D)-x(1:D-1).^2) - 2*(1-x(1:D-1));
+      x1.copy(x.narrow(1, 1, d - 1))
+      x1.cmul(x1).mul(-1).add(x.narrow(1, 2, d - 1)).cmul(x.narrow(1, 1, d - 1)).mul(-400)
+      x0.copy(x.narrow(1, 1, d - 1)).mul(-1).add(1).mul(-2)
+      x1.add(x0)
+      dxout.narrow(1, 1, d - 1).copy(x1)
+      // df(2:D) = df(2:D) + 200*(x(2:D)-x(1:D-1).^2);
+      x0.copy(x.narrow(1, 1, d - 1))
+      x0.cmul(x0).mul(-1).add(x.narrow(1, 2, d - 1)).mul(200)
+      dxout.narrow(1, 2, d - 1).add(x0)
+      (fout, dxout)
+    }
+val x = Tensor(2).fill(0)
+val config = T("learningRate" -> 1e-1)
+for (i <- 1 to 10) {
+  adagrad.optimize(feval, x, config, config)
+}
+x after optimize: 0.27779138
+0.07226955
+[com.intel.analytics.bigdl.tensor.DenseTensor$mcF$sp of size 2]
+```
+
+## LBFGS ##
+
+**Scala:**
+```scala
+val optimMethod = new LBFGS(maxIter=20, maxEval=Double.MaxValue,
+                            tolFun=1e-5, tolX=1e-9, nCorrection=100,
+                            learningRate=1.0, lineSearch=None, lineSearchOptions=None)
+```
+
+**Python:**
+```python
+optim_method = LBFGS(max_iter=20, max_eval=Double.MaxValue, \
+                 tol_fun=1e-5, tol_x=1e-9, n_correction=100, \
+                 learning_rate=1.0, line_search=None, line_search_options=None)
+```
+
+This implementation of L-BFGS relies on a user-provided line search function
+(state.lineSearch). If this function is not provided, then a simple learningRate
+is used to produce fixed size steps. Fixed size steps are much less costly than line
+searches, and can be useful for stochastic problems.
+
+The learning rate is used even when a line search is provided.This is also useful for
+large-scale stochastic problems, where opfunc is a noisy approximation of f(x). In that
+case, the learning rate allows a reduction of confidence in the step size.
+
+**Parameters:**
+
+* maxIter - Maximum number of iterations allowed. Default: 20
+* maxEval - Maximum number of function evaluations. Default: Double.MaxValue
+* tolFun - Termination tolerance on the first-order optimality. Default: 1e-5
+* tolX - Termination tol on progress in terms of func/param changes. Default: 1e-9
+* learningRate - the learning rate. Default: 1.0
+* lineSearch - A line search function. Default: None
+* lineSearchOptions - If no line search provided, then a fixed step size is used. Default: None
+
+**Scala example:**
+```scala
+val optimMethod = new LBFGS(maxIter=20, maxEval=Double.MaxValue,
+                            tolFun=1e-5, tolX=1e-9, nCorrection=100,
+                            learningRate=1.0, lineSearch=None, lineSearchOptions=None)
+optimizer.setOptimMethod(optimMethod)
+```
+
+**Python example:**
+```python
+optim_method = LBFGS(max_iter=20, max_eval=DOUBLEMAX, \
+                 tol_fun=1e-5, tol_x=1e-9, n_correction=100, \
+                 learning_rate=1.0, line_search=None, line_search_options=None)
+                  
+optimizer = Optimizer(
+    model=mlp_model,
+    training_rdd=train_data,
+    criterion=ClassNLLCriterion(),
+    optim_method=optim_method,
+    end_trigger=MaxEpoch(20),
+    batch_size=32)
+```
+
+## Ftrl ##
+
+**Scala:**
+```scala
+val optimMethod = new Ftrl(
+  learningRate = 1e-3, learningRatePower = -0.5,
+  initialAccumulatorValue = 0.1, l1RegularizationStrength = 0.0,
+  l2RegularizationStrength = 0.0, l2ShrinkageRegularizationStrength = 0.0)
+```
+
+**Python:**
+```python
+optim_method = Ftrl(learningrate = 1e-3, learningrate_power = -0.5, \
+                 initial_accumulator_value = 0.1, l1_regularization_strength = 0.0, \
+                 l2_regularization_strength = 0.0, l2_shrinkage_regularization_strength = 0.0)
+```
+
+An implementation of (Ftrl)[https://www.eecs.tufts.edu/~dsculley/papers/ad-click-prediction.pdf.]
+Support L1 penalty, L2 penalty and shrinkage-type L2 penalty.
+
+**Parameters:**
+
+* learningRate: learning rate
+* learningRatePower: double, must be less or equal to zero. Default is -0.5.
+* initialAccumulatorValue: double, the starting value for accumulators, require zero or positive values. Default is 0.1.
+* l1RegularizationStrength: double, must be greater or equal to zero. Default is zero.
+* l2RegularizationStrength: double, must be greater or equal to zero. Default is zero.
+* l2ShrinkageRegularizationStrength: double, must be greater or equal to zero. Default is zero. This differs from l2RegularizationStrength above. L2 above is a stabilization penalty, whereas this one is a magnitude penalty.
+
+**Scala example:**
+```scala
+val optimMethod = new Ftrl(learningRate = 5e-3, learningRatePower = -0.5,
+  initialAccumulatorValue = 0.01)
+optimizer.setOptimMethod(optimMethod)
+```
+
+**Python example:**
+```python
+optim_method = Ftrl(learningrate = 5e-3, \
+    learningrate_power = -0.5, \
+    initial_accumulator_value = 0.01)
+                  
+optimizer = Optimizer(
+    model=mlp_model,
+    training_rdd=train_data,
+    criterion=ClassNLLCriterion(),
+    optim_method=optim_method,
+    end_trigger=MaxEpoch(20),
+    batch_size=32)
+```
+
+## ParallelAdam ##
+Multi-Thread version of [Adam](#adam).
+
+**Scala:**
+```scala
+val optim = new ParallelAdam(learningRate=1e-3, learningRateDecay=0.0, beta1=0.9, beta2=0.999, Epsilon=1e-8, parallelNum=Engine.coreNumber())
+```
+**Python:**
+```python
+optim = ParallelAdam(learningrate=1e-3, learningrate_decay=0.0, beta1=0.9, beta2=0.999, epsilon=1e-8, parallel_num=get_node_and_core_number()[1], bigdl_type="float")
+```
diff --git a/docs/readthedocs/source/doc/DLlib/Overview/regularizers.md b/docs/readthedocs/source/doc/DLlib/Overview/regularizers.md
new file mode 100644
index 00000000..162aa164
--- /dev/null
+++ b/docs/readthedocs/source/doc/DLlib/Overview/regularizers.md
@@ -0,0 +1,266 @@
+## L1 Regularizer ##
+
+**Scala:**
+```scala
+val l1Regularizer = L1Regularizer(rate)
+```
+**Python:**
+```python
+regularizerl1 = L1Regularizer(rate)
+```
+
+L1 regularizer is used to add penalty to the gradWeight to avoid overfitting.
+
+In our code implementation, gradWeight = gradWeight + alpha * abs(weight)
+
+For more details, please refer to [wiki](https://en.wikipedia.org/wiki/Regularization_(mathematics)).
+
+**Scala example:**
+```scala
+
+import com.intel.analytics.bigdl.dllib.utils.RandomGenerator.RNG
+import com.intel.analytics.bigdl.dllib.tensor._
+import com.intel.analytics.bigdl.dllib.optim._
+import com.intel.analytics.bigdl.numeric.NumericFloat
+import com.intel.analytics.bigdl.dllib.nn._
+
+RNG.setSeed(100)
+
+val input = Tensor(3, 5).rand
+val gradOutput = Tensor(3, 5).rand
+val linear = Linear(5, 5, wRegularizer = L1Regularizer(0.2), bRegularizer = L1Regularizer(0.2))
+
+val output = linear.forward(input)
+val gradInput = linear.backward(input, gradOutput)
+
+scala> input
+input: com.intel.analytics.bigdl.tensor.Tensor[Float] =
+0.54340494      0.67115563      0.2783694       0.4120464       0.4245176
+0.52638245      0.84477615      0.14860484      0.004718862     0.15671109
+0.12156912      0.18646719      0.67074907      0.21010774      0.82585275
+[com.intel.analytics.bigdl.tensor.DenseTensor$mcF$sp of size 3x5]
+
+scala> gradOutput
+gradOutput: com.intel.analytics.bigdl.tensor.Tensor[Float] =
+0.4527399       0.13670659      0.87014264      0.5750933       0.063681036
+0.89132196      0.62431186      0.20920213      0.52334774      0.18532822
+0.5622963       0.10837689      0.0058171963    0.21969749      0.3074232
+[com.intel.analytics.bigdl.tensor.DenseTensor$mcF$sp of size 3x5]
+
+scala> linear.gradWeight
+res2: com.intel.analytics.bigdl.tensor.Tensor[Float] =
+0.9835552       1.3616763       0.83564335      0.108898684     0.59625006
+0.21608911      0.8393639       0.0035243928    -0.11795368     0.4453743
+0.38366735      0.9618148       0.47721142      0.5607486       0.6069793
+0.81469804      0.6690552       0.18522228      0.08559488      0.7075894
+-0.030468717    0.056625083     0.051471338     0.2917061       0.109963015
+[com.intel.analytics.bigdl.tensor.DenseTensor of size 5x5]
+
+```
+
+**Python example:**
+```python
+
+from bigdl.dllib.nn.layer import *
+from bigdl.dllib.nn.criterion import *
+from bigdl.dllib.optim.optimizer import *
+from bigdl.dllib.util.common import *
+
+input = np.random.uniform(0, 1, (3, 5)).astype("float32")
+gradOutput = np.random.uniform(0, 1, (3, 5)).astype("float32")
+linear = Linear(5, 5, wRegularizer = L1Regularizer(0.2), bRegularizer = L1Regularizer(0.2))
+output = linear.forward(input)
+gradInput = linear.backward(input, gradOutput)
+
+> linear.parameters()
+{u'Linear@596d857b': {u'bias': array([ 0.3185505 , -0.02004393,  0.34620118, -0.09206461,  0.40776938], dtype=float32),
+  u'gradBias': array([ 2.14087653,  1.82181644,  1.90674937,  1.37307787,  0.81534696], dtype=float32),
+  u'gradWeight': array([[ 0.34909648,  0.85083449,  1.44904375,  0.90150446,  0.57136625],
+         [ 0.3745544 ,  0.42218602,  1.53656614,  1.1836741 ,  1.00702667],
+         [ 0.30529332,  0.26813674,  0.85559171,  0.61224306,  0.34721529],
+         [ 0.22859855,  0.8535381 ,  1.19809723,  1.37248564,  0.50041491],
+         [ 0.36197871,  0.03069445,  0.64837945,  0.12765063,  0.12872688]], dtype=float32),
+  u'weight': array([[-0.12423037,  0.35694697,  0.39038274, -0.34970999, -0.08283543],
+         [-0.4186025 , -0.33235055,  0.34948507,  0.39953214,  0.16294235],
+         [-0.25171402, -0.28955361, -0.32243955, -0.19771226, -0.29320192],
+         [-0.39263198,  0.37766701,  0.14673658,  0.24882999, -0.0779015 ],
+         [ 0.0323218 , -0.31266898,  0.31543773, -0.0898933 , -0.33485892]], dtype=float32)}}
+```
+
+
+
+
+## L2 Regularizer ##
+
+**Scala:**
+```scala
+val l2Regularizer = L2Regularizer(rate)
+```
+**Python:**
+```python
+regularizerl2 = L2Regularizer(rate)
+```
+
+L2 regularizer is used to add penalty to the gradWeight to avoid overfitting.
+
+In our code implementation, gradWeight = gradWeight + alpha * weight * weight
+
+For more details, please refer to [wiki](https://en.wikipedia.org/wiki/Regularization_(mathematics)).
+
+**Scala example:**
+```scala
+
+import com.intel.analytics.bigdl.dllib.utils.RandomGenerator.RNG
+import com.intel.analytics.bigdl.dllib.tensor._
+import com.intel.analytics.bigdl.dllib.optim._
+import com.intel.analytics.bigdl.numeric.NumericFloat
+import com.intel.analytics.bigdl.dllib.nn._
+
+RNG.setSeed(100)
+
+val input = Tensor(3, 5).rand
+val gradOutput = Tensor(3, 5).rand
+val linear = Linear(5, 5, wRegularizer = L2Regularizer(0.2), bRegularizer = L2Regularizer(0.2))
+
+val output = linear.forward(input)
+val gradInput = linear.backward(input, gradOutput)
+
+scala> input
+input: com.intel.analytics.bigdl.tensor.Tensor[Float] =
+0.54340494      0.67115563      0.2783694       0.4120464       0.4245176
+0.52638245      0.84477615      0.14860484      0.004718862     0.15671109
+0.12156912      0.18646719      0.67074907      0.21010774      0.82585275
+[com.intel.analytics.bigdl.tensor.DenseTensor$mcF$sp of size 3x5]
+
+scala> gradOutput
+gradOutput: com.intel.analytics.bigdl.tensor.Tensor[Float] =
+0.4527399       0.13670659      0.87014264      0.5750933       0.063681036
+0.89132196      0.62431186      0.20920213      0.52334774      0.18532822
+0.5622963       0.10837689      0.0058171963    0.21969749      0.3074232
+[com.intel.analytics.bigdl.tensor.DenseTensor$mcF$sp of size 3x5]
+
+scala> linear.gradWeight
+res0: com.intel.analytics.bigdl.tensor.Tensor[Float] =
+1.0329735       0.047239657     0.8979603       0.53614384      1.2781229
+0.5621818       0.29772854      0.69706535      0.30559152      0.8352279
+1.3044653       0.43065858      0.9896795       0.7435816       1.6003494
+0.94218314      0.6793372       0.97101355      0.62892824      1.3458569
+0.73134506      0.5975239       0.9109101       0.59374434      1.1656629
+[com.intel.analytics.bigdl.tensor.DenseTensor of size 5x5]
+
+```
+
+**Python example:**
+```python
+from bigdl.dllib.nn.layer import *
+from bigdl.dllib.nn.criterion import *
+from bigdl.dllib.optim.optimizer import *
+from bigdl.dllib.util.common import *
+
+input = np.random.uniform(0, 1, (3, 5)).astype("float32")
+gradOutput = np.random.uniform(0, 1, (3, 5)).astype("float32")
+linear = Linear(5, 5, wRegularizer = L2Regularizer(0.2), bRegularizer = L2Regularizer(0.2))
+output = linear.forward(input)
+gradInput = linear.backward(input, gradOutput)
+
+> linear.parameters()
+{u'Linear@787aab5e': {u'bias': array([-0.43960261, -0.12444571,  0.22857292, -0.43216187,  0.27770036], dtype=float32),
+  u'gradBias': array([ 0.51726723,  1.32883406,  0.57567948,  1.7791357 ,  1.2887038 ], dtype=float32),
+  u'gradWeight': array([[ 0.45477036,  0.22262168,  0.21923628,  0.26152173,  0.19836383],
+         [ 1.12261093,  0.72921795,  0.08405925,  0.78192139,  0.48798928],
+         [ 0.34581488,  0.21195598,  0.26357424,  0.18987852,  0.2465664 ],
+         [ 1.18659711,  1.11271608,  0.72589797,  1.19098675,  0.33769298],
+         [ 0.82314551,  0.71177536,  0.4428404 ,  0.764337  ,  0.3500182 ]], dtype=float32),
+  u'weight': array([[ 0.03727285, -0.39697152,  0.42733836, -0.34291714, -0.13833708],
+         [ 0.09232076, -0.09720675, -0.33625153,  0.06477787, -0.34739712],
+         [ 0.17145753,  0.10128133,  0.16679128, -0.33541158,  0.40437087],
+         [-0.03005157, -0.36412898,  0.0629965 ,  0.13443278, -0.38414535],
+         [-0.16630849,  0.06934392,  0.40328237,  0.22299488, -0.1178569 ]], dtype=float32)}}
+```
+
+## L1L2 Regularizer ##
+
+**Scala:**
+```scala
+val l1l2Regularizer = L1L2Regularizer(l1rate, l2rate)
+```
+**Python:**
+```python
+regularizerl1l2 = L1L2Regularizer(l1rate, l2rate)
+```
+
+L1L2 regularizer is used to add penalty to the gradWeight to avoid overfitting.
+
+In our code implementation, we will apply L1regularizer and L2regularizer sequentially.
+
+For more details, please refer to [wiki](https://en.wikipedia.org/wiki/Regularization_(mathematics)).
+
+**Scala example:**
+```scala
+
+import com.intel.analytics.bigdl.dllib.utils.RandomGenerator.RNG
+import com.intel.analytics.bigdl.dllib.tensor._
+import com.intel.analytics.bigdl.dllib.optim._
+import com.intel.analytics.bigdl.numeric.NumericFloat
+import com.intel.analytics.bigdl.dllib.nn._
+
+RNG.setSeed(100)
+
+val input = Tensor(3, 5).rand
+val gradOutput = Tensor(3, 5).rand
+val linear = Linear(5, 5, wRegularizer = L1L2Regularizer(0.2, 0.2), bRegularizer = L1L2Regularizer(0.2, 0.2))
+
+val output = linear.forward(input)
+val gradInput = linear.backward(input, gradOutput)
+
+scala> input
+input: com.intel.analytics.bigdl.tensor.Tensor[Float] =
+0.54340494      0.67115563      0.2783694       0.4120464       0.4245176
+0.52638245      0.84477615      0.14860484      0.004718862     0.15671109
+0.12156912      0.18646719      0.67074907      0.21010774      0.82585275
+[com.intel.analytics.bigdl.tensor.DenseTensor$mcF$sp of size 3x5]
+
+scala> gradOutput
+gradOutput: com.intel.analytics.bigdl.tensor.Tensor[Float] =
+0.4527399       0.13670659      0.87014264      0.5750933       0.063681036
+0.89132196      0.62431186      0.20920213      0.52334774      0.18532822
+0.5622963       0.10837689      0.0058171963    0.21969749      0.3074232
+[com.intel.analytics.bigdl.tensor.DenseTensor$mcF$sp of size 3x5]
+
+scala> linear.gradWeight
+res1: com.intel.analytics.bigdl.tensor.Tensor[Float] =
+1.069174        1.4422078       0.8913989       0.042112567     0.53756505
+0.14077617      0.8959319       -0.030221784    -0.1583686      0.4690558
+0.37145022      0.99747723      0.5559263       0.58614403      0.66380215
+0.88983417      0.639738        0.14924419      0.027530536     0.71988696
+-0.053217214    -8.643427E-4    -0.036953792    0.29753304      0.06567569
+[com.intel.analytics.bigdl.tensor.DenseTensor of size 5x5]
+```
+
+**Python example:**
+```python
+from bigdl.dllib.nn.layer import *
+from bigdl.dllib.nn.criterion import *
+from bigdl.dllib.optim.optimizer import *
+from bigdl.dllib.util.common import *
+
+input = np.random.uniform(0, 1, (3, 5)).astype("float32")
+gradOutput = np.random.uniform(0, 1, (3, 5)).astype("float32")
+linear = Linear(5, 5, wRegularizer = L1L2Regularizer(0.2, 0.2), bRegularizer = L1L2Regularizer(0.2, 0.2))
+output = linear.forward(input)
+gradInput = linear.backward(input, gradOutput)
+
+> linear.parameters()
+{u'Linear@1356aa91': {u'bias': array([-0.05799473, -0.0548001 ,  0.00408955, -0.22004321, -0.07143869], dtype=float32),
+  u'gradBias': array([ 0.89119786,  1.09953558,  1.03394508,  1.19511735,  2.02241182], dtype=float32),
+  u'gradWeight': array([[ 0.89061081,  0.58810186, -0.10087357,  0.19108151,  0.60029608],
+         [ 0.95275503,  0.2333075 ,  0.46897018,  0.74429053,  1.16038764],
+         [ 0.22894514,  0.60031962,  0.3836292 ,  0.15895618,  0.83136207],
+         [ 0.49079862,  0.80913013,  0.55491877,  0.69608945,  0.80458677],
+         [ 0.98890561,  0.49226439,  0.14861123,  1.37666655,  1.47615671]], dtype=float32),
+  u'weight': array([[ 0.44654208,  0.16320795, -0.36029238, -0.25365737, -0.41974261],
+         [ 0.18809238, -0.28065765,  0.27677274, -0.29904234,  0.41338971],
+         [-0.03731538,  0.22493915,  0.10021331, -0.19495697,  0.25470355],
+         [-0.30836752,  0.12083009,  0.3773002 ,  0.24059358, -0.40325543],
+         [-0.13601269, -0.39310011, -0.05292636,  0.20001481, -0.08444868]], dtype=float32)}}
+```
diff --git a/docs/readthedocs/source/doc/DLlib/Overview/visualization.md b/docs/readthedocs/source/doc/DLlib/Overview/visualization.md
new file mode 100644
index 00000000..23006116
--- /dev/null
+++ b/docs/readthedocs/source/doc/DLlib/Overview/visualization.md
@@ -0,0 +1,40 @@
+## **Visualizing training with TensorBoard**
+With the summary info generated, we can then use [TensorBoard](https://pypi.python.org/pypi/tensorboard) to visualize the behaviors of the BigDL program.  
+
+* **Installing TensorBoard**
+
+Prerequisites:
+
+1. Python verison: 2.7, 3.4, 3.5, or 3.6
+2. Pip version >= 9.0.1
+
+To install TensorBoard using Python 2, you may run the command:
+```bash
+pip install tensorboard==1.0.0a4
+```
+
+To install TensorBoard using Python 3, you may run the command:
+```bash
+pip3 install tensorboard==1.0.0a4
+```
+
+Please refer to [this page](https://github.com/intel-analytics/BigDL/tree/master/spark/dl/src/main/scala/com/intel/analytics/bigdl/visualization#known-issues) for possible issues when installing TensorBoard.
+
+* **Launching TensorBoard**
+
+You can launch TensorBoard using the command below:
+```bash
+tensorboard --logdir=/tmp/bigdl_summaries
+```
+After that, navigate to the TensorBoard dashboard using a browser. You can find the URL in the console output after TensorBoard is successfully launched; by default the URL is http://your_node:6006
+
+* **Visualizations in TensorBoard**
+
+Within the TensorBoard dashboard, you will be able to read the visualizations of each run, including the “Loss” and “Throughput” curves under the SCALARS tab (as illustrated below):
+![Scalar](../Image/tensorboard-scalar.png)
+
+And “weights”, “bias”, “gradientWeights” and “gradientBias” under the DISTRIBUTIONS and HISTOGRAMS tabs (as illustrated below):
+![histogram1](../Image/tensorboard-histo1.png)
+![histogram2](../Image/tensorboard-histo2.png)
+
+---
\ No newline at end of file