diff --git a/docs/readthedocs/source/doc/UserGuide/databricks.md b/docs/readthedocs/source/doc/UserGuide/databricks.md index 06b22131..44aa5ddd 100644 --- a/docs/readthedocs/source/doc/UserGuide/databricks.md +++ b/docs/readthedocs/source/doc/UserGuide/databricks.md @@ -14,6 +14,9 @@ You can run BigDL program on the [Databricks](https://databricks.com/) cluster a [Init script](https://learn.microsoft.com/en-us/azure/databricks/clusters/init-scripts) is used to Install BigDL or other libraries. First, you need to put the **init script** into [DBFS](https://docs.databricks.com/dbfs/index.html), you can use one of the following ways. +__Note:__ +* To use the Python 3.8 environment (recommended version by BigDL), please set the Databricks runtime version to 9.1 LTS or 10.4 LTS. + **a. Generate init script in Databricks notebook** Create a Databricks notebook and execute @@ -61,7 +64,7 @@ Create a file **init.sh**(or any other filename) in your computer, the file cont cp /databricks/python/lib/python3.8/site-packages/bigdl/share/*/lib/*.jar /databricks/jars ``` -Then upload **init.sh** to DBFS. In Databricks left panel, click **Data > DBFS > Choose or create upload directory > Right click > Upload here**. +Then upload **init.sh** to DBFS. In Databricks left panel, click **Data > DBFS > Choose or create upload directory > Right click > Upload here**. ![](images/upload-init-script.png) @@ -69,6 +72,10 @@ Now the init script is in DBFS, right click the init.sh and choose **Copy path** ![](images/copy-script-path.png) +__Notes:__ +* If Databricks returns an init script failure, please check your Databricks runtime and Python version. +* If your Databricks runtime version is 11.2 or later, click the `Edit` button and change the version to 9.1 LTS or 10.4 LTS. + ### 3. Set Spark configuration In the left panel, click **Compute > Choose your cluster > edit > Advanced options > Spark > Confirm**. You can provide custom [Spark configuration properties](https://spark.apache.org/docs/latest/configuration.html) in a cluster configuration. Please set it according to your cluster resource and program needs.