From 6aeabae58b090d5de79f2b1eff71ab4bbef7b2d1 Mon Sep 17 00:00:00 2001 From: jenniew Date: Sun, 12 Jun 2022 17:06:24 -0700 Subject: [PATCH] Azure PPML support (#4660) * add script * update * update * update scripts and doc * update pass * update * update docker * update * aks script * update create aks * add doc * update --- .../source/doc/PPML/Overview/azure_ppml.md | 98 ++++++++----------- docs/readthedocs/source/index.rst | 1 + 2 files changed, 44 insertions(+), 55 deletions(-) diff --git a/docs/readthedocs/source/doc/PPML/Overview/azure_ppml.md b/docs/readthedocs/source/doc/PPML/Overview/azure_ppml.md index 449743f2..b2780509 100644 --- a/docs/readthedocs/source/doc/PPML/Overview/azure_ppml.md +++ b/docs/readthedocs/source/doc/PPML/Overview/azure_ppml.md @@ -13,23 +13,27 @@ Azure PPML solution integrate BigDL ***PPML*** technology with Azure Services(Az ## 2. Setup -### 2.1 Create Azure VM with BigDL PPML image -#### 2.1.1 Create Resource Group -Create resource group or use your existing resource group. Create resource group with Azure CLI: +### 2.1 Install Azure CLI +Before you setup your environment, please install Azure CLI on your machine according to [guide](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli). + +Then run `az login` to login to Azure system before you run following Azure commands. + +### 2.2 Create Azure VM with BigDL PPML image +#### 2.2.1 Create Resource Group +Create resource group or use your existing resource group. Example code to create resource group with Azure CLI: ``` -BigDLresourceGroupName="bigdl-rg-es2-test" region="eastus2" az group create \ - --name $BigDLresourceGroupName \ + --name myResourceGroup \ --location $region \ --output none ``` -#### 2.1.2 Create Linux client with sgx support -Create Linux VM through Azure CLI/Portal/Powershell. Please choose East US 2 region. -For size of the VM, please choose DC-Series VM with more than 4 vCPU cores. +#### 2.2.2 Create Linux client with sgx support +Create Linux VM through Azure [CLI](https://docs.microsoft.com/en-us/azure/developer/javascript/tutorial/nodejs-virtual-machine-vm/create-linux-virtual-machine-azure-cli)/[Portal](https://docs.microsoft.com/en-us/azure/virtual-machines/linux/quick-create-portal)/Powershell. Please choose East US 2 region. +For size of the VM, please choose DC-Series VM with more than 4 vCPU cores. -#### 2.1.3 Pull BigDL PPML image and start +#### 2.2.3 Pull BigDL PPML image and start * Login to the created VM, pull BigDL PPML image using such command: ```bash docker pull intelanalytics/bigdl-ppml-trusted-big-data-ml-python-graphene:2.1.0-SNAPSHOT @@ -56,53 +60,30 @@ sudo docker run -itd \ $DOCKER_IMAGE bash ``` -### 2.2 Create AKS(Azure Kubernetes Services) +### 2.3 Create AKS(Azure Kubernetes Services) Create AKS or use existing one. -The steps to create AKS is as below -* Create Service Principle +You can run `/ppml/trusted-big-data-ml/azure/create-aks.sh` to create AKS with confidential computing support. + +Note: Please use same VNet information of your client to create AKS. ```bash -az ad sp create-for-rbac +/ppml/trusted-big-data-ml/azure/create-aks.sh \ +--resource-group myResourceGroup \ +--vnet-resource-group myVnetResourceGroup \ +--vnet-name myVnetName \ +--subnet-name mySubnetName \ +--cluster-name myAKSName \ +--vm-size myAKSNodeVMSize \ +--node-count myAKSInitNodeCount ``` -The output is like below, please note down the 'appId'. +You can check the information by run: ```bash -{ -"appId": "b1876d8d-66bc-4352-9ce4-8f0192b2546d", -"displayName": "azure-cli-2022-03-04-01-21-55", -"password": "0t~OHjoWuKYNO.b6r7OZG_uOAn5AbnTmHp", -"tenant": "076293d2-5bf8-4aed-b73f-d8e82dacfc7e" -} -``` -* Assign your service princile to the VNet -```bash -VNET_ID=$(az network vnet show --resource-group myResourceGroup --name myAKSVnet --query id -o tsv) -SUBNET_ID=$(az network vnet subnet show --resource-group myResourceGroup --vnet-name myAKSVnet --name myAKSSubnet --query id -o tsv) -az role assignment create --assignee --scope "/subscriptions/xxx/resourceGroups/myResourceGroup/providers/Microsoft.Network/virtualNetworks/myAKSSubnet" --role "Network Contributor" -``` -* Create AKS -Example command to create AKS: -```bash -az aks create \ - --resource-group myResourceGroup \ - --name myAKSCluster \ - --node-count 3 \ - --network-plugin kubenet \ - --service-cidr 10.0.0.0/16 \ - --dns-service-ip 10.0.0.10 \ - --pod-cidr 10.244.0.0/16 \ - --docker-bridge-address 172.17.0.1/16 \ - --vnet-subnet-id $SUBNET_ID \ - --service-principal +/ppml/trusted-big-data-ml/azure/create-aks.sh --help ``` -* Enable Confidential Computing addon on AKS -```bash -az aks enable-addons --addons confcom --name myAKSCluster--resource-group myResourceGroup -``` - -## 2.3 Create Azure Data Lake Store Gen 2 -### 2.3.1 Create Data Lake Storage account or use existing one. +## 2.4 Create Azure Data Lake Store Gen 2 +### 2.4.1 Create Data Lake Storage account or use existing one. The example command to create Data Lake store is as below: ```bash az dls account create --account myDataLakeAccount --location myLocation --resource-group myResourceGroup @@ -126,7 +107,7 @@ Example command to upload directory ```bash az storage fs directory upload -f myFS --account-name myDataLakeAccount -s "path/to/directory" -d myDirectory --recursive ``` -### 2.3.2 Access data in Hadoop through ABFS(Azure Blob Filesystem) driver +### 2.4.2 Access data in Hadoop through ABFS(Azure Blob Filesystem) driver You can access Data Lake Storage in Hadoop filesytem by such URI: ```abfs[s]://file_system@account_name.dfs.core.windows.net///``` #### Authentication The ABFS driver supports two forms of authentication so that the Hadoop application may securely access resources contained within a Data Lake Storage Gen2 capable account. @@ -141,8 +122,8 @@ az storage account keys list -g MyResourceGroup -n myDataLakeAccount ``` Use one of the keys in authentication. -## 2.4 Create Azure Key Vault -### 2.4.1 Create or use an existing Azure key vault +## 2.5 Create Azure Key Vault +### 2.5.1 Create or use an existing Azure key vault Example command to create key vault ```bash az keyvault create -n myKeyVault -g myResourceGroup -l location @@ -154,7 +135,7 @@ Take note of the following properties for use in the next section: * The name of your Azure key vault resource * The Azure tenant ID that the subscription belongs to -### 2.4.2 Set access policy for the client VM +### 2.5.2 Set access policy for the client VM * Login to the client VM, and get the system identity: ```bash az vm identity assign -g myResourceGroup -n myVM @@ -174,8 +155,8 @@ Example command: az keyvault set-policy --name myKeyVault --object-id --secret-permissions all --key-permissions all --certificate-permissions all ``` -### 2.4.3 AKS access key vault -#### 2.4.3.1 Set access for AKS VM ScaleSet +### 2.5.3 AKS access key vault +#### 2.5.3.1 Set access for AKS VM ScaleSet ##### a. Find your VM ScaleSet in your AKS, and assign system managed identity to VM scale set. ```bash az vm identity assign -g myResourceGroup -n myAKSVMSS @@ -196,7 +177,7 @@ Example command: ```bash az keyvault set-policy --name myKeyVault --object-id --secret-permissions get --key-permissions all --certificate-permissions all ``` -#### 2.4.3.2 Set access for AKS +#### 2.5.3.2 Set access for AKS ##### a. Enable Azure Key Vault Provider for Secrets Store CSI Driver support Example command: ```bash @@ -290,6 +271,9 @@ ARGS= DATA_LAKE_NAME= DATA_LAKE_ACCESS_KEY= KEY_VAULT_NAME= +PRIMARY_KEY_PATH= +DATA_KEY_PATH= + LOCAL_IP= RUNTIME_SPARK_MASTER= @@ -353,6 +337,10 @@ export TF_MKL_ALLOC_MAX_BYTES=10737418240 && \ --conf spark.hadoop.fs.azure.account.auth.type.${DATA_LAKE_NAME}.dfs.core.windows.net=SharedKey \ --conf spark.hadoop.fs.azure.account.key.${DATA_LAKE_NAME}.dfs.core.windows.net=${DATA_LAKE_ACCESS_KEY} \ --conf spark.hadoop.fs.azure.enable.append.support=true \ + --conf spark.bigdl.kms.type=AzureKeyManagementService \ + --conf spark.bigdl.kms.azure.vault=$KEY_VAULT_NAME \ + --conf spark.bigdl.kms.key.primary=$PRIMARY_KEY_PATH \ + --conf spark.bigdl.kms.key.data=$DATA_KEY_PATH \ --class $SPARK_JOB_MAIN_CLASS \ --verbose \ local://$SPARK_EXTRA_JAR_PATH \ diff --git a/docs/readthedocs/source/index.rst b/docs/readthedocs/source/index.rst index 49d0ed7d..615978dd 100644 --- a/docs/readthedocs/source/index.rst +++ b/docs/readthedocs/source/index.rst @@ -93,6 +93,7 @@ BigDL Documentation doc/PPML/QuickStart/trusted-serving-on-k8s-guide.md doc/PPML/QuickStart/tpc-h_with_sparksql_on_k8s.md doc/PPML/QuickStart/tpc-ds_with_sparksql_on_k8s.md + doc/PPML/Overview/azure_ppml.md .. toctree:: :maxdepth: 1