ayo/ipex-llm
1
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

[PPML]add gramine bigdata toolkit (#6135)

Browse source 
* java scala spark hadoop hive

* flink

* tmp copy

* bigdata toolkit

* change image tag

* change build script

* typo

* add log4j2.xml

* change https to http

* fix

* exchange location of spark and flink

* add maven proxy

* change proxy name

* update settings

* add scala version arg

* add submit.sh

* delete test script

* change home and add dependencies

* add custom images

* fix

* add sgx related files

* path change

* arg move

* move unused env and directory

* remove scala

* remove settings

* upgrade spark

* delete settings

* delete work directory

* modify path

* remove java

* remove java in first stage

* readd jdk in first stage

* fix

* build base image script

* add readme

* remove flink

* fix

* remove unused env

* CI/CD

* trigger test

* Trigger test

* fix proxy

* enable non-privilege

* remove directories in level 1

* change resuorces

* remove test code

* change resources limits
This commit is contained in:
Shaojie Cui 2022-10-25 09:44:34 +08:00 committed by GitHub
parent b4d01ac706
commit 6e32cfc215
1 changed files with 37 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

37
.github/workflows/manually_build.yml vendored
View file

@ -63,7 +63,44 @@ jobs:
sudo docker push ${base_image}:${TAG} sudo docker push ${base_image}:${TAG}
sudo docker tag ${base_image}:${TAG} 10.239.45.10/arda/${base_image}:${TAG} sudo docker tag ${base_image}:${TAG} 10.239.45.10/arda/${base_image}:${TAG}
sudo docker push 10.239.45.10/arda/${base_image}:${TAG} sudo docker push 10.239.45.10/arda/${base_image}:${TAG}
echo "########################################################"
echo "####### bigdl-ppml-trusted-bigdata-gramine-base ########"
echo "########################################################"
cd ../trusted-bigdata
export bigdata_base_image=intelanalytics/bigdl-ppml-trusted-bigdata-gramine-base
sudo docker build \
--build-arg http_proxy=${HTTP_PROXY} \
--build-arg https_proxy=${HTTPS_PROXY} \
--build-arg HTTP_PROXY_HOST=${HTTP_PROXY_HOST_2} \
--build-arg HTTP_PROXY_PORT=${HTTP_PROXY_PORT_2} \
--build-arg HTTPS_PROXY_HOST=${HTTP_PROXY_HOST_2} \
--build-arg HTTPS_PROXY_PORT=${HTTP_PROXY_PORT_3} \
--build-arg JDK_VERSION=8u192 \
--build-arg JDK_URL=${JDK_URL} \
--build-arg SPARK_JAR_REPO_URL=${SPARK_JAR_REPO_URL} \
--build-arg no_proxy=${LOCAL_IP} \
--build-arg BASE_IMAGE_NAME=${base_image} \
--build-arg BASE_IMAGE_TAG=${TAG} \
-t ${bigdata_base_image}:${TAG} -f ./Dockerfile .
sudo docker push ${bigdata_base_image}:${TAG}
sudo docker tag ${bigdata_base_image}:${TAG} 10.239.45.10/arda/${bigdata_base_image}:${TAG}
sudo docker push 10.239.45.10/arda/${bigdata_base_image}:${TAG}
cd custom-image
openssl genrsa -3 -out enclave-key.pem 3072
export bigdata_custom_image=intelanalytics/bigdl-ppml-trusted-bigdata-gramine-reference
sudo docker build \
--build-arg BASE_IMAGE_NAME=${bigdata_base_image} \
--build-arg BASE_IMAGE_TAG=${TAG} \
--build-arg SGX_MEM_SIZE=32G \
--build-arg SGX_LOG_LEVEL=error \
-t ${bigdata_custom_image}:${TAG} \
-f ./Dockerfile .
sudo docker push ${bigdata_custom_image}:${TAG}
sudo docker tag ${bigdata_custom_image}:${TAG} 10.239.45.10/arda/${bigdata_custom_image}:${TAG}
sudo docker push 10.239.45.10/arda/${bigdata_custom_image}:${TAG}
sudo docker rmi -f ${base_image}:${TAG} sudo docker rmi -f ${base_image}:${TAG}
sudo docker rmi -f ${bigdata_base_image}:${TAG}
sudo docker rmi -f ${bigdata_custom_image}:${TAG}
bigdl-ppml-trusted-deep-learning-gramine-base: bigdl-ppml-trusted-deep-learning-gramine-base:
if: ${{ github.event.inputs.artifact == 'bigdl-ppml-trusted-deep-learning-gramine-base' || github.event.inputs.artifact == 'all' }} if: ${{ github.event.inputs.artifact == 'bigdl-ppml-trusted-deep-learning-gramine-base' || github.event.inputs.artifact == 'all' }}