cc540d0c34
* Update open in colab button from HTML-style hyperlinks to markdown image with link for notebooks in readthedocs folder * Update "Open in Colab" button for notebooks under python folder with markdown image with link
393 lines
18 KiB
Text
393 lines
18 KiB
Text
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "yzsDlbsUBsuF"
|
|
},
|
|
"source": [
|
|
"[](https://colab.research.google.com/github/intel-analytics/BigDL/blob/main/python/nano/notebooks/hpo/custom.ipynb)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "zcHLkcVjB7Jg"
|
|
},
|
|
"source": [
|
|
""
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "rICdZgjQBfUl"
|
|
},
|
|
"source": [
|
|
"# BigDL-Nano Hyperparameter Tuning (Tensorflow Subclassing Model) Quickstart\n",
|
|
"In this notebook we demonstrates how to use Nano HPO to tune the hyperparameters in tensorflow training. The model is built by subclassing tensorflow.keras.Model.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "XZHU5E3ABfUm"
|
|
},
|
|
"source": [
|
|
"## Step 0: Prepare Environment\n",
|
|
"You can install the latest pre-release version with nano support using below commands.\n",
|
|
"\n",
|
|
"We recommend to run below commands, especially `source bigdl-nano-init` before jupyter kernel is started, or some of the optimizations may not take effect."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"colab": {
|
|
"base_uri": "https://localhost:8080/"
|
|
},
|
|
"id": "3UyXJWA4Cjx2",
|
|
"outputId": "4518ce08-cfc8-4f6c-c40c-01006ffabc81"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Install latest pre-release version of bigdl-nano\n",
|
|
"!pip install --pre bigdl-nano[tensorflow]\n",
|
|
"!pip install setuptools==58.0.4\n",
|
|
"!pip install protobuf==3.20.1\n",
|
|
"!source bigdl-nano-init"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"colab": {
|
|
"base_uri": "https://localhost:8080/"
|
|
},
|
|
"id": "phw-whHaCnpR",
|
|
"outputId": "b517e45c-c98a-4b12-820b-a1a8009fc5aa"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Install other dependecies for Nano HPO\n",
|
|
"!pip install ConfigSpace\n",
|
|
"!pip install optuna"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "xKKCHTU9BfUn"
|
|
},
|
|
"source": [
|
|
"## Step 1: Init Nano AutoML\n",
|
|
"We need to enable Nano HPO before we use it for tensorflow training."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"colab": {
|
|
"base_uri": "https://localhost:8080/"
|
|
},
|
|
"id": "gF1XGVtuBfUn",
|
|
"outputId": "c610c446-328d-43e4-e701-51fe180aef1f"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import bigdl.nano.automl as automl\n",
|
|
"import bigdl.nano.automl.hpo as hpo\n",
|
|
"automl.hpo_config.enable_hpo_tf()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "3InpJGOuBfUp"
|
|
},
|
|
"source": [
|
|
"## Step 2: Prepare data\n",
|
|
"We use fashion MNIST dataset for demonstration."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"colab": {
|
|
"base_uri": "https://localhost:8080/"
|
|
},
|
|
"id": "A4AjqFBTBfUp",
|
|
"outputId": "d80596f1-98e4-4358-a583-706e24c3da0c"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"from tensorflow import keras\n",
|
|
"(x_train, y_train), (x_test, y_test) = keras.datasets.fashion_mnist.load_data()\n",
|
|
"\n",
|
|
"CLASSES = 10\n",
|
|
"\n",
|
|
"img_x, img_y = x_train.shape[1], x_train.shape[2]\n",
|
|
"x_train = x_train.reshape(-1, img_x, img_y,1).astype(\"float32\") / 255\n",
|
|
"x_test = x_test.reshape(-1, img_x, img_y,1).astype(\"float32\") / 255"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "hwrtVD9yBfUq"
|
|
},
|
|
"source": [
|
|
"## Step 3: Build model and specify search spaces\n",
|
|
"We now create our model. \n",
|
|
"\n",
|
|
"Decorate the model class with hpo.tfmodel, and you will be able to specify search spaces in init arguments when creating the model, as shown below. For more details, refer to [user doc](https://bigdl.readthedocs.io/en/latest/doc/Nano/QuickStart/hpo.html).\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "uRFIRllGr0xu"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import tensorflow as tf\n",
|
|
"from tensorflow.keras.datasets import mnist\n",
|
|
"from tensorflow.keras.layers import Conv2D, Dropout, MaxPooling2D\n",
|
|
"from tensorflow.keras.layers import Dense\n",
|
|
"from tensorflow.keras.layers import Flatten"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "SNmZHFMMBfUr"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import tensorflow as tf\n",
|
|
"@hpo.tfmodel()\n",
|
|
"class MyModel(tf.keras.Model):\n",
|
|
"\n",
|
|
" def __init__(self, filters, kernel_size, strides, activation):\n",
|
|
" super().__init__()\n",
|
|
" self.conv1 = Conv2D(\n",
|
|
" filters=filters,\n",
|
|
" kernel_size=kernel_size,\n",
|
|
" strides=strides,\n",
|
|
" activation=activation)\n",
|
|
" self.pool1 = MaxPooling2D(pool_size=2)\n",
|
|
" self.drop1 = Dropout(0.3)\n",
|
|
" self.flat = Flatten()\n",
|
|
" self.dense1 = Dense(256, activation='relu')\n",
|
|
" self.drop3 = Dropout(0.5)\n",
|
|
" self.dense2 = Dense(CLASSES, activation=\"softmax\")\n",
|
|
"\n",
|
|
" def call(self, inputs):\n",
|
|
" x = self.conv1(inputs)\n",
|
|
" x = self.pool1(x)\n",
|
|
" x = self.drop1(x)\n",
|
|
" x = self.flat(x)\n",
|
|
" x = self.dense1(x)\n",
|
|
" x = self.drop3(x)\n",
|
|
" x = self.dense2(x)\n",
|
|
" return x\n",
|
|
"model = MyModel(\n",
|
|
" filters=hpo.space.Categorical(32, 64),\n",
|
|
" kernel_size=hpo.space.Categorical(2, 4),\n",
|
|
" strides=hpo.space.Categorical(1, 2),\n",
|
|
" activation=hpo.space.Categorical(\"relu\", \"linear\")\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "msSa9fyeGHTU"
|
|
},
|
|
"source": [
|
|
"## Step 4: Compile model\n",
|
|
"We now compile our model with loss function, optimizer and metrics. If you want to tune learning rate and batch size, refer to [user guide](https://bigdl.readthedocs.io/en/latest/doc/Nano/QuickStart/hpo.html#search-the-learning-rate)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"id": "PfeBIEmoGHxA"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"from tensorflow.keras.optimizers import RMSprop\n",
|
|
"model.compile(\n",
|
|
" loss=\"sparse_categorical_crossentropy\",\n",
|
|
" optimizer=RMSprop(learning_rate=0.001),\n",
|
|
" metrics=[\"accuracy\"],\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "tuZXTIjXBfUs"
|
|
},
|
|
"source": [
|
|
"## Step 5: Run hyperparameter search\n",
|
|
"Run hyperparameter search by calling `model.search`. Set the `target_metric` and `direction` so that HPO optimizes the `target_metric` in the specified `direction`. Each trial will use a different set of hyperparameters in the search space range. Use `n_parallels` to set the nubmer of parallel processes to run trials. After search completes, you can use `search_summary` to retrive the search results for analysis. For more details, refer to [user doc](https://bigdl.readthedocs.io/en/latest/doc/Nano/QuickStart/hpo.html)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"colab": {
|
|
"base_uri": "https://localhost:8080/"
|
|
},
|
|
"id": "hfBS_1JLBfUs",
|
|
"outputId": "897ac4d1-5766-44f2-d6fb-ed7b2a437b9e"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"%%time\n",
|
|
"from bigdl.nano.automl.hpo.backend import PrunerType\n",
|
|
"model.search(\n",
|
|
" n_trials=5,\n",
|
|
" target_metric='val_accuracy',\n",
|
|
" direction=\"maximize\",\n",
|
|
" pruner=PrunerType.HyperBand,\n",
|
|
" pruner_kwargs={'min_resource':1, 'max_resource':100, 'reduction_factor':3},\n",
|
|
" x=x_train,\n",
|
|
" y=y_train,\n",
|
|
" batch_size=128,\n",
|
|
" epochs=5,\n",
|
|
" validation_split=0.2,\n",
|
|
" verbose=False,\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"colab": {
|
|
"base_uri": "https://localhost:8080/"
|
|
},
|
|
"id": "jeK1hT42HCD5",
|
|
"outputId": "204acc43-94db-4cb8-f058-46c02e43c6ab"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"print(model.search_summary())"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"colab": {
|
|
"base_uri": "https://localhost:8080/",
|
|
"height": 345
|
|
},
|
|
"id": "n_vwMwJVLaCc",
|
|
"outputId": "8996e8ac-5ced-4cfb-c454-de4df5860012"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"study = model.search_summary()\n",
|
|
"study.trials_dataframe(attrs=(\"number\", \"value\", \"params\", \"state\"))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "NVDG1hl-BfUt"
|
|
},
|
|
"source": [
|
|
"## Step 6: fit with the best hyperparameters\n",
|
|
"After search, `model.fit` will autotmatically use the best hyperparmeters found in search to fit the model."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"colab": {
|
|
"base_uri": "https://localhost:8080/"
|
|
},
|
|
"id": "aAet3uX5BfUt",
|
|
"outputId": "2401cd38-09a6-4251-b459-c97e64d89cc9"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"history = model.fit(x_train, y_train,\n",
|
|
" batch_size=128, epochs=5, validation_split=0.2)\n",
|
|
"\n",
|
|
"test_scores = model.evaluate(x_test, y_test, verbose=2)\n",
|
|
"print(\"Test loss:\", test_scores[0])\n",
|
|
"print(\"Test accuracy:\", test_scores[1])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {
|
|
"id": "9JKTKAGQBfUt"
|
|
},
|
|
"source": [
|
|
"Check out the summary of the model. The model has already been built with the best hyperparameters found by nano hpo."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"colab": {
|
|
"base_uri": "https://localhost:8080/"
|
|
},
|
|
"id": "d8O_cz4DBfUu",
|
|
"outputId": "dd7d6add-a32c-4618-dc88-d19bec8e4c3e"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"print(model.summary())"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"You can find the running output from [here](https://github.com/intel-analytics/BigDL/blob/main/python/nano/notebooks/hpo/custom.ipynb), or run the notebook by yourself in [Google Colab](https://colab.research.google.com/github/intel-analytics/BigDL/blob/main/python/nano/notebooks/hpo/custom.ipynb)."
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"colab": {
|
|
"collapsed_sections": [],
|
|
"name": "custom.ipynb",
|
|
"provenance": []
|
|
},
|
|
"kernelspec": {
|
|
"display_name": "Python 3 (ipykernel)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.7.13"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 0
|
|
}
|