{ "cells": [ { "cell_type": "raw", "id": "4996c1b9", "metadata": {}, "source": [ "Run in Google Colab" ] }, { "cell_type": "markdown", "id": "7a505b6f", "metadata": {}, "source": [ "# Basic usage\n", "\n", "`SciKeras` is designed to maximize interoperability between `sklearn` and `Keras/TensorFlow`. The aim is to keep 99% of the flexibility of `Keras` while being able to leverage most features of `sklearn`. Below, we show the basic usage of `SciKeras` and how it can be combined with `sklearn`.\n", "\n", "This notebook shows you how to use the basic functionality of `SciKeras`.\n", "\n", "## Table of contents\n", "\n", "* [1. Setup](#1.-Setup)\n", "* [2. Training a classifier and making predictions](#2.-Training-a-classifier-and-making-predictions)\n", " * [2.1 A toy binary classification task](#2.1-A-toy-binary-classification-task)\n", " * [2.2 Definition of the Keras classification Model](#2.2-Definition-of-the-Keras-classification-Model)\n", " * [2.3 Defining and training the neural net classifier](#2.3-Defining-and-training-the-neural-net-classifier)\n", " * [2.4 Making predictions, classification](#2.4-Making-predictions-classification)\n", "* [3 Training a regressor](#3.-Training-a-regressor)\n", " * [3.1 A toy regression task](#3.1-A-toy-regression-task)\n", " * [3.2 Definition of the Keras regression Model](#3.2-Definition-of-the-Keras-regression-Model)\n", " * [3.3 Defining and training the neural net regressor](#3.3-Defining-and-training-the-neural-net-regressor)\n", " * [3.4 Making predictions, regression](#3.4-Making-predictions-regression)\n", "* [4. Saving and loading a model](#4.-Saving-and-loading-a-model)\n", " * [4.1 Saving the whole model](#4.1-Saving-the-whole-model)\n", " * [4.2 Saving using Keras' saving methods](#4.2-Saving-using-Keras-saving-methods)\n", "* [5. Usage with an sklearn Pipeline](#5.-Usage-with-an-sklearn-Pipeline)\n", "* [6. Callbacks](#6.-Callbacks)\n", "* [7. Usage with sklearn GridSearchCV](#7.-Usage-with-sklearn-GridSearchCV)\n", " * [7.1 Special prefixes](#7.1-Special-prefixes)\n", " * [7.2 Performing a grid search](#7.2-Performing-a-grid-search)\n", "\n", "## 1. Setup" ] }, { "cell_type": "code", "execution_count": 1, "id": "4acaeaab", "metadata": { "execution": { "iopub.execute_input": "2024-04-11T22:25:25.035898Z", "iopub.status.busy": "2024-04-11T22:25:25.035390Z", "iopub.status.idle": "2024-04-11T22:25:27.286455Z", "shell.execute_reply": "2024-04-11T22:25:27.285791Z" } }, "outputs": [], "source": [ "try:\n", " import scikeras\n", "except ImportError:\n", " !python -m pip install scikeras" ] }, { "cell_type": "markdown", "id": "02d2108e", "metadata": {}, "source": [ "Silence TensorFlow logging to keep output succinct." ] }, { "cell_type": "code", "execution_count": 2, "id": "bc26adba", "metadata": { "execution": { "iopub.execute_input": "2024-04-11T22:25:27.290489Z", "iopub.status.busy": "2024-04-11T22:25:27.289759Z", "iopub.status.idle": "2024-04-11T22:25:27.293787Z", "shell.execute_reply": "2024-04-11T22:25:27.293101Z" } }, "outputs": [], "source": [ "import warnings\n", "from tensorflow import get_logger\n", "get_logger().setLevel('ERROR')\n", "warnings.filterwarnings(\"ignore\", message=\"Setting the random state for TF\")" ] }, { "cell_type": "code", "execution_count": 3, "id": "77cc903d", "metadata": { "execution": { "iopub.execute_input": "2024-04-11T22:25:27.296475Z", "iopub.status.busy": "2024-04-11T22:25:27.296003Z", "iopub.status.idle": "2024-04-11T22:25:27.603873Z", "shell.execute_reply": "2024-04-11T22:25:27.603220Z" } }, "outputs": [], "source": [ "import numpy as np\n", "from scikeras.wrappers import KerasClassifier, KerasRegressor\n", "import keras" ] }, { "cell_type": "markdown", "id": "ed3a2383", "metadata": {}, "source": [ "## 2. Training a classifier and making predictions\n", "\n", "### 2.1 A toy binary classification task\n", "\n", "We load a toy classification task from `sklearn`." ] }, { "cell_type": "code", "execution_count": 4, "id": "9f5d96b4", "metadata": { "execution": { "iopub.execute_input": "2024-04-11T22:25:27.607397Z", "iopub.status.busy": "2024-04-11T22:25:27.606555Z", "iopub.status.idle": "2024-04-11T22:25:27.660538Z", "shell.execute_reply": "2024-04-11T22:25:27.659783Z" } }, "outputs": [ { "data": { "text/plain": [ "((1000, 20), (1000,), 0.5)" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import numpy as np\n", "from sklearn.datasets import make_classification\n", "\n", "\n", "X, y = make_classification(1000, 20, n_informative=10, random_state=0)\n", "\n", "X.shape, y.shape, y.mean()" ] }, { "cell_type": "markdown", "id": "8fcc0cba", "metadata": {}, "source": [ "### 2.2 Definition of the Keras classification Model\n", "\n", "We define a vanilla neural network with.\n", "\n", "Because we are dealing with 2 classes, the output layer can be constructed in\n", "two different ways:\n", "\n", "1. Single unit with a `\"sigmoid\"` nonlinearity. The loss must be `\"binary_crossentropy\"`.\n", "2. Two units (one for each class) and a `\"softmax\"` nonlinearity. The loss must be `\"sparse_categorical_crossentropy\"`.\n", "\n", "In this example, we choose the first option, which is what you would usually\n", "do for binary classification. The second option is usually reserved for when\n", "you have >2 classes." ] }, { "cell_type": "code", "execution_count": 5, "id": "0347d71a", "metadata": { "execution": { "iopub.execute_input": "2024-04-11T22:25:27.663026Z", "iopub.status.busy": "2024-04-11T22:25:27.662680Z", "iopub.status.idle": "2024-04-11T22:25:27.667357Z", "shell.execute_reply": "2024-04-11T22:25:27.666543Z" } }, "outputs": [], "source": [ "import keras\n", "\n", "\n", "def get_clf(meta, hidden_layer_sizes, dropout):\n", " n_features_in_ = meta[\"n_features_in_\"]\n", " n_classes_ = meta[\"n_classes_\"]\n", " model = keras.models.Sequential()\n", " model.add(keras.layers.Input(shape=(n_features_in_,)))\n", " for hidden_layer_size in hidden_layer_sizes:\n", " model.add(keras.layers.Dense(hidden_layer_size, activation=\"relu\"))\n", " model.add(keras.layers.Dropout(dropout))\n", " model.add(keras.layers.Dense(1, activation=\"sigmoid\"))\n", " return model" ] }, { "cell_type": "markdown", "id": "cc7fbb31", "metadata": {}, "source": [ "### 2.3 Defining and training the neural net classifier\n", "\n", "We use `KerasClassifier` because we're dealing with a classifcation task. The first argument should be a callable returning a `Keras.Model`, in this case, `get_clf`. As additional arguments, we pass the number of loss function (required) and the optimizer, but the later is optional. We must also pass all of the arguments to `get_clf` as keyword arguments to `KerasClassifier` if they don't have a default value in `get_clf`. Note that if you do not pass an argument to `KerasClassifier`, it will not be avilable for hyperparameter tuning. Finally, we also pass `random_state=0` for reproducible results." ] }, { "cell_type": "code", "execution_count": 6, "id": "8de11748", "metadata": { "execution": { "iopub.execute_input": "2024-04-11T22:25:27.669582Z", "iopub.status.busy": "2024-04-11T22:25:27.669389Z", "iopub.status.idle": "2024-04-11T22:25:27.672807Z", "shell.execute_reply": "2024-04-11T22:25:27.672155Z" } }, "outputs": [], "source": [ "from scikeras.wrappers import KerasClassifier\n", "\n", "\n", "clf = KerasClassifier(\n", " model=get_clf,\n", " loss=\"binary_crossentropy\",\n", " hidden_layer_sizes=(100,),\n", " dropout=0.5,\n", ")" ] }, { "cell_type": "markdown", "id": "97775308", "metadata": {}, "source": [ "As in `sklearn`, we call `fit` passing the input data `X` and the targets `y`." ] }, { "cell_type": "code", "execution_count": 7, "id": "3a0b8c75", "metadata": { "execution": { "iopub.execute_input": "2024-04-11T22:25:27.675530Z", "iopub.status.busy": "2024-04-11T22:25:27.674980Z", "iopub.status.idle": "2024-04-11T22:25:28.291738Z", "shell.execute_reply": "2024-04-11T22:25:28.291043Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\r", "\u001b[1m 1/32\u001b[0m \u001b[37m━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[1m14s\u001b[0m 479ms/step - loss: 0.6073" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "\u001b[1m32/32\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 901us/step - loss: 0.6832 \n" ] } ], "source": [ "clf.fit(X, y);" ] }, { "cell_type": "markdown", "id": "5e21db44", "metadata": {}, "source": [ "Also, as in `sklearn`, you may call `predict` or `predict_proba` on the fitted model.\n", "\n", "### 2.4 Making predictions, classification" ] }, { "cell_type": "code", "execution_count": 8, "id": "cb48ec16", "metadata": { "execution": { "iopub.execute_input": "2024-04-11T22:25:28.294693Z", "iopub.status.busy": "2024-04-11T22:25:28.294136Z", "iopub.status.idle": "2024-04-11T22:25:28.367874Z", "shell.execute_reply": "2024-04-11T22:25:28.367293Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\r", "\u001b[1m1/1\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 28ms/step" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "\u001b[1m1/1\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 28ms/step\n" ] }, { "data": { "text/plain": [ "array([1, 0, 0, 0, 0])" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "y_pred = clf.predict(X[:5])\n", "y_pred" ] }, { "cell_type": "code", "execution_count": 9, "id": "829b064c", "metadata": { "execution": { "iopub.execute_input": "2024-04-11T22:25:28.370165Z", "iopub.status.busy": "2024-04-11T22:25:28.369963Z", "iopub.status.idle": "2024-04-11T22:25:28.414449Z", "shell.execute_reply": "2024-04-11T22:25:28.413813Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\r", "\u001b[1m1/1\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 11ms/step" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "\u001b[1m1/1\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 12ms/step\n" ] }, { "data": { "text/plain": [ "array([[0.41824633, 0.5817537 ],\n", " [0.83141077, 0.1685892 ],\n", " [0.82014996, 0.17985004],\n", " [0.9229777 , 0.07702232],\n", " [0.8843903 , 0.11560971]], dtype=float32)" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "y_proba = clf.predict_proba(X[:5])\n", "y_proba" ] }, { "cell_type": "markdown", "id": "1c1670fe", "metadata": {}, "source": [ "## 3 Training a regressor\n", "\n", "### 3.1 A toy regression task" ] }, { "cell_type": "code", "execution_count": 10, "id": "b4f5beb4", "metadata": { "execution": { "iopub.execute_input": "2024-04-11T22:25:28.417410Z", "iopub.status.busy": "2024-04-11T22:25:28.417208Z", "iopub.status.idle": "2024-04-11T22:25:28.427718Z", "shell.execute_reply": "2024-04-11T22:25:28.426913Z" } }, "outputs": [ { "data": { "text/plain": [ "((1000, 20), (1000,), -649.0148244404172, 615.4505181286091)" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from sklearn.datasets import make_regression\n", "\n", "\n", "X_regr, y_regr = make_regression(1000, 20, n_informative=10, random_state=0)\n", "\n", "X_regr.shape, y_regr.shape, y_regr.min(), y_regr.max()" ] }, { "cell_type": "markdown", "id": "c4ed2307", "metadata": {}, "source": [ "### 3.2 Definition of the Keras regression Model\n", "\n", "Again, define a vanilla neural network. The main difference is that the output layer always has a single unit and does not apply any nonlinearity." ] }, { "cell_type": "code", "execution_count": 11, "id": "7f797552", "metadata": { "execution": { "iopub.execute_input": "2024-04-11T22:25:28.438075Z", "iopub.status.busy": "2024-04-11T22:25:28.436872Z", "iopub.status.idle": "2024-04-11T22:25:28.442791Z", "shell.execute_reply": "2024-04-11T22:25:28.442261Z" } }, "outputs": [], "source": [ "def get_reg(meta, hidden_layer_sizes, dropout):\n", " n_features_in_ = meta[\"n_features_in_\"]\n", " model = keras.models.Sequential()\n", " model.add(keras.layers.Input(shape=(n_features_in_,)))\n", " for hidden_layer_size in hidden_layer_sizes:\n", " model.add(keras.layers.Dense(hidden_layer_size, activation=\"relu\"))\n", " model.add(keras.layers.Dropout(dropout))\n", " model.add(keras.layers.Dense(1))\n", " return model" ] }, { "cell_type": "markdown", "id": "d45f188f", "metadata": {}, "source": [ "### 3.3 Defining and training the neural net regressor\n", "\n", "Training a regressor has nearly the same data flow as training a classifier. The differences include using `KerasRegressor` instead of `KerasClassifier` and adding `keras.metrics.R2Score` as a metric. Most of the Scikit-learn regressors use the coefficient of determination or R^2 as a metric function, which measures correlation between the true labels and predicted labels." ] }, { "cell_type": "code", "execution_count": 12, "id": "ed68b2ef", "metadata": { "execution": { "iopub.execute_input": "2024-04-11T22:25:28.446829Z", "iopub.status.busy": "2024-04-11T22:25:28.445757Z", "iopub.status.idle": "2024-04-11T22:25:28.450712Z", "shell.execute_reply": "2024-04-11T22:25:28.450190Z" } }, "outputs": [], "source": [ "import keras\n", "import keras.models\n", "from scikeras.wrappers import KerasRegressor\n", "\n", "\n", "reg = KerasRegressor(\n", " model=get_reg,\n", " loss=\"mse\",\n", " metrics=[keras.metrics.R2Score],\n", " hidden_layer_sizes=(100,),\n", " dropout=0.5,\n", ")" ] }, { "cell_type": "code", "execution_count": 13, "id": "5b6d004e", "metadata": { "execution": { "iopub.execute_input": "2024-04-11T22:25:28.454603Z", "iopub.status.busy": "2024-04-11T22:25:28.453561Z", "iopub.status.idle": "2024-04-11T22:25:29.195595Z", "shell.execute_reply": "2024-04-11T22:25:29.194992Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\r", "\u001b[1m 1/32\u001b[0m \u001b[37m━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[1m17s\u001b[0m 580ms/step - loss: 39734.1211 - r2_score: -0.0590" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "\u001b[1m32/32\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 907us/step - loss: 42994.3867 - r2_score: -3.3056e-04\n" ] } ], "source": [ "reg.fit(X_regr, y_regr);" ] }, { "cell_type": "markdown", "id": "8f4f60a9", "metadata": {}, "source": [ "### 3.4 Making predictions, regression\n", "\n", "You may call `predict` or `predict_proba` on the fitted model. For regressions, both methods return the same value." ] }, { "cell_type": "code", "execution_count": 14, "id": "5237d02d", "metadata": { "execution": { "iopub.execute_input": "2024-04-11T22:25:29.198476Z", "iopub.status.busy": "2024-04-11T22:25:29.197920Z", "iopub.status.idle": "2024-04-11T22:25:29.273877Z", "shell.execute_reply": "2024-04-11T22:25:29.273300Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\r", "\u001b[1m1/1\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 29ms/step" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "\u001b[1m1/1\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 29ms/step\n" ] }, { "data": { "text/plain": [ "array([ 0.38163126, -0.36355966, 1.3995478 , 0.07941712, -0.1637274 ],\n", " dtype=float32)" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "y_pred = reg.predict(X_regr[:5])\n", "y_pred" ] }, { "cell_type": "markdown", "id": "d2927dee", "metadata": {}, "source": [ "## 4. Saving and loading a model\n", "\n", "Save and load either the whole model by using pickle, or use Keras' specialized save methods on the `KerasClassifier.model_` or `KerasRegressor.model_` attribute that is created after fitting. You will want to use Keras' model saving utilities if any of the following apply:\n", "\n", "1. You wish to save only the weights or only the training configuration of your model.\n", "2. You wish to share your model with collaborators. Pickle is a relatively unsafe protocol and it is not recommended to share or load pickle objects publically.\n", "3. You care about performance, especially if doing in-memory serialization.\n", "\n", "For more information, see Keras' [saving documentation](https://www.tensorflow.org/guide/keras/save_and_serialize).\n", "\n", "### 4.1 Saving the whole model" ] }, { "cell_type": "code", "execution_count": 15, "id": "a5e0cb77", "metadata": { "execution": { "iopub.execute_input": "2024-04-11T22:25:29.276778Z", "iopub.status.busy": "2024-04-11T22:25:29.276216Z", "iopub.status.idle": "2024-04-11T22:25:29.413771Z", "shell.execute_reply": "2024-04-11T22:25:29.413185Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\r", "\u001b[1m1/1\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 34ms/step" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "\u001b[1m1/1\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 34ms/step\n" ] }, { "data": { "text/plain": [ "array([ 0.38163126, -0.36355966, 1.3995478 , 0.07941712, -0.1637274 ],\n", " dtype=float32)" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import pickle\n", "\n", "\n", "bytes_model = pickle.dumps(reg)\n", "new_reg = pickle.loads(bytes_model)\n", "new_reg.predict(X_regr[:5]) # model is still trained" ] }, { "cell_type": "markdown", "id": "72d0c78b", "metadata": {}, "source": [ "### 4.2 Saving using Keras' saving methods\n", "\n", "This efficiently and safely saves the model to disk, including trained weights.\n", "You should use this method if you plan on sharing your saved models." ] }, { "cell_type": "code", "execution_count": 16, "id": "5fee554d", "metadata": { "execution": { "iopub.execute_input": "2024-04-11T22:25:29.416678Z", "iopub.status.busy": "2024-04-11T22:25:29.416123Z", "iopub.status.idle": "2024-04-11T22:25:29.543045Z", "shell.execute_reply": "2024-04-11T22:25:29.542434Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\r", "\u001b[1m 1/32\u001b[0m \u001b[37m━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[1m1s\u001b[0m 39ms/step" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "\u001b[1m32/32\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 650us/step\n" ] } ], "source": [ "# Save to disk\n", "pred_old = reg.predict(X_regr)\n", "reg.model_.save(\"/tmp/my_model.keras\") # saves just the Keras model" ] }, { "cell_type": "code", "execution_count": 17, "id": "02a3865b", "metadata": { "execution": { "iopub.execute_input": "2024-04-11T22:25:29.546044Z", "iopub.status.busy": "2024-04-11T22:25:29.545636Z", "iopub.status.idle": "2024-04-11T22:25:29.724071Z", "shell.execute_reply": "2024-04-11T22:25:29.723346Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\r", "\u001b[1m 1/32\u001b[0m \u001b[37m━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[1m0s\u001b[0m 27ms/step" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "\u001b[1m32/32\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step \n" ] } ], "source": [ "# Load the model back into memory\n", "new_reg_model = keras.saving.load_model(\"/tmp/my_model.keras\")\n", "# Now we need to instantiate a new SciKeras object\n", "# since we only saved the Keras model\n", "reg_new = KerasRegressor(new_reg_model)\n", "# use initialize to avoid re-fitting\n", "reg_new.initialize(X_regr, y_regr)\n", "pred_new = reg_new.predict(X_regr)\n", "np.testing.assert_allclose(pred_old, pred_new)" ] }, { "cell_type": "markdown", "id": "d4d904c3", "metadata": {}, "source": [ "## 5. Usage with an sklearn Pipeline\n", "\n", "It is possible to put the `KerasClassifier` inside an `sklearn Pipeline`, as you would with any `sklearn` classifier.\n" ] }, { "cell_type": "code", "execution_count": 18, "id": "862b1a37", "metadata": { "execution": { "iopub.execute_input": "2024-04-11T22:25:29.727550Z", "iopub.status.busy": "2024-04-11T22:25:29.727166Z", "iopub.status.idle": "2024-04-11T22:25:30.520808Z", "shell.execute_reply": "2024-04-11T22:25:30.520106Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\r", "\u001b[1m 1/32\u001b[0m \u001b[37m━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[1m17s\u001b[0m 570ms/step - loss: 0.7106" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "\u001b[1m32/32\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 816us/step - loss: 0.7350 \n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\r", "\u001b[1m 1/32\u001b[0m \u001b[37m━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[1m0s\u001b[0m 27ms/step" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r", "\u001b[1m32/32\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step \n" ] } ], "source": [ "from sklearn.pipeline import Pipeline\n", "from sklearn.preprocessing import StandardScaler\n", "\n", "\n", "pipe = Pipeline([\n", " ('scale', StandardScaler()),\n", " ('clf', clf),\n", "])\n", "\n", "\n", "y_proba = pipe.fit(X, y).predict(X)" ] }, { "cell_type": "markdown", "id": "8bfa1be6", "metadata": {}, "source": [ "To save the whole pipeline, including the Keras model, use `pickle`.\n", "\n", "## 6. Callbacks\n", "\n", "Adding a new callback to the model is straightforward. Below we define a threashold callback\n", "to avoid training past a certain accuracy. This a rudimentary for of\n", "[early stopping](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/EarlyStopping)." ] }, { "cell_type": "code", "execution_count": 19, "id": "cc077c93", "metadata": { "execution": { "iopub.execute_input": "2024-04-11T22:25:30.523700Z", "iopub.status.busy": "2024-04-11T22:25:30.523157Z", "iopub.status.idle": "2024-04-11T22:25:30.527797Z", "shell.execute_reply": "2024-04-11T22:25:30.527187Z" } }, "outputs": [], "source": [ "class MaxValLoss(keras.callbacks.Callback):\n", "\n", " def __init__(self, monitor: str, threashold: float):\n", " self.monitor = monitor\n", " self.threashold = threashold\n", "\n", " def on_epoch_end(self, epoch, logs=None):\n", " if logs[self.monitor] > self.threashold:\n", " print(\"Threashold reached; stopping training\") \n", " self.model.stop_training = True" ] }, { "cell_type": "markdown", "id": "64805bea", "metadata": {}, "source": [ "Define a test dataset:" ] }, { "cell_type": "code", "execution_count": 20, "id": "fec11443", "metadata": { "execution": { "iopub.execute_input": "2024-04-11T22:25:30.530887Z", "iopub.status.busy": "2024-04-11T22:25:30.530411Z", "iopub.status.idle": "2024-04-11T22:25:30.534554Z", "shell.execute_reply": "2024-04-11T22:25:30.533938Z" } }, "outputs": [], "source": [ "from sklearn.datasets import make_moons\n", "\n", "\n", "X, y = make_moons(n_samples=100, noise=0.2, random_state=0)" ] }, { "cell_type": "markdown", "id": "0c6921d0", "metadata": {}, "source": [ "And try fitting it with and without the callback:" ] }, { "cell_type": "code", "execution_count": 21, "id": "4363e249", "metadata": { "execution": { "iopub.execute_input": "2024-04-11T22:25:30.537219Z", "iopub.status.busy": "2024-04-11T22:25:30.536692Z", "iopub.status.idle": "2024-04-11T22:25:32.161886Z", "shell.execute_reply": "2024-04-11T22:25:32.161202Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Trained 20 epochs\n", "Final accuracy: 0.8999999761581421\n" ] } ], "source": [ "kwargs = dict(\n", " model=get_clf,\n", " loss=\"binary_crossentropy\",\n", " dropout=0.5,\n", " hidden_layer_sizes=(100,),\n", " metrics=[\"binary_accuracy\"],\n", " fit__validation_split=0.2,\n", " epochs=20,\n", " verbose=False,\n", " random_state=0\n", ")\n", "\n", "# First test without the callback\n", "clf = KerasClassifier(**kwargs)\n", "clf.fit(X, y)\n", "print(f\"Trained {len(clf.history_['loss'])} epochs\")\n", "print(f\"Final accuracy: {clf.history_['val_binary_accuracy'][-1]}\") # get last value of last fit/partial_fit call" ] }, { "cell_type": "markdown", "id": "c61a1dbf", "metadata": {}, "source": [ "And with:" ] }, { "cell_type": "code", "execution_count": 22, "id": "f1b6fce4", "metadata": { "execution": { "iopub.execute_input": "2024-04-11T22:25:32.164296Z", "iopub.status.busy": "2024-04-11T22:25:32.164095Z", "iopub.status.idle": "2024-04-11T22:25:33.030278Z", "shell.execute_reply": "2024-04-11T22:25:33.029498Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Threashold reached; stopping training\n", "Trained 4 epochs\n", "Final accuracy: 0.8999999761581421\n" ] } ], "source": [ "# Test with the callback\n", "\n", "cb = MaxValLoss(monitor=\"val_binary_accuracy\", threashold=0.75)\n", "\n", "clf = KerasClassifier(\n", " **kwargs,\n", " callbacks=[cb]\n", ")\n", "clf.fit(X, y)\n", "print(f\"Trained {len(clf.history_['loss'])} epochs\")\n", "print(f\"Final accuracy: {clf.history_['val_binary_accuracy'][-1]}\") # get last value of last fit/partial_fit call" ] }, { "cell_type": "markdown", "id": "49b8ad18", "metadata": {}, "source": [ "For information on how to write custom callbacks, have a look at the\n", "[Advanced Usage](https://nbviewer.jupyter.org/github/adriangb/scikeras/blob/master/notebooks/Advanced_Usage.ipynb) notebook.\n", "\n", "## 7. Usage with sklearn GridSearchCV\n", "\n", "### 7.1 Special prefixes\n", "\n", "SciKeras allows to direct access to all parameters passed to the wrapper constructors, including deeply nested routed parameters. This allows tunning of\n", "paramters like `hidden_layer_sizes` as well as `optimizer__learning_rate`.\n", "\n", "This is exactly the same logic that allows to access estimator parameters in `sklearn Pipeline`s and `FeatureUnion`s.\n", "\n", "This feature is useful in several ways. For one, it allows to set those parameters in the model definition. Furthermore, it allows you to set parameters in an `sklearn GridSearchCV` as shown below.\n", "\n", "To differentiate paramters like `callbacks` which are accepted by both `keras.Model.fit` and `keras.Model.predict` you can add a `fit__` or `predict__` routing suffix respectively. Similar, the `model__` prefix may be used to specify that a paramter is destined only for `get_clf`/`get_reg` (or whatever callable you pass as your `model` argument).\n", "\n", "For more information on parameter routing with special prefixes, see the [Advanced Usage Docs](https://www.adriangb.com/scikeras/stable/advanced.html#routed-parameters)\n", "\n", "### 7.2 Performing a grid search\n", "\n", "Below we show how to perform a grid search over the learning rate (`optimizer__learning_rate`), the model's number of hidden layers (`model__hidden_layer_sizes`), the model's dropout rate (`model__dropout`)." ] }, { "cell_type": "code", "execution_count": 23, "id": "884cdbdc", "metadata": { "execution": { "iopub.execute_input": "2024-04-11T22:25:33.033177Z", "iopub.status.busy": "2024-04-11T22:25:33.032631Z", "iopub.status.idle": "2024-04-11T22:25:33.048745Z", "shell.execute_reply": "2024-04-11T22:25:33.048155Z" } }, "outputs": [], "source": [ "from sklearn.model_selection import GridSearchCV\n", "\n", "\n", "clf = KerasClassifier(\n", " model=get_clf,\n", " loss=\"binary_crossentropy\",\n", " optimizer=\"adam\",\n", " optimizer__learning_rate=0.1,\n", " model__hidden_layer_sizes=(100,),\n", " model__dropout=0.5,\n", " verbose=False,\n", ")" ] }, { "cell_type": "markdown", "id": "39a18cf8", "metadata": {}, "source": [ "*Note*: We set the verbosity level to zero (`verbose=False`) to prevent too much print output from being shown." ] }, { "cell_type": "code", "execution_count": 24, "id": "6e07c5a8", "metadata": { "execution": { "iopub.execute_input": "2024-04-11T22:25:33.051407Z", "iopub.status.busy": "2024-04-11T22:25:33.050943Z", "iopub.status.idle": "2024-04-11T22:25:55.197116Z", "shell.execute_reply": "2024-04-11T22:25:55.196357Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Fitting 5 folds for each of 8 candidates, totalling 40 fits\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "WARNING:tensorflow:5 out of the last 13 calls to .one_step_on_iterator at 0x7faf09385080> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "WARNING:tensorflow:5 out of the last 5 calls to .one_step_on_data_distributed at 0x7faf09233e20> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "WARNING:tensorflow:5 out of the last 13 calls to .one_step_on_iterator at 0x7f940a505080> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "WARNING:tensorflow:5 out of the last 5 calls to .one_step_on_data_distributed at 0x7f940a5afe20> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n", "WARNING:tensorflow:5 out of the last 13 calls to .one_step_on_iterator at 0x7f5bbcc6bf60> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n", "WARNING:tensorflow:5 out of the last 5 calls to .one_step_on_data_distributed at 0x7f5bbcb43240> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "WARNING:tensorflow:5 out of the last 13 calls to .one_step_on_iterator at 0x7faf48ffbf60> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "WARNING:tensorflow:5 out of the last 5 calls to .one_step_on_data_distributed at 0x7faf490d3240> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "WARNING:tensorflow:5 out of the last 13 calls to .one_step_on_iterator at 0x7faf09163ec0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n", "WARNING:tensorflow:6 out of the last 6 calls to .one_step_on_data_distributed at 0x7faf090407c0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n", "WARNING:tensorflow:5 out of the last 13 calls to .one_step_on_iterator at 0x7f940a4e3ec0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "WARNING:tensorflow:6 out of the last 6 calls to .one_step_on_data_distributed at 0x7f940a3c47c0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "WARNING:tensorflow:5 out of the last 13 calls to .one_step_on_iterator at 0x7f5bbccaf7e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n", "WARNING:tensorflow:6 out of the last 6 calls to .one_step_on_data_distributed at 0x7f5bbc990720> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n", "WARNING:tensorflow:5 out of the last 13 calls to .one_step_on_iterator at 0x7faf4903f7e0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "WARNING:tensorflow:6 out of the last 6 calls to .one_step_on_data_distributed at 0x7faf48d20720> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "0.86 {'model__dropout': 0, 'model__hidden_layer_sizes': (100,), 'optimizer__learning_rate': 0.1}\n" ] } ], "source": [ "params = {\n", " 'optimizer__learning_rate': [0.05, 0.1],\n", " 'model__hidden_layer_sizes': [(100, ), (50, 50, )],\n", " 'model__dropout': [0, 0.5],\n", "}\n", "\n", "gs = GridSearchCV(clf, params, scoring='accuracy', n_jobs=-1, verbose=True)\n", "\n", "gs.fit(X, y)\n", "\n", "print(gs.best_score_, gs.best_params_)" ] }, { "cell_type": "markdown", "id": "2c964158", "metadata": {}, "source": [ "Of course, we could further nest the `KerasClassifier` within an `sklearn.pipeline.Pipeline`,\n", "in which case we just prefix the parameter by the name of the net (e.g. `clf__model__hidden_layer_sizes`)." ] } ], "metadata": { "jupytext": { "formats": "ipynb,md" }, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.2" } }, "nbformat": 4, "nbformat_minor": 5 }