Run in Google Colab

SciKeras Benchmarks

SciKeras wraps Keras Models, but does not alter their performance since all of the heavy lifting still happens within Keras/Tensorflow. In this notebook, we compare the performance and accuracy of a pure-Keras Model to the same model wrapped in SciKeras.

Table of contents

1. Setup

[1]:
try:
    import scikeras
except ImportError:
    !python -m pip install scikeras[tensorflow]

Silence TensorFlow logging to keep output succinct.

[2]:
import warnings
from tensorflow import get_logger
get_logger().setLevel('ERROR')
warnings.filterwarnings("ignore", message="Setting the random state for TF")
[3]:
import numpy as np
from scikeras.wrappers import KerasClassifier, KerasRegressor
from tensorflow import keras

2. Dataset

We will be using the MNIST dataset available within Keras.

[4]:
(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()
# Scale images to the [0, 1] range
x_train = x_train.astype("float32") / 255
x_test = x_test.astype("float32") / 255
# Make sure images have shape (28, 28, 1)
x_train = np.expand_dims(x_train, -1)
x_test = np.expand_dims(x_test, -1)
# Reduce dataset size for faster benchmarks
x_train, y_train = x_train[:2000], y_train[:2000]
x_test, y_test = x_test[:500], y_test[:500]

3. Define Keras Model

Next we will define our Keras model (adapted from keras.io):

[5]:
num_classes = 10
input_shape = (28, 28, 1)


def get_model():
    model = keras.Sequential(
        [
            keras.Input(input_shape),
            keras.layers.Conv2D(32, kernel_size=(3, 3), activation="relu"),
            keras.layers.MaxPooling2D(pool_size=(2, 2)),
            keras.layers.Conv2D(64, kernel_size=(3, 3), activation="relu"),
            keras.layers.MaxPooling2D(pool_size=(2, 2)),
            keras.layers.Flatten(),
            keras.layers.Dropout(0.5),
            keras.layers.Dense(num_classes, activation="softmax"),
        ]
    )
    model.compile(
        loss="sparse_categorical_crossentropy", optimizer="adam"
    )
    return model

4. Keras benchmarks

[6]:
fit_kwargs = {"batch_size": 128, "validation_split": 0.1, "verbose": 0, "epochs": 5}
[7]:
from sklearn.metrics import accuracy_score
from scikeras.utils.random_state import tensorflow_random_state
[8]:
from time import time

with tensorflow_random_state(seed=0):  # we force a TF random state to be able to compare accuracy
    model = get_model()
    start = time()
    model.fit(x_train, y_train, **fit_kwargs)
    print(f"Training time: {time()-start:.2f}")
    y_pred = np.argmax(model.predict(x_test), axis=1)
print(f"Accuracy: {accuracy_score(y_test, y_pred)}")
Training time: 7.50
16/16 [==============================] - 0s 7ms/step
Accuracy: 0.884

5. SciKeras benchmark

[9]:
clf = KerasClassifier(
    model=get_model,
    random_state=0,
    **fit_kwargs
)
[10]:
start = time()
clf.fit(x_train, y_train)
print(f"Training time: {time()-start:.2f}")
y_pred = clf.predict(x_test)
print(f"Accuracy: {accuracy_score(y_test, y_pred)}")
Training time: 7.68
Accuracy: 0.884

As you can see, the overhead for SciKeras is <1 sec, and the accuracy is identical.