Meta Estimators in SciKeras¶

In this notebook, we implement sklearn ensemble and tree meta-estimators backed by a Keras MLP model.

Table of contents¶

1. Setup
2. Defining the Keras Model
- 2.1 Building a boosting ensemble
3. Testing with a toy dataset
4. Bagging ensemble

1. Setup¶

[1]:

try:
    import scikeras
except ImportError:
    !python -m pip install scikeras[tensorflow]

WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1734039976.467631    4400 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1734039976.471887    4400 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered

Silence TensorFlow logging to keep output succinct.

[2]:

import warnings
from tensorflow import get_logger
get_logger().setLevel('ERROR')
warnings.filterwarnings("ignore", message="Setting the random state for TF")

[3]:

import numpy as np
from scikeras.wrappers import KerasClassifier, KerasRegressor
import keras

2. Defining the Keras Model¶

We borrow our MLPClassifier implementation from the MLPClassifier notebook.

[4]:

from typing import Dict, Iterable, Any


def get_clf_model(hidden_layer_sizes: Iterable[int], meta: Dict[str, Any], compile_kwargs: Dict[str, Any]):
    model = keras.Sequential()
    inp = keras.layers.Input(shape=(meta["n_features_in_"],))
    model.add(inp)
    for hidden_layer_size in hidden_layer_sizes:
        layer = keras.layers.Dense(hidden_layer_size, activation="relu")
        model.add(layer)
    if meta["target_type_"] == "binary":
        n_output_units = 1
        output_activation = "sigmoid"
        loss = "binary_crossentropy"
    elif meta["target_type_"] == "multiclass":
        n_output_units = meta["n_classes_"]
        output_activation = "softmax"
        loss = "sparse_categorical_crossentropy"
    else:
        raise NotImplementedError(f"Unsupported task type: {meta['target_type_']}")
    out = keras.layers.Dense(n_output_units, activation=output_activation)
    model.add(out)
    model.compile(loss=loss, optimizer=compile_kwargs["optimizer"])
    return model

Next we wrap this Keras model with SciKeras

[5]:

clf = KerasClassifier(
    model=get_clf_model,
    hidden_layer_sizes=(100, ),
    optimizer="adam",
    optimizer__learning_rate=0.001,
    verbose=0,
    random_state=0,
)

2.1 Building a boosting ensemble¶

Because SciKeras estimators are fully compliant with the Scikit-Learn API, we can make use of Scikit-Learn’s built in utilities. In particular example, we will use AdaBoostClassifier from sklearn.ensemble.AdaBoostClassifier, but the process is the same for most Scikit-Learn meta-estimators.

[6]:

from sklearn.ensemble import AdaBoostClassifier

adaboost = AdaBoostClassifier(estimator=clf, random_state=0)

3. Testing with a toy dataset¶

Before continouing, we will run a small test to make sure we get somewhat reasonable results.

[7]:

from sklearn.datasets import make_moons

X, y = make_moons()

single_score = clf.fit(X, y).score(X, y)

adaboost_score = adaboost.fit(X, y).score(X, y)

print(f"Single score: {single_score:.2f}")
print(f"AdaBoost score: {adaboost_score:.2f}")

/home/runner/work/scikeras/scikeras/.venv/lib/python3.12/site-packages/sklearn/ensemble/_weight_boosting.py:527: FutureWarning: The SAMME.R algorithm (the default) is deprecated and will be removed in 1.6. Use the SAMME algorithm to circumvent this warning.
  warnings.warn(

Single score: 0.53
AdaBoost score: 0.87

We see that the score for the AdaBoost classifier is slightly higher than that of an individual MLPRegressor instance. We can explore the individual classifiers, and see that each one is composed of a Keras Model with it’s own individual weights.

[8]:

print(adaboost.estimators_[0].model_.get_weights()[0][0, :5])  # first sub-estimator
print(adaboost.estimators_[1].model_.get_weights()[0][0, :5])  # second sub-estimator

[-0.00939173  0.04660247  0.08650086 -0.20484188  0.11110867]
[ 0.23005284  0.2013653   0.21442483  0.03201675 -0.09931928]

4. Bagging ensemble¶

For comparison, we run the same test with an ensemble built using sklearn.ensemble.BaggingClassifier.

[9]:

from sklearn.ensemble import BaggingClassifier

bagging = BaggingClassifier(estimator=clf, random_state=0, n_jobs=-1)

bagging_score = bagging.fit(X, y).score(X, y)

print(f"Bagging score: {bagging_score:.2f}")

WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1734040053.229018    6471 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1734040053.250741    6471 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1734040053.533584    6470 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1734040053.542010    6470 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1734040053.896569    6472 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1734040053.905468    6472 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1734040053.942108    6473 cuda_dnn.cc:8310] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1734040053.949797    6473 cuda_blas.cc:1418] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered

Bagging score: 0.73

WARNING:tensorflow:5 out of the last 9 calls to <function TensorFlowTrainer.make_predict_function.<locals>.one_step_on_data_distributed at 0x7f31102fbec0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
WARNING:tensorflow:6 out of the last 12 calls to <function TensorFlowTrainer.make_predict_function.<locals>.one_step_on_data_distributed at 0x7f31102fbec0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
WARNING:tensorflow:5 out of the last 9 calls to <function TensorFlowTrainer.make_predict_function.<locals>.one_step_on_data_distributed at 0x7fd612e36c00> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
WARNING:tensorflow:6 out of the last 12 calls to <function TensorFlowTrainer.make_predict_function.<locals>.one_step_on_data_distributed at 0x7fd612e36c00> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.