{
 "cells": [
  {
   "cell_type": "raw",
   "id": "c5dbfde4",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/adriangb/scikeras/blob/docs-deploy/refs/heads/master/notebooks/AutoEncoders.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\">Run in Google Colab</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55771ea7",
   "metadata": {},
   "source": [
    "# Autoencoders in SciKeras\n",
    "\n",
    "Autencoders are an approach to use nearual networks to distill data into it's most important features, thereby compressing the data.\n",
    "We will be following the [Keras tutorial](https://blog.keras.io/building-autoencoders-in-keras.html) on the topic, which goes much more in depth and breadth than we will here.\n",
    "You are highly encouraged to check out that tutorial if you want to learn about autoencoders in the general sense.\n",
    "\n",
    "## Table of contents\n",
    "\n",
    "* [1. Setup](#1.-Setup)\n",
    "* [2. Data](#2.-Data)\n",
    "* [3. Define Keras Model](#3.-Define-Keras-Model)\n",
    "* [4. Training](#4.-Training)\n",
    "* [5. Explore Results](#5.-Explore-Results)\n",
    "* [6. Deep AutoEncoder](#6.-Deep-AutoEncoder)\n",
    "\n",
    "## 1. Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "c116efcc",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-28T16:09:37.224953Z",
     "iopub.status.busy": "2023-06-28T16:09:37.224683Z",
     "iopub.status.idle": "2023-06-28T16:09:43.418824Z",
     "shell.execute_reply": "2023-06-28T16:09:43.418147Z"
    }
   },
   "outputs": [],
   "source": [
    "try:\n",
    "    import scikeras\n",
    "except ImportError:\n",
    "    !python -m pip install scikeras"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "755c16ea",
   "metadata": {},
   "source": [
    "Silence TensorFlow logging to keep output succinct."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "8f68ddf8",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-28T16:09:43.423392Z",
     "iopub.status.busy": "2023-06-28T16:09:43.422795Z",
     "iopub.status.idle": "2023-06-28T16:09:43.428105Z",
     "shell.execute_reply": "2023-06-28T16:09:43.427553Z"
    }
   },
   "outputs": [],
   "source": [
    "import warnings\n",
    "from tensorflow import get_logger\n",
    "get_logger().setLevel('ERROR')\n",
    "warnings.filterwarnings(\"ignore\", message=\"Setting the random state for TF\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "70c3c94c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-28T16:09:43.431749Z",
     "iopub.status.busy": "2023-06-28T16:09:43.431336Z",
     "iopub.status.idle": "2023-06-28T16:09:44.412015Z",
     "shell.execute_reply": "2023-06-28T16:09:44.411354Z"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from scikeras.wrappers import KerasClassifier, KerasRegressor\n",
    "from tensorflow import keras"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e5995c6",
   "metadata": {},
   "source": [
    "## 2. Data\n",
    "\n",
    "We load the dataset from the Keras tutorial. The dataset consists of images of cats and dogs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "b6970b56",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-28T16:09:44.417783Z",
     "iopub.status.busy": "2023-06-28T16:09:44.416547Z",
     "iopub.status.idle": "2023-06-28T16:09:45.119005Z",
     "shell.execute_reply": "2023-06-28T16:09:45.118361Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "    8192/11490434 [..............................] - ETA: 0s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      " 4603904/11490434 [===========>..................] - ETA: 0s"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\r",
      "11490434/11490434 [==============================] - 0s 0us/step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(60000, 784)\n",
      "(10000, 784)\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.keras.datasets import mnist\n",
    "import numpy as np\n",
    "\n",
    "\n",
    "(x_train, _), (x_test, _) = mnist.load_data()\n",
    "x_train = x_train.astype('float32') / 255.\n",
    "x_test = x_test.astype('float32') / 255.\n",
    "x_train = x_train.reshape((len(x_train), np.prod(x_train.shape[1:])))\n",
    "x_test = x_test.reshape((len(x_test), np.prod(x_test.shape[1:])))\n",
    "print(x_train.shape)\n",
    "print(x_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5805a586",
   "metadata": {},
   "source": [
    "## 3. Define Keras Model\n",
    "\n",
    "We will be defining a very simple autencoder. We define _three_ model architectures:\n",
    "\n",
    "1. An encoder: a series of densly connected layers culminating in an \"output\" layer that determines the encoding dimensions.\n",
    "2. A decoder: takes the output of the encoder as it's input and reconstructs the original data.\n",
    "3. An autoencoder: a chain of the encoder and decoder that directly connects them for training purposes.\n",
    "\n",
    "The only variable we give our model is the encoding dimensions, which will be a hyperparemter of our final transformer.\n",
    "\n",
    "The encoder and decoder are views to the first/last layers of the autoencoder model.\n",
    "They'll be directly used in `transform` and `inverse_transform`, so we'll create some SciKeras models with those layers\n",
    "and save them as in `encoder_model_` and `decoder_model_`. All three models are created within `_keras_build_fn`.\n",
    "\n",
    "For a background on chaining Functional Models like this, see [All models are callable](https://keras.io/guides/functional_api/#all-models-are-callable-just-like-layers) in the Keras docs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "b1ac464b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-28T16:09:45.123916Z",
     "iopub.status.busy": "2023-06-28T16:09:45.122717Z",
     "iopub.status.idle": "2023-06-28T16:09:45.135669Z",
     "shell.execute_reply": "2023-06-28T16:09:45.135039Z"
    }
   },
   "outputs": [],
   "source": [
    "from typing import Dict, Any\n",
    "\n",
    "from sklearn.base import TransformerMixin\n",
    "from sklearn.metrics import mean_squared_error\n",
    "from scikeras.wrappers import BaseWrapper\n",
    "\n",
    "\n",
    "class AutoEncoder(BaseWrapper, TransformerMixin):\n",
    "    \"\"\"A class that enables transform and fit_transform.\n",
    "    \"\"\"\n",
    "\n",
    "    encoder_model_: BaseWrapper\n",
    "    decoder_model_: BaseWrapper\n",
    "    \n",
    "    def _keras_build_fn(self, encoding_dim: int, meta: Dict[str, Any]):\n",
    "        n_features_in = meta[\"n_features_in_\"]\n",
    "\n",
    "        encoder_input = keras.Input(shape=(n_features_in,))\n",
    "        encoder_output = keras.layers.Dense(encoding_dim, activation='relu')(encoder_input)\n",
    "        encoder_model = keras.Model(encoder_input, encoder_output)\n",
    "\n",
    "        decoder_input = keras.Input(shape=(encoding_dim,))\n",
    "        decoder_output = keras.layers.Dense(n_features_in, activation='sigmoid', name=\"decoder\")(decoder_input)\n",
    "        decoder_model = keras.Model(decoder_input, decoder_output)\n",
    "        \n",
    "        autoencoder_input = keras.Input(shape=(n_features_in,))\n",
    "        encoded_img = encoder_model(autoencoder_input)\n",
    "        reconstructed_img = decoder_model(encoded_img)\n",
    "\n",
    "        autoencoder_model = keras.Model(autoencoder_input, reconstructed_img)\n",
    "\n",
    "        self.encoder_model_ = BaseWrapper(encoder_model, verbose=self.verbose)\n",
    "        self.decoder_model_ = BaseWrapper(decoder_model, verbose=self.verbose)\n",
    "\n",
    "        return autoencoder_model\n",
    "    \n",
    "    def _initialize(self, X, y=None):\n",
    "        X, _ = super()._initialize(X=X, y=y)\n",
    "        # since encoder_model_ and decoder_model_ share layers (and their weights)\n",
    "        # X_tf here come from random weights, but we only use it to initialize our models\n",
    "        X_tf = self.encoder_model_.initialize(X).predict(X)\n",
    "        self.decoder_model_.initialize(X_tf)\n",
    "        return X, X\n",
    "\n",
    "    def initialize(self, X):\n",
    "        self._initialize(X=X, y=X)\n",
    "        return self\n",
    "\n",
    "    def fit(self, X, *, sample_weight=None) -> \"AutoEncoder\":\n",
    "        super().fit(X=X, y=X, sample_weight=sample_weight)\n",
    "        # at this point, encoder_model_ and decoder_model_\n",
    "        # are both \"fitted\" because they share layers w/ model_\n",
    "        # which is fit in the above call\n",
    "        return self\n",
    "\n",
    "    def score(self, X) -> float:\n",
    "        # Note: we use 1-MSE as the score\n",
    "        # With MSE, \"larger is better\", but Scikit-Learn\n",
    "        # always maximizes the score (e.g. in GridSearch)\n",
    "        return 1 - mean_squared_error(self.predict(X), X)\n",
    "\n",
    "    def transform(self, X) -> np.ndarray:\n",
    "        X: np.ndarray = self.feature_encoder_.transform(X)\n",
    "        return self.encoder_model_.predict(X)\n",
    "\n",
    "    def inverse_transform(self, X_tf: np.ndarray):\n",
    "        X: np.ndarray = self.decoder_model_.predict(X_tf)\n",
    "        return self.feature_encoder_.inverse_transform(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa964f2f",
   "metadata": {},
   "source": [
    "Next, we wrap the Keras Model with Scikeras. Note that for our encoder/decoder estimators, we do not need to provide a loss function since no training will be done.\n",
    "We do however need to have the `fit_model` and `encoding_dim` so that these will be settable by `BaseWrapper.set_params`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "9bce1d00",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-28T16:09:45.140770Z",
     "iopub.status.busy": "2023-06-28T16:09:45.139503Z",
     "iopub.status.idle": "2023-06-28T16:09:45.144647Z",
     "shell.execute_reply": "2023-06-28T16:09:45.144065Z"
    }
   },
   "outputs": [],
   "source": [
    "autoencoder = AutoEncoder(\n",
    "    loss=\"binary_crossentropy\",\n",
    "    encoding_dim=32,\n",
    "    random_state=0,\n",
    "    epochs=5,\n",
    "    verbose=False,\n",
    "    optimizer=\"adam\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5974f464",
   "metadata": {},
   "source": [
    "## 4. Training\n",
    "\n",
    "To train the model, we pass the input images as both the features and the target.\n",
    "This will train the layers to compress the data as accurately as possible between the encoder and decoder.\n",
    "Note that we only pass the `X` parameter, since we defined the mapping `y=X` in `KerasTransformer.fit` above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "b45a1a6b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-28T16:09:45.149216Z",
     "iopub.status.busy": "2023-06-28T16:09:45.148001Z",
     "iopub.status.idle": "2023-06-28T16:10:17.253182Z",
     "shell.execute_reply": "2023-06-28T16:10:17.252288Z"
    }
   },
   "outputs": [],
   "source": [
    "_ = autoencoder.fit(X=x_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de1837e3",
   "metadata": {},
   "source": [
    "Next, we round trip the test dataset and explore the performance of the autoencoder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "bf663af0",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-28T16:10:17.257799Z",
     "iopub.status.busy": "2023-06-28T16:10:17.257052Z",
     "iopub.status.idle": "2023-06-28T16:10:18.706874Z",
     "shell.execute_reply": "2023-06-28T16:10:18.706041Z"
    }
   },
   "outputs": [],
   "source": [
    "roundtrip_imgs = autoencoder.inverse_transform(autoencoder.transform(x_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd48ba4c",
   "metadata": {},
   "source": [
    "## 5. Explore Results\n",
    "\n",
    "Let's compare our inputs to lossy decoded outputs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "7391520a",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-28T16:10:18.710854Z",
     "iopub.status.busy": "2023-06-28T16:10:18.710603Z",
     "iopub.status.idle": "2023-06-28T16:10:21.654487Z",
     "shell.execute_reply": "2023-06-28T16:10:21.653708Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABiEAAAE/CAYAAAAg+mBzAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABOqElEQVR4nO3dd7wdVbk38AkQQiABUggQEkIA6SUYQPQCFxFFUBAUhAtyFRAbWK6KBREQLvC5qGAH8bWiKEqRKlKkioiRJl1qCAkhQAip1Lx/ve911vPoGU727HOSfL//reez9pyVs9dZM7NX9vwGLFy4cGEFAAAAAADQYcv09QAAAAAAAIAlk00IAAAAAACgFTYhAAAAAACAVtiEAAAAAAAAWmETAgAAAAAAaIVNCAAAAAAAoBU2IQAAAAAAgFbYhAAAAAAAAFqxXJNOr776ajV16tRq6NCh1YABA9oeE/3YwoULq9mzZ1ejR4+ullmm3T0s847/p1vzzpzjH5l3dJtzLH3BWke3WevoC9Y6+oJ5R7c5x9IXms67RpsQU6dOrcaOHduxwbH4e/zxx6sxY8a0+jPMO0ptzztzjox5R7c5x9IXrHV0m7WOvmCtoy+Yd3Sbcyx9oad512hbbOjQoR0bEEuGbswJ845S23PCnCNj3tFtzrH0BWsd3Watoy9Y6+gL5h3d5hxLX+hpTjTahPC1GkrdmBPmHaW254Q5R8a8o9ucY+kL1jq6zVpHX7DW0RfMO7rNOZa+0NOcEEwNAAAAAAC0wiYEAAAAAADQCpsQAAAAAABAK2xCAAAAAAAArbAJAQAAAAAAtMImBAAAAAAA0AqbEAAAAAAAQCtsQgAAAAAAAK2wCQEAAAAAALTCJgQAAAAAANCK5fp6ALCk+uxnPxtqgwcPDrUtttii1t5nn30aHf/000+vtf/0pz+FPmeddVajYwEAAAAAtME3IQAAAAAAgFbYhAAAAAAAAFphEwIAAAAAAGiFTQgAAAAAAKAVgqmhA84555xQaxowXXr11Vcb9fvwhz9ca++yyy6hz3XXXRdqkydP7tW4oLTBBhuE2n333Rdqn/zkJ0Pt29/+ditjov9aaaWVau2vfvWroU+5rlVVVf31r3+ttffdd9/Q57HHHlvE0QEAAEurYcOGhdraa6/dq2Nl9yb/9V//VWvfddddoc8DDzwQanfccUevxgD9kW9CAAAAAAAArbAJAQAAAAAAtMImBAAAAAAA0AqbEAAAAAAAQCsEU0MvlEHUvQ2hrqoY5Pv73/8+9Fl33XVDbY899qi111tvvdDnwAMPDLWTTz75tQ4RUltttVWoZcHqU6ZM6cZw6OfWXHPNWvuwww4LfbL5M3HixFr7ne98Z+jz3e9+dxFHx+Lm9a9/faidf/75obbOOut0YTT/2tve9rZa+9577w19Hn/88W4Nh8VEeZ1XVVV10UUXhdoRRxwRameccUat/corr3RuYLRm1KhRofbrX/861G666aZQO/PMM2vtRx99tGPj6qRVVlkl1Hbcccda+/LLLw99XnrppdbGBCz53vGOd9Tae+65Z+iz0047hdr666/fq5+XBUyPGzeu1h40aFCjYy277LK9GgP0R74JAQAAAAAAtMImBAAAAAAA0AqbEAAAAAAAQCtkQkAPtt5661Dbe++9e3zd3XffHWrZsweffvrpWnvOnDmhz/LLLx9qN998c6295ZZbhj4jRozocZzQWxMmTAi1uXPnhtoFF1zQhdHQn6y22mqh9tOf/rQPRsKSatdddw21ps/W7bby2f6HHHJI6LP//vt3azj0U+U12/e+971Gr/vOd74Taj/60Y9q7fnz5/d+YLRm2LBhtXZ275BlKEyfPj3U+mMGRDb2v/71r6FWXjOUWVBVVVUPPvhg5wbGa7byyiuHWpkzuNlmm4U+u+yyS6jJ92BRlDmYhx9+eOiT5c4NHjy41h4wYEBnB1bYYIMNWj0+LK58EwIAAAAAAGiFTQgAAAAAAKAVNiEAAAAAAIBW2IQAAAAAAABa0W+DqffZZ59QywJmpk6dWmsvWLAg9PnFL34Rak8++WSoCbwis+aaa4ZaGWSUBclloZnTpk3r1Rg+85nPhNomm2zS4+suvfTSXv08yJSBc0cccUToc9ZZZ3VrOPQTn/jEJ0Jtr732CrVtt922Iz9vxx13DLVllon/p+KOO+4Iteuvv74jY6C7llsuXq7uvvvufTCS3imDWD/96U+HPiuttFKozZ07t7Ux0f+Ua9uYMWMave6Xv/xlqGX3Q/StkSNHhto555xTaw8fPjz0yQLKP/7xj3duYC06+uijQ238+PGh9uEPf7jWdk/etw488MBQO/HEE0Nt7NixPR4rC7R+5plnejcwqOK58ZOf/GQfjeR/3XfffaGWfT7EkmP99dcPtew8v/fee9faO+20U+jz6quvhtoZZ5wRan/84x9r7cX1XOmbEAAAAAAAQCtsQgAAAAAAAK2wCQEAAAAAALTCJgQAAAAAANCKfhtMfcopp4TaOuus06tjlWFXVVVVs2fPDrX+GB4zZcqUUMt+N5MmTerGcJZKF198caiVQTTZfHr22Wc7Nob9998/1AYOHNix40MTG220Ua2dBamWIYss+U477bRQywK2OuXd7353o9pjjz0Wavvtt1+tXQYG0z+9+c1vDrU3vvGNoZZdH/UHw4YNq7U32WST0GfFFVcMNcHUS65BgwaF2pe+9KVeHeuss84KtYULF/bqWLTn9a9/fahlAZWl448/voXRtGPTTTettT/zmc+EPhdccEGouXbsO2XIb1VV1Te+8Y1QGzFiRKg1WWe+/e1vh9oRRxxRa3fynpn+qQzszcKky9Ddqqqqyy+/PNReeOGFWnvWrFmhT3b9VN63XnHFFaHPXXfdFWp//vOfQ+22226rtefPn99oDCweNttss1Ar163s3jMLpu6tN7zhDaH28ssv19r3339/6HPjjTeGWvn39uKLLy7i6BaNb0IAAAAAAACtsAkBAAAAAAC0wiYEAAAAAADQin6bCXHYYYeF2hZbbBFq9957b6298cYbhz5Nn8G53Xbb1dqPP/546DN27NhQa6J8fldVVdWMGTNCbc011+zxWJMnTw41mRDdlT1rvFOOPPLIUNtggw16fF32vMKsBr31uc99rtbO/g6sRUu2yy67LNSWWabd/8/wzDPP1Npz5swJfcaNGxdq48ePD7Vbbrml1l522WUXcXS0oXwW6y9/+cvQ56GHHgq1k046qbUxLYp3vetdfT0E+pnNN9881CZOnNjj67L7id/97ncdGROdM2rUqFB7z3ve0+PrDj300FDL7hf7gzL/oaqq6qqrrurxdVkmRJatR3d89rOfDbXhw4d37PhlFldVVdXb3/72WvvEE08MfbIsib5+jjnNZJmBZf7ClltuGfrsvffejY5/880319rZZ32PPvpoqK299tq1dpa92mamHX0v+zz58MMPD7Vs3Vp55ZV7PP4TTzwRajfccEOt/cgjj4Q+5WcsVZXnFm677ba1drZW77777qF2xx131NpnnHFG6NNNvgkBAAAAAAC0wiYEAAAAAADQCpsQAAAAAABAK2xCAAAAAAAArei3wdRXX311o1rp8ssvb3T8YcOGhdqECRNq7SwMZJtttml0/NKCBQtC7YEHHgi1Mmg7CxvJwhhZfL3zne+stY8//vjQZ/nllw+1p556qtb+4he/GPrMmzdvEUfH0mqdddYJta233rrWztawuXPntjUk+sC///u/19obbrhh6JOFuPU22C0LyirD7GbNmhX67LzzzqH2pS99qcef99GPfjTUTj/99B5fR7uOPvroWjsLOSyDLasqDy3vtuy6rfw7EnxIk5DiTLke0j99/etfD7X3ve99oVbea/7mN79pbUydtsMOO4Ta6quvXmv/5Cc/CX1+/vOftzUkGhg3blytffDBBzd63Z133hlq06dPr7V32WWXRsdaZZVVau0sHPsXv/hFqD355JONjk/3ZJ9RnH322aFWBlGfdNJJoU+TYPtMFkKdmTx5cq+Oz+Lr+9//fq2dhZ+PHDmy0bHKz6L/9re/hT5HHXVUqGWfA5fe9KY3hVp2j/qjH/2o1i4/v66quC5XVVV997vfrbXPO++80GfGjBk9DbNjfBMCAAAAAABohU0IAAAAAACgFTYhAAAAAACAVtiEAAAAAAAAWtFvg6nbNnPmzFC75pprenxdk3DsprJQujIwOws8Oeecczo2BvpeGfabBTxlynlw3XXXdWxMUAapZroZYET7sjDyX/3qV7V20/CuzGOPPVZrZ6FYX/nKV0Jt3rx5r/nYVVVVH/rQh0JttdVWq7VPOeWU0GeFFVYIte985zu19ksvvdTjmGhmn332CbXdd9+91n7wwQdDn0mTJrU2pkWRBaKXQdTXXntt6PPcc8+1NCL6ox133LHHPi+++GKoZfOL/mfhwoWhlgXST506tdbO3vNuGzx4cKhlYZsf+9jHQq38dx9yyCGdGxgdUQaZDh06NPS54YYbQi27Lyivl/7jP/4j9MnmznrrrVdrr7HGGqHPhRdeGGq77bZbqD377LOhRnuGDBlSa3/xi18Mfd75zneG2tNPP11rf+1rXwt9mlzvQ1Xl92qf+9znQu2DH/xgrT1gwIDQJ/s84/TTTw+1r371q7X23LlzexxnUyNGjAi1ZZddNtSOO+64Wvvyyy8PfcaNG9excbXFNyEAAAAAAIBW2IQAAAAAAABaYRMCAAAAAABohU0IAAAAAACgFUttMHW3jRo1KtS+973vhdoyy9T3hY4//vjQRwDT4uu3v/1tqL3tbW/r8XU/+9nPQu3oo4/uxJAgtfnmm/fYJwv1ZfG13HLxkqC3QdTXXXddqO2///61dhlStyiyYOqTTz451E499dRae8UVVwx9snl90UUX1doPPfTQax0i/8S+++4bauX7kl0v9QdZmPuBBx4Yaq+88kqt/d///d+hj7DzJdeb3vSmRrVSFnp4++23d2JI9BPveMc7au0rrrgi9MlC67PQzN4qA4d32mmn0Ge77bZrdKxzzz23E0OiRYMGDaq1sxD10047rdGxFixYUGv/+Mc/Dn2yc/y6667b47GzkOL+ENy+tNtrr71q7S984Quhz+TJk0Nthx12qLVnzZrV0XGxdMnOU0ceeWSolUHUTzzxROjznve8J9RuueWW3g+uUAZMjx07NvTJPuu77LLLQm3YsGE9/rwsfPuss86qtbPrim7yTQgAAAAAAKAVNiEAAAAAAIBW2IQAAAAAAABaIROiSw4//PBQW2211UJt5syZtfb999/f2pho15prrhlq2TOAy2dzZs9Jz54fPWfOnEUYHfyv7Fm/Bx98cKjddttttfaVV17Z2phYfEyaNCnUDjnkkFDrZAZEE2WOQ1XF5/Vvs8023RoOVVWtssoqodbkWeOdfP55J33oQx8KtSxH5d577621r7nmmtbGRP/T23Wmv857evbNb34z1N785jeH2ujRo2vtHXfcMfTJnu+85557LsLo/vXxs4yAzMMPPxxqRx11VEfGRHv+4z/+o8c+ZVZJVeW5hk1svfXWvXrdzTffHGrufftekzyj8n6xqqpqypQpbQyHpVSZs1BVMX8t8/LLL4faG97whlDbZ599Qm2jjTbq8fjz588PtY033vhftqsqv0deffXVe/x5menTp4da+VliX+fQ+SYEAAAAAADQCpsQAAAAAABAK2xCAAAAAAAArbAJAQAAAAAAtEIwdQv+7d/+LdS+8IUvNHrtXnvtVWvfddddnRgSfeC8884LtREjRvT4up///Oeh9tBDD3VkTJDZZZddQm348OGhdvnll9faCxYsaG1M9A/LLNPz/1XIAr36gyzMs/z3NPn3VVVVHXfccbX2QQcd1OtxLc0GDRoUamuttVao/fKXv+zGcBbZeuut16ifa7mlW9Ng1ueee67WFky9+PrrX/8aaltssUWoTZgwodZ++9vfHvoceeSRoTZjxoxQ++lPf/oaRvi/zjrrrFr7jjvuaPS6m266KdTcr/R/5fk1CznfZpttQi0LZd18881r7b333jv0GTZsWKiVa13W57DDDgu1cq5WVVXdc889oUZ7ssDeUraOHXvssbX2hRdeGPrcfvvtvR4XS5c//OEPoXbNNdeEWvkZx9prrx36fOtb3wq1hQsX9jiGLAg7C8xuomkI9auvvlprX3DBBaHPJz7xiVCbNm1ar8bVFt+EAAAAAAAAWmETAgAAAAAAaIVNCAAAAAAAoBU2IQAAAAAAgFYIpm7B7rvvHmoDBw4MtauvvjrU/vSnP7UyJtqVhXq9/vWvb/Taa6+9ttYug5ugbVtuuWWoZYFM5557bjeGQx/5yEc+EmplANbiZI899gi1rbbaqtbO/n1ZrQympndmz54dalkQYRngOnz48NDn2Wef7di4mhg1alSoNQlorKqquvHGGzs9HPqx7bffvtY+4IADGr1u1qxZtfaUKVM6Nib63syZM0OtDNLMgjU///nPtzamqqqqddddt9YeMGBA6JOt05/97GfbGhItuuqqq2rtct2pqhg4XVV5AHST8Nby51VVVR1++OG19iWXXBL6vO51rwu1LHA1u3alPauttlqtnV0zDxo0KNSOOeaYWvvoo48Ofc4444xQu/nmm0OtDBd+8MEHQ5+777471EqbbrppqGWfxTkX9z/z588Ptb333jvUVl111Vr7C1/4Qujzb//2b6H2zDPPhNrkyZNr7WyeZ5+pbLvttqHWW2eeeWatfdRRR4U+zz33XMd+Xlt8EwIAAAAAAGiFTQgAAAAAAKAVNiEAAAAAAIBWyITogMGDB9fab3/720OfF198MdSyZ/+/9NJLnRsYrRkxYkStnT2PLcsByZTPWZ0zZ06vxwVNrLHGGrX2DjvsEPrcf//9oXbBBRe0Nib6Xpah0B+Vz6OtqqraZJNNQi1bl5uYMWNGqDk3d0b2DNeHHnoo1N7znvfU2pdeemnoc+qpp3ZsXJtttlmolc9JX2eddUKfJs/DrqrFO1uF1668RlxmmWb/5+vKK69sYzjwL5XPas/WtSyXIjtX0v+VeUrvfe97Q58sA26VVVbp8djf/va3Qy2bOwsWLKi1zz///NAne3b7rrvuGmrrrbderZ1dU9A5X/va12rtT3/60706TnZe/NjHPtao1qZsXSvzO6uqqvbff/8ujIZFVeYjZOtKJ/3sZz8LtSaZEFlmXva39ZOf/KTWfuWVV5oPrh/xTQgAAAAAAKAVNiEAAAAAAIBW2IQAAAAAAABaYRMCAAAAAABohWDqDjjyyCNr7a222ir0ufzyy0Ptpptuam1MtOszn/lMrb3NNts0et1vf/vbUMsCyqFNH/jAB2rtUaNGhT6/+93vujQaeG2+9KUvhdrhhx/eq2M9+uijofb+978/1CZPntyr49Oz7Bw4YMCAWvsd73hH6PPLX/6yY2N4+umnQ60MZx05cmSvj18GybFk22effXrsU4YlVlVVff/7329hNPC/9t1331D7z//8z1o7C8h85plnWhsTfeuqq64KtWwNO+CAA0KtXMfKkPOqiiHUmRNOOCHUNt5441Dbc889Q638mdk1HJ1TBvuec845oc/ZZ58dasstV//YcezYsaFPFlbdbauttlqoZX8PRx99dK393//9362Nif7pc5/7XKj1NrD8Ix/5SKh18j6nv+n7v3QAAAAAAGCJZBMCAAAAAABohU0IAAAAAACgFTYhAAAAAACAVgimfo2ycMQvf/nLtfbzzz8f+hx//PGtjYnu+/SnP92r1x1xxBGhNmfOnEUdDrwm48aN67HPzJkzuzAS6Nlll11Wa2+44YYdO/Y999wTajfeeGPHjk/P7rvvvlB773vfW2tPmDAh9Fl//fU7NoZzzz23xz4//elPQ+3AAw9sdPz58+e/5jGxeBgzZkyoZQGupSlTpoTapEmTOjIm+Gd22223HvtccskloXbrrbe2MRz6qSysOqt1SnaOzAKPs2DqN7/5zbX28OHDQ59nn312EUbHP3rllVdq7ey8tcEGG/R4nLe85S2hNnDgwFA77rjjQm2bbbbp8fidNGDAgFCbOHFiV8dA3/vgBz9Ya5fh5FUVA9gzd999d6idf/75vR/YYsg3IQAAAAAAgFbYhAAAAAAAAFphEwIAAAAAAGiFTQgAAAAAAKAVgqn/hREjRoTat771rVBbdtlla+0yRLOqqurmm2/u3MBYbGVhWS+99FJHjj1r1qxGx85Cn1ZZZZUej7/qqquGWm8DustQq6qqqs9//vO19rx583p1bHr2zne+s8c+F198cRdGQn+SBa8ts0zP/1ehSdBlVVXVmWeeWWuPHj260evKMbz66quNXtfEHnvs0bFj0Z7bb7+9Ua1NDz/8cK9fu9lmm9Xad91116IOh37iTW96U6g1WTd/+9vftjAa+Ney8/XcuXNr7a9//evdGg78U7/+9a9DLQum3m+//WrtI444IvQ5/vjjOzcwOuLqq69u1G/ChAmhVgZTv/zyy6HPj3/841D7wQ9+UGt/6lOfCn0OOOCARuNiybbtttuGWnluHDJkSKNjzZkzp9b+yEc+Evq88MILr2F0iz/fhAAAAAAAAFphEwIAAAAAAGiFTQgAAAAAAKAVMiH+QZntcPnll4c+48ePD7WHHnqo1v7yl7/c2YGxxLjzzjtbO/ZvfvObUJs2bVqorb766qFWPk+zLzz55JO19oknnthHI1mybL/99qG2xhpr9MFI6O9OP/30UDvllFN6fN0ll1wSak1yG3qb7bAomRBnnHFGr1/L0i3LTMlqGRkQS64sP6709NNPh9o3v/nNNoYD/1/23OnsHuCpp56qtW+99dbWxgRNZdd62TXpu971rlr72GOPDX1+9atfhdoDDzywCKOjW6644opQKz8jWG65+JHmYYcdFmrrr79+rb3TTjv1elxTpkzp9Wvp/7LMwKFDh/b4ujJjqapils0f//jH3g9sCeGbEAAAAAAAQCtsQgAAAAAAAK2wCQEAAAAAALTCJgQAAAAAANAKwdT/YL311qu1J06c2Oh1n/70p2vtMqiaJc9ll11Wa5ehWH1h33337dixXn755VBrEgZ70UUXhdqkSZMa/cwbbrihUT9em7333jvUll122Vr7tttuC32uv/761sZE/3T++eeH2pFHHllrr7baat0azj81Y8aMULv33ntD7UMf+lCoTZs2rZUxseRbuHBhoxpLl1133bXHPpMnTw61WbNmtTEc+P+yYOpszbr00kt7PFYWyDls2LBQy+Y6dMrtt98easccc0yt/dWvfjX0Oemkk0LtoIMOqrXnz5+/aIOjFdn1/a9//eta+73vfW+jY735zW/usc8rr7wSatka+YUvfKHRz6T/y85vn/vc53p1rF/84hehdu211/bqWEsy34QAAAAAAABaYRMCAAAAAABohU0IAAAAAACgFTYhAAAAAACAViy1wdTjxo0LtSuuuKLH15UhnVVVVZdccklHxsTi493vfnetnYXXDBw4sFfH3nTTTUNtv/3269WxfvSjH4Xao48+2uPrzjvvvFC77777ejUGumfFFVcMtd13373H15177rmhlgVzsWR77LHHQm3//fevtffaa6/Q55Of/GRbQ0qdeOKJofbd7363q2Ng6bPCCis06ifccsmVXdett956Pb5uwYIFofbSSy91ZEywqMrrvQMPPDD0+a//+q9Qu/vuu0Pt/e9/f+cGBg387Gc/q7U//OEPhz7lfXtVVdXxxx9fa995552dHRgdkV1TfepTn6q1hwwZEvpsvfXWoTZq1KhaO/tM5Kyzzgq144477l8PksVGNlfuueeeUGvyOV62ZpRzk5xvQgAAAAAAAK2wCQEAAAAAALTCJgQAAAAAANCKpTYT4kMf+lCorb322j2+7rrrrgu1hQsXdmRMLL5OOeWUVo9/wAEHtHp8lgzZM6ZnzpwZahdddFGt/c1vfrO1MbF4u/766/9lu6ryPKXsHLvHHnvU2uU8rKqqOvPMM0NtwIABtXb27E5o28EHHxxqzz33XKidcMIJXRgNfeHVV18NtUmTJoXaZpttVms/+OCDrY0JFtUHP/jBWvvQQw8NfX74wx+GmrWO/mDGjBm19i677BL6ZM/+//znP19rZ1ko9E/Tp0+vtcv7i6qqqoMOOijUtttuu1r7K1/5Sujz1FNPLeLo6M923nnnUBszZkyoNfl8N8tKyjLAiHwTAgAAAAAAaIVNCAAAAAAAoBU2IQAAAAAAgFbYhAAAAAAAAFqxVARTb7/99qH28Y9/vA9GAtCeLJj6TW96Ux+MhKXJ5Zdf3qgGi7O//OUvoXbqqaeG2jXXXNON4dAHXnnllVD70pe+FGploOFf//rX1sYE/8wRRxwRascff3yoXX/99bX26aefHvrMnDkz1F588cVFGB20Y/LkyaF21VVXhdqee+5Za2+yySahzz333NO5gdFVZ511VqMaS5cTTjgh1JqEUFdVVX31q1+ttV3v955vQgAAAAAAAK2wCQEAAAAAALTCJgQAAAAAANAKmxAAAAAAAEArlopg6h122CHUhgwZ0uPrHnrooVCbM2dOR8YEAMDiYY899ujrIdAPTZ06NdQOOeSQPhgJ1N14442htvPOO/fBSKBv7bPPPqF2xx131Nrrr79+6COYGpYsw4cPD7UBAwaE2lNPPRVq3/jGN9oY0lLJNyEAAAAAAIBW2IQAAAAAAABaYRMCAAAAAABohU0IAAAAAACgFUtFMHVTZUDRW97yltDn2Wef7dZwAAAAAOiF559/PtTGjx/fByMB+tKpp57aqHbCCSeE2rRp01oZ09LINyEAAAAAAIBW2IQAAAAAAABaYRMCAAAAAABoxVKRCXHyySc3qgEAAAAAsGQ47bTTGtVol29CAAAAAAAArbAJAQAAAAAAtMImBAAAAAAA0IpGmxALFy5sexwsZroxJ8w7Sm3PCXOOjHlHtznH0hesdXSbtY6+YK2jL5h3dJtzLH2hpznRaBNi9uzZHRkMS45uzAnzjlLbc8KcI2Pe0W3OsfQFax3dZq2jL1jr6AvmHd3mHEtf6GlODFjYYOvq1VdfraZOnVoNHTq0GjBgQMcGx+Jn4cKF1ezZs6vRo0dXyyzT7tO8zDv+n27NO3OOf2Te0W3OsfQFax3dZq2jL1jr6AvmHd3mHEtfaDrvGm1CAAAAAAAAvFaCqQEAAAAAgFbYhAAAAAAAAFphEwIAAAAAAGiFTQgAAAAAAKAVNiEAAAAAAIBW2IQAAAAAAABaYRMCAAAAAABohU0IAAAAAACgFTYhAAAAAACAVtiEAAAAAAAAWmETAgAAAAAAaIVNCAAAAAAAoBU2IQAAAAAAgFbYhAAAAAAAAFphEwIAAAAAAGiFTQgAAAAAAKAVNiEAAAAAAIBW2IQAAAAAAABaYRMCAAAAAABohU0IAAAAAACgFTYhAAAAAACAVtiEAAAAAAAAWmETAgAAAAAAaIVNCAAAAAAAoBU2IQAAAAAAgFbYhAAAAAAAAFphEwIAAAAAAGiFTQgAAAAAAKAVNiEAAAAAAIBW2IQAAAAAAABaYRMCAAAAAABohU0IAAAAAACgFTYhAAAAAACAVizXpNOrr75aTZ06tRo6dGg1YMCAtsdEP7Zw4cJq9uzZ1ejRo6tllml3D8u84//p1rwz5/hH5h3d5hxLX7DW0W3WOvqCtY6+YN7Rbc6x9IWm867RJsTUqVOrsWPHdmxwLP4ef/zxasyYMa3+DPOOUtvzzpwjY97Rbc6x9AVrHd1mraMvWOvoC+Yd3eYcS1/oad412hYbOnRoxwbEkqEbc8K8o9T2nDDnyJh3dJtzLH3BWke3WevoC9Y6+oJ5R7c5x9IXepoTjTYhfK2GUjfmhHlHqe05Yc6RMe/oNudY+oK1jm6z1tEXrHX0BfOObnOOpS/0NCcEUwMAAAAAAK1olAkBAAAAAN2W/e/ahQsX9sFIAOgt34QAAAAAAABaYRMCAAAAAABohU0IAAAAAACgFTIhoI/1lB5fVZ53CQAAwNLJ/TDA4s83IQAAAAAAgFbYhAAAAAAAAFphEwIAAAAAAGiFTQgAAAAAAKAVgqnhH2Qh0cssE/fqhg8fXmt/6lOfCn322muvUFtppZVCbdCgQbX2ww8/HPo899xzofbMM8/U2hdccEHoc/XVV4fa/PnzQ+2VV16ptV999dXQB8q/hexvI/sbyoLkyjknbI6qyudUuUa+8MILoY81CwCg/8uu9Xr7uvL+oby/gDaU97tN5mZVuV+BqvJNCAAAAAAAoCU2IQAAAAAAgFbYhAAAAAAAAFphEwIAAAAAAGiFYGqWak2CsUaOHBlqhxxySK293377hT5jx44NtYEDB4ZaGWy0xhprhD5ZiNHcuXNr7fXWWy/0eeqpp0Lt9ttvD7UsrLpTll122VDL/j2CiftOFia9wgorhFo5p7fYYovQZ8011wy12267LdTuuuuuWnvWrFmhjzmxeMjW0WytK9fS//zP/wx93vrWt4batGnTau0zzzwz9PnjH/8Yai+//HIcLIulbI3K5l3Wr4km56RsPcp+XpMxCCYkmyflurnSSiuFPiuuuGKoPf/886E2b968WltY6+Ihu2Zebrl4u56tIeU5r79eQzVZI/vr2OmMcv42PXdn86LJvXz2uiZzzDxc+mTzaeWVVw61N77xjT0eK/scpqy99NJLoc+CBQtCzX0ySxLfhAAAAAAAAFphEwIAAAAAAGiFTQgAAAAAAKAVNiEAAAAAAIBWCKZmqVYG+iy//PKhz+DBg0OtDHKeMWNG6DNkyJBQy8LlymMNGjQo9MkCVqdPn15r33zzzaHPc889F2ovvPBCqLUZkimEunuyYLfeBq9lc658L9ddd93QJ6tloZkPPfRQrT179uzQR5Bm/5PNsWzNGj16dKiVodPvfe97Q5/hw4eHWrmWjho1KvTJgrAFUy++ymDALKB39dVXD7XsHD5z5sxaOzsvvvjii6FWrotNw7Gzv4eyVoYGV1UeTijAun/p7Tm26bFWWWWVWvvd73536LPjjjuG2i233BJqZ599dq399NNPhz6uxborC50eNmxYrZ2FnY4ZMybUbrvttlC79957a+05c+aEPp28rmpyPZCt3eW/uariXMzW6azmOrF7moZHlzoZAJ293+Vrs/Nydv9d/nuy64fsHDx37twex0C7mrx3m222Wah95CMfqbW333770GettdYKtey6rhxD9vlKdl1Xzp9HH3009CnP31VVVWeccUaoweLKNyEAAAAAAIBW2IQAAAAAAABaYRMCAAAAAABoRb/JhCifq9b0eX4jR47ssU/2rPFZs2aFmudKLn3KZzhmz+7LnqM7ZcqUWvvKK68MfSZPnhxqf/vb30Ltscceq7Wz5xpOnDgx1Hbbbbdae8UVVwx9smdUdvu5ldnPa/JcUc/XrOvts1h7K1sPy2fsl+tvVeXP7L377rtD7Zlnnqm1Pft88ZXNzU022STUjjzyyFp77NixjY5VPj96jz32CH3+8Ic/hNqCBQtCzbqyeCjPZ69//etDn2233TbUsmeg//GPf6y1szWqt9lF2euytbOcwyNGjAh9ypynqorPDrZOtid7Xn+p7Wuq173udbX25z//+dBn1VVXDbWVV1451H7xi190bFz0rDx3ZfNpjTXWCLWTTz651t58881Dnyx3Lpt3jzzySK2dZc9kymM1nefZ+Xro0KG19oQJE0Kf9ddfP9TKsd95552hT/bZgHv3RZe9j9n8zbIOy7ySLF+pvM/N+i3KOlqeF5seq5xP2TP9sywA87C7mmTP7L///qHPqaeeGmrl+bPpvXWTOZV9Bpkps5+yLLxsPv30pz8NtaZrPH2ryfVB08yxcm4srve1vgkBAAAAAAC0wiYEAAAAAADQCpsQAAAAAABAK2xCAAAAAAAAreiTYOqBAweGWhnat80224Q+2223XaiV4VZZqMe9994bamVQYVXFIOEsoCgL/yj7ZeHGWcBMGd5VVTGssAwlrKo8vLAMY1xcQ0r6WtMwtttvv73WnjRpUuiTvU9NglKzsJrRo0eH2hve8IZaOws2ygI4TzzxxFDLwjzblIV6NQnGW1pk61i3w7yzn1cG0G222Wahz5///OdQ+/vf/x5q5Tq5NL/fi5PsfRo1alSoHX300aE2bty4WrtJEGxVVdVKK61Ua++5556hT7aGlYGfVVVVTzzxRK0t6LfvZeeDMsB1v/32C3023njjULvwwgtDbdq0abV2FpzZ2/Wn6evKuZ+FtZbXFVVVVTfffHOtLWz9tWt6Pm1yju3kepGtf4ccckitvfbaa/f6+O4L2pOtWeX8yc6LH/7wh0Nthx12qLUHDx4c+mTXUFdddVWoPfvss7V22+e37G+mvI/dfvvtQ5+RI0eG2qOPPlprP//886HPyy+//BpHSBPZWlTOy6qqqmOPPTbUyvuA7DOQs88+O9T+53/+p9ZuGr7eRPa67G+hrGXzOZtz1tKe9faeNXtdFvi85ZZb1trHHXdc6LPyyiv3ePxsDNkcnjlzZqjNnz+/1l5xxRVDnzKEOlN+/lhVVfXBD36wx59H7zQNI8/6Lb/88rV2+blIVeVr57777ltrZ5+fZPNu6tSpofbd73631s6uBbJ7hfLeJ1vbunlP7JsQAAAAAABAK2xCAAAAAAAArbAJAQAAAAAAtMImBAAAAAAA0IrWg6mzsKMsuGWDDTaotSdOnBj6ZMHUa621Vq09aNCgHo9dVVX1pje9KdTKwOx11lkn9MlCbsqgj0ceeST0+dvf/hZqq622WqiNHz++1i5Dxqqqqk455ZRQu+aaa2rtLNyEnjUNKJo1a9a/bFdVHn7ZW2WgTVVV1ete97paOwvQyYJvskD0MgC97dCtpqFh/HNNg5V6+15ma+nOO+9ca5frb1VV1f333x9qWWhwb8fVJFSM9pShXFVVVSeddFKoZefwpkHUpXJtWGGFFUKfPfbYo9EYfvOb39TaP/7xj0Of5557LtReeeWVnoZJLzUJxXzb294W+syePTvU7rnnnlArA06zc022nvZ2bclCazfddNNae9dddw19st9DGVadhc3xrzUNv2xyTu3kPBkyZEiolefYbE5kYYLlulZVVfXCCy/0alylTv6blxTZ76T8u8/e3w033DDUymvyp59+OvQpQ3yrqqoefvjhUGtyHd3bwNimyvvY7H77wQcfDLXy2nHevHkdHRf/q5yrWZDq+eefH2pZyG45n7I5+P73vz/UynPZt7/97dBn+vTpodbbOeAesz3ZmlJ+ppb1a3pdPXjw4FDbZJNNau0swPfxxx8PtfJ6MPucLZv71157baiV5+Lsc71ddtkl1MqxZ/chDz30UKhZ/3qnXO+y+TR8+PBQmzBhQqgdfPDBtXb22XS2TpZjaHptN2LEiFD7xje+UWs/+uijoc8NN9wQauedd16tfe+994Y+5d/HP9OJueibEAAAAAAAQCtsQgAAAAAAAK2wCQEAAAAAALSi9UyI7Bl88+fPD7UyRyF7llX2nN2111671s6e85U90z97pvSoUaN6PFb2nPRS9jzp7Dm+WebEeuutV2tnz5crnyVXVTETgs7J5nD5HncygyN7zv5BBx0UauXzFmfMmBH6lM+Nq6r8ebPd5tmc/1qTZ1g3zYRo8jznbG0dOXJkqJXPMc9ed99994VaJ9/v8t+TjSH7eZ6l2RnZs/n32WefUGuS/5C9J9lzzMtMkex8mv28cePGhdqRRx5Za++4446hzzHHHBNqd955Z61tDeucLCfsXe96V62dZRllzzMt36eqiufnJs9zzzR9fvFKK60UajvttFOtPXbs2NAn+zeWY7eOdUb2e2xyXmx63i1lxypzvaqqqtZYY40ej5U9s/eyyy4LtU7lLjnHRk1yZbLfW/as6DL74JJLLgl97rjjjkZj6K0m710291ddddVQO+SQQ2rt7D720ksvDbUnnnii1naObU/5vv3gBz/osc8/U86d7Bouy0jcfffda+3suu7MM88MtWeeeSbUynPz0rQW9QfLLRc/Tiw/n6uqeG2UPcu+zKisqnxuXHTRRbX29ddfH/pkn8c1uR7MPqdskvOZ5R9meQ/lZ4nZsc3h3snuBUePHl1rf/CDHwx93v72t4fammuuGWrl58fZZ8VZtkM5F7P84KeeeirUsvvYjTfeuNbecsstQ5/y31xVMYtp2rRpoU/TDE+ZEAAAAAAAQL9lEwIAAAAAAGiFTQgAAAAAAKAVNiEAAAAAAIBWtB5MnQVXZIEdZVhuFrz2wAMPhFoZXJUFBGchJcsvv3yolYE5W221VY8/r6piYNiTTz4Z+owYMSLUsrCRDTbYoNbOflezZs1qNC46o+kc7q0yvO6LX/xi6JMF3zz++OO19gEHHBD6TJo0KdQ6OXa6p82QqiyYKws1KsOqZ8+eHfpkAemdCsjMCO9qV7n2nH766aHPwIEDGx2rDCy86aabQp+zzjor1Mq1bsMNNwx9ytD0qqqqTTbZJNTKc/GECRNCnyOOOCLUDj/88Fo7C1+kZ9nf9Pjx40NtzJgxtXYWVvjb3/421J599tlQK9eIbAxZrcl1VRY+m82piRMn1tpZ8GEWEmeetSM7b5Tvd9MQ6qxfk5DrbH0q71eyOfjnP/851LIAzk5xjm2mfI+zkPEsmLpc27KA1TJ4d1E0mddZn+HDh4faV7/61VDbfvvta+0sVPuqq64KtSbBr7x22Xv5xje+sdbOAlizv/vsM5Yrrrii1r766qtDn3XWWSfU3vzmN9fa++67b+gzffr0ULvwwgtDrfwMqZN/L0RlEHX5XlZVHv77+9//vtYug3KrKn/vslp5rTdz5szQJ5vD5d9Ddv/S2/nT9G+m/BzGObaZ8r3Lrquyz1aPOuqoWvsd73hH6LPyyiuHWnadfsMNN9TaN954Y+hz8cUXh9qUKVNq7WyOZWP4/Oc/H2pbbLFFrZ3N4VGjRoVauQ5n143dnIu+CQEAAAAAALTCJgQAAAAAANAKmxAAAAAAAEArbEIAAAAAAACtaD2YOpMFrZWBVFmQSxZM2MkAjTLw5O9//3uj15X/nqZBxlloWRlKlwV13XPPPT2Ogd7JArzaDrgqA9Df9a53hT7Z+/utb32r1r7lllsavY4lV7b2NAnmKtedqsqDm1ZfffVa++677w592gzIzDT9N9OzbP1797vfXWuvtdZajY6VrZunnXZarX3KKaeEPtl7N2jQoFr71ltvDX1uv/32UCvDyKqqqrbbbrtauwxbr6o8ZC/7G+G1KwMNq6qq1l9//VAr50EZTl5Vebhpdu3YZD3IzpVNru2yf8+hhx4aauU8u/fee0Ofa6+9NtSya0faUb6/Tc8tTc6x5RpWVVV1yCGHhFoZMJgFk3/zm98MtTavU51jmynPEePHj2/0ujJcswyxr6qquvLKK0MtC80s591KK60U+mThl3PmzKm1hw0bFvqU9xxVVVVve9vbehzX97///dDn/vvvDzVzqh1ZeGsZRJ2d/7Kg3/e9732hVgapL7/88qHP1772tVBbd911a+1srZswYUKolUHYVWXudFsZ/vuDH/wg9MmuXa655ppaOztvNX0vs/uVJn3Kv4ds7rc9n8zX3infz+z+LfsMbc8996y1s/PbggULQu3SSy8NtWOOOabWfuKJJ0KfJvcTmeycvscee4RaeT2ZzfPs502bNq3Wzj5j7ub1nm9CAAAAAAAArbAJAQAAAAAAtMImBAAAAAAA0AqbEAAAAAAAQCv6JJg60yQQrttj6KTRo0eHWhbGWPr9738fanfeeWeoCbnpjLZ/j1mo6/e+971ae8iQIaHPQw89FGq/+tWvam1zYOnSyTCtFVdcMdR22mmnUCuDxm644YbQJws66q0sbKk/nCuWVKuuumqoleHRTd6TqsqDNL/85S/X2llwXRYAPXfu3Fo7m/tZMPW5554baptuummtna23Q4cODbU11lij1n744YdDH3o2ePDgUNtss816fF02n8qQtapqth5kfXob7Dt27NhQ22abbUKtDN389a9/Hfo88sgjoWZ96zuLco4t18lsnpRrUebpp58OtUmTJvV6XL1hDjZT/p7K81ZV5SHBq6yySq1dhmhWVT4Xp0+fHmqrrbZarT18+PDQ57nnngu1J598stY+/PDDQ5+NN9441LLrgWuvvbbWvuyyy0Kfl156KdRoR3ZNtfrqq9fajz/+eOhzwgknhNpVV10VauW5s7xWqqqq2nvvvUOtvBbI1pmBAweGWjZ3rFHtyf7Gt91221o7CwjOru/L9Sjr01STe8Emgb1NQ33pf8p1rKqq6oADDgi1Mog6e8+z82J5LquqeE3WdK6U5/7llosfwX/sYx8LtXXXXTfUyvFnc3/27NmhVoZo9/U8900IAAAAAACgFTYhAAAAAACAVtiEAAAAAAAAWtFvMiGWJNnzFw877LBQy547XT6T7Mwzzwx9ymcL0/ey58uNGjUq1L7zne+E2kYbbVRrz5kzJ/TJnm3+/PPPv5Yh/n/ZWEuer9n/dDILoZwDa6+9duizzjrrhNpTTz1Va2fP+u3tMz6bzMuqMjc7Jft9Z8+yL58xnf3+y+dMVlVVvec97wm1JnMjezZ/k+f1Z8/fzp5ffOyxx9ba2e9h0KBBoVau557f30z5+y2ff15VVTVixIhQmzdvXq19zTXXhD7dfq54dm23/fbbh1qWMzJ58uRaOzundzJPh9euk+fY8nm/e+yxR+iT3QOUa91FF10U+mTPLu6tphk/9Kx8vvIDDzwQ+mTZRWVeyJgxY0Kf7FnR2XmqfO50lpuT5YyUa/B6660X+mTrX3a/ctRRR9Xa1rW+lT17vJyrV1xxRehz0003hVq2Nqy00kq19o9+9KPQJ8sbK481c+bM0Cf7e8muI8t539uMJ6Isx+atb31rrZ2tRdl6seWWW/Z47E5yLluyZfcO2VxskqGQye6JywzNbD3Kjj9jxoxa+x3veEfos88++4Ratn43yZ/685//HGp33HFHrd3JbNHe8E0IAAAAAACgFTYhAAAAAACAVtiEAAAAAAAAWmETAgAAAAAAaMVSEUydBa9lgTmlLNgoC+woj58FemUhndmxLr300lo7CzbLgkTorvI9Hz58eOjzta99LdTe8pa39Hjsq6++OtR+9rOfhdqCBQtq7Ww+ZfM8C4LqbZgw/V+T4N2ddtop9Bk4cGCo/e53v6u1H3/88dCnk6FGQsXak72/hx12WKiV68ULL7wQ+hx++OGhNn/+/EUYXWdk63IZGpyth+XaWlXWyN4q158yxLKqYuh3VcXrryx4rW3l2IcNGxb6HHrooaGWBeP94Q9/qLWzwFjr3eIpO8eOHDmy1t5zzz0bva4MZ/3hD38Y+nTyHsCc653s91auWdn10QUXXBBq5TkpW2cGDx4catlaWo4hCxzO7m0POOCAWjs7L2bz7re//W2o3X333bW2OdY92ZqywgorhFqTcNUJEyaE2mabbRZqX/jCF2rtN7zhDT0Ns6qquNZde+21oU/279l8881D7U9/+lOt3RfXC0uTVVZZpdbO3qfsHmOHHXaotUePHh36TJ48OdSyNaS360q5tmVrXRYGnP2N+Dyuu8p5lt2PPvXUU6E2fvz4Wjs7B2bv+X777Rdq5fk6m4fZuMqxl2twVeVzMTt+ub5dfvnloc+XvvSlUJs6dWqo9TTOfzaGTvBNCAAAAAAAoBU2IQAAAAAAgFbYhAAAAAAAAFphEwIAAAAAAGjFEhdMnQVqNA3MKYM3suCSTBlm8tGPfjT0GTp0aKjdeuutoXbsscfW2i+++GKjMTTRzbCRJV0ZPHnccceFPm9/+9tDLXsP7rvvvlq7nANVlYfJlIFITed+08B1lgxZ0NHqq69ea2eB6XPmzAm1Mlw1C19qKpubJfOyPVlofRbIXK4Xzz//fOhz0003hVq337vll18+1D7+8Y+H2qqrrtrjsbJg4SxEjJ6V8yCbY7NmzQq1cePG1do77bRT6PPwww+H2rx583ocQ7b2ZH8P5Tw48MADQ5+tt9461LKwwscee6zWfumll0IfFk/ZOXbixIm19vrrrx/6ZNdi5X3Bgw8+GPo4L/ZP5d99tq5dd911oXb77bfX2tn6lF2PZUGpZS2bYyuvvHKorbvuurV2uf5WVVXNnj071L7+9a+HmrWt7zQ9t5XBwtttt13os+uuu4bamDFjQq38fCM7/2Xn6h//+Me19hNPPBH67Ljjjo3GVY7hoosuCn2afqZDzx555JFaO1uLsqDfESNG1NqXXnpp6HPJJZeEWvYZyJ133llrZ+fKbK0r58pWW20V+qyzzjqhdsEFF4TabbfdVmsvyj0xPSvXlnIOVFVVHXLIIaFWfsax9tprhz7ZPeQuu+wSaptsskmtnX2enN1Dln8P2XVjtnY+99xzofbLX/6y1j7mmGNCn+yaob8FqfsmBAAAAAAA0AqbEAAAAAAAQCtsQgAAAAAAAK3oN5kQ5XMMe/vM0+x12TMSs6yF8llZTY81fvz4Wvtd73pX6DNz5sxQy57hlT33rjeyZ415jmznlM+Ty/IfhgwZEmrTp08PtcMOO6zWLp8dXVW9f5al/If+r5NZLU2fB7veeuvV2htvvHHokz1P8O9//3ut3fT5gk3ySszL7soyDrLnWJbzYNq0aaHP4MGDQ61J5kdT5dxYYYUVQp8PfOADoXbAAQeEWvn3kM3hcp5XVVU99NBD/3JMNJOtK/fee2+obbTRRrV2do7N1q3s2dNPP/10rZ29508++WQcbOHQQw8NtexvJnsucPl309+ezboka/scm82BMsMky6LJnp1fPiM7yzhpyjm2u5rkCs6fP79RradjL4rs540ePbrWzsZ+4YUXhtr9998fak3GKhOse7Jn85fnxOyaKst/yJ6bXuaEZXkMp556aqiV98Ovf/3rQ58NN9ww1LJ8nfLfePXVV/c4TnqvzLEp51NVVdWwYcNCrXyfNthgg9DnyCOPDLVsLSjPn9nnelmtPFZ2L5S9bvPNNw+1j33sY7V29vmNdaxzyt/l3LlzQ5+s9pOf/KTWzs4/2eemJ510UqiVa9IRRxwR+mRZEmussUatnc2L7D7ky1/+cqj95je/qbWzf3MTfZ0V7JsQAAAAAABAK2xCAAAAAAAArbAJAQAAAAAAtMImBAAAAAAA0Io+CabOwj/KWhbal4VlNAnQ6GQAYBbAedxxx9XaWRjPjTfeGGplsE9V9X6sZbhI9jvOvPzyy736eUuT7D0vg2LKwJmqyn+3F198caiVoZxNQ6j7Q7BbGTLV9O82s7SEN3UysLeJbC2YMGFCrT1ixIjQZ/LkyaFWBsk1fc+6/W+mZ1kY22abbRZqZRBhNp+yv/vevufZnCrXmTe+8Y2hz8knnxxq2dpdysJhzz///FCbMWNGj8ciKt/P2bNnhz533XVXqL3uda+rtbOQzCwoMJsb5Vx85JFHQp/vfOc7odbTcf5ZrQw/ryrXWt1Uvidth++tvPLKobbbbrvV2lmgaxbSfuWVV9banbwepLv66zXt1ltv3WMtm5tZSOeCBQtCrVOh7/3199efNT1H3XLLLbX2rFmzQp8mIdRVVVV/+tOfau3s/JqFoZfXktnrsoDg7Pw6ZMiQWjtbk7M53eQzl05eyy6Osn/HNddcU2uXwb9VVVW77rprqI0cObLWzj47yYLUs/dg0KBBtfbAgQNDn6xWvufZPM9q2bXlTjvtVGv//Oc/D31c+7Wnt39j2euytSB77+68885a+7zzzgt9ttpqq1Ar5362Jmb3IWUIdVX1Poi61NdrlG9CAAAAAAAArbAJAQAAAAAAtMImBAAAAAAA0AqbEAAAAAAAQCv6JJi6iSz8smlAW6dkwTTvec97Qu0tb3lLrZ0FPGWhPVnAU2+Vv6/s99fXASSLqy222CLUyjCiFVZYIfTJ3t9JkyaFWjmvmwaL9TbELetXHisL/srCi8sQ9qeffjr0yYLrsjDYcgzZ6xY33Q6LzN7bJgFbWYBvFhY7b968RRhdXblGNQ2Kso61J/u7L3/fWeDfaqutFmrPPvtsqJUhg01DfcePH19rZyHU2bgy5Zy6+eabQ5/TTz891LKARF677Pd44403htrf/va3WnvUqFGhz+qrrx5qO++8c6iV69tll10W+vzxj38MtXLtvPXWW0OfTTfdNNSyNSqb1yy6Tl4b9dY666wTamuttVatnY1z8uTJofb444/X2osy9iYB3U2CWemcbl8TrrjiiqF2/PHHh1p5D3P55ZeHPtl87WQwqHD1RZf9XmfOnBlq1113Xa1dBg1XVfNA3fJnZp8/ZOtMWXv00UdDnx/+8Iehdvjhh4faU089VWuvuuqqoc+TTz4Zak3mbzYvs9ctqfcm2XtX/i5PPPHE0OeMM84ItfIzg3333Tf0eec73xlqG2ywQaiVodPZOLPw3/J1Te9DyvDzqqqq3XbbrdY+55xzQh/B1EuW8l4zO59uuOGGoVbOs4svvjj0ye49OxVC3R/5JgQAAAAAANAKmxAAAAAAAEArbEIAAAAAAACtsAkBAAAAAAC0ok+CqZuEjfZFQFUZpvS6170u9DnmmGNCrQw9PPfcc0OfK664ItQ6GVZT/v4E4XTORhttFGpl6FUWxJWFVa+//vqhNnLkyFo7C23O5uIb3vCGWvv2228PfbKgr2xubLLJJrX27rvvHvr8+7//e6iV4dHf+973Qp977rkn1LLQ2jJ8NgtKW9w0DTRr8+etscYaoVbOnWxMWQhrGaLeVG/D32jP888/H2oPP/xwqJXzJwuo/9WvfhVqWSjdlVdeWWsPHTo09PnQhz4UamV4XRakns2fbK279957a+39998/9Jk1a1ao0RnZWpCFB5bnlhkzZoQ+999/f6hNmjQp1Mprzjlz5vTYp6piiPaDDz7YY5+qys/h5fVAt88NS6r+cI7daqutQm3QoEG1drYWXX311aFWznuWfOX9Q3Y/kc3pJtdjw4cPD7XsfqL8mTfeeGOvfl5T2d9ROYalKfy3U7LfT7amdPL3WIb4Ng2mLsfwwgsvhD6XXHJJqGXHX2uttWrtLJB9+eWXD7Xy/N00ML2TfwuLo/LfP2/evNAnq02ZMqXWvvvuu0Of0047LdTe9ra3hdonPvGJWnvMmDGhTxkiXFXx/iGbT02Vx88+92n774/2ZPe7l156aa295ZZbhj7ZnCoDpo877rjQJ7svX5L5JgQAAAAAANAKmxAAAAAAAEArbEIAAAAAAACt6JNMiEz5fLS2n5eWPeOvfE7cV77yldAne756+dz9k046KfSZPXv2axzha5M9b5HO+Nvf/hZq5TP+smeUL7dc/PN63/veF2pl1sK4ceNCn1GjRoVa+RzO7Hma2bzLnmU5ZMiQWrt8nvE/M3PmzFp74403Dn2yrIoNNtgg1LLxL056m2nQybUuew7h3nvvHWqjR4+utbPffZbl0cl1psmxPDezPdnacMopp4Ta2WefXWuvttpqoU+WdXPiiSeG2rHHHltrZ+tm9tze8m8rmxfZs/lvueWWUDvooINq7enTp4c+5l3fK9+D7LyV1bJ50Nv3s1yjnnvuudAne95v9vz/clwyIbqn7XPsdttt12O/7Bx7/vnnh1qbuUv0vex9KteZplkITZ5fX2atVVX+nPRy3k2dOrXHY/+zMTSR/R0NHDiwxz7Z+s6/1sm1IZsD2b1ukzE0GVeWK9Dk3qTJZzxZv+zcndWa3O9Zk6Mm13VZZmSWDbLrrrvW2ll+Z5YN0uS9y+5Ps+yyu+66q9YuP5epquYZKSXzp7uy9SG7Rttmm21q7ez9zeb1UUcdVWtnma1LG9+EAAAAAAAAWmETAgAAAAAAaIVNCAAAAAAAoBU2IQAAAAAAgFb0m2DqbsuCaXbbbbdae+eddw59sjCZz372s7V2FujF4uuBBx4ItQsvvLDW3muvvUKfLBBpzTXXDLWxY8fW2lnITZMgpTLUraryoJ1sDpcBSFkQ19NPPx1qd955Z639+9//PvSZNm1aqE2ePDnUnn/++VBb3PU2jK23sve7DO+qqjjHnnnmmdBnypQpHRuXgK3+J1sHbrvttlD7y1/+Umtn8ylbe7L1L6s1Uc6fLAz4//yf/xNqJ598cqiVQdTm5pKlzRDiYcOGhT5z584NtSzAes6cObV2FmDYJKyQum7/zpZffvlQW2uttUKtHFd2jn3sscc6N7AGrHX9U/m+LMr7VJ6LDz744NBnpZVWCrUy8HncuHE9Hjt7XVXF8Wf3L01q5mtn9DZEuWkIddmvyZxoKntddn86fPjwWnvMmDGhz5NPPhlq5Xk4uw9dlKB4OiObU+U6tsIKK4Q+TUKDsxDh8nqtqvJA9Ntvv73Wfumll0KfJn9b2d9aNi46I5sXH/jAB0LtjW98Y4+vzd7fiy++ONS+853vvIYRLh18EwIAAAAAAGiFTQgAAAAAAKAVNiEAAAAAAIBW2IQAAAAAAABasdQGU5chRlVVVZ/73Odq7Szk5uabbw61a665ptYWTrRkyYInP/3pT9faP/jBD0Kf7bffPtQmTJgQajvssEOtvfrqq4c+WYhOOc+yQKTZs2eHWhnMWlUxYPr6668Pfe6+++5Q+/vf/15rz5o1K/RZWsKVuv13nwVZjRgxItSyuVOGr1199dWhTxau2ibrZt/L3vOPfvSjtfbPf/7z0CcL78oCXJsEJGbrxVNPPVVrH3rooaHPtddeG2pZgLV5RlNlkHp2Pn3iiSdC7a677gq1Rx55pNYWQr14WmWVVUItC2stgzQff/zx0Gf+/PmdG1jCWrdky86nZSDvLrvsEvpk14RlQO8GG2wQ+qy88sqhNnPmzFArz+HZPMzO8+X52vx97XobAp71yeZJdt4q17pOntuytXXVVVcNtfL+NztXDx48ONReeOGFHsfQJNx4Sdbkuj3Tyb/fLIz87LPPrrUnTpwY+mSf45Vz4y9/+Uvoc99994Xa7373u1ArPwPJAq2tY32vnMNZcH35uV5VxfNiVcX3swwnr6qq2nfffXt8Hb4JAQAAAAAAtMQmBAAAAAAA0AqbEAAAAAAAQCuWikyI7NnU//M//xNqW265ZY/Huuiii0KtfB4iS5bsOW7lM/X//Oc/hz5ZrYmmz+Ysn1U3cODARsdv8gzM7HmX2bjK341n3nVP9n5kz80877zzQq3Mtsme859ljHSSubJ4KJ95/9a3vjX0yZ6vmT1Tevz48bV2liHT5Hn68+bNC33Mp/6pybOne3tu6eR7nj37tXyecJaZkmUl/eEPfwi18nnnvf090D3ZdVeWu1ReD1ZVVT355JO1dvaM6WzOlXPA+8+iyNasNddcM9TKa/5BgwaFPmVGTlVV1bPPPhtqTeZsb/tYI/+17HeRrTNlrWn+Q9Nab2TvbZbjkOWQZGtwaciQIaFWXktmPy+7F8qe/b+kyuZUOV/a/hvM5till15aa2fP5s/O1+V9R5YfkmWBZvcdsr0WD+XnY3vvvXfoM3LkyFDLPgsr872yY2WfxRD5JgQAAAAAANAKmxAAAAAAAEArbEIAAAAAAACtsAkBAAAAAAC0YokLps4CmD7zmc+E2kEHHRRqZXDJ/PnzQ5+rrrpqEUYHPcsCnrJwnLLWdkC68Lf+JQvEmjZtWqidccYZoVa+l9n88n6TyQK3Hn300UY1lj7lOpIFYGa1ck1qez3Kjl+GXV5//fWhz9///vdQK0OJq6qqpk+fXmsLNOz/svfo4YcfDrWjjjoq1FZaaaVa+4knngh9sjBV512ayubKlClTau0f/ehHoc/BBx8cauW14znnnBP6zJgxI9S6vY75+3jtst9ZWcvCl7P7gjbf7yyYesGCBaH22GOPhVp5zh02bFjoU67JVRXvm7PfQxZSvLTrD9cv5XvlPoSqyu8n1llnnVp7n332CX1WWGGFUHvhhRdC7dvf/natnX3uQjO+CQEAAAAAALTCJgQAAAAAANAKmxAAAAAAAEArbEIAAAAAAACtWOyDqcsAkokTJ4Y+RxxxRKiVIdSZLPzokUceeQ2jA+ieLCysPwSIAVRV87DLbgeQZmMoQzGza8IslC4Lt8xqLH7mzZsXavfcc0+Pr3MephvKoN3TTz899LnyyitDbbnl6h8HZAHs8+fPD7VOrtNCpxdd9jtscn7ti3NwGUSdBVNnwbBZWHXp2WefDbVBgwaFWvlvbHuOA+0qz2VVVVX7779/rT1hwoTQJ/tcOFsPbr311lq7POfSnG9CAAAAAAAArbAJAQAAAAAAtMImBAAAAAAA0AqbEAAAAAAAQCsW+2DqJubMmRNqWZBIWTvllFMavQ4AgNeuvwY/luN6+eWXQ5+sxtJF6DT9VRbie/fdd/f4uv66JvPaLS7rUzbnmtbKUOvs3zxv3rxQW1x+N0Az2ee0v/71r2vtXXbZJfTZaKONQu20004LtRtvvHERRsc/8k0IAAAAAACgFTYhAAAAAACAVtiEAAAAAAAAWrHYZ0KUz/O79dZbQ5+JEyeG2gorrBBq8+fPr7Wz5wd6TiYAAACLE/ex9AflPFyUeVm+9pVXXun1sYAly3333Vdr77jjjn00Ev6Rb0IAAAAAAACtsAkBAAAAAAC0wiYEAAAAAADQikaZEIvT8yOzsWa1Mksi67c4/bu7rRu/G79/Sm3PCXOOjHlHtznH0hesdXSbtY6+YK2jL5h3dJtzLH2hpznRaBNi9uzZHRlMN2RhRHPnzm1Uo7nZs2dXq6yySus/A/5R2/POnCNj3tFtzrH0BWsd3Watoy9Y6+gL5h3d5hxLX+hp3g1Y2GDr6tVXX62mTp1aDR06tBowYEBHB8jiZeHChdXs2bOr0aNHV8ss0+7TvMw7/p9uzTtzjn9k3tFtzrH0BWsd3Watoy9Y6+gL5h3d5hxLX2g67xptQgAAAAAAALxWgqkBAAAAAIBW2IQAAAAAAABaYRMCAAAAAABohU0IAAAAAACgFTYhAAAAAACAVtiEAAAAAAAAWmETAgAAAAAAaMX/BVHjykDmYDv7AAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 2000x400 with 20 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "\n",
    "n = 10  # How many digits we will display\n",
    "plt.figure(figsize=(20, 4))\n",
    "for i in range(n):\n",
    "    # Display original\n",
    "    ax = plt.subplot(2, n, i + 1)\n",
    "    plt.imshow(x_test[i].reshape(28, 28))\n",
    "    plt.gray()\n",
    "    ax.get_xaxis().set_visible(False)\n",
    "    ax.get_yaxis().set_visible(False)\n",
    "\n",
    "    # Display reconstruction\n",
    "    ax = plt.subplot(2, n, i + 1 + n)\n",
    "    plt.imshow(roundtrip_imgs[i].reshape(28, 28))\n",
    "    plt.gray()\n",
    "    ax.get_xaxis().set_visible(False)\n",
    "    ax.get_yaxis().set_visible(False)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44028996",
   "metadata": {},
   "source": [
    "What about the compression? Let's check the sizes of the arrays."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "4bc1b868",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-28T16:10:21.659924Z",
     "iopub.status.busy": "2023-06-28T16:10:21.658324Z",
     "iopub.status.idle": "2023-06-28T16:10:22.373196Z",
     "shell.execute_reply": "2023-06-28T16:10:22.372534Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x_test size (in MB): 29.91\n",
      "encoded_imgs size (in MB): 1.22\n",
      "Compression ratio: 1/25\n"
     ]
    }
   ],
   "source": [
    "encoded_imgs = autoencoder.transform(x_test)\n",
    "print(f\"x_test size (in MB): {x_test.nbytes/1024**2:.2f}\")\n",
    "print(f\"encoded_imgs size (in MB): {encoded_imgs.nbytes/1024**2:.2f}\")\n",
    "cr = round((encoded_imgs.nbytes/x_test.nbytes), 2)\n",
    "print(f\"Compression ratio: 1/{1/cr:.0f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a97f75f3",
   "metadata": {},
   "source": [
    "## 6. Deep AutoEncoder"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31575169",
   "metadata": {},
   "source": [
    "We can easily expand our model to be a deep autoencoder by adding some hidden layers. All we have to do is add a parameter `hidden_layer_sizes` and use it in `_keras_build_fn` to build hidden layers.\n",
    "For simplicity, we use a single `hidden_layer_sizes` parameter and mirror it across the encoding layers and decoding layers, but there is nothing forcing us to build symetrical models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "06ca0fa0",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-28T16:10:22.376919Z",
     "iopub.status.busy": "2023-06-28T16:10:22.376660Z",
     "iopub.status.idle": "2023-06-28T16:10:22.386105Z",
     "shell.execute_reply": "2023-06-28T16:10:22.385470Z"
    }
   },
   "outputs": [],
   "source": [
    "from typing import List\n",
    "\n",
    "\n",
    "class DeepAutoEncoder(AutoEncoder):\n",
    "    \"\"\"A class that enables transform and fit_transform.\n",
    "    \"\"\"\n",
    "    \n",
    "    def _keras_build_fn(self, encoding_dim: int, hidden_layer_sizes: List[str], meta: Dict[str, Any]):\n",
    "        n_features_in = meta[\"n_features_in_\"]\n",
    "\n",
    "        encoder_input = keras.Input(shape=(n_features_in,))\n",
    "        x = encoder_input\n",
    "        for layer_size in hidden_layer_sizes:\n",
    "            x = keras.layers.Dense(layer_size, activation='relu')(x)\n",
    "        encoder_output = keras.layers.Dense(encoding_dim, activation='relu')(x)\n",
    "        encoder_model = keras.Model(encoder_input, encoder_output)\n",
    "\n",
    "        decoder_input = keras.Input(shape=(encoding_dim,))\n",
    "        x = decoder_input\n",
    "        for layer_size in reversed(hidden_layer_sizes):\n",
    "            x = keras.layers.Dense(layer_size, activation='relu')(x)\n",
    "        decoder_output = keras.layers.Dense(n_features_in, activation='sigmoid', name=\"decoder\")(x)\n",
    "        decoder_model = keras.Model(decoder_input, decoder_output)\n",
    "\n",
    "        autoencoder_input = keras.Input(shape=(n_features_in,))\n",
    "        encoded_img = encoder_model(autoencoder_input)\n",
    "        reconstructed_img = decoder_model(encoded_img)\n",
    "\n",
    "        autoencoder_model = keras.Model(autoencoder_input, reconstructed_img)\n",
    "\n",
    "        self.encoder_model_ = BaseWrapper(encoder_model, verbose=self.verbose)\n",
    "        self.decoder_model_ = BaseWrapper(decoder_model, verbose=self.verbose)\n",
    "\n",
    "        return autoencoder_model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "10b3e628",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-28T16:10:22.394290Z",
     "iopub.status.busy": "2023-06-28T16:10:22.392139Z",
     "iopub.status.idle": "2023-06-28T16:11:14.941846Z",
     "shell.execute_reply": "2023-06-28T16:11:14.940886Z"
    }
   },
   "outputs": [],
   "source": [
    "deep = DeepAutoEncoder(\n",
    "    loss=\"binary_crossentropy\",\n",
    "    encoding_dim=32,\n",
    "    hidden_layer_sizes=[128],\n",
    "    random_state=0,\n",
    "    epochs=5,\n",
    "    verbose=False,\n",
    "    optimizer=\"adam\",\n",
    ")\n",
    "_ = deep.fit(X=x_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "02431451",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-06-28T16:11:14.946240Z",
     "iopub.status.busy": "2023-06-28T16:11:14.945529Z",
     "iopub.status.idle": "2023-06-28T16:11:16.478067Z",
     "shell.execute_reply": "2023-06-28T16:11:16.477268Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1-MSE for training set (higher is better)\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AutoEncoder: 0.9899\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Deep AutoEncoder: 0.9914\n"
     ]
    }
   ],
   "source": [
    "print(\"1-MSE for training set (higher is better)\\n\")\n",
    "score = autoencoder.score(X=x_test)\n",
    "print(f\"AutoEncoder: {score:.4f}\")\n",
    "\n",
    "score = deep.score(X=x_test)\n",
    "print(f\"Deep AutoEncoder: {score:.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "474c690f",
   "metadata": {},
   "source": [
    "Suprisingly, our score got worse. It's possible that that because of the extra trainable variables, our deep model trains slower than our simple model.\n",
    "\n",
    "Check out the [Keras tutorial](https://blog.keras.io/building-autoencoders-in-keras.html) to see the difference after 100 epochs of training, as well as more architectures and applications for AutoEncoders!"
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "formats": "ipynb,md"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.17"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}