From 32c75c226cc39c1aae833b404a2b38c3daf1cafb Mon Sep 17 00:00:00 2001 From: Iden Kalemaj Date: Tue, 17 Dec 2024 08:25:17 -0800 Subject: [PATCH 01/10] Add LoRA to the BERT fine-tuning tutorial (#698) Summary: Pull Request resolved: https://github.com/pytorch/opacus/pull/698 Update the BERT fine-tuning tutorial to show how LoRA can be used with DP-SGD. Reviewed By: HuanyuZhang Differential Revision: D67281956 fbshipit-source-id: e7f099ba2e7e816de96cd61f4adff4ab84e9e7d5 --- tutorials/building_text_classifier.ipynb | 4468 ++++++++++++++++++---- 1 file changed, 3660 insertions(+), 808 deletions(-) diff --git a/tutorials/building_text_classifier.ipynb b/tutorials/building_text_classifier.ipynb index a8a3fa45..585d54d7 100644 --- a/tutorials/building_text_classifier.ipynb +++ b/tutorials/building_text_classifier.ipynb @@ -1,815 +1,3667 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# Building text classifier with Differential Privacy" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "In this tutorial, we will train a text classifier with Differential Privacy by taking a model pre-trained on public text data and fine-tuning it for a different task.\n", - "\n", - "When training a model with differential privacy, we almost always face a trade-off between model size and accuracy on the task. The exact details depend on the problem, but a rule of thumb is that the fewer parameters the model has, the easier it is to get good performance with DP.\n", - "\n", - "Most state-of-the-art NLP models are quite deep and large (e.g. [BERT-base](https://github.com/google-research/bert) has over 100M parameters), which makes the task of training text models on private datasets rather challenging.\n", - "\n", - "One way of addressing this problem is to divide the training process into two stages. First, we will pre-train the model on a public dataset, exposing the model to generic text data. Assuming that the generic text data is public, we will not be using differential privacy at this step. Then, we freeze most of the layers, leaving only a few upper layers to be trained on the private dataset using DP-SGD. This way we can get the best of both worlds - we have a deep and powerful text understanding model, while only training a small number of parameters with differentially private algorithm.\n", - "\n", - "In this tutorial, we will take the pre-trained [BERT-base](https://github.com/google-research/bert) model and fine-tune it to recognize textual entailment on the [SNLI](https://nlp.stanford.edu/projects/snli/) dataset.\n", - "\n", - "We also fine-tune it with Ghost Clipping DP-SGD, a memory-efficient implementation of DP-SGD, which enables the use of large batch sizes. " - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Dataset" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "First, we need to download the dataset (we'll use Stanford NLP mirror)" - ] - }, - { - "cell_type": "code", - "execution_count": 1, - "metadata": {}, - "outputs": [], - "source": [ - "STANFORD_SNLI_URL = \"https://nlp.stanford.edu/projects/snli/snli_1.0.zip\"\n", - "DATA_DIR = \"data\"" - ] - }, - { - "cell_type": "code", - "execution_count": 2, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Downloading and extracting ...\n", - "Completed!\n" - ] - } - ], - "source": [ - "import zipfile\n", - "import urllib.request\n", - "import os\n", - "\n", - "import warnings\n", - "warnings.simplefilter(\"ignore\")\n", - "\n", - "def download_and_extract(dataset_url, data_dir):\n", - " print(\"Downloading and extracting ...\")\n", - " filename = \"snli.zip\"\n", - " urllib.request.urlretrieve(dataset_url, filename)\n", - " with zipfile.ZipFile(filename) as zip_ref:\n", - " zip_ref.extractall(data_dir)\n", - " os.remove(filename)\n", - " print(\"Completed!\")\n", - "\n", - "download_and_extract(STANFORD_SNLI_URL, DATA_DIR)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The dataset comes in two formats (`tsv` and `json`) and has already been split into train/dev/test. Let’s verify that’s the case." - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "['snli_1.0_dev.txt',\n", - " 'README.txt',\n", - " 'snli_1.0_dev.jsonl',\n", - " 'Icon\\r',\n", - " '.DS_Store',\n", - " 'snli_1.0_test.txt',\n", - " 'snli_1.0_train.jsonl',\n", - " 'snli_1.0_test.jsonl',\n", - " 'snli_1.0_train.txt']" - ] - }, - "execution_count": 3, - "metadata": {}, - "output_type": "execute_result" + "metadata": { + "kernelspec": { + "name": "python3", + "display_name": "python3", + "languaage": "python" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.10.6" + }, + "colab": { + "provenance": [], + "gpuType": "T4" + }, + "accelerator": "GPU", + "widgets": { + "application/vnd.jupyter.widget-state+json": { + "a6f080fa6f4b4de399af5d1d7850b960": { + "model_module": "@jupyter-widgets/controls", + "model_name": "HBoxModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HBoxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HBoxView", + "box_style": "", + "children": [ + "IPY_MODEL_47fec328e2464db3861b16e68e6cc65d", + "IPY_MODEL_3ec2b8f4e38d4b05a09c83e6925960a6", + "IPY_MODEL_6fee830ab9f545cea62081d8cf5b3240" + ], + "layout": "IPY_MODEL_cf024e76fe9b4766ac035f617391deb7" + } + }, + "47fec328e2464db3861b16e68e6cc65d": { + "model_module": "@jupyter-widgets/controls", + "model_name": "HTMLModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_fc0b17bfb44c45dd9825c6f1719b61cc", + "placeholder": "​", + "style": "IPY_MODEL_2dac3a8089c34d0b9ce81653bde67603", + "value": "config.json: 100%" + } + }, + "3ec2b8f4e38d4b05a09c83e6925960a6": { + "model_module": "@jupyter-widgets/controls", + "model_name": "FloatProgressModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "FloatProgressModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "ProgressView", + "bar_style": "success", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_ffb2ab66b3ca4d9899658fe58c43acb1", + "max": 570, + "min": 0, + "orientation": "horizontal", + "style": "IPY_MODEL_4b6aad944453432dbf957da380d059a1", + "value": 570 + } + }, + "6fee830ab9f545cea62081d8cf5b3240": { + "model_module": "@jupyter-widgets/controls", + "model_name": "HTMLModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_cf3dae1b960440d7984e9ed287b54cee", + "placeholder": "​", + "style": "IPY_MODEL_36254c0a04c840f3bf4038096c736873", + "value": " 570/570 [00:00<00:00, 33.8kB/s]" + } + }, + "cf024e76fe9b4766ac035f617391deb7": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "fc0b17bfb44c45dd9825c6f1719b61cc": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "2dac3a8089c34d0b9ce81653bde67603": { + "model_module": "@jupyter-widgets/controls", + "model_name": "DescriptionStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "ffb2ab66b3ca4d9899658fe58c43acb1": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "4b6aad944453432dbf957da380d059a1": { + "model_module": "@jupyter-widgets/controls", + "model_name": "ProgressStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "ProgressStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "bar_color": null, + "description_width": "" + } + }, + "cf3dae1b960440d7984e9ed287b54cee": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "36254c0a04c840f3bf4038096c736873": { + "model_module": "@jupyter-widgets/controls", + "model_name": "DescriptionStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "e3ffd50ee822433fabd9c1ee4a39612e": { + "model_module": "@jupyter-widgets/controls", + "model_name": "HBoxModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HBoxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HBoxView", + "box_style": "", + "children": [ + "IPY_MODEL_f248160605a2450a8411e4f5d58a5cfa", + "IPY_MODEL_a7b6e7aa521647649bb4157b6504d4e8", + "IPY_MODEL_1c1e86bca0534caaa7ad435fd7e67bf2" + ], + "layout": "IPY_MODEL_22ca9e6c6c1f4bc6b0f3db1a09a5e562" + } + }, + "f248160605a2450a8411e4f5d58a5cfa": { + "model_module": "@jupyter-widgets/controls", + "model_name": "HTMLModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_e9699698559c4860bdf6a312c492e7da", + "placeholder": "​", + "style": "IPY_MODEL_874e45fe39844927ad1fd10d4899a428", + "value": "tokenizer_config.json: 100%" + } + }, + "a7b6e7aa521647649bb4157b6504d4e8": { + "model_module": "@jupyter-widgets/controls", + "model_name": "FloatProgressModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "FloatProgressModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "ProgressView", + "bar_style": "success", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_2dca6b2477344b45b1c17f124e27ce72", + "max": 49, + "min": 0, + "orientation": "horizontal", + "style": "IPY_MODEL_c7afefe6f907441b9e466605cb4f5c7f", + "value": 49 + } + }, + "1c1e86bca0534caaa7ad435fd7e67bf2": { + "model_module": "@jupyter-widgets/controls", + "model_name": "HTMLModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_df3c4c12e06245b1a8f4a4a7d71a530c", + "placeholder": "​", + "style": "IPY_MODEL_e63eb3e5c06140249f6d8d4c04fe8693", + "value": " 49.0/49.0 [00:00<00:00, 2.50kB/s]" + } + }, + "22ca9e6c6c1f4bc6b0f3db1a09a5e562": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "e9699698559c4860bdf6a312c492e7da": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "874e45fe39844927ad1fd10d4899a428": { + "model_module": "@jupyter-widgets/controls", + "model_name": "DescriptionStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "2dca6b2477344b45b1c17f124e27ce72": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "c7afefe6f907441b9e466605cb4f5c7f": { + "model_module": "@jupyter-widgets/controls", + "model_name": "ProgressStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "ProgressStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "bar_color": null, + "description_width": "" + } + }, + "df3c4c12e06245b1a8f4a4a7d71a530c": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "e63eb3e5c06140249f6d8d4c04fe8693": { + "model_module": "@jupyter-widgets/controls", + "model_name": "DescriptionStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "d951c3592058414ab00cf754e9b70685": { + "model_module": "@jupyter-widgets/controls", + "model_name": "HBoxModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HBoxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HBoxView", + "box_style": "", + "children": [ + "IPY_MODEL_c3f9146e082346a3bed274efb2265376", + "IPY_MODEL_1602c2298e9443f78007fdbf101a0c2b", + "IPY_MODEL_d5dda55bd4de4f12bf3718fb386c5bf9" + ], + "layout": "IPY_MODEL_943e61866ed74be4b10ba383450cb4c3" + } + }, + "c3f9146e082346a3bed274efb2265376": { + "model_module": "@jupyter-widgets/controls", + "model_name": "HTMLModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_0727d77eaf28466c93c2c6021661ac9a", + "placeholder": "​", + "style": "IPY_MODEL_3ab2e1a9ba694463ab5f3ec78ad0a8f4", + "value": "vocab.txt: 100%" + } + }, + "1602c2298e9443f78007fdbf101a0c2b": { + "model_module": "@jupyter-widgets/controls", + "model_name": "FloatProgressModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "FloatProgressModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "ProgressView", + "bar_style": "success", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_30b4db204fa644128198abf6d82664bf", + "max": 213450, + "min": 0, + "orientation": "horizontal", + "style": "IPY_MODEL_0517fdac88784fe6b51ad1f989f99cb7", + "value": 213450 + } + }, + "d5dda55bd4de4f12bf3718fb386c5bf9": { + "model_module": "@jupyter-widgets/controls", + "model_name": "HTMLModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_ce1c55bc51dc49a9b261f104f49d38d8", + "placeholder": "​", + "style": "IPY_MODEL_4b7e11bb32bc43c9ad0449bd39bc4d40", + "value": " 213k/213k [00:00<00:00, 613kB/s]" + } + }, + "943e61866ed74be4b10ba383450cb4c3": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "0727d77eaf28466c93c2c6021661ac9a": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "3ab2e1a9ba694463ab5f3ec78ad0a8f4": { + "model_module": "@jupyter-widgets/controls", + "model_name": "DescriptionStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "30b4db204fa644128198abf6d82664bf": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "0517fdac88784fe6b51ad1f989f99cb7": { + "model_module": "@jupyter-widgets/controls", + "model_name": "ProgressStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "ProgressStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "bar_color": null, + "description_width": "" + } + }, + "ce1c55bc51dc49a9b261f104f49d38d8": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "4b7e11bb32bc43c9ad0449bd39bc4d40": { + "model_module": "@jupyter-widgets/controls", + "model_name": "DescriptionStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "3dbf36a5c0884579ab2f36c2e91c04fb": { + "model_module": "@jupyter-widgets/controls", + "model_name": "HBoxModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HBoxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HBoxView", + "box_style": "", + "children": [ + "IPY_MODEL_d71c49bd9f8c438898250c3874c06240", + "IPY_MODEL_1b70fa16b803466ea31649dcd644e3d7", + "IPY_MODEL_7a728bea623646c182c508e34b582fc9" + ], + "layout": "IPY_MODEL_8494caffe83a4743a11d2751b38c56bb" + } + }, + "d71c49bd9f8c438898250c3874c06240": { + "model_module": "@jupyter-widgets/controls", + "model_name": "HTMLModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_1580380472df40e38cbde67659c5221d", + "placeholder": "​", + "style": "IPY_MODEL_349b262479b9418badd6a3acff386dd2", + "value": "tokenizer.json: 100%" + } + }, + "1b70fa16b803466ea31649dcd644e3d7": { + "model_module": "@jupyter-widgets/controls", + "model_name": "FloatProgressModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "FloatProgressModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "ProgressView", + "bar_style": "success", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_3370a9d70dd04d5195bc3f1f81b18728", + "max": 435797, + "min": 0, + "orientation": "horizontal", + "style": "IPY_MODEL_ee235d5ffc5142c895175cbea5c94dfe", + "value": 435797 + } + }, + "7a728bea623646c182c508e34b582fc9": { + "model_module": "@jupyter-widgets/controls", + "model_name": "HTMLModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_f6c5ef333a2b45f1bf196fdd58873688", + "placeholder": "​", + "style": "IPY_MODEL_e24ff9dae78241d9b5a6a7199c888e45", + "value": " 436k/436k [00:00<00:00, 1.24MB/s]" + } + }, + "8494caffe83a4743a11d2751b38c56bb": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "1580380472df40e38cbde67659c5221d": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "349b262479b9418badd6a3acff386dd2": { + "model_module": "@jupyter-widgets/controls", + "model_name": "DescriptionStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "3370a9d70dd04d5195bc3f1f81b18728": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "ee235d5ffc5142c895175cbea5c94dfe": { + "model_module": "@jupyter-widgets/controls", + "model_name": "ProgressStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "ProgressStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "bar_color": null, + "description_width": "" + } + }, + "f6c5ef333a2b45f1bf196fdd58873688": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "e24ff9dae78241d9b5a6a7199c888e45": { + "model_module": "@jupyter-widgets/controls", + "model_name": "DescriptionStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "d4a768f261614ac69b3004fbf2323c89": { + "model_module": "@jupyter-widgets/controls", + "model_name": "HBoxModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HBoxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HBoxView", + "box_style": "", + "children": [ + "IPY_MODEL_81a7d1b27cd94916ac3c330aa2551cf0", + "IPY_MODEL_eac4d3f8e59a4d4c81178cc76600182f", + "IPY_MODEL_5bcca2ee852144c28bfd40de0978cadc" + ], + "layout": "IPY_MODEL_05fa1e761bc14c929b19abf0d8a93f5f" + } + }, + "81a7d1b27cd94916ac3c330aa2551cf0": { + "model_module": "@jupyter-widgets/controls", + "model_name": "HTMLModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_32ba8daa0c6e4a9c9f62df588a198b1b", + "placeholder": "​", + "style": "IPY_MODEL_e1e93f905b494126a6a9e1e0a2f92022", + "value": "model.safetensors: 100%" + } + }, + "eac4d3f8e59a4d4c81178cc76600182f": { + "model_module": "@jupyter-widgets/controls", + "model_name": "FloatProgressModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "FloatProgressModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "ProgressView", + "bar_style": "success", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_9ca2cb3f116547d3bb062f4da11762d7", + "max": 435755784, + "min": 0, + "orientation": "horizontal", + "style": "IPY_MODEL_554184f1c9b44bd3a8116773572347eb", + "value": 435755784 + } + }, + "5bcca2ee852144c28bfd40de0978cadc": { + "model_module": "@jupyter-widgets/controls", + "model_name": "HTMLModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_bb89b007667e4bf0b696ad84dfb2f91d", + "placeholder": "​", + "style": "IPY_MODEL_610f073056924398b9229978dff5ff4d", + "value": " 436M/436M [00:02<00:00, 177MB/s]" + } + }, + "05fa1e761bc14c929b19abf0d8a93f5f": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "32ba8daa0c6e4a9c9f62df588a198b1b": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "e1e93f905b494126a6a9e1e0a2f92022": { + "model_module": "@jupyter-widgets/controls", + "model_name": "DescriptionStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "9ca2cb3f116547d3bb062f4da11762d7": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "554184f1c9b44bd3a8116773572347eb": { + "model_module": "@jupyter-widgets/controls", + "model_name": "ProgressStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "ProgressStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "bar_color": null, + "description_width": "" + } + }, + "bb89b007667e4bf0b696ad84dfb2f91d": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "610f073056924398b9229978dff5ff4d": { + "model_module": "@jupyter-widgets/controls", + "model_name": "DescriptionStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + } + } } - ], - "source": [ - "snli_folder = os.path.join(DATA_DIR, \"snli_1.0\")\n", - "os.listdir(snli_folder)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Let's now take a look inside. [SNLI dataset](https://nlp.stanford.edu/projects/snli/) provides ample syntactic metadata, but we'll only use raw input text. Therefore, the only fields we're interested in are **sentence1** (premise), **sentence2** (hypothesis), and **gold_label** (label chosen by the majority of annotators).\n", - "\n", - "The label defines the relation between premise and hypothesis: either *contradiction*, *neutral*, or *entailment*." - ] }, - { - "cell_type": "code", - "execution_count": 4, - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
sentence1sentence2gold_label
0A person on a horse jumps over a broken down a...A person is training his horse for a competition.neutral
1A person on a horse jumps over a broken down a...A person is at a diner, ordering an omelette.contradiction
2A person on a horse jumps over a broken down a...A person is outdoors, on a horse.entailment
3Children smiling and waving at cameraThey are smiling at their parentsneutral
4Children smiling and waving at cameraThere are children presententailment
\n", - "
" + "nbformat": 4, + "nbformat_minor": 0, + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "originalKey": "c21b7ad1-cba1-43cd-b602-42294e10cc9a", + "outputsInitialized": false, + "isAgentGenerated": false, + "language": "markdown", + "showInput": false, + "id": "IccO-A2JpH_1" + }, + "source": [ + "# Building a text classifier with Differential Privacy" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "originalKey": "bfca9bac-8231-4ebe-a67b-eb070f4a5958", + "outputsInitialized": false, + "isAgentGenerated": false, + "language": "markdown", + "showInput": false, + "id": "PDqDnN-FpH_2" + }, + "source": [ + "In this tutorial, we will train a text classifier with Differential Privacy by taking a model pre-trained on public text data and fine-tuning it for a different task.\n", + "\n", + "When training a model with differential privacy, we almost always face a trade-off between model size and accuracy on the task. The exact details depend on the problem, but a rule of thumb is that the fewer parameters the model has, the easier it is to get good performance with DP.\n", + "\n", + "Most state-of-the-art NLP models are quite deep and large (e.g. [BERT-base](https://github.com/google-research/bert) has over 100M parameters), which makes the task of training text models on private datasets rather challenging.\n", + "\n", + "One way of addressing this problem is to divide the training process into two stages. First, we will pre-train the model on a public dataset, exposing the model to generic text data. Assuming that the generic text data is public, we will not be using differential privacy at this step. Then, we freeze most of the layers, leaving only a few upper layers to be trained on the private dataset using DP-SGD. This way we can get the best of both worlds - we have a deep and powerful text understanding model, while only training a small number of parameters with differentially private algorithm.\n", + "\n", + "In this tutorial, we will take the pre-trained [BERT-base](https://github.com/google-research/bert) model and fine-tune it to recognize textual entailment on the [SNLI](https://nlp.stanford.edu/projects/snli/) dataset.\n", + "\n", + "We further demonstrate fine-tuning results with\n", + "\n", + "- Ghost Clipping DP-SGD, a memory-efficient implementation of DP-SGD, which enables the use of large batch sizes.\n", + "- LoRA (low-rank adaptation), a method for parameter-efficienct fine-tuning which can be used in conjucture with DP-SGD to further reduce the number of trainable parameters" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "originalKey": "239e9b8e-09ba-4c61-8bee-d64dd51e73e3", + "outputsInitialized": false, + "isAgentGenerated": false, + "language": "markdown", + "id": "PA1qSy0ipH_2" + }, + "source": [ + "## Dataset" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "originalKey": "fdcba1fc-0d70-4ca0-b27f-724355d95e7d", + "outputsInitialized": false, + "isAgentGenerated": false, + "language": "markdown", + "id": "Fp-i3-N5pH_3" + }, + "source": [ + "First, we need to download the dataset (we'll use Stanford NLP mirror)" + ] + }, + { + "cell_type": "code", + "metadata": { + "originalKey": "0b3afde6-52df-4226-acc9-68346e5d91cc", + "outputsInitialized": false, + "isAgentGenerated": false, + "language": "python", + "executionStartTime": 1734022773178, + "executionStopTime": 1734022773601, + "serverExecutionDuration": 2.28899018839, + "requestMsgId": "0b3afde6-52df-4226-acc9-68346e5d91cc", + "id": "dIAKXrvNpH_3" + }, + "source": [ + "STANFORD_SNLI_URL = \"https://nlp.stanford.edu/projects/snli/snli_1.0.zip\"\n", + "DATA_DIR = \"data\"" ], - "text/plain": [ - " sentence1 \\\n", - "0 A person on a horse jumps over a broken down a... \n", - "1 A person on a horse jumps over a broken down a... \n", - "2 A person on a horse jumps over a broken down a... \n", - "3 Children smiling and waving at camera \n", - "4 Children smiling and waving at camera \n", - "\n", - " sentence2 gold_label \n", - "0 A person is training his horse for a competition. neutral \n", - "1 A person is at a diner, ordering an omelette. contradiction \n", - "2 A person is outdoors, on a horse. entailment \n", - "3 They are smiling at their parents neutral \n", - "4 There are children present entailment " - ] - }, - "execution_count": 4, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import pandas as pd\n", - "train_path = os.path.join(snli_folder, \"snli_1.0_train.txt\")\n", - "dev_path = os.path.join(snli_folder, \"snli_1.0_dev.txt\")\n", - "\n", - "df_train = pd.read_csv(train_path, sep='\\t')\n", - "df_test = pd.read_csv(dev_path, sep='\\t')\n", - "\n", - "df_train[['sentence1', 'sentence2', 'gold_label']][:5]" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Model" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "BERT (Bidirectional Encoder Representations from Transformers) is a state-of-the-art approach to various NLP tasks. It uses a Transformer architecture and relies heavily on the concept of pre-training. \n", - "\n", - "We'll use a pre-trained BERT-base model, provided in the huggingface [transformers](https://github.com/huggingface/transformers) repo.\n", - "It gives us a PyTorch implementation for the classic BERT architecture, as well as a tokenizer and weights, pre-trained on a public English corpus (Wikipedia).\n", - "\n", - "Please follow these [installation instructions](https://github.com/huggingface/transformers#installation) before proceeding." - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "To use data.metrics please install scikit-learn. See https://scikit-learn.org/stable/index.html\n", - "100%|██████████| 433/433 [00:00<00:00, 455171.34B/s]\n", - "100%|██████████| 213450/213450 [00:00<00:00, 37577090.82B/s]\n", - "100%|██████████| 435779157/435779157 [00:11<00:00, 39433911.33B/s]\n" - ] - } - ], - "source": [ - "from transformers import BertConfig, BertTokenizer, BertForSequenceClassification\n", - "\n", - "model_name = \"bert-base-cased\"\n", - "config = BertConfig.from_pretrained(\n", - " model_name,\n", - " num_labels=3,\n", - ")\n", - "tokenizer = BertTokenizer.from_pretrained(\n", - " \"bert-base-cased\",\n", - " do_lower_case=False,\n", - ")\n", - "model = BertForSequenceClassification.from_pretrained(\n", - " \"bert-base-cased\",\n", - " config=config,\n", - ")" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The model has the following structure. It uses a combination of word, positional and token *embeddings* to create a sequence representation, then passes the data through 12 *transformer encoders* and finally uses a *linear classifier* to produce the final label.\n", - "As the model is already pre-trained and we only plan to fine-tune a few upper layers, we want to freeze all layers, except for the last encoder and above (`BertPooler` and `Classifier`)." - ] - }, - { - "cell_type": "code", - "execution_count": 6, - "metadata": {}, - "outputs": [ - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAABHAAAAPkCAYAAADBJskBAAAgAElEQVR4AeydCdiWY97/255K0kZFtihRUUlikhClxVooZUuWomxJjV0kStaUpaSUiKLFEBJS8aR1JmMnM4Mx7wzvyOt9j+M/zv/xPc15z/3cz3Wvz71cy6fjuI97u5bf+Tm/d8/3+l7ndV7VDP8gAAEIQAACEIAABCAAAQhAAAIQgAAEfE2gmq+rozgIQAACEIAABCAAAQhAAAIQgAAEIAABQ4CDCCAAAQhAAAIQgAAEIAABCEAAAhCAgM8JEOD4vIMoDwIQgAAEIAABCEAAAhCAAAQgAAEIEOCgAQhAAAIQgAAEIAABCEAAAhCAAAQg4HMCBDg+7yDKgwAEIAABCEAAAhCAAAQgAAEIQAACBDhoAAIQgAAEIAABCEAAAhCAAAQgAAEI+JwAAY7PO4jyIAABCEAAAhCAAAQgAAEIQAACEIAAAQ4agAAEIAABCEAAAhCAAAQgAAEIQAACPidAgOPzDqI8CEAAAhCAAAQgAAEIQAACEIAABCBAgIMGIAABCEAAAhCAAAQgAAEIQAACEICAzwkQ4Pi8gygPAhCAAAQgAAEIQAACEIAABCAAAQgQ4KABCEAAAhCAAAQgAAEIQAACEIAABCDgcwIEOD7vIMqDAAQgAAEIQAACEIAABCAAAQhAAAIEOGgAAhCAAAQgAAEIQAACEIAABCAAAQj4nAABjs87iPIgAAEIQAACEIAABCAAAQhAAAIQgAABDhqAAAQgAAEIQAACEIAABCAAAQhAAAI+J0CA4/MOojwIQAACEIAABCAAAQhAAAIQgAAEIECAgwYgAAEIQAACEIAABCAAAQhAAAIQgIDPCRDg+LyDKA8CEIAABCAAAQhAAAIQgAAEIAABCBDgoAEIQAACEIAABCAAAQhAAAIQgAAEIOBzAgQ4Pu8gyoMABCAAAQhAAAIQgAAEIAABCEAAAgQ4aAACEIAABCAAAQhAAAIQgAAEIAABCPicAAGOzzuI8iAAAQhAAAIQgAAEIAABCEAAAhCAAAEOGoAABCAAAQhAAAIQgAAEIAABCEAAAj4nQIDj8w6iPAhAAAIQgAAEIAABCEAAAhCAAAQgQICDBiAAAQhAAAIQgAAEIAABCEAAAhCAgM8JEOD4vIMoDwIQgAAEIAABCEAAAhCAAAQgAAEIEOCgAQhAAAIQgAAEIAABCEAAAhCAAAQg4HMCBDg+7yDKgwAEIAABCEAAAhCAAAQgAAEIQAACBDhoAAIQgAAEIAABCEAAAhCAAAQgAAEI+JwAAY7PO4jyIAABCEAAAhCAAAQgAAEIQAACEIAAAQ4agAAEIAABCEAAAhCAAAQgAAEIQAACPidAgOPzDqI8CEAAAhCAAAQgAAEIQAACEIAABCBAgIMGIAABCEAAAhCAAAQgAAEIQAACEICAzwkQ4Pi8gygPAhCAAAQgAAEIQAACEIAABCAAAQgQ4KABCEAAAhCAAAQgAAEIQAACEIAABCDgcwIEOD7vIMqDAAQgAAEIQAACEIAABCAAAQhAAAIEOGgAAhCAAAQgAAEIQAACEIAABCAAAQj4nAABjs87iPIgAAEIQAACEIAABCAAAQhAAAIQgAABDhqAAAQgAAEIQAACEIAABCAAAQhAAAI+J0CA4/MOojwIQAACEIAABCAAAQhAAAIQgAAEIECAgwYgAAEIQAACEIAABCAAAQhAAAIQgIDPCRDg+LyDKA8CEIAABCAAAQhAAAIQgAAEIAABCBDgoAEIQAACEIBAgQh8/PHH5q233irQ1tksBCAAAQhAAAIQgECUCBDgRKm3aSsEIAABCBSVwB133GFGjhxZ1H2yMwhAAAIQgAAEIACBcBIgwAlnv9IqCEAAAhDwAYFDDjnENGnSxAeVUAIEIAABCEAAAhCAQNAJEOAEvQepHwIQgAAEfElg7dq1plq1avaxcOFCX9ZIURCAAAQgAAEIQAACwSFAgBOcvqJSCEAAAhAIEIErr7wyFuAMGDAgQJVTKgQgAAEIQAACEICAHwkQ4PixV6gJAhCAAAQCT2D33XePBTgaifPdd98Fvk00AAIQgAAEIAABCECgdAQIcErHnj1DAAIQgEBICSxZsqRCeKMA58EHHwxpa2kWBCAAAQhAAAIQgEAxCBDgFIMy+4AABCAAgUgRGDJkSKUA56ijjooUAxoLAQhAAAIQgAAEIJBfAgQ4+eXJ1iAAAQhAIOIEfvzxR1OzZs1KAY5G4WzdujXidGg+BCAAAQhAAAIQgECuBAhwciXHehCAAAQgAAEPAjNnzvQMbxTg3HDDDR5r8BEEIAABCEAAAhCAAATSEyDASc+IJSAAAQhAAAIZE+jVq1fSAKdVq1YZb4cFIQABCEAAAhCAAAQgEE+AACeeBq8hAAEIQAACVSDw+eefJw1vNAJHj9dee60Ke2BVCEAAAhCAAAQgAIGoEiDAiWrP024IQAACEMg7gbvuuittgDN8+PC875cNQgACEIAABCAAAQiEnwABTvj7mBZCAAIQgECRCHTq1CltgFO/fn3z//7f/ytSRewGAhCAAAQgAAEIQCAsBAhwwtKTtAMCEIAABEpKoLy8PG144y6jmjdvXklrZecQgAAEIAABCEAAAsEjQIATvD6jYghAAAIQ8CGBMWPGZBzgnHTSST5sASVBAAIQgAAEIAABCPiZAAGOn3uH2iAAAQhAIDAE9tprr4wDHI3E+fOf/xyYtlEoBCAAAQhAAAIQgEDpCRDglL4PqAACEIAABAJO4KWXXsoqvFGAM3Xq1IC3mvIhAAEIQAACEIAABIpJgACnmLTZFwQgAAEIhJLAeeedl3WAc/jhh4eSBY2CAAQgAAEIQAACECgMAQKcwnBlqxCAAAQgEBECP//8s6lTp07WAY5G4WzYsCEilGgmBCAAAQhAAAIQgEBVCRDgVJUg60MAAhCAQKQJPPnkkzmFNwpwxo4dG2l2NB4CEIAABCAAAQhAIHMCBDiZs2JJCEAAAhCAQCUCTz/9tNFjwYIFscczzzxj9FBIo8eUKVPMwoUL7eO5554z7vH8889X2h4fQAACEIAABCAAAQhAwIsAAY4XFT6DAAQgAAEIVJHAL7/8EgtwtmzZUsWtsToEIAABCEAAAhCAQNQJEOBEXQG0HwIQgAAECkKAAKcgWNkoBCAAAQhAAAIQiCwBApzIdj0NhwAEIACBQhNwl1Bt3ry50Lti+xCAAAQgAAEIQAACISdAgBPyDqZ5EIAABCBQOgIEOKVjz54hAAEIQAACEIBA2AgQ4IStR2kPBCAAAQj4hoALcDZt2uSbmigEAhCAAAQgAAEIQCCYBAhwgtlvVA0BCEAAAgEgUL16dTuRMQFOADqLEiEAAQhAAAIQgIDPCRDg+LyDKA8CEIAABIJLgAAnuH1H5RCAAAQgAAEIQMBvBAhw/NYj1AMBCEAAAqEhUKNGDTsCZ+PGjaFpEw2BAAQgAAEIQAACECgNAQKc0nBnrxCAAAQgEAECBDgR6GSaCAEIQAACEIAABIpEgACnSKDZDQQgAAEIRI9AzZo17QicDRs2RK/xtBgCEIAABCAAAQhAIK8ECHDyipONQQACEIAABP5DgADnPyx4BQEIQAACEIAABCBQNQIEOFXjx9oQgAAEIACBpARcgPP+++8nXYYvIAABCEAAAhCAAAQgkAkBApxMKLEMBCAAAQhAIAcCtWrVspdQEeDkAI9VIAABCEAAAhCAAAQqECDAqYCDNxCAAAQgAIH8ESDAyR9LtgQBCEAAAhCAAASiToAAJ+oKoP0QgAAEIFAwAmVlZXYEzvr16wu2DzYMAQhAAAIQgAAEIBANAgQ40ehnWgkBCEAAAiUgQIBTAujsEgIQgAAEIAABCISUAAFOSDuWZkEAAhCAQOkJ1K5d247AKS8vL30xVAABCEAAAhCAAAQgEGgCBDiB7j6KhwAEIAABPxMgwPFz71AbBCAAAQhAAAIQCBYBApxg9RfVQgACEIBAgAi4AOe9994LUNWUCgEIQAACEIAABCDgRwIEOH7sFWoKBIGvv/7arFmzxsybN89MmDDBDBs2zBx77LE8YIAG0EBMAzVq1LCXUHXu3Dn2Gf9P8P8kGkADw4cPNxMnTjTz5883q1evNtu3bw+E96FICEAAAhAoLQECnNLyZ+8BI/Dzzz+bWbNmmRNOOMEelFWrVo1nGKABNIAG0AAaQANV1kDHjh3N1KlTzXfffRcwd0S5EIAABCBQLAIEOMUizX4CTeCjjz4yt956q2nTpk0lg9a4YQPTuUM7c+apJ5obx4zgAQM0gAbQABpAA2ggpQYGnd7Xeof69etV8hUtWrQw48ePN9u2bQu0d6J4CEAAAhDIPwECnPwzZYshIyATVbdOnZjBarBLfXP+4NPMi09NM3989yXzv19v5gEDNIAG0AAaQANoICcNfLHpNfPqoplm+DkDTXygU1ZWZq655pqQuSqaAwEIQAACVSFAgFMVeqwbagI//PCDGTJkSCy4OfrIw8zU28eZzze+mpNBI+gh6EIDaAANoAE0gAZSaeCf29ebOdMnmdP7/+dS7R49ephvv/021J6LxkEAAhCAQGYECHAy48RSESOwdetW06hRo1h4M+OeWwhtOLOKBtAAGkADaAANFE0Di558wDTbrYn1IvXq1TMrVqyImBujuRCAAAQgkEiAACeRCO8jT2DatGmx4OagA/Y3G954vmhmLdVZOb7jrC0aQANoAA2ggWhpYNuapab9Qa1jvmTKlCmR92kAgAAEIBBlAgQ4Ue592l6JwNy5c2MmafCAfgQ3nGlFA2gADaABNIAGSq6BE3seFfMnK1eurORf+AACEIAABKJBgAAnGv1MKzMgsGXLlpg5uvi8M0tu1jjLGq2zrPQ3/Y0G0AAaQAOpNHDt5cNiPmX79u0ZOBsWgQAEIACBsBEgwAlbj9KenAm0bv3rEOUDW7ckvOFsKxpAA2gADaABNOA7DQw8uZcNcbp165az32FFCEAAAhAILgECnOD2HZXnkcDgwYOtIapdVmZ+/84S3xm2VGfk+I4ztmgADaABNIAGoqOBzh3aWc8ycuTIPDohNgUBCEAAAkEgQIAThF6ixoISmD17dmxI8sOTbyK84YwrGkADaAANoAE04FsNrH99oWncsIH1LjNnziyoR2LjEIAABCDgLwIEOP7qD6opAYF+/fpZE3TB2af51qxxZjU6Z1bpa/oaDaABNIAG0mng3jvGWe/So0ePEjgndgkBCEAAAqUiQIBTKvLs1xcEysvLY6NvXl00kwCHM65oAA2gATSABtCA7zXw6fsrTFlZLethlixZ4gtPRREQgAAEIFB4AgQ4hWfMHnxMYOzYsb+ewerWxfdmLd3ZOL7njC0aQANoAA2ggeho4NS+Pa2HGThwoI+dFqVBAAIQgEA+CRDg5JMm2woUgR07dpiWLVta8zPn4UkEOJxxRQNoAA2gATSABgKjgfmPTrEeplq1ambt2rWB8mAUCwEIQAACuREgwMmNG2uFgMDy5cut8alfv15gzBpnVqNzZpW+pq/RABpAA2ggnQYaNdzFeplx48aFwJnRBAhAAAIQSEeAACcdIb4PLYHp06db09OvVw8CHM64ogE0gAbQABpAA4HTwOn9T7BeZtCgQaH1azQMAhCAAAT+Q4AA5z8seBUxAuPHj7emZ/g5AwNn2NKdkeN7ztqiATSABtAAGgi/Bq4bPdx6ma5du0bMxdFcCEAAAtEkQIATzX6n1cYYna3SdeM3jhlBgMNZVzSABtAAGkADaCBwGnj8/gnWyzRr1gxvBwEIQAACESBAgBOBTqaJ3gS6dOliTc/Dk28KnGHjrGr4z6rSx/QxGkADaAANpNPAm0vnWC+jE1Lff/+9t+HhUwhAAAIQCA0BApzQdCUNyZZA48aNrelZPOdBAhzOuqIBNIAG0AAaQAOB08Bftr0ZC3A2bdqUrRVieQhAAAIQCBgBApyAdRjl5o+Azlbp8eqimYEzbOnOyPE9Z23RABpAA2gADURDA87PrFq1Kn8miS1BAAIQgIAvCRDg+LJbKKoYBJzhIcCJhsHlQIZ+RgNoAA2ggTBqwPkZApxiuEf2AQEIQKC0BAhwSsufvZeQgDM8BDgY+jAaetqErtEAGkAD0dCA8zMEOCU0lewaAhCAQJEIEOAUCTS78R8BZ3gIcKJhcDmQoZ/RABpAA2ggjBpwfoYAx39ek4ogAAEI5JsAAU6+ibK9wBBwhocAB0MfRkNPm9A1GkADaCAaGnB+hgAnMBaUQiEAAQjkTIAAJ2d0rBh0As7wEOBEw+ByIEM/owE0gAbQQBg14PwMAU7QnSn1QwACEEhPgAAnPSOWCCkBZ3gIcDD0YTT0tAldowE0gAaioQHnZwhwQmpYaRYEIACBOAIEOHEweBktAs7wEOBEw+ByIEM/owE0gAbQQBg14PwMAU60fCythQAEokmAACea/U6rjTHO8BDgYOjDaOhpE7pGA2gADURDA87PEOBgbyEAAQiEnwABTvj7mBYmIeAMDwFONAwuBzL0MxpAA2gADYRRA87PEOAkMXx8DAEIQCBEBAhwQtSZNCU7As7wEOBg6MNo6GkTukYDaAANREMDzs8Q4GTnA1kaAhCAQBAJEOAEsdeoOS8EnOEhwImGwS3FgczfPl5j/vKHN2MPvS9FHaXe58hhg820u2+sctt/+tNG84/P3s1qO7979lFz/uDTzPbNr2e1XqmZFWr/YliobSduN1/9nrjdsL4vZt+ElWFU2+X8DAFOXuwhG4EABCDgawIEOL7uHoorJAFneAhwCHDSmf4fvnjPrH3lafPC3IfM1tUvmv/5c/qD4I/LXzZ169SJzbUkvQ08uVfRDp7TtamY3zdqsIs569Q+ObddIUyXTu1N7bIyy7PeTnVNy332NMd1P8KIc6q23DNhrF1H/ZZquSh8N/P+280Jx/ymaByq2u9R6BPXxhefmmYOP/Rg8+P294vWP27fPAf/b6DzMwQ4hXSNbBsCEICAPwgQ4PijH6iiBASc4SHACb55LdQByM9/2WRuunakadK4kQ1jFCLsuUdzowDhikvOsaNBjjqis/lswwrPgy6tr7BHjwfuvJ4A5+vstbZ8wQyzS/2dzfgrL7Ih2n99stb8cd1y8+bSOWbRnAeMwrVU/U+A8yvzb/74tlGg8sSDd3jy6nvC0aZGjRqejyFnnOS5Tiru+o4AJ3O9//eX5WaP5k3N1Nuvy4l1ur7g+8z7IoisnJ8hwCmBmWSXEIAABIpMgACnyMDZnX8IOMNDgBNuY1sVM37lpeeazh3amTdenG3+uX197MDqw3dfMmcP6Gf223cvU3/nemb9yoWx75Lt76G7bohsgPP7d5aYzze+mpaRFzsFZGMuuyCndbW9bz54y2x+64UK/ee1n7B/Job7t9zbJLtMR/r+09Y37EOXnEn77n2uo0Kq0u9h7w+v9ils3LVJI6OQ0ut7PuNvVTINOD9DgOMfj0klEIAABApFgACnUGTZru8JOMNDgIMp9jLFd9xwpenW9dCUB/43j73MXp5DgFM4DemANhO+Xn3IZ7/2i0Z3NG7YwNx7x7iMgoERwwaZ66++JKNlYZw/7Wt+J402m3HPzbDPYbRelLXo/AwBju+tJwVCAAIQqDIBApwqI2QDQSXgDA8BTv4OQMJioDe88bwpK6tl9JyqTbp8RyNw0i2nbeQ6AkdzvMx/dIq5+5YxRnOYvPvqMylrSqxXoyBmT7vTrv/kw5NiYUimoyreWjbXPHrvrWbKbWPNs7PuNZ++/4rdf7r1v/1wtZ04WJMH66H3ibUle68DWbeeLsNZ+cITsffu81SXTu34akOl5fVZsv15fa7lFRyJnS5/Uw1//fCdtNv4/vP3YvvWBNbx2xaz1xfPMuqT+M8L/XrujLusnhPrSbbfqgQ4ufa7Rkqpb8VPdYn1S888Yu6bON6yT9Xfie3QJXYLn7jPXo60ZN4088Wm1zLmrf1oxN3Dk28yGhGjy/Qy6Xct47T53UcVJyvX+6XzH7aTmSfWmvj+3EGn2PmeEj/nPX+nUmnA+RkCnKA6UuqGAAQgkDkBApzMWbFkyAg4w0OAgzFONMaa32bYkAEZHfQte3p6RhOPZhvgTLzxKrPv3i3MzvV2Mt2PPMxestWvVw+z266NjZ7T3dHq621vmd7HHWXP6Guy30Gn9TUHtt7PjhjSe71+53fzk7Zx3YoFdhnNy6H5UQac1Ms0b7abqVWrpjlv0KmmSaOGRnP8JLLTe11ipnmCqlevbh/6rZ3U+1jPZb3WP7jtARXWdduJf9alVV7r6rOLzj0jtr7W0f6zCb5WL3/KtDuwlZ2TRAw1+bRqUl/ooD7VJNaaJ8nV2bbN/rZGhUGaS0mjKzT5si690+iuLzcV585YPY8+wk74nIxX4ue5Bji59ruCsjq1a9t+0uiTBY/fYxo2qG/E75Q+x5kGu+xs2SlYSaw1/r2CmpNPPNYur9+MNN/1sA5mp7p1zPBzBqb8zXywdpnpdWw3W8cB++9r9ar1NbGwRoEpRI3fV/xrXWqmINdp7dILzrLLKjDTNhQGd2x/oGnWdFdzwdmnm1Rh4uI5D1oO7732bNL9xe+b1/z9kgacnyHACZlRpTkQgAAEPAgQ4HhA4aNoEHCGhwAHA5x4EKRw4LH7bsvrAVS2Ac5zs+83ry2aVekSLo1Q0EHh4NP7pqxPB+2645Amr41vn0IDHaBL/7q7U/x37rVGwOgAetwVF1UKpzSfzG8O72TXT3Ug6ralZ43gySbAiV9XQZH2Gf9Ztq/3arF7xgHO049NsZcbTZ9yc6X5Yla/NM8cekhbc+YpJ6atRyN2FEDoIF5hTaeDDzIKhlS7uCkkVFCRbVuyXf6rLSttsKBbeme6bq4BTuL2s+33U/v2NIe0bWMDrvjL5jQq5vH7JthgRu1J3I/eK4DRZWJXjzy/0ogZzb+ku6C1arl30vllNHpGI9zcCLP4fSg4kg4zGWmnyyoV4CgAVfip36q7U5qCHrVRI4Titx//Wu3Qb3PsqAuTLhO/PK/5+yUNOD9DgBMN/0orIQCBaBMgwIl2/0e69c7wEOBggBMPgjRCQpcOJX5elffZBjip9qW7XmlUQbIARSNj9P3GVYs826A5UTS6R3dy8trP28vm2tE2ybavQEXbTzUSJX672R7Ix69bzABHXJvu1sReuhNfQ/xrBWIavZQufFGAoxEX0tKFQwdUCuLit1nI16pD/9dlc3ejUgY4LXZvFgs8ErkoDNGlTYmfa2JmjbS5ccyISt+5ZfWbUIiTTZDl1tWzRs7cNn5U0u27ZRXgdGjXxgagXrW65ZI9qy0asaMRb8mW4XP+ZiVqwPkZApxI21oaDwEIRIQAAU5EOppmVibgDA8BDmY40Qwr3Fjz8tN5PYDKZ4CjEQnSb+JcG/Ht0KU/GuWRbK4ajQZIdgmUm/Q21WUjyUZCxNfgXgclwNGtytONbFKbXpj7kL2kyrXP61nBiS6pufPGq/OqI699pfps9kMTrVY0F0yq5eK/K2WAo99JfC3xrxWgKAyL/0yvNZJszz2apw0UNc9O3Tp1MpqLJnEf114+zP6eEj9PfK8AR6PXko1uS1ze633r/fYxh3VsV6mdXsvyGX+/pAHnZwhwKns9PoEABCAQNgIEOGHrUdqTMQFneAhwMMCJB0G61EIH4ImfV+V9LgHOi09NM7qls+as0eU4ml9Fc3xcNeK8tAGOLgnRehrBokupLh8+xI7CeGXhY7GJYlO1R5f7aL6Wffbaw172oQNYXVa09pWn0x4oJ243KAGOQi/NlaLL51I9NLJCI5CShWNqv/Tj5sBJ5FHM97qbmv6v2/Tm4oz1nEmAk+x25PFty7bfdXlRqjsw3X79FZ5zU6mNmvMmVZ+57zS/jS5NjK/TvVaoqVE8p/U73nTu0M6GdNquRt/oMjgFom7ZZM/uEqpk32fy+fE9jrQj4DJZlmX4+yUNOD9DgJOxBWRBCEAAAoElQIAT2K6j8KoScIaHAAcDnHgQpMDj/om/TXuwlrheqvfZBDgaWaN5ZjRhsUaF6O5ROuhcvmCG0YiKURcNTRvguFp0x6OnHrnb3DpulB1dogPY3ZvvZjRZqlsm2bMO0jXRseYfUR0Kj3SJkSZ2zWZemmwP5OPrKeYlVBr1oLb1Ob57Rg+FZPG1xr/2S4DzyNRbrFZefX5m0lrj69brdAGOdKiwJXG9xPfZ9nuuAY7mvdm/5d4Z9Zn6VvUn1vrEg3fYSapP7NndPDjpeqPwVL+555+839x18zV2EuJiBTi6BEuTaCfWyHv+ViXTgPMzBDhVdYasDwEIQMD/BAhw/N9HVFggAs7wEOBgihNNsSYQPXtAv7weQGUT4GgEwOn9TzCaTDixNr3P5BIqr/XcZwoXdMnJMzOnem7fLef1rFEnCgV0Z6BMb8+c7YF8/H6LGeBopIVGecTvP9fXfglwXl74qA1wNHoq07akC3AUCPbo1iXt9rLt91wDHIWauoNUpu1LXE4TDut29alCrmwuoXJ3oUrcT6bvdbcz3fkt0+VZjr9hzs8Q4BTIMLJZCEAAAj4iQIDjo86glOIScIaHAAfzm3gApDsHNWncyKxa8mTagygtm+quMm7bmQY4f/90nb2VceLdo9x29JwuwNHogWQTFLvtaFRPsjtD6VbZf/79qpRt11wkk2+9NuUybl/ZHsi79fRczABHl9noblGZTs4cX2fia78EOO6uRtdcdn5GfaV2BC3A2bZ2mb2kLdWIqMT+iX+vu62NvnhoSj7FCnB0lzj9bcp1suX4dvE6On/bnJ8hwCmuj2RvEIAABEpBgACnFNTZpy8IOMNDgBMdk5vNAY3m4tBcOLoFcbL1yl9faEeyzHrg9qTLuHUzDXC+/XC1qVmzhvnDmqWe29TlVbobj/SbbBLji849w5xxSm/P9V09994xLmmA06ZVS3sZiVvW61mjhMIW4Ci40aVr140enpKdeO5iFFAAACAASURBVGx5+4VKtxmP5+SXAEc1qT9P6XNc2ja5+oMW4KhujZrTfDXJ7pzm2qZRY4nhqMKbIWeclJSPAlFdzliMS6h02ZZ+2xo55Wrmmb9R6TTg/AwBji/sJUVAAAIQKCgBApyC4mXjfibgDA8BDuY4mTm+/upL7AiQeY9MrjAqQ5c2zZ1xl2ncsEHSuwz99cN3zNfb3oo9Jt10tQ1M4j/Ta687QR3bvasNYOJHwfxz+3o7d4fukHX0bw6zB3kfvfc781+frK10oKeJeHUHJM2V8/3n71X4XiGFatedcpIdJGqeHF3GoUulEuvT/n571cVmj+ZNjUYLeLFTsBTfTt3CWhMEx3+WrO3a3t8+/s/6Yqxbume6rg7gE5fVralXPPd4hc810smrdo2m2qvF7ubcQacYrxEdGpWltmhkkNeduDQ6SsxfeuYRG5wk1pIsdPOqJV+fSXvqb9WWbJu6NM7VqkvJNK+Me5/4rHDT6xKqXPtdGtM+dNnQlNvGegaTYqrfo4IWL83r96GaFMDpkqjEdn747ktGwWa9neqapfMfrvD964tnmdplZVYj8aOvFPZo8m+x69j+QHPJ+WfZOuOX0X5Uv+pzNWri8URm+i6xJq/3GumjycMTf3dey/IZf7ucBpyfIcDxs+ukNghAAAL5IUCAkx+ObCWABJzhIcDBBDsT7PWs+TV0QFV/53r2DP8Rh3UwZWW1zEEH7Jf0MqW3l801tWrVtCGL01myZ685V3TgeNQRnU2NGjWM9tf1sA72sioFEQpf3CVU2qYOSBNHCelOOmMuu8CGQKr1wNb72fBIB7gKXjQiQ5OzerVXn/U8+gjzwJ3X27teafu6I49GcKgWzX2j7zeuWuS5vg6etc9k7Y3/XHUm1qBQSnd4il/O67VCicR19V5zB3ktn/iZ2qGgyGsbCiJ0sF+ndm07mawuNdMEx7vU39ns2qSRvbzlsw0rKq2reWYS95P4Xmx0Jy+v/RbqM13mp4Ai1W3hFZ4k1prqfWKAU5V+n3jjVRX2LUbxIYzCJXF39Ugfn6x/pRJDBSsKC9VPCjo1IbFG5WjibelYc+ysfmlepfXEXXrXb1wjbRTQaY4o/f402k1BpYIVt38Fo/F9pfmy3HfJnlVDutFBql/71Z2s4rfPa/5GpdOA0x0BTgDNKCVDAAIQyJIAAU6WwFg8PASc4SHAwRynM8c6G66RGZr0d+ET93mOvEi3jVy+X/Py02b2tDvNs7PuNQo2ctmGwh3dOlzz0Kj+bG8DvnX1i+aFuQ/Z0Ti6lER3tcqljiCuo4BHo390F67fPfuo54icoLRLQYRGbgWl3qrUqcBn/cqF9u5tz82+32geoMRRM17bV9C1ZN40ezty9XuyUVpe6+bjM/3fovDq0/crh1P52D7bCO/fOudnCHDC41FpCQQgAIFkBAhwkpHh89ATcIaHACe8ppYDFvoWDfyqAQVvGhWmy4Vg4s/fxaGHtGXy4q/92Td+/804P0OAE3rrSgMhAAEIGAIcRBBZAs7wEOBgmP1uzqkPjeZDA5rXpl+vHgQ4PgwJNB9V82a7GY0Cykdfs41o/Z/h/AwBTmQtLQ2HAAQiRIAAJ0KdTVMrEnCGhwAnWkaXAxv6O6oa0ES/urQoqu33c7s175Juh+7nGqnNv/93Oj9DgFPR5/EOAhCAQBgJEOCEsVdpU0YEnOEhwPGvKeWAgb5BA2gADaABNJBaA87PEOBkZP9YCAIQgECgCRDgBLr7KL4qBJzhIcBJbQwxzvBBA2gADaABNOBfDTg/Q4BTFVfIuhCAAASCQYAAJxj9RJUFIOAMDwGOf00pBwz0DRpAA2gADaCB1BpwfoYApwBmkU1CAAIQ8BkBAhyfdQjlFI+AMzwEOKmNIcYZPmgADaABNIAG/KsB52cIcIrnIdkTBCAAgVIRIMApFXn2W3ICzvAQ4PjXlHLAQN+gATSABtAAGkitAednCHBKbi0pAAIQgEDBCRDgFBwxO/ArAWd4CHBSG0OMM3zQABpAA2gADfhXA87PEOD41XFSFwQgAIH8ESDAyR9LthQwAs7wEOD415RywEDfoAE0gAbQABpIrQHnZwhwAmZEKRcCEIBADgQIcHKAxirhIOAMDwFOamOIcYYPGkADaAANoAH/asD5GQKccPhTWgEBCEAgFQECnFR0+C7UBJzhIcDxrynlgIG+QQNoAA2gATSQWgPOzxDghNq20jgIQAAClgABDkKILAFneAhwUhtDjDN80AAaQANoAA34VwPOzxDgRNbS0nAIQCBCBAhwItTZNLUiAWd4CHD8a0o5YKBv0AAaQANoAA2k1oDzMwQ4FX0e7yAAAQiEkQABThh7lTZlRKBx48ZGpmfxnAcN5jC1OYQPfNAAGkADaAAN+E8Df9n2pvUy8jObNm3KyP+wEAQgAAEIBJcAAU5w+47Kq0igS5cu1vQ8PPkmApyv/WdKOVCgT9AAGkADaAANpNbAm0vnxAKc77//vorOiNUhAAEIQMDvBAhw/N5D1FcwAoMGDbKm58YxIwhwCHDQABpAA2gADaCBwGng8fsnWC/TrFmzgvklNgwBCEAAAv4hQIDjn76gkiITGD9+vDU9w88ZGDjDxhnJ1Gck4QMfNIAG0AAaiIIGrhs93HqZrl27FtlFsTsIQAACECgFAQKcUlBnn74gMH36dGt6+vXqQYDDWVc0gAbQABpAA2ggcBo4vf8J1stoVDH/IAABCEAg/AQIcMLfx7QwCYHly5db01O/fr3AGbYonFWkjZw9RwNoAA2gATSQWgONGu5ivcy4ceOSuB0+hgAEIACBMBEgwAlTb9KWrAjs2LHDtGzZ0hqfOQ9PIsThzCsaQANoAA2gATQQGA3Mf3SK9TC6A9XatWuz8kAsDAEIQAACwSRAgBPMfqPqPBEYO3asNT89unUJjGHjbGTqs5HwgQ8aQANoAA1EQQOn9u1pPczAgQPz5IrYDAQgAAEI+J0AAY7fe4j6CkqgvLw8dvbq1UUzCXE484oG0AAaQANoAA34XgOfvr/ClJXVsh5myZIlBfVKbBwCEIAABPxDgADHP31BJSUi0K9fP2uALjj7NN8btiicUaSNnDlHA2gADaABNJBaA/feMc56lx49epTIPbFbCEAAAhAoBQECnFJQZ5++IjB79uzYKJyHJ99EiMOZVzSABtAAGkADaMC3Glj/+kLTuGED611mzpzpK09FMRCAAAQgUFgCBDiF5cvWA0Jg8ODB1gjVLiszv39niW9NG2ckU5+RhA980AAaQANoIOwa6NyhnfUsI0eODIjLokwIQAACEMgXAQKcfJFkO4En0Lp1a2uIDmzdkgCHM69oAA2gATSABtCA7zQw8ORe1qt069Yt8L6LBkAAAhCAQPYECHCyZ8YaISWwZcsWa4p0O86LzzvTd6Yt7GcUaR9nzdEAGkADaAANJNfAtZcPi/mU7du3h9SN0SwIQAACEEhFgAAnFR2+ixyBuXPnxszR4AH9CHE4+4oG0AAaQANoAA2UXAMn9jwq5k9WrlwZOX9GgyEAAQhA4FcCBDgoAQIJBKZNmxYzSQcdsL/Z8MbzJTdunJFMfkYSNrBBA2gADaCBsGpg25qlpv1Bv17irRHCU6ZMSXAtvIUABCAAgSgRIMCJUm/T1owJbN261TRq1CgW5My45xZCHM7AogE0gAbQABpAA0XTwKInHzDNdmtivUi9evXMihUrMvYxLAgBCEAAAuEkQIATzn6lVXkg8MMPP5ghQ4bEQpyjjzzMTL19nPl846tFM29hPaNIuzhbjgbQABpAA2igsgb+uX29mTN9kjm9/wkx/9GjRw/z7bff5sHZsAkIQAACEAg6AQKcoPcg9RecwPjx401ZWVnMSDXYpb45f/Bp5sWnppk/vvsSYQ5nY9EAGkADaAANoIGcNfDFptfMq4tmmuHnDDT169eL+Q15j2uuuabgPocdQAACEIBAcAgQ4ASnr6i0hAQ++OADoyCnRYsWMWOla9H1aNywgencoZ0589QTzY1jRvCAARpAA2gADaABNJBSA4NO72u9Q3xg43yFvIY8x7Zt20rofNg1BCAAAQj4kQABjh97hZp8TWDmzJmmS5culYIcZ7x4/jXYggMc0AAaQANoAA1kroGOHTuaqVOnmu+++87XPojiIAABCECgdAQIcErHnj0HnMCPP/5oPv30U7NmzRqzePFiM2PGDHPrrbfygAEaQAMxDQwaNMj069cv9p7/I/g/Eg2gAWlAnmHRokXm7bffNhrl+7e//S3grojyIQABCECgGAQIcIpBmX1AAAIQgEAkCSjA6d69eyTbTqMhAAEIQAACEIAABPJLgAAnvzzZGgQgAAEIQMAS0J3s3OUjf/jDH6ACAQhAAAIQgAAEIACBKhEgwKkSPlaGAAQgAAEIeBN49NFHYwHOjTfe6L0Qn0IAAhCAAAQgAAEIQCBDAgQ4GYJiMQhAAAIQgEA2BHr27BkLcA444IBsVmVZCEAAAhCAAAQgAAEIVCJAgFMJCR9AAAIQgAAEqkbgk08+iYU37jKqlStXVm2jrA0BCEAAAhCAAAQgEGkCBDiR7n4aDwEIQAAChSBwxx13VApwLr744kLsim1CAAIQgAAEIAABCESEAAFORDqaZkIAAhCAQPEIHHLIIZUCnAYNGph//etfxSuCPUEAAhCAAAQgAAEIhIoAAU6oupPGQAACEIBAqQmsXbu2UnjjLqOaP39+qctj/xCAAAQgAAEIQAACASVAgBPQjqNsCEAAAhDwJ4GrrroqaYBzyimn+LNoqoIABCAAAQhAAAIQ8D0BAhzfdxEFQgACEIBAkAjsscceSQMcjcT5+uuvg9QcaoUABCAAAQhAAAIQ8AkBAhyfdARlQAACEIBA8AksXbo0ZXijAOfee+8NfkNpAQQgAAEIQAACEIBA0QkQ4BQdOTuEAAQgAIGwEhg6dGjaAKdr165hbT7tggAEIAABCEAAAhAoIAECnALCZdMQgAAEIBAdAjt27DC1atVKG+BoFM7GjRujA4aWQgACEIAABCAAAQjkhQABTl4wshEIQAACEIg6gVmzZmUU3ijAGTduXNRx0X4IQAACEIAABCAAgSwJEOBkCYzFIQABCEAAAl4EevfunXGAs++++3ptgs8gAAEIQAACEIAABCCQlAABTlI0fAEBCEAAAhDIjMAXX3yRcXijETh6vPzyy5ltnKUgAAEIQAACEIAABCBgjCHAQQYQgAAEIACBKhK46667sg5wzj///CruldUhAAEIQAACEIAABKJEgAAnSr1NWyEAAQhAoCAEDj300KwDnLp165qff/65IPWwUQhAAAIQgAAEIACB8BEgwAlfn9IiCEAAAhAoIoHy8vKswxt3GdWTTz5ZxErZFQQgAAEIQAACEIBAkAkQ4AS596gdAhCAAARKTmDMmDGmffv2scfBBx9s3MMFNW3atDEdOnQwHTt2tI9OnToZPfr06VPy+ikAAhCAAAQgAAEIQCAYBAhwgtFPVAkBCEAAAgEj8Msvv8RG5mzZsiVg1VMuBCAAAQhAAAIQgIDfCBDg+K1HqAcCEIAABEJBgAAnFN1IIyAAAQhAAAIQgIBvCBDg+KYrKAQCEIAABMJGwF1CtXnz5rA1jfZAAAIQgAAEIAABCBSZAAFOkYGzOwhAAAIQiA4BApzo9DUthQAEIAABCEAAAoUmQIBTaMJsHwIQgAAEIkuAACeyXU/DIQABCEAAAhCAQN4JEODkHSkbhAAEIAABCPxKoHr16nYi402bNoEEAhCAAAQgAAEIQAACVSJAgFMlfKwMAQhAAAIQSE6AACc5G76BAAQgAAEIQAACEMiOAAFOdrxYGgIQgAAEIJAxgRo1atgROBs3bsx4HRaEAAQgAAEIQAACEICAFwECHC8qfAYBCEAAAhDIAwECnDxAZBMQgAAEIAABCEAAApYAAQ5CgAAEIAABCBSIgAtwNmzYUKA9sFkIQAACEIAABCAAgagQIMCJSk/TTghAAAIQKDqBmjVr2kuoCHCKjp4dQgACEIAABCAAgdARIMAJXZfSIAhAAAIQ8AsBAhy/9AR1QAACEIAABCAAgeATIMAJfh/SAghAAAIQ8CmBWrVq2RE477//vk8rpCwIQAACEIAABCAAgaAQIMAJSk9RJwQgAAEIBI4AAU7guoyCIQABCEAAAhCAgG8JEOD4tmsoDAIQgAAEgk7ABTjr168PelOoHwIQgAAEIAABCECgxAQIcErcAeweAhCAAATCS6CsrMxeQkWAE94+pmUQgAAEIAABCECgWAQIcIpFmv1AAAIQgEDkCLgAp7y8PHJtp8EQgAAEIAABCEAAAvklQICTX55sDQIQgAAEIBAjULt2bTsChwAnhoQXEIAABCAAAQhAAAI5EiDAyREcq0EAAhCAAATSESDASUeI7yEAAQhAAAIQgAAEMiVAgJMpKZaDQBoC33zzjVm3bp1ZtWoVDxigATRgNeAuoZoxYwaaQBNoAA3ENLB9+/Y0roKvIQABCEAAApUJEOBUZsInEEhLYNGiRebKK680p556qunQoYNp0KCBvUyiWrVqPMMADaABNIAG0AAaSKsBjdBr27at6d+/vxk5cqRR0PvTTz+l9SAsAAEIQAAC0SVAgBPdvqflWRL49ttvzX333WcOP/zwtKaMIIcgCw2gATSABtAAGshWA61btzYTJkwwX375ZZYuhcUhAAEIQCAKBAhwotDLtLFKBLZu3Wquvvpq07x58wrBTc+jjzAXDh1gbhs/ysx7ZLIpf+1Z8/1n75r//XozDxigATSABtAAGkADSTWwbe0ys+zp6eb+O39rrrjkHNO/9zGm/s71Yj6jUcOGZvTo0Wbz5s1V8jCsDAEIQAAC4SJAgBOu/qQ1eSawfPlys88++8QM1QGt9jXXXTHcrFuxIKkpI8AhwEIDaAANoAE0gAay1cAn618xk2+91nQ7vFPMdzRtupuZOXNmnt0Nm4MABCAAgaASIMAJas9Rd8EJTJ8+PWagjuzS0Y6y+elPGwluOKOKBtAAGkADaAANFFQDK5573Jx1Wp+YDxkxYkTBfQ87gAAEIAAB/xMgwPF/H1FhCQgMGDAgZpouPu9M8/dP1xXUqGV7lo7lObOLBtAAGkADaCD8Ghh10ZCYH+nSpUsJHBG7hAAEIAABPxEgwPFTb1CLLwjETzg4dcJ1BDecZUUDaAANoAE0gAZKpoF7bx8XC3HkUfgHAQhAAALRJcBfgej2PS33IKCzWy7AWTznwZKZNc6qhv+sKn1MH6MBNIAG0ECmGti2ZmnMn2iUMP8gAAEIQCCaBAhwotnvtNqDgK4vd+GNznZlaqpYDgOOBtAAGkADaAANFFoD27esjPkUzdPHPwhAAAIQiB4BApzo9Tkt9iCgOzy48EaTBhbahLF9jD4aQANoAA2gATSQrQaWL5gR8yu6Uyb/IAABCEAgWgQIcKLV37TWg8DmzZtN06ZNrSE6rntXwhvmOUADaAANoAE0gAZ8q4FJN11tPcs+++xjtm7d6uFs+AgCEIAABMJKgAAnrD1LuzImMHbsWGuE2h/U2nz47ku+NWzZnqVjec7sogE0gAbQABoIpwaGnHGS9S6jR4/O2O+wIAQgAAEIBJ8AAU7w+5AWVIHAjh07TMuWLa0Jmno7d5zC6IfT6NOv9CsaQANoIFwaWPHc49a7NGrU0Hz55ZdVcEKsCgEIQAACQSJAgBOk3qLWvBOYPXu2NUB7tdjdfPPB24y+Ycg8GkADaAANoAE0EAgNdDu8k/UwEyZMyLs/YoMQgAAEIOBPAgQ4/uwXqioSgX79+lnzc90VwwNh1jiDGq4zqPQn/YkG0AAaQAO5amDyrddaD9O6dWvz008/Fck5sRsIQAACECglAQKcUtJn3yUlUF5ebo1P7bIys+WtFwhwOOOKBtAAGkADaAANBEYDn6x/xdTfuZ71MjNmzCipp2LnEIAABCBQHAIEOMXhzF58SGD+/PnW9JzY86jAmLVcz9KxHmd40QAaQANoAA2ETwP9evWwXubSSy/1odOiJAhAAAIQyDcBApx8E2V7gSEwadIka3rOG3wqAQ5nXNEAGkADaAANoIHAaeCyC8/+9WTUiScGxn9RKAQgAAEI5E6AACd3dqwZcAIjRoywpof5b8J3RpKzzPQpGkADaAANREEDd98yxnqZNm3aBNyVUT4EIAABCGRCgAAnE0osE0oCvXv3tqaH24dj8qNg8mkjOkcDaAANhE8Di+c8aL1MWVlZKL0ajYIABCAAgYoECHAq8uBdhAi0atXKmp6nHrk7cEOmMeHhM+H0KX2KBtAAGkAD2Wrg9+8ssV6mWrVq5osvvoiQi6OpEIAABKJJgAAnmv1Oq42JGZ5XF80kwGHeAzSABtAAGkADaCCQGlB4o8eqVavwdxCAAAQgEHICBDgh72Cal5yAMzwEOJzxzPaMJ8ujGTSABtAAGvCLBpyfIcBJ7vn4BgIQgEBYCBDghKUnaUfWBJzhIcDBhPvFhFMHWkQDaAANoIFsNeD8DAFO1laQFSAAAQgEjgABTuC6jILzRcAZHgIczHK2Zpnl0QwaQANoAA34RQPOzxDg5Mshsh0IQAAC/iVAgOPfvqGyAhNwhocABxPuFxNOHWgRDaABNIAGstWA8zMEOAU2jmweAhCAgA8IEOD4oBMooTQEnOEhwMEsZ2uWWR7NoAE0gAbQgF804PwMAU5p/CR7hQAEIFBMAgQ4xaTNvnxFwBkeAhxMuF9MOHWgRTSABtAAGshWA87PEOD4ymZSDAQgAIGCECDAKQhWNhoEAs7wEOBglrM1yyyPZtAAGkADaMAvGnB+hgAnCO6TGiEAAQhUjQABTtX4sXaACTjDQ4CDCfeLCacOtIgG0AAaQAPZasD5GQKcAJtSSocABCCQIQECnAxBsVj4CDjDQ4CDWc7WLLM8mkEDaAANoAG/aMD5GQKc8HlVWgQBCEAgkQABTiIR3keGgDM8BDiYcL+YcOpAi2gADaABNJCtBpyfIcCJjIWloRCAQIQJEOBEuPOj3nRneAhwMMvZmmWWRzNoAA2gATTgFw04P0OAE3VnS/shAIEoECDAiUIv00ZPAs7wEOBgwv1iwqkjP1r86U8bzV/+8GaFhz6D72ZTTA4jhw020+6+Ee5fZ6brYvYNv4XM+iQonJyfIcDxtHt8CAEIQCBUBAhwQtWdNCYbAs7wEOCEy8gGxXBTZ+F0d8n5Zxn3+3bPm95cHPkgYeb9t5sTjvlN0Tg0arCLOevUPkXbX5B/U999tMbstmtj83H5y/DKMPAKcn/nu3b3/xwBTjYukGUhAAEIBJMAAU4w+42q80DAGR4CnMIdSOfbpBZje68tmmVql5WZGjVqeD6aNG5kunU91Dz58CTfH2iNumioeW72/VnVuX3z66b+zvU8216zZg3Tcp89Tf9ex5g3l87JarvF6Lv4ffzPnzca92jWdFcT9QDnmz++bRSoPPHgHZ791veEoz37XL+DIWec5LlOPG+v1wQ42f3felq/482pfXvmxNqLP59lxz/IvJyfIcDJgzlkExCAAAR8ToAAx+cdRHmFI+AMDwFOdExupgb9mw/eMkvmTbNhxZ+2vmHiH39Ys9SGN61a7m0uHDrA1wdbOiB8cNL1Wdeo0QAaCaDfyCfrX6nQ/ndffcZMvvVa06RRQ3Pb+FFZbzvTPsjncgQ4m82Yyy4w+7fcO+klVP/cvj7Wz+cPPs1ceem5sfc/bn8/p37+/TtLzOcbX81p3Xz2f1C29d5rz5rq1aubt5fNhRmjcLLSgPMzBDiF84xsGQIQgIBfCBDg+KUnqKPoBJzhIcAhwPE6wNNBlEIar+/0meZY2avF7mbp/IeTLpNs3WJ9nmuAo/r+8dm7NsD54Yv3PNv3wdplpnHDBkYjlorVnlz3E/UA57+/LLd9de8d4zLqqxHDBpnrr74ko2Vz7RPW8/5/99juXc3ZA/rBngAnKw04P0OAU3QryQ4hAAEIFJ0AAU7RkbNDvxBwhocAx/tAIuoHWOkCHPG55rLzzdUjz09ptP/+6Trz1rK55uHJN5nH75tg1q1YYHRAnY7vXz98x+hyJj00IiZ+eb1XcKQQKf5z91qfaz1dFnPHDVfGtuO2p2cFNG55r+d0AY7WuXHMCHPRuWck3Y4mZd3wxvNm9kMT7WS2Yvr9596BkFcNWlbraCJcbUPbymWi11wCnB1fbTDrVy40s6fdaR6483qz8oUnjPrEq874z1Sz45zYPxrJ8vriWUYjU+LXKfTruTPuMmVltZLqJXH/VQlwvv1wdaz94qD3idv3eq9Rb1re6UOsX3rmEXPfxPGWfbIg0Wtbf1y33Cx84j4z9fbr7Ei6Lza9llEN2pb288aLs+3v9Z4JY82iOQ8Utd81T1HdOnWMeHi1jc/4e+WlAednCHD84jCpAwIQgEDhCBDgFI4tW/Y5AWd4CHAwxF6GOJMARyNc7rr5mqQHWrMeuN1eanTQAfsZLduvVw+zz157mL333N0enHrtV5/pki3NQ6PLKaTTSy84y+5DgcCg0/rag/GO7Q80CiYuOPt0o7DBbUsBgQ7W3bpaX68TH4e0bRNbx60b/5xJgKP5VHod281zOwo/OrRrYydm1cS5p/Q5zl6S1nS3JmbB4/d4rhO//6cfm2LX3W/fvey8IMf3ONK+V7u17fhl073ONsBZvfwp0+7AVmaP5k1N7+OOMgNP7mUObnuA2bneTkYH9ZpbJ9k+u3RqH2Pdts3+djn1z03XjjS71N/ZMlCbNI/Sl5teT7qdZNvP5fOeRx9hjut+RMb7yjXA+fDdl0y9nerG2i/tndT72LT7VX/WqV3ban3GPTdbfTRsUN+In3TTYJedLTsFK6nar6Dm5BOPtct3P/Iw+1vpelgHs1PdOmb4OQPN3z6uGITGb0sjyqRl1XHA/vvauvVbO/zQg82uTRqZ+Y9OSbnvfPW788Q2PgAAIABJREFUwln9VqfcNjbl/uJr5zV/w5yfIcDxufGkPAhAAAJ5IECAkweIbCKYBJzhIcDB/HodAKULcJ5/8n6jg8xta5d5HmhdPnyIadOqpXll4WMVvv/5L5vMnOmTbPjy0F03VPjOq46bx15mAxyN3FGgoINKd6caBT2a9FQjDrzWLeQlVNrf2QP7m99edXGlfWv+IB10K9xKDDtefGqabUeqS3S0zRa7N6t0eZq2deeNV9ttZ3PpWjYBjoIjXRo2fcrNlUb7rH5pnjn0kLbmzFNOrNTmRP4asaMAQqGbwppOBx9kFAxpOQU6V1xyTkZBVuJ2s33/1ZaVNhDQLb0zXTfXACdx+4/ee2tGAY5bT1pWsKiAKz6k06gYjV6TptQet3z8s7ukTyPiEkdKaR4e3Q1Ll0T+1ydrPdfX6B+Nfvn0/Vcqfa/gSHM+aQRY/D69Xuej3/fco7lR8OS1fT7j75WXBpyfIcAJph+laghAAALZECDAyYYWy4aKgDM8BDgYYi9DrACnebPd7EG2RozoMe+RyXYkhUbS6G5UOljzWveFuQ/ZAOKzDSs8v9c6a1952oY4Lozx2o4+U4CjkSwKi3QZVrLlvD4vZICjkSi6y9DW1S9WqOnrbW9ZbqnufvXRe7+z63qNqNBnClA0ebJXm/TZMzOnmt2b75bxZSaZBjjqL40Q0qU7yfatuzkd2Hq/tOGLtKH9KozQZNeaJDjZNgv5uerQ/3W6nCjT/ZQywFFwl+w3ofDS6zegy+oUeOiSvmRtVHCqECebICt+Wxrplsmk3fno92OOOtyO+onfP6/5O5VKA87PEOCEyqbSGAhAAAKeBAhwPLHwYRQIOMNDgIMx9jLGCnA0F8VRR3SOPX5zeCez794tjG6nrYNz3Y3J6w49Wk4jOLy2G/+ZRgtcduHZKZdTgKPw5nfPPppyufjtutf5CHBefX6mncNH8/hoNNHt119hL8dRuLV8wYxKNekgOpMRKprbpEe3LpXWP/o3h5lMRiad3v+ElAfsjoGeMw1wxl95kRl8et9KNcVvS68V0OmSqsTP49/rQF6XwmjEUPznxX6tuYP0f51GRWW671IGOKn6XgGK153f9NvQqJXE0V6J7dW8MvpNJ85NlLic1/trLx9mR015fRf/WT76XW1Un2n+rPht85q/Vck04PwMAU4U3CtthAAEok6AACfqCohw+53hIcDBFHuZ4lSXUGk0hQINzW2j+V3iDxz1WnOlKNx57L7bUj50a2evECO+HncJVfxnmb7OR4DTer997Jwgmhek/UGt7cTIE2+8KukkyJo4WZeipKtRl7zokhiNjHDL6rXmiUk2AsMtp+dHpt5i5xSK/yzZ60wDHM13o7lS0vWbRoFoXhWv8M7VoAN5NweO+6wUz5rEWv/XbXpzcYxzujoyCXAymUw6l0uoNAdOsvoUHg4bMqDS92qj5rxJ12/6XjpOduc0XZKoAFK/m84d2tmQTtvV6BtdBqfL3pLV5j7PR78rqFKfJY5uc/vgmb9ZiRpwfoYAJ8KmlqZDAAKRIUCAE5mupqGJBJzhIcDBDCeaYb1PFeC45TXRr+a5iR8dozk2NPJCYUCf47unfZxz1skpDwpLHeBkc/cfcRGPVUueTNkmLacAQCOZdNDseGoeEk3AHB/quO8Sn3UQrgAt8XOv95kGOId1bGcnrc2k37SM6vXanz7Lx4F8sm1n87mCLv1fp5FUma6XLsDRyCvNV5Nue8UKcDSSbf+We6f9rbl+9Ro5pgm5Fbye2LO7eXDS9UZzNUljmutKczlp8uxiBTijLhpq+yyXkULp+oTvw/n3zvkZApxEp8d7CEAAAuEjQIATvj6lRRkScIaHACechraqByqZBDjah86Wnzfo1AoHszqY1PpVrUHrBy3A0V2HNNIhXdt1K23NoRO/nIIbXS6muxnFf+71WqNgdMchr+8SP8s0wNFIC43ySFw/l/d+CXBeXvioDQMyuaTPtTNdgPPUI3enHTmmbRUrwNFlYsnuhubalOpZE4RLi6lCrmwuoarqyCsFTQqTUtXMd/zditeA8zMEOBkaQBaDAAQgEGACBDgB7jxKrxoBZ3gIcDDC8UbYvc40wNHtoXXZjVtPz0PPPNnoLlTxn+X6OmgBzi3XXW5v/ZyuvXffMsbo9taJy+l215ogOfHzxPf9ex1jbh03Ku1yWi/TAEfBk+4WFX9JXOJ+M33vlwBHl6rp/7prLjs/I1ZqX9ACHN0JTpe0pRoRlarfxl1xkRl98dCUfIoZ4LjLFVPVzHf83YrXgPMzBDhV84WsDQEIQCAIBAhwgtBL1FgQAs7wEOBghOONsHudSYCjy3802ka3nnbr6fnLTa+b3XZtXOk22PHL6LVGnKSbm6QqAY4mQ730grMq1JZYQ7L3ujxMv5FsL6H67qM1Zu89dzdzZ9yVdL8afaORNmtefrrSMu/8br79Ltnt2VXv7Gl32n1oX8nqj/880wBHwY0moL5u9PC0293y9guVbjMev0+/BDiqSZe1ndLnuLRtcvUHLcBR3WNHXWjnwdEt2l07vJ6/2PSa0Z3E4r9TeDPkjJMqfBb/vS6n0u+5GJdQqX5dRvjbqy5OWk98bbzm75c04PwMAU5B7CIbhQAEIOArAgQ4vuoOiikmAWd4CHAwwIkHQX/98B2z7Onp9hbQui124kOjGjRaQ3Ow6LbEXnO2aO6MXZs0spdY/feX5RUOxnSQNmf6JDvJ7ZFdOlb4TrVoe99//p59XH/1Jeb8wadVqkHfJ9ad+F6329ZddxSKxH+noEK3ytYdl6bcVnm0i+bx+fT9V+xBwecbX43t+9sPV1fYTvw241+veO5xG8JoQtjEA2rdil0Hw5oIOX6d+NealFbLJAZjmjT4hmsutdtOdrmLlknsL21L8/LEf66AKn6f7vUf1y03e7XY3Zw76BTPER3ajuY3atKooflqy8pK21Dgpb4RXwUn8fvU60xDJ1dPPp4n3XS1ZZYqjIvnpkvJNK9MYu3uvSYa9pp8W21zy+hZty4Xq/jP9Drx96L3+lwTYEuPXozEVL8FBS3SZyIXTSyumhTA6ZKoxO91Wd5F555h6u1Ut1Kw+vriWaZ2WZmRbuNHXyns0Ug6hY2aA+eS88+ydcYv4/aTr35fOv9hO4eWdOi2zTN/o9JpwPkZApxiukj2BQEIQKA0BAhwSsOdvfqAgDM8BDiY43hzrIlLNbmu04fXs+6UpLleNMdH/LqJrzWKRLfFrr9zPdOlU3ujy340P0ad2rVNq5Z7GwUVXgfVZw/ol3L/qqnpbk0qhSOJ+9d7hSja3x7Nm9q7NulAVPW02L2ZvaxGt1aOX0+Xoegg16vdNWrUMIvnPFhh+fh141+r7TqY1r4UUh3f40hbs8IRhRvxy3q91kSzWlbt1J2+tA1tS3cDSjU6p1+vHp61J7anccMGNmjx2rcCBB3si1u7A1vZvtYEx+p3hXIK7T7bsKJSGzTPTOJ+Et9rdMXaVyqPPPKqI1+faTJcBRTzH604Uix++wpPEmtN9T4xwFFooralWsd9J03G71thnvtOz9pOfAijcEnc3TK6XOqT9a9U2Ia2p2BFoZH6ad+9W9gJiXUXKWlImtbEy6tfmldpPa37wJ3XW30p7FPopNuSS++DTutrR9TpEiq3f00yHF9/Pvtdoeqx3btW2H78vnjN3ysvDThtEuD4wFxSAgQgAIECEyDAKTBgNu9fAs7wEOBgiL0McT4/00gN3anq8fsm2NuPe40wyOf+ErelUEbBiwInBVTxd35KXDbf7zWSZ9GcB+yIo81vpb7sKHHfulOV1tFoJY1o0rYSlynk+799vMb2l/pN/ZfrHCuFrDHTbSuIUJiY6fJBXk6Bz/qVC82TD08yz82+32jEnNeomcQ2KuhaMm+aHV331rK55u+frisqL116qRFzmiQ6sTbe83cqlQacnyHA8a/npDIIQAAC+SJAgJMvkmwncASc4SHAwRinMsZ8hz7CoAHNO1SrVk2jy4XC0J4wtkFz7HRo1yajsCmM7adNuf9f6/wMAU7grCgFQwACEMiaAAFO1shYISwEnOEhwMndNGK4YYcGgqMBzWujS8zoM//1mUZ76RJBAjb/9U0Qfi/OzxDghMWh0g4IQAACyQkQ4CRnwzchJ+AMDwEOhjkIBp0a0WlVNaCJfnVpUVW3w/qF0WL8vD8wLgzjsHJ1foYAJ+TGleZBAAIQMMYQ4CCDyBJwhocAB6McVlNPu9A2GkADaCD8GnB+hgAnspaWhkMAAhEiQIAToc6mqRUJOMNDgBN+c8sBDH2MBtAAGkADYdWA8zMEOBV9Hu8gAAEIhJEAAU4Ye5U2ZUTAGR4CHEx9WE097ULbaAANoIHwa8D5GQKcjOwfC0EAAhAINAECnEB3H8VXhYAzPAQ44Te3HMDQx2gADaABNBBWDTg/Q4BTFVfIuhCAAASCQYAAJxj9RJUFIOAMDwEOpj6spp52oW00gAbQQPg14PwMAU4BzCKbhAAEIOAzAgQ4PusQyikeAWd4CHDCb245gKGP0QAaQANoIKwacH6GAKd4HpI9QQACECgVAQKcUpFnvyUn4AwPAQ6mPqymnnahbTSABtBA+DXg/AwBTsmtJQVAAAIQKDgBApyCI2YHfiXgDA8BTvjNLQcw9DEaQANoAA2EVQPOzxDg+NVxUhcEIACB/BEgwMkfS7YUMALO8BDgYOrDauppF9pGA2gADYRfA87PEOAEzIhSLgQgAIEcCBDg5ACNVcJBwBkeApzwm1sOYOhjNIAG0AAaCKsGnJ8hwAmHP6UVEIAABFIRIMBJRYfvQk3AGR4CHEx9WE097ULbaAANoIHwa8D5GQKcUNtWGgcBCEDAEiDAQQiRJdCqVSsj0zPn4UkGgxt+g0sf08doAA2gATQQNg38/p0l1svIz3zxxReR9XQ0HAIQgEBUCBDgRKWnaWclAr1797am565briHA+RpTHzZTT3vQNBpAA2gg/BpYPOdB62XKysoq+Rw+gAAEIACB8BEgwAlfn9KiDAmMGDHCmp6rRpxHgEOAgwbQABpAA2gADQROAzoJpdE3bdq0ydD9sBgEIAABCASZAAFOkHuP2qtEYNKkSdb0DDq9b+AMG2dVw39WlT6mj9EAGkADaCCdBoafM9B6mRNPPLFKnoiVIQABCEAgGAQIcILRT1RZAALz58+3pqdHty4EOJx1RQNoAA2gATSABgKnAXkYjcC59NJLC+CU2CQEIAABCPiNAAGO33qEeopGoLy83JoeGZ8tb70QONOW7qwc33PmFg2gATSABtBAeDXwwdplMR8zY8aMovkndgQBCEAAAqUjQIBTOvbs2QcE+vXrZ80P8+CE1+By8ELfogE0gAbQQBg1MHb0hdbDtG7d2vz0008+cFWUAAEIQAAChSZAgFNowmzf1wRmz55tzU/9+vXMNx+8zSgchs+jATSABtAAGkADgdBAuwNbWQ8zYcIEX3stioMABCAAgfwRIMDJH0u2FEACO3bsMC1btrQGaOrt1wXCsIXxLCJt4uw4GkADaAANoIHMNbDs6enWuzRq1NB8+eWXAXRglAwBCEAAArkQIMDJhRrrhIrA2LFjrQlqf1Br8+G7LxHicOYVDaABNIAG0AAa8LUGhpxxkvUuo0ePDpUnozEQgAAEIJCaAAFOaj58GwECmzdvNk2bNrVG6LjuXX1t2Dg7mfnZSVjBCg2gATSABsKogUk3XW09yz777GO2bt0aAadGEyEAAQhAwBEgwHEkeI40gZkzZ1ozpDtSnXVaH0IczryiATSABtAAGkADvtPA8gUzYn5l+fLlkfZuNB4CEIBAFAkQ4ESx12mzJ4ERI0bETNG9t4/znWkL41lE2sTZcTSABtAAGkADmWlg+5aVMZ8yffp0Ty/DhxCAAAQgEG4CBDjh7l9alyWBLl26xMzRgsfvIcTh7CsaQANoAA2gATRQcg2Uv/ZszJ8MGDAgS3fD4hCAAAQgEBYCBDhh6UnakTcCuozKPcZcPqzkpo0zk5mdmYQTnNAAGkADaCCMGrjiknNivkT+hH8QgAAEIBBdAvwViG7f0/IUBHR2y4U4xx7V1axbsYAghzOwaAANoAE0gAbQQNE08On7K8zJfY6L+RGNEuYfBCAAAQhEmwABTrT7n9anIKDry12Is0v9nc11Vww3W956oWjGLYxnEWkTZ8fRABpAA2gADaTWwDcfvG2m3n6dObjtATEfonn6+AcBCEAAAhAgwEEDEEhBQHd40G06XZBTu6zMXHD2aebVRTMJcjgLiwbQABpAA2gADeRNAzpJpJNFe7XYPeY7mjZtanSnTP5BAAIQgAAERIAABx1AIA2BrVu3mtGjR5tGjRrFDJUCnZ5HH2EuHDrA3DZ+lJn3yGSjCQa//+zdvBk5zlCmPkMJH/igATSABtBAUDWwbe0ys+zp6eb+O39rNMdN/97HGJ0kcieMGjVqaL3H5s2b07gUvoYABCAAgSgRIMCJUm/T1ioR+Mc//mEmTpxoWrduHTNYzmjx/J+Jn2EBCzSABtAAGkADuWlAHmPChAnmyy+/rJJnYWUIQAACEAgnAQKccPYrrSowAV1ade2115rzzjvP9OnTx3Tu3Nnstddepnbt2oQ7cXfxwsDnZuDhBjc0gAbQQLg10Lx5c9OhQwfTu3dvc84555irrrrKzJgxw/z0008FdjBsHgIQgAAEgkyAACfIvUftEIAABCDgawK33XabufTSS31dI8VBAAIQgAAEIAABCASDAAFOMPqJKiEAAQhAIIAE2rVrZxo2bGh++eWXAFZPyRCAAAQgAAEIQAACfiJAgOOn3qAWCEAAAhAIDYHVq1fHLqlcsGBBaNpFQyAAAQhAAAIQgAAESkOAAKc03NkrBCAAAQiEnMCoUaNiAc6pp54a8tbSPAhAAAIQgAAEIACBQhMgwCk0YbYPAQhAAAKRJNCsWbNYgKMJab/55ptIcqDREIAABCAAAQhAAAL5IUCAkx+ObAUCEIAABCAQI7B48eIK4Y0CnPvuuy/2PS8gAAEIQAACEIAABCCQLQECnGyJsTwEIAABCEAgDYHBgwdXCnCOPPLINGvxNQQgAAEIQAACEIAABJITIMBJzoZvIAABCEAAAlkT+OGHH0z16tUrBTgahbN58+ast8cKEIAABCAAAQhAAAIQEAECHHQAAQhAAAIQyCOBxx57zDO8UYAzfvz4PO6JTUEAAhCAAAQgAAEIRIkAAU6Uepu2QgACEIBAwQkcf/zxSQOc/fbbr+D7ZwcQgAAEIAABCEAAAuEkQIATzn6lVRCAAAQgUAICn376adLwRiNw9HjllVdKUBm7hAAEIAABCEAAAhAIOgECnKD3IPVDAAIQgIBvCEycODFtgDNs2DDf1EshEIAABCAAAQhAAALBIUCAE5y+olIIQAACEPA5gQ4dOqQNcHbaaSfzf//3fz5vCeVBAAIQgAAEIAABCPiNAAGO33qEeiAAAQhAIJAE1q1blza8cZdRzZkzJ5BtpGgIQAACEIAABCAAgdIRIMApHXv2DAEIQAACISJw1VVXZRzg9O3bN0QtpykQgAAEIAABCEAAAsUgQIBTDMrsAwIQgAAEQk9gjz32yDjA0Uic7du3h54JDYQABCAAAQhAAAIQyB8BApz8sWRLEIAABCAQUQJLly7NKrxRgDN58uSI0qLZEIAABCAAAQhAAAK5ECDAyYUa60AAAhCAAATiCAwdOjTrAKdz585xW+AlBCAAAQhAAAIQgAAEUhMgwEnNh28hAAEIQAACKQns2LHD1KpVK+sAR6NwysvLU26bLyEAAQhAAAIQgAAEIOAIEOA4EjxDAAIQgAAEciAwa9asnMIbBThjxozJYY+sAgEIQAACEIAABCAQRQIEOFHsddoMAQhAAAJ5I7B48WKjxwsvvBB7vPjii/a1Qho9HnjgAaN5cvRYtmxZ7LF8+fK81cGGIAABCEAAAhCAAATCTYAAJ9z9S+sgAAEIQKBEBH755ZfYyJwtW7aUqAp2CwEIQAACEIAABCAQFgIEOGHpSdoBAQhAAAK+IkCA46vuoBgIQAACEIAABCAQeAIEOIHvQhoAAQhAAAJ+JeAuodq8ebNfS6QuCEAAAhCAAAQgAIGAECDACUhHUSYEIAABCASPAAFO8PqMiiEAAQhAAAIQgIBfCRDg+LVnqAsCEIAABAJPwAU4mzZtCnxbaAAEIAABCEAAAhCAQGkJEOCUlj97hwAEIACBEBOoXr26nciYACfEnUzTIAABCEAAAhCAQJEIEOAUCTS7gQAEIACB6BEgwIlen9NiCEAAAhCAAAQgUCgCBDiFIst2IQABCEAg8gRq1KhhR+Bs3Lgx8iwAAAEIQAACEIAABCBQNQIEOFXjx9oFJPDBBx+Y66+/3lxyySXm9NNPN127djV77rmnPRgq4G7ZNAQgAIG8ESDAyRtKNgQBCEAAAhCAAAQiT4AAJ/IS8B+AF1980fTt29cGNW4C0MRn/1VNRRCAAAQqE6hZs6b9v2zDhg2Vv+QTCEAAAhCAAAQgAAEIZEGAACcLWCxaOAI//vijufvuu02nTp0qBDddunQx55xzjrnpppvMrFmzzBtvvGE+//zzwhXCliEAAQjkkQABTh5hsikIQAACEIAABCAQcQIEOBEXgB+av2bNGnPEEUdUCG769+9vnnnmGT+URw0QgAAEcibgApz3338/522wIgQgAAEIQAACEIAABESAAAcdlJTAjBkzjDvA0WVSw4cPN6tWrSppTewcAhCAQL4I1KpVy4bTBDj5Isp2IAABCEAAAhCAQHQJEOBEt+9L3vIRI0bERt0oxFGYwz8IQAACYSJAgBOm3qQtEIAABCAAAQhAoLQECHBKyz+yez/ppJNi4U2rVq3Mpk2bIsuChkMAAuElUFZWZv+vW79+fXgbScsgAAEIQAACEIAABIpCgACnKJjZSTyB5557LhbenHHGGfFf8RoCEIBAqAgQ4ISqO2kMBCAAAQhAAAIQKCkBApyS4o/ezj/44INYeHPllVdGDwAthgAEIkWgdu3a9v+88vLySLWbxkIAAhCAAAQgAAEI5J8AAU7+mbLFJAT+9a9/me7du9uDmZNPPjnJUnwMAQhAIDwECHDC05e0BAIQgAAEIAABCJSaAAFOqXsgQvsfNWqUDW/at29vfvzxxwi1nKZCAAJRJeACnPfeey+qCGg3BCAAAQhAAAIQgECeCBDg5Akkm0lNYMWKFTa82WmnncyaNWtSLxyQb7/55huzbt06s2DBAjNp0iSju2r16dPHHHvssTxggAbQgNVAjRo17P99nTt3RhNoAg2ggZgGhg8fbiZOnGjmz59vVq9ebbZv3x4Q90OZEIAABCBQSgIEOKWkH6F9Dx061B7EjB07NtCt/vnnn82sWbPMCSecYNtTrVo1nmGABtAAGkADaAANVFkDHTt2NFOnTjXfffddoL0SxUMAAhCAQOEIEOAUji1b/jeBjRs3xkzNhg0bAsnlo48+Mrfeeqtp0+aAWFtceLNL/Z3NIe3amFP6HGfGXHaBuXHMCB4wQANoAA2gATSABpJqYMgZJ5kju3Q0zZvuWslXtGjRwowfP95s27YtkJ6JoiEAAQhAoHAECHAKx5Yt/5vAVVddZc1Jv379AslEJqpunToxg9Vgl/rm/MGnmRefmmY+WLvM/O/Xm3nAAA2gATSABtAAGshJA9u3rDRvLHnSjLl8mGnVcu+Y3ygrKzPXXHNNIL0TRUMAAhCAQGEIEOAUhitb/TeBr776yjRp0sSakSVLlgSKyw8//GCGDBkSM1JHH3mYmXr7OPP5xldzMmgEPQRdaAANoAE0gAbQQCoN/HP7ejNn+iRzev//XKrdo0cP8+233wbKQ1EsBCAAAQgUhgABTmG4stV/E3juuedsANKqVatAMdm6datp1KhRLLx56K4bCG04s4oG0AAaQANoAA0UTQOLnnzANNvt15Ng9erVM7ohBP8gAAEIQCDaBAhwot3/BW/95MmTbQgycODAgu8rXzuYNm1aLLipW7eOeXPpnKKZtVRn5fiOs7ZoAA2gATSABqKlgW1rlpr2B7WO+ZIpU6bky+6wHQhAAAIQCCABApwAdlqQSr7sssus6Rg1alQgyp47d27MJLVt08r8/dN1hDecbUUDaAANoAE0gAZKqoETex4V8ycrV64MhKeiSAhAAAIQyD8BApz8M2WLcQT69u1rDcedd94Z96k/X27ZsiVmjlrtt09JjRpnWKN1hpX+pr/RABpAA2ggnQauvXxYzKds377dn2aKqiAAAQhAoKAECHAKipeNt23b1pqN2bNn+x5G69a/DlHeud5OhDecaUUDaAANoAE0gAZ8p4GBJ/eyvqpbt26+91UUCAEIQAAC+SdAgJN/pmwxjkDdunWt0fD7xHuDBw+OndVaMu9h3xm2dGfl+J4zt2gADaABNIAGoqGBzh3aWc8ycuTIOMfFSwhAAAIQiAIBApwo9HIJ21itWjVrMlatWlXCKlLvWqODXJ33TLiO8IYzrmgADaABNIAG0IBvNbD+9YWmccMG1rvMnDkztcnhWwhAAAIQCBUBApxQdaf/GuOCET8HOP369bMm6IKzT/OtWeOsajTOqtLP9DMaQANoAA1kooH77/yt9S49evTwn/mjIghAAAIQKBgBApyCoWXDIuD3AKe8vDxW46uLZhLgcMYVDaABNIAG0AAa8L0Gtm9ZaRo3amg9zJIlSzCdEIAABCAQEQIEOBHp6FI10+8BztixY6356XvC0b43a5mckWMZztyiATSABtAAGoiGBi4cOsB6mIEDB5bK5rFfCEAAAhAoMgECnCIDj9ru/Bzg7Nixw7Rs2dKan6ceuZsAhzOuaAANoAE0gAbQQGA0sOzp6dbDyGutXbs2ahaT9kIAAhCIJAECnEh2e/Ea7ecAZ/ny5db4HNaxfWDMGmdVo3FWlX6Vv0zEAAAgAElEQVSmn9EAGkADaCATDXTpdLD1MuPGjSueuWNPEIAABCBQMgIEOCVDH40d+znAmT791zNXA0/uRYDDGVc0gAbQABpAA2ggcBoYMrC/DXAGDRoUDWNJKyEAAQhEnAABTsQFUOjm+znAGT9+vDU9lw0/O3CGLZOzcizD2Vs0gAbQABpAA+HWwG+vvsR6ma5duxba0rF9CEAAAhDwAQECHB90QphL8HOAo7NVqm/C+NEEOJx1RQNoAA2gATSABgKngcfvn2C9TLNmzcJsJ2kbBCAAAQj8mwABDlIoKAE/BzhdunSxpueRqbcGzrBxRjXcZ1TpX/oXDaABNIAGMtHAm0vnWC8jv/X9998X1NOxcQhAAAIQKD0BApzS90GoK/BzgNO4cWNrel6Y+xABDmdd0QAaQANoAA2ggcBp4C/b3owFOJs2bQq1p6RxEIAABCBgDAEOKigoAT8HOK62VxfNDJxhy+SsHMtw9hYNoAE0gAbQQPg14PzMqlWrCurp2DgEIAABCJSeAAFO6fsg1BX42VS42ghwwm9uOYChj9EAGkADaCCsGnB+hgAn1JaaxkEAAhCwBAhwEEJBCfjZVLjaCHAw9WE19bQLbaMBNIAGwq8B52cIcApqadk4BCAAAV8QIMDxRTeEtwg/mwpXGwFO+M0tBzD0MRpAA2gADYRVA87PEOCE10/TMghAAAKOAAGOI8FzQQj42VS42ghwMPVhNfW0C22jATSABsKvAednCHAKYmXZKAQgAAFfESDA8VV3hK8YP5sKVxsBTvjNLQcw9DEaQANoAA2EVQPOzxDghM9H0yIIQAACiQQIcBKJ8D6vBPxsKlxtBDiY+rCaetqFttEAGkAD4deA8zMEOHm1sGwMAhCAgC8JEOD4slvCU5SfTYWrjQAn/OaWAxj6GA2gATSABsKqAednCHDC459pCQQgAIFkBAhwkpHh87wQ8LOpcLUR4GDqw2rqaRfaRgNoAA2EXwPOzxDg5MW6shEIQAACviZAgOPr7gl+cX42Fa42Apzwm1sOYOhjNIAG0AAaCKsGnJ8hwAm+b6YFEIAABNIRIMBJR4jvq0TAz6bC1UaAg6kPq6mnXWgbDaABNBB+DTg/Q4BTJcvKyhCAAAQCQYAAJxDdFNwi/WwqXG0EOOE3txzA5LePf/7LJvM/f95o4Jpfrql4/vSnjeYfn70L86+LxzxVf/BdZv3w4/b3zV/+8KZ9/PeX5b7Ur35bQe9P52cIcILrl6kcAhCAQKYECHAyJcVyORHws6lwtRHgZGbEg25wqT9//XzuoFPMbeNHBf6gJwia+N2zj5oundqb2mVlRv9n1duprmm5z57muO5HmI/LX6YPCHR8rYE+x3e3upV2p0+52Ze1Ht75EPPc7Pt9WVum/0c5P0OAk5NVZSUIQAACgSJAgBOo7gpesX42Fa42vwY4x/c40tSoUcPzUVZWy7Rp1dJccPbp5otNr/naeKq+Iw7rkFWNby2baw9Yvdq/U906ZtObi7PaXqYmmOXShzwrnnvc1KpV03z47kuefdCs6a6emvXqS332x3XLPbdDX2w2yxfMMLvU39mMv/Iis/aVp81/fbLW8npz6RyzaM4D5ocv3oNdiQOc26+/Iiu9T7v7xsj1mUbr9e91jG8DnHvvGGda7N4s0CPcnJ8hwAmeT6ZiCEAAAtkSIMDJlhjLZ0XAz6bC1ebXAEfDzf+09Q1zzFGHmwcnXW9f670e2ze/bt5eNtdcfN6ZpnHDBmbzWy/49qDg/ZXPmaa7Ncm6vm/++LZt65a3X7BncNVmtf3bD1dnvS3CgPTBTKaMDj/0YKMROMmW323XxmbVkicr6PXWcaPMKX2Oq/CZ+rLBLjubD9YuS7qtZPuIyudHHdHZjLnsAviUOKRJpTddTvjn368yN1xzqRl0Wt9KGnf/Z+tZIcZDd90Qyf48qfexvg1wFIQ2b7abueOGKwPbN87PEOBkZVFZGAIQgEAgCRDgBLLbglO0n02Fq82vAY47aOh1bDcze9qdSY2lTKeGgLvl/faca4Dj2qH5E9RXzLmSvxDGsc32+Z3fzbd9oT5Ntq4CHIVu8d/fM2GsOevUPhU+0/cNG9QnwEkRTuzapJFZv3JhJW7xbHld+t+F+uDOG6825w06NWVfDTipFwFOCr2XUss3j73M7L3n7oH9O+P8DAFOcPwxlUIAAhDIlQABTq7kWC8jAn42Fa62oAc4uqyiZs0a5m8fr0l58KBLVRY+cZ+Zevt1Zsm8aRldeqXJHTXyxT0SQxQdyGu0hc5CJ5pvndXUerrkRgeibhvxzzornbhe4vuqBjiqWXXOeuB2c/ctY8y8RyannTtkx1cbKtSriTgT6/rn9vWxZb7astKTgdb5+6frjC4Je3jyTebx+yaYdSsWmEwm8/zrh+/Etv/dRxX7Vu+Xzn/YTgyaWFch31907hmmY/sDK7GI32c+Apx8tD2Xfnd9qhEVapN0rcv11G8KUbOZc0Z9rN/ZA3deb+6f+FvbX64fvfTkGGqiYvcbadRgF7PyhSdi793nmVw69fW2t4zmz9HlIfrdZ1J7rr93/Y5Vm9r1zQdvmScfnmSemTnVat+1a83LT5tH773Vfu8+S/Ys/i8984it/enHppitq19M+vuK34bWS8bo842v2kvSMmEXv81MX2cS4Jx5yomeo1DETHV///mvl8RJ/2r/fRPH2/7PpmZtY/Xyp+x+Ztxzs9Hldm7y61S6c+2UTqQX6Ub6kY7cd5k8K3DUiNHXFs2qcIlfJiNwiv03Kr49uiS0evXq5sWnpmXV3vhtlPK18zMEOBlZUxaCAAQgEGgCBDiB7j7/F+9nU+FqC3qAo3BCB3oKHbwMpOagOfnEY+3lKt2PPMwO8+96WAejuWSGnzMwZfBzzWXnW1MrYyte7sB2weP3mL1a7G4v3zr0kLZm371bmDdenF1h/ycc85sK62obiQ8FTzpA86rbfZZrgKOD8aFnnmxr1PB41XP2wP6m59FH2HlFrhs9POl+77r5Glur2qyah5xxUqVlzzild2wZzeWiUSauZves0KhJo4bmoAP2M6f1O97069XD7LPXHvZMrw7Q3HKJzzogrr9zvdj2L73gLLusWOgyDc2BpCBF881oHqRkfZ+43aq810GgLnm65brLk9at7etyOR1wx+8r2QgcXf6XOAdOVdtelX7XKCH1t+bD0CV8fU842rZZo+A6d2hnfwPq93S81e8KLdu22d/2+4k9u9v+VHsHn97XbjeeT/zrg9seYGtQHU5/ib8bXVoVv078a9V245gR9vd9SNs2RqGB5qDaud5ORrVLQ/HLx7/O5feuQFS/Y9Wq+WCkSe1Pc3Rp/wqtzjnrZPs76H3cUfa3p2Ahfr/utYKKsaMutLUf1rGdHbV1bPeudl21eVuKy+00J5DmZnLc5j/66/8rChT0/12d2rXtSEXpU4Ga22e+nr0CHAUxCmfcPhTeKux17/Ws+lSb+ImL/m/VyDRpR5cd6jeneZAS/3+N34Z7rXX1f12rlnubU/v2tI92B7aynz3x4B2m/UGtzR/WLK2wf7eudDHw5F5WJ0d26Wh1Iy3q78RN145Mq3mFNvrdqG7pTKNC9XdJ/88qCE0V4JTqb5Rru3vWb1x/K937ID1LP3oQ4PjfF1MhBCAAgaoSIMCpKkHWT0nAz6bC1RbkAEejbhRM6CDey2xqfhEdNF498nyjg4n4ZXRGWgesMvsaxRP/nddrhRRfbnrdjBg2yGiUhQ7c3Bndmfffbo2613oKmHKZA8dtK9cARyMwHpl6iz1AShwhpLbrQCPVXVEWz3nQ7N9y75QHLmq/AhmdqXb1uufLhw+xB7GvLHyswneqZc70SfZAN5P5MDS0XwGODv72aN7UBjhuNIXCDh2oJYYgroZ8PuuAWL8ZHSSm2q604HThlksW4Dx2320m1S18c2l7VftdB9Q6YNadnjT/THx9v39niZ2TSr8n17bEZx1oK9CQfuK/U7Ci7xTo/ebwThW+i18u/rWWzWZ+K9V69G8OswGKao3fluaO0txFOsCX/uO/83qd7e+99X77xEIIbU/9oNBF/1coAHOjSzQxs8KkxFFoCt4U2ijsSgxqtK60oDBDvwOveuM/U+gjvSpI0740B5P7/08jgTQXTfzy+XjtFeAo+Lj28mEZ7Uu/YwVe++27V4XL5hRqaQSY2p4Y/sTXrdBOQbpG38R/rtfSnS4P0u9Xk2Enfv/ZhhVWF7oEzHFyy+hySAVgPbp1qfBbcN/rWf+PHrD/vpX2rb8X6nsF4OpXr/9v/fA3yrVFAb9CQI2adJ8F5dn5GQKclJaULyEAAQiEggABTii60b+N8LOpcLUFIcBRGKADZ/eQEdblLDrI1IgMHfwkGk0dzMl4y9gnfufeK0xQiDNy2OCky7hldUCnM7M6s6rh/u7zdM+lCnDS1TX7oYlGd/pKtZzOyCqQSLaMAiKvO2y9MPchezZaB0b/n73zAJuiOt93YuhgQUSjCKggKCKKgFgQG4iCIoLYBRUxEESi2EWDFQUFFSwYsCOKWLFFg9iwaxCxYAkxMYr1r4nY9fyvZ/yddXe+2T67e2bm/q5rr9mddt5zn2f3e+eZc85kO1YXUuqtYM2YbPvpwrVLpw7eXXkNw8q2X6XXKw59Z2RwFFtWNgMn33kqUfd87a76qZ6n/WlkYD1ljKingd8UtHVRLxTNdWI/+5dqw3y6s8cUa+Do0e565HiuHkJ/POpg76LalpFtWez3XQaO/7dGPV3EKn3icXFTvV56dH4GI/X+Ue84v/mXHp96eWzdebOc9dP+MnD0vdT3S5O9p5+jUu9l4KhcDT+yr/0G9DHHHn1oQeXLwBGrbL8H+p3P9v3Xb4mY5vpd1lBA6TrIwNFNgOOOOSxrnGoT9bQ857SxdfaRISeDXmZyEFu1t0xF9YzyGziu/I+ycctwE6OXF90RWBe7n4tLxa0XBo67+TCRQQACEAiLAAZOWCQ5TyABl5MKG1sUDBwNR9DdbPtSMq07sqqDunwHXVSrV0ir9dfLOymjuvg3atgw59AKJay6oNOFqb2TXmgS66qBozlkdDGYqx4alqEeSum9MOz+ukjW3XKdx66zS/Ww8F+s2G3pS/Xk0AV1+jr/e5kYGlIR1MvHv28lP6uXl/Rm59MopqxyDJyw656v3fVdUi+VXPXTEDYN+wjaRz0gfr/eOt68S0HbpaVcw5jSjynGwFGPFg3FyddjR99fDX/UHCXpZfnfF/t9l4HjN0s0b5BMGf+59bv03CO3pdarl4l6PmQb3pN+vHrpaH6d9HX+9zJSNHQomxni3z+MzzJwZBjZ32gtxbkYAydXjzyZc0cdOjiw3jJ/ZBzmq4eeWmiHwdp9H54/y+u5ozaw64KWMtykL3/PKfXa0dxGQcfYdepRpd8O/2+iK/+jbJyau0lxyoC366KyVNx6YeAEpqKshAAEIBArAhg4sWpO9yrjclJhY4uCgZPtKVRKxtVTQMME/HOq6OlUmvNGw1TyvdT9vZALOn/yX0hyWw0DJ8hgUWy62NBFkeYdkamiYVO6M6+72XrlM3B0DvXC0VAMf121Tr0d/Os1dETtMXniiXm5a4iOhib4z5H+WQaOnQMnfX2132v+Eg2HKaXccgycUupeTrvLwFFPiFz11IVsrmFIGvKiOYzUc+rgwf29nin6Dhf7yPRiDBwNDdIQmlxx220aKqJ5nuznoKUMnGK+7+UYODK9ZH7k+53Sds2l4u/p44/fDqHyr6/k56AhVJr/pRgDJ9vcQIpbBs2RhwQbOOqJ+ezDt+Zsz2x1n3Tm8d7Qumzb09fLePP34NHQrLdfeChv2dKm38Bx5X+UraMmfFZeMOOiCXnrY49xZWnzGQwc9/JgIoIABCAQNgEMnLCJcr4MAi4nFTa2KBs4NnnUUB7/vBrq3aE5XPbcvVdBL3WFt+cLWhZ7QWfPUQ0DR0O7nv9b5tAeeyGuO/G6SNFdez3RR08G0lAEXQgWYuCoF44uTtNNIpk0Mr20zdbTLjWfkIYLyPAohL0meLXHBi1dMXDs/BC5hrgExa911TRwym13e3y2umh9PgNH+6inkuY/0rCfsSMP9XqiaEJYzQeiCZJznd9uK8bAufGqCwN7u9hzpS9lgBwz/JeJsdPXp78v9vtejoGjHg+a8LaQ74v2yWc+uWLgPDhvZsFPNVIvmlIMHH0f9Xvjn7smvS1zvddQXM0RlGsfu01ms+Y+s5+lY/Wcyjac0O6npdrEb+C48j/Kxqmhb8oL8k2sb/d3aWnzGQycjBSUDxCAAARiSQADJ5bN6k6lXE4qbGxxMHDU/V1DqtKHDGiuD00gGVaSWewFnS23GgZOx/YbZ/Qg0sWzhi/IuMl2cZFvKI2NX0sNWdNTXOw6TUKcy/yRceYfTmKPLXbpioGjnl76zhQyzMVfx2oZOGG0e1gGjp+BPmsuGM1npclqg7b71xVj4KgHhnpD+M8R9Fm9z6acXfepaen7Fvt9L8fA0VxRmjg6aC6v9JgKfe+KgeOPV+aufkv86/W5VANHx/qHpAWdP9s69RTU0/qybU9fr6GB6UPftE29f5Y/90De46VNv4Hjyv8oW0f1otJvnHqz2XVRWdp8BgPHnfyXSCAAAQhUigAGTqXIcl6PgMtJhY0tLgZOk8aNvKdE2YRT8w7ojn+uiS3tvoUsi72gs+eshYGj4WR6zLaNIWhZjIEz//pLvR436oUjQ0hDsW6bPTXr+XVBpAv1oHKLXeeKgaNeXvrO3H3TjKLrVS0DJ4x2L8fAkT5OP/6YvHxkdhQy5KUYA0fmh34D/D3R/HqTiaCnmS265/qccRb7fS/HwFGMeppbru+Uvx65Prtq4GjeJPXeCoq9HANHj7uXERN03nzr9IQqGUD5nrykp39JX/4eeBrWlWvuHpVvJ1D2Gziu/I+yjEYcNsT7jfvgtV8f/W63ub60+QwGDsk3BCAAgfgTwMCJfxvXtIYuJxU2tjgYOBf9ebw3aac/yTzp2KO8eXByPZVGx+jCIt+wjmIv6GwsOneuSV/tftmWhTxG3N8D5/brLvXuDKcPe0o//9In7/buhOfqRZO+v97rzrnuGKt7fefNN83as0f76vG5mi8maILj9PPKDNLFTfo6/3tXDBz1ktDjsWXG+GPM97laBk4Y7V6OgSOzVL8ruSYSltGiSZLDNnDUBupVo949/olm09tHF9wD99w1bxsW+30v18DRd0Xz4OR6cpvqoV5Wbz57f874k2bgyITRELR8c9FoDqGgJw5qInwNpUrXSfp79fDcYrP2gb22NAlyi7XXyjqpt36D9eRC6clv4KgMF/5H2bpqiNgO23bNysHu5+LS5jMYODVNeSkcAhCAQFUIYOBUBXNyC3E5qbCxuWrg6EJPdwJ1MaJJFfU+/SVj46+3/8WbJFMXhP7H8irJ1DmUlGp+HN1B9SeeuhBS4q47q0Fmg47XU2v0UgIu4yM9Bj3Byn/OoM8yPIYfNKjOPA0yOtRtXXPY+J9upTkdVJYe26y20oSq6WWnv9fFY/okzIpLBorOnX7ej5cv9lhquIYSdcWl8+S64LX1US8cPQ1MF8jp80DY7f6l9teFjZ4e4z+/DDUNw1JPHtXdf6yMHctdPTrELr2+ep9eL//xlfqsO/2FDstTnW3MYqAeBvZzrrv95dS93HaXOfC3O2d7T5FSrEHGp8pQLwp93/wGofQsrWpYSVAPl3defMgM6Luz953WPEpB7fTJW4tTnJqvuYZ5fMGNqc+WnxgFHav1miNGkyf7e+Lo+6PHWmsy2WyPfC71+y7z1z6RLb2HhiZu1iOk/fOzqMeHLvz9fDXpr3rizJk5pU79dA49xlrfqVOOO7rOdp3LfmdUpp6MZHnZpb+8IIbFrhNznV/zCmmydFtW0NI+ySm9DHu8vlsy4PQblb5d71Uv/Q5o/iT1oPJv12dNiK5ho0E95FSu/g/I5NG8PP7jpQdxH7JP3zpPSNOQKf3m9e/bO6tprac3SVf6f5R+bg2t0u+s5luy9fP/z3Dlf5SGN+p7rQnI0+sQlfc2n8HASW6+Tc0hAIHkEMDASU5b16SmLicVNjZXDRw9ftfGmG2pCXqPO+Ywo67o2RJNXShecu7J3hwTSrL77dbL65XTcp21PeNGF9ZP3n9zneOV9Ovx4tnKtuunnndKnWP9sciE0UWAJrzU43VlSmlOhAb163sTr/ovOnTRqn1tGYUs0w0cla8Ji1VfDSMTS5klMqF0N1hPUpFhZc+rC15/zEGf1Qtns003zvtodnus2kUXk3oikZ5YpYt3xaELBT2eXE9hCXp8r55cZGPLtlT7VeKC1MYetJQppV44hQzLU32zxS4DTRdMQWWUW/dS213tIMMkPWYZHukxaphK+vZRRx6YsV3GlMxETZKtJ1mt3Xwt77umdpd2pEU9jj1b3XXBq33Sywh6r7md0uNKf6/vu4wAmbIyVXThrO+eesGp940MovT97ftSv+8yWzSJro1Tk4Pbc8rA0XqVnf4bIwNH6/cfuEdqX3uMhsHpeyt967ujycD1Wb8V23brYm66+qI6x6jOMi9sDNmWg/rvXudYW26pSz0dKlt5Qev9Q6jOn/CnjOPFKt1slyEm08qeS/rI1tNGbGRca66a3Xtv52lPmtb8aCeOOTLn91bGkbQpztKLdCP9SEfSUzbD0XLTU/mkfRlBOlY9dqR/O0GydKE6SCv+IVcu/I9S7yD9LuUyl21dXVxafWDg1CTVpVAIQAACVSWAgVNV3MkrzOWkwsbmqoETdpKoCwEND9HdUg010SON8yXlYcagO80yZjQZsIYhvbzojtAmLc0Wp3qB6E6/7qrqwlC9lrLtW8h6PVJc/ArZN32ff72y0Dxw20wvDjEIusuevr+r79XjRBfK+R7jXOv4w273UuqjGDRMSlqX/tTzoRDjq5Sygo5Rr42nHphjZl16rvc0LH/Ph6BjXFmn34o3nrnP6KlA+r7pd0s9NVyJz+U4xMn+zqunoMwg9SwrNGbpRD1ppBuZbsX29FOvThl30n6x/19q+T9Kc6aNHDa0YE6F8qzWfjafwcBJXp5NjSEAgeQRwMBJXptXtcYuJxU2tqQYONVKJONajnoJ6Q6zf9hMXOubrV4amqKhGP5hMdn2Z/2SyF4U0na0XRI0oJ6F6tmUb/4ll1nYfAYDp6opLoVBAAIQqAkBDJyaYE9OoS4nFTY2DBwuUgpJzDX0SnelC9k3zvvornqPrp3NhWedkHgWcW5n6sbvYlI0oHl67FCvqNbZ5jMYOMnJr6kpBCCQXAIYOMlt+6rU3OWkwsaGgcOFij9pVy8bzUei4QB6aSiCnpCzbPG9ZQ/D8pcVxc8aEqYJeaMYOzHzfUcDaCBdA5qoudjhXunHu/De5jMYOFVJbSkEAhCAQE0JYODUFH/8C3c5qbCxYeCQzPsTcD1ud801mnmTf2oC0PSXJulMf9KO/1g+oyc0gAbQABqopgZsPoOBE/+8mhpCAAIQwMBBAxUl4HJSYWPDwCHRrmaiTVnoDQ2gATSABsLUgM1nMHAqmtJycghAAAJOEMDAcaIZ4huEy0mFjQ0Dh0Q6zESac6EnNIAG0AAaqKYGbD6DgRPffJqaQQACELAEMHAsCZYVIeByUmFjw8Ah0a5mok1Z6A0NoAE0gAbC1IDNZzBwKpLKclIIQAACThHAwHGqOeIXjMtJhY0NA4dEOsxEmnOhJzSABtAAGqimBmw+g4ETvzyaGkEAAhDwE8DA8RPhc6gEXE4qbGwYOCTa1Uy0KQu9oQE0gAbQQJgasPkMBk6oKSwngwAEIOAkAQwcJ5slPkG5nFTY2DBwSKTDTKQ5F3pCA2gADaCBamrA5jMYOPHJn6kJBCAAgWwEMHCykWF9KARcTipsbBg4JNrVTLQpC72hATSABtBAmBqw+QwGTiipKyeBAAQg4DQBDBynmyf6wbmcVNjYMHBIpMNMpDkXekIDaAANoIFqasDmMxg40c+bqQEEIACBfAQwcPIRYntZBFxOKmxsGDgk2tVMtCkLvaEBNIAG0ECYGrD5DAZOWSkrB0MAAhCIBAEMnEg0U3SDdDmpsLFh4JBIh5lIcy70hAbQABpAA9XUgM1nMHCimy8TOQQgAIFCCWDgFEqK/Uoi4HJS0bx5c6P47rzhclPNRIuySOzRABpAA2gADaCBMDTwn9ce83IZ5TNLliwpKVfjIAhAAAIQiA4BDJzotFUkI3XZwOnevbuX9Fwx+UwMnA9IpMNIpDkHOkIDaAANoIFqauCxe29IGThffvllJHNFgoYABCAAgcIJYOAUzoo9SyDgsoFz4IEHeknPhPGjMHAwcNAAGkADaAANoIHIaeAvl57j5TIbbLBBCVkah0AAAhCAQNQIYOBErcUiFq/LBs7pp5/uJT0jDhsSuYStmnf3KIu7yWgADaABNIAG3NTAyWNHeLlMr169IpYhEi4EIAABCJRCAAOnFGocUzABlw2cWbNmeUlP/769MXC464oG0AAaQANoAA1ETgP7Dejj5TLDhg0rODdjRwhAAAIQiC4BDJzotl0kInfZwNHTGhRfs2ZNIpewcSfUzTuhtAvtggbQABpAA9XUwFprru7lMhMnToxEXkiQEIAABCBQHgEMnPL4cXQeAi4bOAq9Q4cOXuJzwxWTMHG484oG0AAaQANoAA1ERgNzZk7xcpiGDRua5cuX58nI2AwBCEAAAnEggIETh1Z0uA6uGzgXX3yxl/z03qF7ZBK2at7ZoyzuJKMBNIAG0AAacFMD++61m5fDHHXUUQ5ngoQGAQhAAAJhEsDACZMm56pDwHUD59133/WSH8X58B2zMHG484oG0AAaQANoAA04r4F3XvyrqV+/npfDPPLII3XyL1ZAAAIQgEA8CWDgxLNdnamV6waOQA0ZMsRLgI44eJDzCRt3Qd28C0q70C5oAA2gATRQTTRk7AgAACAASURBVA1MPe8UL3fp06ePMzkfgUAAAhCAQOUJYOBUnnGiS4iCgXPjjTemeuFcfM7JmDjceUUDaAANoAE0gAac1cALf5tnmq+5hpe7zJ49O9F5JpWHAAQgkDQCGDhJa/Eq1zcKBo6QHHTQQSkT556br3A2aavm3T3K4m4yGkADaAANoAH3NLBNl05ezjJ69OgqZ3UUBwEIQAACtSaAgVPrFoh5+VExcNQM7du39xKipk0aY+Bw5xUNoAE0gAbQABpwTgND9unr5So77LBDzDNIqgcBCEAAAkEEMHCCqLAuNAJRMnBeeeWVVC+cdhu3cS5p4y6oe3dBaRPaBA2gATSABqqlgRPHHJnKU957773QcjVOBAEIQAAC0SGAgROdtopkpFEycAQ4fT6cNhuub/7x8sMYOdyBRQNoAA2gATSABmqqgZ7duqTMm4ULF0YyJyRoCEAAAhAonwAGTvkMOUMOAlEzcFSVGTNmpJIkxX/z1ZNrmrRV684e5XAXGQ2gATSABtCAWxqYM3OKWXedtVN5yZQpU3JkXWyCAAQgAIG4E8DAiXsL17h+UTRwhGzp0qVmrbXWSiVMY0YcbN558SGMHO7AogE0gAbQABpAAxXXwH/fe8Gcd8a4VB7SpEkT89e//rXGWR3FQwACEIBArQlg4NS6BWJeflQNHDXLF198YQ455JBU8tR8rTXMmBGHmMUPzql44sYdULfugNIetAcaQANoAA1UQwO6WXTOqWNNx003TuUfvXv3NitXrox5xkj1IAABCECgEAIYOIVQYp+SCUTZwLGVPvXUU03Tpk1TiZTqNHRgP3PXjdPN608vwMzhTiwaQANoAA2gATRQsgbee2WhefSe672bRLpZZHMn5R4nnHCCTUdYQgACEIAABAwGDiKoKAGbhCxatKii5VT65P/617/M5MmTTY8ePVKJla3b6s2ami07dTAD99zVjP/jEWbC+FG8YIAG0AAaQANoAA1k1cAh++9ttuu+lVmvZYs6eYVyDeUcK1asqHR6w/khAAEIQCBiBDBwItZgUQvXmhxRN3DSub/wwgvmuOOOM1tttZVZY41f75TZurL8TZ1kFCYwQQNoAA2gATQQrIFmuhG05ZbmiCOOMPfff396ysF7CEAAAhCAQAYBDJwMHHwIm4BN1uJk4PgZffrpp+b55583t912m7nwwgvNxIkTecEADaABTwODBw82++yzD3pAD2gADWRo4MYbbzRPPvmkee+99/xpBZ8hAAEIQAACWQlg4GRFw4YwCCTBwAmDE+eAAATiSWDvvfc2u+22WzwrR60gAAEIQAACEIAABKpKAAOnqriTVxgGTvLanBpDAAK/EHj//fdTwwnfeustsEAAAhCAAAQgAAEIQKAsAhg4ZeHj4HwEMHDyEWI7BCAQVwJTp05NGTjnnHNOXKtJvSAAAQhAAAIQgAAEqkQAA6dKoJNaDAZOUlueekMAAj179kwZOFtssQVAIAABCEAAAhCAAAQgUBYBDJyy8HFwPgIYOPkIsR0CEIgjgSVLlqTMG/s7+MQTT8SxqtQJAhCAAAQgAAEIQKBKBDBwqgQ6qcXYC5c4P4UqqW1LvSEAgewETjvttDoGzpgxY7IfwBYIQAACEIAABCAAAQjkIYCBkwcQm8sjgIFTHj+OhgAEokmgXbt2dQycFi1aRLMyRA0BCEAAAhCAAAQg4AQBDBwnmiG+QWDgxLdtqRkEIBBM4JFHHqlj3tjfwttvvz34INZCAAIQgAAEIAABCEAgDwEMnDyA2FweAXvRwhCq8jhyNAQgEB0CI0eOzGrgDBkyJDoVIVIIQAACEIAABCAAAacIYOA41RzxCwYDJ35tSo0gAIHsBH766SezxhprZDVw9Jv4ySefZD8BWyAAAQhAAAIQgAAEIJCFAAZOFjCsDocABk44HDkLBCAQDQJz587Nad7oN3HGjBnRqAxRQgACEIAABCAAAQg4RQADx6nmiF8wGDjxa1NqBAEIZCcwaNCgvAZOr169sp+ALRCAAAQgAAEIQAACEMhCAAMnCxhWh0MAAyccjpwFAhBwn8BHH32U17yxv4nLli1zv0JECAEIQAACEIAABCDgFAEMHKeaI37B2IsVJjGOX9tSIwhAIJPA5ZdfXrCBM2HChMyD+QQBCEAAAhCAAAQgAIE8BDBw8gBic3kEMHDK48fREIBAdAjsuOOOBRs4m266aXQqRqQQgAAEIAABCEAAAk4QwMBxohniGwQGTnzblppBAAK/EtCQKPt7V+hy4cKFv56AdxCAAAQgAAEIQAACEMhDAAMnDyA2l0fAXsgwhKo8jhwNAQi4TeCss84q2sAZOXKk25UiOghAAAIQgAAEIAABpwhg4DjVHPELBgMnfm1KjSAAgboEOnbsWLSBs/rqq5uffvqp7slYAwEIQAACEIAABCAAgQACGDgBUFgVHgEMnPBYciYIQMBNAuphaH/ril3ecsstblaKqCAAAQhAAAIQgAAEnCOAgeNck8QrIHsxwxCqeLUrtYEABH4lMGrUKLPNNtuYbt26ea/u3bsbvfTZ/gZ26dLFbLvttqZnz56p13bbbWcGDhz464l4BwEIQAACEIAABCAAgRwEMHBywGFT+QTsxQsGTvksOQMEIBAtAqtWrUoZOG+99Va0gidaCEAAAhCAAAQgAAHnCGDgONck8QoIAyde7UltIACBwglg4BTOij0hAAEIQAACEIAABPITwMDJz4g9yiCAgVMGPA6FAAQiTeDrr79O9cBZvnx5pOtC8BCAAAQgAAEIQAACtSeAgVP7Noh1BBg4sW5eKgcBCOQggIGTAw6bIAABCEAAAhCAAASKJoCBUzQyDiiGAAZOMbTYFwIQiBOBdAPnzTffjFPVqAsEIAABCEAAAhCAQA0IYODUAHqSisTASVJrU1cIQCCdwDfffJMaQoWBk06G9xCAAAQgAAEIQAACpRDAwCmFGscUTAADp2BU7AgBCMSMAAZOzBqU6kAAAhCAAAQgAIEaE8DAqXEDxL14DJy4tzD1gwAEshH49ttvUz1w3njjjWy7sR4CEIAABCAAAQhAAAIFEcDAKQgTO5VKAAOnVHIcBwEIRJ0ABk7UW5D4IQABCEAAAhCAgFsEMHDcao/YRYOBE7smpUIQgECBBNINnNdff73Ao9gNAhCAAAQgAAEIQAACwQQwcIK5sDYkAhg4IYHkNBCAQOQIfPfdd6khVBg4kWs+AoYABCAAAQhAAALOEcDAca5J4hUQBk682pPaQAAChRNIN3Bee+21wg9kTwhAAAIQgAAEIAABCAQQwMAJgMKq8Ahg4ITHkjNBAALRIvD999+neuBg4ESr7YgWAhCAAAQgAAEIuEgAA8fFVolRTBg4MWpMqgIBCBRFIN3AWbZsWVHHsjMEIAABCEAAAhCAAAT8BDBw/ET4HCoBDJxQcXIyCEAgQgQwcCLUWIQKAQhAAAIQgAAEIkAAAycCjRTlEDFwotx6xA4BCJRD4IcffkgNoaIHTjkkORYCEIAABCAAAQhAQAQwcNBBRQlg4FQULyeHAAQcJpBu4Lz66qsOR0poEIAABCAAAQhAAAJRIICBE4VWinCMGDgRbjxChwAEyiLw448/pnrgYOCUhZKDIQABCEAAAhCAAATogYMGKk0AA6fShDk/BCDgKoF0A2fp0qWuhklcEIAABCAAAQhAAAIRIUAPnIg0VFTDTJKB89FHH5lnn33WLFq0iBcM0AAaMAsXLkz1wJk9ezaaQBNoAA2kNPDee+9FNbUjbghAAAIQqCEBDJwawk9C0XE1cO68804zbtw4M3DgQNO5c2fTtGnT1IWarTPL38DkNzDge4AG0AAaQAPBGmjQoIHZfPPNzYABA8zo0aPNFVdcYT7++OMkpIfUEQIQgAAESiSAgVMiOA4rjIBN2tQrJep/ixcvNn/84x9Ny5YtMSYwJtAAGkADaAANoIHQNaAcQ7lGHPKmqOd9xA8BCEDARQIYOC62SoxiioOBM3fuXLPLLrtkJGnrtWprhhw13hw1/gJz/Hl/MWdffY+5/I5nzU2PrTBzF/+HFwzQABpAA2gADaCBrBq46t6/m0nX/9WccvHN5g+nTTUH/uFU02HL7hm5hnKPm266KUZZIVWBAAQgAIFyCWDglEuQ43MSiLqBM3369IxkSsnViBMnmev+9nbWpAwDBwMLDaABNIAG0AAaKEUDJ1wwy/TovVdG7nH66afnzLXYCAEIQAACySGAgZOctq5JTaNs4EycODGVQMm4UVJVSjLGMSTxaAANoAE0gAbQQDEaUM/eHfoMSuUhgwcPrkkeR6EQgAAEIOAWAQwct9ojdtFE1cBJHzLVZ9AwetzQDR7zDg2gATSABtBA1TWw19CjUyaOJjzmDwIQgAAEkk0AAyfZ7V/x2kfRwLExazl83DlVT9aKuUPHvtzRRQNoAA2gATQQbw0M/9O5KRNHuQl/EIAABCCQXAL8F0hu21el5tYMicrTFLp3/3UCwZMuuh7zhrutaAANoAE0gAbQQM01MO22p1ImDsOpqpLCUggEIAABJwlg4DjZLPEJKkoGzqhRo1LJke52cUcz3nc0aV/aFw2gATSABqKkgasXLEnlKVdeeWV8kkVqAgEIQAACBRPAwCkYFTuWQiAqBs6sWbNSSZEmDYxSQkesXICgATSABtAAGkiGBk6bNjeVr9x3332lpGYcAwEIQAACESaAgRPhxotC6FEwcJYsWWJatmzpJUSdu/XCvKGrPBpAA2gADaABNOCsBg4dc6aXs7Rp08YsXbo0CukgMUIAAhCAQEgEMHBCAslpgglEwcA56aSTvESo9Sabmctuf8bZhI27q8m4u0o7085oAA2gATSQTwM79Rvi5S5jx44NTsBYCwEIQAACsSSAgRPLZnWnUq4bOF999ZXZaKONvCSIJ06RMOdLmNmORtAAGkADaMAFDUy4fJ6Xu6y55prmn//8pzuJH5FAAAIQgEBFCWDgVBQvJ3fdwLnuuuu8BKjFuhuYWQ+9Ru8busyjATSABtAAGkADkdBAxy49vBzmnHPOIeGEAAQgAIGEEMDASUhD16qarhs4/fv395KfQcPGRiJZc+GuHzFw9xkNoAE0gAbQQO01cPjYiV4O0759e7Nq1apapXqUCwEIQAACVSSAgVNF2EksymUD5/nnn/cSn3r16puLb3kcA4c7rmgADaABNIAG0EBkNDDjzudNo8ZNvVzmqquuSmKaSZ0hAAEIJI4ABk7imry6FXbZwJkzZ46X9Gy93a6RSda441n7O560AW2ABtAAGkADrmhgmx36eLnMH/7wh+omeJQGAQhAAAI1IYCBUxPsySnUZQNn0qRJXtKzy4ADMXC444oG0AAaQANoAA1ETgP9hhzp5TL9+vVLTnJJTSEAAQgkmAAGToIbvxpVd9nAGTVqlJf0DDqc+W9cuZNIHNzVRgNoAA2gATRQuAYOO/bPXi7ToUOHaqR1lAEBCEAAAjUmgIFT4waIe/EuGzh77LGHl/Tw+PDCE0WSalihATSABtAAGnBHAydddL2Xy9SvXz/uKSX1gwAEIAABYwwGDjKoKAGXDZx27dp5Sc9xZ18VuS7TJM/uJM+0BW2BBtAAGkADtdLA1Fuf9HIZ5VsrVqyoaE7HySEAAQhAoPYEMHBq3waxjsBlA8fGdub0+Rg4zHuABtAAGkADaAANRFIDNp9ZtGhRrHNKKgcBCEAAAvTAQQMVJuByUmFjw8Dhzmmt7pxSLtpDA2gADaCBcjVg8xkMnAontZweAhCAgAME6IHjQCPEOQSXkwobGwYOyXO5yTPHoyE0gAbQABqolQZsPoOBE+eMmrpBAAIQ+IUABg5KqCgBl5MKGxsGDkl3rZJuykV7aAANoAE0UK4GbD6DgVPRlJaTQwACEHCCAAaOE80Q3yBcTipsbBg4JM/lJs8cj4bQABpAA2igVhqw+QwGTnzzaWoGAQhAwBLAwLEkWFaEgMtJhY0NA4eku1ZJN+WiPTSABtAAGihXAzafwcCpSCrLSSEAAQg4RQADx6nmiF8wLicVNjYMHJLncpNnjkdDaAANoAE0UCsN2HwGAyd+eTQ1ggAEIOAngIHjJ8LnUAm4nFTY2DBwSLprlXRTLtpDA2gADaCBcjVg8xkMnFBTWE4GAQhAwEkCGDhONkt8gnI5qbCxYeCQPJebPHM8GkIDaAANoIFaacDmMxg48cmfqQkEIACBbAQwcLKRYX0oBFxOKmxsGDgk3bVKuikX7aEBNIAG0EC5GrD5DAZOKKkrJ4EABCDgNAEMHKebJ/rBuZxU2NgwcEiey02eOR4NoQE0gAbQQK00YPMZDJzo583UAAIQgEA+Ahg4+QixvSwCLicVNjYMHJLuWiXdlIv20AAaQANooFwN2HwGA6eslJWDIQABCESCAAZOJJopukG6nFTY2DBwSJ7LTZ6TdvwtT71vbnny3yZp9a5lfec8+S9z/cJ3YL6Y36ta6rDYsm9+/D1zzQOveq8bH1vhpH713Sq2Xq7tb/MZDJzo5stEDgEIQKBQAhg4hZJiv5IIuJxU2NgwcLggci0Zdz2enfsfYA485pTIX/S4zlnxnX7prabd5lubevXrG/1mNWzU2Ky7fhvTuXsvM/2O52gDDB2nNbD19rt5upV2R5482clY23fqasZPutbJ2Ar9jbL5DAZOSakqB0EAAhCIFAEMnEg1V/SCdTmpsLG5auBs2aO3+e1qqwW+flevntmgTTuz694HmyvvednpxFPxbdq5W1ExnjPzXu+CNaj+DRo2MlPmLCrqfIUmweyX38ybcPnt5ne/q2cum/9MYBusuXbLQM0GtaXWXXb704HnoS3+Y06beotp3KSZ2W/4ceaCax801/3tLY/X2VffY0666Hpz46J/wK7GBs5Bo04rSu9Hn3RR4tpMvfW22bGvswbOEX8616zd8veR7uFm8xkMnOjlyUQMAQhAoFgCGDjFEmP/ogi4nFTY2Fw1cNTdfOZ9r5gtttnBHDV+kvden/W6asHfzTnXLDB9Bh1umq6+prl4zmPOXhRMvnGhWaP5OkXHN+uh1726XnLL494dXNVZdZ/91zeKPhdmQH5jplBG7Tp1NeqBk23/NdZqYSZedXeGXg8YeYrpvtOeGevUlo2brm4unYeBk43lZlv1NPsc+sesrLMdx/rw9J6PpYYTzrx/qRly5PFmhz6D6mjc/mZrKRNjxIkXJrI9u/Xaw1kDR0boWmuvaw4efXpk28bmMxg4RaWo7AwBCEAgkgQwcCLZbNEJ2uWkwsbmqoFjLxy69NzFjDlretbEUkmnuoDb/V1blmrg2Hpo/gS1FXOuVO+i1LL3L8+bdb/XFmpT/zb7WQaOTDf7Wcth4842O/TZN2Od1jdphoGTzsn/fvU11zYX3fi3Otz8+/G59t+NQ/44wewy4MCcbdVz1wEYODXuMZXtuzL06JPMOuu1iuz/GZvPYOBEJz8mUghAAAKlEsDAKZUcxxVEwOWkwsYWdQNHwypWW+135tpHlue8eNBQlfGTZpvh484xp1x8U0FDrzS5o3q+2JffRNGFvHpb6C60PzHWXU0dpyE3uhC150hf6q60/zj/53INHMWsOEdPuMwcduyfzbhzrs47d8jNT7yXEa8m4vTHddNj/0ztc/WCJYEMdMz1C982GhJ29MkXmVGnTzMXXPeQKWQyz9l/fTN1/msffjOjfH0+9ZKbvYlB/XFV8vPu+x5u2m66RUYs/vLCMHDCqHsp7W7bVD0qVC/pWsP11G4yUYuZc0ZtrO/ZUeMvMEeecL7XXrYdg/RkOWqiYvsdadJsDfPnK+5MfbbrCxk69ZcHlnnz52h4iL73hcRe6vdd32PFpnr95cHXzLF/nmGOP/8vnvZtvc6f/YD5w2mXeNvtumxL8T992lyj2P907jXmkrlPZP1+pZ9Dx2VjdMXdL3lD0gphl37OQt8XYuBsv/vAwF4oYqa4b3j0XU930r9X/+PP89q/mJh1jnP/cp9XzjGnTDEabmcnv86lO1tP6UR6EXvNvyQd2W2FLGU4qsfoWTPmZwzxK6QHTrX/R6XXR0NCf/vb35pTptxUVH3Tz1HL9zafwcApKDVlJwhAAAKRJoCBE+nmcz94l5MKG1vUDRyZE7rQk+kQlEBqDpruO/XzhqtsvvV2Xjf/TbfYxmgumd0HHpbT+NnnkNFeUqvEVrzshe3x511jWqy7gTd8a+MOW5qW67c2E6+8K6P8LtvunHGszuF/yXjSBVpQ3HZdqQaOLsZ777m/F6O6xyueXv2GmC277+TNK7Lv4WOzlnvYsWd5sarOinmnfvvX2Xe73fb5dZ/VVvN6mdiY7VKmUbM11jKtNtrUbLtzf7PNDn3MOr/f0LvTqws0u59/qQviRo2bps7fd7/h3r5ioWEamgNJRormm9E8SNna3n/ecj7rIlBDng4YeXLWuHV+DZfTBXd6Wdl64Gj4n38OnHLrXk67q5eQ2lvzYWgIX9cddvfqrF5wG3fs4n0H1O75eKvdZVpuuFEHr9233m43rz1V3x377uedN51P+vvW7TbzYlAcVn/+742GVqUfk/5ese1/1Hjv+92m3eZGpoHmoGrYqIlR7NJQ+v7p70v5vssQ1fdYsWo+GGlS5WmOLpUv06r3XkO978FWPXf1vnsyFtLLte9lVOx72LFe7JtstpXXa2uLbjt6x6rO025bHHicjtecQJqbyXIbd+5Mb18ZCvq9q1+/gddTUfqUoWbLDGsZZODIiJE5Y8uQeSuz137WUvEpNvETF/22qmeatKNhh/rOaR4k/+9r+jnsex2r37rfb7iR6dF7L++14cYdvXV/PPMy03qTzcy0257KKN8eK11st+venk46bNnd0420qP8T+484Ma/mZdroe6O4pTP1CtX/Jf3OygjNZeDU6n+Urbtd6juu/5X2c5SW0o9eGDju58WlRrhq1Srz8ssvm7lz55qJEydW9TV58mSzYMECs2zZMvP111+XWgWOgwAEQiKAgRMSSE4TTMDlpMLGFmUDR71uZEzoIj4o2dT8Irpo3PvgUUYXE+n76I60LliV7KsXT/q2oPeacPaqe/9u9hh8hFEvC1242Tu6o8+41EvUg46TwVTKHDj2XKUaOOqBccypF3sXSP4eQqq7LjRyPRXlpMk3mPVatc154aL6y5DRnWobr13uOXSEdxE74bJ5GdsUy9iJV3gXuoXMh6Gu/TJwdPHXfJ31PAPH9qaQ2aELNb8JYmMIc6kLYn1ndJGY67zSgtWF3S+bgfOH06aaXI/wLaXu5ba7Lqh1wawnPWn+mfT4pt76pDcnlb5Ptm7+pS60ZWhIP+nbZKxomwy9jlv2yNiWvl/6e+1bzPxWinXzrbf3DBTFmn4uzR2luYt0gS/9p28Lel/s9/33G26cMiF0PrWDTBf9VsgAs71LNDGzzCR/LzQZbzJtZHb5jRodKy3IzND3ICje9HUyfaRXGWkqS3Mw2d8/9QTSXDTp+4fxPsjAkfEx8NAxBZWl77EMr3U3aJsxbE6mlnqAqe5+8yc9bpl2MtLV+yZ9vd5LdxoepO+vJsP2b7/irhc9XWgImOVk99FwSBlgnbpun/FdsNu11O/o+q03qVO2/l+o7WWAq12Dfm9d+B9l6yKDXyagek3adVFZ2nwGAyc4F4362nnz5plNNtnE+w7btq7VsnXr1mb69OlRR0r8EIg0AQycSDef+8HbfzAuJhU2tigYODIDdOFsX0qENZxFF5nqkaGLH3+iqYs5Jd5Djjqhzja7r8wEmTj9hhyZdR+7ry7odGdWd1bV3d+uz7eslYGTL64xZ15u9KSvXPvpjqwMiWz7yCAKesLWyZNv9O5G68Io27G6kFJvBWvGZNtPF65t23fy7sprGFa2/Sq9XnHoOyODo9iyshk4+c5Tibrna3fVT/Xcb/i4wHrKGFFPA78paOuiXiia68R+9i/Vhvl0Z48p1sDRo931yPFcPYT67X+Ud1Fty8i2LPb7LgPH/1ujni5ilT7xuLipXpNvypxHSb1/1DvOb/6lx6deHht16JyzftpfBo6+l/p+abL39HNU6r0MHJWr4Uf2te0uA8xeQ48uqHwZOGKV7fdAv/PZvv/6LRHTXL/LGgooXQcZOLoJ0P/AkVnjVJuop+VBfzi1zj4y5GTQy0wOYqv2lqmonlF+A8eV/1E2bhluYjTl5kcD62L3c3GpuPVyMddyP1N1O8IBAwZ4bav2Xa9lC9Nn5+3NhPGjqvr606hhpvcO3U2zZk1SsQwdOtRtcEQHgRgTwMCJceO6UDWXkwobWxQMHA1H0N1s+1IyrTuyqoO6fAddVKtXyNrrrp93UkZ18a/foGHOoRVKWHVBpwtTeye90CTWVQNHc8joYjBXPTQsQz2U0nth2P11kay75TqPXWeX6mHhv1ix29KX6smhC+r0df73MjE0pCKol49/30p+Vi8v6c3Op1FMWeUYOGHXPV+767ukXiq56qchbBr2EbSPekCs1WJdb96loO3SUq5hTOnHFGPgqEeLhuLk67Gj76+GP2qOkvSy/O+L/b7LwPGbJZo3SKaM/9z6Xbrw+odT69XLRD0fsg3vST9evXQ0v076Ov97GSkaOpTNDPHvH8ZnGTgyjOxvtJbiXIyBk6tHnsy53QYeGlhvmT8yDvPVQ08ttMNg7b76/6eeO/nm2ZHhJn35e06p147mNrLnC1qqR5V+O/y/ia78j7Ixa+4mxSkD3q6LylJx64WB40LmG14MHTt29NpVbXvF5DPNtx8sqenr9WfuM/vutVsqpmnTpoVXWc4EAQgUTAADp2BU7FgKAZeTChtbFAycbE+hUjKungIaJuCfU0VPp9KcNxqmku+l7u+FXND5k/9CkttqGDhBBoti08WGLoo074hMFQ2b0p153c3WK5+Bo3N4vXAmXFYnodfwDPV28DPQ0BG1x+FjJ+blriE6GprgP0f6Zxk4dg6cbY001AAAIABJREFU9PXVfq/5SzQcppRyyzFwSql7Oe0uA0c9IXLVUxeyuYYhaciL5jBSz6leewz25qTRd7jYR6YXY+BoaJAuxHPFbbdpqIjmebKfg5YycIr5vpdj4Mj0kvmR73dK2zWXioYLBcVs19khVPZzNZZBQ6g0/0sxBk62uYEUvwyaXfc5JLDe6ok56bq/Bm7LV/dDx5zpDa3Lt5+2y3jz9+DR0Kzpdz6ft2xp02/guPI/ytZdEz4rLzj6pNr1dLSxFLu0+QwGTimZqpvHnHzyySmjpNbGjb/8e26+IhUbmnNTP0QVbwIYOPFu35rXzuWkwsYWZQPHJnkayuOfV0O9OzSHy9bb71bQS13h7fmClsVe0NlzVMPA0dCuC294JCN+eyGuO/G6SNFdez3RR08G0lAEXQgWYuD80gtn44xeODJpZHppm62nXWo+IQ0XkOFRCHtN8GqPDVq6YuDY+SFyDXEJil/rqmnglNvu9vhsddH6fAaO9lFPJc1/pGE/ex0w0uuJoglhNR+IJkjOdX67rRgDZ+zEKwN7u9hzpS9lgPTdb1jOGIr9vpdj4KjHgya8LeT7on3ymU+uGDhnXHZbwU81Ui+aUgwcfR/1e+Ofuya9vXO911BczRGUax+7TWaz5j6zn6Vj9ZzKNpzQ7qel2sRv4LjyP8rGqaFvygvyTaxv93dpafMZLqZrnvaGEsDnn3+eMkiOHz28pr1u/OaN/TzisCFejDvuuGModeYkEIBA4QQwcApnxZ4lEHA5qbCxxcHAUfd3DalKHzKguT40gWRYSWaxF3S23GoYOBu0bZ/Rg0gXzxq+IOMm28VFvqE0Nn4tNWRNT3Gx6zQJcS7zR8aZfziJPbbYpSsGjnp66TtTyDAXfx2rZeCE0e5hGTh+BvqsuWA0n5Umqw3a7l9XjIGjHhjqDeE/R9Bn9T4bdtzEnPsW+30vx8DRXFGaODpoLq+g+POtc8XA8ccpc1e/Jf71+lyqgaNj/UPSgs6fbZ16Cuppfdm2p6/X0MD0oW/apt4/l89/Nu/x0qbfwHHlf5Sto3pR6TdOvdnsuqgsbT6DgVNCourgIS+++KKnRc05Yw0T15aP3XuDF+M666zjIEFCgkC8CWDgxLt9a147l5MKG1tcDJyGjRp7T4myCafmHdAd/1wTW9p9C1kWe0Fnz1kLA0fDyfSYbRtD0LIYA+fEC6/zetxoqJYMIQ3FOuGCWVnPrwsiXagHlVvsOlcMHPXy0nfmlCk3FV2vahk4YbR7OQaO9DH4iD/l5SOzo5AhL8UYODI/9Bvg74nm15tMBD3NbOJVd+eMs9jvezkGjmLU09xyfaf89cj12VUDR/MmqfdWUOzlGDh63L2MmKDz5lunJ1TJAMr35CU9/Uv68vfA07CuXHP3qHw7gbLfwHHlf5RltPvAw7zfuL88sKwklvY8tVjafAYDp+ZpbygB6KlTatNN27V11sCRodS0SWMvzvfffz+UenMSCECgMAIYOIVxYq8SCbicVNjY4mDgHHbsn71JO/2J476HHevNg5PrqTQ6RhcW+YZ1FHtBZ2PRuXNN+mr3y7Ys5DHi/h444ydd690ZzjY3ziVzn/DuhOfqReOPR3fOdcdY3etbt9ssa88eHafH52q+mKAJjtPPKzNIFzfp6/zvXTFw1EtCj8eWGeOPMd/nahk4YbR7OQaOzFL9ruSaSFhGiyZJDtvAURuoV4169/gnmk1vH11wd99pz7xtWOz3vVwDR98VzYOT68ltqod6WV02/5mc8SfNwJEJoyFo+eai0RxCQU8c1ET4GkqVrpP09+rh2XqTzQJ7ben/5+prrp11Um/9BuvJhdKT38BRGS78j7J11RCxjl22zcrB7ufi0uYzGDglJquOHTZx4kTvf4me/ORaz5v0eBSftIfuHBMQ4cSeAAZO7Ju4thV0Oamwsblq4OhCT3cCdTGiSRX1Pv0lY2PC5bd7k2TqgtD/WF4lmTqHl5Ru2cPoDqo/8dSFkBJ33VkNMht0vJ5ao5cScBkf6THoCVb+cwZ9luGxy4CD6szTIKND3dY1h43KSD9WczqoLD22WW2lCVUzyk7joYvH9EmYFZcMFJ07/bzXPvymx1LDNZSoKy6dM9cFr41JvXD0NDBdIKfPA2G3+5faXxc2enqM//wy1DQMSz15VHf/sTJ2LHf16BA7f93T6+U/vlKfdae/0GF5qrONWQzUw8B+znW3v5y6l9vuMgfOmnGH9xQpxRpkfHpPbavfwPu++Q1C6Vla1bCSoB4uM+56wWyzY1/vO615lILa6dpHlqc4NV19TXPOzHtTny0/MQo6Vus1R4wmT/b3xNH3R4+11mSy2R75XOr3XeavfSJbeg8NTdysR0j752dRjw/97vr5atJf9cQZd+7MOvXTOfQYa32nBg07rs52nct+Z1Smnoxkedmlv7wghsWuE3OdX/MKabJ0W1bQ0j7JKb0Me7y+WzLg9BuVvl3vVS/9Dmj+JPWg8m/XZ02IrmGjQT3kVK7+D8jk0bw8/uOlB3Hfbte96zwhTUOm9Ju3zQ59sprWenqTdKX/R+nn1tAq/c5qviVbP///DFf+R2l4o3pHaQLy9DpE5b3NZ7iQrm3OG1bpGDhhkeQ8EIgnAQyceLarM7VyOamwsblq4OjxuzbGbEtN0Nv/wJFGXdGzJZq6UBw+7hxvjgkl2Vtvt5vXK2eN5ut4xo0urM+bdX+d472LjQYN88ZwxJ/OrXOsPxaZMLoI0ISXeryuTCnNiVCvfn1v4lX/RYcuWrVvtnoHrU83cFS+JixWfTWMTCxllsiE0t1gPUlFhpU9jy54/TEHfVYvnFYbbZr30ez2WLWLLib1RCI9sUoX74pDFwp6PLmewhL0+F49ucjGlm2p9qvEBamNPWgpU0q9cAoZlqf6ZotdBpoumILKKLfupba72kGGSXrMMjzSY9QwlfTteww+ImO7jCmZiZokW0+yarZmc++7pnaXdqRFPY49W911wat90ssIeq+5ndLjSn+v77uMAJmyMlV04ex99+rV855iJIMofX/7vtTvu8wWTaJr49Tk4PacMnC0Xj3w0n9jZOBo/Xa77ZPa1x6jYXD63krf+u5oMnB91m/FpltsY447+6o6x6jOMi9sDNmW2+7cv86xttxSl3o6VLbygtb7h1AdMvqMjOPFKt1slyEm08qeS/rI1tNGbGRca66aLXv09rQnTWt+tIGHjsn5vZVxJG2Ks/Qi3Ug/0pH0lM1wtNz0VD5pX0aQjlWPHenfTpAsXagO0op/yJUL/6PUO0i/S7nMZVtXF5dWHxg4zqS/ZQWCgVMWPg6GQOwJYODEvolrW0GXkwobm6sGTthJoi4ENDxEd0s11ESPNM6XlIcZg+40y5jRZMAahjTl5kdDm7Q0W5zqBaL21V1VXRiq11K2fQtZr0eKi18h+6bvc/WCJeb0S2/14hCDoLvs6fu7+l49TnShPOSoE4pmUM06hd3upcSuGDRMSlqX/tTzoRDjq5Sygo5Rrw2ZJqPPuNR7Gpa/50PQMa6s02/FZbc/bfRUIH3f9LulnhquxOdyHOJkf+fVU1BmkHqWFRqzdKKeNNKN9FNsTz/16pRxJ+0X+/+llv+jNGdan0HZh5EVyq9W+9l8BgOntjlvWKVj4IRFkvNAIJ4EMHDi2a7O1MrlpMLGlhQDp1aJZVzKVS8h3WH2D5uJS/0KrYeGpmgohn9YTKHHs99/Cr6YhhWs0EDlNaCeherZlG/+JZfbwuYzGDjOpL9lBYKBUxY+DoZA7Alg4MS+iWtbQZeTChsbBk7lE2SXE99CY9PQK92VLnT/uO6nu+rtOnU1hx17VuJZxLWNqRe/iUnSgObpsUO9olpvm89g4NQ25w2rdAycsEhyHgjEkwAGTjzb1ZlauZxU2NgwcLhY8Sft6mWj+Ug0HEAvDUXQE3Km3fZU2cOw/GVF8bOGhGlC3ijGTsx839EAGkjXgCZqLna4V/rxLry3+QwGjjPpb1mBYOCUhY+DIRB7Ahg4sW/i2lbQ5aTCxoaBQzLvT8D1uN0mzVb3Jv/UBKDpL03Smf6kHf+xfEZPaAANoAE0UE0N2HwGA6e2OW9YpWPghEWS80AgngQwcOLZrs7UyuWkwsaGgUOiXc1Em7LQGxpAA2gADYSpAZvPYOA4k/6WFQgGTln4an7wqaeeak488cSax2ED+OGHH8zChQvNgw8+aF555RW7uqjl+++/7x2vc+j1wgsvFHU8O4dLAAMnXJ6czUfA5aTCxoaBQyIdZiLNudATGkADaAANVFMDNp/BwPEloRH9iIGT2XBvv/22+eabbzJXOvxp4MCBZs8993Qmwg8//NAccMABpmfPnubQQw8tKa5bb73VDBo0yHv17t3b9O3bt6TzcFA4BDBwwuHIWbIQcDmpsLFh4JBoVzPRpiz0hgbQABpAA2FqwOYzGDhZktGIrcbAyWywIUOGmEceeSRzpcOfzj77bHPWWWc5F+Fdd91VsoGTXpnFixdj4KQDqcF7DJwaQE9SkS4nFTY2DBwS6TATac6FntAAGkADaKCaGrD5DAZOPDJsDJzMdlTPj4ceeihzJZ+KJoCBUzQyZw/AwHG2aeIRmMtJhY0NA4dEu5qJNmWhNzSABtAAGghTAzafwcCJR+6MgZPZjmEbON9++21mAQGffvzxR/OPf/zDaPjW999/H7BHZVatWLHCfPXVVwWfvJg4gwycf//73+bzzz8vuDztWEoPnJ9//tm8++675tNPPy2qLHYOJoCBE8yFtSERcDmpsLFh4JBIh5lIcy70hAbQABpAA9XUgM1nMHBCSl5rfBoMHGM08e4222xj2rdvb5o2bWo22GAD770+29fWW29tgswYGS+bb755ar+PP/7Ya1HN46J5YFq0aGF23HFH88wzz9Rp6dmzZ3vDg9ZZZx2z2WabmdatW5uGDRua8ePH5zRypk2blipP8eWba+af//yn6dSpk3eMjI2LL77YtGzZ0nTo0ME0a9bMbL/99p7hUSfA/1tRSpzWwJHpc9ppp3lli9N6661nNt10U3PPPfdkKy5jfTEGjp1/R8w7d+5s2rZtazbccEOvvj/99FPGeflQOAEMnMJZsWcJBFxOKmxsGDgk2tVMtCkLvaEBNIAG0ECYGrD5DAZOCYmqg4dg4PzSKJ999plRD5F+/fqZm266yXuvz/aVqzfHypUrvf022mgjo14tw4cPN127djX6jsg4eOqpp8ywYcPqtP4VV1zhzbeTPmnyW2+9ZQYMGGBOOOGEOvvbFeqlY+O67777zM4772w3ZV0qRpkaMmsOOuigVO+UL774wsgQatOmjfnuu+8Cjy8lThk4++23n9l3333NyJEjzZdffpk6t8ysjh07mvPOOy+1LtubQg2cpUuXembNpZdemlGP9957z+y///5mjz328NoiWzmsz04AAyc7G7aEQMDlpMLGhoFDIh1mIs250BMaQANoAA1UUwM2n8HACSFxdeAUGDiZjVDOECr1hunSpYs5+uijM0yEzBLyf5KJo15AhfzpEdu77LJLIbt6vYwOPvjgQCNDPZD0yO5i/nLFKQNHvxVjxowJPKUMsVatWpmXXnopcLtdWYiBo55RW221lbn33nvtYRlLDak68MADzfTp0zPW86EwAhg4hXFirxIJuJxU2NgwcEi0q5loUxZ6QwNoAA2ggTA1YPMZDJwSk1XHDsPAyWyQcg2cQnqVZJb4y6dVq1aZ//73v0Zmg15NmjQJ2q3OumINnDfffLPOObRiwoQJ5pJLLgnclr6y0Dhl4Ky++uo559iZOXOmGTFiRPrp67wvxMB5+OGHze67717n2PQV6rHUrl279FW8L5AABk6BoNitNAIuJxU2NgwcEukwE2nOhZ7QABpAA2igmhqw+QwGTmm5qmtHYeBktki5Bs4HH3yQecIcnzSUaO+99zaaA0fztqjXzdprr+314qm2gaNHkWczcEqJUwZOr169ctTemFdffdV079495z6FGDgXXnihN8fP0KFDTa6XDCUZUPwVRwADpzhe7F0kAZeTCrm+im/sxCtNNRMtyiKxRwNoAA2gATSABsLQwCW3PO7lMspnNLcEf9EngIGT2YbVMnBkcGiC3WuuucbreWOjUO+b5cuXV70HTjYDp9Q4dVzfvn1ttQKX77zzjtHk0Ln+CjFwZsyY4c1zs2DBApPvhYGTi3bwNgycYC6sDYHA//73v1RSsWTJkhDOGO4pNHmWEp7Dj5uIgbOYRDqMRJpzoCM0gAbQABqopgZOuGC2l8s0btw43CSJs9WMAAZOJvpqGTgyb954443Mwv/vUy2GUGUzcEqNUwaOnnSV6+lP8+bN8yZ8DoTwfysLMXCefvpp06NHj1ynYVsZBDBwyoDHobkJvP322ykD5/PPP8+9cw22jho1yotvn0P/iIGDgYMG0AAaQANoAA1ETgMHjz7dy2W22GKLGmRSFFkJAhg4mVQ12e3cuXMzVxb4SZMYFzKESnPdNG/ePOujwjUZrwtDqMqJUwaOblzrkeVBf5p4WL8jCxcuDNqcWleIgSOTSMO15syZkzqON+ERwMAJjyVn8hF44oknvB8KjSF18W/KlClefDv1GxK5hK2ad/coi7vJaAANoAE0gAbc1MDuAw/zcpl99tnHxVSLmEoggIGTCW3q1Kneo7b9j9T+5JNPjB5Rfffdd2cekPapUANHh+hpVVdeeWXa0cabn+XYY4/1HuntgoFTTpwycHbddVfTrVu3OvX86KOPTP/+/Y1ubuf7K8TA0Tneffddb0jaVVddZX788ceM02rYlOb30WPe1Y78FUcAA6c4XuxdBAF1w5PT62oXujvvvNOLr3O3Xhg43HVFA2gADaABNIAGIqeBLbvv5OUy48aNKyJDY1eXCWDgZLbODz/8YEaOHOmZKDIqDzjgAO/x27pBfMQRR5h//etfGQfIhNA2vVZbbTWv50zTpk2NffXs2TNjf/tBE/h26NDBtG3b1hxyyCHeZL7NmjUzp5xyirHTQugcGtKV/nfFFVekzq3tjRo18sq15Wmpx3N/+eWXqcPUo0jnVnwa/jh48ODUNr3ZeeedTf369U2DBg2M35wtNk7xWXfddU3Dhg3NoYceav7f//t/Zs899zSbbbaZ0SPM+/Xr5w2tUs8cDRXz/02bNq1O/X73u99lrGvdunXgZMQffvihV7c2bdqYgQMHmsMPP9yIv4Zy6XO+R5b7Y+HzLwQwcFBCxQhcfvnlXlKx//77V6yMck78xRdfePHJZOJJVG7eWeSOL+2CBtAAGkADaCBYAxdc+1Aqj7njjjvKSYk41iECGDjBjaFeHE8++aR54IEHzIoVKwLNhuAjC1/7/fffG/Uw0U3ol19+2fh7/RR+psruGUacug66//77PaYyqCr599lnn5nHHnvMqBeQJoT298ipZNlxPDcGThxb1ZE6HXPMMV5icfzxxzsSUd0w5DzLwNl174Mid9eNhDY4oYULXNAAGkADaCAJGhh0+Fgvh3G1p3PdrIs1hRDAwCmEEvtAILkEMHCS2/YVrbntaihzREOVXP2T86wY69Wrby6+5XFMHLrPowE0gAbQABpAA85rYM6T/zLrt2nn5TAa4sBffAhg4MSnLakJBCpBAAOnElQ5p9GEVTJGNAO5638a76pYBw0b63zCloQ7itSRO+doAA2gATSABnJrYNw5V3u5i+a2WLlypeupFvEVQQADpwhY7AqBBBLAwElgo1ejyprhXKbINddcU43iyirjjDPO8GKt36ChmXDZbZg43HlFA2gADaABNIAGnNXATY//02y2VU8vd3F5mHpZyVmCD8bASXDjU3UIFEAAA6cASOxSHAH7dKeOHTuab775priDa7D3kiVLvNnQZTht0La9swkbdyNz342ED3zQABpAA2ggCRrY++DRnnmjJ7ssXbq0BpkTRVaSwJQpU7z23a77VubbD5Y4+9pqi45enM8991wlcXBuCEDARwADxweEj+UR0Czj22yzjfeDfv7555d3sioePWvWLC9mmThb9dwFE4c7r2gADaABNIAG0IBzGjj+vGtS+cp9991XxUyJoqpFQE9ZUj7atvUGzpo3H735ZEqHepoRfxCAQPUIYOBUj3UiStpzzz29H3SZOFH7QR81alTqn9HBo093LmlLwl1F6sjdczSABtAAGkADwRq44u4XU3nKlVdemYi8MomVfPvtt712btCgvnn9mfucNHHuvOFyL8ZNN900iU1EnSFQUwIYODXFH6/Chw0blkosonpXqHv37qk6/Om8azBxuPuKBtAAGkADaAAN1FwDF97wSCo/GTx4cLwSSGpTh0DPnr/McbTvXrs5aeD06Lqlp8fx48fXiZ0VEIBAZQlg4FSWb2LOfvPNN6cSi1NPPTXS9Va3VfsaeOiYmidt3IkMvhMJF7igATSABtBAEjTQ/8BjUnmJ8hP+4k/Aziep9r7n5iucMnEmjP+lx/p6661nfvjhh/g3BjWEgGMEEvdf4H//+5+56667imqG1157zcyYMcMcd9xx5uqrrzaff/55UcfHfed58+alEosoPDa8kPbQ3S1r4mzRbUdzwbUPYeRwBxYNoAE0gAbQABqomgZm3PWi6b5Tv1Q+ol7C/CWHQHrP9hGHDTEL77rWLH/ugaqbOV+ueN4se+oec9vsqWaTthum9Pjss88mpzGoKQQcIpAIA+fTTz81s2fPNnvvvbdp3ry5ady4ccFN8Oc//9l7QtGIESPMzJkzzejRo03btm3NokWLCj5HnHe88MILUz/kcbsrpPHl1sRp3KSZGTRsrLn4lserlrgl4a4ideTuORpAA2gADaCBTA3Meug1M3zcOaZNu81TeYjm6eMveQR0/WJzUZeWevgHfxCAQG0IJMLA2X///Y1cbHVHXLVqlWnatGlBtM8991yjMajff/99xv4vvfSSad26tXn//fcz1ifpw8qVK83hhx+e+qcS1/HYmstHj+m0/zTr1atvdt37IHPm9PkYOdyFRQNoAA2gATSABkLTgG4S6WZRi3U3SOUdLVu2NFwsJynDrlvX5cuXm/3228+0atUqpQubl1ZzudFGG5mBAweaZcuW1Q2SNRCAQNUIJMLA8dMsxMDRmE79UGkm+KC/008/3VxwwQVBm2K97oMPPjCXXHKJ6dKlS+qfSNyfhLB06VJz0kkneXpI/0fZuVsvs0OfQWavoUebg/5wqhl1xjRz2tRbPHNHBg8vGKABNIAG0AAaQANBGjju7Ku8XjaDDh9rdhlwoNl6u12NbhLZPEM5qHKPJUuWxDqvpHLFEVAerlEA1Xw9//zzTB9RXDOxNwQqSgADJwfeXA7zrbfeaoYPH57j6HhtWrx4sTn22GNNixYtUsmFeqZE9WlTpbaO2n3AgAEpBjbRYvnrxM+wgAUaQANoAA2ggdI00L9/f3PdddeZr776qtRUheMgAAEIQCDGBDBwSmzcMWPGmClTptQ5WnPltG/f3nttu+225ueff/b20XL77bdPbTvyyCPrHOvCCg0x+/vf/25uu+02c/7555ujjz7a9OnTJ8Ow6Ny5sznjjDOMeqYk9U8TWWsybPVGkrElU6dTp06mUaNGGaxIYEtLYOEGNzSABtAAGoizBtq1a2f22GMPo7ltJk2aZObMmWPU04E/CEAAAhCAQC4CGDi56GTZ9sQTT5iNN97Y6IlW/r///ve/5ogjjjC77LKL+eijjzI2f/zxx2b33Xf35uP58ssvM7a58EFxZ0uW1l9/fTNy5Ehz991388hAFxqLGCAAAQhAAAIQgAAEIAABCEAgUQQwcIpsbk1grKFDWmb7U28bPa1q3333Nd999523m5aagEx3WmyvnGzH12q9xlpnM3C0fvPNN/cm0tP8QPxBAAIQgAAEIAABCEAAAhCAAAQgUD0CGDhFsH7uuec880bzwRTyN378eLPnnnuazz77zFuecMIJhRxW032+/vpr8+qrr3rDgzRETIaTfwiVHsV+yimnMLFeTVuKwiEAAQhAAAIQgAAEIAABCEAgSQQwcAps7UcffdRsuOGG5plnninwiF92k9HRrFkzz/Ao6kDHdg6axJhHWzrWSIQDAQhAAAIQgAAEIAABCEAAArElgIFTQNMuWLDAtGrVyrz44osF7P3rLl988YXZYYcdTL9+/bylPkf9zz5GfMstt0wNt1IvHf4gAAEIQAACEIAABCAAAQhAAAIQqBwBDJw8bG+55RbPvCn2iUuffvqp6datm5k6dapXwrRp00z37t2N1sfh79///rc3x4+dM0d14w8CEIAABCAAAQhAAAIQgAAEIACByhDAwMnB9eqrr/Ye+/3222/n2KvuppUrV5ouXbqYK6+8MmPjzJkzvfXaHpe/448/PtUTR2YOfxCAAAQgAAEIQAACEIAABCAAAQiETyCRV9xNmzbNS/L66683Xbt2NR9++GHgvhpOdeyxx9bZpp4pelrTddddV2ebVtx0002mU6dORvvF5W/JkiUpE0ePIucPAhCAAAQgAAEIQAACEIAABCAAgXAJJMLA6d+/v5FpY1/qKWLfazlo0KA6VPXEqIYNG2bsl35Mo0aN6hynp1Q1adLE/Pa3v/WO69y5c8Z5ZQjpHKuttpq3X7ETImeczLEPMqTscKpbb73VsegIBwIQgAAEIAABCEAAAhCAAAQgEG0CiTBwot1E0Yl+/vz5KRPn73//e3QCJ1IIQAACEIAABCAAAQhAAAIQgIDjBDBwHG+gqIV34okneibOzjvvbFatWhW18IkXAhCAAAQgAAEIQAACEIAABCDgJAEMHCebJbpBfffdd2annXbyTJzzzz8/uhUhcghAAAIQgAAEIAABCEAAAhCAgEMEMHAcaoy4hDJv3jzPwOnYsaP55ptv4lIt6gEBCEAAAhCAAAQgAAEIQAACEKgZAQycmqGPd8FbbLGFZ+Jcc8018a4otYMABCAAAQhWlyE6AAAgAElEQVRAAAIQgAAEIAABCFSBAAZOFSAnsYiJEyd6Bk6vXr2SWH3qDAEIQAACEIAABCAAAQhAAAIQCJUABk6oODmZJfDll196Bo4eLX7nnXfa1SwhAAEIQAACEIAABCAAAQhAAAIQKIEABk4J0DikMAIjRozwTJxx48YVdgB7QQACEIAABCAAAQhAAAIQgAAEIBBIAAMnEAsrwyAwffp0z8DZd999wzgd54AABCAAAQhAAAIQgAAEIAABCCSWAAZOYpu+8hWfP3++Z+B06dKl8oVRAgQgAAEIQAACEIAABCAAAQhAIMYEMHBi3Li1rtrTTz/tGThrrLFGrUOhfAhAAAIQgAAEIAABCEAAAhCAQKQJYOBEuvncDn7FihWegaOJjD/88EO3gyU6CEAAAhCAAAQgAAEIOEhg1apV5uWXXzZz5841etJrNV+TJ082CxYsMMuWLTNff/21g3QICQLJIoCBk6z2rmptv/vuu5SB88wzz1S1bAqDAAQgAAEIQAACEIBA1AnMmzfPbLLJJqmcWjdGa/Vq3bq10RyX/EEAArUjgIFTO/aJKNn+g1m0aFEi6kslIQABCEAAAhCAAAQgEAaBAQMGpMyaRo2bmt9vuLHp1HX7qr46dO5uWq7f2tSr3yAVy9ChQ8OoHueAAARKIICBUwI0DimcAAZO4azYEwIQgAAEIAABCEAAAiKgh4DYPProky8ycxf/p6avS+c9bXr03isV0+zZs2koCECgBgQwcGoAPUlF2n889MBJUqtTVwhAAAIQgAAEIACBUglcdNFFKaOk1saNv/xTLr45Fdvy5ctLrSLHQQACJRLAwCkRHIcVRgADpzBO7AUBCEAAAhCAAAQgAAFNFNyqVSvPJNn74FE17XXjN2/s590HHubFt99++9FgEIBAlQlg4FQZeNKKw8BJWotTXwhAAAIQgAAEIACBUgm88sornjnSqElTJ80bmThnX32PF6OMJv4gAIHqEsDAqS7vxJWGgZO4JqfCEIAABCAAAQhAAAIlErjjjjs8c2T91ps4a+DIxGnUuIkX5wcffFBiTTkMAhAohQAGTinUOKZgAhg4BaNiRwhAAAIQgAAEIACBhBOYOHGiZ4zoaVN2yJKLS8WnPJ95LhMuWKpfdQIYOFVHnqwCMXCS1d7UFgIQgAAEIAABCECgdAIYOKWz40gIJIEABk4SWrmGdcTAqSF8ioYABCAAAQhAAAIQiBQBDJxINRfBQqDqBDBwqo48WQVi4CSrvaktBCAAAQhAAAIQgEDpBDBwSmfHkRBIAgEMnCS0cg3riIFTQ/gUDQEIQAACEIAABCAQKQIYOJFqLoKFQNUJYOBUHXmyCsTASVZ7U1sIQAACEIAABCAAgdIJYOCUzo4jIZAEAhg4SWjlGtYRA6eG8CkaAhCAAAQgAAEIQCBSBDBwItVcBAuBqhPAwKk68mQViIGTrPamthCAAAQgAAEIQAACpRPAwCmdHUdCIAkEMHCS0Mo1rCMGTg3hUzQEIAABCEAAAhCAQKQIYOBkb67vv//evPrqq2bRokXe8qOPPjI///xz9gPYAoEYEsDAiWGjulQlDByXWoNYIAABCEAAAhCAAARcJoCBE9w6jz32mNloo43M73//e9OvXz/TrVs3s+GGG5rdd989+ADWQiCmBDBwYtqwrlQLA8eVliAOCEAAAhCAAAQgAAHXCWDg1G2hTz/91Ky99trmnnvuMT/99FPdHVgDgQQRwMBJUGPXoqoYOLWgTpkQgAAEIAABCEAAAlEkgIFTt9Xuvfde07dv37obWAOBBBLAwElgo1ezyhg41aRNWRCAAAQgAAEIQAACUSaAgVO39ebMmWNGjBhRd0MJa7799tuCjvriiy/MO++8U7UePx9++KF57733CorNv9OKFSvMV1995V/N55gSwMCJacO6Ui0MHFdagjggAAEIQAACEIAABFwngIHzSwtpqFSPHj1M+/btzXrrrWfWWGMN770+29eZZ55Zpzn/8Y9/mM033zy1z8cff+ztc+utt5qePXuaFi1amB133NE888wzdY7Virlz55p27dp5c+1sscUWpnnz5ubggw829jxBB+25556p8mxs6csbb7wx6DAjM+mMM87wytL+irtly5bm6KOPNjKQ/H///Oc/TadOnbyy3n33XXPxxRd7+3fo0ME0a9bMbL/99kbr+Ys3AQyceLdvzWuHgVPzJiAACEAAAhCAAAQgAIGIEMDA+bWhPv/8c/Pvf//bXH755eaggw7y3uuzfX3zzTe/7pz2buXKld4+mvRYvVOGDx9uunbt6j29SsbQU089ZYYNG5Z2xC9vR48e7Zk8y5YtS22TyTJp0iRvwuQ33ngjtT79jYyXxYsXp+Ky8c2cOdO0bdvWfPDBB+m7e+/VY2arrbYyKvOTTz5Jbdd6GVOKPeg41a1z586eWSMmmh9IfzJ8pk2bZtq0aWO+++671Pl4Ez8CGDjxa1OnaoSB41RzEAwEIAABCEAAAhCAgMMEMHDqNk6pQ6jUq6VLly5ej5Z8psYdd9zhPdlKjyoP+lMPnu7du5sffvihzuZevXrV6aHzwgsvmNatW5vXX3+9zv5accwxx5gTTzwxcJtWXnnllWbIkCGB27fZZhuvV1DQhM7a9uCDDwYex8p4EMDAiUc7OlsLDBxnm4bAIAABCEAAAhCAAAQcI4CBU7dByjFwzjvvvLonDFij4VrPPfdcwJZfV/Xp08c89NBDv674v3c///xzxjoNY1LPG/X0Cfr7+uuvvaFZWub609CooHlxZNK8+eabgYdOmDDBXHLJJYHbWBkPAhg48WhHZ2uBgeNs0xAYBCAAAQhAAAIQgIBjBDBw6jZIOQZO0DAkfwnqVaPHlP/444/+TRmfzz77bG84VcZK3wcNh9pss82MevRk+3v22We9uWuGDh1qcr023nhj88ADD9Q5TS4D56yzzsLAqUMsXiswcOLVns7VBgPHuSYhIAhAAAIQgAAEIAABRwlg4NRtmEobOJpLZ5111qlbsG/N5MmTjQySbH+rVq0y2223nZkxY0a2Xbz1mmOnY8eOZsGCBXlfxfbAwcDJiT4WGzFwYtGM7lYCA8fdtiEyCEAAAhCAAAQgAAG3CGDg1G2PShs4KlFPnHrrrbfqFp62Rr1l7rrrrrQ1v75V752BAweaU0899deVWd5pXz1VK+hJU1kOyVhND5wMHIn7gIGTuCavboUxcKrLm9IgAAEIQAACEIAABKJLAAOnbttVw8CZOnWqZ8DULf2XNU8//bTZZJNNzH//+9/AXUaNGmUOO+ww458PZ+nSpWb8+PF1jjnttNPMyJEj66wvZAUGTiGU4rsPBk5829aJmmHgONEMBAEBCEAAAhCAAAQgEAECGDh1G6kaBo6e6LTbbruZo446ynz55ZcZQWgeGj1R6tFHH81Ybz+cf/75pm/fviboCVaPPPJIoDGkfbfddlszduzYOj1xZALNnz/fbL311t4QK1uOXWLgWBLJXGLgJLPdq1ZrDJyqoaYgCMSOwMqVK82TTz4Z+ivosZvFwnvttddCjyvf0y8KjbESzN55551Ci2c/CEAAAhAogwAGzi/w9PhtTSzctGlT07BhQ1OvXj3vvT7rtfrqqwc+Lls9YVq0aOG9VlttNdOkSZOM43r27Jm1dfSocfWMWXfddU2/fv28HjV6DPmWW25pXnrppcDjvvrqK6PrnUaNGmWUY+PU+n333Tfw2G+//dacccYZXnl6wtWRRx5pdt11V7PBBht45s7tt9+ecdzcuXNNs2bNjOrVuHFjM3jw4IztO++8s6lfv75p0KCB2WeffTK28SE+BDBw4tOWTtYEA8fJZiEoCESCgAycBx98MPRXWAZO2LGFZeA8/vjjoTPDwInEV4YgIQCBGBDAwKl9I2qYlB4Brvlu9P/PPywq7Aj1OPEXXnjByLB5+eWXjSZV5g8C2Qhg4GQjw/pQCGDghIKRk0AgkQQwcEprdgyc0rhxFAQgAAEXCGDguNAKxAABdwlg4LjbNrGIDAMnFs1IJSBQEwIYOKVhx8ApjRtHQQACEHCBAAaOC61ADBBwlwAGjrttE4vIMHBi0YxUAgI1IYCBUxp2DJzSuHEUBCAAARcIYOC40ArEAAF3CWDguNs2sYgMAycWzUglIFATAhg4pWHHwCmNG0dBAAIQcIEABo4LrUAMEHCXAAaOu20Ti8gwcGLRjFQCAjUhgIFTGnYMnNK4cRQEIAABFwhMnjzZe6pRh87dzdzF/3H21XbTLbw4n332WRewEQMEEkMAAycxTV2bimLg1IY7pUIgDgQwcEprRQyc0rhxFAQgAAEXCCxYsMAzRlqu39pZ82b2X9/wYlSe/9lnn7mAjRggkBgCGDiJaeraVBQDpzbcKRUCcSCAgVNaK2LglMaNoyAAAQi4QGDZsmWeOVKvfgNz6bynnTRxTrroei/G1q1bu4CMGCCQKAIYOIlq7upXFgOn+swpEQJxIYCBU1pLYuCUxo2jIAABCLhA4OuvvzYyRpRD9+i9l5MGTvtOXb34Bg8e7AIyYoBAoghg4CSquatfWQyc6jOnRAjEhQAGTmktiYFTGjeOggAEIOAKgenTp3sGifLoUy6+2SkTZ8hR41Oxvf/++64gIw4IJIYABk5imro2FcXAqQ13SoVAHAhg4JTWihg4pXHjKAhAAAIuERg6dGjKKNl94GHm7KvvqZmRc+3Db5oTLphl1mvVNhXTokWLXMJFLBBIDAEMnMQ0dW0qioFTG+6UCoE4EMDAKa0VMXBK48ZREIAABFwjMHv27JRhopy6UeMmplPX7av62rhjl4wYFMesWbNcQ0U8EEgMAQycxDR1bSqKgVMb7pQKgTgQwMAprRUxcErjxlEQgAAEXCSwfPlys99++5lWrVrVMVJsnl2N5UYbbWQGDhxoNMkyfxCAQO0IYODUjn0iSrb/UOhmmYjmppIQCJUABk5pODFwSuPGURCAAARcJ/DBBx8Y5dTVfD3//PPm888/dx0N8UEgMQQwcBLT1LWpKAZObbhTKgTiQAADp7RWxMApjRtHQQACEIAABCAAAdcJYOC43kIRjw8DJ+INSPgQqCEBDJzS4GPglMaNoyAAAQhAAAIQgIDrBDBwXG+hiMeHgRPxBiR8CNSQAAZOafAxcErjxlEQgAAEIAABCEDAdQIYOK63UMTjw8CJeAMSPgRqSAADpzT4GDilceMoCEAAAhCAAAQg4DoBDBzXWyji8WHgRLwBCR8CNSSAgVMafAyc0rhxFAQgAAEIQAACEHCdAAaO6y0U8fgwcCLegIQPgRoSwMApDT4GTmncOAoCEIAABCAAAQi4TgADx/UWinh8GDgRb0DCh0ANCcjAeeutt0J//fTTT2XXqhJx6Zxh/FUitnfeeSeM0DgHBCAAAQhAAAIQgEAZBDBwyoDHofkJYODkZ8QeEIAABCAAAQhAAAIQgAAEIACBfAQwcPIRYntZBDBwysLHwRCAAAQgAAEIQAACEIAABCAAAY8ABg5CqCgBDJyK4uXkEIAABCAAAQhAAAIQgAAEIJAQAhg4CWnoWlUTA6dW5CkXAhCAAAQgAAEIQAACEIAABOJEAAMnTq3pYF0wcBxsFEKCAAQgAAEIQAACEIAABCAAgcgRwMCJXJNFK2AMnGi1F9FCAAIQgAAEIAABCEAAAhCAgJsEMHDcbJfYRIWBE5umpCIQgAAEIAABCEAAAhCAAAQgUEMCGDg1hJ+EojFwktDK1BECEIAABCAAAQhAAAIQgAAEKk0AA6fShBN+fgychAuA6kMAAhCAAAQgAAEIQAACEIBAKAQwcELByEmyEcDAyUaG9RCAAAQgAAEIQAACEIAABCAAgcIJYOAUzoo9SyCAgVMCNA6BAAQgAAEIQAACEIAABCAAAQj4CGDg+IDwMVwCGDjh8uRsEIAABCAAAQhAAAIQgAAEIJBMAhg4yWz3qtUaA6dqqCkIAhCAAAQgAAEIQAACEIAABGJMAAMnxo3rQtUwcFxoBWKAAAQgAAEIQAACEIAABCAAgagTwMCJegs6Hj8GjuMNRHgQgAAEIAABCEAAAhCAAAQgEAkCGDiRaKboBomBE922I3IIQAACEIAABCAAAQhAAAIQcIcABo47bRHLSDBwYtmsVAoCEIAABCAAAQhAAAIQgAAEqkwAA6fKwJNWHAZO0lqc+kIAAhCAAAQgAAEIQAACEIBAJQhg4FSCKudMEcDASaHgDQQgAAEIQAACEIAABCAAAQhAoGQCGDglo+PAQghg4BRCiX0gAAEIQAACEIAABCAAAQhAAAK5CWDg5ObD1jIJYOCUCZDDIQABCEAAAhCAAAQgAAEIQAACxhgMHGRQUQIYOBXFy8khAAEIQAACEIAABCAAAQhAICEEMHAS0tC1qiYGTq3IUy4EIAABCEAAAhCAAAQgAAEIxIkABk6cWtPBumDgONgohAQBCEAAAhCAAAQgAAEIQAACkSOAgRO5JotWwBg40WovooUABCAAAQhAAAIQgAAEIAABNwlg4LjZLrGJCgMnNk1JRSAAAQhAAAIQgAAEIAABCECghgQwcGoIPwlFY+AkoZWpIwQgAAEIQAACEIAABCAAAQhUmgAGTqUJJ/z8GDgJFwDVhwAEIAABCEAAAhCAAAQgAIFQCGDghIKRk2QjgIGTjQzrIQABCEAAAhCAAAQgAAEIQAAChRPAwCmcFXuWQAADpwRoHAIBCEAAAhCAAAQgAAEIQAACEPARwMDxAeFjuAQwcMLlydkgAAEIQAACEIAABCAAAQhAIJkEMHCS2e5VqzUGTtVQUxAEIAABCEAAAhCAAAQgAAEIxJgABk6MG9eFqmHguNAKxAABCEAAAhCAAAQgAAEIQAACUSeAgRP1FnQ8fgwcxxuI8CAAAQhAAAIQgAAEIAABCEAgEgQwcCLRTNENEgMnum1H5BCAAAQgAAEIQAACEIAABCDgDgEMHHfaIpaRYODEslmpFAQgAAEIQAACEIAABCAAAQhUmQAGTpWBJ604DJyktTj1hQAEIAABCEAAAhCAAAQgAIFKEMDAqQRVzpkigIGTQsEbCEAAAhCAAAQgAAEIQAACEIBAyQQwcEpGx4GFEMDAKYQS+0AAAhCAAAQgAAEIQAACEIAABHITwMDJzYetZRLAwCkTIIdDAAIQgAAEIAABCEAAAhCAAASMMRg4yKCiBDBwKoqXk0MAAhCAAAQgAAEIQAACEIBAQghg4CSkoWtVTQycWpGnXAhAAAIQgAAEIAABCEAAAhCIEwEMnDi1poN1wcBxsFEICQIQgAAEIAABCEAAAhCAAAQiRwADJ3JNFq2AMXCi1V5ECwEIQAACEIAABCAAAQhAAAJuEsDAcbNdYhMVBk5smpKKQAACEIAABCAAAQhAAAIQgEANCWDg1BB+EorGwElCK1NHCEAAAhCAAAQgAAEIQAACEKg0AQycShNO+PkxcBIuAKoPAQhAAAIQgAAEIAABCEAAAqEQwMAJBSMnyUYAAycbGdZDAAIQgAAEIAABCEAAAhCAAAQKJ4CBUzgr9iyBAAZOCdA4BAIQgAAEIAABCEAAAhCAAAQg4COAgeMDwsdwCSTFwPnyyy/N22+/bRYvXmwWLVrECwZoAA2gATSABtBAVg28/vrr5pNPPgk36eJsEIAABCAQewIYOLFv4tpWMI4GzjvvvGNOPPFE06NHD9OmTRvTqFEjY+vJ8jew+A0M+B6gATSABtBAYRpo0KCBad26tenWrZsZMWKEefzxx2ubuFE6BCAAAQg4TQADx+nmiX5wNoFTr5So/915551m4MCBWQ2KZms2N+06dTWdum7PCwZoAA2gATSABtBAVg20WHeDrPmEco358+dHPW0ifghAAAIQqAABDJwKQOWUvxKIg4EzZ84cs/3222ckWq032cwMOWq8GX3GpebPV95lrr7vFTN38X94wQANoAE0gAbQABooSAPX/e0tc+END5vjz7vGHDz6dLPFNjtm5BrKPW644YZfkyreQQACEIBA4glg4CReApUFEHUDZ9KkSalkqkHDRqZXvyFm/AWzC0rMMHQwtNAAGkADaAANoIFiNDDxqrtN/wNHmhbrtUrlH2PHjq1sssbZIQABCEAgMgQwcCLTVNEMNMoGziGHHJJKnvoMGmYun/8sxg13VdEAGkADaAANoIGKa0C9cw485pRUHqLeOPxBAAIQgAAEMHDQQEUJRNHA+cc//pExMfGIEydVPFEr5u4c+3I3Fw2gATSABtBAMjRwzClTUiZO/fr1zSuvvFLRvI2TQwACEICA2wQwcNxun8hHFzUDR0+YsjFrefbV92DecKcVDaABNIAG0AAaqJkGpt32lFmz+Tqp/GTlypWRzw+pAAQgAAEIlEYAA6c0bhxVIAFrhkTlKVRrr712KkG67m9v1yxZ485qMu6s0s60MxpAA2gADRSqgY5deng5ih47zh8EIAABCCSTAAZOMtu9arWOkoGzxx57pMybc/9yH+YNd1vRABpAA2gADaABpzTQfJ31vFxl3LhxVcvlKAgCEIAABNwhgIHjTlvEMpKoGDinn356yrzR48ELvRvGftw5RQNoAA2gATSABqqlgUvnPZ3KV6ZNmxbL3JFKQQACEIBAdgIYONnZsCUEAlEwcO6///5UMtR3v+GYN9xtRQNoAA2gATSABpzVwAkXzErlLffdd18I2RqngAAEIACBqBDAwIlKS0U0zigYOEcccYSXCHXdoY+zyVq17uxRDneR0QAaQANoAA24rwH1FlaOdcABB0Q0QyRsCEAAAhAohQAGTinUOKZgAq4bOCtWrDBNmjb1kqDjzr4KA4c7rmgADaABNIAG0IDzGjh/9oOpXjjPPvtswXkZO0IAAhCAQLQJYOBEu/2cj951A2fy5MleArR+602cT9a4I+r+HVHaiDZCA2gADaCBamlg862383KYMWPGOJ8PEiAEIAABCIRDAAMnHI6cJQsB1w2cHj1+eSTnYceehYHDHVc0gAbQABpAA2ggMhoY/qdzPANn9dVXN++++26WTIzVEIAABCAQJwIYOHFqTQfr4rKB88gjj6S6H8966LXIJGzVurNHOdxFRgNoAA2gATTgrgauX/h2Ko+ZOHGig1kgIUEAAhCAQNgEMHDCJsr5Mgi4bODMnj3bS3y6bLsz5g13XNEAGkADaAANoIHIaWDr7Xbzcpnhw4dn5F98gAAEIACBeBLAwIlnuzpTK5cNHN2tUnw79z8gcgkbd0TdvSNK29A2aAANoAE0UC0N7HXA0V4u07t3b2dyPwKBAAQgAIHKEcDAqRxbzmyMl1TIJFm0aJFzPOzjw/c9fCwGDndd0QAaQANoAA2ggchp4Kjxk7xcq02bNs7lWQQEAQhAAALhE8DACZ8pZ0wj4HIPnJ122slLeo44/rzIJWzVurNHOdxFRgNoAA2gATTgrgbOuOy21M2yb775Ji0D4y0EIAABCMSRAAZOHFvVoTq5bOBssMEGXtJz/Hl/wcDhrisaQANoAA2gATQQOQ3MuOuFlIHzxhtvOJQBEgoEIAABCFSCAAZOJahyzhQBlw0cG9uZ0+dHLmHjbqi7d0NpG9oGDaABNIAGqqkBm8+4OFw9lRDyBgIQgAAEQiGAgRMKRk6SjYDLSYWNDQOHRLuaiTZloTc0gAbQABoIUwM2n8HAyZaNsh4CEIBAfAhg4MSnLZ2sictJhY0NA4dEOsxEmnOhJzSABtAAGqimBmw+g4HjZCpMUBCAAARCJYCBEypOTuYn4HJSYWPDwCHRrmaiTVnoDQ2gATSABsLUgM1nMHD8WSifIQABCMSPAAZO/NrUqRq5nFTY2DBwSKTDTKQ5F3pCA2gADaCBamrA5jMYOE6lwAQDAQhAoCIEMHAqgpWTWgIuJxU2NgwcEu1qJtqUhd7QABpAA2ggTA3YfAYDx2afLCEAAQjElwAGTnzb1omauZxU2NgwcEikw0ykORd6QgNoAA2ggWpqwOYzGDhOpL4EAQEIQKCiBDBwKoqXk7ucVNjYMHBItKuZaFMWekMDaAANoIEwNWDzGQwc8m4IQAAC8SeAgRP/Nq5pDV1OKmxsGDgk0mEm0pwLPaEBNIAG0EA1NWDzGQycmqa8FA4BCECgKgQwcKqCObmFuJxU2NgwcEi0q5loUxZ6QwNoAA2ggTA1YPMZDJzk5tvUHAIQSA4BDJzktHVNaupyUmFjw8AhkQ4zkeZc6AkNoAE0gAaqqQGbz2Dg1CTVpVAIQAACVSWAgVNV3MkrzOWkwsaGgUOiXc1EOw5l3fLU++aWJ/9t4lCXqNRhzpP/MtcvfAfmi/m9iopmFefNj79nrnngVe9142MrnNSvvltRYhoUq81nMHCSl2dTYwhAIHkEMHCS1+ZVrbHLSYWNDQOHC6KghJh12XWxc/8DzIHHnBL5i54otPHpl95q2m2+talXv77Rb1bDRo3Nuuu3MZ279zLT73iONsDQcVoDW2+/m6dbaXfkyZOdjLV9p65m/KRrnYyt0N8om89g4FQ1xaUwCEAAAjUhgIFTE+zJKdTlpMLG5qqBs2WP3ua3q60W+PpdvXpmgzbtzK57H2yuvOdlpxNPxbdp525FxXjOzHu9C9ag+jdo2MhMmbOoqPMVmgSzX3bTxrKZcPnt5ne/q2cum/9MYBusuXbLQM0GtaXWXXb704HnseUleXna1FtM4ybNzH7DjzMXXPugue5vb3m8zr76/7d3JlBXFGf6j8qmyKZGRcUFIoIEVAQXxA1ZBBRwQ9wAQWUXCRgRNxQRxAUxKgqKomI0bokLJpLEqCEmMRtJXIhJjMk/48zJmTMzJ5NhJpkz9T9PZepOf/1139t3r+7+cc53+t6+vbz1q6ebt56uqv6aufr2x8zjb/wWdk02cCbNvLYsvV929e25qzP11ht4wghvDZypVy0ze3x231T3cHP5DAZOfvJrSgoBCOSXAAZOfuu+Ibibc6gAACAASURBVCX3Oalwsflq4Ki7+UOvbDP9Bg4x0xausJ/1XX9rX/6puWXdy2b4hEtMx05dzJ2bvuNto2DV498ynbvtVXZ8D3/9fVvWu5560z7BVZlV9ke+8UHZx8qzCVDrsvc6/CijHjhxx+3cdU+zdO1XW+h14uXXmEEnnt5inepy146dzD1fwcCJY9nniGPNmRfNjmUdtx/rSxuRtWKk4YQPvfpzc86lC8yQ4RNaadzds7WUiTF90cpc1ufRQ0d6a+DICO26x97mgllLUls3Lp/BwGlIastJIAABCDSVAAZOU/Fn/+Q+JxUuNl8NHNfAGHDsKWbOjV+KTSyVdKoLuNvet2WlBo4rh+ZPUF0x50rjGqWOfXh568Ov2rpQnYZ/c99l4Mh0c9+1nDz/ZjNk+PgW67R+t90xcIKcwp87ddnD3P74N1txC2/H9+ZfGxfOvt6cMvb8onV17KljMXCa3GMq7lo577KrzV777J/a/2dcPoOBk/28mhJCAAIQwMBBA3Ul4HNS4WJLu4GjYRU777yL2bBle9HGg4aqLFzxiJky/xZzzZ1PJBp6pckd1fPF/YVNFDXk1dtCT6HDibGeamo/DblRQ9QdI7jUU+nwfuHv1Ro4illxzrp+jbl47k1m/i0Plpw75Mm3PmkRrybiDMf1xHd+V9jmwZd/FslA+zz2rY+MhoRd9sXbzcwlq81tj37dJJnM85FvfFg4/obXP2xxfn1ffNeTdmLQcFz1/H7a+EvMQYf2axFL+Hy1MHBqUfZK6t3VqXpUqFzStYbrqd5kopYz54zqWNfZtIW3mUu/sNzWl6vHKD05jpqo2F0ju+3e2dx0/wuF7259kqFT6zf/0mj+HA0P0XWfJPZKr3ddx4pN5Vr/2ntm7k33mQXL11vtu3Itf2SzmXHtXfZ3ty5uKf5LVn/Zxn7VsnXmri+/FXt9BY+h/eIY3f/VHxsNSUvCLnjMpJ+TGDjHnzYusheKmCnujd/+jdWd9G/Lv+BWW//lxKxjLFv/ij3PFdfcYTTczk1+XUx3rpzSifQi3Ug/0pH7LclShqN6jN5433MtWCfpgdPo/6OC5dGQ0J122slcc8cTZZU3eIxmfnb5DAZOXVNaDg4BCEDACwIYOF5UQ3aD8DmpcLGl3cCROaGGnkyHqARSc9AMOnGUHa7S98jjbDf/Q/sNNJpL5rRxFxc1fs68cJZNapXYipdr2C64dZ3Zc+/97PCtQ3r3N5/t3sMsfeDFFucfcMzJLfbVMcJ/Mp7UQIuK262r1MBRY/yk08+1Map7vOIZOuoc03/QiXZekfGXzIs978Vzb7SxqsyK+cRR57ba9rhhZ/7fNjvvbHuZuJjdUqbR7p27mv0PPtQcc/IYM3DIcLPXvgfYJ71qoLntwks1iDvs2rFw/BFnTbHbioWGaWgOJBkpmm9G8yDF1X34uNV8VyNQQ54mXv7F2Lh1fA2XU4M7eK64Hjga/heeA6fasldT7+olpPrWfBgawnfUkNNsmdUL7pDDBthrQPVeirfqXablAQf3tvV+5HHDbH2qvCeMOMseN8gn+LlHrz42BsXh9Be+bjS0KrhP8LNiO3faQnt9H9irr5FpoDmo2nfYzSh2aSi4ffBzJde7DFFdx4pV88FIkzqf5ujS+WVanTT6PHsdHHHsqfbak7EQPK/7LKNi/MVzbew9+xxhe231O/oEu6/KvPqZrZH7aX/NCaS5mRy3+csestvKUND9rm3bdranovQpQ82ds1bLKANHRozMGXcOmbcye913LRWfYhM/cdG9VT3TpB0NO9Q1p3mQwvfX4DHcZ+2re92+BxxsBp802v4dcMhhdt3sG9aYHj37mNXPfLfF+d2+0sVxp55hddK7/yCrG2lR/0+cO31RSc3LtNF1o7ilM/UK1f9Lus/KCC1m4DTr/yhXdrfUNa7/K933NC2lH/1h4GQ3n6ZkEIAABBwBDBxHgmVdCPicVLjY0mzgqNeNjAk14qOSTc0vokbjGRfMNGpMBLfRE2k1WJXsqxdP8Leoz5pwdu1LPzUjz55q1MtCDTf3RHfWdffYRD1qPxlMlcyB445VqYGjHhhXLL7TNpDCPYRUdjU0ir0V5epVG80++x9UtOGi8suQ0ZNqF69bnn7edNuIvX7NV1r8pljmLb3fNnSTzIehrv0ycNT467bXPtbAcb0pZHaooRY2QVwMtVyqQaxrRo3EYseVFpwu3HZxBs6Ma+82xV7hW0nZq613NajVYNabnjT/TDC+u59+285JpevJlS28VENbhob0E/xNxop+k6F3WP/BLX4Lbhf8rG3Lmd9KsfY98nhroCjW4LE0d5TmLlIDX/oP/hb1udzrfd8DDimYEDqe6kGmi+4VMsBc7xL1gpGZFO6FJuNNpo3MrrBRo32lBZkZug6i4g2uk+kjvcpI07k0B5O7/6knkOaiCW5fi89RBo6Mj3EXzUl0Ll3HMrz23u+gFsPmZGqpB5jKHjZ/gnHLtJORrt43wfX6LN1peJCuX02GHf79/hd/ZHWhIWCOk9tGwyFlgB1+1PEtrgX3u5a6j3bv0bPVufX/hepeBrjqNep+68P/Ua4sMvhlAqrXpFuXlqXLZzBw6pLKclAIQAACXhHAwPGqOrIXjM9JhYstDQaOzAA1nN2fEmENZ1EjUz0y1PgJJ5pqzCnxPmfaF1r95raVmSATZ9Q5l8Zu47ZVg05PZvVkVd393fpSy2YZOKXimnPDvUZv+iq2nZ7IypCI20YGUdQbtr646nH7NFoNo7h91ZBSbwVnxsRtp4brQZ873D6V1zCsuO3qvV5x6JqRwVHuueIMnFLHqUfZS9W7yqdynjVlfmQ5ZYyop0HYFHRlUS8UzXXivoeXqsNSunP7lGvg6NXueuV4sR5Co86dZhvV7hxxy3Kvdxk44XuNerqIVXDicXFTuVY90XIeJfX+Ue+4sPkXjE+9PA7u/fmi5dP2MnB0Xer60mTvwWPU67MMHJ1Xw4/c3zGnjDWjz7ss0fll4IhV3P1A9/m461/3EjEtdl/WUEDpOsrA0UOAMedfHhun6kQ9LSfNWNxqGxlyMuhlJkexVX3LVFTPqLCB48v/US5uGW5idMeT344si9vOx6Xi1h8GTvbyaEoEAQhAIEwAAydMhO81JeBzUuFiS4OBo+EIeprt/pRM64msyqAu31GNavUK2WPv7iUnZVQX/7bt2hcdWqGEVQ06NUzdk/SkSayvBo7mkFFjsFg5NCxDPZSCvTDc9mok62m5juPWuaV6WIQbK+634FI9OdSgDq4Lf5aJoSEVUb18wtvW87t6eUlvbj6Ncs5VjYFT67KXqnddS+qlUqx8GsKmYR9R26gHRNc997bzLkX9Li0VG8YU3KccA0c9WjQUp1SPHV2/Gv6oOUqC5wp/Lvd6l4ETNks0b5BMmfCxdV9a+djrhfXqZaKeD3HDe4L7q5eO5tcJrgt/lpGioUNxZkh4+1p8l4Ejw8jdo7UU53IMnGI98mTODRt3UWS5Zf7IOCxVDr210A2Dddvq/z/13Ck1z44MN+kr3HNKvXY0t5E7XtRSPap07wjfE335P8rFrLmbFKcMeLcuLUvFrT8MnJqmsBwMAhCAgJcEMHC8rJbsBOVzUuFiS4OBE/cWKiXj6imgYQLhOVX0dirNeaNhKqX+1P09SYMunPwnSW4bYeBEGSyKTY0NNYo074hMFQ2b0pN5Pc3WXykDR8ewvXCuX9MqodfwDPV2CDPQ0BHVxyXzlpbkriE6GpoQPkbwuwwcNwdOcH2jP2v+Eg2HqeS81Rg4lZS9mnqXgaOeEMXKqYZssWFIGvKiOYzUc2royLPtnDS6hst9ZXo5Bo6GBqkhXixu95uGimieJ/c9aikDp5zrvRoDR6aXzI9S9yn9rrlUNFwoKma3zg2hct8bsYwaQqX5X8oxcOLmBlL8MmhOPfPCyHKrJ+aKR78R+Vupsl805wY7tK7Udvpdxlu4B4+GZn3phR+WPLe0GTZwfPk/ypVdEz4rL7js6ub1dHSxlLt0+QwGTnbyZ0oCAQhAII4ABk4cGdbXhIDPSYWLLc0GjkvyNJQnPK+GendoDpcjjx+W6E9d4d3xopblNujcMRph4Gho18qNW1rE7xriehKvRoqe2uuNPnozkIYiqCGYxMD5ey+cQ1r0wpFJI9NLv7lyuqXmE9JwARkeSdhrgle3b9TSFwPHzQ9RbIhLVPxa10gDp9p6d/vHlUXrSxk42kY9lTT/kYb9jJ54ue2JoglhNR+IJkgudnz3WzkGzrylD0T2dnHHCi5lgIw4a3LRGMq93qsxcNTjQRPeJrletE0p88kXA+e6Nc8kfquRetFUYuDoetT9Jjx3TbC+i33WUFzNEVRsG/ebzGbNfea+S8fqORU3nNBtp6XqJGzg+PJ/lItTQ9+UF5SaWN9t79PS5TMYODVJXTkIBCAAAa8JYOB4XT3pD87npMLFlgUDR93fNaQqOGRAc31oAslaJZnlNujceRth4Ox30Oda9CBS41nDF2TcxDUuSg2lcfFrqSFreouLW6dJiIuZPzLOwsNJ3L7lLn0xcNTTS9dMkmEu4TI2ysCpRb3XysAJM9B3zQWj+aw0WW3U7+F15Rg46oGh3hDhY0R9V++zyVcuLbptudd7NQaO5orSxNFRc3lFxV9qnS8GTjhOmbu6l4TX63ulBo72DQ9Jizp+3Dr1FNTb+uJ+D67X0MDg0Df9pt4/9z73/ZL7S5thA8eX/6NcGdWLSvc49WZz69KydPkMBk7682ZKAAEIQKAUAQycUoT4vSoCPicVLrasGDjtO+xq3xLlEk7NO6An/sUmtnTbJlmW26Bzx2yGgaPhZHrNtoshalmOgbNo5aO2x42GaskQ0lCsL9z2cOzx1SBSQz3qvOWu88XAUS8vXTPX3PFE2eVqlIFTi3qvxsCRPs6eelVJPjI7kgx5KcfAkfmhe0C4J1pYbzIR9DazpWu/WjTOcq/3agwcxai3uRW7psLlKPbdVwNH8yap91ZU7NUYOHrdvYyYqOOWWqc3VMkAKvXmJb39S/oK98DTsK5ic/fo/G4C5bCB48v/UY7RaeMutve49Zt/WRFLd5xmLF0+g4FTVcrKzhCAAARSQQADJxXVlN4gfU4qXGxZMHAunnuTnbQznDiOv3iunQen2FtptI8aFqWGdZTboHOx6NjFJn1128Utk7xGPNwDZ+GKDfbJcNzcOHd9+S37JLxYL5pwPHpyrifG6l7fo1ef2J492k+vz9V8MVETHAePKzNIjZvguvBnXwwc9ZLQ67FlxoRjLPW9UQZOLeq9GgNHZqnuK8UmEpbRokmSa23gqA7Uq0a9e8ITzQbrRw3uQSeeXrIOy73eqzVwdK1oHpxib25TOdTLas1z7xSNP28GjkwYDUErNReN5hCKeuOgJsLXUKqgToKf1cOzR88+kb229P9npy57xE7qrXuw3lwoPYUNHJ3Dh/+jXFk1ROywAcfEcnDb+bh0+QwGTnrzZSKHAAQgkJQABk5SUmxXEQGfkwoXm68Gjhp6ehKoxogmVdTn4J+MjevvfdZOkqkGYfi1vEoydQyblPYfbPQENZx4qiGkxF1PVqPMBu2vt9boTwm4jI9gDHqDVfiYUd9leJwydlKreRpkdKjbuuaw0TmC+2pOB51Lr21WXWlC1RbnDvBQ4zE4CbPikoGiYwePu+H1Dy1LDddQoq64dMxiDV4Xk3rh6G1gaiAH54Fwv4eX2l4NG709Jnx8GWoahqWePCp7eF8ZO467enSIXbjswXKF96/Xdz3pTzosT2V2MYuBehi478We9ldT9mrrXebAjfc9b98ipVijjE/71ra27ez1FjYIpWdpVcNKonq43Pfiu2bgCSPsNa15lKLqacOW7QVOHTt1Mbc89FLhu+MnRlH7ar3miNHkyeGeOLp+9FprTSYb98rnSq93mb/ujWzBHhqauFmvkA7Pz6IeH7rvhvlq0l/1xJm/7KFW5dMx9BprXVMTJl/Z6ncdy10zOqfejOR4uWX4fFEMy10n5jq+5hXSZOnuXFFL9yan4Dnc/rq2ZMDpHhX8XZ9VLt0HNH+SelCFf9d3TYiuYaNRPeR0Xv0/IJNH8/KE95cexP24U89o9YY0DZnSPW/gkOGxprXe3iRd6f+j4LE1tEr3Wc235MoX/j/Dl/+jNLxRvaM0AXmwDGn57PIZDJyKUlV2ggAEIJAqAhg4qaqu9AXrc1LhYvPVwNHrd12McUtN0Dvm/MuNuqLHJZpqKE6Zf4udY0JJ9pHHDbO9cjp328saN2pY3/rwq632t42Ndu1LxjD1qmWt9g3HIhNGjQBNeKnX68qU0pwIbdq2tROvhhsdarRq27hyR60PGjg6vyYsVnk1jEwsZZbIhNLTYL1JRYaVO44avOGYo76rF87+Bx9a8tXsbl/VixqTeiOR3lilxrviUENBryfXW1iiXt+rNxe52OKWqr96NEhd7FFLmVLqhZNkWJ7KGxe7DDQ1mKLOUW3ZK6131YMMk2DMMjyCMWqYSvD3kWdPbfG7jCmZiZokW2+y2r1LN3utqd6lHWlRr2OPK7savNomeI6oz5rbKRhX8LOudxkBMmVlqqjhbK+9Nm3sW4xkEAW3d58rvd5ltmgSXRenJgd3x5SBo/XqgRe8x8jA0frjhp1Z2Nbto2Fwum6lb107mgxc33WvOLTfQHPlzWtb7aMyy7xwMcQtjzl5TKt93XkrXertUHHni1ofHkJ14azrWuwvVkGzXYaYTCt3LOkjrqeN2Mi41lw1/QefZLUnTWt+tHEXzSl63co4kjbFWXqRbqQf6Uh6ijMcHTe9lU/alxGkfdVjR/p3EyRLFyqDtBIecuXD/1HqHaT7UjFz2ZXVx6XTBwZO+vJkIoYABCBQLgEMnHKJsX1ZBHxOKlxsvho4tU4S1RDQ8BA9LdVQE73SuFRSXssY9KRZxowmA9YwpDue/HbNJi2Ni1O9QFS/eqqqhqF6LcVtm2S9Xikufkm2DW7z4Ms/M0vuedrGIQZRT9mD2/v6WT1O1FA+Z9oXymbQyDLVut4riV0xaJiUtC79qedDEuOrknNF7aNeGzJNZl13j30bVrjnQ9Q+vqzTvWLNs98zeiuQrjfdt9RTw5f4fI5DnNx9Xj0FZQapZ1nSmKUT9aSRbqSfcnv6qVenjDtpv9z/X5r5f5TmTBs+IX4YWVJ+zdrO5TMYOGWlqGwMAQhAIJUEMHBSWW3pCdrnpMLFlhcDp1mJZVbOq15CesIcHjaTlfIlLYeGpmgoRnhYTNL92e6PiRvTsIIVGqi/BtSzUD2bSs2/5HNduHwGAyc9+TGRQgACEKiUAAZOpeTYLxEBn5MKFxsGTv0TZJ8T36SxaeiVnkon3T6r2+mpeq/DjzIXz70x9yyyWseUi3tinjSgeXrcUK+0ltvlMxg4iVJTNoIABCCQagIYOKmuPv+D9zmpcLFh4NBYCSft6mWj+Ug0HEB/GoqgN+Ssfua7VQ/DCp8rjd81JEwT8qYxdmLmekcDaCCoAU3UXO5wr+D+Pnx2+QwGjv95MRFCAAIQqJYABk61BNm/KAGfkwoXGwYOyXw4AdfrdnfbvZOd/FMTgAb/NEln8E074X35jp7QABpAA2igkRpw+QwGTtGUlB8hAAEIZIIABk4mqtHfQvicVLjYMHBItBuZaHMu9IYG0AAaQAO11IDLZzBw/M2HiQwCEIBArQhg4NSKJMeJJOBzUuFiw8Ahka5lIs2x0BMaQANoAA00UgMun8HAiUxFWQkBCEAgUwQwcDJVnf4VxuekwsWGgUOi3chEm3OhNzSABtAAGqilBlw+g4HjXx5MRBCAAARqTQADp9ZEOV4LAj4nFS42DBwS6Vom0hwLPaEBNIAG0EAjNeDyGQycFikoXyAAAQhkkgAGTiar1Z9C+ZxUuNgwcEi0G5locy70hgbQABpAA7XUgMtnMHD8yX+JBAIQgEC9CGDg1Issx7UEfE4qXGwYOCTStUykORZ6QgNoAA2ggUZqwOUzGDgk3xCAAASyTwADJ/t13NQS+pxUuNgwcEi0G5locy70hgbQABpAA7XUgMtnMHCamvJycghAAAINIYCB0xDM+T2Jz0mFiw0Dh0S6lok0x0JPaAANoAE00EgNuHwGAye/+TYlhwAE8kMAAyc/dd2UkvqcVLjYMHBItBuZaHMu9IYG0AAaQAO11IDLZzBwmpLqclIIQAACDSWAgdNQ3Pk7mc9JhYsNA4dEupaJNMdCT2gADaABNNBIDbh8BgMnf3k2JYYABPJHAAMnf3Xe0BL7nFTsuuuuRvFde/dTppGJFucisUcDaAANoAE0gAZqoYGN3/61zWWUz7zzzjsNzfE4GQQgAAEINJ4ABk7jmefqjD4bOP369bNJz8zrVmPgbCWRrkUizTHQERpAA2gADTRSAysfe71g4Hz66ae5yjEpLAQgAIE8EsDAyWOtN7DMPhs448aNs0nPpBmLMXAwcNAAGkADaAANoIHUaWD+LQ/aXKZz584NzO44FQQgAAEINIsABk6zyOfkvD4bOAsWLLBJz6hzLk1dwtbIp3uci6fJaAANoAE0gAb81MDEy662ucyAAQNykllSTAhAAAL5JoCBk+/6r3vpfTZwVq9ebZOeY04Zg4HDU1c0gAbQABpAA2ggdRo4cdQ5NpcZP3583XM6TgABCEAAAs0ngIHT/DrIdAQ+GzjPP/+8TXq69+iZuoSNJ6F+PgmlXqgXNIAG0AAaaKQG9juwl81l5s+fn+l8ksJBAAIQgMDfCWDgoIS6EvDZwPn4449Nx44dbeLDq8RJuBuZcHMu9IYG0AAaQAPVamDpAy/aHEa51gsvvFDXfI6DQwACEICAHwQwcPyoh8xG4bOBI+hTp061yc+QERPohUPXeTSABtAAGkADaCA1Ghh25oU2hxk6dGhm80gKBgEIQAACLQlg4LTkwbcaE/DdwHn11VcLT6/ufOrN1CRt1T61Y3+e/KIBNIAG0AAaSK8GNr31e9N1z71tDrNu3boaZ28cDgIQgAAEfCWAgeNrzWQkLt8NHGEePHiwTYAmTJ6HgcOTVzSABtAAGkADaMB7Dcy96X6buxx22GFmx44dGckaKQYEIAABCJQigIFTihC/V0UgDQbOqlWrbBLUpk1bs/iuJ71P2nhimt4nptQddYcG0AAaQAPVauDRb35kevcfZHOX5cuXV5WnsTMEIAABCKSLAAZOuuorddGmwcD585//bI499libCO3x2e5m/WvvYeLw9BUNoAE0gAbQABrwUgPDJ0y2Ocvw4cPpfZO6zJiAIQABCFRHAAOnOn7sXYJAGgwcFWHr1q1ml112sQnRIb37e5mwVfvEjv156osG0AAaQANoIN0amDL/FpurtGnTxmzfvr1EFsbPEIAABCCQNQIYOFmrUc/KkxYDR9jWrl1rkyLFPHTk2Zg4PHlFA2gADaABNIAGvNHA1bc/VshTtmzZ4lnGRzgQgAAEINAIAhg4jaCc43OkycBRNc2cObOQHJ1+3nRvkjaemKb7iSn1R/2hATSABtBANRqYctWyQn6ycuXKHGeWFB0CEIBAvglg4OS7/ute+rQZOAIyZsyYQpI08IQRRpMFVpN0sS9JOxpAA2gADaABNFCpBoYMn1DIS0aOHFn33I0TQAACEICAvwQwcPytm0xElkYDR+AffvjhQrJ0wCG9zc0Pfg0Th270aAANoAE0gAbQQMM0sObZd8znjx5ayEcWLVqUidyQQkAAAhCAQOUEMHAqZ8eeCQik1cBR0d5++23TvXt3mzi1a9/BnHHBTHPLQy81LHGr9Ekd+/GUFw2gATSABtBAejVwz1e2mgtmLTE9evaxOUiXLl3M448/niDrYhMIQAACEMg6AQycrNdwk8vXrVs3m3y8/PLLTY6kstN/8sknZuzYsYWnXzKk+g860UxftNI89OrPMXN4EosG0AAaQANoAA3URAMLbl1nNFyqTdu2hbxDrwrftm1bZUkMe0EAAhCAQOYIYOBkrkr9KtCgQYNsErJ+/Xq/Aiszmk2bNrUyctq2a2/2P+hQc+Rxw8yIs6aYyVfebK658wlzw5ee4w8GaAANoAE0gAbQQKwGrlh8l5kweZ45/rRxpmefI0zHTl0Lpo0eFsm42bhxY5nZCptDAAIQgEDWCWDgZL2Gm1y+888/3yYky5Yta3IktTm9hlXNmjXLqDuzGx7G8jOw+AwMuA7QABpAA2igeg1MmjTJbN68uTZJC0eBAAQgAIHMEcDAyVyV+lWgxYsX28b97Nmz/Qqsymg0tOrZZ581q1atMirb6NGjTd++fU2HDh0wMzAz0AAaQANoAA2ggaIa6NWrl9EbpWbOnGlWrFhh1NP33XffrTI7YXcIQAACEMg6AQycrNdwk8v3wAMP2ATm7LPPbnIknB4CEIBA4wnMmDHDTJ48ufEn5owQgAAEIAABCEAAApkjgIGTuSr1q0CvvPKKNXB69+7tV2BEAwEIQKABBLp27WratWtn/vKXvzTgbJwCAhCAAAQgAAEIQCDLBDBwsly7HpRtx44dRuaNxsW/8cYbHkRECBCAAAQaQ+CZZ54pDKHYsGFDY07KWSAAAQhAAAIQgAAEMksAAyezVetPwZYuXWobMRdccIE/QREJBCAAgToT0NBRN6mr5rrgHwQgAAEIQAACEIAABKohgIFTDT32TURg+/bthcl933///UT7sBEEIACBNBP405/+VDBvnInz29/+Ns1FInYIQAACEIAABCAAgSYTwMBpcgXk5fTTpk2zjZklS5bkpciUEwIQyDGB++67r5WBc9ttt+WYCEWHAAQgAAEIQAACEKiWAAZOtQTZPxGBLVu22MaMJvN87bXXEu3DRhCAAATSSuCkk05qZeAcccQRaS0OcUMAAhCAAAQgAAEIeEAAA8eDSshLCK4XzgEHHGD++Z//OS/FppwQgEDOCHzwwQetzBs3jOqdd97JR+xkpQAAIABJREFUGQ2KCwEIQAACEIAABCBQKwIYOLUiyXFKEvjb3/5WeCPVwIEDS27PBhCAAATSSMBN3O5Mm+By/vz5aSwSMUMAAhCAAAQgAAEIeEAAA8eDSshTCHqVuGvMXHTRRXkqOmWFAARyQqBv376F+5y737nlvvvumxMKFBMCEIAABCAAAQhAoNYEMHBqTZTjlSSwbNmyQuNmxowZJbdnAwhAAAJpIfDWW28V7m/OtAkvX3zxxbQUhzghAAEIQAACEIAABDwigIHjUWXkKZRRo0YVGjmDBw82n3zySZ6KT1khAIGMEpgzZ07h3hY2btz3SZMmZbT0FAsCEIAABCAAAQhAoJ4EMHDqSZdjFyWwcuXKQkOnTZs25plnnim6PT9CAAIQ8J3AXnvtVbivOcMmavmv//qvvheF+CAAAQhAAAIQgAAEPCOAgeNZheQtHL1eXOaNa+BMnz7daJ4c/kEAAhBIG4Hnn3++cC9z97S45YMPPpi24hEvBCAAAQhAAAIQgECTCWDgNLkCOL0x27dvN8OHD2/R8Bk7dqx5+umnwQMBCEAgNQTOP//8FvexOPNG60855ZTUlItAIQABCEAAAhCAAAT8IICB40c95D6KHTt2mFWrVhnNhxNs9AwaNMicd955Zt68eWb58uVm3bp1hglAcy8XAEDAOwL/8i//0uLeFbyPxX3+8MMPvSsHAUEAAhCAAAQgAAEI+EsAA8ffusltZBpCpaFUHTt2jG0Q5RYOBYcABLwk8NBDD8Xer+IMnJtvvtnLshAUBCAAAQhAAAIQgICfBDBw/KwXovpfAps3bzb33nuv7YEzevRo87nPfc42kgAEAQhAwCcCw4YNK9vA6du3r09FIBYIQAACEIAABCAAAc8JYOB4XkGEBwEIQAACfhP46KOPyjZvXK+cN9980+/CER0EIAABCEAAAhCAgDcEMHC8qQoCgQAEIACBNBK49dZbKzZwZs2alcYiEzMEIAABCEAAAhCAQBMIYOA0ATqnhAAEIACB7BD45JNPjPv7/e9/b/T3hz/8wfzqV78qGDtvv/22+eMf/1j4+4d/+Aejv08//TQ7ICgJBCAAAQhAAAIQgEBdCWDg1BUvB4cABCAAgbwS+Mtf/lIwcGTm8A8CEIAABCAAAQhAAALVEMDAqYYe+0IAAhCAAARiCPzHf/xHwcDZvn17zFashgAEIAABCEAAAhCAQDICGDjJOLEVBCAAAQhAoCwCGDhl4WJjCEAAAhCAAAQgAIESBDBwSgDiZwhAAAIQgEAlBDBwKqHGPhCAAAQgAAEIQAACcQQwcOLIsB4CEIAABCBQBYEdO3YUhlB9+OGHVRyJXSEAAQhAAAIQgAAEIGAMBg4qgAAEIAABCNSBAAZOHaBySAhAAAIQgAAEIJBjAhg4Oa58ig4BCEAAAvUj8J//+Z+FHjgffPBB/U7EkSEAAQhAAAIQgAAEckEAAycX1UwhIQABCECg0QQwcBpNnPNBAAIQgAAEIACBbBPAwMl2/VI6CEAAAhBoEoGggfP+++83KQpOCwEIQAACEIAABCCQFQIYOFmpScoBAQhAAAJeEfiv//qvwhAqDByvqoZgIAABCEAAAhCAQCoJYOCkstoIGgIQgAAEfCcQNHDee+8938MlPghAAAIQgAAEIAABzwlg4HheQYQHAQhAAALpJICBk856I2oIQAACEIAABCDgKwEMHF9rhrggAAEIQCDVBP76178WhlDRAyfVVUnwEIAABCAAAQhAwAsCGDheVANBQAACEIBA1ggEDZxf/vKXWSse5YEABCAAAQhAAAIQaDABDJwGA+d0EIAABCCQDwJ/+9vfCj1wMHDyUeeUEgIQgAAEIAABCNSTAAZOPelybAhAAAIQyC2BoIHzi1/8IrccKDgEIAABCEAAAhCAQG0IYODUhiNHgQAEIAABCLQggIHTAgdfIAABCEAAAhCAAASqJICBUyVAdocABCAAAQhEEfjv//7vwhCqn//851GbsA4CEIAABCAAAQhAAAKJCWDgJEbFhhCAAAQgAIHkBDBwkrNiSwhAAAIQgAAEIACB0gQwcEozYgsItCLwySefmGeffdasWrXKzJ4924wePdr07dvXdOjQofDE/TOf+QyfYYAG0AAaQANoAA200kCvXr3MyJEjzcyZM82KFSvMpk2bzLvvvtsq32AFBCAAAQhAIEgAAydIg88QKEHg7bffNrNmzTJdunRplYxh2GBYoQE0gAbQABpAA9VoYNKkSWbz5s0lshF+hgAEIACBvBLAwMlrzVPusgjoydjYsWNbmDZt27U3+x90qDnyuGFmxFlTzOQrbzbX3PmEueFLz/EHAzSABtAAGkADaCBWA1csvstMmDzPHH/aONOzzxGmY6euLXKM4cOHm40bN5aVq7AxBCAAAQhknwAGTvbrmBJWQUBDpcLGTf9BJ5rpi1aah179ufny1j/yBwM0gAbQABpAA2igag0suHWdGTJ8gmnTtm3BzJGRs23btioyGXaFAAQgAIEsEcDAyVJtUpaaEtBwqe7du9skql37DuaMC2aaWx56qeoEDdMH0wsNoAE0gAbQABqI08A9X9lqLpi1xPTo2cfmIF26djWPP/54TXMcDgYBCEAAAukkgIGTznoj6joTePjhhwtPvw44pLe5+cGvYdzwdBUNoAE0gAbQABpomAbWPPuO+fzRQwv5yKJFi+qc/XB4CEAAAhDwnQAGju81RHwNJzBmzJhCsjTwhBHm0W9+1LBkLe5pHOt5UosG0AAaQANoIJ8a0LAqNzGy3lzFPwhAAAIQyC8BDJz81j0ljyCg13m6JOn086Zj3PCkFQ2gATSABtAAGmi6BqZctayQn6xcuTIig2EVBCAAAQjkgQAGTh5qmTImIrB27dpCcjR05NlNT9Z40prPJ63UO/WOBtAAGkADURq4+vbHCnnKli1bEuU2bAQBCEAAAtkigIGTrfqkNBUS2Lp1q9lll11sYnRI7/6YNzxtRQNoAA2gATSABrzTwJT5t9hcpU2bNmb79u0VZj3sBgEIQAACaSWAgZPWmiPumhH485//bI499libEO3x2e5m/WvveZewRT2JYx1PaNEAGkADaAAN5E8DwydMtjmLXjG+Y8eOmuVDHAgCEIAABPwngIHjfx0RYZ0JrFq16n+fZrU1i+96EvOGJ65oAA2gATSABtCAtxrQyxV69x9kc5fly5fXOUvi8BCAAAQg4BMBDByfaoNYmkJg8ODBNgmaMHmet8kaT1jz94SVOqfO0QAaQANoIE4Dc2+63+Yuhx12GL1wmpI9clIIQAACzSGAgdMc7pzVEwKvvvqqTYD05qk7n3oTA4cnrmgADaABNIAG0ID3Gtj01u9N1z33tjnMunXrPMmqCAMCEIAABOpNAAOn3oQ5vtcEpk6dapOfISMmeJ+sxT2FYz1PaNEAGkADaAAN5E8Dw8680OYwQ4cO9TrXIjgIQAACEKgdAQyc2rHkSCkj8PHHH5uOHTva5OeGLz2HgcMTVzSABtAAGkADaCA1Glj6wIs2h1Ev4hdeeCFlWRjhQgACEIBAJQQwcCqhxj6ZIPD888/bxKd7j56pSdZ4wpq/J6zUOXWOBtAAGkADcRrY78BeNpeZP39+JnIzCgEBCEAAAsUJYOAU58OvGSawevVqm/Qcc8oYDByeuKIBNIAG0AAaQAOp08CJo86xucz48eMznLFRNAhAAAIQcAQwcBwJlrkjsGDBApv0jDrn0tQlbHFP4ljPU1o0gAbQABpAA/nRwMTLrra5zIABA3KXx1FgCEAAAnkkgIGTx1qnzJbAuHHjbNIzacZiDByeuqIBNIAG0AAaQAOp08D8Wx60uUznzp3J7iAAAQhAIAcEMHByUMkUMZpAv379bNIz87rVqUvYeLqan6er1DV1jQbQABpAA3EaWPnY6zaX0UTGn376aXTCw1oIQAACEMgMAQyczFQlBSmXwK677mqTnmvvfgoDh6euaAANoAE0gAbQQOo0sPHbvy4YOO+88065qRDbQwACEIBAyghg4KSswgi3dgT0tEp/vEKcJ5txTzZZjzbQABpAA2jAdw24fOaNN96oXZLEkSAAAQhAwEsCGDheVgtBNYKAS3gwcEjOfU/OiQ+NogE0gAbQQJwGXD6DgdOI7JFzQAACEGguAQyc5vLn7E0k4BIeDByS4rikmPVoAw2gATSABnzXgMtnMHCamFRyaghAAAINIoCB0yDQnMY/Ai7hwcAhOfc9OSc+NIoG0AAaQANxGnD5DAaOf7kmEUEAAhCoNQEMnFoT5XipIeASHgwckuK4pJj1aAMNoAE0gAZ814DLZzBwUpOCEigEIACBiglg4FSMjh3TTsAlPBg4JOe+J+fEh0bRABpAA2ggTgMun8HASXtmSvwQgAAEShPAwCnNiC0ySsAlPBg4JMVxSTHr0QYaQANoAA34rgGXz2DgZDRhpVgQgAAEAgQwcAIw+JgvAi7hwcAhOfc9OSc+NIoG0AAaQANxGnD5DAZOvvJYSgsBCOSTAAZOPuudUhtjXMKDgUNSHJcUsx5toAE0gAbQgO8acPkMBg7pLQQgAIHsE8DAyX4dU8IYAi7hwcAhOfc9OSc+NIoG0AAaQANxGnD5DAZOTMLHaghAAAIZIoCBk6HKpCjlEXAJDwYOSXFcUsx6tIEGGqeBO578thlz/uUG5o1jDutssHb5DAZOeXkgW0MAAhBIIwEMnDTWGjHXhIBLeDBwspHAJm2IPPX2H8y6zb+wf4996yMvG4tPvvlJIcbHv/Nx2TE+9q1fF/ZXeUuxWXLP0+aUsZPM2pd/WnLbUsfK+++b3v59LhlWq1np5rhhZ5ozL5qdeX73vfiuvd5uvO+5ppT10i8sz5VR9vgbvy3cD3XvX//ae2VxT8M17fIZDJyapIccBAIQgIDXBDBwvK4egqsnAZfwYODky8AZf8m8wvxHp583vaxEvlHmxJHHDyvEePkXV5UVowyfjp26FPZf/cx3S+4/ef7Ndvu7vvxWyW0bxSCN55FxdsAhh5mbH/xa7jhWo1nV9T1f+Z5p32FXs/al7JuIKzdusdfbZVff3hSdDDrxdNN1j72bcu5mXNf7H3xo4X6o//c7d92zrLJ/7vCjzMIVG8rap9HldPkMBk49s0aODQEIQMAPAhg4ftQDUTSBgEt4mmXgTLz8GrPTzjtH/u288y5mz733MwOHDDe3Pvyq14mjElUluA++/LPEcU6aeW1kueN41Lqho14pZ02Zb3w1cMRUMQ48YYQp18Cx+373/9n9d+3YyWDgNM6gPGfaF8y+Bxxiwk/sZy5ZbXRNB/Xd54hjW1wvun523W33Ftt02LWjuf+rP26xXaMbhuWcrxrNDp9wiTn1zAtjy3rIYQNasAmy7NxtL3P4UcebWdfdY5767v+LPUY5ZanntpUaOKPPu6wmRkLeDBzVpbSpvzs2vVG2gTP1qmVmj8/ua2TQ1lMX1Rzb5TMYOE1IJjklBCAAgQYTwMBpMHBO5w8Bl/A0y8BRQ+OhV39uzppypRk66hzz0CvbWvzd/vg3zYWzrze77d7ZzLnhXm8TRyWdu3fuau7c9J3EMbqyn3PpAjNk+IQW5Q5zkIkxfdHKxMdOmgSfPfUqrw0clePooSMrMnAcg6QGjoYUqP6e+M7vas7ZxZL1pa7l9h12syZCVFml4b5HHl/QetTQuA2vf2h/f+BrPzE77bRTKoe0VaJZmVfqfXP302/H6k/DYHRvOOjQfmbe0gcKHLVO+2ndfgf2MuoJFDbQouqjmetU97reHvnGB7HljYrvmJPHmGkLV5S1T9Rx7n/xR4mM3ah9077urqfeLNvAkfbUY+mCWUuqZl8vfi6fwcDxJ8ckEghAAAL1IoCBUy+yHNd7Ai7haZaB4xI59UYp9uR56dqvmt1272TWb/6lt8ljuQaOK7sMqlPGnl+0XMeeOhYDZ2tlvUiSGjiuPlhWxlncpOUue3zWPPnWJ5F61nwnGsqRhLHmSNE1n2Rb37apxMDRsMZjThmbqLy9+h5pFt/1ZOS2amirp855l10d+btvrMqNp1YGTrnnzdL2lRg4Kr80tdc++9tePD7ycPkMBo73qScBQgACEKiaAAZO1Qg5QFoJuITHdwNHyaIaLV9c9XjRRol6ACxZ/WWj7t5XLVtnNJ9JkuEE2k+T1+pPDaBgcqrhG9fe/VSr9dpG3dHdfppz5cb7ni98d+u1jGvQ6hhJDJzjTxvXoheKjqfjuiFbmjtDw4x0fvfkXT1JFt+1ycxf9lBsr5JwD5wNW7aba+58wkxbeJu56f4XTNIJjhWPekvNufFLhX0f+caHLTgGmUZ91v56sq5GfrAOkjSGNXmsNKweHhqaoXpx5yhm4DiOSevKHfPRb/7K8ndlFOtl61+x8V+9amPiCUKlzVVPfMv2WJl1/Rqz5rl3CnG7c5TSj3peiPvFc28yc2+6z9aD4hQTF2+jlur9MXpi/BuU1jz7PTtEKhiPGMqglBaD62956CVzwMG9W6wL/q7PMnQ1+bSu94UrHjFfev4HRbfXPro+gvUd1Ip+X/X4t4wM4yT3jWo0GyyLrjv1Mkw6VLSYgaPj6hpSXQTPEf4sVmImdmJYjjmu3jPuPqHJgGUmqeeUzlFKd+otFOSfdEiOJt7VfkcNOc32Agkew30udSz97rbVUrGEuZT6Lv3omlWPUA1rvWXdy2bjt39T9Djq3afzue2kefv/1IJb7X02eL8rdn5pVfrUvULX+/xbHkyk+fAxKzVwdH9Sr7hr7niiaHnD52vUd5fPYOCkNSMlbghAAALJCWDgJGfFlhkj4BIe3w0cJa6adPG2Da9FJo5KgMdfPNe0a9/B9OxzhBkyfLzpd/QJdliT5tlY/czWyP2UWF59+2Nml13a2MRUPGR4aL0aZ4f2G2jatm1n57fRHBNqrAST0fOvuMbup6RW+2oZ9VdsnpkoA0cJfvAtIbc9+vWCWaPzjzrnUnueNm3bminzbzGaJ+SIY0+15VVPJjVMDu79ebPP/geZwwYcYxS7jhGMXZ+dgSMD4qTR59keD+KlngB773eQ2b1LN2uEhfcLfpdxoUlru+21j43huFPPMD169bFDaTQxcLiBHNxXn2XaaG4FNdb1Bh7NJaTGrHokqBFdzMBRY+q0cRebTl32sPtpf8Wi7wuWr7flLWbgnDb+khb1pTpc8eg3WnEKxqzyqKzadujIs21jrnuPnrYMmldDT6ilgclXLi16HA3h+PygobbOBhxzsi2nOBz0ucONep+obpymRp49tdWx1OBWnWvOGB1Hw/D2O+hzdh991+ekhkCwfJV+lumheJc+8GKrWN0x1fAXG2d8af0XbnvYznWl4UPBhqyuQ3Fx+waXMt7OnbbQXu8H9uprZHAe+vmjreakATX2g9sHP5954axCnStembf6fcGt62wcMmIP6d3ffLZ7j9iyVKPZYCzus+4Bul+576WWpQwc6adtu/aRJpTY6BrVULfe/QdZdrpede88d/qiomaz4pJ5oOtL16t6wxx53DB7/xG3E0acZQ2WuPhlMNp5kP73Pin+l8wrfp3oWDKmd2nzf/dod12E77XSQty5tV6mVnAf3RuKbR/+Tf8n6PrU/0XSpq73vbsfaO+v0k94e33XPvo/RDFfcc0dVmfqWSZ+2l8x6Boudt3Y+/Pp59qJ2TWMSefWkOP+g060++peGXXuuHWVGjg6nnp3DTpxVFnni4uj1uvFWH8YOBlLVCkOBCAAgQgCGDgRUFiVDwIu4fHZwFEjfsLkK82+Bxwc2SBRcivTRg2JsFGjJ57q9q0kOcrACCeQakSp4ahGiho4mmTZNTaXP7LZTqgb3sd9r+UQKjWkxl00p2iSrDlCVH9K/p3poPkkZOaogaHJZF1sSvCV7LvvbimTQMZNv4FDbEPu4a+/32IbNa7VWLtw1nUt1rv91ctJDTf1/nE9f9xvMg/UEFbj2q0LL7WfzA+ZQMHf9BaeAceeYk4cda6t16hJjKUL/a7twm/t0fHUWFNPomIGTvCc+qxJsx3L8G/h7zLOpEnx0ZNw97viUu+lrnvuHdtjTD2nZKppQtagaaF9ZfzIeFPjuJjxp/pUQy5cZ2Ihw0faUM8KF1e9lyPOmmLPWaonh4ZYqQeDi0cmmrSuxndwWJAa9nqtu9vOLaUzzaMjwyY8X4z0f/KYiXaujiQTH2sSYMdL14zq0fUg0WTAUQ3jajTryhBc6v4lQ1A9MoLri30uZeBcsfhOa9yGjyHTUAaAhmy6+5rbRo16GdaaCDl8LbttZDLIgFEvM7dOSxlq+k33wMP6D27xW3C78Gddv0kMnOB+ui5qMQeO7Q1WhoGjHke6l1w898ZWprR6pKgOw73IgnEPPmm01biMcZk67jdd/5rgW8d2PSrdb24p01h1qv10j3DrtZTOZQZF3SOD2wU/V2PgyDjSA4+kvTOD5633Z5fPYODkI3+llBCAQL4JYODku/5zXXqX8Phg4KjXgJ5iuj91UT/jgpmm1+FH2cQ3rkGmJ+pqzLqGV1SSqIRfPVLU0Ij63a2TgaOGoRqZ6hrv1idZVmPg6LwazuD+1ANGjfti53WTvF6/5istttNwFA0zCO6rYUXqTRNcp8+ul4d6LYR/c99laMjMuuPJb7fYRo1BmRDFGp4yF9QTJOrptIalaf+4YQxqqKihrifmUY0TTXyt38MNGhe3jitzRRpP8hYq7VeugaNeAWrYuXMGl5d98XbbKya4zn1WY049h9z38FLGk+KOM3BUZvWYCNeJO456uqgHSSNf5S0Ny8xzMcQtZbYGjZq99j3ADn+TiRMs79hJV0Q2iNXrTQZGsWt51LnTrLEXF4NbLwNHvVDU60tDXNz6uGW1mo06rhrmMjqjfotbV8zA0fAp3YvU2yW8vwy/MefHD3HTPVSxTJqxuNW+OpbmKtP9JXxc912a7z/4pNjf3XZumRYDR6akjK9ir9G+97nv256DcT1pdM2rh13cMD/1oBM/x6acpf6vLId7NQaOHi7o3hR37ykn7lpv6/IZDJxcp7UUHgIQyAkBDJycVDTFbE3AJTw+GDhKkDV8x/3pabCMFMWobuuLVj7aKrnV00s9DUzSQFfD0Q2riUsc1QjVEJy4JDtuP62vxsBROV25tZSRkMTAUdf8cEwaQqWeQ8H1miMnakJYGTjqsVNsyImOIyNNPXWCx5SBoiETwXVRnzVvkYZohH9TL4AZ197Van1wO/WoUv1HGTh6kq1yBbcPf1bjWPsn0Yf2LdfAKda7SOaJXqcdjklGgepN856Ef3PfZdBost+goeF+c0sNn1JjPM64lIEVZ265Y9RyqV5DMltLHVM9KFx9ao4q9SBTGTRMJshLw3zCDVoZU2JX6m1v6nmnupSZUSweGTgyJLR9se3cb9Vq1h3HLdWzQj3Fgj243G/FljJwNAeK5gnSn0wDmdQyAdSTI6rHnO7xMvWCPb6izqHeUWIs1uHf1bNNPcs0D0v4N31Xz51S95LgfmkxcNSbsdi17so0dcGttgeT+x5cysAp9iZBGZPDxl0UyTV4nKjPMkT1gCLqt6h11Rg4MgZ1Ty01H13Ueeu9TnHpDwOnda7HGghAAAJZI4CBk7UapTyJCbiExwcDJ+4tVGrozlt6vzUgJl7+xRZJqhoUMj9mXHt3yT/NpaJ5M4olkTJw3Bw4xbaL+q0aAyf8FioN3WiUgaM5cqLKE1ynngcy0YLrZCCoF0kp9mqEq7dI2GjQXDFfeuGHLY4ZPL77rEana/C7dZpMU3NPuO9xSxlx0ni9DBzNgRN3bvXg0hCr8O9JG1saClTMwJERFJxDR9tqWJd6ZCU1JMKxVfNdc67IdCl1DOnaDTVRvG4+DfWokcnoTDn1hAs3EjWMUXoodQ79rqEeGu5SbFsZOG4OnGLbud+q0aw7RnApQ1nGV9yQpeC2wc8ycNSbQ8MP9adebrp3ydBTz7jgtu7zRXNusMPL3Pdiyz327h4735iG+8j01f1A+tc9VZNou3ordtzwb2kxcNSjUUPqwvGHv4uBDLQo41QGjubACe/jvpd6E6MMNRlAMs01TE3DpnSNyLTTX6MMHBnTuqdqAmcXuy9Ll89g4CROAdkQAhCAQGoJYOCktuoIvFoCLuHx2cBxyaHmu1DDIdhzQQ08TXh75PHDEv2VatD5YuBct+aZkm/60BCqWvTACfescbyDSzVMxDm4Tj2a1OMiKfvgEBUNrVLPqaiGTvAc+qw6CRs4mptH5kV42/B3HT9pDy3tW24PnEoMHA0nS2KaaR6bYgaOK6uuiytvXmt7Xalxpwa9ekmE5ylx29drqUa/hrSVOr6uQWdYSjvB+Uw0dFC9GHQM1YWG/gWPN2/pA5FzOQW3cZ9lLIw4a3KL/d1vblmOgVOtZt05g0tdP5U0hIsNoQoeP/hZcw2Fe+YFfw9+1jw4xXoF6W1OMgpVd3rrmIawyqSVIROekyl43PDntBg46iWlSbrD8Ye/y4jTHEFRw0KrMXA0942d6P2Qw4yMOBl/mmdLwzdlkOvhRKMMHA3zVd6g+c/C5W/2d5fPYOBUmxmyPwQgAAH/CWDg+F9HRFgnAi7hSYOBo+RQ4/w1wbBLFPW0WUMwNBGoW1fN0hcDJ1wGvVI6PE9GrQwczQESPl/4u3qNhIfHnHrGBXZOjPC2Sb+rB43mjSi1vXo9hA0c9dzR3Cml9lVPHWncpx44Ghamnial3s6lp+xJDJwoBmrcyVApNWQwat9K18mU0kSupfaXMaAGv3oU6M1TwVena0idTB2xkfEWHo6j+Zikh1Ln0O/qlVDqTWDlGDg6ZjWaDcd8/b3P2t6DpYY0hffT90oMHE0WfNLp5yZiJwNw5WOvJ9rWxacJpKXXUm+CcttrmRYDRwaJehoGY4/6LDM1bHS77So1cGSWycyUcRNneCft1ediqWYIld6YpnuqesO54/mydPnGj/7pAAAfVElEQVQMBk6dEkYOCwEIQMAjAhg4HlUGoTSWgEt40mLg6G1JGk4VTBjVkFePjOC6Sj/7auBEmTVR61TucufAUSM62Dsmip2GvYR7M6hBo6e+pYyIqOO5OIvNCaFt7tj0hm0shA0c/ab5UkrNhaIeAr4ZOGqEaU4gDR+KY6PeDZq8Oc7A0VtvSk1QrLkq1PCMO0et12vIkliXejuN5mxRjwa9IUu9hYJxSNPSo3p8aUhW8Dd9llGr38M9c8LbyfCUmVSq10S5Bo6urWo0G4xTb0/TsJnguqSfKzFwNE+OTL1S9aO39YlxeMijepe4oW/F4tR1mfRNbmkxcDR0V6/8LlZu/aZ5iWRORm1XqYGjSeIPOrRf5DHdeRpp4GjYrK7zUm+bc7E1cunyGQycxuaRnA0CEIBAMwhg4DSDOuf0goBLeNJg4GhOETXqwq9aVfKqeXDi5n5wCaSeZAaf9rv1wWU1Bo4aLpXMn6Mnmm5ISTCW4OcosyZqnfYp18CRUeDmIQme033WPEN6u9B9L77bohEh40a9RKJetez2dUs98Q3P8yHN6S1RKofbLrjU9no7kBrZUQaOGlWa6Dp8XHcMvbVMcat8PvXAUXxqJKsXjoYjuHjdUq9f1wTNGqIWZ+BoOEyxN4fpWFOvWtZQA0fGqu4npcwV1YveaqaJsaPKp7lV9KYbMXBMgkv1qlEvj6hJdt12ugaSNLjLNXCq1ayLT71bNFdK+FXe7vdSy0oMHB1T17m0E3d89Qbq0bNPZM8lmbyq32KmqQw2TUZfTwNHE/3qlfVxZUi6vpzXiG94/UPb80tD+OKO//feN51ie6ZUauDozVfq+RV3n9NE4Oqd2aghVBpel2QIaByneq53+QwGjhfpJUFAAAIQqCsBDJy64uXgPhNwCU+zDBz1RtCTPD3Z1dNYfQ7/qbGj7uNqjMcl0DIR1BMnykBRI0mvxZVZMGHyla0ScE2eqklf9ac5PDSMIxxDsVcWu4RUry7Wk9Kw0SHjaPYNa+yTWfU+cNu7smuuDs1dEj5n8Lt7M4zbV8m81rVp29buF+xar8aryqnzuu3d5JrrX3vPrnPnViNaQ6E0bGXgkOGt5m7Q62nFXROXumMFl2oEqXu/JtyN6sWjHhCa7FgTPIeNNx1HvUQ0Ka2GkwSPq6FVaiSo148mEFWj3cUe3G74hEvs27vCQ7F0PBkAGm6nhrJMETXCgvuqToOM9VnzTGjf4PqoHgtOU3qDUVRc6r2w+K5NtuEVNyeI5q+Q8Sj20qfKKB2ItyaN1jURZXCoDHoKLmNKPaOk22C5ZKzpOpFBpLmUgr/V87MMFb2qvlSvEmlXw6OkG/UkCsdkr+V99jd9jzyu1W/aVtoVMxk9YbNIExJrHh1pKmoeEu0vk8Fd7zJw1AAO1ndUfQZjrFazOpaGd5150ezI8gXPFf4sxopVjXUNRQvGnaQnnJjoPqnJpsPD03SflTGm+0DwfuJiWPvS3w0cmQlRPZt03xt4wgg7Z1VcLGIbjFkmpIzY4LpSppaGBcrI1zXtYtNS51RvFV1DUUPnVOfB89y24TU7/Da4Tp/jhuPqvqBrSvfr8P8HqovOXfeMfPuXu9e6+1j4PqTYFZv7P1C9x4LlEjN77NnXt7jWdRzNn6QhxLpXqlef4o8yNvV/QbCc+n9I12pwnT6He10F49BnDZPTvGtx/x+Et2/0d5fPYOD4nHUSGwQgAIHaEMDAqQ1HjpJCAi7haZaBo0k1XQxxSw2F0ES74UZ+ODlU8q6GW+due1kjRsaBvsvkUE8NTfQa3kdJvxqSced26485eUyrfcPHUhKup/7aR0+x1cjUUBElvGoYaXLWYOKvhq47fpJlcMLiUedc2mJfdd138cjA0fFkWLmEXAaOO8e0hbcZvQLcfZdJoLhkhqiBoqe5YidzQfON6PXO7thRSzUk9FRf8ekV7Bq2o94jalgoBsVarHeUTBb1XlLDUo0csVPjwk24quMpVhkW4eErahyNnTTDPvXXftpfx7HmwJ1/Nwdk4Gh/6UCTgboyqKHvGBRbav/gxNkauiQDIriPeru442qpng7ud8UdNyGsDAfpQgbY319RfXeBVTEDR41IGQBqAO/Spo19C5E46em4rhfpbtHKR1vEFIyvXp/1FiS9HafU8aUr6SVoMrp9NH+P2Kkh7taFl7pu1UjXUB8Zdap3XWNiIf0H6yu4rzVC27Uv1I2ro/AyXJ/BY+hzNZqV6am4ZYiEj1vqu67NcKzue1JDSNerTFtdD2ImdmKomMQ0znyRkanrVJPmyujUNSqTTaaN4tIkxjquGvlR5RBTF2uxpSYB1hDCqGO4ddK/9COty3CSca4J7hXXmRfOamWqyjTU/wvFzut+0zBdd57wUvNXqdehzqW5wzQnm46r+43+/wlvr+96pbs7tpbSqHrguW11j9Z90m0jjuG38+ma0P9l+k1DtHSNyXxUD0UZUeqF6vYPvy1QhpTqyv1ebKlhfS6uqKV6Quq+HmVqR23f6HWubBg4KUxGCRkCEIBAmQQwcMoExubZIeASnmYZOLVO8NSgVwNJQ1P0pFwN9rgnqrU+tzuehgupJ5B63egpcfiJqtvOx6VY6em6et6osRvXbT8qdjWa9WRXT2c1v0lUj5yo/dw6DW/T64g1/CKuEem2DS9V72rIaBhPqWFy4X19/W5fJT/x8qINKsWuIUlq8KrnmHoniEO5/GrFQNqXYdWo+4lMU11jesWzGJTqPVOrcrrjVKJZDf/RMCB3jGYtxUqmuNiJoVgmjUW9PHSd6k1Eut7V06vc6z3pueK20/n0pjXp/sb7novtcRW3fzXr1dvo6tsfs/cbDSkr5z5Z6XnFXNeVvb+u/nKrHlSVHrec/WSUyegvZ59GbuvyGQyc7OSolAQCEIBAHAEMnDgyrM88AZfwNKrB1chkjnP90dtEm7opXTfq2aDGadpYaShksV4MaStPLeOV0aiebUnnZKrluTlW6WsORvGM1KNPPYCK9aZsNj+Xz2DgZD51pYAQgAAEDAYOIsgtAZfwYODEJ67NTko5f/7qRvPjqLGUxt5EauDpVcrqCYR2W2s3TT3yqL/W9ZdXJppnxw1r9ZWBy2cwcHKb0lJwCEAgRwQwcHJU2RS1JQGX8GDgkKj7mpRnOS7NPaT5ifSnoX8aynLeZVfbiZc1tCWtZdewkqj5bdJaHuLm/ph3DWiS62YNzUzK3uUzGDgt8zy+QQACEMgiAQycLNYqZUpEwCU8GDg0UJImyWxXO63Yt8h07GQNG02WrElYNZFo8G1l8K4db1jCEg1kVwMun8HASZT+sREEIACBVBPAwEl19RF8NQRcwoOBk92klgYLdYsG0AAaQANZ14DLZzBwqskK2RcCEIBAOghg4KSjnoiyDgRcwoOBQ3Kf9eSe8qFxNIAG0EB2NeDyGQycOiSLHBICEICAZwQwcDyrEMJpHAGX8GDgZDeppcFC3aIBNIAG0EDWNeDyGQycxuWQnAkCEIBAswhg4DSLPOdtOgGX8GDgkNxnPbmnfGgcDaABNJBdDbh8BgOn6aklAUAAAhCoOwEMnLoj5gS+EnAJDwZOdpNaGizULRpAA2gADWRdAy6fwcDxNeMkLghAAAK1I4CBUzuWHCllBFzCg4FDcp/15J7yoXE0gAbQQHY14PIZDJyUJaKECwEIQKACAhg4FUBjl2wQcAkPBk52k1oaLNQtGkADaAANZF0DLp/BwMlGfkopIAABCBQjgIFTjA6/ZZqAS3gwcEjus57cUz40jgbQABrIrgZcPoOBk+m0lcJBAAIQsAQwcBBCbgm4hAcDJ7tJLQ0W6hYNoAE0gAayrgGXz2Dg5DalpeAQgECOCGDg5KiyKWpLAi7hwcAhuc96ck/50DgaQANoILsacPkMBk7LPI9vEIAABLJIAAMni7VKmRIR2G+//YySngW3rjckttlNbKlb6hYNoAE0gAayqoH7XnzX5jLKZz744INE+Q8bQQACEIBAeglg4KS37oi8SgInnniiTXqmLrgVA2cryX1Wk3vKhbbRABpAA9nVwHVrnikYODt27KgyM2J3CEAAAhDwnQAGju81RHx1IzB16lSb9Iy/ZB4GDgYOGkADaAANoAE0kDoNTFu4wuYyBx54YN3yJQ4MAQhAAAL+EMDA8acuiKTBBJYuXWqTnpPHTExdwsbT1Ow+TaVuqVs0gAbQABpIqoHREy+zucxJJ53U4CyK00EAAhCAQDMIYOA0gzrn9ILAI488YpOeAcecjIHDU1c0gAbQABpAA2ggdRo48rhhNpeZMmWKF7kVQUAAAhCAQH0JYODUly9H95jAli1bbNKjif8e/vp7qUvakj6dYzue5KIBNIAG0AAayJ4GHvvWR4U8Rr2K+QcBCEAAAtkngIGT/TqmhEUIDB482CY/F8+9EQOHJ69oAA2gATSABtBAajQw5apbbA7TqVMn85vf/KZItsNPEIAABCCQFQIYOFmpScpREYFVq1bZ5Kd7j56pSdh4ipq9p6jUKXWKBtAAGkAD5Wqg75HH2Rxmzpw5FeVA7AQBCEAAAukjgIGTvjoj4hoS+Pjjj03Hjh1tAnTlzWsxcXjyigbQABpAA2gADXivgeWPvGZzFw0D//73v1/DzIhDQQACEICAzwQwcHyuHWJrCAH3OvGjhgz3PmEr9+kc2/NEFw2gATSABtBA9jRwzrSF1sCZOHFiQ3IlTgIBCEAAAn4QwMDxox6IookEXn311cJTrBFnTcHE4ckrGkADaAANoAE04K0GvnDbw4W85ZVXXmliBsWpIQABCECg0QQwcBpNnPN5SWDJkiWFZEhPtXhamb2nldQpdYoG0AAaQANp18A9X/leIV9ZvXq1lzkVQUEAAhCAQP0IYODUjy1HThmBkSNHFpKiZetfwcTh6SsaQANoAA2gATTglQa67bWPzVXmz5+fsiyLcCEAAQhAoBYEMHBqQZFjZIbAHnvsUTBxHv3mR14lbWl/akj8PPlGA2gADaABNFC5Bg4bMNjmKD169MhM3kVBIAABCECgPAIYOOXxYuuME/j1r39dMHD0ZoebH/waJg5PX9EAGkADaAANoIGmaWD1M981XbrtVchP/vEf/zHj2RjFgwAEIACBOAIYOHFkWJ9bAr/97W9Nhw4dConS9EUrmpa08aSy8ieVsIMdGkADaAANpF0DV1xzRyEfadu2rdm2bVtu8zMKDgEIQAACxmDgoAIIxBC48MILC0nT8AmTzb3PfR8jhyewaAANoAE0gAbQQN018Og3f2XOv+KaQh5y/PHHx2QrrIYABCAAgTwRwMDJU21T1rIJrFixopA8tWvfwQwddY5ZeNsjdU/c0v7EkPh56o0G0AAaQANooHwNLF37VTPm/MvNnvvsX8g/5s2bV3b+wg4QgAAEIJBNAhg42axXSlVDAps2bTJ68qU5cdxfj559jF43Puu6e8xND7xoHnxlG6YOT2TRABpAA2gADaCBxBpQL5uVG183C25dZy6YtcT0G3hCIc9QvqHcY+PGjTXMaDgUBCAAAQiknQAGTtprkPgbRuCFF14w48aNa5FcOUNHy927dDO9Dj/KHH7U8fzBAA2gATSABtAAGojVwJ577xebTyjXeO655xqW33AiCEAAAhBIDwEMnPTUFZF6QkBvqlq0aJEZPHiwOfDAA1tMeBw0dPj8fz2WYAELNIAG0AAaQAOtNdCuXTuj14IfffTRZvr06ebNN9/0JNshDAhAAAIQ8JEABo6PtUJMqSPwb//2b+ajjz4yW7duNW+88QZ/MEADaMBqYOHChWbq1KnoAT2gATTQQgPvv/+++dOf/pS6fIeAIQABCECguQQwcJrLn7NDAAIQgECGCZx22mmmT58+GS4hRYMABCAAAQhAAAIQaBQBDJxGkeY8EIAABCCQKwIabumGjHznO9/JVdkpLAQgAAEIQAACEIBA7Qlg4NSeKUeEAAQgAAEImOXLlxcMnJkzZ0IEAhCAAAQgAAEIQAACVRHAwKkKHztDAAIQgAAEogkMGDCgYOB069YteiPWQgACEIAABCAAAQhAICEBDJyEoNgMAhCAAAQgkJTAO++8UzBv3DCqp59+OunubAcBCEAAAhCAAAQgAIFWBDBwWiFhBQQgAAEIQKA6AldddVUrA+ess86q7qDsDQEIQAACEIAABCCQawIYOLmufgoPAQhAAAL1INC9e/dWBo564vzTP/1TPU7HMSEAAQhAAAIQgAAEckAAAycHlUwRIQABCECgcQReeumlSPNGBs6aNWsaFwhnggAEIAABCEAAAhDIFAEMnExVJ4WBAAQgAIFmE7joootiDZwhQ4Y0OzzODwEIQAACEIAABCCQUgIYOCmtOMKGAAQgAAH/CPz7v/+7adOmTayBo14427Zt8y9wIoIABCAAAQhAAAIQ8J4ABo73VUSAEIAABCCQFgIbNmwoat7IwFmyZElaikOcEIAABCAAAQhAAAIeEcDA8agyCAUCEIAABNJNYNSoUSUNnJ49e6a7kEQPAQhAAAIQgAAEINAUAhg4TcHOSSEAAQhAIGsEfve735U0b9QDR3+vv/561opPeSAAAQhAAAIQgAAE6kwAA6fOgDk8BCAAAQjkg8Dtt9+e2MCZNm1aPqBQSghAAAIQgAAEIACBmhHAwKkZSg4EAQhAAAJ5JjBw4MDEBk7Hjh3NX//61zzjouwQgAAEIAABCEAAAmUSwMApExibQwACEIAABMIE3n333cTmjRtG9cQTT4QPw3cIQAACEIAABCAAAQjEEsDAiUXDDxCAAAQgAIFkBBYtWlS2gTN27NhkB2crCEAAAhCAAAQgAAEIGGMwcJABBCAAAQhAoEoCPXr0KNvAUU+cP/zhD1Wemd0hAAEIQAACEIAABPJCAAMnLzVNOSEAAQhAoC4ENm/eXJF5IwPnzjvvrEtMHBQCEIAABCAAAQhAIHsEMHCyV6eUCAIQgAAEGkjg8ssvN1dccUXhb8aMGcb9ufluJk6caGbNmmX/Zs+ebfQ3Z84cM3fu3AZGyqkgAAEIQAACEIAABNJMAAMnzbVH7BCAAAQg4C2B//mf/yn0zNm2bZu3cRIYBCAAAQhAAAIQgEA6CGDgpKOeiBICEIAABFJGAAMnZRVGuBCAAAQgAAEIQMBzAhg4nlcQ4UEAAhCAQHoJuCFUP/vZz9JbCCKHAAQgAAEIQAACEPCCAAaOF9VAEBCAAAQgkEUCGDhZrFXKBAEIQAACEIAABJpDAAOnOdw5KwQgAAEI5ICAM3B++tOf5qC0FBECEIAABCAAAQhAoJ4EMHDqSZdjQwACEIBArgnstNNOdiJjDJxcy4DCQwACEIAABCAAgZoQwMCpCUYOAgEIQAACEGhNAAOnNRPWQAACEIAABCAAAQhURgADpzJu7AUBCEAAAhAoSWDnnXe2PXB+8pOflNyWDSAAAQhAAAIQgAAEIFCMAAZOMTr8BgEIQAACEKiCAAZOFfDYFQIQgAAEIAABCECgBQEMnBY4+AIBCEAAAhCoHQFn4Pz4xz+u3UE5EgQgAAEIQAACEIBALglg4OSy2ik0BCAAAQg0gsAuu+xih1Bh4DSCNueAAAQgAAEIQAAC2SaAgZPt+qV0EIAABCDQRALOwPnRj37UxCg4NQQgAAEIQAACEIBAFghg4GShFikDBCAAAQh4SaBNmza2Bw4GjpfVQ1AQgAAEIAABCEAgVQQwcFJVXQQLAQhAAAJpIoCBk6baIlYIQAACEIAABCDgNwEMHL/rh+ggAAEIQCDFBNq2bWt74Lz77rspLgWhQwACEIAABCAAAQj4QAADx4daIAYIQAACEMgkAQycTFYrhYIABCAAAQhAAAJNIYCB0xTsnBQCEIAABPJAwBk4P/zhD/NQXMoIAQhAAAIQgAAEIFBHAhg4dYTLoSEAAQhAIN8E2rVrZ4dQYeDkWweUHgIQgAAEIAABCNSCAAZOLShyDAhAAAIQgEAEAWfg/OAHP4j4lVUQgAAEIAABCEAAAhBITgADJzkrtoQABCAAAQiURaB9+/a2Bw4GTlnY2BgCEIAABCAAAQhAIIIABk4EFFZBAAIQgAAEakFg/PjxZsKECQYDpxY0OQYEIAABCEAAAhDINwEMnHzXP6WHAAQgAAEIQAACEIAABCAAAQhAIAUEMHBSUEmECAEIQAACEIAABCAAAQhAAAIQgEC+CWDg5Lv+KT0EIAABCEAAAhCAAAQgAAEIQAACKSCAgZOCSiJECEAAAhCAAAQgAAEIQAACEIAABPJNAAMn3/VP6SEAAQhAAAIQgAAEIAABCEAAAhBIAQEMnBRUEiFCAAIQgAAEIAABCEAAAhCAAAQgkG8CGDj5rn9KDwEIQAACEIAABCAAAQhAAAIQgEAKCGDgpKCSCBECEIAABCAAAQhAAAIQgAAEIACBfBPAwMl3/VN6CEAAAhCAAAQgAAEIQAACEIAABFJAAAMnBZVEiBCAAAQgAAEIQAACEIAABCAAAQjkmwAGTr7rn9JDAAIQgAAEIAABCEAAAhCAAAQgkAICGDgpqCRChAAEIAABCEAAAhCAAAQgAAEIQCDfBDBw8l3/lB4CEIAABCAAAQhAAAIQgAAEIACBFBDAwElBJREiBCAAAQhAAAIQgAAEIAABCEAAAvkmgIGT7/qn9BCAAAQgAAEIQAACEIAABCAAAQikgAAGTgoqiRAhAAEIQAACEIAABCAAAQhAAAIQyDcBDJx81z+lhwAEIAABCEAAAhCAAAQgAAEIQCAFBDBwUlBJhAgBCEAAAhCAAAQgAAEIQAACEIBAvglg4OS7/ik9BCAAAQhAAAIQgAAEIAABCEAAAikggIGTgkoiRAhAAAIQgAAEIAABCEAAAhCAAATyTQADJ9/1T+khAAEIQAACEIAABCAAAQhAAAIQSAEBDJwUVBIhQgACEIAABCAAAQhAAAIQgAAEIJBvAhg4+a5/Sg8BCEAAAhCAAAQgAAEIQAACEIBACghg4KSgkggRAhCAAAQgAAEIQAACEIAABCAAgXwTwMDJd/1TeghAAAIQgAAEIAABCEAAAhCAAARSQAADJwWVRIgQgAAEIAABCEAAAhCAAAQgAAEI5JsABk6+65/SQwACEIAABCAAAQhAAAIQgAAEIJACAhg4KagkQoQABCAAAQhAAAIQgAAEIAABCEAg3wQwcPJd/5QeAhCAAAQgAAEIQAACEIAABCAAgRQQwMBJQSURIgQgAAEIQAACEIAABCAAAQhAAAL5JoCBk+/6p/QQgAAEIAABCEAAAhCAAAQgAAEIpIAABk4KKokQIQABCEAAAhCAAAQgAAEIQAACEMg3AQycfNc/pYcABCAAAQhAAAIQgAAEIAABCEAgBQQwcFJQSYQIAQhAAAIQgAAEIAABCEAAAhCAQL4JYODku/4pPQQgAAEIQAACEIAABCAAAQhAAAIpIICBk4JKIkQIQAACEIAABCAAAQhAAAIQgAAE8k0AAyff9U/pIQABCEAAAhCAAAQgAAEIQAACEEgBAQycFFQSIUIAAhCAAAQgAAEIQAACEIAABCCQbwIYOPmuf0oPAQhAAAIQgAAEIAABCEAAAhCAQAoIYOCkoJIIEQIQgAAEIAABCEAAAhCAAAQgAIF8E8DAyXf9U3oIQAACEIAABCAAAQhAAAIQgAAEUkAAAycFlUSIEIAABCAAAQhAAAIQgAAEIAABCOSbAAZOvuuf0kMAAhCAAAQgAAEIQAACEIAABCCQAgIYOCmoJEKEAAQgAAEIQAACEIAABCAAAQhAIN8EMHDyXf+UHgIQgAAEIAABCEAAAhCAAAQgAIEUEMDASUElESIEIAABCEAAAhCAAAQgAAEIQAAC+SaAgZPv+qf0EIAABCAAAQhAAAIQgAAEIAABCKSAAAZOCiqJECEAAQhAAAIQgAAEIAABCEAAAhDINwEMnHzXP6WHAAQgAAEIQAACEIAABCAAAQhAIAUEMHBSUEmECAEIQAACEIAABCAAAQhAAAIQgEC+CWDg5Lv+KT0EIAABCEAAAhCAAAQgAAEIQAACKSCAgZOCSiJECEAAAhCAAAQgAAEIQAACEIAABPJNAAMn3/VP6SEAAQhAAAIQgAAEIAABCEAAAhBIAQEMnBRUEiFCAAIQgAAEIAABCEAAAhCAAAQgkG8CGDj5rn9KDwEIQAACEIAABCAAAQhAAAIQgEAKCGDgpKCSCBECEIAABCAAAQhAAAIQgAAEIACBfBPAwMl3/VN6CEAAAhCAAAQgAAEIQAACEIAABFJAAAMnBZVEiBCAAAQgAAEIQAACEIAABCAAAQjkmwAGTr7rn9JDAAIQgAAEIAABCEAAAhCAAAQgkAICGDgpqCRChAAEIAABCEAAAhCAAAQgAAEIQCDfBDBw8l3/lB4CEIAABCAAAQhAAAIQgAAEIACBFBDAwElBJREiBCAAAQhAAAIQgAAEIAABCEAAAvkmgIGT7/qn9BCAAAQgAAEIQAACEIAABCAAAQikgAAGTgoqiRAhAAEIQAACEIAABCAAAQhAAAIQyDcBDJx81z+lhwAEIAABCEAAAhCAAAQgAAEIQCAFBDBwUlBJhAgBCEAAAhCAAAQgAAEIQAACEIBAvglg4OS7/ik9BCAAAQhAAAIQgAAEIAABCEAAAikggIGTgkoiRAhAAAIQgAAEIAABCEAAAhCAAATyTQADJ9/1T+khAAEIQAACEIAABCAAAQhAAAIQSAGB/w9Q7DYjrPr9SwAAAABJRU5ErkJggg==", - "text/plain": [ - "" - ] - }, - "execution_count": 6, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "from IPython.display import Image\n", - "Image(filename='img/BERT.png')" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Total parameters count: 108312579\n", - "Trainable parameters count: 7680771\n" - ] - } - ], - "source": [ - "trainable_layers = [model.bert.encoder.layer[-1], model.bert.pooler, model.classifier]\n", - "total_params = 0\n", - "trainable_params = 0\n", - "\n", - "for p in model.parameters():\n", - " p.requires_grad = False\n", - " total_params += p.numel()\n", - "\n", - "for layer in trainable_layers:\n", - " for p in layer.parameters():\n", - " p.requires_grad = True\n", - " trainable_params += p.numel()\n", - "\n", - "print(f\"Total parameters count: {total_params}\") # ~108M\n", - "print(f\"Trainable parameters count: {trainable_params}\") # ~7M" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Thus, by using a pre-trained model we reduce the number of trainable params from over 100 million to just above 7.5 million. This will help both performance and convergence with added noise." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Prepare the data" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Before we begin training, we need to preprocess the data and convert it to the format our model expects. \n", - "\n", - "(Note: it'll take 5-10 minutes to run on a laptop)" - ] - }, - { - "cell_type": "code", - "execution_count": 8, - "metadata": {}, - "outputs": [], - "source": [ - "LABEL_LIST = ['contradiction', 'entailment', 'neutral']\n", - "MAX_SEQ_LENGHT = 128\n", - "\n", - "import torch\n", - "import torch.nn as nn\n", - "import transformers\n", - "from torch.utils.data import TensorDataset\n", - "from transformers.data.processors.utils import InputExample\n", - "from transformers.data.processors.glue import glue_convert_examples_to_features\n", - "\n", - "\n", - "def _create_examples(df, set_type):\n", - " \"\"\" Convert raw dataframe to a list of InputExample. Filter malformed examples\n", - " \"\"\"\n", - " examples = []\n", - " for index, row in df.iterrows():\n", - " if row['gold_label'] not in LABEL_LIST:\n", - " continue\n", - " if not isinstance(row['sentence1'], str) or not isinstance(row['sentence2'], str):\n", - " continue\n", - "\n", - " guid = f\"{index}-{set_type}\"\n", - " examples.append(\n", - " InputExample(guid=guid, text_a=row['sentence1'], text_b=row['sentence2'], label=row['gold_label']))\n", - " return examples\n", - "\n", - "def _df_to_features(df, set_type):\n", - " \"\"\" Pre-process text. This method will:\n", - " 1) tokenize inputs\n", - " 2) cut or pad each sequence to MAX_SEQ_LENGHT\n", - " 3) convert tokens into ids\n", - "\n", - " The output will contain:\n", - " `input_ids` - padded token ids sequence\n", - " `attention mask` - mask indicating padded tokens\n", - " `token_type_ids` - mask indicating the split between premise and hypothesis\n", - " `label` - label\n", - " \"\"\"\n", - " examples = _create_examples(df, set_type)\n", - "\n", - " #backward compatibility with older transformers versions\n", - " legacy_kwards = {}\n", - " from packaging import version\n", - " if version.parse(transformers.__version__) < version.parse(\"2.9.0\"):\n", - " legacy_kwards = {\n", - " \"pad_on_left\": False,\n", - " \"pad_token\": tokenizer.convert_tokens_to_ids([tokenizer.pad_token])[0],\n", - " \"pad_token_segment_id\": 0,\n", - " }\n", - "\n", - " return glue_convert_examples_to_features(\n", - " examples=examples,\n", - " tokenizer=tokenizer,\n", - " label_list=LABEL_LIST,\n", - " max_length=MAX_SEQ_LENGHT,\n", - " output_mode=\"classification\",\n", - " **legacy_kwards,\n", - " )\n", - "\n", - "def _features_to_dataset(features):\n", - " \"\"\" Convert features from `_df_to_features` into a single dataset\n", - " \"\"\"\n", - " all_input_ids = torch.tensor([f.input_ids for f in features], dtype=torch.long)\n", - " all_attention_mask = torch.tensor(\n", - " [f.attention_mask for f in features], dtype=torch.long\n", - " )\n", - " all_token_type_ids = torch.tensor(\n", - " [f.token_type_ids for f in features], dtype=torch.long\n", - " )\n", - " all_labels = torch.tensor([f.label for f in features], dtype=torch.long)\n", - " dataset = TensorDataset(\n", - " all_input_ids, all_attention_mask, all_token_type_ids, all_labels\n", - " )\n", - "\n", - " return dataset\n", - "\n", - "train_features = _df_to_features(df_train, \"train\")\n", - "test_features = _df_to_features(df_test, \"test\")\n", - "\n", - "train_dataset = _features_to_dataset(train_features)\n", - "test_dataset = _features_to_dataset(test_features)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Choosing batch size\n", - "\n", - "Let's talk about batch sizes for a bit.\n", - "\n", - "In addition to all the considerations you normally take into account when choosing batch size, training models with DP adds another one - privacy cost. \n", - "\n", - "Because of the threat model we assume and the way we add noise to the gradients, larger batch sizes (to a certain extent) generally help convergence. We add the same amount of noise to each gradient update (scaled to the norm of one sample in the batch) regardless of the batch size. What this means is that as the batch size increases, the relative amount of noise added decreases. while preserving the same epsilon guarantee. \n", - "\n", - "You should, however, keep in mind that increasing batch size has its price in terms of epsilon, which grows at `O(sqrt(batch_size))` as we train (therefore larger batches make it grow faster). The good strategy here is to experiment with multiple combinations of `batch_size` and `noise_multiplier` to find the one that provides the best possible quality at acceptable privacy guarantee.\n", - "\n", - "There's another side to this - memory. Opacus computes and stores *per sample* gradients, so for every normal gradient, Opacus will store `n=batch_size` per-sample gradients on each step, thus increasing the memory footprint by at least `O(batch_size)`. In reality, however, the peak memory requirement is `O(batch_size^2)` compared to a non-private model. This is because some intermediate steps in per sample gradient computation involve operations on two matrices, each with batch_size as one of the dimensions.\n", - "\n", - "The good news is, we can pick the most appropriate batch size, regardless of memory constraints. Opacus has built-in support for *virtual* batches. Using it we can separate physical steps (gradient computation) and logical steps (noise addition and parameter updates): use larger batches for training, while keeping memory footprint low. Below we will specify two constants:\n", - "\n", - "- `MAX_PHYSICAL_BATCH_SIZE` defines the maximum batch size we can afford from a memory standpoint, and only affects computation speed\n", - "- `BATCH_SIZE`, on the other hand, will affect only convergence and privacy guarantee.\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": 9, - "metadata": {}, - "outputs": [], - "source": [ - "BATCH_SIZE = 32\n", - "MAX_PHYSICAL_BATCH_SIZE = 8" - ] - }, - { - "cell_type": "code", - "execution_count": 10, - "metadata": {}, - "outputs": [], - "source": [ - "from torch.utils.data import DataLoader, RandomSampler, SequentialSampler\n", - "from opacus.utils.uniform_sampler import UniformWithReplacementSampler\n", - "\n", - "\n", - "train_dataloader = DataLoader(train_dataset, batch_size=BATCH_SIZE)\n", - "test_dataloader = DataLoader(test_dataset, sampler=SequentialSampler(test_dataset), batch_size=BATCH_SIZE)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Training" - ] - }, - { - "cell_type": "code", - "execution_count": 11, - "metadata": {}, - "outputs": [], - "source": [ - "# Move the model to appropriate device\n", - "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n", - "model = model.to(device)\n", - "\n", - "# Set the model to train mode (HuggingFace models load in eval mode)\n", - "model = model.train()\n", - "# Define optimizer\n", - "optimizer = torch.optim.AdamW(model.parameters(), lr=5e-4, eps=1e-8)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "First, we specify some training parameters ready to run the training loop for three epochs" - ] - }, - { - "cell_type": "code", - "execution_count": 12, - "metadata": {}, - "outputs": [], - "source": [ - "EPOCHS = 3\n", - "LOGGING_INTERVAL = 5000 # once every how many steps we run evaluation cycle and report metrics\n", - "EPSILON = 7.5\n", - "DELTA = 1 / len(train_dataloader) # Parameter for privacy accounting. Probability of not achieving privacy guarantees" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Let’s now define the evaluation cycle." - ] - }, - { - "cell_type": "code", - "execution_count": 13, - "metadata": {}, - "outputs": [], - "source": [ - "import numpy as np\n", - "from tqdm.notebook import tqdm\n", - "\n", - "def accuracy(preds, labels):\n", - " return (preds == labels).mean()\n", - "\n", - "# define evaluation cycle\n", - "def evaluate(model):\n", - " model.eval()\n", - "\n", - " loss_arr = []\n", - " accuracy_arr = []\n", - "\n", - " for batch in test_dataloader:\n", - " batch = tuple(t.to(device) for t in batch)\n", - "\n", - " with torch.no_grad():\n", - " inputs = {'input_ids': batch[0],\n", - " 'attention_mask': batch[1],\n", - " 'token_type_ids': batch[2],\n", - " 'labels': batch[3]}\n", - "\n", - " outputs = model(**inputs)\n", - " loss, logits = outputs[:2]\n", - "\n", - " preds = np.argmax(logits.detach().cpu().numpy(), axis=1)\n", - " labels = inputs['labels'].detach().cpu().numpy()\n", - "\n", - " loss_arr.append(loss.item())\n", - " accuracy_arr.append(accuracy(preds, labels))\n", - "\n", - " model.train()\n", - " return np.mean(loss_arr), np.mean(accuracy_arr)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Next, we will define and attach PrivacyEngine. There are two parameters you need to consider here:\n", - "\n", - "- `noise_multiplier`. It defines the trade-off between privacy and accuracy. Adding more noise will provide stronger privacy guarantees, but will also hurt model quality. In this run, the PrivacyEngine will determine this value based on the target values of `EPSILON`, `DELTA`, and `EPOCHS`. For the default settings, this will set `noise_multiplier` to about 0.4. \n", - "- `max_grad_norm`. Defines the maximum magnitude of L2 norms to which we clip per sample gradients. There is a bit of tug of war with this threshold: on the one hand, a low threshold means that we will clip many gradients, hurting convergence, so we might be tempted to raise it. However, recall that we add noise with `std=noise_multiplier * max_grad_norm` so we will pay for the increased threshold with more noise. In most cases you can rely on the model being quite resilient to clipping (after the first few iterations your model will tend to adjust so that its gradients stay below the clipping threshold), so you can often just keep the default value (`=1.0`) and focus on tuning `batch_size` and `noise_multiplier` instead. That being said, sometimes clipping hurts the model so it may be worth experimenting with different clipping thresholds, like we are doing in this tutorial.\n", - "\n", - "These two parameters define the scale of the noise we add to gradients: the noise will be sampled from a Gaussian distribution with `std=noise_multiplier * max_grad_norm`.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 14, - "metadata": {}, - "outputs": [], - "source": [ - "from opacus import PrivacyEngine\n", - "\n", - "MAX_GRAD_NORM = 0.1\n", - "\n", - "privacy_engine = PrivacyEngine()\n", - "\n", - "model, optimizer, train_dataloader = privacy_engine.make_private_with_epsilon(\n", - " module=model,\n", - " optimizer=optimizer,\n", - " data_loader=train_dataloader,\n", - " target_delta=DELTA,\n", - " target_epsilon=EPSILON,\n", - " epochs=EPOCHS,\n", - " max_grad_norm=MAX_GRAD_NORM,\n", - ")" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Now we can train the model." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from opacus.utils.batch_memory_manager import BatchMemoryManager\n", - "\n", - "for epoch in range(1, EPOCHS+1):\n", - " losses = []\n", - "\n", - " with BatchMemoryManager(\n", - " data_loader=train_dataloader,\n", - " max_physical_batch_size=MAX_PHYSICAL_BATCH_SIZE,\n", - " optimizer=optimizer\n", - " ) as memory_safe_data_loader:\n", - " for step, batch in enumerate(tqdm(memory_safe_data_loader)):\n", - " optimizer.zero_grad()\n", - "\n", - " batch = tuple(t.to(device) for t in batch)\n", - " inputs = {'input_ids': batch[0],\n", - " 'attention_mask': batch[1],\n", - " 'token_type_ids': batch[2],\n", - " 'labels': batch[3]}\n", - "\n", - " outputs = model(**inputs) # output = loss, logits, hidden_states, attentions\n", - "\n", - " loss = outputs[0]\n", - " loss.backward()\n", - " losses.append(loss.item())\n", - "\n", - " optimizer.step()\n", - "\n", - " if step > 0 and step % LOGGING_INTERVAL == 0:\n", - " train_loss = np.mean(losses)\n", - " eps = privacy_engine.get_epsilon(DELTA)\n", - "\n", - " eval_loss, eval_accuracy = evaluate(model)\n", - "\n", - " print(\n", - " f\"Epoch: {epoch} | \"\n", - " f\"Step: {step} | \"\n", - " f\"Train loss: {train_loss:.3f} | \"\n", - " f\"Eval loss: {eval_loss:.3f} | \"\n", - " f\"Eval accuracy: {eval_accuracy:.3f} | \"\n", - " f\"ɛ: {eps:.2f}\"\n", - " )" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "For the test accuracy, after training for three epochs you should expect something close to the results below.\n", - "\n", - "You can see that we can achieve quite strong privacy guarantee at epsilon=7.5 with a moderate accuracy cost of 11 percentage points compared to non-private model trained in a similar setting (upper layers only) and 16 points compared to best results we were able to achieve using the same architecture.\n", - "\n", - "*NB: When not specified, DP-SGD is trained with upper layers only*" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "| Model | Noise multiplier | Batch size | Accuracy | Epsilon |\n", - "| --- | --- | --- | --- | --- |\n", - "| no DP, train full model | N/A | 32 | 90.1% | N/A |\n", - "| no DP, train upper layers only | N/A | 32 | 85.4% | N/A |\n", - "| DP-SGD | 1.0 | 32 | 70.5% | 0.7 |\n", - "| **DP-SGD (this tutorial)** | **0.4** | **32** | **74.3%** | **7.5** |\n", - "| DP-SGD | 0.3 | 32 | 75.8% | 20.7 |\n", - "| DP-SGD | 0.1 | 32 | 78.3% | 2865 |\n", - "| DP-SGD | 0.4 | 8 | 67.3% | 5.9 |" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Ghost Clipping\n", - "\n", - "In this section, we show how to use Fast Gradient Clipping and Ghost Clipping DP-SGD. The training loop is nearly identical to the existing one in Opacus, which was based on the (non-private) PyTorch training loop. To use Fast Gradient Clipping, we need to pass grad_sample_mode = 'ghost' in the make_private function.\n", - "\n", - "\n", - "The other change is that privacy engine's make_private function takes the loss criterion as input too and sanitizes it. This allows us to repurpose loss.backward to do two backward passes, and a loss rescaling in between. The first backward computes per-sample gradient norms, where as the second backward on the rescaled loss computes the aggregard clipped gradient" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "device = torch.device(\"cuda:0\")\n", - "os.environ[\"CUDA_LAUNCH_BLOCKING\"] = \"1\"\n", - "\n", - "optimizer = torch.optim.AdamW(model.parameters(), lr=5e-4, eps=1e-8)\n", - "model = model.train()\n", - "\n", - "privacy_engine = PrivacyEngine()\n", - "criterion = nn.CrossEntropyLoss(reduction=\"mean\")\n", - "\n", - "model_gc, optimizer_gc, criterion_gc, train_dataloader = (\n", - " privacy_engine.make_private_with_epsilon(\n", - " module=model,\n", - " optimizer=optimizer,\n", - " data_loader=train_dataloader,\n", - " criterion=criterion,\n", - " target_delta=DELTA,\n", - " target_epsilon=EPSILON,\n", - " epochs=EPOCHS,\n", - " max_grad_norm=MAX_GRAD_NORM,\n", - " grad_sample_mode=\"ghost\",\n", - " )\n", - ")\n", - "\n", - "model_gc = model_gc.to(device)\n", - "model_gc = model_gc.train()\n", - "\n", - "for epoch in range(1, EPOCHS + 1):\n", - " losses = []\n", - " for step, batch in enumerate(tqdm(train_dataloader)):\n", - " optimizer_gc.zero_grad()\n", - " batch = tuple(t.to(device) for t in batch)\n", - " inputs = {\n", - " \"input_ids\": batch[0],\n", - " \"attention_mask\": batch[1],\n", - " \"token_type_ids\": batch[2],\n", - " \"labels\": batch[3],\n", - " }\n", - " outputs = model_gc(**inputs) # output = loss, logits, hidden_states, attentions\n", - " loss = criterion_gc(outputs[1], batch[3])\n", - " loss.backward()\n", - " optimizer_gc.step()\n", - " losses.append(loss.item())\n", - "\n", - " if step > 0 and step % LOGGING_INTERVAL == 0:\n", - " train_loss = np.mean(losses)\n", - " eval_loss, eval_accuracy = evaluate(model_gc)\n", - " eps = privacy_engine.get_epsilon(DELTA)\n", - " print(\n", - " f\"Epoch: {epoch} | \"\n", - " f\"Step: {step} | \"\n", - " f\"Train loss: {train_loss:.3f} | \"\n", - " f\"Eval loss: {eval_loss:.3f} | \"\n", - " f\"Eval accuracy: {eval_accuracy:.3f} | \"\n", - " f\"ɛ: {eps:.2f}\"\n", - " )" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Epoch: 1 | Step: 500 | Train loss: 1.209 | Eval loss: 1.409 | Eval accuracy: 0.443 | ɛ: 5.25\n", - "Epoch: 1 | Step: 1000 | Train loss: 1.273 | Eval loss: 1.496 | Eval accuracy: 0.481 | ɛ: 6.12\n", - "Epoch: 1 | Step: 1500 | Train loss: 1.316 | Eval loss: 1.514 | Eval accuracy: 0.537 | ɛ: 6.72" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3 (ipykernel)", - "language": "python", - "name": "python3" + "execution_count": null, + "outputs": [] }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.10.6" + { + "cell_type": "code", + "metadata": { + "originalKey": "02a529bf-d250-4e0b-b7cb-f047265eedaf", + "outputsInitialized": true, + "isAgentGenerated": false, + "language": "python", + "executionStartTime": 1734022774784, + "executionStopTime": 1734022777897, + "serverExecutionDuration": 2957.9766383395, + "requestMsgId": "02a529bf-d250-4e0b-b7cb-f047265eedaf", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "DgEx_iaepH_3", + "outputId": "648e2934-e9f2-476e-cf69-29faca0c4a82" + }, + "source": [ + "import zipfile\n", + "import urllib.request\n", + "import os\n", + "\n", + "import warnings\n", + "warnings.simplefilter(\"ignore\")\n", + "\n", + "def download_and_extract(dataset_url, data_dir):\n", + " print(\"Downloading and extracting ...\")\n", + " filename = \"snli_1.0.zip\"\n", + " urllib.request.urlretrieve(dataset_url, filename)\n", + " with zipfile.ZipFile(filename) as zip_ref:\n", + " zip_ref.extractall(data_dir)\n", + " os.remove(filename)\n", + " print(\"Completed!\")\n", + "\n", + "download_and_extract(STANFORD_SNLI_URL, DATA_DIR)" + ], + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Downloading and extracting ...\n", + "Completed!\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "originalKey": "a3734548-7eef-4b88-a1ba-6ef28a6dd954", + "outputsInitialized": false, + "isAgentGenerated": false, + "language": "markdown", + "id": "JcSMVhyKpH_4" + }, + "source": [ + "The dataset comes in two formats (`tsv` and `json`) and has already been split into train/dev/test. Let’s verify that’s the case." + ] + }, + { + "cell_type": "code", + "metadata": { + "originalKey": "4efc1661-0aae-4c89-b2ab-79c4bf4bf548", + "outputsInitialized": true, + "isAgentGenerated": false, + "language": "python", + "executionStartTime": 1734022779737, + "executionStopTime": 1734022779877, + "serverExecutionDuration": 9.7765219397843, + "requestMsgId": "4efc1661-0aae-4c89-b2ab-79c4bf4bf548", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "33zpUvz3pH_4", + "outputId": "9535d490-3e57-4f23-b348-8e6f6f2b9e9d" + }, + "source": [ + "snli_folder = os.path.join(DATA_DIR, \"snli_1.0\")\n", + "os.listdir(snli_folder)" + ], + "execution_count": null, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "['snli_1.0_test.jsonl',\n", + " 'snli_1.0_train.txt',\n", + " '.DS_Store',\n", + " 'snli_1.0_dev.jsonl',\n", + " 'Icon\\r',\n", + " 'README.txt',\n", + " 'snli_1.0_dev.txt',\n", + " 'snli_1.0_test.txt',\n", + " 'snli_1.0_train.jsonl']" + ] + }, + "metadata": {}, + "execution_count": 8 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "originalKey": "a975a807-7365-4504-a47d-067e70c0d8bf", + "outputsInitialized": false, + "isAgentGenerated": false, + "language": "markdown", + "id": "Qnv2-RWWpH_5" + }, + "source": [ + "Let's now take a look inside. [SNLI dataset](https://nlp.stanford.edu/projects/snli/) provides ample syntactic metadata, but we'll only use raw input text. Therefore, the only fields we're interested in are **sentence1** (premise), **sentence2** (hypothesis), and **gold_label** (label chosen by the majority of annotators).\n", + "\n", + "The label defines the relation between premise and hypothesis: either *contradiction*, *neutral*, or *entailment*." + ] + }, + { + "cell_type": "code", + "metadata": { + "originalKey": "82b717b7-ddc0-4096-a623-de39767ccc25", + "outputsInitialized": true, + "isAgentGenerated": false, + "language": "python", + "executionStartTime": 1734022781467, + "executionStopTime": 1734022786972, + "serverExecutionDuration": 5307.5669091195, + "requestMsgId": "82b717b7-ddc0-4096-a623-de39767ccc25", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 206 + }, + "id": "7jhnTgT3pH_5", + "outputId": "d5523231-85c0-4003-cebc-e1baf814e8d9" + }, + "source": [ + "import pandas as pd\n", + "train_path = os.path.join(snli_folder, \"snli_1.0_train.txt\")\n", + "dev_path = os.path.join(snli_folder, \"snli_1.0_dev.txt\")\n", + "\n", + "df_train = pd.read_csv(train_path, sep='\\t')\n", + "df_test = pd.read_csv(dev_path, sep='\\t')\n", + "\n", + "df_train[['sentence1', 'sentence2', 'gold_label']][:5]" + ], + "execution_count": null, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + " sentence1 \\\n", + "0 A person on a horse jumps over a broken down a... \n", + "1 A person on a horse jumps over a broken down a... \n", + "2 A person on a horse jumps over a broken down a... \n", + "3 Children smiling and waving at camera \n", + "4 Children smiling and waving at camera \n", + "\n", + " sentence2 gold_label \n", + "0 A person is training his horse for a competition. neutral \n", + "1 A person is at a diner, ordering an omelette. contradiction \n", + "2 A person is outdoors, on a horse. entailment \n", + "3 They are smiling at their parents neutral \n", + "4 There are children present entailment " + ], + "text/html": [ + "\n", + "
\n", + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
sentence1sentence2gold_label
0A person on a horse jumps over a broken down a...A person is training his horse for a competition.neutral
1A person on a horse jumps over a broken down a...A person is at a diner, ordering an omelette.contradiction
2A person on a horse jumps over a broken down a...A person is outdoors, on a horse.entailment
3Children smiling and waving at cameraThey are smiling at their parentsneutral
4Children smiling and waving at cameraThere are children presententailment
\n", + "
\n", + "
\n", + "\n", + "
\n", + " \n", + "\n", + " \n", + "\n", + " \n", + "
\n", + "\n", + "\n", + "
\n", + " \n", + "\n", + "\n", + "\n", + " \n", + "
\n", + "\n", + "
\n", + "
\n" + ], + "application/vnd.google.colaboratory.intrinsic+json": { + "type": "dataframe", + "summary": "{\n \"name\": \"df_train[['sentence1', 'sentence2', 'gold_label']][:5]\",\n \"rows\": 5,\n \"fields\": [\n {\n \"column\": \"sentence1\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 2,\n \"samples\": [\n \"Children smiling and waving at camera\",\n \"A person on a horse jumps over a broken down airplane.\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"sentence2\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 5,\n \"samples\": [\n \"A person is at a diner, ordering an omelette.\",\n \"There are children present\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"gold_label\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 3,\n \"samples\": [\n \"neutral\",\n \"contradiction\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}" + } + }, + "metadata": {}, + "execution_count": 9 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "originalKey": "c6251c2b-155c-44c9-a6da-18188598d6fa", + "outputsInitialized": false, + "isAgentGenerated": false, + "language": "markdown", + "id": "T2yWe4xMpH_6" + }, + "source": [ + "## Model" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "originalKey": "53ecea2a-3d50-4cbc-bfe5-e13292e1f27d", + "outputsInitialized": false, + "isAgentGenerated": false, + "language": "markdown", + "id": "LezVNRtypH_6" + }, + "source": [ + "BERT (Bidirectional Encoder Representations from Transformers) is a state-of-the-art approach to various NLP tasks. It uses a Transformer architecture and relies heavily on the concept of pre-training.\n", + "\n", + "We'll use a pre-trained BERT-base model, provided in the huggingface [transformers](https://github.com/huggingface/transformers) repo.\n", + "It gives us a PyTorch implementation for the classic BERT architecture, as well as a tokenizer and weights, pre-trained on a public English corpus (Wikipedia).\n", + "\n", + "Please follow these [installation instructions](https://github.com/huggingface/transformers#installation) before proceeding." + ] + }, + { + "cell_type": "code", + "metadata": { + "originalKey": "755dd641-c9ec-4681-9bd0-652255767547", + "showInput": true, + "customInput": null, + "language": "python", + "executionStartTime": 1734032316314, + "executionStopTime": 1734032317257, + "serverExecutionDuration": 818.86289687827, + "requestMsgId": "755dd641-c9ec-4681-9bd0-652255767547", + "outputsInitialized": true, + "isAgentGenerated": false, + "colab": { + "base_uri": "https://localhost:8080/", + "height": 232, + "referenced_widgets": [ + "a6f080fa6f4b4de399af5d1d7850b960", + "47fec328e2464db3861b16e68e6cc65d", + "3ec2b8f4e38d4b05a09c83e6925960a6", + "6fee830ab9f545cea62081d8cf5b3240", + "cf024e76fe9b4766ac035f617391deb7", + "fc0b17bfb44c45dd9825c6f1719b61cc", + "2dac3a8089c34d0b9ce81653bde67603", + "ffb2ab66b3ca4d9899658fe58c43acb1", + "4b6aad944453432dbf957da380d059a1", + "cf3dae1b960440d7984e9ed287b54cee", + "36254c0a04c840f3bf4038096c736873", + "e3ffd50ee822433fabd9c1ee4a39612e", + "f248160605a2450a8411e4f5d58a5cfa", + "a7b6e7aa521647649bb4157b6504d4e8", + "1c1e86bca0534caaa7ad435fd7e67bf2", + "22ca9e6c6c1f4bc6b0f3db1a09a5e562", + "e9699698559c4860bdf6a312c492e7da", + "874e45fe39844927ad1fd10d4899a428", + "2dca6b2477344b45b1c17f124e27ce72", + "c7afefe6f907441b9e466605cb4f5c7f", + "df3c4c12e06245b1a8f4a4a7d71a530c", + "e63eb3e5c06140249f6d8d4c04fe8693", + "d951c3592058414ab00cf754e9b70685", + "c3f9146e082346a3bed274efb2265376", + "1602c2298e9443f78007fdbf101a0c2b", + "d5dda55bd4de4f12bf3718fb386c5bf9", + "943e61866ed74be4b10ba383450cb4c3", + "0727d77eaf28466c93c2c6021661ac9a", + "3ab2e1a9ba694463ab5f3ec78ad0a8f4", + "30b4db204fa644128198abf6d82664bf", + "0517fdac88784fe6b51ad1f989f99cb7", + "ce1c55bc51dc49a9b261f104f49d38d8", + "4b7e11bb32bc43c9ad0449bd39bc4d40", + "3dbf36a5c0884579ab2f36c2e91c04fb", + "d71c49bd9f8c438898250c3874c06240", + "1b70fa16b803466ea31649dcd644e3d7", + "7a728bea623646c182c508e34b582fc9", + "8494caffe83a4743a11d2751b38c56bb", + "1580380472df40e38cbde67659c5221d", + "349b262479b9418badd6a3acff386dd2", + "3370a9d70dd04d5195bc3f1f81b18728", + "ee235d5ffc5142c895175cbea5c94dfe", + "f6c5ef333a2b45f1bf196fdd58873688", + "e24ff9dae78241d9b5a6a7199c888e45", + "d4a768f261614ac69b3004fbf2323c89", + "81a7d1b27cd94916ac3c330aa2551cf0", + "eac4d3f8e59a4d4c81178cc76600182f", + "5bcca2ee852144c28bfd40de0978cadc", + "05fa1e761bc14c929b19abf0d8a93f5f", + "32ba8daa0c6e4a9c9f62df588a198b1b", + "e1e93f905b494126a6a9e1e0a2f92022", + "9ca2cb3f116547d3bb062f4da11762d7", + "554184f1c9b44bd3a8116773572347eb", + "bb89b007667e4bf0b696ad84dfb2f91d", + "610f073056924398b9229978dff5ff4d" + ] + }, + "id": "bxwD3rYepH_6", + "outputId": "96ef0f4f-6ff3-435d-c2bc-8fed92ca0d05" + }, + "source": [ + "from transformers import BertConfig, BertTokenizer, BertForSequenceClassification\n", + "\n", + "model_name = \"bert-base-cased\"\n", + "config = BertConfig.from_pretrained(\n", + " model_name,\n", + " num_labels=3,\n", + ")\n", + "tokenizer = BertTokenizer.from_pretrained(\n", + " \"bert-base-cased\",\n", + " do_lower_case=False,\n", + ")\n", + "model = BertForSequenceClassification.from_pretrained(\n", + " \"bert-base-cased\",\n", + " config=config,\n", + ")" + ], + "execution_count": null, + "outputs": [ + { + "output_type": "display_data", + "data": { + "text/plain": [ + "config.json: 0%| | 0.00/570 [00:00" + ] + }, + "metadata": {}, + "execution_count": 20 + } + ] + }, + { + "cell_type": "code", + "metadata": { + "originalKey": "6332ec74-d2e5-48a6-bf20-bee1ddf327c6", + "outputsInitialized": true, + "isAgentGenerated": false, + "language": "python", + "executionStartTime": 1734032318121, + "executionStopTime": 1734032318289, + "serverExecutionDuration": 8.0086700618267, + "requestMsgId": "6332ec74-d2e5-48a6-bf20-bee1ddf327c6", + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "F8iA_l3xpH_7", + "outputId": "8d8dd938-f073-4f16-8a9c-dbeaee2bfc4a" + }, + "source": [ + "trainable_layers = [model.bert.encoder.layer[-1], model.bert.pooler, model.classifier]\n", + "total_params = 0\n", + "trainable_params = 0\n", + "\n", + "for p in model.parameters():\n", + " p.requires_grad = False\n", + " total_params += p.numel()\n", + "\n", + "for layer in trainable_layers:\n", + " for p in layer.parameters():\n", + " p.requires_grad = True\n", + " trainable_params += p.numel()\n", + "\n", + "print(f\"Total parameters count: {total_params:,}\") # ~108M\n", + "print(f\"Trainable parameters count: {trainable_params:,}\") # ~7M" + ], + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Total parameters count: 108,312,579\n", + "Trainable parameters count: 7,680,771\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "originalKey": "123236c7-ba61-47e0-89a8-204627f2d9f0", + "outputsInitialized": false, + "isAgentGenerated": false, + "language": "markdown", + "id": "r_KuCSszpH_7" + }, + "source": [ + "Thus, by using a pre-trained model we reduce the number of trainable params from over 100 million to just above 7.5 million. This will help both performance and convergence with added noise." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "originalKey": "b2909681-24bd-45ad-98d6-872d92fd237e", + "outputsInitialized": false, + "isAgentGenerated": false, + "language": "markdown", + "id": "P63ItKGTpH_7" + }, + "source": [ + "## Prepare the data" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "originalKey": "a7d059bf-0844-4457-b77b-6755b5bbb674", + "outputsInitialized": false, + "isAgentGenerated": false, + "language": "markdown", + "id": "49gvBz19pH_7" + }, + "source": [ + "Before we begin training, we need to preprocess the data and convert it to the format our model expects.\n", + "\n", + "(Note: it'll take 5-10 minutes to run on a laptop)" + ] + }, + { + "cell_type": "code", + "metadata": { + "originalKey": "42698e36-08f9-4a83-9a0a-d5f32c384758", + "showInput": true, + "customInput": null, + "language": "python", + "executionStartTime": 1734022792696, + "executionStopTime": 1734022819705, + "serverExecutionDuration": 1299.7262682766, + "requestMsgId": "42698e36-08f9-4a83-9a0a-d5f32c384758", + "outputsInitialized": false, + "isAgentGenerated": false, + "id": "5SPPcMkFpH_7" + }, + "source": [ + "import torch\n", + "import torch.nn as nn\n", + "import transformers\n", + "from torch.utils.data import TensorDataset\n", + "from transformers.data.processors.utils import InputExample\n", + "from transformers.data.processors.glue import glue_convert_examples_to_features" + ], + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "code", + "metadata": { + "originalKey": "9871bf56-ba44-4a14-92dd-6626c58883ee", + "outputsInitialized": false, + "isAgentGenerated": false, + "language": "python", + "executionStartTime": 1734022793887, + "executionStopTime": 1734023174330, + "serverExecutionDuration": 354598.03974768, + "requestMsgId": "9871bf56-ba44-4a14-92dd-6626c58883ee", + "id": "RMPbfeMvpH_7" + }, + "source": [ + "LABEL_LIST = ['contradiction', 'entailment', 'neutral']\n", + "MAX_SEQ_LENGHT = 128\n", + "\n", + "\n", + "\n", + "\n", + "def _create_examples(df, set_type):\n", + " \"\"\" Convert raw dataframe to a list of InputExample. Filter malformed examples\n", + " \"\"\"\n", + " examples = []\n", + " for index, row in df.iterrows():\n", + " if row['gold_label'] not in LABEL_LIST:\n", + " continue\n", + " if not isinstance(row['sentence1'], str) or not isinstance(row['sentence2'], str):\n", + " continue\n", + "\n", + " guid = f\"{index}-{set_type}\"\n", + " examples.append(\n", + " InputExample(guid=guid, text_a=row['sentence1'], text_b=row['sentence2'], label=row['gold_label']))\n", + " return examples\n", + "\n", + "def _df_to_features(df, set_type):\n", + " \"\"\" Pre-process text. This method will:\n", + " 1) tokenize inputs\n", + " 2) cut or pad each sequence to MAX_SEQ_LENGHT\n", + " 3) convert tokens into ids\n", + "\n", + " The output will contain:\n", + " `input_ids` - padded token ids sequence\n", + " `attention mask` - mask indicating padded tokens\n", + " `token_type_ids` - mask indicating the split between premise and hypothesis\n", + " `label` - label\n", + " \"\"\"\n", + " examples = _create_examples(df, set_type)\n", + "\n", + " #backward compatibility with older transformers versions\n", + " legacy_kwards = {}\n", + " from packaging import version\n", + " if version.parse(transformers.__version__) < version.parse(\"2.9.0\"):\n", + " legacy_kwards = {\n", + " \"pad_on_left\": False,\n", + " \"pad_token\": tokenizer.convert_tokens_to_ids([tokenizer.pad_token])[0],\n", + " \"pad_token_segment_id\": 0,\n", + " }\n", + "\n", + " return glue_convert_examples_to_features(\n", + " examples=examples,\n", + " tokenizer=tokenizer,\n", + " label_list=LABEL_LIST,\n", + " max_length=MAX_SEQ_LENGHT,\n", + " output_mode=\"classification\",\n", + " **legacy_kwards,\n", + " )\n", + "\n", + "def _features_to_dataset(features):\n", + " \"\"\" Convert features from `_df_to_features` into a single dataset\n", + " \"\"\"\n", + " all_input_ids = torch.tensor([f.input_ids for f in features], dtype=torch.long)\n", + " all_attention_mask = torch.tensor(\n", + " [f.attention_mask for f in features], dtype=torch.long\n", + " )\n", + " all_token_type_ids = torch.tensor(\n", + " [f.token_type_ids for f in features], dtype=torch.long\n", + " )\n", + " all_labels = torch.tensor([f.label for f in features], dtype=torch.long)\n", + " dataset = TensorDataset(\n", + " all_input_ids, all_attention_mask, all_token_type_ids, all_labels\n", + " )\n", + "\n", + " return dataset\n", + "\n", + "train_features = _df_to_features(df_train, \"train\")\n", + "test_features = _df_to_features(df_test, \"test\")\n", + "\n", + "train_dataset = _features_to_dataset(train_features)\n", + "test_dataset = _features_to_dataset(test_features)" + ], + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "originalKey": "06f80462-8e47-4f1a-8687-f5b891481ab5", + "outputsInitialized": false, + "isAgentGenerated": false, + "language": "markdown", + "id": "yU5kzHNhpH_7" + }, + "source": [ + "## Choosing batch size\n", + "\n", + "Let's talk about batch sizes for a bit.\n", + "\n", + "In addition to all the considerations you normally take into account when choosing batch size, training models with DP adds another one - privacy cost.\n", + "\n", + "Because of the threat model we assume and the way we add noise to the gradients, larger batch sizes (to a certain extent) generally help convergence. We add the same amount of noise to each gradient update (scaled to the norm of one sample in the batch) regardless of the batch size. What this means is that as the batch size increases, the relative amount of noise added decreases. while preserving the same epsilon guarantee.\n", + "\n", + "You should, however, keep in mind that increasing batch size has its price in terms of epsilon, which grows at `O(sqrt(batch_size))` as we train (therefore larger batches make it grow faster). The good strategy here is to experiment with multiple combinations of `batch_size` and `noise_multiplier` to find the one that provides the best possible quality at acceptable privacy guarantee.\n", + "\n", + "There's another side to this - memory. Opacus computes and stores *per sample* gradients, so for every normal gradient, Opacus will store `n=batch_size` per-sample gradients on each step, thus increasing the memory footprint by at least `O(batch_size)`. In reality, however, the peak memory requirement is `O(batch_size^2)` compared to a non-private model. This is because some intermediate steps in per sample gradient computation involve operations on two matrices, each with batch_size as one of the dimensions.\n", + "\n", + "The good news is, we can pick the most appropriate batch size, regardless of memory constraints. Opacus has built-in support for *virtual* batches. Using it we can separate physical steps (gradient computation) and logical steps (noise addition and parameter updates): use larger batches for training, while keeping memory footprint low. Below we will specify two constants:\n", + "\n", + "- `MAX_PHYSICAL_BATCH_SIZE` defines the maximum batch size we can afford from a memory standpoint, and only affects computation speed\n", + "- `BATCH_SIZE`, on the other hand, will affect only convergence and privacy guarantee.\n", + "\n" + ] + }, + { + "cell_type": "code", + "metadata": { + "originalKey": "b06c0410-a2d1-407a-b543-199e23605ad5", + "outputsInitialized": false, + "isAgentGenerated": false, + "language": "python", + "executionStartTime": 1734032324563, + "executionStopTime": 1734032324694, + "serverExecutionDuration": 2.0098211243749, + "requestMsgId": "b06c0410-a2d1-407a-b543-199e23605ad5", + "id": "TefYWR8mpH_7" + }, + "source": [ + "BATCH_SIZE = 32\n", + "MAX_PHYSICAL_BATCH_SIZE = 8" + ], + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "code", + "metadata": { + "originalKey": "07602a22-7950-426b-9d1a-32188f163cb8", + "outputsInitialized": false, + "isAgentGenerated": false, + "language": "python", + "executionStartTime": 1734032325733, + "executionStopTime": 1734032325869, + "serverExecutionDuration": 2.5164256803691, + "requestMsgId": "07602a22-7950-426b-9d1a-32188f163cb8", + "id": "CcH-whfRpH_7" + }, + "source": [ + "from torch.utils.data import DataLoader, RandomSampler, SequentialSampler\n", + "from opacus.utils.uniform_sampler import UniformWithReplacementSampler\n", + "\n", + "train_dataloader = DataLoader(train_dataset, batch_size=BATCH_SIZE)\n", + "test_dataloader = DataLoader(test_dataset, sampler=SequentialSampler(test_dataset), batch_size=BATCH_SIZE)" + ], + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "originalKey": "16320607-1a02-4588-8b7d-3e9ab859c7e7", + "outputsInitialized": false, + "isAgentGenerated": false, + "language": "markdown", + "id": "9Atc-d7QpH_8" + }, + "source": [ + "## Training" + ] + }, + { + "cell_type": "code", + "metadata": { + "originalKey": "e37dd036-25c4-47fa-996e-aced5bd1856b", + "outputsInitialized": false, + "isAgentGenerated": false, + "language": "python", + "executionStartTime": 1734032332678, + "executionStopTime": 1734032332917, + "serverExecutionDuration": 123.88719897717, + "requestMsgId": "e37dd036-25c4-47fa-996e-aced5bd1856b", + "id": "Ibx6k7GspH_8" + }, + "source": [ + "# Move the model to appropriate device\n", + "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n", + "model = model.to(device)\n", + "\n", + "# Set the model to train mode (HuggingFace models load in eval mode)\n", + "model = model.train()\n", + "# Define optimizer\n", + "optimizer = torch.optim.AdamW(model.parameters(), lr=5e-4, eps=1e-8)" + ], + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "originalKey": "4352fbca-a402-4f32-a150-243fbd6cf721", + "outputsInitialized": false, + "isAgentGenerated": false, + "language": "markdown", + "id": "0ghT3eN1pH_8" + }, + "source": [ + "First, we specify some training parameters ready to run the training loop for three epochs" + ] + }, + { + "cell_type": "code", + "metadata": { + "originalKey": "b2807795-97b2-4969-af72-2a434d161c58", + "outputsInitialized": false, + "isAgentGenerated": false, + "language": "python", + "executionStartTime": 1734032333878, + "executionStopTime": 1734032335582, + "serverExecutionDuration": 2.3859869688749, + "requestMsgId": "b2807795-97b2-4969-af72-2a434d161c58", + "id": "cNQ-Rb5LpH_8" + }, + "source": [ + "EPOCHS = 3\n", + "LOGGING_INTERVAL = 5000 # once every how many steps we run evaluation cycle and report metrics\n", + "EPSILON = 7.5\n", + "DELTA = 1 / len(train_dataloader) # Parameter for privacy accounting. Probability of not achieving privacy guarantees" + ], + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "originalKey": "5d20a64b-7e9a-4a02-a367-a01ec40d10d0", + "outputsInitialized": false, + "isAgentGenerated": false, + "language": "markdown", + "id": "VszTFifzpH_8" + }, + "source": [ + "Let’s now define the evaluation cycle." + ] + }, + { + "cell_type": "code", + "metadata": { + "originalKey": "04f41504-002f-4da3-89b2-71a306e3bb9f", + "outputsInitialized": false, + "isAgentGenerated": false, + "language": "python", + "executionStartTime": 1734032340042, + "executionStopTime": 1734032340181, + "serverExecutionDuration": 3.5456721670926, + "requestMsgId": "04f41504-002f-4da3-89b2-71a306e3bb9f", + "id": "ri93DMyTpH_8" + }, + "source": [ + "import numpy as np\n", + "from tqdm.notebook import tqdm\n", + "\n", + "def accuracy(preds, labels):\n", + " return (preds == labels).mean()\n", + "\n", + "# define evaluation cycle\n", + "def evaluate(model):\n", + " model.eval()\n", + "\n", + " loss_arr = []\n", + " accuracy_arr = []\n", + "\n", + " for batch in test_dataloader:\n", + " batch = tuple(t.to(device) for t in batch)\n", + "\n", + " with torch.no_grad():\n", + " inputs = {'input_ids': batch[0],\n", + " 'attention_mask': batch[1],\n", + " 'token_type_ids': batch[2],\n", + " 'labels': batch[3]}\n", + "\n", + " outputs = model(**inputs)\n", + " loss, logits = outputs[:2]\n", + "\n", + " preds = np.argmax(logits.detach().cpu().numpy(), axis=1)\n", + " labels = inputs['labels'].detach().cpu().numpy()\n", + "\n", + " loss_arr.append(loss.item())\n", + " accuracy_arr.append(accuracy(preds, labels))\n", + "\n", + " model.train()\n", + " return np.mean(loss_arr), np.mean(accuracy_arr)" + ], + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "originalKey": "13b3b5e7-70d5-4ee3-951b-8db81148a974", + "outputsInitialized": false, + "isAgentGenerated": false, + "language": "markdown", + "id": "Rl8ncwc1pH_8" + }, + "source": [ + "Next, we will define and attach PrivacyEngine. There are two parameters you need to consider here:\n", + "\n", + "- `noise_multiplier`. It defines the trade-off between privacy and accuracy. Adding more noise will provide stronger privacy guarantees, but will also hurt model quality. In this run, the PrivacyEngine will determine this value based on the target values of `EPSILON`, `DELTA`, and `EPOCHS`. For the default settings, this will set `noise_multiplier` to about 0.4.\n", + "- `max_grad_norm`. Defines the maximum magnitude of L2 norms to which we clip per sample gradients. There is a bit of tug of war with this threshold: on the one hand, a low threshold means that we will clip many gradients, hurting convergence, so we might be tempted to raise it. However, recall that we add noise with `std=noise_multiplier * max_grad_norm` so we will pay for the increased threshold with more noise. In most cases you can rely on the model being quite resilient to clipping (after the first few iterations your model will tend to adjust so that its gradients stay below the clipping threshold), so you can often just keep the default value (`=1.0`) and focus on tuning `batch_size` and `noise_multiplier` instead. That being said, sometimes clipping hurts the model so it may be worth experimenting with different clipping thresholds, like we are doing in this tutorial.\n", + "\n", + "These two parameters define the scale of the noise we add to gradients: the noise will be sampled from a Gaussian distribution with `std=noise_multiplier * max_grad_norm`.\n" + ] + }, + { + "cell_type": "code", + "metadata": { + "originalKey": "12c0094d-2f1d-4cf9-aa41-5bb010b3ad97", + "outputsInitialized": false, + "isAgentGenerated": false, + "language": "python", + "executionStartTime": 1734032342308, + "executionStopTime": 1734032384336, + "serverExecutionDuration": 41907.956605777, + "requestMsgId": "12c0094d-2f1d-4cf9-aa41-5bb010b3ad97", + "id": "nai3pjOqpH_8" + }, + "source": [ + "from opacus import PrivacyEngine\n", + "\n", + "MAX_GRAD_NORM = 0.1\n", + "\n", + "privacy_engine = PrivacyEngine()\n", + "\n", + "model, optimizer, train_dataloader = privacy_engine.make_private_with_epsilon(\n", + " module=model,\n", + " optimizer=optimizer,\n", + " data_loader=train_dataloader,\n", + " target_delta=DELTA,\n", + " target_epsilon=EPSILON,\n", + " epochs=EPOCHS,\n", + " max_grad_norm=MAX_GRAD_NORM,\n", + ")" + ], + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "originalKey": "fff606f7-8ca5-4e01-abdd-aeab3e36f0cc", + "outputsInitialized": false, + "isAgentGenerated": false, + "language": "markdown", + "id": "JDHUkp59pH_8" + }, + "source": [ + "Now we can train the model." + ] + }, + { + "cell_type": "code", + "source": [ + "from opacus.utils.batch_memory_manager import BatchMemoryManager\n", + "\n", + "for epoch in range(1, EPOCHS+1):\n", + " losses = []\n", + "\n", + " with BatchMemoryManager(\n", + " data_loader=train_dataloader,\n", + " max_physical_batch_size=MAX_PHYSICAL_BATCH_SIZE,\n", + " optimizer=optimizer\n", + " ) as memory_safe_data_loader:\n", + " for step, batch in enumerate(tqdm(memory_safe_data_loader)):\n", + " optimizer.zero_grad()\n", + "\n", + " batch = tuple(t.to(device) for t in batch)\n", + " inputs = {'input_ids': batch[0],\n", + " 'attention_mask': batch[1],\n", + " 'token_type_ids': batch[2],\n", + " 'labels': batch[3]}\n", + "\n", + " outputs = model(**inputs) # output = loss, logits, hidden_states, attentions\n", + "\n", + " loss = outputs[0]\n", + " loss.backward()\n", + " losses.append(loss.item())\n", + "\n", + " optimizer.step()\n", + "\n", + " if step > 0 and step % LOGGING_INTERVAL == 0:\n", + " train_loss = np.mean(losses)\n", + " eps = privacy_engine.get_epsilon(DELTA)\n", + "\n", + " eval_loss, eval_accuracy = evaluate(model)\n", + "\n", + " print(\n", + " f\"Epoch: {epoch} | \"\n", + " f\"Step: {step} | \"\n", + " f\"Train loss: {train_loss:.3f} | \"\n", + " f\"Eval loss: {eval_loss:.3f} | \"\n", + " f\"Eval accuracy: {eval_accuracy:.3f} | \"\n", + " f\"ɛ: {eps:.2f}\"\n", + " )" + ], + "metadata": { + "id": "KuwJNsdSscEu" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "originalKey": "4017f2a8-7ffc-4752-82d4-dac5b10e77bd", + "outputsInitialized": false, + "isAgentGenerated": false, + "language": "markdown", + "showInput": false, + "id": "3c6wH8tfpH_8" + }, + "source": [ + "For the test accuracy, after training for three epochs you should expect something close to the results below.\n", + "\n", + "You can see that we can achieve quite strong privacy guarantee at epsilon=7.5 with a moderate accuracy cost of 11 percentage points compared to non-private model trained in a similar setting (upper layers only) and 16 points compared to best results we were able to achieve using the same architecture.\n", + "\n", + "*NB: When not specified, DP-SGD is trained with upper layers only*" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "originalKey": "a5a2c5d2-9e45-41be-9c31-846808588385", + "outputsInitialized": false, + "isAgentGenerated": false, + "language": "markdown", + "showInput": false, + "id": "ZE2OFPcMpH_9" + }, + "source": [ + "| Model | Noise multiplier | Batch size | Accuracy | Epsilon |\n", + "| --- | --- | --- | --- | --- |\n", + "| no DP, train full model | N/A | 32 | 90.1% | N/A |\n", + "| no DP, train upper layers only | N/A | 32 | 85.4% | N/A |\n", + "| DP-SGD | 1.0 | 32 | 70.5% | 0.7 |\n", + "| **DP-SGD (this tutorial)** | **0.4** | **32** | **74.3%** | **7.5** |\n", + "| DP-SGD | 0.3 | 32 | 75.8% | 20.7 |\n", + "| DP-SGD | 0.1 | 32 | 78.3% | 2865 |\n", + "| DP-SGD | 0.4 | 8 | 67.3% | 5.9 |" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "originalKey": "4cfbe9ac-18cd-487c-865b-3da5e13e07b8", + "outputsInitialized": false, + "isAgentGenerated": false, + "language": "markdown", + "id": "8wuqVkG4pH_9" + }, + "source": [ + "## Ghost Clipping\n", + "\n", + "In this section, we show how to use Fast Gradient Clipping and Ghost Clipping DP-SGD. The training loop is nearly identical to the existing one in Opacus, which was based on the (non-private) PyTorch training loop. To use Fast Gradient Clipping, we need to pass grad_sample_mode = 'ghost' in the make_private function.\n", + "\n", + "\n", + "The other change is that privacy engine's make_private function takes the loss criterion as input too and sanitizes it. This allows us to repurpose loss.backward to do two backward passes, and a loss rescaling in between. The first backward computes per-sample gradient norms, where as the second backward on the rescaled loss computes the aggregard clipped gradient" + ] + }, + { + "cell_type": "code", + "metadata": { + "originalKey": "8dec9e28-dbed-43ce-8b8b-172ced512e6d", + "showInput": true, + "customInput": null, + "language": "python", + "executionStartTime": 1733867931416, + "executionStopTime": 1733867931607, + "serverExecutionDuration": 2.2143041715026, + "requestMsgId": "8dec9e28-dbed-43ce-8b8b-172ced512e6d", + "outputsInitialized": false, + "isAgentGenerated": false, + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "cZWxTEFqpH_9", + "outputId": "6ad5c023-07ff-485f-9acd-a98acd05de8d" + }, + "source": [ + "# Let's import the model again and freeze layers as before\n", + "model = BertForSequenceClassification.from_pretrained(\n", + " \"bert-base-cased\",\n", + " config=config,\n", + ")\n", + "\n", + "trainable_layers = [model.bert.encoder.layer[-1], model.bert.pooler, model.classifier]\n", + "\n", + "for p in model.parameters():\n", + " p.requires_grad = False\n", + " total_params += p.numel()\n", + "\n", + "for layer in trainable_layers:\n", + " for p in layer.parameters():\n", + " p.requires_grad = True\n", + " trainable_params += p.numel()" + ], + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stderr", + "text": [ + "Some weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-cased and are newly initialized: ['classifier.bias', 'classifier.weight']\n", + "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n" + ] + } + ] + }, + { + "cell_type": "code", + "source": [ + "device = torch.device(\"cuda:0\")\n", + "os.environ[\"CUDA_LAUNCH_BLOCKING\"] = \"1\"\n", + "\n", + "optimizer = torch.optim.AdamW(model.parameters(), lr=5e-4, eps=1e-8)\n", + "model = model.train()\n", + "\n", + "privacy_engine = PrivacyEngine()\n", + "criterion = nn.CrossEntropyLoss(reduction=\"mean\")\n", + "\n", + "model_gc, optimizer_gc, criterion_gc, train_dataloader = (\n", + " privacy_engine.make_private_with_epsilon(\n", + " module=model,\n", + " optimizer=optimizer,\n", + " data_loader=train_dataloader,\n", + " criterion=criterion,\n", + " target_delta=DELTA,\n", + " target_epsilon=EPSILON,\n", + " epochs=EPOCHS,\n", + " max_grad_norm=MAX_GRAD_NORM,\n", + " grad_sample_mode=\"ghost\",\n", + " )\n", + ")\n", + "\n", + "model_gc = model_gc.to(device)\n", + "model_gc = model_gc.train()\n", + "\n", + "for epoch in range(1, EPOCHS + 1):\n", + " losses = []\n", + " for step, batch in enumerate(tqdm(train_dataloader)):\n", + " optimizer_gc.zero_grad()\n", + " batch = tuple(t.to(device) for t in batch)\n", + " inputs = {\n", + " \"input_ids\": batch[0],\n", + " \"attention_mask\": batch[1],\n", + " \"token_type_ids\": batch[2],\n", + " \"labels\": batch[3],\n", + " }\n", + " outputs = model_gc(**inputs) # output = loss, logits, hidden_states, attentions\n", + " loss = criterion_gc(outputs[1], batch[3])\n", + " loss.backward()\n", + " optimizer_gc.step()\n", + " losses.append(loss.item())\n", + "\n", + " if step > 0 and step % LOGGING_INTERVAL == 0:\n", + " train_loss = np.mean(losses)\n", + " eval_loss, eval_accuracy = evaluate(model_gc)\n", + " eps = privacy_engine.get_epsilon(DELTA)\n", + " print(\n", + " f\"Epoch: {epoch} | \"\n", + " f\"Step: {step} | \"\n", + " f\"Train loss: {train_loss:.3f} | \"\n", + " f\"Eval loss: {eval_loss:.3f} | \"\n", + " f\"Eval accuracy: {eval_accuracy:.3f} | \"\n", + " f\"ɛ: {eps:.2f}\"\n", + " )" + ], + "metadata": { + "id": "lAgYg2IUuI3P" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "originalKey": "f3a32c29-aaf7-4c89-b736-5d558021202e", + "showInput": false, + "customInput": null, + "language": "markdown", + "outputsInitialized": false, + "isAgentGenerated": false, + "id": "H6DxmW2LpIAA" + }, + "source": [ + "\n", + "Epoch: 1 | Step: 5000 | Train loss: 1.559 | Eval loss: 1.508 | Eval accuracy: 0.683 | ɛ: 4.83\n", + "\n", + "Epoch: 1 | Step: 10000 | Train loss: 1.625 | Eval loss: 1.635 | Eval accuracy: 0.723 | ɛ: 5.46\n", + "\n", + "Epoch: 1 | Step: 15000 | Train loss: 1.655 | Eval loss: 1.649 | Eval accuracy: 0.735 | ɛ: 5.86\n", + "\n", + "Epoch: 2 | Step: 5000 | Train loss: 1.742 | Eval loss: 1.676 | Eval accuracy: 0.739 | ɛ: 6.29\n", + "\n", + "Epoch: 2 | Step: 10000 | Train loss: 1.746 | Eval loss: 1.681 | Eval accuracy: 0.743 | ɛ: 6.54\n", + "\n", + "Epoch: 2 | Step: 15000 | Train loss: 1.759 | Eval loss: 1.683 | Eval accuracy: 0.745 | ɛ: 6.76\n", + "\n", + "Epoch: 3 | Step: 5000 | Train loss: 1.784 | Eval loss: 1.769 | Eval accuracy: 0.745 | ɛ: 7.05\n", + "\n", + "Epoch: 3 | Step: 10000 | Train loss: 1.789 | Eval loss: 1.695 | Eval accuracy: 0.748 | ɛ: 7.24\n", + "\n", + "Epoch: 3 | Step: 15000 | Train loss: 1.792 | Eval loss: 1.714 | Eval accuracy: 0.749 | ɛ: 7.42" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "originalKey": "0db42c5e-e10d-4298-b11a-962100627c27", + "showInput": false, + "customInput": null, + "language": "markdown", + "outputsInitialized": false, + "isAgentGenerated": false, + "id": "P99rSXt9pIAA" + }, + "source": [ + "## Low-Rank Adaptation (LoRA) with DP-SGD\n", + "\n", + "\n", + "\n", + "In this section, we show that DP-SGD fine-tuning is compatible with LoRA and other parameter-efficient fine-tuning techinques (PEFT). LoRA can be set up with only a few lines and there are no conceptual changes in the privacy analysis. When full fine-tuning of large models is costly, PEFT methods speed up training by training only a small number of extra parameters, while maintaining on par accuracy with full fine-tuning. In the context of DP-SGD, PEFT have the potential to further improve accuracy, since training fewer parameters means less noise is infused in the computation.\n", + "\n", + "We will use the [`peft`](https://huggingface.co/docs/peft/main/en/index) libarary from HuggingFace, compatible with the `transformers` library." + ] + }, + { + "cell_type": "code", + "metadata": { + "originalKey": "a99e5687-82c3-4a60-b606-615c90bcda46", + "showInput": true, + "customInput": null, + "language": "python", + "executionStartTime": 1734035446835, + "executionStopTime": 1734035447482, + "serverExecutionDuration": 524.47218215093, + "requestMsgId": "a99e5687-82c3-4a60-b606-615c90bcda46", + "outputsInitialized": true, + "isAgentGenerated": false, + "id": "RUguJUh-pIAA", + "outputId": "ae21e4a8-0754-4d0d-e71a-0dbc9d69ef21" + }, + "source": [ + "# reset the model\n", + "model = BertForSequenceClassification.from_pretrained(\n", + " \"bert-base-cased\",\n", + " config=config,\n", + ")" + ], + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stderr", + "text": [ + "Some weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-cased and are newly initialized: ['classifier.bias', 'classifier.weight']\nYou should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "originalKey": "3ed0fe44-7bd7-47f3-ac79-2039fe102fda", + "showInput": false, + "customInput": null, + "language": "markdown", + "outputsInitialized": false, + "isAgentGenerated": false, + "id": "kZ5hxMaypIAA" + }, + "source": [ + "Recall that the total number of model parameters is ~108 M." + ] + }, + { + "cell_type": "code", + "metadata": { + "originalKey": "ca5a2e9f-a10f-414a-a608-68c0e592801b", + "showInput": true, + "customInput": null, + "language": "python", + "executionStartTime": 1734035448142, + "executionStopTime": 1734035448269, + "serverExecutionDuration": 3.400239162147, + "requestMsgId": "ca5a2e9f-a10f-414a-a608-68c0e592801b", + "outputsInitialized": true, + "isAgentGenerated": false, + "id": "718Vz1cgpIAA", + "outputId": "ce2731bc-859a-49bb-e9e9-28c9e439aab1" + }, + "source": [ + "total_params = sum(p.numel() for p in model.parameters() if p.requires_grad)\n", + "print(f\"Total parameters count: {total_params:,}\") # ~108M" + ], + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Total parameters count: 108,312,579\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "originalKey": "d28ddd8b-a38a-489b-b0e9-64a4758a1cb2", + "showInput": false, + "customInput": null, + "language": "markdown", + "outputsInitialized": false, + "isAgentGenerated": false, + "id": "YLeZo1RapIAA" + }, + "source": [ + "After enabling LoRA, the total number of trainable parameters decreases 100-fold to ~1M.\n", + "\n", + "Note some key hyper-parameters when using LoRA such as the rank $r$ of the decomposition matrix. These parameters might need tuning. " + ] + }, + { + "cell_type": "code", + "metadata": { + "originalKey": "19206705-bf23-4979-bcfe-ae4c392091ef", + "showInput": true, + "customInput": null, + "language": "python", + "executionStartTime": 1734035450177, + "executionStopTime": 1734035451571, + "serverExecutionDuration": 1273.7480518408, + "requestMsgId": "19206705-bf23-4979-bcfe-ae4c392091ef", + "outputsInitialized": true, + "isAgentGenerated": false, + "id": "dNTL2qtApIAA", + "outputId": "ac3714b0-7dc5-472e-b604-2ac665e9fa8d" + }, + "source": [ + "from peft import get_peft_model, LoraConfig, TaskType\n", + "\n", + "lora_config = LoraConfig(\n", + " task_type=TaskType.SEQ_CLS, # our particular task is sequence classification\n", + " inference_mode=False, # Enable training mode\n", + " r=32, # Low-rank dimension\n", + " lora_alpha=32, # Alpha scaling factor\n", + " lora_dropout=0.05, # Dropout for LoRA layers\n", + ")\n", + "\n", + "model_with_lora = get_peft_model(model, lora_config)\n", + "trainable_params = sum(p.numel() for p in model_with_lora.parameters() if p.requires_grad)\n", + "print(f\"Total trainable parameters with LoRA: {trainable_params:,}\") # ~1M" + ], + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Total trainable parameters with LoRA: 1,181,955\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "originalKey": "4e46f5d6-decc-442a-a7a8-a710302718c1", + "showInput": false, + "customInput": null, + "language": "markdown", + "outputsInitialized": false, + "isAgentGenerated": false, + "id": "spK3IRN_pIAA" + }, + "source": [ + "Similar to before, we will freeze all but the last attention layer of the model. This further reduces the number of trainable parameters to ~100k, compared to ~7M without LoRA.\n" + ] + }, + { + "cell_type": "code", + "metadata": { + "originalKey": "5d457fd9-e3f0-4aa2-a601-7231fc579323", + "showInput": true, + "customInput": null, + "language": "python", + "executionStartTime": 1734035452652, + "executionStopTime": 1734035452895, + "serverExecutionDuration": 5.7214903645217, + "requestMsgId": "5d457fd9-e3f0-4aa2-a601-7231fc579323", + "outputsInitialized": true, + "isAgentGenerated": false, + "id": "MeB6CaOcpIAB", + "outputId": "1fdc4384-ce55-4380-9f7d-7c4ff844ca15" + }, + "source": [ + "attention_layers_to_freeze = model_with_lora.base_model.bert.encoder.layer[:-1]\n", + "\n", + "# Freeze the parameters in the first 11 attention layers\n", + "for param in attention_layers_to_freeze.parameters():\n", + " param.requires_grad = False\n", + "\n", + "\n", + "trainable_params = sum(p.numel() for p in model_with_lora.parameters() if p.requires_grad)\n", + "print(f\"Total trainable parameters with LoRA after freezing: {trainable_params:,}\") # ~1M" + ], + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Total trainable parameters with LoRA after freezing: 100,611\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "originalKey": "714ebb6a-7512-491e-946b-b802b6bbc1bc", + "showInput": false, + "customInput": null, + "language": "markdown", + "outputsInitialized": false, + "isAgentGenerated": false, + "id": "VWXYrtQ8pIAB" + }, + "source": [ + "Now that we have prepared the model with the LoRA setup, it is business as usual for training with DP-SGD. We use DP-SGD with Ghost Clipping.\n" + ] + }, + { + "cell_type": "code", + "metadata": { + "originalKey": "697d14ed-0b45-43ca-a700-c898ad30dfb8", + "showInput": true, + "customInput": null, + "language": "python", + "outputsInitialized": false, + "isAgentGenerated": false, + "executionStartTime": 1734035472216, + "executionStopTime": 1734035472334, + "serverExecutionDuration": 2.4106916971505, + "requestMsgId": "697d14ed-0b45-43ca-a700-c898ad30dfb8", + "id": "SdDoJGk2pIAB" + }, + "source": [ + "EPOCHS = 3\n", + "LOGGING_INTERVAL = 5000 # once every how many steps we run evaluation cycle and report metrics\n", + "DELTA = 1 / len(train_dataloader) # Parameter for privacy accounting. Probability of not achieving privacy guarantees\n", + "MAX_GRAD_NORM = 0.1" + ], + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "code", + "metadata": { + "originalKey": "409821f0-9513-463b-9077-39d1bf2e5a64", + "showInput": true, + "customInput": null, + "language": "python", + "executionStartTime": 1734035480604, + "executionStopTime": 1734039134648, + "serverExecutionDuration": 3653747.3752331, + "requestMsgId": "409821f0-9513-463b-9077-39d1bf2e5a64", + "outputsInitialized": true, + "isAgentGenerated": false, + "customOutput": null, + "id": "rtSdOuV2pIAB" + }, + "source": [ + "device = torch.device(\"cuda:0\")\n", + "os.environ[\"CUDA_LAUNCH_BLOCKING\"] = \"1\"\n", + "\n", + "optimizer = torch.optim.AdamW(model.parameters(), lr=5e-4, eps=1e-8)\n", + "model = model.train()\n", + "\n", + "privacy_engine = PrivacyEngine()\n", + "criterion = nn.CrossEntropyLoss(reduction=\"mean\")\n", + "\n", + "model_lora, optimizer_lora, criterion_lora, train_dataloader = (\n", + " privacy_engine.make_private_with_epsilon(\n", + " module=model,\n", + " optimizer=optimizer,\n", + " data_loader=train_dataloader,\n", + " criterion=criterion,\n", + " target_delta=DELTA,\n", + " target_epsilon=EPSILON,\n", + " epochs=EPOCHS,\n", + " max_grad_norm=MAX_GRAD_NORM,\n", + " grad_sample_mode=\"ghost\",\n", + " )\n", + ")\n", + "\n", + "model_lora = model_lora.to(device)\n", + "model_lora = model_lora.train()\n", + "\n", + "for epoch in range(1, EPOCHS + 1):\n", + " losses = []\n", + " for step, batch in enumerate(tqdm(train_dataloader)):\n", + " optimizer_lora.zero_grad()\n", + " batch = tuple(t.to(device) for t in batch)\n", + " inputs = {\n", + " \"input_ids\": batch[0],\n", + " \"attention_mask\": batch[1],\n", + " \"token_type_ids\": batch[2],\n", + " \"labels\": batch[3],\n", + " }\n", + " outputs = model_lora(**inputs) # output = loss, logits, hidden_states, attentions\n", + " loss = criterion_lora(outputs[1], batch[3])\n", + " loss.backward()\n", + " optimizer_lora.step()\n", + " losses.append(loss.item())\n", + "\n", + " if step > 0 and step % LOGGING_INTERVAL == 0:\n", + " train_loss = np.mean(losses)\n", + " eval_loss, eval_accuracy = evaluate(model_lora)\n", + " eps = privacy_engine.get_epsilon(DELTA)\n", + " print(\n", + " f\"Epoch: {epoch} | \"\n", + " f\"Step: {step} | \"\n", + " f\"Train loss: {train_loss:.3f} | \"\n", + " f\"Eval loss: {eval_loss:.3f} | \"\n", + " f\"Eval accuracy: {eval_accuracy:.3f} | \"\n", + " f\"ɛ: {eps:.2f}\"\n", + " )" + ], + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "originalKey": "110a245f-fa68-4c59-8cb2-07a95c18a85e", + "showInput": false, + "customInput": null, + "language": "markdown", + "outputsInitialized": false, + "isAgentGenerated": false, + "id": "YDOhAShopIAB" + }, + "source": [ + "\n", + "Epoch: 1 | Step: 5000 | Train loss: 1.370 | Eval loss: 1.208 | Eval accuracy: 0.512 | ɛ: 4.83\n", + "\n", + "Epoch: 1 | Step: 10000 | Train loss: 1.551 | Eval loss: 1.584 | Eval accuracy: 0.621 | ɛ: 5.46\n", + "\n", + "Epoch: 1 | Step: 15000 | Train loss: 1.601 | Eval loss: 1.523 | Eval accuracy: 0.661 | ɛ: 5.86\n", + "\n", + "\n", + "Epoch: 2 | Step: 5000 | Train loss: 1.713 | Eval loss: 1.556 | Eval accuracy: 0.708 | ɛ: 6.29\n", + "\n", + "Epoch: 2 | Step: 10000 | Train loss: 1.720 | Eval loss: 1.573 | Eval accuracy: 0.712 | ɛ: 6.54\n", + "\n", + "Epoch: 2 | Step: 15000 | Train loss: 1.723 | Eval loss: 1.510 | Eval accuracy: 0.725 | ɛ: 6.76\n", + "\n", + "\n", + "Epoch: 3 | Step: 5000 | Train loss: 1.704 | Eval loss: 1.492 | Eval accuracy: 0.735 | ɛ: 7.05\n", + "\n", + "Epoch: 3 | Step: 10000 | Train loss: 1.705 | Eval loss: 1.535 | Eval accuracy: 0.734 | ɛ: 7.24\n", + "\n", + "Epoch: 3 | Step: 15000 | Train loss: 1.704 | Eval loss: 1.492 | Eval accuracy: 0.740 | ɛ: 7.42" + ] + }, + { + "cell_type": "markdown", + "source": [ + "We achieve on-par accuracy with ghost clipping (and vanilla DP-SGD), while training ~100x fewer parameters." + ], + "metadata": { + "id": "cXd0DWIUsnBH" + } + }, + { + "cell_type": "markdown", + "metadata": { + "originalKey": "1340f00d-22db-4623-b2ec-8fd5dd4088ae", + "showInput": false, + "customInput": null, + "language": "markdown", + "id": "HPAik4kTpIAC" + }, + "source": [ + "## Final notes\n", + "\n", + "Notice that there is a significant gap in model accuracy, of about 15 percentage points, between DP and non-DP training.\n", + "\n", + "This gap could be further improved by:\n", + "\n", + "- using larger batch sizes to overcome the effect of noise.\n", + "- hyper-parameter tuning of learning rate and clipping norm (in conjuction with larger batch size).\n", + "- training more attention layers (and exploring the noise-to-model-capacity tradeoff introduced by training more parameters). This strategy is a good fit with LoRA, where we can train more layers with fewer parameters.\n", + "\n", + "We invite you to play along and further improve the accuracy of DP-SGD training." + ] } - }, - "nbformat": 4, - "nbformat_minor": 2 + ] } From 144bd2aee0b0cb3be2c9506b73da9d562cebf534 Mon Sep 17 00:00:00 2001 From: Huanyu Zhang Date: Thu, 19 Dec 2024 08:30:52 -0800 Subject: [PATCH 02/10] Delete CircleCI configs since GithubActions CI are now live (#701) Summary: Pull Request resolved: https://github.com/pytorch/opacus/pull/701 Reviewed By: iden-kalemaj Differential Revision: D67430600 fbshipit-source-id: 19c10c550a4a438b3125ce487d5a0f239e547367 --- .circleci/config.yml | 517 ------------------------------------ .circleci/flake8_config.ini | 119 --------- 2 files changed, 636 deletions(-) delete mode 100644 .circleci/config.yml delete mode 100644 .circleci/flake8_config.ini diff --git a/.circleci/config.yml b/.circleci/config.yml deleted file mode 100644 index 99f1f43f..00000000 --- a/.circleci/config.yml +++ /dev/null @@ -1,517 +0,0 @@ -version: 2.1 - -# ------------------------------------------------------------------------------------- -# Commands -# ------------------------------------------------------------------------------------- - -commands: - - py_3_9_setup: - description: "Install and switch to Python 3.9; also install pip and pytest." - steps: - - run: - name: "Setup Python v3.9 environment" - command: | - cd /opt/circleci/.pyenv && git pull && cd - - pyenv install -s 3.9.4 - pyenv global 3.9.4 - pyenv local 3.9.4 - pyenv versions - echo "In venv: $(pyenv local) - $(python -V), $(pip -V)" - sudo "$(which python)" -m pip install --upgrade pip - sudo "$(which python)" -m pip install pytest - sudo "$(which python)" -m pip install coverage - sudo "$(which python)" -m pip install coveralls - - run_nvidia_smi: - description: "Prints GPU capabilities from nvidia-smi" - steps: - - run: - name: "Run Nvidia-SMI" - command: | - nvidia-smi - - pip_dev_install: - description: "Install dependencies via pip, including extra deps. Also supports more options, such as building on top of PyTorch nightly." - parameters: - args: - type: string - default: "" - steps: - - run: - name: "Install dependencies via pip" - command: ./scripts/install_via_pip.sh << parameters.args >> - - lint_flake8: - description: "Lint with flake8" - steps: - - run: - name: "Lint with flake8" - command: flake8 --config ./.circleci/flake8_config.ini - - lint_black: - description: "Lint with black" - steps: - - run: - name: "Lint with black" - command: black --check --diff --color . - - isort: - description: "Check import order with isort" - steps: - - run: - name: "Check import order with isort" - command: isort -v -l 88 -o opacus --lines-after-imports 2 -m 3 --trailing-comma --check-only . - - unit_tests: - description: "Run unit tests" - steps: - - run: - name: "Unit tests & doctests" - no_output_timeout: 1h - command: | - mkdir unittest-reports - coverage run -m pytest --doctest-modules -p conftest --junitxml=unittest-reports/junit.xml opacus - coverage report -i -m - - - store_test_results: - path: unittest-reports - - store_artifacts: - path: unittest-reports - - command_unit_tests_multi_gpu: - description: "Run multi gpu unit tests" - steps: - - run: - name: "Unit test multi_gpu" - no_output_timeout: 1h - command: | - mkdir unittest-multigpu-reports - coverage run -m unittest opacus.tests.multigpu_gradcheck.GradientComputationTest.test_gradient_correct - coverage report -i -m - - coveralls_upload_parallel: - description: "upload coverage to coveralls" - steps: - - run: - name: "coveralls upload" - no_output_timeout: 5m - command: | - pip install coveralls --user - COVERALLS_PARALLEL=true COVERALLS_FLAG_NAME="${CIRCLE_JOB}" coveralls - - mnist_integration_test: - description: "Runs MNIST example end to end" - parameters: - device: - default: "cpu" - type: string - steps: - - run: - name: MNIST example - command: | - mkdir -p runs/mnist/data - mkdir -p runs/mnist/test-reports - echo "Using $(python -V) ($(which python))" - echo "Using $(pip -V) ($(which pip))" - python examples/mnist.py --lr 0.25 --sigma 0.7 -c 1.5 --batch-size 64 --epochs 1 --data-root runs/mnist/data --n-runs 1 --device <> - python -c "import torch; accuracy = torch.load('run_results_mnist_0.25_0.7_1.5_64_1.pt'); exit(0) if (accuracy[0]>0.78 and accuracy[0]<0.95) else exit(1)" - when: always - - store_test_results: - path: runs/mnist/test-reports - - store_artifacts: - path: runs/mnist/test-reports - - mnist_lightning_integration_test: - description: "Runs MNIST-Lightning example end to end" - parameters: - device: - default: "cpu" - type: string - steps: - - run: - name: MNIST-Lightning example - command: | - mkdir -p runs/mnist/data - mkdir -p runs/mnist/test-reports - echo "Using $(python -V) ($(which python))" - echo "Using $(pip -V) ($(which pip))" - python examples/mnist_lightning.py fit --trainer.accelerator <> --model.lr 0.25 --model.sigma 0.7 --model.max_per_sample_grad_norm 1.5 --model.sample_rate 0.004 --trainer.max_epochs 1 --data.data_dir runs/mnist/data --data.sample_rate 0.004 - python -c "import torch; exit(0)" - when: always - - store_test_results: - path: runs/mnist-lightning/test-reports - - store_artifacts: - path: runs/mnist-lightning/test-reports - - cifar10_integration_test: - description: "Runs CIFAR10 example end to end" - parameters: - device: - default: "cpu" - type: string - steps: - - run: - name: CIFAR10 example - command: | - mkdir -p runs/cifar10/data - mkdir -p runs/cifar10/logs - mkdir -p runs/cifar10/test-reports - echo "Using $(python -V) ($(which python))" - echo "Using $(pip -V) ($(which pip))" - pip install tensorboard - python examples/cifar10.py --lr 0.1 --sigma 1.5 -c 10 --batch-size 2000 --epochs 10 --data-root runs/cifar10/data --log-dir runs/cifar10/logs --device <> - python -c "import torch; model = torch.load('model_best.pth.tar'); exit(0) if (model['best_acc1']>0.4 and model['best_acc1']<0.49) else exit(1)" - python examples/cifar10.py --lr 0.1 --sigma 1.5 -c 10 --batch-size 2000 --epochs 10 --data-root runs/cifar10/data --log-dir runs/cifar10/logs --device <> --grad_sample_mode no_op - python -c "import torch; model = torch.load('model_best.pth.tar'); exit(0) if (model['best_acc1']>0.4 and model['best_acc1']<0.49) else exit(1)" - when: always - - store_test_results: - path: runs/cifar10/test-reports - - store_artifacts: - path: runs/cifar10/test-reports - - dcgan_integration_test: - description: "Runs dcgan example end to end" - parameters: - device: - default: "cpu" - type: string - steps: - - run: - name: dcgan example - command: | - mkdir -p runs/dcgan/data - mkdir -p runs/dcgan/test-reports - echo "Using $(python -V) ($(which python))" - echo "Using $(pip -V) ($(which pip))" - python examples/dcgan.py --lr 2e-4 --sigma 0.7 -c 1.5 --batch-size 32 --epochs 1 --data-root runs/dcgan/data --device <> - when: always - - store_test_results: - path: runs/dcgan/test-reports - - store_artifacts: - path: runs/dcgan/test-reports - - imdb_integration_test: - description: "Runs imdb example end to end" - parameters: - device: - default: "cpu" - type: string - steps: - - run: - name: imdb example - command: | - mkdir -p runs/imdb/data - mkdir -p runs/imdb/test-reports - echo "Using $(python -V) ($(which python))" - echo "Using $(pip -V) ($(which pip))" - pip install --user datasets transformers - python examples/imdb.py --lr 0.02 --sigma 1.0 -c 1.0 --batch-size 64 --max-sequence-length 256 --epochs 2 --data-root runs/imdb/data --device <> - python -c "import torch; accuracy = torch.load('run_results_imdb_classification.pt'); exit(0) if (accuracy>0.54 and accuracy<0.66) else exit(1)" - when: always - - store_test_results: - path: runs/imdb/test-reports - - store_artifacts: - path: runs/imdb/test-reports - - charlstm_integration_test: - description: "Runs charlstm example end to end" - parameters: - device: - default: "cpu" - type: string - steps: - - run: - name: charlstm example - command: | - mkdir -p runs/charlstm/data - wget https://download.pytorch.org/tutorial/data.zip -O runs/charlstm/data/data.zip - unzip runs/charlstm/data/data.zip -d runs/charlstm/data - rm runs/charlstm/data/data.zip - mkdir -p runs/charlstm/test-reports - echo "Using $(python -V) ($(which python))" - echo "Using $(pip -V) ($(which pip))" - pip install scikit-learn - python examples/char-lstm-classification.py --epochs=20 --learning-rate=2.0 --hidden-size=128 --delta=8e-5 --batch-size 400 --n-layers=1 --sigma=1.0 --max-per-sample-grad-norm=1.5 --data-root="runs/charlstm/data/data/names/" --device=<> --test-every 5 - python -c "import torch; accuracy = torch.load('run_results_chr_lstm_classification.pt'); exit(0) if (accuracy>0.60 and accuracy<0.80) else exit(1)" - when: always - - store_test_results: - path: runs/charlstm/test-reports - - store_artifacts: - path: runs/charlstm/test-reports - - benchmark_layers_integration_test: - description: "Runs benchmark end to end" - parameters: - device: - default: "cpu" - type: string - layers: - type: string - grad_sample_modes: - default: "baseline hooks" - type: string - report_column: - default: "hooks/baseline" - type: string - runtime_ratio_threshold: - type: string - memory_ratio_threshold: - type: string - steps: - - run: - name: benchmarks - command: | - mkdir -p benchmarks/results/raw - echo "Using $(python -V) ($(which python))" - echo "Using $(pip -V) ($(which pip))" - python benchmarks/run_benchmarks.py --batch_size 16 --layers <> --config_file ./benchmarks/config.json --root ./benchmarks/results/raw/ --cont - IFS=$' ';layers=(<>); rm -rf /tmp/report_layers; mkdir -p /tmp/report_layers; IFS=$'\n'; files=`( echo "${layers[*]}" ) | sed 's/.*/.\/benchmarks\/results\/raw\/&*/'` - cp -v ${files[@]} /tmp/report_layers - report_id=`IFS=$'-'; echo "${layers[*]}"` - python benchmarks/generate_report.py --path-to-results /tmp/report_layers --save-path benchmarks/results/report-${report_id}.csv --format csv - python benchmarks/generate_report.py --path-to-results /tmp/report_layers --save-path benchmarks/results/report-${report_id}.pkl --format pkl - - python benchmarks/check_threshold.py --report-path "./benchmarks/results/report-"$report_id".pkl" --metric runtime --threshold <> --column <> - when: always - - store_artifacts: - path: benchmarks/results/ -# ------------------------------------------------------------------------------------- -# Jobs -# ------------------------------------------------------------------------------------- - -jobs: - - lint_py39_torch_release: - docker: - - image: cimg/python:3.9 - steps: - - checkout - - pip_dev_install - - lint_flake8 - - lint_black - - isort - - unittest_py38_torch_release: - docker: - - image: cimg/python:3.8 - steps: - - checkout - - pip_dev_install - - unit_tests - - unittest_py39_torch_release: - docker: - - image: cimg/python:3.9 - steps: - - checkout - - pip_dev_install - - unit_tests - - unittest_py39_torch_nightly: - docker: - - image: cimg/python:3.9 - steps: - - checkout - - pip_dev_install: - args: "-n" - - unit_tests - - prv_accountant_values: - docker: - - image: cimg/python:3.9 - steps: - - checkout - - py_3_9_setup - - pip_dev_install - - run: - name: "Unit test prv accountant" - no_output_timeout: 1h - command: | - python -m unittest opacus.tests.prv_accountant - - integrationtest_py39_torch_release_cpu: - docker: - - image: cimg/python:3.9 - steps: - - checkout - - py_3_9_setup - - pip_dev_install - - mnist_integration_test: - device: "cpu" - - integrationtest_py39_torch_release_cuda: - machine: - resource_class: gpu.nvidia.small.multi - image: linux-cuda-12:default - steps: - - checkout - - py_3_9_setup - - pip_dev_install - - run_nvidia_smi - - mnist_integration_test: - device: "cuda" - - cifar10_integration_test: - device: "cuda" - - imdb_integration_test: - device: "cuda" - - charlstm_integration_test: - device: "cuda" - - dcgan_integration_test: - device: "cuda" - - micro_benchmarks_py39_torch_release_cuda: - machine: - resource_class: gpu.nvidia.small.multi - image: linux-cuda-12:default - steps: - - checkout - - py_3_9_setup - - pip_dev_install - - run_nvidia_smi - - benchmark_layers_integration_test: - device: "cuda" - layers: "groupnorm instancenorm layernorm" - grad_sample_modes: "baseline hooks" - runtime_ratio_threshold: "3.0" - memory_ratio_threshold: "1.6" - - benchmark_layers_integration_test: - device: "cuda" - layers: "linear" - grad_sample_modes: "baseline hooks" - runtime_ratio_threshold: "3.6" - memory_ratio_threshold: "13.0" - - benchmark_layers_integration_test: - device: "cuda" - layers: "mha dpmha" - report_column: "dp_baseline/baseline" - grad_sample_modes: "baseline hooks" - runtime_ratio_threshold: "3.0" - memory_ratio_threshold: "1.6" - - benchmark_layers_integration_test: - device: "cuda" - layers: "mha dpmha" - report_column: "dp_hooks/baseline" - grad_sample_modes: "baseline hooks" - runtime_ratio_threshold: "3.5" - memory_ratio_threshold: "2.0" - - benchmark_layers_integration_test: - device: "cuda" - layers: "gru dpgru" - report_column: "dp_baseline/baseline" - grad_sample_modes: "baseline hooks" - runtime_ratio_threshold: "55.2" - memory_ratio_threshold: "1.2" - - benchmark_layers_integration_test: - device: "cuda" - layers: "gru dpgru" - report_column: "dp_hooks/baseline" - grad_sample_modes: "baseline hooks" - runtime_ratio_threshold: "140" - memory_ratio_threshold: "1.6" - - benchmark_layers_integration_test: - device: "cuda" - layers: "lstm dplstm" - report_column: "dp_baseline/baseline" - grad_sample_modes: "baseline hooks" - runtime_ratio_threshold: "48.6" - memory_ratio_threshold: "1.2" - - benchmark_layers_integration_test: - device: "cuda" - layers: "lstm dplstm" - report_column: "dp_hooks/baseline" - grad_sample_modes: "baseline hooks" - runtime_ratio_threshold: "126.0" - memory_ratio_threshold: "1.8" - - benchmark_layers_integration_test: - device: "cuda" - layers: "rnn dprnn" - report_column: "dp_baseline/baseline" - grad_sample_modes: "baseline hooks" - runtime_ratio_threshold: "21.4" - memory_ratio_threshold: "1.2" - - benchmark_layers_integration_test: - device: "cuda" - layers: "rnn dprnn" - report_column: "dp_hooks/baseline" - grad_sample_modes: "baseline hooks" - runtime_ratio_threshold: "98.5" - memory_ratio_threshold: "1.2" - - benchmark_layers_integration_test: - device: "cuda" - layers: "embedding" - grad_sample_modes: "baseline hooks" - runtime_ratio_threshold: "8.0" - memory_ratio_threshold: "15.0" - - unittest_multi_gpu: - machine: - resource_class: gpu.nvidia.medium.multi - image: linux-cuda-12:default - steps: - - checkout - - py_3_9_setup - - pip_dev_install - - run_nvidia_smi - - command_unit_tests_multi_gpu - - finish_coveralls_parallel: - docker: - - image: cimg/python:3.9 - steps: - - run: - name: "finish coveralls parallel" - no_output_timeout: 5m - command: | - pip install coveralls --user - coveralls --finish - - -aliases: - - - &exclude_ghpages - branches: - ignore: - - gh-pages - -# ------------------------------------------------------------------------------------- -# Workflows -# ------------------------------------------------------------------------------------- - -workflows: - commit: - when: - not: - equal: [ scheduled_pipeline, << pipeline.trigger_source >> ] - jobs: - - lint_py39_torch_release: - filters: *exclude_ghpages - - unittest_py38_torch_release: - filters: *exclude_ghpages - - unittest_py39_torch_release: - filters: *exclude_ghpages - - unittest_py39_torch_nightly: - filters: *exclude_ghpages - - unittest_multi_gpu: - filters: *exclude_ghpages - - integrationtest_py39_torch_release_cpu: - filters: *exclude_ghpages - - integrationtest_py39_torch_release_cuda: - filters: *exclude_ghpages - - prv_accountant_values: - filters: *exclude_ghpages - - nightly: - when: - equal: [ scheduled_pipeline, << pipeline.trigger_source >> ] - jobs: - - unittest_py39_torch_nightly: - filters: *exclude_ghpages - - integrationtest_py39_torch_release_cpu: - filters: *exclude_ghpages - - integrationtest_py39_torch_release_cuda: - filters: *exclude_ghpages - - lint_py39_torch_release: - filters: *exclude_ghpages - - micro_benchmarks_py39_torch_release_cuda: - filters: *exclude_ghpages diff --git a/.circleci/flake8_config.ini b/.circleci/flake8_config.ini deleted file mode 100644 index 988ed7f3..00000000 --- a/.circleci/flake8_config.ini +++ /dev/null @@ -1,119 +0,0 @@ -[flake8] -select = B,C,E,F,P,W,B9 -max-line-length = 80 -# Main Explanation Docs: https://github.com/grantmcconnaughey/Flake8Rules -ignore = - # Black conflicts and overlaps. - # Found in https://github.com/psf/black/issues/429 - # B950: Line too long. (Use `arc lint`'s LINEWRAP instead) - B950, - # E111: Indentation is not a multiple of four. - E111, - # E115: Expected an indented block (comment). - E115, - # E117: Over-indented. - E117, - # E121: Continuation line under-indented for hanging indent. - E121, - # E122: Continuation line missing indentation or outdented. - E122, - # E123: Closing bracket does not match indentation of opening bracket's line. - E123, - # E124: Closing bracket does not match visual indentation. - E124, - # E125: Continuation line with same indent as next logical line. - E125, - # E126: Continuation line over-indented for hanging indent. - E126, - # E127: Continuation line over-indented for visual indent. - E127, - # E128: Continuation line under-indented for visual indent. - E128, - # E129: Visually indented line with same indent as next logical line. - E129, - # E201: Whitespace after '('. - E201, - # E202: Whitespace before ')'. - E202, - # E203: Whitespace before ':'. - E203, - # E221: Multiple spaces before operator. - E221, - # E222: Multiple spaces after operator. - E222, - # E225: Missing whitespace around operator. - E225, - # E226: Missing whitespace around arithmetic operator. - E226, - # E227: Missing whitespace around bitwise or shift operator. - E227, - # E231: Missing whitespace after ',', ';', or ':'. - E231, - # E241: Multiple spaces after ','. - E241, - # E251: Unexpected spaces around keyword / parameter equals. - E251, - # E261: At least two spaces before inline comment. - E261, - # E262: Inline comment should start with '# '. - E262, - # E265: Block comment should start with '# '. - E265, - # E271: Multiple spaces after keyword. - E271, - # E272: Multiple spaces before keyword. - E272, - # E301: Expected 1 blank line, found 0. - E301, - # E302: Expected 2 blank lines, found 0. - E302, - # E303: Too many blank lines (3). - E303, - # E305: Expected 2 blank lines after end of function or class. - E305, - # E306: Expected 1 blank line before a nested definition. - E306, - # E501: Line too long (82 > 79 characters). - E501, - # E502: The backslash is redundant between brackets. - E502, - # E701: Multiple statements on one line (colon). - E701, - # E702: Multiple statements on one line (semicolon). - E702, - # E703: Statement ends with a semicolon. - E703, - # E704: Multiple statements on one line (def). - E704, - # W291: Trailing whitespace. - W291, - # W292: No newline at end of file. - W292, - # W293: Blank line contains whitespace. - W293, - # W391: Blank line at end of file. - W391, - - # Too opinionated. - # E265: Block comment should start with '# '. - E265, - # E266: Too many leading '#' for block comment. - E266, - # E402: Module level import not at top of file. - E402, - # E722: Do not use bare except, specify exception instead. (Duplicate of B001) - E722, - # F811: Redefinition of unused name from line n. - F811, - # P207: (Duplicate of B003) - P207, - # P208: (Duplicate of C403) - P208, - # W503: Line break occurred before a binary operator. - W503 - -exclude = - .hg, - __pycache__, - -max-complexity = 12 From f86ddf4380cdbcb2fd37bb886b8da7fa7d20b613 Mon Sep 17 00:00:00 2001 From: Iden Kalemaj Date: Thu, 19 Dec 2024 11:23:42 -0800 Subject: [PATCH 03/10] Separate function for preparing criterion in PrivacyEngine (#703) Summary: Pull Request resolved: https://github.com/pytorch/opacus/pull/703 Having a separate function for preparing the criterion makes it easy to build custom extensions of PrivacyEnginge for methods that require a different DPLoss class, e.g., adaptive clipping. Reviewed By: EnayatUllah Differential Revision: D67458234 fbshipit-source-id: 9fca64fcde7714708ac1cb9a35a991099606f449 --- opacus/privacy_engine.py | 29 +++++++++++++++++++++++++++-- 1 file changed, 27 insertions(+), 2 deletions(-) diff --git a/opacus/privacy_engine.py b/opacus/privacy_engine.py index 1af891c4..558c8f8e 100644 --- a/opacus/privacy_engine.py +++ b/opacus/privacy_engine.py @@ -212,6 +212,26 @@ def _prepare_model( loss_reduction=loss_reduction, ) + def _prepare_criterion( + self, + *, + module: GradSampleModule, + optimizer: DPOptimizer, + criterion=nn.CrossEntropyLoss(), + loss_reduction: str = "mean", + **kwargs, + ) -> DPLossFastGradientClipping: + """ + Args: + module: GradSampleModule used for training, + optimizer: DPOptimizer used for training, + criterion: Loss function used for training, + loss_reduction: "mean" or "sum", indicates if the loss reduction (for aggregating the gradients) + + Prepare the DP loss class, which packages the two backward passes for fast gradient clipping. + """ + return DPLossFastGradientClipping(module, optimizer, criterion, loss_reduction) + def is_compatible( self, *, @@ -403,9 +423,14 @@ def make_private( self.accountant.get_optimizer_hook_fn(sample_rate=sample_rate) ) if grad_sample_mode == "ghost": - criterion = DPLossFastGradientClipping( - module, optimizer, criterion, loss_reduction + criterion = self._prepare_criterion( + module=module, + optimizer=optimizer, + criterion=criterion, + loss_reduction=loss_reduction, + **kwargs, ) + return module, optimizer, criterion, data_loader return module, optimizer, data_loader From 2a2583013bcb84b2b191b84a30706ef50526b965 Mon Sep 17 00:00:00 2001 From: Iden Kalemaj Date: Fri, 20 Dec 2024 02:04:34 -0800 Subject: [PATCH 04/10] Add research folder (#700) Summary: Pull Request resolved: https://github.com/pytorch/opacus/pull/700 Add a research folder for open-source contributions of new methods enhancing DP-SGD. Reviewed By: HuanyuZhang Differential Revision: D67400804 fbshipit-source-id: 49e9dbd7914822c73538a723380ae7578962c1f7 --- research/README.md | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) create mode 100644 research/README.md diff --git a/research/README.md b/research/README.md new file mode 100644 index 00000000..3d16e32b --- /dev/null +++ b/research/README.md @@ -0,0 +1,22 @@ +# New Methods and Extensions of Opacus + +This directory contains novel methods built on top of Opacus that enhance DP-SGD. These contributions, made by the community, stem from research demonstrating potential improvements in differentially private model training. By consolidating these methods within the Opacus repository, we facilitate new research and provide a broader array of tools for DP-ML practitioners. + + +## Contributions +We warmly welcome and encourage contributions of new methods! To contribute, please follow these steps: + +1. Fork the repo and create your branch from `main`. +2. Place the new method in a separate subfolder within the `research` directory. +3. The new folder should include a `README.md` that explains the method at a high level, demonstrates usage (e.g., introducing new parameters to the `PrivacyEngine`), and cites relevant sources. The subfolder name should aptly represent the method. +4. If you have added code that should be tested, add unit tests. +5. If you have changed APIs, document the API change in the PR. Also update the documentation and make sure the documentation builds. +6. Ensure the test suite passes. +7. Make sure your code passes both `black` and `flake8` formatting checks. + +More detailed PR instructions can be found [here](https://github.com/pytorch/opacus/blob/main/CONTRIBUTING.md). + +Feel free to reach out with any questions about the process or to discuss whether your method is a good fit for the repository. + +## Notes +Please note that the code provided in this directory will not be maintained by the Opacus team, which may lead to compatibility issues with future changes. If you have any questions, please reach out to the PR contributor. From ed1bd0bad2d73780c384317f50496ea027e7c6f3 Mon Sep 17 00:00:00 2001 From: Iden Kalemaj Date: Fri, 20 Dec 2024 02:32:57 -0800 Subject: [PATCH 05/10] Remove **kwargs from optim_class initialization (#702) Summary: Pull Request resolved: https://github.com/pytorch/opacus/pull/702 None of the optimizer classes accept **kwargs or have special arguments so I am removing **kwargs from optim_class(). Otherwise, the current code throws an error when creating a custom PrivacyEngine that takes in additional arguments. Reviewed By: EnayatUllah Differential Revision: D67456352 fbshipit-source-id: 8d2985a3f1b5b0af7004b249b6da13aee02debd7 --- opacus/privacy_engine.py | 1 - 1 file changed, 1 deletion(-) diff --git a/opacus/privacy_engine.py b/opacus/privacy_engine.py index 558c8f8e..bdddafe4 100644 --- a/opacus/privacy_engine.py +++ b/opacus/privacy_engine.py @@ -136,7 +136,6 @@ def _prepare_optimizer( loss_reduction=loss_reduction, generator=generator, secure_mode=self.secure_mode, - **kwargs, ) def _prepare_data_loader( From b3c390efb70319ea6b4f84b30df849e7960b6253 Mon Sep 17 00:00:00 2001 From: Huanyu Zhang Date: Fri, 20 Dec 2024 07:06:29 -0800 Subject: [PATCH 06/10] Padding on the research folder (D67400804) (#705) Summary: Pull Request resolved: https://github.com/pytorch/opacus/pull/705 Slightly change the language of the ``readme``. Reviewed By: iden-kalemaj Differential Revision: D67520069 fbshipit-source-id: ee8ac1b878daa78270ef8c145a339a917de899bb --- research/README.md | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/research/README.md b/research/README.md index 3d16e32b..952b0d40 100644 --- a/research/README.md +++ b/research/README.md @@ -9,14 +9,10 @@ We warmly welcome and encourage contributions of new methods! To contribute, ple 1. Fork the repo and create your branch from `main`. 2. Place the new method in a separate subfolder within the `research` directory. 3. The new folder should include a `README.md` that explains the method at a high level, demonstrates usage (e.g., introducing new parameters to the `PrivacyEngine`), and cites relevant sources. The subfolder name should aptly represent the method. -4. If you have added code that should be tested, add unit tests. -5. If you have changed APIs, document the API change in the PR. Also update the documentation and make sure the documentation builds. -6. Ensure the test suite passes. -7. Make sure your code passes both `black` and `flake8` formatting checks. More detailed PR instructions can be found [here](https://github.com/pytorch/opacus/blob/main/CONTRIBUTING.md). Feel free to reach out with any questions about the process or to discuss whether your method is a good fit for the repository. ## Notes -Please note that the code provided in this directory will not be maintained by the Opacus team, which may lead to compatibility issues with future changes. If you have any questions, please reach out to the PR contributor. +Please note that the code provided in this directory will not be maintained by the Opacus team, which may lead to compatibility issues with future changes. If you have any questions, please reach out to the PR contributor directly. From 6b756a7c506c9d610487e5a6b5e66771df87e720 Mon Sep 17 00:00:00 2001 From: Iden Kalemaj Date: Fri, 20 Dec 2024 08:30:05 -0800 Subject: [PATCH 07/10] Add *kwargs to get_epsilon (#704) Summary: Pull Request resolved: https://github.com/pytorch/opacus/pull/704 Add **kwargs as argument to the get_epsilon() function in accountants. It allows for building custom PrivacyEngines that take in additional arguments. Reviewed By: EnayatUllah Differential Revision: D67514874 fbshipit-source-id: 6ebee60d9662ec90ae10b7dcbbb1a939b33f9bfa --- opacus/accountants/gdp.py | 2 +- opacus/accountants/prv.py | 7 ++++++- opacus/accountants/rdp.py | 5 ++++- 3 files changed, 11 insertions(+), 3 deletions(-) diff --git a/opacus/accountants/gdp.py b/opacus/accountants/gdp.py index c39dc765..8a147ae0 100644 --- a/opacus/accountants/gdp.py +++ b/opacus/accountants/gdp.py @@ -44,7 +44,7 @@ def step(self, *, noise_multiplier: float, sample_rate: float): else: self.history = [(noise_multiplier, sample_rate, 1)] - def get_epsilon(self, delta: float, poisson: bool = True) -> float: + def get_epsilon(self, delta: float, poisson: bool = True, **kwargs) -> float: """ Return privacy budget (epsilon) expended so far. diff --git a/opacus/accountants/prv.py b/opacus/accountants/prv.py index ee0e0d6d..b1d09ebf 100644 --- a/opacus/accountants/prv.py +++ b/opacus/accountants/prv.py @@ -81,7 +81,12 @@ def step(self, *, noise_multiplier: float, sample_rate: float): self.history.append((noise_multiplier, sample_rate, 1)) def get_epsilon( - self, delta: float, *, eps_error: float = 0.01, delta_error: float = None + self, + delta: float, + *, + eps_error: float = 0.01, + delta_error: float = None, + **kwargs, ) -> float: """ Return privacy budget (epsilon) expended so far. diff --git a/opacus/accountants/rdp.py b/opacus/accountants/rdp.py index a5b9cb03..1697e352 100644 --- a/opacus/accountants/rdp.py +++ b/opacus/accountants/rdp.py @@ -68,7 +68,10 @@ def get_privacy_spent( return float(eps), float(best_alpha) def get_epsilon( - self, delta: float, alphas: Optional[List[Union[float, int]]] = None + self, + delta: float, + alphas: Optional[List[Union[float, int]]] = None, + **kwargs, ): """ Return privacy budget (epsilon) expended so far. From 53b3c25432bb75dd92b22131d02fcdc39c8dbe5f Mon Sep 17 00:00:00 2001 From: Enayat Ullah Date: Tue, 7 Jan 2025 10:51:24 -0800 Subject: [PATCH 08/10] Fixing Ghost Clipping with Batch Memory Manager Summary: Ghost Clipping with Batch memory manager had an error, resulting in major accuracy loss. The issue was in the accumulate function, the command `p.summed_grad +=p.grad`, wasn't working as expected, since `p.grad` is modified in every iteration. The fix is to copy it and modify in-place. Reviewed By: HuanyuZhang Differential Revision: D67778159 fbshipit-source-id: b103cf95905c0b1feb9745249ec1669c95c11979 --- opacus/optimizers/optimizer_fast_gradient_clipping.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/opacus/optimizers/optimizer_fast_gradient_clipping.py b/opacus/optimizers/optimizer_fast_gradient_clipping.py index aa415e33..21489779 100644 --- a/opacus/optimizers/optimizer_fast_gradient_clipping.py +++ b/opacus/optimizers/optimizer_fast_gradient_clipping.py @@ -14,6 +14,7 @@ from __future__ import annotations +import copy import logging from typing import Callable, Optional @@ -112,9 +113,9 @@ def accumulate(self): """ for p in self.params: if p.summed_grad is not None: - p.summed_grad += p.grad + p.summed_grad.add_(p.grad.data) else: - p.summed_grad = p.grad + p.summed_grad = copy.deepcopy(p.grad.data) def zero_grad(self, set_to_none: bool = False): """ From 9b7854347e1ab172ceb8e1ac1ed4118cd3a96544 Mon Sep 17 00:00:00 2001 From: Enayat Ullah Date: Thu, 9 Jan 2025 08:30:20 -0800 Subject: [PATCH 09/10] Website and Github update (#677) Summary: Pull Request resolved: https://github.com/pytorch/opacus/pull/677 Two updates: 1. Github page: Added a line that the latest version supports fast gradient and ghost clipping. 2. Wesbite: Removed the line about passing in custom alphas in the privacy accountant in the FAQs section of website. Reviewed By: HuanyuZhang Differential Revision: D63790553 fbshipit-source-id: 566914c9c92451f3b90804cab6c560f57ba597e2 --- README.md | 2 + docs/faq.md | 3 +- tutorials/building_text_classifier.ipynb | 1716 +--------------------- website/package.json | 26 +- 4 files changed, 19 insertions(+), 1728 deletions(-) diff --git a/README.md b/README.md index 5ef7651d..a671d35c 100644 --- a/README.md +++ b/README.md @@ -10,6 +10,7 @@ [Opacus](https://opacus.ai) is a library that enables training PyTorch models with differential privacy. It supports training with minimal code changes required on the client, has little impact on training performance, and allows the client to online track the privacy budget expended at any given moment. + ## Target audience This code release is aimed at two target audiences: 1. ML practitioners will find this to be a gentle introduction to training a model with differential privacy as it requires minimal code changes. @@ -99,6 +100,7 @@ If you want to learn more about DP-SGD and related topics, check out our series - [PriCon 2020 Tutorial: Differentially Private Model Training with Opacus](https://www.youtube.com/watch?v=MWPwofiQMdE&list=PLUNOsx6Az_ZGKQd_p4StdZRFQkCBwnaY6&index=52) - [Differential Privacy on PyTorch | PyTorch Developer Day 2020](https://www.youtube.com/watch?v=l6fbl2CBnq0) - [Opacus v1.0 Highlights | PyTorch Developer Day 2021](https://www.youtube.com/watch?v=U1mszp8lzUI) +- [Enabling Fast Gradient Clipping and Ghost Clipping in Opacus](https://pytorch.org/blog/clipping-in-opacus/) ## FAQ diff --git a/docs/faq.md b/docs/faq.md index 1de387a8..ea12bebe 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -108,7 +108,8 @@ Opacus computes and stores *per-sample* gradients under the hood. What this mean Although we report expended privacy budget using the (epsilon, delta) language, internally, we track it using Rényi Differential Privacy (RDP) [[Mironov 2017](https://arxiv.org/abs/1702.07476), [Mironov et al. 2019](https://arxiv.org/abs/1908.10530)]. In short, (alpha, epsilon)-RDP bounds the [Rényi divergence](https://en.wikipedia.org/wiki/R%C3%A9nyi_entropy#R%C3%A9nyi_divergence) of order alpha between the distribution of the mechanism’s outputs on any two datasets that differ in a single element. An (alpha, epsilon)-RDP statement is a relaxation of epsilon-DP but retains many of its important properties that make RDP particularly well-suited for privacy analysis of DP-SGD. The `alphas` parameter instructs the privacy engine what RDP orders to use for tracking privacy expenditure. -When the privacy engine needs to bound the privacy loss of a training run using (epsilon, delta)-DP for a given delta, it searches for the optimal order from among `alphas`. There’s very little additional cost in expanding the list of orders. We suggest using a list `[1 + x / 10.0 for x in range(1, 100)] + list(range(12, 64))`. You can pass your own alphas by passing `alphas=custom_alphas` when calling `privacy_engine.make_private_with_epsilon`. +When the privacy engine needs to bound the privacy loss of a training run using (epsilon, delta)-DP for a given delta, it searches for the optimal order from among `alphas`. There’s very little additional cost in expanding the list of orders. We suggest using a list `[1 + x / 10.0 for x in range(1, 100)] + list(range(12, 64))`. + A call to `privacy_engine.get_epsilon(delta=delta)` returns a pair: an epsilon such that the training run satisfies (epsilon, delta)-DP and an optimal order alpha. An easy diagnostic to determine whether the list of `alphas` ought to be expanded is whether the returned value alpha is one of the two boundary values of `alphas`. diff --git a/tutorials/building_text_classifier.ipynb b/tutorials/building_text_classifier.ipynb index 585d54d7..b4eb0d58 100644 --- a/tutorials/building_text_classifier.ipynb +++ b/tutorials/building_text_classifier.ipynb @@ -21,1721 +21,7 @@ "provenance": [], "gpuType": "T4" }, - "accelerator": "GPU", - "widgets": { - "application/vnd.jupyter.widget-state+json": { - "a6f080fa6f4b4de399af5d1d7850b960": { - "model_module": "@jupyter-widgets/controls", - "model_name": "HBoxModel", - "model_module_version": "1.5.0", - "state": { - "_dom_classes": [], - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "HBoxModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/controls", - "_view_module_version": "1.5.0", - "_view_name": "HBoxView", - "box_style": "", - "children": [ - "IPY_MODEL_47fec328e2464db3861b16e68e6cc65d", - "IPY_MODEL_3ec2b8f4e38d4b05a09c83e6925960a6", - "IPY_MODEL_6fee830ab9f545cea62081d8cf5b3240" - ], - "layout": "IPY_MODEL_cf024e76fe9b4766ac035f617391deb7" - } - }, - "47fec328e2464db3861b16e68e6cc65d": { - "model_module": "@jupyter-widgets/controls", - "model_name": "HTMLModel", - "model_module_version": "1.5.0", - "state": { - "_dom_classes": [], - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "HTMLModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/controls", - "_view_module_version": "1.5.0", - "_view_name": "HTMLView", - "description": "", - "description_tooltip": null, - "layout": "IPY_MODEL_fc0b17bfb44c45dd9825c6f1719b61cc", - "placeholder": "​", - "style": "IPY_MODEL_2dac3a8089c34d0b9ce81653bde67603", - "value": "config.json: 100%" - } - }, - "3ec2b8f4e38d4b05a09c83e6925960a6": { - "model_module": "@jupyter-widgets/controls", - "model_name": "FloatProgressModel", - "model_module_version": "1.5.0", - "state": { - "_dom_classes": [], - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "FloatProgressModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/controls", - "_view_module_version": "1.5.0", - "_view_name": "ProgressView", - "bar_style": "success", - "description": "", - "description_tooltip": null, - "layout": "IPY_MODEL_ffb2ab66b3ca4d9899658fe58c43acb1", - "max": 570, - "min": 0, - "orientation": "horizontal", - "style": "IPY_MODEL_4b6aad944453432dbf957da380d059a1", - "value": 570 - } - }, - "6fee830ab9f545cea62081d8cf5b3240": { - "model_module": "@jupyter-widgets/controls", - "model_name": "HTMLModel", - "model_module_version": "1.5.0", - "state": { - "_dom_classes": [], - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "HTMLModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/controls", - "_view_module_version": "1.5.0", - "_view_name": "HTMLView", - "description": "", - "description_tooltip": null, - "layout": "IPY_MODEL_cf3dae1b960440d7984e9ed287b54cee", - "placeholder": "​", - "style": "IPY_MODEL_36254c0a04c840f3bf4038096c736873", - "value": " 570/570 [00:00<00:00, 33.8kB/s]" - } - }, - "cf024e76fe9b4766ac035f617391deb7": { - "model_module": "@jupyter-widgets/base", - "model_name": "LayoutModel", - "model_module_version": "1.2.0", - "state": { - "_model_module": "@jupyter-widgets/base", - "_model_module_version": "1.2.0", - "_model_name": "LayoutModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "LayoutView", - "align_content": null, - "align_items": null, - "align_self": null, - "border": null, - "bottom": null, - "display": null, - "flex": null, - "flex_flow": null, - "grid_area": null, - "grid_auto_columns": null, - "grid_auto_flow": null, - "grid_auto_rows": null, - "grid_column": null, - "grid_gap": null, - "grid_row": null, - "grid_template_areas": null, - "grid_template_columns": null, - "grid_template_rows": null, - "height": null, - "justify_content": null, - "justify_items": null, - "left": null, - "margin": null, - "max_height": null, - "max_width": null, - "min_height": null, - "min_width": null, - "object_fit": null, - "object_position": null, - "order": null, - "overflow": null, - "overflow_x": null, - "overflow_y": null, - "padding": null, - "right": null, - "top": null, - "visibility": null, - "width": null - } - }, - "fc0b17bfb44c45dd9825c6f1719b61cc": { - "model_module": "@jupyter-widgets/base", - "model_name": "LayoutModel", - "model_module_version": "1.2.0", - "state": { - "_model_module": "@jupyter-widgets/base", - "_model_module_version": "1.2.0", - "_model_name": "LayoutModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "LayoutView", - "align_content": null, - "align_items": null, - "align_self": null, - "border": null, - "bottom": null, - "display": null, - "flex": null, - "flex_flow": null, - "grid_area": null, - "grid_auto_columns": null, - "grid_auto_flow": null, - "grid_auto_rows": null, - "grid_column": null, - "grid_gap": null, - "grid_row": null, - "grid_template_areas": null, - "grid_template_columns": null, - "grid_template_rows": null, - "height": null, - "justify_content": null, - "justify_items": null, - "left": null, - "margin": null, - "max_height": null, - "max_width": null, - "min_height": null, - "min_width": null, - "object_fit": null, - "object_position": null, - "order": null, - "overflow": null, - "overflow_x": null, - "overflow_y": null, - "padding": null, - "right": null, - "top": null, - "visibility": null, - "width": null - } - }, - "2dac3a8089c34d0b9ce81653bde67603": { - "model_module": "@jupyter-widgets/controls", - "model_name": "DescriptionStyleModel", - "model_module_version": "1.5.0", - "state": { - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "DescriptionStyleModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "StyleView", - "description_width": "" - } - }, - "ffb2ab66b3ca4d9899658fe58c43acb1": { - "model_module": "@jupyter-widgets/base", - "model_name": "LayoutModel", - "model_module_version": "1.2.0", - "state": { - "_model_module": "@jupyter-widgets/base", - "_model_module_version": "1.2.0", - "_model_name": "LayoutModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "LayoutView", - "align_content": null, - "align_items": null, - "align_self": null, - "border": null, - "bottom": null, - "display": null, - "flex": null, - "flex_flow": null, - "grid_area": null, - "grid_auto_columns": null, - "grid_auto_flow": null, - "grid_auto_rows": null, - "grid_column": null, - "grid_gap": null, - "grid_row": null, - "grid_template_areas": null, - "grid_template_columns": null, - "grid_template_rows": null, - "height": null, - "justify_content": null, - "justify_items": null, - "left": null, - "margin": null, - "max_height": null, - "max_width": null, - "min_height": null, - "min_width": null, - "object_fit": null, - "object_position": null, - "order": null, - "overflow": null, - "overflow_x": null, - "overflow_y": null, - "padding": null, - "right": null, - "top": null, - "visibility": null, - "width": null - } - }, - "4b6aad944453432dbf957da380d059a1": { - "model_module": "@jupyter-widgets/controls", - "model_name": "ProgressStyleModel", - "model_module_version": "1.5.0", - "state": { - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "ProgressStyleModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "StyleView", - "bar_color": null, - "description_width": "" - } - }, - "cf3dae1b960440d7984e9ed287b54cee": { - "model_module": "@jupyter-widgets/base", - "model_name": "LayoutModel", - "model_module_version": "1.2.0", - "state": { - "_model_module": "@jupyter-widgets/base", - "_model_module_version": "1.2.0", - "_model_name": "LayoutModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "LayoutView", - "align_content": null, - "align_items": null, - "align_self": null, - "border": null, - "bottom": null, - "display": null, - "flex": null, - "flex_flow": null, - "grid_area": null, - "grid_auto_columns": null, - "grid_auto_flow": null, - "grid_auto_rows": null, - "grid_column": null, - "grid_gap": null, - "grid_row": null, - "grid_template_areas": null, - "grid_template_columns": null, - "grid_template_rows": null, - "height": null, - "justify_content": null, - "justify_items": null, - "left": null, - "margin": null, - "max_height": null, - "max_width": null, - "min_height": null, - "min_width": null, - "object_fit": null, - "object_position": null, - "order": null, - "overflow": null, - "overflow_x": null, - "overflow_y": null, - "padding": null, - "right": null, - "top": null, - "visibility": null, - "width": null - } - }, - "36254c0a04c840f3bf4038096c736873": { - "model_module": "@jupyter-widgets/controls", - "model_name": "DescriptionStyleModel", - "model_module_version": "1.5.0", - "state": { - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "DescriptionStyleModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "StyleView", - "description_width": "" - } - }, - "e3ffd50ee822433fabd9c1ee4a39612e": { - "model_module": "@jupyter-widgets/controls", - "model_name": "HBoxModel", - "model_module_version": "1.5.0", - "state": { - "_dom_classes": [], - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "HBoxModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/controls", - "_view_module_version": "1.5.0", - "_view_name": "HBoxView", - "box_style": "", - "children": [ - "IPY_MODEL_f248160605a2450a8411e4f5d58a5cfa", - "IPY_MODEL_a7b6e7aa521647649bb4157b6504d4e8", - "IPY_MODEL_1c1e86bca0534caaa7ad435fd7e67bf2" - ], - "layout": "IPY_MODEL_22ca9e6c6c1f4bc6b0f3db1a09a5e562" - } - }, - "f248160605a2450a8411e4f5d58a5cfa": { - "model_module": "@jupyter-widgets/controls", - "model_name": "HTMLModel", - "model_module_version": "1.5.0", - "state": { - "_dom_classes": [], - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "HTMLModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/controls", - "_view_module_version": "1.5.0", - "_view_name": "HTMLView", - "description": "", - "description_tooltip": null, - "layout": "IPY_MODEL_e9699698559c4860bdf6a312c492e7da", - "placeholder": "​", - "style": "IPY_MODEL_874e45fe39844927ad1fd10d4899a428", - "value": "tokenizer_config.json: 100%" - } - }, - "a7b6e7aa521647649bb4157b6504d4e8": { - "model_module": "@jupyter-widgets/controls", - "model_name": "FloatProgressModel", - "model_module_version": "1.5.0", - "state": { - "_dom_classes": [], - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "FloatProgressModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/controls", - "_view_module_version": "1.5.0", - "_view_name": "ProgressView", - "bar_style": "success", - "description": "", - "description_tooltip": null, - "layout": "IPY_MODEL_2dca6b2477344b45b1c17f124e27ce72", - "max": 49, - "min": 0, - "orientation": "horizontal", - "style": "IPY_MODEL_c7afefe6f907441b9e466605cb4f5c7f", - "value": 49 - } - }, - "1c1e86bca0534caaa7ad435fd7e67bf2": { - "model_module": "@jupyter-widgets/controls", - "model_name": "HTMLModel", - "model_module_version": "1.5.0", - "state": { - "_dom_classes": [], - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "HTMLModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/controls", - "_view_module_version": "1.5.0", - "_view_name": "HTMLView", - "description": "", - "description_tooltip": null, - "layout": "IPY_MODEL_df3c4c12e06245b1a8f4a4a7d71a530c", - "placeholder": "​", - "style": "IPY_MODEL_e63eb3e5c06140249f6d8d4c04fe8693", - "value": " 49.0/49.0 [00:00<00:00, 2.50kB/s]" - } - }, - "22ca9e6c6c1f4bc6b0f3db1a09a5e562": { - "model_module": "@jupyter-widgets/base", - "model_name": "LayoutModel", - "model_module_version": "1.2.0", - "state": { - "_model_module": "@jupyter-widgets/base", - "_model_module_version": "1.2.0", - "_model_name": "LayoutModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "LayoutView", - "align_content": null, - "align_items": null, - "align_self": null, - "border": null, - "bottom": null, - "display": null, - "flex": null, - "flex_flow": null, - "grid_area": null, - "grid_auto_columns": null, - "grid_auto_flow": null, - "grid_auto_rows": null, - "grid_column": null, - "grid_gap": null, - "grid_row": null, - "grid_template_areas": null, - "grid_template_columns": null, - "grid_template_rows": null, - "height": null, - "justify_content": null, - "justify_items": null, - "left": null, - "margin": null, - "max_height": null, - "max_width": null, - "min_height": null, - "min_width": null, - "object_fit": null, - "object_position": null, - "order": null, - "overflow": null, - "overflow_x": null, - "overflow_y": null, - "padding": null, - "right": null, - "top": null, - "visibility": null, - "width": null - } - }, - "e9699698559c4860bdf6a312c492e7da": { - "model_module": "@jupyter-widgets/base", - "model_name": "LayoutModel", - "model_module_version": "1.2.0", - "state": { - "_model_module": "@jupyter-widgets/base", - "_model_module_version": "1.2.0", - "_model_name": "LayoutModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "LayoutView", - "align_content": null, - "align_items": null, - "align_self": null, - "border": null, - "bottom": null, - "display": null, - "flex": null, - "flex_flow": null, - "grid_area": null, - "grid_auto_columns": null, - "grid_auto_flow": null, - "grid_auto_rows": null, - "grid_column": null, - "grid_gap": null, - "grid_row": null, - "grid_template_areas": null, - "grid_template_columns": null, - "grid_template_rows": null, - "height": null, - "justify_content": null, - "justify_items": null, - "left": null, - "margin": null, - "max_height": null, - "max_width": null, - "min_height": null, - "min_width": null, - "object_fit": null, - "object_position": null, - "order": null, - "overflow": null, - "overflow_x": null, - "overflow_y": null, - "padding": null, - "right": null, - "top": null, - "visibility": null, - "width": null - } - }, - "874e45fe39844927ad1fd10d4899a428": { - "model_module": "@jupyter-widgets/controls", - "model_name": "DescriptionStyleModel", - "model_module_version": "1.5.0", - "state": { - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "DescriptionStyleModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "StyleView", - "description_width": "" - } - }, - "2dca6b2477344b45b1c17f124e27ce72": { - "model_module": "@jupyter-widgets/base", - "model_name": "LayoutModel", - "model_module_version": "1.2.0", - "state": { - "_model_module": "@jupyter-widgets/base", - "_model_module_version": "1.2.0", - "_model_name": "LayoutModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "LayoutView", - "align_content": null, - "align_items": null, - "align_self": null, - "border": null, - "bottom": null, - "display": null, - "flex": null, - "flex_flow": null, - "grid_area": null, - "grid_auto_columns": null, - "grid_auto_flow": null, - "grid_auto_rows": null, - "grid_column": null, - "grid_gap": null, - "grid_row": null, - "grid_template_areas": null, - "grid_template_columns": null, - "grid_template_rows": null, - "height": null, - "justify_content": null, - "justify_items": null, - "left": null, - "margin": null, - "max_height": null, - "max_width": null, - "min_height": null, - "min_width": null, - "object_fit": null, - "object_position": null, - "order": null, - "overflow": null, - "overflow_x": null, - "overflow_y": null, - "padding": null, - "right": null, - "top": null, - "visibility": null, - "width": null - } - }, - "c7afefe6f907441b9e466605cb4f5c7f": { - "model_module": "@jupyter-widgets/controls", - "model_name": "ProgressStyleModel", - "model_module_version": "1.5.0", - "state": { - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "ProgressStyleModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "StyleView", - "bar_color": null, - "description_width": "" - } - }, - "df3c4c12e06245b1a8f4a4a7d71a530c": { - "model_module": "@jupyter-widgets/base", - "model_name": "LayoutModel", - "model_module_version": "1.2.0", - "state": { - "_model_module": "@jupyter-widgets/base", - "_model_module_version": "1.2.0", - "_model_name": "LayoutModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "LayoutView", - "align_content": null, - "align_items": null, - "align_self": null, - "border": null, - "bottom": null, - "display": null, - "flex": null, - "flex_flow": null, - "grid_area": null, - "grid_auto_columns": null, - "grid_auto_flow": null, - "grid_auto_rows": null, - "grid_column": null, - "grid_gap": null, - "grid_row": null, - "grid_template_areas": null, - "grid_template_columns": null, - "grid_template_rows": null, - "height": null, - "justify_content": null, - "justify_items": null, - "left": null, - "margin": null, - "max_height": null, - "max_width": null, - "min_height": null, - "min_width": null, - "object_fit": null, - "object_position": null, - "order": null, - "overflow": null, - "overflow_x": null, - "overflow_y": null, - "padding": null, - "right": null, - "top": null, - "visibility": null, - "width": null - } - }, - "e63eb3e5c06140249f6d8d4c04fe8693": { - "model_module": "@jupyter-widgets/controls", - "model_name": "DescriptionStyleModel", - "model_module_version": "1.5.0", - "state": { - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "DescriptionStyleModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "StyleView", - "description_width": "" - } - }, - "d951c3592058414ab00cf754e9b70685": { - "model_module": "@jupyter-widgets/controls", - "model_name": "HBoxModel", - "model_module_version": "1.5.0", - "state": { - "_dom_classes": [], - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "HBoxModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/controls", - "_view_module_version": "1.5.0", - "_view_name": "HBoxView", - "box_style": "", - "children": [ - "IPY_MODEL_c3f9146e082346a3bed274efb2265376", - "IPY_MODEL_1602c2298e9443f78007fdbf101a0c2b", - "IPY_MODEL_d5dda55bd4de4f12bf3718fb386c5bf9" - ], - "layout": "IPY_MODEL_943e61866ed74be4b10ba383450cb4c3" - } - }, - "c3f9146e082346a3bed274efb2265376": { - "model_module": "@jupyter-widgets/controls", - "model_name": "HTMLModel", - "model_module_version": "1.5.0", - "state": { - "_dom_classes": [], - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "HTMLModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/controls", - "_view_module_version": "1.5.0", - "_view_name": "HTMLView", - "description": "", - "description_tooltip": null, - "layout": "IPY_MODEL_0727d77eaf28466c93c2c6021661ac9a", - "placeholder": "​", - "style": "IPY_MODEL_3ab2e1a9ba694463ab5f3ec78ad0a8f4", - "value": "vocab.txt: 100%" - } - }, - "1602c2298e9443f78007fdbf101a0c2b": { - "model_module": "@jupyter-widgets/controls", - "model_name": "FloatProgressModel", - "model_module_version": "1.5.0", - "state": { - "_dom_classes": [], - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "FloatProgressModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/controls", - "_view_module_version": "1.5.0", - "_view_name": "ProgressView", - "bar_style": "success", - "description": "", - "description_tooltip": null, - "layout": "IPY_MODEL_30b4db204fa644128198abf6d82664bf", - "max": 213450, - "min": 0, - "orientation": "horizontal", - "style": "IPY_MODEL_0517fdac88784fe6b51ad1f989f99cb7", - "value": 213450 - } - }, - "d5dda55bd4de4f12bf3718fb386c5bf9": { - "model_module": "@jupyter-widgets/controls", - "model_name": "HTMLModel", - "model_module_version": "1.5.0", - "state": { - "_dom_classes": [], - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "HTMLModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/controls", - "_view_module_version": "1.5.0", - "_view_name": "HTMLView", - "description": "", - "description_tooltip": null, - "layout": "IPY_MODEL_ce1c55bc51dc49a9b261f104f49d38d8", - "placeholder": "​", - "style": "IPY_MODEL_4b7e11bb32bc43c9ad0449bd39bc4d40", - "value": " 213k/213k [00:00<00:00, 613kB/s]" - } - }, - "943e61866ed74be4b10ba383450cb4c3": { - "model_module": "@jupyter-widgets/base", - "model_name": "LayoutModel", - "model_module_version": "1.2.0", - "state": { - "_model_module": "@jupyter-widgets/base", - "_model_module_version": "1.2.0", - "_model_name": "LayoutModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "LayoutView", - "align_content": null, - "align_items": null, - "align_self": null, - "border": null, - "bottom": null, - "display": null, - "flex": null, - "flex_flow": null, - "grid_area": null, - "grid_auto_columns": null, - "grid_auto_flow": null, - "grid_auto_rows": null, - "grid_column": null, - "grid_gap": null, - "grid_row": null, - "grid_template_areas": null, - "grid_template_columns": null, - "grid_template_rows": null, - "height": null, - "justify_content": null, - "justify_items": null, - "left": null, - "margin": null, - "max_height": null, - "max_width": null, - "min_height": null, - "min_width": null, - "object_fit": null, - "object_position": null, - "order": null, - "overflow": null, - "overflow_x": null, - "overflow_y": null, - "padding": null, - "right": null, - "top": null, - "visibility": null, - "width": null - } - }, - "0727d77eaf28466c93c2c6021661ac9a": { - "model_module": "@jupyter-widgets/base", - "model_name": "LayoutModel", - "model_module_version": "1.2.0", - "state": { - "_model_module": "@jupyter-widgets/base", - "_model_module_version": "1.2.0", - "_model_name": "LayoutModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "LayoutView", - "align_content": null, - "align_items": null, - "align_self": null, - "border": null, - "bottom": null, - "display": null, - "flex": null, - "flex_flow": null, - "grid_area": null, - "grid_auto_columns": null, - "grid_auto_flow": null, - "grid_auto_rows": null, - "grid_column": null, - "grid_gap": null, - "grid_row": null, - "grid_template_areas": null, - "grid_template_columns": null, - "grid_template_rows": null, - "height": null, - "justify_content": null, - "justify_items": null, - "left": null, - "margin": null, - "max_height": null, - "max_width": null, - "min_height": null, - "min_width": null, - "object_fit": null, - "object_position": null, - "order": null, - "overflow": null, - "overflow_x": null, - "overflow_y": null, - "padding": null, - "right": null, - "top": null, - "visibility": null, - "width": null - } - }, - "3ab2e1a9ba694463ab5f3ec78ad0a8f4": { - "model_module": "@jupyter-widgets/controls", - "model_name": "DescriptionStyleModel", - "model_module_version": "1.5.0", - "state": { - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "DescriptionStyleModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "StyleView", - "description_width": "" - } - }, - "30b4db204fa644128198abf6d82664bf": { - "model_module": "@jupyter-widgets/base", - "model_name": "LayoutModel", - "model_module_version": "1.2.0", - "state": { - "_model_module": "@jupyter-widgets/base", - "_model_module_version": "1.2.0", - "_model_name": "LayoutModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "LayoutView", - "align_content": null, - "align_items": null, - "align_self": null, - "border": null, - "bottom": null, - "display": null, - "flex": null, - "flex_flow": null, - "grid_area": null, - "grid_auto_columns": null, - "grid_auto_flow": null, - "grid_auto_rows": null, - "grid_column": null, - "grid_gap": null, - "grid_row": null, - "grid_template_areas": null, - "grid_template_columns": null, - "grid_template_rows": null, - "height": null, - "justify_content": null, - "justify_items": null, - "left": null, - "margin": null, - "max_height": null, - "max_width": null, - "min_height": null, - "min_width": null, - "object_fit": null, - "object_position": null, - "order": null, - "overflow": null, - "overflow_x": null, - "overflow_y": null, - "padding": null, - "right": null, - "top": null, - "visibility": null, - "width": null - } - }, - "0517fdac88784fe6b51ad1f989f99cb7": { - "model_module": "@jupyter-widgets/controls", - "model_name": "ProgressStyleModel", - "model_module_version": "1.5.0", - "state": { - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "ProgressStyleModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "StyleView", - "bar_color": null, - "description_width": "" - } - }, - "ce1c55bc51dc49a9b261f104f49d38d8": { - "model_module": "@jupyter-widgets/base", - "model_name": "LayoutModel", - "model_module_version": "1.2.0", - "state": { - "_model_module": "@jupyter-widgets/base", - "_model_module_version": "1.2.0", - "_model_name": "LayoutModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "LayoutView", - "align_content": null, - "align_items": null, - "align_self": null, - "border": null, - "bottom": null, - "display": null, - "flex": null, - "flex_flow": null, - "grid_area": null, - "grid_auto_columns": null, - "grid_auto_flow": null, - "grid_auto_rows": null, - "grid_column": null, - "grid_gap": null, - "grid_row": null, - "grid_template_areas": null, - "grid_template_columns": null, - "grid_template_rows": null, - "height": null, - "justify_content": null, - "justify_items": null, - "left": null, - "margin": null, - "max_height": null, - "max_width": null, - "min_height": null, - "min_width": null, - "object_fit": null, - "object_position": null, - "order": null, - "overflow": null, - "overflow_x": null, - "overflow_y": null, - "padding": null, - "right": null, - "top": null, - "visibility": null, - "width": null - } - }, - "4b7e11bb32bc43c9ad0449bd39bc4d40": { - "model_module": "@jupyter-widgets/controls", - "model_name": "DescriptionStyleModel", - "model_module_version": "1.5.0", - "state": { - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "DescriptionStyleModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "StyleView", - "description_width": "" - } - }, - "3dbf36a5c0884579ab2f36c2e91c04fb": { - "model_module": "@jupyter-widgets/controls", - "model_name": "HBoxModel", - "model_module_version": "1.5.0", - "state": { - "_dom_classes": [], - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "HBoxModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/controls", - "_view_module_version": "1.5.0", - "_view_name": "HBoxView", - "box_style": "", - "children": [ - "IPY_MODEL_d71c49bd9f8c438898250c3874c06240", - "IPY_MODEL_1b70fa16b803466ea31649dcd644e3d7", - "IPY_MODEL_7a728bea623646c182c508e34b582fc9" - ], - "layout": "IPY_MODEL_8494caffe83a4743a11d2751b38c56bb" - } - }, - "d71c49bd9f8c438898250c3874c06240": { - "model_module": "@jupyter-widgets/controls", - "model_name": "HTMLModel", - "model_module_version": "1.5.0", - "state": { - "_dom_classes": [], - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "HTMLModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/controls", - "_view_module_version": "1.5.0", - "_view_name": "HTMLView", - "description": "", - "description_tooltip": null, - "layout": "IPY_MODEL_1580380472df40e38cbde67659c5221d", - "placeholder": "​", - "style": "IPY_MODEL_349b262479b9418badd6a3acff386dd2", - "value": "tokenizer.json: 100%" - } - }, - "1b70fa16b803466ea31649dcd644e3d7": { - "model_module": "@jupyter-widgets/controls", - "model_name": "FloatProgressModel", - "model_module_version": "1.5.0", - "state": { - "_dom_classes": [], - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "FloatProgressModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/controls", - "_view_module_version": "1.5.0", - "_view_name": "ProgressView", - "bar_style": "success", - "description": "", - "description_tooltip": null, - "layout": "IPY_MODEL_3370a9d70dd04d5195bc3f1f81b18728", - "max": 435797, - "min": 0, - "orientation": "horizontal", - "style": "IPY_MODEL_ee235d5ffc5142c895175cbea5c94dfe", - "value": 435797 - } - }, - "7a728bea623646c182c508e34b582fc9": { - "model_module": "@jupyter-widgets/controls", - "model_name": "HTMLModel", - "model_module_version": "1.5.0", - "state": { - "_dom_classes": [], - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "HTMLModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/controls", - "_view_module_version": "1.5.0", - "_view_name": "HTMLView", - "description": "", - "description_tooltip": null, - "layout": "IPY_MODEL_f6c5ef333a2b45f1bf196fdd58873688", - "placeholder": "​", - "style": "IPY_MODEL_e24ff9dae78241d9b5a6a7199c888e45", - "value": " 436k/436k [00:00<00:00, 1.24MB/s]" - } - }, - "8494caffe83a4743a11d2751b38c56bb": { - "model_module": "@jupyter-widgets/base", - "model_name": "LayoutModel", - "model_module_version": "1.2.0", - "state": { - "_model_module": "@jupyter-widgets/base", - "_model_module_version": "1.2.0", - "_model_name": "LayoutModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "LayoutView", - "align_content": null, - "align_items": null, - "align_self": null, - "border": null, - "bottom": null, - "display": null, - "flex": null, - "flex_flow": null, - "grid_area": null, - "grid_auto_columns": null, - "grid_auto_flow": null, - "grid_auto_rows": null, - "grid_column": null, - "grid_gap": null, - "grid_row": null, - "grid_template_areas": null, - "grid_template_columns": null, - "grid_template_rows": null, - "height": null, - "justify_content": null, - "justify_items": null, - "left": null, - "margin": null, - "max_height": null, - "max_width": null, - "min_height": null, - "min_width": null, - "object_fit": null, - "object_position": null, - "order": null, - "overflow": null, - "overflow_x": null, - "overflow_y": null, - "padding": null, - "right": null, - "top": null, - "visibility": null, - "width": null - } - }, - "1580380472df40e38cbde67659c5221d": { - "model_module": "@jupyter-widgets/base", - "model_name": "LayoutModel", - "model_module_version": "1.2.0", - "state": { - "_model_module": "@jupyter-widgets/base", - "_model_module_version": "1.2.0", - "_model_name": "LayoutModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "LayoutView", - "align_content": null, - "align_items": null, - "align_self": null, - "border": null, - "bottom": null, - "display": null, - "flex": null, - "flex_flow": null, - "grid_area": null, - "grid_auto_columns": null, - "grid_auto_flow": null, - "grid_auto_rows": null, - "grid_column": null, - "grid_gap": null, - "grid_row": null, - "grid_template_areas": null, - "grid_template_columns": null, - "grid_template_rows": null, - "height": null, - "justify_content": null, - "justify_items": null, - "left": null, - "margin": null, - "max_height": null, - "max_width": null, - "min_height": null, - "min_width": null, - "object_fit": null, - "object_position": null, - "order": null, - "overflow": null, - "overflow_x": null, - "overflow_y": null, - "padding": null, - "right": null, - "top": null, - "visibility": null, - "width": null - } - }, - "349b262479b9418badd6a3acff386dd2": { - "model_module": "@jupyter-widgets/controls", - "model_name": "DescriptionStyleModel", - "model_module_version": "1.5.0", - "state": { - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "DescriptionStyleModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "StyleView", - "description_width": "" - } - }, - "3370a9d70dd04d5195bc3f1f81b18728": { - "model_module": "@jupyter-widgets/base", - "model_name": "LayoutModel", - "model_module_version": "1.2.0", - "state": { - "_model_module": "@jupyter-widgets/base", - "_model_module_version": "1.2.0", - "_model_name": "LayoutModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "LayoutView", - "align_content": null, - "align_items": null, - "align_self": null, - "border": null, - "bottom": null, - "display": null, - "flex": null, - "flex_flow": null, - "grid_area": null, - "grid_auto_columns": null, - "grid_auto_flow": null, - "grid_auto_rows": null, - "grid_column": null, - "grid_gap": null, - "grid_row": null, - "grid_template_areas": null, - "grid_template_columns": null, - "grid_template_rows": null, - "height": null, - "justify_content": null, - "justify_items": null, - "left": null, - "margin": null, - "max_height": null, - "max_width": null, - "min_height": null, - "min_width": null, - "object_fit": null, - "object_position": null, - "order": null, - "overflow": null, - "overflow_x": null, - "overflow_y": null, - "padding": null, - "right": null, - "top": null, - "visibility": null, - "width": null - } - }, - "ee235d5ffc5142c895175cbea5c94dfe": { - "model_module": "@jupyter-widgets/controls", - "model_name": "ProgressStyleModel", - "model_module_version": "1.5.0", - "state": { - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "ProgressStyleModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "StyleView", - "bar_color": null, - "description_width": "" - } - }, - "f6c5ef333a2b45f1bf196fdd58873688": { - "model_module": "@jupyter-widgets/base", - "model_name": "LayoutModel", - "model_module_version": "1.2.0", - "state": { - "_model_module": "@jupyter-widgets/base", - "_model_module_version": "1.2.0", - "_model_name": "LayoutModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "LayoutView", - "align_content": null, - "align_items": null, - "align_self": null, - "border": null, - "bottom": null, - "display": null, - "flex": null, - "flex_flow": null, - "grid_area": null, - "grid_auto_columns": null, - "grid_auto_flow": null, - "grid_auto_rows": null, - "grid_column": null, - "grid_gap": null, - "grid_row": null, - "grid_template_areas": null, - "grid_template_columns": null, - "grid_template_rows": null, - "height": null, - "justify_content": null, - "justify_items": null, - "left": null, - "margin": null, - "max_height": null, - "max_width": null, - "min_height": null, - "min_width": null, - "object_fit": null, - "object_position": null, - "order": null, - "overflow": null, - "overflow_x": null, - "overflow_y": null, - "padding": null, - "right": null, - "top": null, - "visibility": null, - "width": null - } - }, - "e24ff9dae78241d9b5a6a7199c888e45": { - "model_module": "@jupyter-widgets/controls", - "model_name": "DescriptionStyleModel", - "model_module_version": "1.5.0", - "state": { - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "DescriptionStyleModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "StyleView", - "description_width": "" - } - }, - "d4a768f261614ac69b3004fbf2323c89": { - "model_module": "@jupyter-widgets/controls", - "model_name": "HBoxModel", - "model_module_version": "1.5.0", - "state": { - "_dom_classes": [], - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "HBoxModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/controls", - "_view_module_version": "1.5.0", - "_view_name": "HBoxView", - "box_style": "", - "children": [ - "IPY_MODEL_81a7d1b27cd94916ac3c330aa2551cf0", - "IPY_MODEL_eac4d3f8e59a4d4c81178cc76600182f", - "IPY_MODEL_5bcca2ee852144c28bfd40de0978cadc" - ], - "layout": "IPY_MODEL_05fa1e761bc14c929b19abf0d8a93f5f" - } - }, - "81a7d1b27cd94916ac3c330aa2551cf0": { - "model_module": "@jupyter-widgets/controls", - "model_name": "HTMLModel", - "model_module_version": "1.5.0", - "state": { - "_dom_classes": [], - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "HTMLModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/controls", - "_view_module_version": "1.5.0", - "_view_name": "HTMLView", - "description": "", - "description_tooltip": null, - "layout": "IPY_MODEL_32ba8daa0c6e4a9c9f62df588a198b1b", - "placeholder": "​", - "style": "IPY_MODEL_e1e93f905b494126a6a9e1e0a2f92022", - "value": "model.safetensors: 100%" - } - }, - "eac4d3f8e59a4d4c81178cc76600182f": { - "model_module": "@jupyter-widgets/controls", - "model_name": "FloatProgressModel", - "model_module_version": "1.5.0", - "state": { - "_dom_classes": [], - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "FloatProgressModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/controls", - "_view_module_version": "1.5.0", - "_view_name": "ProgressView", - "bar_style": "success", - "description": "", - "description_tooltip": null, - "layout": "IPY_MODEL_9ca2cb3f116547d3bb062f4da11762d7", - "max": 435755784, - "min": 0, - "orientation": "horizontal", - "style": "IPY_MODEL_554184f1c9b44bd3a8116773572347eb", - "value": 435755784 - } - }, - "5bcca2ee852144c28bfd40de0978cadc": { - "model_module": "@jupyter-widgets/controls", - "model_name": "HTMLModel", - "model_module_version": "1.5.0", - "state": { - "_dom_classes": [], - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "HTMLModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/controls", - "_view_module_version": "1.5.0", - "_view_name": "HTMLView", - "description": "", - "description_tooltip": null, - "layout": "IPY_MODEL_bb89b007667e4bf0b696ad84dfb2f91d", - "placeholder": "​", - "style": "IPY_MODEL_610f073056924398b9229978dff5ff4d", - "value": " 436M/436M [00:02<00:00, 177MB/s]" - } - }, - "05fa1e761bc14c929b19abf0d8a93f5f": { - "model_module": "@jupyter-widgets/base", - "model_name": "LayoutModel", - "model_module_version": "1.2.0", - "state": { - "_model_module": "@jupyter-widgets/base", - "_model_module_version": "1.2.0", - "_model_name": "LayoutModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "LayoutView", - "align_content": null, - "align_items": null, - "align_self": null, - "border": null, - "bottom": null, - "display": null, - "flex": null, - "flex_flow": null, - "grid_area": null, - "grid_auto_columns": null, - "grid_auto_flow": null, - "grid_auto_rows": null, - "grid_column": null, - "grid_gap": null, - "grid_row": null, - "grid_template_areas": null, - "grid_template_columns": null, - "grid_template_rows": null, - "height": null, - "justify_content": null, - "justify_items": null, - "left": null, - "margin": null, - "max_height": null, - "max_width": null, - "min_height": null, - "min_width": null, - "object_fit": null, - "object_position": null, - "order": null, - "overflow": null, - "overflow_x": null, - "overflow_y": null, - "padding": null, - "right": null, - "top": null, - "visibility": null, - "width": null - } - }, - "32ba8daa0c6e4a9c9f62df588a198b1b": { - "model_module": "@jupyter-widgets/base", - "model_name": "LayoutModel", - "model_module_version": "1.2.0", - "state": { - "_model_module": "@jupyter-widgets/base", - "_model_module_version": "1.2.0", - "_model_name": "LayoutModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "LayoutView", - "align_content": null, - "align_items": null, - "align_self": null, - "border": null, - "bottom": null, - "display": null, - "flex": null, - "flex_flow": null, - "grid_area": null, - "grid_auto_columns": null, - "grid_auto_flow": null, - "grid_auto_rows": null, - "grid_column": null, - "grid_gap": null, - "grid_row": null, - "grid_template_areas": null, - "grid_template_columns": null, - "grid_template_rows": null, - "height": null, - "justify_content": null, - "justify_items": null, - "left": null, - "margin": null, - "max_height": null, - "max_width": null, - "min_height": null, - "min_width": null, - "object_fit": null, - "object_position": null, - "order": null, - "overflow": null, - "overflow_x": null, - "overflow_y": null, - "padding": null, - "right": null, - "top": null, - "visibility": null, - "width": null - } - }, - "e1e93f905b494126a6a9e1e0a2f92022": { - "model_module": "@jupyter-widgets/controls", - "model_name": "DescriptionStyleModel", - "model_module_version": "1.5.0", - "state": { - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "DescriptionStyleModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "StyleView", - "description_width": "" - } - }, - "9ca2cb3f116547d3bb062f4da11762d7": { - "model_module": "@jupyter-widgets/base", - "model_name": "LayoutModel", - "model_module_version": "1.2.0", - "state": { - "_model_module": "@jupyter-widgets/base", - "_model_module_version": "1.2.0", - "_model_name": "LayoutModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "LayoutView", - "align_content": null, - "align_items": null, - "align_self": null, - "border": null, - "bottom": null, - "display": null, - "flex": null, - "flex_flow": null, - "grid_area": null, - "grid_auto_columns": null, - "grid_auto_flow": null, - "grid_auto_rows": null, - "grid_column": null, - "grid_gap": null, - "grid_row": null, - "grid_template_areas": null, - "grid_template_columns": null, - "grid_template_rows": null, - "height": null, - "justify_content": null, - "justify_items": null, - "left": null, - "margin": null, - "max_height": null, - "max_width": null, - "min_height": null, - "min_width": null, - "object_fit": null, - "object_position": null, - "order": null, - "overflow": null, - "overflow_x": null, - "overflow_y": null, - "padding": null, - "right": null, - "top": null, - "visibility": null, - "width": null - } - }, - "554184f1c9b44bd3a8116773572347eb": { - "model_module": "@jupyter-widgets/controls", - "model_name": "ProgressStyleModel", - "model_module_version": "1.5.0", - "state": { - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "ProgressStyleModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "StyleView", - "bar_color": null, - "description_width": "" - } - }, - "bb89b007667e4bf0b696ad84dfb2f91d": { - "model_module": "@jupyter-widgets/base", - "model_name": "LayoutModel", - "model_module_version": "1.2.0", - "state": { - "_model_module": "@jupyter-widgets/base", - "_model_module_version": "1.2.0", - "_model_name": "LayoutModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "LayoutView", - "align_content": null, - "align_items": null, - "align_self": null, - "border": null, - "bottom": null, - "display": null, - "flex": null, - "flex_flow": null, - "grid_area": null, - "grid_auto_columns": null, - "grid_auto_flow": null, - "grid_auto_rows": null, - "grid_column": null, - "grid_gap": null, - "grid_row": null, - "grid_template_areas": null, - "grid_template_columns": null, - "grid_template_rows": null, - "height": null, - "justify_content": null, - "justify_items": null, - "left": null, - "margin": null, - "max_height": null, - "max_width": null, - "min_height": null, - "min_width": null, - "object_fit": null, - "object_position": null, - "order": null, - "overflow": null, - "overflow_x": null, - "overflow_y": null, - "padding": null, - "right": null, - "top": null, - "visibility": null, - "width": null - } - }, - "610f073056924398b9229978dff5ff4d": { - "model_module": "@jupyter-widgets/controls", - "model_name": "DescriptionStyleModel", - "model_module_version": "1.5.0", - "state": { - "_model_module": "@jupyter-widgets/controls", - "_model_module_version": "1.5.0", - "_model_name": "DescriptionStyleModel", - "_view_count": null, - "_view_module": "@jupyter-widgets/base", - "_view_module_version": "1.2.0", - "_view_name": "StyleView", - "description_width": "" - } - } - } - } + "accelerator": "GPU" }, "nbformat": 4, "nbformat_minor": 0, diff --git a/website/package.json b/website/package.json index 9dd4fb7d..b1b47913 100644 --- a/website/package.json +++ b/website/package.json @@ -9,21 +9,23 @@ "rename-version": "docusaurus-rename-version" }, "devDependencies": { - "docusaurus": "^1.9.0" + "docusaurus": "^1.14.7" }, "dependencies": { - "prismjs": "^1.23.0", - "bl": "^1.2.3" + "@babel/helper-compilation-targets": "^8.0.0-alpha.14", + "bl": "^5.0.0", + "browserslist": "^4.21.4", + "prismjs": "^1.29.0" }, "resolutions": { - "trim-newlines": "3.0.1", - "normalize-url": "4.5.1", - "highlight.js" : "10.5.0", - "react-dev-utils": "11.0.4", - "immer": "8.0.1", - "prismjs": "1.23.0", - "bl": "1.2.3", - "glob-parent": "5.1.2", - "browserslist": "4.16.5" + "trim-newlines": "^4.0.2", + "normalize-url": "^6.1.0", + "highlight.js": "^11.8.0", + "react-dev-utils": "^12.0.0", + "immer": "^10.0.0", + "prismjs": "^1.29.0", + "bl": "^5.0.0", + "glob-parent": "^6.0.2", + "browserslist": "^4.21.4" } } From 3934851902599ad915981055506fecf57cf298dd Mon Sep 17 00:00:00 2001 From: Enayat Ullah Date: Thu, 9 Jan 2025 18:04:17 -0800 Subject: [PATCH 10/10] Generate the status badge on Github using Github Actions (#712) Summary: Pull Request resolved: https://github.com/pytorch/opacus/pull/712 Since CircleCI is disabled, we now use tests from Github Actions to indicate its status on the Github main page. Reviewed By: iden-kalemaj Differential Revision: D67990187 fbshipit-source-id: 058268cad1cafd3abf319a60229c1ee24752f033 --- README.md | 71 +++++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 50 insertions(+), 21 deletions(-) diff --git a/README.md b/README.md index a671d35c..c46147a6 100644 --- a/README.md +++ b/README.md @@ -2,32 +2,43 @@
-[![CircleCI](https://dl.circleci.com/status-badge/img/gh/pytorch/opacus/tree/main.svg?style=svg)](https://dl.circleci.com/status-badge/redirect/gh/pytorch/opacus/tree/main) +[![GitHub Actions](https://github.com/pytorch/opacus/actions/workflows/ci_cpu.yml/badge.svg)](https://github.com/pytorch/opacus/actions/workflows/ci_cpu.yml) [![Coverage Status](https://coveralls.io/repos/github/pytorch/opacus/badge.svg?branch=main)](https://coveralls.io/github/pytorch/opacus?branch=main) [![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](CONTRIBUTING.md) [![License](https://img.shields.io/badge/license-apache2-green.svg)](LICENSE) -[Opacus](https://opacus.ai) is a library that enables training PyTorch models with differential privacy. -It supports training with minimal code changes required on the client, has little impact on training performance, and allows the client to online track the privacy budget expended at any given moment. +[Opacus](https://opacus.ai) is a library that enables training PyTorch models +with differential privacy. It supports training with minimal code changes +required on the client, has little impact on training performance, and allows +the client to online track the privacy budget expended at any given moment. ## Target audience + This code release is aimed at two target audiences: -1. ML practitioners will find this to be a gentle introduction to training a model with differential privacy as it requires minimal code changes. -2. Differential Privacy researchers will find this easy to experiment and tinker with, allowing them to focus on what matters. +1. ML practitioners will find this to be a gentle introduction to training a + model with differential privacy as it requires minimal code changes. +2. Differential Privacy researchers will find this easy to experiment and tinker + with, allowing them to focus on what matters. ## Installation + The latest release of Opacus can be installed via `pip`: + ```bash pip install opacus ``` + OR, alternatively, via `conda`: + ```bash conda install -c conda-forge opacus ``` -You can also install directly from the source for the latest features (along with its quirks and potentially occasional bugs): +You can also install directly from the source for the latest features (along +with its quirks and potentially occasional bugs): + ```bash git clone https://github.com/pytorch/opacus.git cd opacus @@ -35,7 +46,10 @@ pip install -e . ``` ## Getting started -To train your model with differential privacy, all you need to do is to instantiate a `PrivacyEngine` and pass your model, data_loader, and optimizer to the engine's `make_private()` method to obtain their private counterparts. + +To train your model with differential privacy, all you need to do is to +instantiate a `PrivacyEngine` and pass your model, data_loader, and optimizer to +the engine's `make_private()` method to obtain their private counterparts. ```python # define your components as usual @@ -55,21 +69,25 @@ model, optimizer, data_loader = privacy_engine.make_private( # Now it's business as usual ``` -The [MNIST example](https://github.com/pytorch/opacus/tree/main/examples/mnist.py) shows an end-to-end run using Opacus. The [examples](https://github.com/pytorch/opacus/tree/main/examples/) folder contains more such examples. +The +[MNIST example](https://github.com/pytorch/opacus/tree/main/examples/mnist.py) +shows an end-to-end run using Opacus. The +[examples](https://github.com/pytorch/opacus/tree/main/examples/) folder +contains more such examples. ### Migrating to 1.0 -Opacus 1.0 introduced many improvements to the library, but also some breaking changes. -If you've been using Opacus 0.x and want to update to the latest release, -please use this [Migration Guide](https://github.com/pytorch/opacus/blob/main/Migration_Guide.md) - +Opacus 1.0 introduced many improvements to the library, but also some breaking +changes. If you've been using Opacus 0.x and want to update to the latest +release, please use this +[Migration Guide](https://github.com/pytorch/opacus/blob/main/Migration_Guide.md) ## Learn more ### Interactive tutorials -We've built a series of IPython-based tutorials as a gentle introduction to training models -with privacy and using various Opacus features. +We've built a series of IPython-based tutorials as a gentle introduction to +training models with privacy and using various Opacus features. - [Building an Image Classifier with Differential Privacy](https://github.com/pytorch/opacus/blob/main/tutorials/building_image_classifier.ipynb) - [Training a differentially private LSTM model for name classification](https://github.com/pytorch/opacus/blob/main/tutorials/building_lstm_name_classifier.ipynb) @@ -79,9 +97,13 @@ with privacy and using various Opacus features. - [Opacus Guide: Module Validator and Fixer](https://github.com/pytorch/opacus/blob/main/tutorials/guide_to_module_validator.ipynb) ## Technical report and citation -The technical report introducing Opacus, presenting its design principles, mathematical foundations, and benchmarks can be found [here](https://arxiv.org/abs/2109.12298). + +The technical report introducing Opacus, presenting its design principles, +mathematical foundations, and benchmarks can be found +[here](https://arxiv.org/abs/2109.12298). Consider citing the report if you use Opacus in your papers, as follows: + ``` @article{opacus, title={Opacus: {U}ser-Friendly Differential Privacy Library in {PyTorch}}, @@ -93,7 +115,8 @@ Consider citing the report if you use Opacus in your papers, as follows: ### Blogposts and talks -If you want to learn more about DP-SGD and related topics, check out our series of blogposts and talks: +If you want to learn more about DP-SGD and related topics, check out our series +of blogposts and talks: - [Differential Privacy Series Part 1 | DP-SGD Algorithm Explained](https://medium.com/pytorch/differential-privacy-series-part-1-dp-sgd-algorithm-explained-12512c3959a3) - [Differential Privacy Series Part 2 | Efficient Per-Sample Gradient Computation in Opacus](https://medium.com/pytorch/differential-privacy-series-part-2-efficient-per-sample-gradient-computation-in-opacus-5bf4031d9e22) @@ -102,13 +125,19 @@ If you want to learn more about DP-SGD and related topics, check out our series - [Opacus v1.0 Highlights | PyTorch Developer Day 2021](https://www.youtube.com/watch?v=U1mszp8lzUI) - [Enabling Fast Gradient Clipping and Ghost Clipping in Opacus](https://pytorch.org/blog/clipping-in-opacus/) - ## FAQ -Check out the [FAQ](https://opacus.ai/docs/faq) page for answers to some of the most frequently asked questions about differential privacy and Opacus. + +Check out the [FAQ](https://opacus.ai/docs/faq) page for answers to some of the +most frequently asked questions about differential privacy and Opacus. ## Contributing -See the [CONTRIBUTING](https://github.com/pytorch/opacus/tree/main/CONTRIBUTING.md) file for how to help out. -Do also check out the README files inside the repo to learn how the code is organized. + +See the +[CONTRIBUTING](https://github.com/pytorch/opacus/tree/main/CONTRIBUTING.md) file +for how to help out. Do also check out the README files inside the repo to learn +how the code is organized. ## License -This code is released under Apache 2.0, as found in the [LICENSE](https://github.com/pytorch/opacus/tree/main/LICENSE) file. + +This code is released under Apache 2.0, as found in the +[LICENSE](https://github.com/pytorch/opacus/tree/main/LICENSE) file.