Skip to content

Commit

Permalink
Merge pull request #8 from Sharing-Sam-Work/getting-started
Browse files Browse the repository at this point in the history 
Minor correction to documentation (reformulations to clarify)
  • Loading branch information
@Sharing-Sam-Work
Sharing-Sam-Work committed Aug 9, 2023
2 parents c625d2e + 5f39af2 commit 71e714b
Show file tree
Hide file tree
Showing 2 changed files with 186 additions and 65 deletions.
138 changes: 114 additions & 24 deletions docs/notebooks/Getting_started_1.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
"\n",
"In this first notebook, our objective is to show how to create 1-Lipschitz neural networks with `deel-lip`. \n",
"\n",
"In the course of this notebook, we will cover the following: \n",
"In particular, we will cover the following: \n",
"1. [📚 Theoretical background](#theoretical_background) \n",
"A brief theoretical background on Lipschitz continuous functions. This section can be safely skipped if one is not interested in the theory.\n",
"2. [🧱 Creating a 1-Lipschitz neural network with `deel-lip` and `keras`](#deel_keras) \n",
Expand Down Expand Up @@ -65,7 +65,7 @@
},
{
"cell_type": "code",
"execution_count": 2,
"execution_count": 1,
"id": "1e3f0694-8547-4d06-b2aa-bfc0d008ff8b",
"metadata": {
"scrolled": true
Expand Down Expand Up @@ -126,10 +126,34 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 2,
"id": "d6f7099e-a425-452d-9ed5-5328f0258a71",
"metadata": {},
"outputs": [],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Model: \"model\"\n",
"_________________________________________________________________\n",
" Layer (type) Output Shape Param # \n",
"=================================================================\n",
" input_2 (InputLayer) [(None, 28, 28, 1)] 0 \n",
" \n",
" dense_3 (Dense) (None, 28, 28, 64) 128 \n",
" \n",
" dense_4 (Dense) (None, 28, 28, 32) 2080 \n",
" \n",
" dense_5 (Dense) (None, 28, 28, 10) 330 \n",
" \n",
"=================================================================\n",
"Total params: 2538 (9.91 KB)\n",
"Trainable params: 2538 (9.91 KB)\n",
"Non-trainable params: 0 (0.00 Byte)\n",
"_________________________________________________________________\n"
]
}
],
"source": [
"inputs = keras.layers.Input(input_shape)\n",
"x = layers.Dense(64, activation='relu')(inputs)\n",
Expand All @@ -151,7 +175,7 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 3,
"id": "a62d3a3f-c0e8-4a3e-9025-758afebf99bd",
"metadata": {},
"outputs": [],
Expand All @@ -164,10 +188,43 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 4,
"id": "c26f52a6-c2ec-49b5-a99f-3353dc3f3044",
"metadata": {},
"outputs": [],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Model: \"sequential_1\"\n",
"_________________________________________________________________\n",
" Layer (type) Output Shape Param # \n",
"=================================================================\n",
" spectral_dense (SpectralDe (None, 28, 28, 64) 257 \n",
" nse) \n",
" \n",
" spectral_dense_1 (Spectral (None, 28, 28, 32) 4161 \n",
" Dense) \n",
" \n",
" spectral_dense_2 (Spectral (None, 28, 28, 10) 661 \n",
" Dense) \n",
" \n",
"=================================================================\n",
"Total params: 5079 (19.84 KB)\n",
"Trainable params: 2538 (9.91 KB)\n",
"Non-trainable params: 2541 (9.93 KB)\n",
"_________________________________________________________________\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"C:\\Users\\kierszbaums\\anaconda.related\\envs\\1_lipschitz\\deel_lip\\lib\\site-packages\\keras\\src\\initializers\\initializers.py:120: UserWarning: The initializer Orthogonal is unseeded and being called multiple times, which will return identical values each time (even if the initializer is unseeded). Please update your code to provide a seed to the initializer, or avoid using the same initializer instance more than once.\n",
" warnings.warn(\n"
]
}
],
"source": [
"K1_model = lip.model.Sequential([ \n",
" keras.layers.Input(shape=input_shape),\n",
Expand Down Expand Up @@ -197,10 +254,37 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 5,
"id": "1a98e31f-40a9-46f5-a4e6-bd91229046e7",
"metadata": {},
"outputs": [],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Model: \"model_1\"\n",
"_________________________________________________________________\n",
" Layer (type) Output Shape Param # \n",
"=================================================================\n",
" input_4 (InputLayer) [(None, 28, 28, 1)] 0 \n",
" \n",
" spectral_dense_3 (Spectral (None, 28, 28, 64) 257 \n",
" Dense) \n",
" \n",
" spectral_dense_4 (Spectral (None, 28, 28, 32) 4161 \n",
" Dense) \n",
" \n",
" spectral_dense_5 (Spectral (None, 28, 28, 10) 661 \n",
" Dense) \n",
" \n",
"=================================================================\n",
"Total params: 5079 (19.84 KB)\n",
"Trainable params: 2538 (9.91 KB)\n",
"Non-trainable params: 2541 (9.93 KB)\n",
"_________________________________________________________________\n"
]
}
],
"source": [
"inputs = keras.layers.Input(input_shape)\n",
"x = lip.layers.SpectralDense(64, activation=activation(),k_coef_lip=1.)(inputs)\n",
Expand All @@ -221,7 +305,7 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 6,
"id": "2ae5d775-fb90-4017-919d-bd34c08865cf",
"metadata": {},
"outputs": [],
Expand All @@ -235,7 +319,7 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 7,
"id": "6ff2460d-b1f3-43d6-964d-5919b24009e9",
"metadata": {},
"outputs": [],
Expand All @@ -248,15 +332,24 @@
"id": "de38b069-1705-408c-8dd3-99ed17cf519f",
"metadata": {},
"source": [
"- specify the Lipschitz constant of the whole model through the `k_coef_lip` attribute of the `Sequential` object, e.g.:"
"- by specifying the Lipschitz constant of the whole model through the `k_coef_lip` attribute of a `Sequential` object, e.g.:"
]
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 8,
"id": "fb882d3e-163f-4aa8-902f-83922ac8da89",
"metadata": {},
"outputs": [],
"outputs": [
{
"ename": "SyntaxError",
"evalue": "invalid syntax (1507601557.py, line 3)",
"output_type": "error",
"traceback": [
"\u001b[1;36m Cell \u001b[1;32mIn[8], line 3\u001b[1;36m\u001b[0m\n\u001b[1;33m ],\u001b[0m\n\u001b[1;37m ^\u001b[0m\n\u001b[1;31mSyntaxError\u001b[0m\u001b[1;31m:\u001b[0m invalid syntax\n"
]
}
],
"source": [
"K1_model = lip.model.Sequential([ \n",
" ....\n",
Expand All @@ -273,7 +366,7 @@
"source": [
"💡\n",
"Keep in mind that all the classes above inherit from their respective `keras` equivalent (e.g. `Dense` for `SpectralDense`). <br>\n",
"As a result, these objects conveniently use the same interface and the same parameters as their keras equivalent, with the additional parameter `k_coef_lip` that controls the Lipschitz constant of the layers.\n",
"As a result, these objects conveniently use the same interface and the same parameters as their keras equivalent, with the additional parameter `k_coef_lip` that controls the Lipschitz constant.\n",
"\n",
"## 🔨 Design rules for 1-Lipschitz neural networks with `deel-lip` <a id='design'></a> <a name='design'></a>\n",
"**Layer selection: `deel-lip` vs `keras`** \n",
Expand All @@ -284,7 +377,7 @@
"\n",
"This is the case for the following keras layers: `MaxPooling`, `GlobalMaxPooling`, `Flatten` and `Input`.\n",
"\n",
"Here is the full list of `keras` layers for which there is a Lipschitz equivalent supported by `deel-lip`, that must be used in order to enforce Lipschitz continuity of the model that incorporates the layer:\n",
"Below is the full list of `keras` layers for which `deel-lip` provides a Lipschitz equivalent. If one wants to ensure a model's Lipschitz continuity, the alternative `deel-lip` layers must be employed instead of the original `keras` counterparts.\n",
"\n",
"| tensorflow.keras.layers | deel.lip.layers |\n",
"| --------------- | --------------- |\n",
Expand All @@ -294,17 +387,14 @@
"\n",
"<br/>\n",
"\n",
"💡 Although there are additional layers available, the ones mentioned above are perfectly suitable and recommended for practical use. Interested readers can find information about the other layers [here](#documentation)\n",
"💡 Although there are additional Lipschitz continuous layers available in `deel-lip`, the ones mentioned above are perfectly suitable and recommended for practical use. Interested readers can find information about the other layers [here](#documentation).\n",
"\n",
"<br> \n",
"\n",
"\n",
"<div style=\"border: 1px solid #f2c94c; background-color: #fff3cd; padding: 10px; border-radius: 5px; margin-bottom: 10px;\">\n",
"<b>🚨 Warning:</b> <br> \n",
"When creating a 1-Lipschitz neural network, one should avoid using the following layers:<br> \n",
" - `Dropout`: Our recommendation is not to use it, as we have not well understood its behaviour yet during learning<br> \n",
" - `BatchNormalization`: It is not 1-Lipschitz<br></div>\n",
"<br/>"
"🚨 **Note:** *When creating a 1-Lipschitz neural network, one should avoid using the following layers:*<br> \n",
"- `Dropout`: Our current recommendation is to avoid using it, as we have not yet fully understood how it affects learning of 1-Lipschitz neural networks\n",
"- `BatchNormalization`: It is not 1-Lipschitz"
]
},
{
Expand All @@ -321,7 +411,7 @@
"- using the `GroupSort2` activation function stored in the `activations` submodule of `deel-lip` for the intermediate layers of a 1-Lipschitz neural network.\n",
"- not using any activation function for the last layer of 1-Lipschitz neural networks.\n",
"\n",
"💡 Interested readers can find information relevant to other activation functions that exist within `deel-lip` [here](https://deel-ai.github.io/deel-lip/api/layers/).\n",
"💡 Interested readers can find information relevant to other 1-Lipschitz activation functions that exist within `deel-lip` [here](https://deel-ai.github.io/deel-lip/api/layers/).\n",
"\n",
"\n",
"**Loss function selection:**\n",
Expand Down
Loading

0 comments on commit 71e714b

Please sign in to comment.