{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "TP.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o6JcTmBgauWE",
        "colab_type": "text"
      },
      "source": [
        "## Minimisation du risque empirique régularisé\n",
        "\n",
        "On s'intéresse aux algorithmes d'apprentissage basées sur la minimisation du rsque empirique régularisé $$\\min_f \\sum_i \\ell(y_i ,f(x_i)) + \\alpha \\Omega(f) $$"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a-AwEELgc1wa",
        "colab_type": "text"
      },
      "source": [
        "Télécharger le jeu de données `concrete` et paratger les données en données d'apprentissage et données test."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZMmeYQnjc2Sg",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        " from sklearn.model_selection import train_test_split \n",
        "\n",
        "data = pd.read_csv(\"concrete.csv\")\n",
        "X = data[[col for col in data.columns if col != \"strength\"]]\n",
        "y = data[\"strength\"]\n",
        "\n",
        "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, shuffle=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NPlZUTi8boEs",
        "colab_type": "text"
      },
      "source": [
        "### Régression ridge\n",
        "\n",
        "* $\\ell(y_i,f(x_i)) = (y_i - \\beta^T x_i)^2$\n",
        "* $\\Omega(f) = \\|\\beta\\|_2^2$ (norme $L_2$)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qV-2nHibf578",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from sklearn.linear_model import Ridge\n",
        "from yellowbrick.features import FeatureImportances\n",
        "\n",
        "model = FeatureImportances(Ridge(alpha=5), relative=False)\n",
        "model.fit(X_train, y_train)\n",
        "model.score(X_test, y_test)\n",
        "model.poof()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hxwkN8Fsi8kA",
        "colab_type": "text"
      },
      "source": [
        "### Régression Lasso\n",
        "\n",
        "* $\\ell(y_i,f(x_i)) = (y_i - \\beta^T x_i)^2$\n",
        "* $\\Omega(f) = \\|\\beta\\|_1$ (norme $L_1$)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7L767S72i7Mm",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from sklearn.linear_model import Lasso \n",
        "\n",
        "model = FeatureImportances(Lasso(alpha=5), relative=False)\n",
        "model.fit(X_train, y_train)\n",
        "model.score(X_test, y_test)\n",
        "model.poof()\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RETc5Inxjvh-",
        "colab_type": "text"
      },
      "source": [
        "* Analyser et Expliquer les résultats obtenus"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "16zahEU_kPvk",
        "colab_type": "text"
      },
      "source": [
        "### Régression ElasticNet\n",
        "\n",
        "* combine une régularization $L_1$ et une régularization $L_2$\n",
        "* coefficients de régularisation $\\alpha$ $(L_2)$ et $\\rho$ $(L_1)$\n",
        "* $\\ell(y_i,f(x_i)) = (y_i - \\beta^T x_i)^2$\n",
        "* $\\Omega(f) = \\alpha \\rho \\|\\beta\\|_1 + \\frac{\\alpha(1-\\rho)}{2}\\|\\beta\\|_2$"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sbLjkmdrkQXU",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "\n",
        "\n",
        "from sklearn.linear_model import ElasticNet \n",
        "\n",
        "model = FeatureImportances(ElasticNet(alpha=15.5, l1_ratio=15.1), relative=False)\n",
        "model.fit(X_train, y_train)\n",
        "model.score(X_test, y_test)\n",
        "model.poof()\n",
        "\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gGh1wLIjl59D",
        "colab_type": "text"
      },
      "source": [
        "### Sélection du paramètre $\\alpha$\n",
        "\n",
        "* Le choix de $\\alpha$ est très important.\n",
        "\n",
        "* On choisit $\\alpha$ qui maximise l'erreur de cross validation parmi plusieurs valeurs définies dans une grille.  "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "c8JNdWkgnAG2",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from yellowbrick.regressor import AlphaSelection \n",
        "from sklearn.linear_model import RidgeCV, LassoCV, ElasticNetCV\n",
        "\n",
        "alphas = np.logspace(-10, 1, 400)\n",
        "\n",
        "viz = AlphaSelection(LassoCV(alphas=alphas))\n",
        "viz.fit(X, y)\n",
        "viz.poof()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GpUHTVqHnJqI",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "# **To do **\n",
        "* Appliquer et comparer la régression Ridge et la régression Lasso sur le jeu de données boston. \n",
        "\n",
        "* Afficher l'erreur de prédiction sur les données de test et les données d'apprentissage en fonction du paramètre de régularisation.\n",
        "\n",
        "* Afficher l'erreur de prédiction sur les données de test et les données d'apprentissage en fonction du nombre de données d'apprentissage dans les deux cas : $\\alpha =0$ et $\\alpha$ optimal choisi par cross-validation.\n",
        "\n",
        "* Lire les sections 3.1 et 3.2 du chapitre 3 du livre \"Pattern Recognition and Machine Learning\" (Bishop, 2006).\n"
      ]
    }
  ]
}