{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b3fea80c-f5ab-4a63-b29e-e5043fd7c96e",
   "metadata": {},
   "source": [
    "## Dataset exploration"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea1a24a8-1942-494a-a795-96d3d1f07dae",
   "metadata": {},
   "source": [
    "### Setup and Data Loading"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61b65190-569d-418e-a75c-c91db5106f25",
   "metadata": {},
   "source": [
    "Import necessary libraries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "04e223dc-b69e-4ebf-ad58-a79e630cb75f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ebf95787-c0dc-4668-b8ea-0f8479a7b978",
   "metadata": {},
   "source": [
    " Load your data into a Pandas DataFrame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "740240e1-ad27-47e2-b395-fb08dcadbf1d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "train_data_orig = pd.read_csv('train_data.csv')\n",
    "test_data_orig = pd.read_csv('test_data.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "8ce84259-79ea-429d-9e42-91ffa23edcb6",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#train_data_orig"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "237e8e44-62e8-4d0f-9da4-bb6fe7b8c8c6",
   "metadata": {},
   "source": [
    "### Data Merging/Concatenation (if necessary)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "5a4300f9-60a4-414a-9a4c-684bbc43029e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# If concatenating vertically (append data in rows)\n",
    "conc_data = pd.concat([train_data_orig, test_data_orig], ignore_index=True)\n",
    "\n",
    "# If merging based on a common column (e.g., 'output')\n",
    "# conc_data = pd.merge(train_data_orig, test_data_orig, on='output', how='inner')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "331c8f02-01a7-49c2-b976-017e7627dbf4",
   "metadata": {},
   "source": [
    "### Normalization\n",
    "Data normalization is a technique used to change the values of numeric columns in the dataset to a common scale."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "3ebbb0aa-5149-4e25-adb4-67ab883dbf19",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Mannual Normalisation:\n",
    "#numerical_cols = ['output', 'input1', 'input2', 'input3', ..., 'input21']\n",
    "\n",
    "# Programatically\n",
    "numerical_cols = train_data_orig.select_dtypes(include=np.number).columns.tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "afbe7ca8-848f-4916-8466-de9d06717a68",
   "metadata": {
    "tags": []
   },
   "source": [
    " Handle any missing values in these columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "1c4a0bf6-3e68-474c-b24c-444478603922",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Fill NAs with the mean value of each column\n",
    "train_data_orig[numerical_cols] = test_data_orig[numerical_cols].apply(lambda x: x.fillna(x.mean()), axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1da0a70-a48c-4b1b-b150-b8477815eed9",
   "metadata": {},
   "source": [
    "**Apply normalization to the selected columns:**\n",
    "\n",
    "Choose a normalization method. The two most common methods are:\n",
    "\n",
    " ***1. Min-Max Scaling***\n",
    "\n",
    "This method rescales the features to a fixed range, usually [0,1].\n",
    "\\begin{equation}\n",
    "X_{\\text{norm}} = \\frac{X - X_{\\text{min}}}{X_{\\text{max}} - X_{\\text{min}}}\n",
    "\\end{equation}\n",
    "\n",
    "Where:\n",
    "- \\( X \\) is the original value.\n",
    "- \\( X_{\\text{min}} \\) is the minimum value in the column.\n",
    "- \\( X_{\\text{max}} \\) is the maximum value in the column.\n",
    "\n",
    " ***2. Z-score Normalization (Standardization)***\n",
    "\n",
    "This method uses the mean and standard deviation of the feature.\n",
    "\n",
    "\\begin{equation}\n",
    "X_{\\text{std}} = \\frac{X - \\mu}{\\sigma}\n",
    "\\end{equation}\n",
    "\n",
    "Where:\n",
    "- \\( X \\) is the original value.\n",
    "- \\( \\mu \\) is the mean of the column.\n",
    "- \\( \\sigma \\) is the standard deviation of the column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "f5c85bb1-c5ba-4dcd-bb63-2b7331e1a575",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Using Min-Max normalization\n",
    "test_data_orig[numerical_cols] = test_data_orig[numerical_cols].apply(\n",
    "    lambda x: (x - x.min()) / (x.max() - x.min()), axis=0\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1593accd-45aa-4467-afe5-17349a9217f7",
   "metadata": {},
   "source": [
    "Saving normalized data to a CSV file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "5c916c35-a519-4d82-bb0c-2becd7b6b72c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#train_data.to_csv('path_to_save/normalized_data.csv', index=False)\n",
    "train_data_orig.to_csv('normalized_train_data.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1e5bde8-fc56-479b-83cf-5251b070a542",
   "metadata": {},
   "source": [
    "### Preliminary Data Exploration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "f851a8b4-537b-4485-ba65-9df7f0dd266a",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 10979 entries, 0 to 10978\n",
      "Data columns (total 22 columns):\n",
      " #   Column   Non-Null Count  Dtype  \n",
      "---  ------   --------------  -----  \n",
      " 0   output   10979 non-null  float64\n",
      " 1   input1   10979 non-null  float64\n",
      " 2   input2   10979 non-null  float64\n",
      " 3   input3   10979 non-null  float64\n",
      " 4   input4   10979 non-null  float64\n",
      " 5   input5   10979 non-null  float64\n",
      " 6   input6   10979 non-null  float64\n",
      " 7   input7   10979 non-null  float64\n",
      " 8   input8   10979 non-null  float64\n",
      " 9   input9   10979 non-null  float64\n",
      " 10  input10  10979 non-null  float64\n",
      " 11  input11  10979 non-null  float64\n",
      " 12  input12  10979 non-null  float64\n",
      " 13  input13  10979 non-null  float64\n",
      " 14  input14  10979 non-null  float64\n",
      " 15  input15  10979 non-null  float64\n",
      " 16  input16  10979 non-null  float64\n",
      " 17  input17  10979 non-null  float64\n",
      " 18  input18  10979 non-null  float64\n",
      " 19  input19  10979 non-null  float64\n",
      " 20  input20  10979 non-null  float64\n",
      " 21  input21  10979 non-null  float64\n",
      "dtypes: float64(22)\n",
      "memory usage: 1.8 MB\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "output     0\n",
       "input1     0\n",
       "input2     0\n",
       "input3     0\n",
       "input4     0\n",
       "input5     0\n",
       "input6     0\n",
       "input7     0\n",
       "input8     0\n",
       "input9     0\n",
       "input10    0\n",
       "input11    0\n",
       "input12    0\n",
       "input13    0\n",
       "input14    0\n",
       "input15    0\n",
       "input16    0\n",
       "input17    0\n",
       "input18    0\n",
       "input19    0\n",
       "input20    0\n",
       "input21    0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Visualize first rows of the data\n",
    "train_data.head()\n",
    "\n",
    "# Obtain summary info about data\n",
    "train_data.info()\n",
    "\n",
    "# Statistical summary\n",
    "train_data.describe()\n",
    "\n",
    "#Check for null or missing values and handle them.\n",
    "data.isnull().sum()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "074a38d4-ac5b-4126-9817-4391f4401609",
   "metadata": {},
   "source": [
    "### Detailed Data Exploration and Visualization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "73ebc06f-9b59-4531-ba26-b0f07f0b7c92",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: matplotlib in /opt/conda/lib/python3.10/site-packages (3.8.0)\n",
      "Requirement already satisfied: seaborn in /opt/conda/lib/python3.10/site-packages (0.13.0)\n",
      "Requirement already satisfied: contourpy>=1.0.1 in /opt/conda/lib/python3.10/site-packages (from matplotlib) (1.1.1)\n",
      "Requirement already satisfied: cycler>=0.10 in /opt/conda/lib/python3.10/site-packages (from matplotlib) (0.12.0)\n",
      "Requirement already satisfied: fonttools>=4.22.0 in /opt/conda/lib/python3.10/site-packages (from matplotlib) (4.43.0)\n",
      "Requirement already satisfied: kiwisolver>=1.0.1 in /opt/conda/lib/python3.10/site-packages (from matplotlib) (1.4.5)\n",
      "Requirement already satisfied: numpy<2,>=1.21 in /opt/conda/lib/python3.10/site-packages (from matplotlib) (1.26.0)\n",
      "Requirement already satisfied: packaging>=20.0 in /opt/conda/lib/python3.10/site-packages (from matplotlib) (23.2)\n",
      "Requirement already satisfied: pillow>=6.2.0 in /opt/conda/lib/python3.10/site-packages (from matplotlib) (10.0.1)\n",
      "Requirement already satisfied: pyparsing>=2.3.1 in /opt/conda/lib/python3.10/site-packages (from matplotlib) (3.1.1)\n",
      "Requirement already satisfied: python-dateutil>=2.7 in /opt/conda/lib/python3.10/site-packages (from matplotlib) (2.8.2)\n",
      "Requirement already satisfied: pandas>=1.2 in /opt/conda/lib/python3.10/site-packages (from seaborn) (2.1.1)\n",
      "Requirement already satisfied: pytz>=2020.1 in /opt/conda/lib/python3.10/site-packages (from pandas>=1.2->seaborn) (2023.3)\n",
      "Requirement already satisfied: tzdata>=2022.1 in /opt/conda/lib/python3.10/site-packages (from pandas>=1.2->seaborn) (2023.3)\n",
      "Requirement already satisfied: six>=1.5 in /opt/conda/lib/python3.10/site-packages (from python-dateutil>=2.7->matplotlib) (1.16.0)\n"
     ]
    }
   ],
   "source": [
    "!pip install matplotlib seaborn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "f6cc0667-31ad-4523-bafc-8a588fdcf8ba",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "bfecfaf0-803b-4293-9fd3-3529c00920d2",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "train_data = pd.read_csv('normalized_train_data.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "d1cb4f55-e647-4c11-bd99-0c3fa2a4e8e4",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 98484 characters]",
      "text/plain": [
       "<Figure size 1000x600 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(10, 6))\n",
    "sns.scatterplot(x='input21', y='output', data=train_data)\n",
    "plt.title('Scatter plot of input21 vs output')\n",
    "plt.xlabel('input21')\n",
    "plt.ylabel('output')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "824f2d43-94ea-46bf-af52-87f261d3119a",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 16852 characters]",
      "text/plain": [
       "<Figure size 1000x600 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(10, 6))\n",
    "sns.barplot(x='input1', y='output', data=train_data)\n",
    "#plt.title('Average of input1 for each category in output')\n",
    "plt.xlabel('input1')\n",
    "plt.ylabel('Average of output')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "db81c7e9-fb3f-42e9-853c-2c2697791f26",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
