{"id":3171,"date":"2020-08-24T18:48:25","date_gmt":"2020-08-24T18:48:25","guid":{"rendered":"https:\/\/cloudxlab.com\/blog\/?p=3171"},"modified":"2020-09-03T11:06:47","modified_gmt":"2020-09-03T11:06:47","slug":"writing-custom-optimizer-in-tensorflow-and-keras","status":"publish","type":"post","link":"https:\/\/cloudxlab.com\/blog\/writing-custom-optimizer-in-tensorflow-and-keras\/","title":{"rendered":"Writing Custom Optimizer in TensorFlow Keras API"},"content":{"rendered":"\n
\"\"<\/figure>\n\n\n\n

Recently, I came up with an idea for a new Optimizer (an algorithm for training neural network). In theory, it looked great but when I implemented it and tested it, it didn’t turn out to be good.<\/p>\n\n\n\n

Some of my learning are:<\/p>\n\n\n\n

  1. Neural Networks are hard to predict.<\/li>
  2. Figuring out how to customize TensorFlow is hard because the main documentation is messy.<\/li>
  3. Theory and Practical are two different things. The more hands-on you are, the higher are your chances of trying out an idea and thus iterating faster.<\/li><\/ol>\n\n\n\n

    I am sharing my algorithm here. Even though this algorithm may not be of much use to you but it would give you ideas on how to implement your own optimizer using Tensorflow Keras.<\/p>\n\n\n\n

    A neural network is basically a set of neurons connected to input and output. We need to adjust the connection strengths such that it gives the least error for a given set of input. To adjust the weight we use the algorithms. One brute force algorithm could be to try all possible combinations of weights (connections strength) but that will be too time-consuming. So, we usually use the greedy algorithm most of these are variants of Gradient Descent. In this article, we will write our custom algorithm to train a neural network. In other words, we will learn how to write our own custom optimizer using TensorFlow Keras.<\/p>\n\n\n\n\n\n\n\n

    Gradient descent is simply this:<\/p>\n\n\n\n

    New_weight = weight  - eta * rate of change of error wrt weight\nw -= \u03b7*\u2202E\/\u2202w<\/code><\/pre>\n\n\n\n
    <\/div><\/div>\n\n\n\n
    Here eta (learning rate) is basically some constant. We will need to figure out. Usually, we keep eta as 0.001.<\/p>\n\n\n\n
    Here is an easy way to visualize it. If there is only one weight, we can visualize it like this:<\/p>\n\n\n\n
    \"Gradient
    Gradient Descent algotirhm<\/figcaption><\/figure>\n\n\n\n
    In this blog, we will learn how to create your own algorithm. Though it is extremely rare to need to customize the optimizer there are around 5-6 variants of Gradient descent algorithm but again if you get an idea of a new clever optimizer it could be a breakthrough.<\/p>\n\n\n\n
    In Gradient Descent, if the eta or learning rate is too high the error might increase instead of decreasing because the next value of weight could go to the other side of minima.<\/p>\n\n\n\n
    \"Gradient
    Sometimes the Gradient Descent Does not converge.<\/figcaption><\/figure>\n\n\n\n
    In my optimizer, I thought that the moment the slope changes sign, we will average the weight.<\/p>\n\n\n\n
    \"Gradient
    This is how our optimizer is going to work.<\/figcaption><\/figure>\n\n\n\n
    If the slope hasn’t changed the sign, the usual gradient descent will apply.<\/p>\n\n\n\n
    Now, let us get to the coding.<\/p>\n\n\n\n
    First import the libraries that we will be using.<\/p>\n\n\n\n
    import tensorflow as tf\nfrom tensorflow import keras\n\n# Common imports\nimport numpy as np\nimport os\n<\/code><\/pre>\n\n\n\n
    We are going to test our optimizer on California housing data. Now, let us load the data and create test and training data sets.<\/p>\n\n\n\n
    from sklearn.datasets import fetch_california_housing\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.preprocessing import StandardScaler\n\nhousing = fetch_california_housing()\nX_train_full, X_test, y_train_full, y_test = train_test_split(\n    housing.data, housing.target.reshape(-1, 1), random_state=42)\nX_train, X_valid, y_train, y_valid = train_test_split(\n    X_train_full, y_train_full, random_state=42)\n\nscaler = StandardScaler()\nX_train_scaled = scaler.fit_transform(X_train)\nX_valid_scaled = scaler.transform(X_valid)\nX_test_scaled = scaler.transform(X_test)\n<\/code><\/pre>\n\n\n\n
    In order to create a custom optimizer we will have to extend from base Optimizer Class which is in keras.optimizers class. <\/p>\n\n\n\n
    class SGOptimizer(keras.optimizers.Optimizer):\n\t\u2026\n\t<< this is where our implementation would be >>>\n\t\u2026 \n<\/code><\/pre>\n\n\n\n
    We will be overriding or implementing these methods:  <\/p>\n\n\n\n