![\"Process](\"https:\/\/blog.cloudxlab.com\/wp-content\/uploads\/2019\/04\/Machine-Learning-Model-Deployment-1.png\")
The complete code for creating a REST service for your Machine Learning model can be found at the below link:<\/p>\n\n\n\n
https:\/\/github.com\/cloudxlab\/ml\/tree\/master\/projects\/deploy_mnist <\/a> Let us say, you have trained, fine-tuned and tested Machine Learning(ML) model – sgd_clf<\/em>, which was trained and tested using SGD Classifier on MNIST dataset. And now you want to deploy it in production, so that consumers of this model could use it. What are different options you have to deploy your ML model in production?<\/p>\n\n\n\n\n\n\n\n Before you start thinking about the deployment of your ML model in production, there are a few more tasks that needs to be performed. <\/p>\n\n\n\n There may be more steps involved, depending on what specific requirements you have, but below are some of the main steps:<\/p>\n\n\n <\/p>\n\n\n Instead of saving the ML model as it is, you can package your ML model (say mnist<\/em>), and create a .pkl file for it, using Python’s joblib<\/em> library. <\/p>\n\n\n\n For creating and restoring the .pkl file, you can either use joblib<\/em> library or pickle <\/em>library of Python. You normally use joblib<\/em> to save an object having large data, else, you use pickle<\/em> library. Here, in this case, we have used joblib<\/em> library.<\/p>\n\n\n\n .pkl file is nothing but a serialized pickle<\/em> file, which if you want, you can compress it further, to save storage space, using say Python’s gzip<\/em> library. After you apply compression, your ML model file name will look like – mnist.pkl.gz The content of your pickle file (in this case your ML model) could be intercepted and modified by anyone over the network if your network is not secured. Hence, it is advisable to use secured (encrypted) network connection while exchanging the pickle file.<\/p>\n\n\n\n In addition to this, the pickle file could be signed (using ‘cryptographic signature’<\/em>) before storing or transmitting, and this signature can be verified before it is restored at the receiver’s end (say your REST API). Cryptographic signature helps in detecting any alterations to your pickle file data.<\/p>\n\n\n\n Cryptographic signature uses a cryptographic algorithm which generates a cryptographic hash of your pickle file data along with shared secret key. SHA-1 cryptographic algorithm is considered to be the best algorithm to create a stronger hash, hence, it is highly advisable to use it. One way to deploy your ML model is, simply save the trained and tested ML model (sgd_clf<\/em>), with a proper relevant name (e.g. mnist<\/em>), in some file location on the production machine. The consumers can read (restore) this ML model file (mnist.pkl<\/em>) from this file location and start using it to make predictions on their dataset. But, simply deploying your ML model file on the production machine may not be sufficient, as only a handful of consumers who have access to your production machine, will be able to use it. <\/p>\n\n\n In most of the cases, the consumers of your model may not limited to your team members, who have access to your production machine. There may be consumers, who are from different departments (and located globally) who don’t have access to your production environment. Also, if you are building the ML model to be consumed “as a service”<\/em> by public (anyone), then, in that case also your consumers will not have access to your production environment.<\/p>\n\n\n\n In this case, where the consumers don’t have access to your production environment, how do you make your deployed ML model available to them?<\/p>\n\n\n\n The answer is – exposing your deployed ML model to the consumers, as a service (say REST service or REST API). REST API increases the reach of your ML model to wider audience and as it can be called from any Application – mobile app, Java application, web application, .Net application, PHP\/javascript, Python etc. The ‘second option’ option seems to be a good option, if your audience is globally located and you want to provide this ML model as a service to a wider audience. Most of the ML models are written using Python libraries now a days, hence, in this article, we will discuss about how to expose your ML model as a REST service using Python frameworks.<\/p>\n\n\n\n The most common Python framework, which is used to create a REST API in Python is, Flask.<\/p>\n\n\n\n Flask<\/strong> is a lightweight micro web framework written in Python, which can be used to create small web applications in Python.<\/p>\n\n\n\n Flask is not only a framework, but it also has a web server, which could be used to develop and test small Python web applications.<\/p>\n\n\n\n Flask also has APIs\/functions using which you can create a REST API (service) in Python, and for testing this REST service, you can deploy in on Flask web server. However, since, Flask is a lightweight web server, it should not be used in production environment, to deploy your web application or REST service. It should only be used in development environment, for development and testing of the web applications or REST API.<\/p>\n\n\n\n For production purposes, you can deploy this REST service (written using Python and Flask framework) on a more matured WSGI protocol compliant application server like Gunicorn or uWSGI<\/strong> along with Nginx<\/strong> as the web server. If you are coming from Java background, then, you can think of:<\/p>\n\n\n <\/p>\n\n\n In your ML model training Python code, you can save your trained and tested ML model (say sgd_clf<\/em>), using a proper file name, on a file location of your production application server using joblib<\/em> library of Python, as shown below:<\/p>\n\n\n\n Here, ML model sgd_clf<\/em> is being saved with a file name mnist_model.pkl <\/em>in file location ‘trained_models\/’ <\/em>on the production machine. This ML model you can retrieve (restore) inside your REST API code to make the predictions on the input digit images. Below is the code in Python (using Flask framework) to create a REST API (predict_image()<\/em>) which predicts a digit from a given image (containing a handwritten digit). Currently, this REST API just takes a ‘image file name’ (‘file’<\/em>) as input (sent in the request) The below line of code loads (retrieves) the stored (saved) ML model mnist_model.pkl<\/em> from the file location ‘..\/trained_models\/’ <\/em>using the Scikit Learn’s (sklearn) library joblib<\/em> and stores it in a variable called model<\/em>.<\/p>\n\n\n\n We can use this model<\/em> object to make the ML predictions using its predict() <\/em>function. <\/p>\n\n\n\n We use something called as decorators<\/em> in Python (like we have annotations in Java\/Spring) to modify behaviour of a function or a class. Below, we are defining a decorator<\/em> ( @app.route(…)<\/em> ) for the predict_image() <\/em>function, which says that any ‘POST’ request URL which matches ‘\/predict’<\/em> pattern, should be redirected to this function – predict_image()<\/em>. <\/p>\n\n\n\n Thus, here, we are defining the ‘End Point’<\/em> for our REST API (predict_image()<\/em>). The ‘End Point URL’<\/em> for your REST API will look like as below To convert an input image to grayscale<\/em> image, which is required by mnist<\/em> ML model (as the model was trained on grayscale<\/em> images), we can use convert()<\/em> function of Image<\/em> module of Python’s Pillow library. Please don’t forget to pass value ‘L’ value to the convert() function. Now, we can make predictions on this input image using the model<\/em> object, using its predict()<\/em> function. <\/p>\n\n\n\n We have used Flask’s jsonify() function to convert our ‘result’ of the ML model to a JSON object. Our REST API predict_image()<\/em> returns result in JSON format. This JSON object (‘digit’: result), you can use to retrieve the result in your calling application. <\/p>\n\n\n\n This REST API can be called from any application – mobile app, Java application, web application, .Net application, PHP\/javascript, Python, etc. The only information your consumers need about this REST API is its ‘end point URL’<\/em> and the credentials to access it. Using a web browser to test a web service isn’t the best idea since web browsers can’t generate all types of http requests easily. Instead, use ‘curl’<\/strong>command of unix to test your REST API. Still some more work needs to be done, after you have created and deployed your REST API, you need to secure it. But, not all REST API functions may need to be secured.<\/p>\n\n\n\n You can use HTTP secure server (https:\/\/\u2026.<\/em>) to encrypt all the communications, to make your REST service communications more secure. <\/p>\n\n\n\n You can secure your REST service using Flask extension called flask-httpauth. Using @auth.login_required<\/em> decorator with your REST API functions, you can specify which functions (APIs) in the REST service are secured (protected). <\/p>\n\n\n\n You can use a ‘token’<\/em> based authentication for requests. In this type of authentication, the client application (which is calling this REST API), for the first request, sends the credentials to the REST service, and in return gets back a token, which it needs to send to the REST service in all its future requests. You can use any of the hashing algorithms (like SHA1) to create these tokens. Also, you may get unauthorized access<\/em> error thrown from these protected REST APIs (functions), which would need to be handled in the code.<\/p>\n\n\n\n In your REST API code file, you can write a separate function to handle these authorization errors. And, to specify, which function in your REST API code file, will handle these authorization errors, you can use @auth.error_handler <\/em>decorator on top of such a function, as shown below: Here, unauthorized() is the function that will be invoked if your protected REST API function throws an unauthorized access<\/em> error. You can return a relevant error message from this function as a response. There are many other security implementations available that you can use according to the level of your security requirements. The main idea is to secure your REST API. When you are creating the deployment plan for your model, you may need to consider two important aspects – Scalability and Fault Tolerance.<\/p>\n\n\n\n To achieve fault tolerance and scalability for your REST service, you can use Docker and Kubernetes for the deployment. Using Docker application, you can package your REST service code\/application in a Docker container. For this you need to create a Dockerfile for your REST service application (code), mentioning the following:<\/p>\n\n\n You can use the above created Dockerfile to create a Docker container using ‘docker build’ <\/em>command. Now, after packaging your REST service in a Docker container, you can deploy and run your Docker containers on any machine or the VM (virtual machine). <\/p>\n\n\n\n Docker enables you to create new Docker containers (for your REST service). And Kubernetes can be used to deploy these Docker containers on a Kubernetes cluster. Since, you can spin (create) new Docker containers on the fly (say when an existing container goes down or you need new containers due to increase in user requests), it provides you the required ‘fault tolerance’<\/em>.<\/p>\n\n\n\n Kubernetes cluster (cluster of machines) provides the required ‘scalability’ <\/em>for your REST service, it can spin (create) new Docker containers based on demand, using the Docker image. Kubernetes master node which also takes care of ‘fault tolerance’<\/em>, if a container goes down, it spins another Docker container.<\/p>\n\n\n\n Kubernetes also has a load balancer mechanism, which takes care of distributing the load (user requests) on the containers.<\/p>\n\n\n\n Since, we are discussing about clusters (of machines) and Kubernetes, thought of sharing a small note on Kubernetes clusters and Spark clusters. Often, when we talk about Big Data and Spark MLLib, people get confused between Kubernetes clusters and Spark clusters. When to use which cluster?<\/p>\n\n\n\n To understand, when (and why<\/em>) we use Kubernetes clusters and when we use Spark clusters, we need to understand their purpose of use. Spark MLLib is basically a library of Spark, which has various Machine Learning algorithms (which are also available in Scikit Learn), customized to run on a Spark cluster i.e. using multiple machines. We use the ML algorithms from Spark MLLib library (in place of normal Scikit Learn version of ML algorithms), when our dataset is so huge that we need Big Data<\/em> kind of processing to reduce the training and prediction time of our ML model.<\/p>\n\n\n\n For Deep Learning models, you can use TensorFlow Distributed instead of Spark MLLib. Hence, in a nutshell, we use Spark MLLib on Spark cluster to reduce the training and prediction time of our ML model.<\/p>\n\n\n\n Whereas, we use Kubernetes cluster is used to achieve ‘scalability’<\/em>, of the finally trained and tested ML model, when we deployed it on production. <\/p>\n\n\n\n When we get multiple requests simultaneously, Kubernetes spins new Docker containers (containing your ML model) and distributes the requests to the multiple containers to reduce the load. Hence, after we have trained and tested our Spark MLLib Machine Learning model, using huge amount of data (Big Data), on a Spark cluster, we can package and deploy the same on a Kubernetes cluster in production. But, in case, your requirement is to run Spark on a Kubernetes cluster of machines, you can do so, as Spark (version 2.3 onwards<\/em>) supports this. You can create Docker containers for Spark nodes and deploy these Docker containers on Kubernetes cluster (of machines). Spark (version 2.3 and above<\/em>) already comes with a Dockerfile that you can use for this purpose. For more details on this, you can refer Spark documentation<\/a>. Below are a few case studies for different types of deployment modes: <\/p>\n\n\n\n Again, complete code for creating a REST service for your Machine Learning model, can be found at the below link:<\/p>\n\n\n\n https:\/\/github.com\/cloudxlab\/ml\/tree\/master\/projects\/deploy_mnist <\/a> For the complete course on Machine Learning, please visit Specialization Course on Machine Learning & Deep Learning<\/a>
<\/p>\n\n\n\n\n
\n
\n
Packaging your ML model<\/strong><\/h2>\n\n\n
<\/p>\n\n\nSecuring your packaged ML model<\/strong><\/h2>\n\n\n
<\/p>\n\n\nOptions to deploy your ML model in production<\/h2>\n
First option<\/h3>\n\n\n
<\/p>\n\n\n\nSecond option<\/h3>\n\n\n
<\/p>\n\n\n\n
<\/p>\n\n\n\n
Hence, in this article, we will be focussing our discussion around this ‘second option’ – exposing your ML model as a REST service.
<\/p>\n\n\nTools and libraries to build REST API in Python<\/h2>\n\n\n
<\/p>\n\n\n\n
<\/p>\n\n\n\n\n
\n
\n
![\"Process](\"https:\/\/blog.cloudxlab.com\/wp-content\/uploads\/2019\/04\/Machine-Learning-Model-Deployment-2.png\")
Building (and testing) your REST API (service) using Flask framework<\/h2>\n\n\n
# dump the model to the file\njoblib.dump(sgd_clf, \"trained_models\/mnist_model.pkl\")<\/code><\/pre>\n\n\n\n
<\/p>\n\n\n\n
<\/p>\n\n\n\n
<\/p>\n\n\n\nimport numpy as np\nfrom sklearn.externals import joblib\nfrom PIL import Image\nfrom flask import Flask, jsonify, request\n\n# Create flask app\napp = Flask(__name__)\n\n# Load the previously trained model from the file\nmodel = joblib.load(\"..\/trained_models\/mnist_model.pkl\")\n\n# \/predict is the end point\n@app.route('\/predict', methods=[\"POST\"])\ndef predict_image():\n # Read the image uploaded by the curl command\n requested_img = request.files['file']\n\n '''\n Convert the uploaded image to grayscale.\n Since in MNIST the training images are grayscaled hence we will have to convert the uploaded image to grayscale\n '''\n grayscale_img = Image.open(requested_img).convert('L')\n\n '''\n Resize the uploaded image to 28x28 pixels.\n Since in MNIST the training images are of 28x28 pixels hence we will have to resize the uploaded image to 28x28 pixels.\n '''\n resized_image = grayscale_img.resize((28,28))\n \n # Convert the image to an array\n img = np.asarray(resized_image)\n\n # Reshape the image to (784, 1)\n img = img.reshape(784,)\n\n # Predict the digit using the trained model\n pred = model.predict(img.reshape(1, -1))\n\n # Get the digit\n result = int(pred.tolist()[0])\n\n # Return the JSON response\n return jsonify({\"digit\": result})\n<\/code><\/pre>\n\n\n\n# Load the previously trained model from the file\nmodel = joblib.load(\"..\/trained_models\/mnist_model.pkl\")<\/code><\/pre>\n\n\n\n
<\/p>\n\n\n\n@app.route('\/predict', methods=[\"POST\"])\ndef predict_image():<\/code><\/pre>\n\n\n\n
<\/p>\n\n\n\nhttp:\/\/mydomain.com:8080\/predict<\/code><\/pre>\n\n\n\n
<\/p>\n\n\n\n
<\/p>\n\n\n\ngrayscale_img = Image.open(requested_img).convert('L')<\/code><\/pre>\n\n\n\n# Predict the digit using the trained model \u00a0\u00a0\npred = model.predict(img.reshape(1, -1))<\/code><\/pre>\n\n\n\n
<\/p>\n\n\n\nreturn jsonify({ \u201cdigit\u201d: result })<\/code><\/pre>\n\n\n\n
<\/p>\n\n\n\n
<\/p>\n\n\n\ncurl -F 'file=@test-images\/7.png' 127.0.0.1:4041\/predict<\/code><\/pre>\n\n\n\n
<\/p>\n\n\n\n
<\/p>\n\n\n\n@auth.error_handler\ndef unauthorized():<\/code><\/pre>\n\n\n\n
<\/p>\n\n\n\n
<\/p>\n\n\nAchieving Scalability and Fault Tolerance for your deployed ML model<\/h2>\n\n\n
<\/p>\n\n\n\n\n
\n
\n
<\/p>\n\n\n\n
<\/p>\n\n\n\n![\"Deployment](\"https:\/\/blog.cloudxlab.com\/wp-content\/uploads\/2019\/04\/Deployment-Architecture-for-REST-service.png\")
Kubernetes, Spark MLLib and TensorFlow Distributed<\/h2>\n\n\n
<\/p>\n\n\n\n
<\/p>\n\n\n\n
<\/p>\n\n\n\n
<\/p>\n\n\n\n
<\/p>\n\n\nCase Studies<\/h2>\n\n\n
<\/p>\n\n\n\n
\n
\n
\n
\n
\n
\n
<\/p>\n\n\n\n
<\/p>\n\n\n\n
<\/p>\n","protected":false},"excerpt":{"rendered":"