Number Plate Reader Methodology<\/h2>\n\n\n\n
We will break down the task of building a custom number plate reader to the following<\/p>\n\n\n\n
- Create a dataset of images with number plates<\/li>
- Annotate the dataset with LabelImg<\/li>
- Train an existing object detection model to detect number plates in a picture<\/li>
- Extract the number plate using the trained model <\/li>
- Run filters and cleanup of the picture <\/li>
- Read the number plate using an OCR tool<\/li>
- Identify shortcomings and explore methods to improve the model<\/li><\/ol>\n\n\n\n
We will cover the former 4 steps in this blog. The latter 3 will be covered in a later blog.<\/p>\n\n\n\n
Creating the Dataset<\/h2>\n\n\n\n
![\"\"](\"https:\/\/blog.cloudxlab.com\/wp-content\/uploads\/2020\/04\/img11.jpeg\")
<\/figure><\/li>![\"\"](\"https:\/\/blog.cloudxlab.com\/wp-content\/uploads\/2020\/04\/img10.jpeg\")
<\/figure><\/li>![\"\"](\"https:\/\/blog.cloudxlab.com\/wp-content\/uploads\/2020\/04\/img8.jpeg\")
<\/figure><\/li>![\"\"](\"https:\/\/blog.cloudxlab.com\/wp-content\/uploads\/2020\/04\/img7.jpeg\")
<\/figure><\/li>![\"\"](\"https:\/\/blog.cloudxlab.com\/wp-content\/uploads\/2020\/04\/img6.jpeg\")
<\/figure><\/li>![\"\"](\"https:\/\/blog.cloudxlab.com\/wp-content\/uploads\/2020\/04\/img5.jpeg\")
<\/figure><\/li><\/ul>Images for number plate reader<\/figcaption><\/figure>\n\n\n\n We can create a dataset on images with number plates like the ones above. You can take such photos with your mobile phone or scrape from the internet. Break the dataset into two directories – train and test. The train directory can have about 80% of the images and the test directory can have the remaining 20%. <\/p>\n\n\n\n
I advise you to be very careful in this step. You should make sure that you shuffle the entire set and your test and train sets are random and not over-representative of a certain type of images. For e.g. you may choose images from multiple sources like mobile images, web scraped, cctv feed etc. If your test set or train set has more images from a single source like web scraping, you will see bad results. Make sure your test and train sets are randomised and as representative of the images you want to use in the end.<\/p>\n\n\n\n
Annotating the Images<\/h2>\n\n\n\n
First create a new environment with virtualenv to manage the workflow. I strongly recommend using a virtualenvwrapper as it makes management of multiple python environments very easy.<\/p>\n\n\n\n
pip3 install virtualenvwrapper\nmkvirtualenv tf_obj<\/code><\/pre>\n\n\n\nInstall LabelImg following the instruction here<\/a>. Choose the appropriate workflow based on your environment. Open LabelImg and open the directory containing the images and annotate them.<\/p>\n\n\n\n
![\"\"](\"https:\/\/blog.cloudxlab.com\/wp-content\/uploads\/2020\/04\/Screenshot-2020-04-18-at-6.19.35-AM.png\")
Annotating images with LabelImg to build a number plate detector<\/figcaption><\/figure>\n\n\n\n We will annotate both the train and test image sets. Enter the label as numberplate and choose Pascal\/Voc as save format. Once your done, you will notice there are xml files associated with each picture. You can read the xml file and understand the parameters in the file. A sample xml file is below.<\/p>\n\n\n\n
![\"\"](\"https:\/\/blog.cloudxlab.com\/wp-content\/uploads\/2020\/04\/Screenshot-2020-04-18-at-9.52.03-PM.png\")
<\/figure>\n\n\n\nTraining<\/h2>\n\n\n\n
Recommended Directory Structure<\/h3>\n\n\n\n
TensorFlow\n\u251c\u2500 models\n\u2502 \u251c\u2500 official\n\u2502 \u251c\u2500 research\n\u2502 \u251c\u2500 samples\n\u2502 \u2514\u2500 tutorials\n\u2514\u2500 workspace\n \u2514\u2500 training_home\n \u251c\u2500 annotations\n \u251c\u2500 images\n \u2502 \u251c\u2500 test\n \u2502 \u2514\u2500 train\n \u251c\u2500 pre-trained-model\n \u251c\u2500 training\n \u251c\u2500 eval\n \u2514\u2500 trained-inference-graph<\/code><\/pre>\n\n\n\nThe images and xml files generated will be saved in the images directory. The test set and train set are kept images\/test and images\/train folders.<\/p>\n\n\n\n
Install Tensorflow and Tensorflow Object Detection API<\/h3>\n\n\n\n
Tensorflow object detection API is a set of libraries that use TensorFlow. At the time of writing this blog tensorflow 2.0 was released but it did not support Tensorflow Object Detection yet. A key library tf.contrib was not part of tensorflow 2.0. For this exercise install version tf 1.15. Continue using the environment we created earlier. <\/p>\n\n\n\n
- Tensorflow installation instructions<\/a>: Note install tensorflow 1.15, <\/li>
- Tensorflow Object Detection API instruction<\/a>: This API models have to be downloaded and kept locally. We will store the models under the directory models as seen in the Recommended Directory Structure. <\/li><\/ul>\n\n\n\n
For your reference, we have provided the entire requirements.txt file here.<\/p>\n\n\n\n
absl-py==0.9.0\nansiwrap==0.8.4\narrow==0.15.5\nastor==0.8.1\nastroid==2.3.3\nastropy==3.2.3\nattrs==19.3.0\nbackcall==0.1.0\nbcolz==1.2.1\nbinaryornot==0.4.4\nbleach==3.1.0\ncachetools==4.0.0\ncertifi==2019.11.28\ncffi==1.13.2\nchardet==3.0.4\nClick==7.0\ncloud-tpu-profiler==1.15.0rc1\ncloudpickle==1.2.2\ncolorama==0.4.3\nconfigparser==4.0.2\nconfuse==1.0.0\ncookiecutter==1.7.0\ncryptography==1.7.1\ncycler==0.10.0\nCython==0.29.15\ndaal==2019.0\ndatalab==1.1.5\ndecorator==4.4.1\ndefusedxml==0.6.0\ndill==0.3.1.1\ndistro==1.0.1\ndocker==4.1.0\nentrypoints==0.3\nenum34==1.1.6\nfairing==0.5.3\nfsspec==0.6.2\nfuture==0.18.2\ngast==0.2.2\ngcsfs==0.6.0\ngitdb2==2.0.6\nGitPython==3.0.5\ngoogle-api-core==1.16.0\ngoogle-api-python-client==1.7.11\ngoogle-auth==1.11.0\ngoogle-auth-httplib2==0.0.3\ngoogle-auth-oauthlib==0.4.1\ngoogle-cloud-bigquery==1.23.1\ngoogle-cloud-core==1.2.0\ngoogle-cloud-dataproc==0.6.1\ngoogle-cloud-datastore==1.10.0\ngoogle-cloud-language==1.3.0\ngoogle-cloud-logging==1.14.0\ngoogle-cloud-monitoring==0.31.1\ngoogle-cloud-spanner==1.13.0\ngoogle-cloud-storage==1.25.0\ngoogle-cloud-translate==2.0.0\ngoogle-compute-engine==20191210.0\ngoogle-pasta==0.1.8\ngoogle-resumable-media==0.5.0\ngoogleapis-common-protos==1.51.0\ngrpc-google-iam-v1==0.12.3\ngrpcio==1.26.0\nh5py==2.10.0\nhorovod==0.19.0\nhtml5lib==1.0.1\nhtmlmin==0.1.12\nhttplib2==0.17.0\nicc-rt==2020.0.133\nidna==2.8\nimageio==2.6.1\nimportlib-metadata==1.4.0\nintel-openmp==2020.0.133\nipykernel==5.1.4\nipython==7.9.0\nipython-genutils==0.2.0\nipython-sql==0.3.9\nipywidgets==7.5.1\nisort==4.3.21\njedi==0.16.0\nJinja2==2.11.0\njinja2-time==0.2.0\njoblib==0.14.1\njson5==0.8.5\njsonschema==3.2.0\njupyter==1.0.0\njupyter-aihub-deploy-extension==0.1\njupyter-client==5.3.4\njupyter-console==6.1.0\njupyter-contrib-core==0.3.3\njupyter-contrib-nbextensions==0.5.1\njupyter-core==4.6.1\njupyter-highlight-selected-word==0.2.0\njupyter-http-over-ws==0.0.7\njupyter-latex-envs==1.4.6\njupyter-nbextensions-configurator==0.4.1\njupyterlab==1.2.6\njupyterlab-git==0.9.0\njupyterlab-server==1.0.6\nKeras==2.3.1\nKeras-Applications==1.0.8\nKeras-Preprocessing==1.1.0\nkeyring==10.1\nkeyrings.alt==1.3\nkiwisolver==1.1.0\nkubernetes==10.0.1\nlazy-object-proxy==1.4.3\nllvmlite==0.31.0\nlxml==4.4.2\nMarkdown==3.1.1\nMarkupSafe==1.1.1\nmatplotlib==3.0.3\nmccabe==0.6.1\nmissingno==0.4.2\nmistune==0.8.4\nmkl==2019.0\nmkl-fft==1.0.6\nmkl-random==1.0.1.1\nmock==3.0.5\nmore-itertools==8.1.0\nnbconvert==5.6.1\nnbdime==1.1.0\nnbformat==5.0.4\nnetworkx==2.4\nnltk==3.4.5\nnotebook==6.0.3\nnumba==0.47.0\nnumpy==1.18.1\noauth2client==4.1.3\noauthlib==3.1.0\nopencv-python==4.1.2.30\nopt-einsum==3.1.0\npackaging==20.1\npandas==0.25.3\npandas-profiling==1.4.0\npandocfilters==1.4.2\npapermill==1.2.1\nparso==0.6.0\npathlib2==2.3.5\npexpect==4.8.0\nphik==0.9.8\npickleshare==0.7.5\nPillow==7.0.0\nplotly==4.5.0\npluggy==0.13.1\npoyo==0.5.0\nprettytable==0.7.2\nprometheus-client==0.7.1\npromise==2.3\nprompt-toolkit==2.0.10\nprotobuf==3.11.2\npsutil==5.6.7\nptyprocess==0.6.0\npy==1.8.1\npyarrow==0.15.1\npyasn1==0.4.8\npyasn1-modules==0.2.8\npycparser==2.19\npycrypto==2.6.1\npycurl==7.43.0\npydaal==2019.0.0.20180713\npydot==1.4.1\nPygments==2.5.2\npygobject==3.22.0\npylint==2.4.4\npyparsing==2.4.6\npyrsistent==0.15.7\npytest==5.3.4\npytest-pylint==0.14.1\npython-apt==1.4.1\npython-dateutil==2.8.1\npytz==2019.3\nPyWavelets==1.1.1\npyxdg==0.25\nPyYAML==5.3\npyzmq==18.1.1\nqtconsole==4.6.0\nrequests==2.22.0\nrequests-oauthlib==1.3.0\nretrying==1.3.3\nrsa==4.0\nscikit-image==0.15.0\nscikit-learn==0.22.1\nscipy==1.4.1\nseaborn==0.9.1\nSecretStorage==2.3.1\nSend2Trash==1.5.0\nsimplegeneric==0.8.1\nsix==1.14.0\nsmmap2==2.0.5\nSQLAlchemy==1.3.13\nsqlparse==0.3.0\ntbb==2019.0\ntbb4py==2019.0\ntenacity==6.0.0\ntensorboard==1.15.0\ntensorflow-datasets==1.2.0\ntensorflow-estimator==1.15.1\ntensorflow-gpu==1.15.2\ntensorflow-hub==0.6.0\ntensorflow-io==0.8.1\ntensorflow-metadata==0.21.1\ntensorflow-probability==0.9.0\ntensorflow-serving-api-gpu==1.14.0\ntermcolor==1.1.0\nterminado==0.8.3\ntestpath==0.4.4\ntextwrap3==0.9.2\ntfds-nightly==1.0.1.dev201903050105\ntornado==5.1.1\ntqdm==4.42.0\ntraitlets==4.3.3\ntyped-ast==1.4.1\nunattended-upgrades==0.1\nuritemplate==3.0.1\nurllib3==1.24.2\nvirtualenv==16.7.9\nwcwidth==0.1.8\nwebencodings==0.5.1\nwebsocket-client==0.57.0\nWerkzeug==0.16.1\nwhichcraft==0.6.1\nwidgetsnbextension==3.5.1\nwitwidget-gpu==1.5.1\nwrapt==1.11.2\nzipp==1.1.0\n<\/code><\/pre>\n\n\n\nThe Label-Map file<\/h3>\n\n\n\n
The label-map file uniquely maps the labels in your model to integers. In our case we have only one label, so our label-map file will look like<\/p>\n\n\n\n
item {\n id: 1\n name: 'numberplate'\n}<\/code><\/pre>\n\n\n\nWe will call this file label-map.pbtxt. In case our model had more types of images we would have corresponding additional entries with unique integers mapping to the additional labels. We will keep this model in the annotations folder.<\/p>\n\n\n\n
Creating the TF.Record<\/h3>\n\n\n\n
TensorFlow uses a binary format called TF.Record. Deep Learning uses large datasets, and storing the data in binary makes it efficient. It is more compact so it needs less disk for storage, and creates an efficient data pipeline to feed the training process. While working with datasets too large to store directly in memory, the training process only loads smaller segments or batches. TF.Records is highly optimised for TensorFlow and enables efficient data saving, loading, merging capabilities. Additionally, TF.Records simplifies processing of sequence data like time series and word encodings. For more details refer to this link<\/a>.<\/p>\n\n\n\n
There are two steps to create TF records. First we create csv files from the xml files, then we create the TF.record files. The code to generate the csv file is in the code block below. The code has the usage at the top. The input is the directory with the xml files and output is in the annotation directory.<\/p>\n\n\n\n
\"\"\"\nUsage:\n# Create train data:\npython xml_to_csv.py -i [PATH_TO_IMAGES_FOLDER]\/train -o [PATH_TO_ANNOTATIONS_FOLDER]\/train_labels.csv\n\n# Create test data:\npython xml_to_csv.py -i [PATH_TO_IMAGES_FOLDER]\/test -o [PATH_TO_ANNOTATIONS_FOLDER]\/test_labels.csv\n\"\"\"\n\nimport os\nimport glob\nimport pandas as pd\nimport argparse\nimport xml.etree.ElementTree as ET\n\n\ndef xml_to_csv(path):\n \"\"\"Iterates through all .xml files (generated by labelImg) in a given directory and combines them in a single Pandas datagrame.\n\n Parameters:\n ----------\n path : {str}\n The path containing the .xml files\n Returns\n -------\n Pandas DataFrame\n The produced dataframe\n \"\"\"\n\n xml_list = []\n for xml_file in glob.glob(path + '\/*.xml'):\n tree = ET.parse(xml_file)\n root = tree.getroot()\n for member in root.findall('object'):\n value = (root.find('filename').text,\n int(root.find('size')[0].text),\n int(root.find('size')[1].text),\n member[0].text,\n int(member[4][0].text),\n int(member[4][1].text),\n int(member[4][2].text),\n int(member[4][3].text)\n )\n xml_list.append(value)\n column_name = ['filename', 'width', 'height',\n 'class', 'xmin', 'ymin', 'xmax', 'ymax']\n xml_df = pd.DataFrame(xml_list, columns=column_name)\n return xml_df\n\n\ndef main():\n # Initiate argument parser\n parser = argparse.ArgumentParser(\n description=\"Sample TensorFlow XML-to-CSV converter\")\n parser.add_argument(\"-i\",\n \"--inputDir\",\n help=\"Path to the folder where the input .xml files are stored\",\n type=str)\n parser.add_argument(\"-o\",\n \"--outputFile\",\n help=\"Name of output .csv file (including path)\", type=str)\n args = parser.parse_args()\n\n if(args.inputDir is None):\n args.inputDir = os.getcwd()\n if(args.outputFile is None):\n args.outputFile = args.inputDir + \"\/labels.csv\"\n\n assert(os.path.isdir(args.inputDir))\n\n xml_df = xml_to_csv(args.inputDir)\n xml_df.to_csv(\n args.outputFile, index=None)\n print('Successfully converted xml to csv.')\n\n\nif __name__ == '__main__':\n main()<\/code><\/pre>\n\n\n\nAfter creating the csv files, we will use another script to create the TF.records. In the script below as you can see from the usage specified we take the csv files and create the record files. Note we use only one label here called numberplate. In case there are more, edit the FLAGS.label accordingly.<\/p>\n\n\n\n
\"\"\"<\/em>\nUsage:<\/em>\n\n# Create train data:<\/em>\npython generate_tfrecord.py --label=<LABEL> --csv_input=<PATH_TO_ANNOTATIONS_FOLDER>\/train_labels.csv --output_path=<PATH_TO_ANNOTATIONS_FOLDER>\/train.record<\/em>\n\n# Create test data:<\/em>\npython generate_tfrecord.py --label=<LABEL> --csv_input=<PATH_TO_ANNOTATIONS_FOLDER>\/test_labels.csv --output_path=<PATH_TO_ANNOTATIONS_FOLDER>\/test.record<\/em>\n\"\"\"<\/em>\n\nfrom<\/strong> __future__<\/strong> import<\/strong> division\nfrom<\/strong> __future__<\/strong> import<\/strong> print_function\nfrom<\/strong> __future__<\/strong> import<\/strong> absolute_import\n\nimport<\/strong> os<\/strong>\nimport<\/strong> io<\/strong>\nimport<\/strong> pandas<\/strong> as<\/strong> pd<\/strong>\nimport<\/strong> tensorflow<\/strong> as<\/strong> tf<\/strong>\nimport<\/strong> sys<\/strong>\nsys.path.append(\"..\/..\/models\/research\")\n\nfrom<\/strong> PIL<\/strong> import<\/strong> Image\nfrom<\/strong> object_detection.utils<\/strong> import<\/strong> dataset_util\nfrom<\/strong> collections<\/strong> import<\/strong> namedtuple, OrderedDict\n\nflags = tf.app.flags\nflags.DEFINE_string('csv_input', '', 'Path to the CSV input')\nflags.DEFINE_string('output_path', '', 'Path to output TFRecord')\nflags.DEFINE_string('label', '', 'Name of class label')\n# if your image has more labels input them as<\/em>\n# flags.DEFINE_string('label0', '', 'Name of class[0] label')<\/em>\n# flags.DEFINE_string('label1', '', 'Name of class[1] label')<\/em>\n# and so on.<\/em>\nflags.DEFINE_string('img_path', '', 'Path to images')\nFLAGS = flags.FLAGS\n\n\n# TO-DO replace this with label map<\/em>\n# for multiple labels add more else if statements<\/em>\ndef<\/strong> class_text_to_int(row_label):\n if<\/strong> row_label == FLAGS.label: # 'numberplate':<\/em>\n return<\/strong> 1\n # comment upper if statement and uncomment these statements for multiple labelling<\/em>\n # if row_label == FLAGS.label0:<\/em>\n # return 1<\/em>\n # elif row_label == FLAGS.label1:<\/em>\n # return 0<\/em>\n else<\/strong>:\n None\n\n\ndef<\/strong> split(df, group):\n data = namedtuple('data', ['filename', 'object'])\n gb = df.groupby(group)\n return<\/strong> [data(filename, gb.get_group(x)) for<\/strong> filename, x in<\/strong> zip(gb.groups.keys(), gb.groups)]\n\n\ndef<\/strong> create_tf_example(group, path):\n with<\/strong> tf.gfile.GFile(os.path.join(path, '{}'.format(group.filename)), 'rb') as<\/strong> fid:\n encoded_jpg = fid.read()\n encoded_jpg_io = io.BytesIO(encoded_jpg)\n image = Image.open(encoded_jpg_io)\n width, height = image.size\n\n filename = group.filename.encode('utf8')\n image_format = b'jpg'\n # check if the image format is matching with your images.<\/em>\n xmins = []\n xmaxs = []\n ymins = []\n ymaxs = []\n classes_text = []\n classes = []\n\n for<\/strong> index, row in<\/strong> group.object.iterrows():\n xmins.append(row['xmin'] \/ width)\n xmaxs.append(row['xmax'] \/ width)\n ymins.append(row['ymin'] \/ height)\n ymaxs.append(row['ymax'] \/ height)\n classes_text.append(row['class'].encode('utf8'))\n classes.append(class_text_to_int(row['class']))\n\n tf_example = tf.train.Example(features=tf.train.Features(feature={\n 'image\/height': dataset_util.int64_feature(height),\n 'image\/width': dataset_util.int64_feature(width),\n 'image\/filename': dataset_util.bytes_feature(filename),\n 'image\/source_id': dataset_util.bytes_feature(filename),\n 'image\/encoded': dataset_util.bytes_feature(encoded_jpg),\n 'image\/format': dataset_util.bytes_feature(image_format),\n 'image\/object\/bbox\/xmin': dataset_util.float_list_feature(xmins),\n 'image\/object\/bbox\/xmax': dataset_util.float_list_feature(xmaxs),\n 'image\/object\/bbox\/ymin': dataset_util.float_list_feature(ymins),\n 'image\/object\/bbox\/ymax': dataset_util.float_list_feature(ymaxs),\n 'image\/object\/class\/text': dataset_util.bytes_list_feature(classes_text),\n 'image\/object\/class\/label': dataset_util.int64_list_feature(classes),\n }))\n return<\/strong> tf_example\n\n\ndef<\/strong> main(_):\n writer = tf.python_io.TFRecordWriter(FLAGS.output_path)\n path = os.path.join(os.getcwd(), FLAGS.img_path)\n examples = pd.read_csv(FLAGS.csv_input)\n grouped = split(examples, 'filename')\n for<\/strong> group in<\/strong> grouped:\n tf_example = create_tf_example(group, path)\n writer.write(tf_example.SerializeToString())\n\n writer.close()\n output_path = os.path.join(os.getcwd(), FLAGS.output_path)\n print<\/strong>('Successfully created the TFRecords: {}'.format(output_path))\n\n\nif<\/strong> __name__ == '__main__':\n tf.app.run()<\/pre>\n\n\n\nRefer to the link<\/a> for more details.<\/p>\n\n\n\n
What is Transfer Training<\/h3>\n\n\n\n
We have converted the annotated images to a format compatible with tensorflow. In this section we will train an existing model to detect and localise a number plate in a picture. We will take an existing pre-trained model and feed it the annotated pictures from above. This process is called transfer training. The premise of transfer training is that the pre-existing model has already been extensively trained with a large set of images. So that model can already classify and detect a number of pictures. Tensorflow object detection API provides us with several models which have been pre-trained exhaustively with a large data set. By leveraging this learning we can get a working model that needs fewer number of images and fewer iterations. A list of these available models can be found in the model zoo<\/a>.<\/p>\n\n\n\n
MOdeL ZOO<\/h4>\n\n\n\n
We have multiple models in the model zoo; some are faster, others more accurate. Some of the models can even do segmentation, which identifies the pixels that make that object. For our application detecting bounding boxes is sufficient. The list of available models speed and accuracy measured and mAP (mean Average Precision) on the coco set is below. Higher the mAP, better the accuracy.<\/p>\n\n\n\n
- Tensorflow Object Detection API instruction<\/a>: This API models have to be downloaded and kept locally. We will store the models under the directory models as seen in the Recommended Directory Structure. <\/li><\/ul>\n\n\n\n
![\"\"](\"https:\/\/blog.cloudxlab.com\/wp-content\/uploads\/2020\/04\/licence_plate_detected.png\")
<\/figure>\n\n\n\n