{"id":3053,"date":"2020-05-12T10:29:42","date_gmt":"2020-05-12T10:29:42","guid":{"rendered":"https:\/\/cloudxlab.com\/blog\/?p=3053"},"modified":"2020-05-19T05:37:19","modified_gmt":"2020-05-19T05:37:19","slug":"video-processing-with-yolov4-and-tensorflow","status":"publish","type":"post","link":"https:\/\/cloudxlab.com\/blog\/video-processing-with-yolov4-and-tensorflow\/","title":{"rendered":"Video processing with YOLOv4 and TensorFlow"},"content":{"rendered":"\n<p>In this blog we will show how to process video with YOLOv4 and tensorflow. <a href=\"https:\/\/arxiv.org\/abs\/2004.10934\">YOLOv4<\/a> is out and it&#8217;s hot. YOLOv4 is significantly better than YOLOv3 as can be seen in the pic below.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img src=\"https:\/\/user-images.githubusercontent.com\/4096485\/80213782-5f1e3480-8642-11ea-8fdf-0e6b9a6b5f4c.png\" alt=\"YOLOv4 video processing tensowrflow\" \/><\/figure>\n\n\n\n<p class=\"has-text-align-center\"><a href=\"https:\/\/github.com\/AlexeyAB\/darknet\">Source<\/a><\/p>\n\n\n\n<!--more-->\n\n\n\n<h2>YOLO v4 Overview<\/h2>\n\n\n\n<p class=\"has-text-align-left\">YOLOv4 uses several of universal features like Weighted-Residual-Connections (WRC), Cross-Stage-Partial-connections (CSP), Cross mini-Batch Normalization (CmBN), Self-adversarial-training (SAT) and Mish-activation. They also use new features: WRC, CSP, CmBN, SAT, Mish activation, Mosaic data augmentation, CmBN, DropBlock regularization, and CIoU loss. A combination of known and new features has enabled them to achieve 43.5%<br>AP (65.7% AP50) for the MS COCO dataset at a realtime speed of \u223c65 FPS on Tesla V100. It is faster and more accurate than YOLOv3 and faster than EfficientDet for similar accuracies.<\/p>\n\n\n\n<p>The authors have tried to design a model that can be trained efficiently on a single GPU. It is optimised to work well in production systems. Users can train and implement YOLOv4 based programs on single GPU systems, keeping the cost low. For more details please see the <a href=\"https:\/\/arxiv.org\/pdf\/2004.10934.pdf\">YOLOv4 paper<\/a>.<\/p>\n\n\n\n<h2>Video processing with YOLO v4 and TensorFlow<\/h2>\n\n\n\n<p>We will use this <a href=\"https:\/\/github.com\/hunglc007\/tensorflow-yolov4-tflite\">implementation<\/a> of YOLO in python and Tensorflow in our work. The github project provides implementation in YOLOv3, YOLOv4. It also has methods to convert YOLO weights files to tflite (tensorflow lite models). Tensorflow lite models are smaller and can be implemented for speed at a cost of accuracy. We can also use Tensorflow lite models on edge devices like mobiles, etc. <\/p>\n\n\n\n<p>We will first create a development environment using <a href=\"https:\/\/virtualenvwrapper.readthedocs.io\/en\/latest\/install.html\">virtualenv<\/a>.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">mkvirtualenv tf-yolo<\/code><\/pre>\n\n\n\n<p>Clone the project.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">git clone https:\/\/github.com\/hunglc007\/tensorflow-yolov4-tflite\ncd tensorflow-yolo4-tflite<\/code><\/pre>\n\n\n\n<p>Install the requirements file.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">pip install -r requirements.txt<\/code><\/pre>\n\n\n\n<p>I found an error in the repository, which was fixed by creating an empty __init__.py in the core directory.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">touch core\/__init__.py<\/code><\/pre>\n\n\n\n<p>I have made a new file for processing video using the existing code to process image. <\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"language-python\">import time\nfrom absl import app, flags, logging\nfrom absl.flags import FLAGS\nimport core\nimport core.utils as utils\nfrom core.yolov4 import YOLOv4, YOLOv3, YOLOv3_tiny, decode\nfrom PIL import Image\nfrom core.config import cfg\nimport cv2\nimport numpy as np\nimport tensorflow as tf\n\nimport imutils\nfrom imutils.video import FPS\nfrom imutils.video import VideoStream\n\n\n\nflags.DEFINE_string('framework', 'tf', '(tf, tflite')\nflags.DEFINE_string('weights', '..\/darknet\/yolov4.weights',\n                    'path to weights file')\nflags.DEFINE_integer('size', 608, 'resize images to')\nflags.DEFINE_boolean('tiny', False, 'yolo or yolo-tiny')\nflags.DEFINE_string('model', 'yolov4', 'yolov3 or yolov4')\nflags.DEFINE_string('video', '.\/data\/road.mp4', 'path to input video')\nflags.DEFINE_string('output', 'out.avi', 'path to output video')\n\ndef main(_argv):\n        #FPS calculator enables\n\n    if FLAGS.tiny:\n        STRIDES = np.array(cfg.YOLO.STRIDES_TINY)\n        ANCHORS = utils.get_anchors(cfg.YOLO.ANCHORS_TINY, FLAGS.tiny)\n    else:\n        STRIDES = np.array(cfg.YOLO.STRIDES)\n        if FLAGS.model == 'yolov4':\n            ANCHORS = utils.get_anchors(cfg.YOLO.ANCHORS, FLAGS.tiny)\n        else:\n            ANCHORS = utils.get_anchors(cfg.YOLO.ANCHORS_V3, FLAGS.tiny)\n    NUM_CLASS = len(utils.read_class_names(cfg.YOLO.CLASSES))\n    XYSCALE = cfg.YOLO.XYSCALE\n    input_size = FLAGS.size\n    video_path = FLAGS.video\n    output=FLAGS.output\n    fps = FPS().start()\n\n    #Declaration of output file to save the analysed video\n\n    cnt=0\n    vs = cv2.VideoCapture(video_path)\n    while True and cnt &lt; 100:\n        try:\n            (grabbed, original_image) = vs.read()\n            original_image_size = original_image.shape[:2]\n            #print (original_image_size)\n        except:\n            break\n        cnt+=1\n\n\n\n        if cnt == 1:\n            fourcc = cv2.VideoWriter_fourcc(*\"MJPG\")\n            writer = cv2.VideoWriter(output, fourcc, 30, original_image_size[::-1], True)\n\n\n\n\n        original_image = cv2.cvtColor(original_image, cv2.COLOR_BGR2RGB)\n        original_image_size = original_image.shape[:2]\n\n        image_data = utils.image_preporcess(np.copy(original_image), [input_size, input_size])\n        image_data = image_data[np.newaxis, ...].astype(np.float32)\n        if FLAGS.framework == 'tf':\n            if cnt == 1:\n                input_layer = tf.keras.layers.Input([input_size, input_size, 3])\n                if FLAGS.tiny:\n                    feature_maps = YOLOv3_tiny(input_layer, NUM_CLASS)\n                    bbox_tensors = []\n                    for i, fm in enumerate(feature_maps):\n                        bbox_tensor = decode(fm, NUM_CLASS, i)\n                        bbox_tensors.append(bbox_tensor)\n                    model = tf.keras.Model(input_layer, bbox_tensors)\n                    utils.load_weights_tiny(model, FLAGS.weights)\n                else:\n                    if FLAGS.model == 'yolov3':\n                        feature_maps = YOLOv3(input_layer, NUM_CLASS)\n                        bbox_tensors = []\n                        for i, fm in enumerate(feature_maps):\n                            bbox_tensor = decode(fm, NUM_CLASS, i)\n                            bbox_tensors.append(bbox_tensor)\n                        model = tf.keras.Model(input_layer, bbox_tensors)\n                        utils.load_weights_v3(model, FLAGS.weights)\n                    elif FLAGS.model == 'yolov4':\n                        feature_maps = YOLOv4(input_layer, NUM_CLASS)\n                        bbox_tensors = []\n                        for i, fm in enumerate(feature_maps):\n                            bbox_tensor = decode(fm, NUM_CLASS, i)\n                            bbox_tensors.append(bbox_tensor)\n                        model = tf.keras.Model(input_layer, bbox_tensors)\n                        utils.load_weights(model, FLAGS.weights)\n\n                model.summary()\n            pred_bbox = model.predict(image_data)\n        else:\n            if cnt==1:\n                # Load TFLite model and allocate tensors.\n                interpreter = tf.lite.Interpreter(model_path=FLAGS.weights)\n                interpreter.allocate_tensors()\n                # Get input and output tensors.\n                input_details = interpreter.get_input_details()\n                output_details = interpreter.get_output_details()\n                print(input_details)\n                print(output_details)\n                interpreter.set_tensor(input_details[0]['index'], image_data)\n                interpreter.invoke()\n            pred_bbox = [interpreter.get_tensor(output_details[i]['index']) for i in range(len(output_details))]\n\n        if FLAGS.model == 'yolov4':\n            pred_bbox = utils.postprocess_bbbox(pred_bbox, ANCHORS, STRIDES, XYSCALE)\n        else:\n            pred_bbox = utils.postprocess_bbbox(pred_bbox, ANCHORS, STRIDES)\n        bboxes = utils.postprocess_boxes(pred_bbox, original_image_size, input_size, 0.25)\n        bboxes = utils.nms(bboxes, 0.213, method='nms')\n\n        image = utils.draw_bbox(original_image, bboxes)\n        image = Image.fromarray(image)\n\n        #image.show()\n        image = cv2.cvtColor(np.array(image), cv2.COLOR_BGR2RGB)\n\n        cv2.imshow(\"output\", cv2.resize(image, (800, 600)))\n        writer.write(image)\n        key = cv2.waitKey(1) &amp; 0xFF\n        if key == ord(\"q\"):\n\t           break\n        fps.update()\n\n    fps.stop()\n\n    print(\"[INFO] elasped time: {:.2f}\".format(fps.elapsed()))\n    print(\"[INFO] approx. FPS: {:.2f}\".format(fps.fps()))\n\n    # do a bit of cleanup\n    cv2.destroyAllWindows()\n\n    # release the file pointers\n    print(\"[INFO] cleaning up...\")\n    writer.release()\n    vs.release()\n\n\nif __name__ == '__main__':\n    try:\n        app.run(main)\n    except SystemExit:\n        pass\n<\/code><\/pre>\n\n\n\n<p>Video processing with YOLOv4 and TensorFlow<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">python detect.py --weights .\/data\/yolov4.weights --framework tf --size 608 --video .\/data\/road.mp4<\/pre>\n\n\n\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"<p>In this blog we will show how to process video with YOLOv4 and tensorflow. YOLOv4 is out and it&#8217;s hot. YOLOv4 is significantly better than YOLOv3 as can be seen in the pic below. Source<\/p>\n","protected":false},"author":26,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[67,1],"tags":[61],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v16.2 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Video processing with YOLOv4 and TensorFlow | CloudxLab Blog<\/title>\n<meta name=\"description\" content=\"In this blog we demonstrate Video processing with YOLOv4 and TensorFlow. YOLOv4 is better than YOLOv3 in both speed and accuracy.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/cloudxlab.com\/blog\/video-processing-with-yolov4-and-tensorflow\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Video processing with YOLOv4 and TensorFlow | CloudxLab Blog\" \/>\n<meta property=\"og:description\" content=\"In this blog we demonstrate Video processing with YOLOv4 and TensorFlow. YOLOv4 is better than YOLOv3 in both speed and accuracy.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/cloudxlab.com\/blog\/video-processing-with-yolov4-and-tensorflow\/\" \/>\n<meta property=\"og:site_name\" content=\"CloudxLab Blog\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/cloudxlab\" \/>\n<meta property=\"article:published_time\" content=\"2020-05-12T10:29:42+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2020-05-19T05:37:19+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/user-images.githubusercontent.com\/4096485\/80213782-5f1e3480-8642-11ea-8fdf-0e6b9a6b5f4c.png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@CloudxLab\" \/>\n<meta name=\"twitter:site\" content=\"@CloudxLab\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\">\n\t<meta name=\"twitter:data1\" content=\"5 minutes\">\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebSite\",\"@id\":\"https:\/\/cloudxlab.com\/blog\/#website\",\"url\":\"https:\/\/cloudxlab.com\/blog\/\",\"name\":\"CloudxLab Blog\",\"description\":\"Learn AI, Machine Learning, Deep Learning, Devops &amp; Big Data\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":\"https:\/\/cloudxlab.com\/blog\/?s={search_term_string}\",\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"ImageObject\",\"@id\":\"https:\/\/cloudxlab.com\/blog\/video-processing-with-yolov4-and-tensorflow\/#primaryimage\",\"inLanguage\":\"en-US\",\"url\":\"https:\/\/user-images.githubusercontent.com\/4096485\/80213782-5f1e3480-8642-11ea-8fdf-0e6b9a6b5f4c.png\",\"contentUrl\":\"https:\/\/user-images.githubusercontent.com\/4096485\/80213782-5f1e3480-8642-11ea-8fdf-0e6b9a6b5f4c.png\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/cloudxlab.com\/blog\/video-processing-with-yolov4-and-tensorflow\/#webpage\",\"url\":\"https:\/\/cloudxlab.com\/blog\/video-processing-with-yolov4-and-tensorflow\/\",\"name\":\"Video processing with YOLOv4 and TensorFlow | CloudxLab Blog\",\"isPartOf\":{\"@id\":\"https:\/\/cloudxlab.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/cloudxlab.com\/blog\/video-processing-with-yolov4-and-tensorflow\/#primaryimage\"},\"datePublished\":\"2020-05-12T10:29:42+00:00\",\"dateModified\":\"2020-05-19T05:37:19+00:00\",\"author\":{\"@id\":\"https:\/\/cloudxlab.com\/blog\/#\/schema\/person\/e2c5cc7b933ebd4b15f9b463dc7cf1b4\"},\"description\":\"In this blog we demonstrate Video processing with YOLOv4 and TensorFlow. YOLOv4 is better than YOLOv3 in both speed and accuracy.\",\"breadcrumb\":{\"@id\":\"https:\/\/cloudxlab.com\/blog\/video-processing-with-yolov4-and-tensorflow\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/cloudxlab.com\/blog\/video-processing-with-yolov4-and-tensorflow\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/cloudxlab.com\/blog\/video-processing-with-yolov4-and-tensorflow\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"item\":{\"@type\":\"WebPage\",\"@id\":\"https:\/\/cloudxlab.com\/blog\/\",\"url\":\"https:\/\/cloudxlab.com\/blog\/\",\"name\":\"Home\"}},{\"@type\":\"ListItem\",\"position\":2,\"item\":{\"@id\":\"https:\/\/cloudxlab.com\/blog\/video-processing-with-yolov4-and-tensorflow\/#webpage\"}}]},{\"@type\":\"Person\",\"@id\":\"https:\/\/cloudxlab.com\/blog\/#\/schema\/person\/e2c5cc7b933ebd4b15f9b463dc7cf1b4\",\"name\":\"Praveen Pavithran\",\"image\":{\"@type\":\"ImageObject\",\"@id\":\"https:\/\/cloudxlab.com\/blog\/#personlogo\",\"inLanguage\":\"en-US\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/03c8d253347dcf9e04ec550cd6144973?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/03c8d253347dcf9e04ec550cd6144973?s=96&d=mm&r=g\",\"caption\":\"Praveen Pavithran\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","_links":{"self":[{"href":"https:\/\/cloudxlab.com\/blog\/wp-json\/wp\/v2\/posts\/3053"}],"collection":[{"href":"https:\/\/cloudxlab.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/cloudxlab.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/cloudxlab.com\/blog\/wp-json\/wp\/v2\/users\/26"}],"replies":[{"embeddable":true,"href":"https:\/\/cloudxlab.com\/blog\/wp-json\/wp\/v2\/comments?post=3053"}],"version-history":[{"count":3,"href":"https:\/\/cloudxlab.com\/blog\/wp-json\/wp\/v2\/posts\/3053\/revisions"}],"predecessor-version":[{"id":3061,"href":"https:\/\/cloudxlab.com\/blog\/wp-json\/wp\/v2\/posts\/3053\/revisions\/3061"}],"wp:attachment":[{"href":"https:\/\/cloudxlab.com\/blog\/wp-json\/wp\/v2\/media?parent=3053"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/cloudxlab.com\/blog\/wp-json\/wp\/v2\/categories?post=3053"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/cloudxlab.com\/blog\/wp-json\/wp\/v2\/tags?post=3053"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}