DL Workbench can lower the precision of a model from FP32 to INT8 with a process called calibration. Calibration accelerates the performance of certain models on hardware that supports INT8. A model in INT8 precision takes up less memory and has higher throughput capacity. Often this performance boost is achieved at the cost of a small accuracy reduction. Post-training Optimization Tool (POT) is designed to accelerate the inference of deep learning models by applying special methods without model retraining or fine-tuning, like post-training quantization.
The POT is aimed to fully automate the model transformation process without a need to change the model on the user's side. The POT is available only in the Intel® distribution of OpenVINO™ toolkit and is not opensourced.
Refer Optimize Model Performance guide to learn, how to achieve INT8 calibration using POT in DL workbench. The guide also contains a workflow video to help you with the process.
To read more about INT8 inference, refer Post-Training Optimization Toolkit.
