Scribd Logo
Upload

Welcome to Scribd!

  • 44 views

    CyPhyss23 DD PPT

    Uploaded by

    Dibya Dillip
    A Deep Learning Approach for Aircraft Surface Defect Detection”, in IEEE Access, Vol. 8, 2020 [6] YOLOv5: https://github.com/ultralytics/yolov5 [7] AutoML2.0: https://github.com/Anthropic/AutoML2 [8] OpenCV: https://opencv.org/ [9] Scikit-Learn: https://scikit-learn.org/stable/ [10] TensorFlow: https://www.tensorflow.org/ [11] Keras: https://keras.io/ [12] PyTorch: https://pytorch.org/ [13] NumPy

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd

    CyPhyss23 DD PPT

    Uploaded by

    Dibya Dillip
    44 views35 pages
    A Deep Learning Approach for Aircraft Surface Defect Detection”, in IEEE Access, Vol. 8, 2020 [6] YOLOv5: https://github.com/ultralytics/yolov5 [7] AutoML2.0: https://github.com/Anthropic/AutoML2 [8] OpenCV: https://opencv.org/ [9] Scikit-Learn: https://scikit-learn.org/stable/ [10] TensorFlow: https://www.tensorflow.org/ [11] Keras: https://keras.io/ [12] PyTorch: https://pytorch.org/ [13] NumPy

    Original Description:

    Cyphyss 2023 ppt @IITKGP

    Original Title

    CyPhyss23_DD_PPT

    Copyright

    © © All Rights Reserved

    Available Formats

    PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    A Deep Learning Approach for Aircraft Surface Defect Detection”, in IEEE Access, Vol. 8, 2020 [6] YOLOv5: https://github.com/ultralytics/yolov5 [7] AutoML2.0: https://github.com/Anthropic/AutoML2 [8] OpenCV: https://opencv.org/ [9] Scikit-Learn: https://scikit-learn.org/stable/ [10] TensorFlow: https://www.tensorflow.org/ [11] Keras: https://keras.io/ [12] PyTorch: https://pytorch.org/ [13] NumPy

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Download as pdf or txt
    44 views35 pages

    CyPhyss23 DD PPT

    Uploaded by

    Dibya Dillip
    A Deep Learning Approach for Aircraft Surface Defect Detection”, in IEEE Access, Vol. 8, 2020 [6] YOLOv5: https://github.com/ultralytics/yolov5 [7] AutoML2.0: https://github.com/Anthropic/AutoML2 [8] OpenCV: https://opencv.org/ [9] Scikit-Learn: https://scikit-learn.org/stable/ [10] TensorFlow: https://www.tensorflow.org/ [11] Keras: https://keras.io/ [12] PyTorch: https://pytorch.org/ [13] NumPy

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Download as pdf or txt
    of 35

    UAV Based Aircraft Surface Inspection Using

    Computer Vision and Deep Learning

    Sqn Ldr Dibya Dillip

    M.Tech (Artificial Intellience),


    Indian Institute of Science, Bangalore

    [20 July 2023]

    Sqn Ldr Dibya Dillip CyPhyss 2023 [20 July 2023] 1 / 35


    Introduction

    Manual inspection of aircraft (military or commercial) is time intensive


    and error prone due to Human factor.

    Inevitable need for automation of aircraft surface inspection.

    Obviates discretionary limitations of human expertise and skills,


    fatigue, willful negligence etc.

    Pre - Deep Learning era approaches were primarily statistical methods


    or classical Machine Learning based methods

    Sqn Ldr Dibya Dillip CyPhyss 2023 [20 July 2023] 2 / 35


    Classical Approaches and Inherent Limitations
    Heuristic Based Method
    Feature Extraction (color and textural) - Edge Detection,
    Histogram Thresholding, Wavelet Transforms etc.
    Classification - pre-defined discriminant functions

    Classical ML Method
    Feature Crafting - filter banks, morphological operations,
    histograms and transforms
    Classification - hand crafted rule based decision making (SVM,
    decision tree, k NN)

    Limitations
    Heavy reliance on crafted features (complex defects, background)
    Inflexibility (not adaptable to diverse aircraft / defects)
    Performance saturation with increasing amount of data
    Sqn Ldr Dibya Dillip CyPhyss 2023 [20 July 2023] 3 / 35
    Problem Statement

    Primary Objective
    To develop a computer vision and deep learning based methodology for
    automated detection of surface defects on aircraft external structure
    using images / videos captured through an Unmanned Aerial Vehicle

    Associated Expectations
    Detection of multiple defect classes viz. Crack, Dent, Paint
    Peel-off, Scratch, Missing Rivet
    Defect recognition and classification (labelling)
    Flexibility of application over multiple aircraft types

    Sqn Ldr Dibya Dillip CyPhyss 2023 [20 July 2023] 4 / 35


    Summary of Related Works

    Sqn Ldr Dibya Dillip CyPhyss 2023 [20 July 2023] 5 / 35


    Methodology

    Custom Dataset Creation - Phase 1


    11 aircraft, 9 types of aircraft (IISc - 3, AFTC - 8)
    2466 Original images filtered to 2023

    Sqn Ldr Dibya Dillip CyPhyss 2023 [20 July 2023] 6 / 35


    Aircrafts Photographed

    Sqn Ldr Dibya Dillip CyPhyss 2023 [20 July 2023] 7 / 35


    Annotations

    Sqn Ldr Dibya Dillip CyPhyss 2023 [20 July 2023] 8 / 35


    Data Augmentation

    Sqn Ldr Dibya Dillip CyPhyss 2023 [20 July 2023] 9 / 35


    Architecture Trials

    End-to-end Deep Learning Architectures


    CNN based SOTA models
    Initial layers - Feature Extraction
    Later layers - Detection Head (Localization using bounding box)
    Two-Pass models - RPN based (RCNN, Fast / Faster RCNN,
    Mask-RCNN etc)
    Single Pass models - Non RPN based (SSD, YOLO, EfficientDet
    etc.)

    Sqn Ldr Dibya Dillip CyPhyss 2023 [20 July 2023] 10 / 35


    End-to-end Deep Learning Architecture Trials

    YOLOv5 Trials - Vanilla version

    Sqn Ldr Dibya Dillip CyPhyss 2023 [20 July 2023] 11 / 35


    End-to-end Deep Learning Architecture Trials

    YOLOv8 Trials - Vanilla version

    Sqn Ldr Dibya Dillip CyPhyss 2023 [20 July 2023] 12 / 35


    Architecture Trials

    Two Stage Architecture


    Complex defect texture - difficult for even Off-the-Shelf SOTA
    object detection models
    Need for customized architecture (Two - Stage)

    Stage 1: CNN based Object Detection pipeline (only ’defect’ class)


    : Pre-Classifier = crop out defect patches
    Stage 2: CNN based Image Classification pipeline (5 defect
    classes) takes cropped image from Stage 1 as input

    Integrate both stages : overlay class labels from Stage 2 with


    bounding boxes from Stage 1 onto original input image

    Sqn Ldr Dibya Dillip CyPhyss 2023 [20 July 2023] 13 / 35


    Stage 1 Architecture Trials : Summary

    Sqn Ldr Dibya Dillip CyPhyss 2023 [20 July 2023] 14 / 35


    Custom Dataset for Stage 2 (Classification)

    Example cropped labeled images

    Sqn Ldr Dibya Dillip CyPhyss 2023 [20 July 2023] 15 / 35


    Stage 2 Architecture Trials : Summary

    Sqn Ldr Dibya Dillip CyPhyss 2023 [20 July 2023] 16 / 35


    Implementation
    Hardware / Environment
    Google Collaboratory, Python 3.10, NVIDIA Tesla T4 GPU, 16GB
    Graphics Memory

    Sqn Ldr Dibya Dillip CyPhyss 2023 [20 July 2023] 17 / 35


    Stage 1 Implementation: YOLOv5-m Object Detection
    Model

    Sqn Ldr Dibya Dillip CyPhyss 2023 [20 July 2023] 18 / 35


    Stage 1 Implementation: YOLOv5-m fine-tuning

    fine-tuning parameters
    maximize batch size optimizing image resolution
    lower initial learning rate without weight decay

    Sqn Ldr Dibya Dillip CyPhyss 2023 [20 July 2023] 19 / 35


    Stage 1 Implementation: Improving Accuracy
    YOLOv5-m Architecture
    Backbone: CSPDarknet, 10 Layers, Feature Extraction
    Neck: PANet, adds contextual and localizing info
    Head: Predictions - bounding boxes and class probabilities

    Sqn Ldr Dibya Dillip CyPhyss 2023 [20 July 2023] 20 / 35


    Stage 1 Implementation: Improving Accuracy

    YOLOv5-m: Loss Computation


    Class Loss: Classification judgement error in multi-class detection
    or classification tasks. Binary Cross Entropy (BCE) loss. Nil for
    single class detection
    Objectness Loss: Error in detecting presence / absence of an
    object instance in a grid cell. Also BCE loss. P3, P4, P5 layers in
    Head make predictions at different scales. Loss at different scales
    are weighted differently
    Location / Box Loss: Error in localizing object instance.
    Measured as Complete IoU (CIoU) loss

    Total Loss: Weighted combination of all three losses

    Sqn Ldr Dibya Dillip CyPhyss 2023 [20 July 2023] 21 / 35


    Improving Stage 1 Accuracy: Novel Approach 1

    Strategy: Increase mAP + decrease Object Loss and Box Loss

    Freezing Backbone Layer


    Improve model’s localizing ability: even distribution of objects in
    image. 985 additional images acquired from AFTC
    Freeze backbone layers weights: Retain feature knowledge
    Update ’neck’ and ’head’ weights for better contextual knowledge,
    localization (bounding box regression and detection capability
    Transfer Learning: Initialize weights from last best model and
    retrain
    Switch to SGD optimizer with higher initial LR and appropriate
    weight decay

    Sqn Ldr Dibya Dillip CyPhyss 2023 [20 July 2023] 22 / 35


    Improving Stage 1 Accuracy: Novel Approach 1

    Accuracy Improvement with Frozen Backbone Training:


    YOLOv5-m

    Sqn Ldr Dibya Dillip CyPhyss 2023 [20 July 2023] 23 / 35


    Improving Stage 1 Accuracy: Novel Approach 2

    k-Fold Cross Validation


    Limited Dataset: prone to overfitting
    Chances of dataset splitting bias

    k-Fold Cross Validation: Benefits


    Uniform subset splitting: Robust evaluation
    Better utilization of available data (both training and evaluation)

    Sqn Ldr Dibya Dillip CyPhyss 2023 [20 July 2023] 24 / 35


    Finalized Stage 1 model ’YOLOv5-m’: Performance
    Plots

    Sqn Ldr Dibya Dillip CyPhyss 2023 [20 July 2023] 25 / 35


    Finalized Stage 1 model ’YOLOv5-m’: Confusion
    Matrix

    Sqn Ldr Dibya Dillip CyPhyss 2023 [20 July 2023] 26 / 35


    Finalized Stage 1 model ’YOLOv5-m’: Predictions

    Sqn Ldr Dibya Dillip CyPhyss 2023 [20 July 2023] 27 / 35


    Stage 2 Implementation: RF AutoML2.0 Classification
    Model
    Classification Dataset Statistics

    Classwise Model Performance

    Sqn Ldr Dibya Dillip CyPhyss 2023 [20 July 2023] 28 / 35


    Stage 2: RF AutoML2.0 Classification Model
    Normalized Confusion Matrix

    Sqn Ldr Dibya Dillip CyPhyss 2023 [20 July 2023] 29 / 35


    Stage 2: RF AutoML2.0 Classification Model

    Prediction Example

    Sqn Ldr Dibya Dillip CyPhyss 2023 [20 July 2023] 30 / 35


    Model Deployment and Real World Demonstration
    Pipeline integration and real world testing
    Both stages integrated on Python 3.10 Google Colab notebook
    Unseen aircraft images used for deployment demo

    Sqn Ldr Dibya Dillip CyPhyss 2023 [20 July 2023] 31 / 35


    Conclusion

    This work aims at presenting a novel system for UAV based


    aircraft surface inspection using Deep Learning methods.
    It is expected to improve aircraft maintenance efficiency and
    defect detection accuracy.
    The importance of accuracy and efficiency in aircraft maintenance,
    be it military or commercial, can never be over emphasized.

    Sqn Ldr Dibya Dillip CyPhyss 2023 [20 July 2023] 32 / 35


    References
    Key References
    [1] Touba Malekzadeh, Milad Abdollahzadeh, Hossein Nejati,
    Ngai-Man Cheung, ”Aircraft Fuselage Defect Detection Using Deep
    Neural Networks,” in GlobalSIP 2017
    [2] Anil Dog̃ru, Soufiane Bouarfa, Ridwan Arizar, Reyhan Aydog̃an,
    ”Using Convolutional Neural Networks to Automate Aircraft
    Maintenance Visual Inspection”, as article in MDPI Aerospace journal,
    puclished 07 Dec 2020
    [3] Yuanpeng Liu, Jingxuan Dong, Yida Li, Xiaoxi Gong, Jun Wang,
    ”A UAV-Based Aircraft Surface Defect Inspection System via External
    Constraints and Deep Learning”, in IEEE Transactions on
    Instrumentation and Measurement, Vol.71 2022
    [4] Capt. Tyler B. Hussey, USAF, ”Surface Defect Detection in
    Aircraft Skin Visual Navigation based on Forced Feature Selection
    through Segmentation”, Theses and Dissertations. 5044, Air Force
    Institute of Technology, USA
    Sqn Ldr Dibya Dillip CyPhyss 2023 [20 July 2023] 33 / 35
    Key References
    [5] Yuri D. V. Yasuda, Fabio A. M. Cappabianco, Luiz Eduardo G.
    Martins, Jorge A. B. Gripp, ”Automated Visual Inspection of Aircraft
    Exterior Using Deep Learning,” as Thesis paper at Institute of Science
    and Technology Federal University of Sao Paulo Sao Jose dos Campos,
    SP, Brazil, 12247-014, 2022
    [6] Meng DING, Boer WU, Juan XU, Abdul Nasser KASULE, Hongfu
    ZUO, ”Visual inspection of aircraft skin: Automated pixel-level defect
    detection by instance segmentation”, in Chinese Journal of
    Aeronautics, Chinese Society of Aeronautics and Astronautics, 2022
    [7] Yong Yao, Siwen Wei, Jing Wang, ”Surface Defect Detection of
    Aircraft Flared Duct Based on Improved YOLOv4 Algorithm”, as
    Thesis paper at Xidian University, China, 2021
    [8] Rafael Padilla, Sergio L. Netto, and Eduardo A. B. da Silva: “A
    survey on performance metrics for object-detection algorithms” in 2020
    International Conference on Systems, Signals and Image Processing
    (IWSSIP),

    Sqn Ldr Dibya Dillip CyPhyss 2023 [20 July 2023] 34 / 35


    References

    [9] R. Ribani and M. Marengoni, ”A Survey of Transfer Learning for


    Convolutional Neural Networks,” 2019 32nd SIBGRAPI Conference on
    Graphics, Patterns, and Images Tutorials (SIBGRAPI-T), Rio de
    Janeiro, Brazil, 2019,
    [10] Hyun-Ki Jung, Gi-Sang Choi, “Improved YOLOv5: Efficient
    Object Detection Using Drone Images under Various Conditions”,
    Applied Science 2022, published in ResearchGate July 2022
    [11] Nagadevi Darapureddy, Nagaprakash Karatapu, Tirumala Krishna
    Battula, “Research of Machine Learning Algorithms using K-Fold
    Cross Validation”, as paper in IJEAT, August 2019

    Sqn Ldr Dibya Dillip CyPhyss 2023 [20 July 2023] 35 / 35

    You might also like