Abstract
Vision-based quality inspection is a key step to ensure the quality control of complex industrial products. However, accurate defect recognition for complex products with information-rich, structure-irregular and significantly different patterns is still a tough problem, since it causes the strong visual interference. This paper proposes a causal deep learning method (CDLM) to tackle the explainable vision-based quality inspection under visual interference. First, a structural causal model for defect recognition of complex industrial products is constructed and a causal intervention strategy to overcome the background interference is generated. Second, a defect-guided recognition neural network (DGRNN) is constructed, which can realize accurate defect recognition under the training of CDLM via feature-wise causal intervention using two sub-networks with feature difference mechanism. Finally, the causality between defect features and defective product labels can guide the DGRNN to complete the accurate and explainable learning of defect in a causal direction of optimization. Quantitative experiments show that the proposed method achieves recognition accuracy of 94.09% and 93.95% on two fabric datasets respectively, which outperforms the cutting-edge inspection models. Besides, Grad-CAM visualization experiments show that the proposed method successfully captures the data causality and realizes the explainable defect recognition.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data availability
The experimental data used in this study are from ‘ZJU-Leaper: A Benchmark Dataset for Fabric Defect Detection and a Comparative Study’ (Zhang et al., 2020a, 2020b). The DOI of this article is https://doi.org/https://doi.org/10.1109/tai.2021.3057027 and the data are available by visiting https://github.com/nico-zck/ZJU-Leaper-Dataset or http://www.qaas.zju.edu.cn/zju-leaper/.
Abbreviations
- CDLM:
-
Causal deep learning method
- IQI:
-
Intelligent quality inspection
- DL:
-
Deep learning
- DNN:
-
Deep neural networks
- CIP:
-
Complex industrial products
- CPF:
-
Complex patterned fabrics
- SCM:
-
Structural causal model
- DGRNN:
-
Defect-guided recognition neural network
- CIM:
-
Causal intervention module
- FDM:
-
Feature difference module
- BK-Net:
-
Background knowledge network
- DD-Net:
-
Defect detection network
- VQA:
-
Visual question and answering
- Init_DS:
-
Initial down-sampling
- FE:
-
Feature extraction
- Conv:
-
Convolution
- BN:
-
Batch normalization
- FC:
-
Fully-connected
- SA:
-
Spatial attention
- CA:
-
Channel attention
- ACC:
-
Accuracy
- PRE:
-
Precision
- REC:
-
Recall
- AUC:
-
Area under curve
- ROC:
-
Receiver operating characteristic
- PAR:
-
Parameter amount
- FLOPs:
-
Floating-point operations
- FPS:
-
Frame per second
References
Abbasnejad, E., Teney, D., Parvaneh, A., Shi, J., & Hengel, A. V. D. (2020). Counterfactual vision and language learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 10044–10054). https://doi.org/10.1109/cvpr42600.2020.01006
Chan, C. H., & Pang, G. K. (2000). Fabric defect detection by Fourier analysis. IEEE Transactions on Industry Applications, 36(5), 1267–1276. https://doi.org/10.1109/28.871274
Cheng, M., Xu, C., Wang, J., Zhang, W., Zhou, Y., & Zhang, J. (2022). MicroCrack-Net: A deep neural network with outline profile-guided feature augmentation and attention-based multiscale fusion for microcrack detection of tantalum capacitors. IEEE Transactions on Aerospace and Electronic Systems, 58(6), 5141–5152. https://doi.org/10.1109/taes.2022.3181117
Cui, P., & Athey, S. (2022). Stable learning establishes some common ground between causal inference and machine learning. Nature Machine Intelligence, 4(2), 110–115. https://doi.org/10.1038/s42256-022-00445-z
Cui, P., Shen, Z., Li, S., Yao, L., Li, Y., Chu, Z., & Gao, J. (2020). Causal inference meets machine learning. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining (pp. 3527–3528). https://doi.org/10.1145/3394486.3406460
Ding, H., Zhang, J., Kazanzides, P., Wu, J. Y., & Unberath, M. (2022). Carts: causality-driven robot tool segmentation from vision and kinematics data. In International conference on medical image computing and computer-assisted intervention (pp. 387–398). Springer. https://doi.org/10.1007/978-3-031-16449-1_37
Gao, Y., Gao, L., Li, X., & Yan, X. (2020). A semi-supervised convolutional neural network-based method for steel surface defect recognition. Robotics and Computer-Integrated Manufacturing, 61, 101825. https://doi.org/10.1016/j.rcim.2019.101825
Gao, Y., Li, X., Wang, X. V., Wang, L., & Gao, L. (2021). A review on recent advances in vision-based defect recognition towards industrial intelligence. Journal of Manufacturing Systems, 62, 753–766. https://doi.org/10.1016/j.jmsy.2021.05.008
He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770–778). https://doi.org/10.1109/cvpr.2016.90
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017). Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4700–4708). https://doi.org/10.1109/cvpr.2017.243
Iandola, F. N., Han, S., Moskewicz, M. W., Ashraf, K., Dally, W. J., & Keutzer, K. (2016). SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and< 0.5 MB model size. arXiv preprint arXiv:1602.07360
Jing, J., Wang, Z., Rätsch, M., & Zhang, H. (2022). Mobile-Unet: An efficient convolutional neural network for fabric defect detection. Textile Research Journal, 92(1–2), 30–42. https://doi.org/10.1177/0040517520928604
Kang, D., Lai, J., Zhu, J., & Han, Y. (2022). An adaptive feature reconstruction network for the precise segmentation of surface defects on printed circuit boards. Journal of Intelligent Manufacturing, 34(7), 3197–3214. https://doi.org/10.1007/s10845-022-02008-w
Karthikeyan, A., Tiwari, A., Zhong, Y., & Bukkapatnam, S. T. (2022). Explainable AI-infused ultrasonic inspection for internal defect detection. CIRP Annals, 71(1), 449–452. https://doi.org/10.1016/j.cirp.2022.04.036
Kuang, K., Li, L., Geng, Z., Xu, L., Zhang, K., Liao, B., Huang, H., Ding, P., Miao, W., & Jiang, Z. (2020). Causal Inference. Engineering, 6(3), 253–263. https://doi.org/10.1016/j.eng.2019.08.016
Kumar, A. (2008). Computer-vision-based fabric defect detection: A survey. IEEE Transactions on Industrial Electronics, 55(1), 348–363. https://doi.org/10.1109/tie.2007.896476
Lee, M., Jeon, J., & Lee, H. (2022). Explainable AI for domain experts: A post Hoc analysis of deep learning for defect classification of TFT–LCD panels. Journal of Intelligent Manufacturing, 33, 1747–1759. https://doi.org/10.1007/s10845-021-01758-3
Li, C., Gao, G., Liu, Z., Yu, M., & Huang, D. (2018). Fabric defect detection based on biological vision modeling. IEEE Access, 6, 27659–27670. https://doi.org/10.1109/access.2018.2841055
Li, Y., Wang, H., Dang, L. M., Piran, M. J., & Moon, H. (2022). A robust instance segmentation framework for underground sewer defect detection. Measurement, 190, 110727. https://doi.org/10.1016/j.measurement.2022.110727
Li, Y., Zhang, D., & Lee, D. J. (2019). Automatic fabric defect detection with a wide-and-compact network. Neurocomputing, 329, 329–338. https://doi.org/10.1016/j.neucom.2018.10.070
Liu, J., Wang, C., Su, H., Du, B., & Tao, D. (2019). Multistage GAN for fabric defect detection. IEEE Transactions on Image Processing, 29, 3388–3400. https://doi.org/10.1109/tip.2019.2959741
Liu, T., Wang, J., Huang, X., Lu, Y., & Bao, J. (2022a). 3DSMDA-Net: An improved 3DCNN with separable structure and multi-dimensional attention for welding status recognition. Journal of Manufacturing Systems, 62, 811–822. https://doi.org/10.1016/j.jmsy.2021.01.017
Liu, T., Zheng, H., Bao, J., Zheng, P., Wang, J., Yang, C., & Gu, J. (2022b). An explainable laser welding defect recognition method based on multi-scale class activation mapping. IEEE Transactions on Instrumentation and Measurement, 71, 1–12. https://doi.org/10.1109/tim.2022.3148739
Liu, T., Zheng, P., Chen, H., & Zhang, L. (2023). An attention-based bilinear feature extraction mechanism for fine-grained laser welding molten pool/keyhole defect recognition. Journal of Manufacturing Processes, 87, 150–159. https://doi.org/10.1016/j.jmapro.2023.01.014
Mak, K. L., Peng, P., & Yiu, K. F. C. (2009). Fabric defect detection using morphological filters. Image and Vision Computing, 27(10), 1585–1592. https://doi.org/10.1016/j.imavis.2009.03.007
Manivannan, S. (2022). Automatic quality inspection in additive manufacturing using semi-supervised deep learning. Journal of Intelligent Manufacturing. https://doi.org/10.1007/s10845-022-02000-4
Ngan, H. Y., Pang, G. K., Yung, S. P., & Ng, M. K. (2005). Wavelet based methods on patterned fabric defect detection. Pattern Recognition, 38(4), 559–576. https://doi.org/10.1016/j.patcog.2004.07.009
Pearl, J. (2009). Causality. Cambridge University Press.
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016). Why should i trust you? Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining (pp. 1135–1144). https://doi.org/10.1145/2939672.2939778
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., & Chen, L. C. (2018). Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4510–4520). https://doi.org/10.1109/cvpr.2018.00474
Schölkopf, B., Locatello, F., Bauer, S., Ke, N. R., Kalchbrenner, N., Goyal, A., & Bengio, Y. (2021). Toward causal representation learning. Proceedings of the IEEE, 109(5), 612–634. https://doi.org/10.1109/jproc.2021.3058954
Schölkopf, B., & von Kügelgen, J. (2022). From statistical to causal learning. arXiv preprint arXiv:2204.00607
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017). Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision (pp. 618–626). https://doi.org/10.1109/ICCV.2017.74
Sun, Y., Qin, W., & Zhuang, Z. (2022). Nonparametric-copula-entropy and network deconvolution method for causal discovery in complex manufacturing systems. Journal of Intelligent Manufacturing, 33, 1699–1713. https://doi.org/10.1007/s10845-021-01751-w
Sun, Y., Qin, W., Zhuang, Z., & Xu, H. (2021). An adaptive fault detection and root-cause analysis scheme for complex industrial processes using moving window KPCA and information geometric causal inference. Journal of Intelligent Manufacturing, 32, 2007–2021. https://doi.org/10.1007/s10845-021-01752-9
Sundarrajan, K., & Rajendran, B. (2023). Explainable efficient and optimized feature fusion network for surface defect detection. The International Journal of Advanced Manufacturing Technology. https://doi.org/10.1007/s00170-023-11789-0
Tang, J., Zhou, H., Wang, T., Jin, Z., Wang, Y., & Wang, X. (2022). Cascaded foreign object detection in manufacturing processes using convolutional neural networks and synthetic data generation methodology. Journal of Intelligent Manufacturing, 34(7), 2925–2941. https://doi.org/10.1007/s10845-022-01976-3
Wang, J., Gao, P., Zhang, J., Lu, C., & Shen, B. (2023). Knowledge augmented broad learning system for computer vision based mixed-type defect detection in semiconductor manufacturing. Robotics and Computer-Integrated Manufacturing, 81, 102513. https://doi.org/10.1016/j.rcim.2022.102513
Wang, J., Xu, C., Zhang, J., & Zhong, R. (2022a). Big data analytics for intelligent manufacturing systems: A review. Journal of Manufacturing Systems, 62, 738–752. https://doi.org/10.1016/j.jmsy.2021.03.005
Wang, J., Zhao, S., Xu, C., Zhang, J., & Zhong, R. (2022b). Brain-inspired interpretable network pruning for smart vision-based defect detection equipment. IEEE Transactions on Industrial Informatics, 19(2), 1666–1673. https://doi.org/10.1109/TII.2022.3188349
Wang, T., Huang, J., Zhang, H., & Sun, Q. (2020). Visual commonsense r-cnn. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 10760–10770). https://doi.org/10.1109/cvpr42600.2020.01077
Wang, Y., Li, X., Gao, Y., Wang, L., & Gao, L. (2021). A new Feature-Fusion method based on training dataset prototype for surface defect recognition. Advanced Engineering Informatics, 50, 101392. https://doi.org/10.1016/j.aei.2021.101392
Wei, B., Hao, K., Gao, L., & Tang, X. S. (2020). Detecting textile micro-defects: A novel and efficient method based on visual gain mechanism. Information Sciences, 541, 60–74. https://doi.org/10.1016/j.ins.2020.06.035
Woo, S., Park, J., Lee, J. Y., & Kweon, I. S. (2018). Cbam: Convolutional block attention module. In Proceedings of the European conference on computer vision (pp. 3–19). https://doi.org/10.1007/978-3-030-01234-2_1
Xames, M. D., Torsha, F. K., & Sarwar, F. (2022). A systematic literature review on recent trends of machine learning applications in additive manufacturing. Journal of Intelligent Manufacturing, 34(6), 2529–2555. https://doi.org/10.1007/s10845-022-01957-6
Yang, X., Zhang, H., Qi, G., & Cai, J. (2021). Causal attention for vision-language tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 9847–9857). https://doi.org/10.1109/cvpr46437.2021.00972
Zeng, Z., Liu, B., Fu, J., & Chao, H. (2021). Reference-Based Defect Detection Network. IEEE Transactions on Image Processing, 30, 6637–6647. https://doi.org/10.1109/tip.2021.3096067
Zhang, C., Cui, J., Wu, J., & Zhang, X. (2023). Attention mechanism and texture contextual information for steel plate defects detection. Journal of Intelligent Manufacturing. https://doi.org/10.1007/s10845-023-02149-6
Zhang, C., Feng, S., Wang, X., & Wang, Y. (2020a). Zju-leaper: A benchmark dataset for fabric defect detection and a comparative study. IEEE Transactions on Artificial Intelligence, 1(3), 219–232. https://doi.org/10.1109/tai.2021.3057027
Zhang, D., Zhang, H., Tang, J., Hua, X. S., & Sun, Q. (2020b). Causal intervention for weakly-supervised semantic segmentation. Advances in Neural Information Processing Systems, 33, 655–666. https://doi.org/10.5555/3495724.3495780
Zhao, S., Yin, L., Zhang, J., Wang, J., & Zhong, R. (2020a). Real-time fabric defect detection based on multi-scale convolutional neural network. IET Collaborative Intelligent Manufacturing, 2(4), 189–196. https://doi.org/10.1049/iet-cim.2020.0062
Zhao, Y., Hao, K., He, H., Tang, X., & Wei, B. (2020b). A visual long-short-term memory based integrated CNN model for fabric defect image classification. Neurocomputing, 380, 259–270. https://doi.org/10.1016/j.neucom.2019.10.067
Zhou, B., Hua, B., Gu, X., Lu, Y., Peng, T., Zheng, Y., Shen, X., & Bao, J. (2021). An end-to-end tabular information-oriented causality event evolutionary knowledge graph for manufacturing documents. Advanced Engineering Informatics, 50, 101441. https://doi.org/10.1016/j.aei.2021.101441
Acknowledgements
This work is supported by National Natural Science Foundation of China under grant [No. 52275478] and [No. 52375485], Key R&D Program of Shandong Province of China under grant [No. 2021CXGC011004], Young Elite Scientists Sponsorship Program by CAST under grand [No. 2021QNRC001] and Key R&D Plan of Xinjiang Uyghur Autonomous Region of China under grant [No. 2022B01057-1].
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Competing interests
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendix
Appendix
The setting of hyperparameters in the training of DNN is always an empirical problem. In this paper, five different learning rate optimization strategies are selected for experiments, namely constant learning rate, i.e., no scheduler, cosine annealing learning rate, exponential learning rate and its parameter gamma is set to 0.98, step learning rate and its parameter step is set to 5 and gamma is set to 0.9, and one cycle learning rate. The comparison results are shown in Fig. 14.
Comparison of different learning rate optimization strategies
The results show that different learning rate optimization strategies have slight differences in the training of CDLM, but all of these strategies can contribute to a decent score. In this paper, the cosine annealing optimization strategy is chosen because it can make the DGRNN achieve faster convergence speed and more efficient fitting result. Besides, as shown in the training results on the FD_2 data set, it can be seen that the cosine annealing strategy can enable DGRNN to achieve a higher recognition accuracy in the early iterative training stage. However, it also has some shortcomings, such as the existence of certain oscillations after convergence. Therefore, trying different learning rate optimization strategies to assist the training of DGRNN in other visual inspection scenarios is encouraged.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Liang, T., Liu, T., Wang, J. et al. Causal deep learning for explainable vision-based quality inspection under visual interference. J Intell Manuf 36, 1363–1384 (2025). https://doi.org/10.1007/s10845-023-02297-9
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10845-023-02297-9