Abstract
The inspection of conductive particles after Anisotropic Conductive Film bonding is a crucial step in TFT-LCD manufacturing for quality assurance. Manual inspection under microscope is labor-intensive, time-consuming and error prone. Automatic inspection methods have been proposed by researchers including deep learning methods. However, inspection results are case dependent and existing deep learning-based methods heavily rely on large training dataset which is not given in many real applications. This is because the data available for analysis is limited on the manufacturing lines. To take on this challenge, this paper proposes a novel deep learning method based on modified Mask R-CNN algorithm which performs pixel-level segmentation to detect conductive particles. Under the proposed method, training dataset is augmented by applying novel parametric space envelope technique through a label-preserving transformation. This helps address small sample size prediction problem as well as class imbalance issue within the training dataset. Experimental results show significant improvement over existing methods under real-world constraint of limited training data (i.e., 99.25% overall particle detection accuracy compared with ~ 90% from existing template matching based auto-inspection method). The developed method provides industries an intelligent way to inspect conductive particle in TFT-LCD manufacturing.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data availability
This study is a joint academy and industry research efforts. The raw image data is available (on request) only for research purposes (NOT for commercial gains).
References
Alzubaidi, L., Zhang, J., Humaidi, A. J., Al-Dujaili, A., Duan, Y., Al-Shamma, O., & Farhan, L. (2021). Review of deep learning: Concepts, CNN architectures, challenges, applications, future directions. Journal of Big Data,8(1), 1–74. https://doi.org/10.1186/s40537-021-00444-8
Bilbao, I., & Bilbao, J. (2017). Overfitting problem and the over-training in the era of data: Particularly for artificial neural networks. In Eighth international conference on intelligent computing and information systems (ICICIS) Cairo Egypt (pp. 173–177). https://doi.org/10.1109/INTELCIS.2017.8260032
Charruau, S. (2002). Modelling electrical conductivity of anisotropic chemical adhesives [for flat panel interconnections]. In 2nd international IEEE conference on polymers and adhesives in microelectronics and photonics. POLYTRONIC 2002. Conference Proceedings (Cat. No.02EX599), Zalaegerszeg, Hungary (pp. 7–12). https://doi.org/10.1109/POLYTR.2002.1020175
Chawla, N. V., Japkowicz, N., & Kotcz, A. (2004). Special issue on learning from imbalanced data sets. ACM SIGKDD Explorations Newsletter,6(1), 1–6. https://doi.org/10.1145/1007730.1007733
Dey, N. (2019). Uneven illumination correction of digital images: A survey of the state-of-the-art. Optik,183, 483–495. https://doi.org/10.1016/j.ijleo.2019.02.118
Duraj, A., & Mach, P. (2008). Analysis and prediction of electrical contact resistance for anisotropic conductive adhesives. In 2008 31st international spring seminar on electronics technology (pp. 358–362). https://doi.org/10.1109/ISSE.2008.5276668
Farin, G. (2014). Curves and surfaces for computer-aided geometric design: A practical guide. Elsevier.
Girshick, R. (2015). Fast R-CNN. In 2015 IEEE international conference on computer vision (ICCV) Santiago Chile (pp. 1440–1448). https://doi.org/10.1109/ICCV.2015.169
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014). Rich feature hierarchies for accurate object detection and semantic segmentation. In IEEE conference on computer vision and pattern recognition (pp. 580–587). https://doi.org/10.1109/CVPR.2014.81
Halevy, A., Norvig, P., & Pereira, F. (2009). The unreasonable effectiveness of data. IEEE Intelligent Systems,24(2), 8–12. https://doi.org/10.1109/MIS.2009.36
He, K., Gkioxari, G., Dollár, P., & Girshick, R. (2017). Mask R-CNN. In IEEE international conference on computer vision (ICCV) (pp. 2961–2969). https://doi.org/10.1109/ICCV.2017.322
He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In IEEE conference on computer vision and pattern recognition (CVPR) (pp. 770–778). https://doi.org/10.1109/CVPR.2016.90
Hwang, H., Kong, M., Kim, K., Park, D., Lee, S., Park, S., & Jeong, U. (2021). Stretchable anisotropic conductive film (S-ACF) for electrical interfacing in high-resolution stretchable circuits. Science Advances. https://doi.org/10.1126/sciadv.abh0171
Jia, L., Sheng, X., Xiong, Z., Wang, Z., & Ding, H. (2014). Particle on bump (POB) technique for ultra-fine pitch chip on glass (COG) applications by conductive particles and adhesives. Microelectronics Reliability,54(4), 825–832. https://doi.org/10.1016/j.microrel.2013.11.015
Jin, S., Jeong, Y. H., Choi, E. S., Kim, B., & Yun, W. S. (2014). Relationship between contrast ratio of conductive particle and contact resistance on COG bonding using ACF. Journal of the Korean Society for Precision Engineering,31(9), 831–838.
Johnson, J. M., & Khoshgoftaar, T. M. (2019). Survey on deep learning with class imbalance. Journal of Big Data,6(1), 1–54. https://doi.org/10.1186/s40537-019-0192-5
Kim, M., Lee, M., An, M., & Lee, H. (2020). Effective automatic defect classification process based on CNN with stacking ensemble model for TFT-LCD panel. Journal of Intelligent Manufacturing,31(5), 1165–1174. https://doi.org/10.1007/s10845-019-01502-y
LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature,521(7553), 436–444. https://doi.org/10.1038/nature14539
Lee, C. Y., & Chien, C. F. (2022). Pitfalls and protocols of data science in manufacturing practice. Journal of Intelligent Manufacturing,33, 1189–1207. https://doi.org/10.1007/s10845-020-01711-w
Lee, M., Jeon, J., & Lee, H. (2022). Explainable AI for domain experts: A post Hoc analysis of deep learning for defect classification of TFT–LCD panels. Journal of Intelligent Manufacturing,33, 1747–1759. https://doi.org/10.1007/s10845-021-01758-3
Leevy, J. L., Khoshgoftaar, T. M., Bauder, R. A., & Seliya, N. (2018). A survey on addressing high-class imbalance in big data. Journal of Big Data,5(1), 1–30. https://doi.org/10.1186/s40537-018-0151-6
Lin, C. S., Huang, K. H., Lin, T. C., Shei, H. J., & Tien, C. L. (2011). An automatic inspection method for the fracture conditions of anisotropic conductive film in the TFT-LCD assembly process. International Journal of Optomechatronics,5(3), 286–298. https://doi.org/10.1080/15599612.2011.604114
Lin, T. Y., Dollár, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2017). Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2117–2125). https://doi.org/10.1109/CVPR.2017.106
Liu, E., Chen, K., Xiang, Z., & Zhang, J. (2020). Conductive particle detection via deep learning for ACF bonding in TFT-LCD manufacturing. Journal of Intelligent Manufacturing,31(4), 1037–1049. https://doi.org/10.1007/s10845-019-01494-9
Luo, C., Franciosa, P., Ceglarek, D., Ni, Z., & Jia, F. (2018). A novel geometric tolerance modeling inspired by parametric space envelope. IEEE Transactions on Automation Science and Engineering,15(3), 1386–1398. https://doi.org/10.1109/TASE.2018.2793920
Minaee, S., Boykov, Y. Y., Porikli, F., Plaza, A. J., Kehtarnavaz, N., & Terzopoulos, D. (2021). Image segmentation using deep learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence,44(7), 3523–3542. https://doi.org/10.1109/TPAMI.2021.3059968
Ni, G., Du, X., Liu, L., Zhang, J., Liu, J., & Liu, Y. (2016). Automated optical inspection of liquid crystal display anisotropic conductive film bonding. Optical Engineering,55(10), 103109.
Ni, G., Liu, L., Du, X., Zhang, J., Liu, J., & Liu, Y. (2017). Accurate AOI inspection of resistance in LCD anisotropic conductive film bonding using differential interference contrast. Optik,130, 786–796.
Nti, I. K., Adekoya, A. F., Weyori, B. A., & Nyarko-Boateng, O. (2022). Applications of artificial intelligence in engineering and manufacturing: A systematic review. Journal of Intelligent Manufacturing,33(6), 1581–1601. https://doi.org/10.1007/s10845-021-01771-6
Ren, S., He, K., Girshick, R., & Sun, J. (2017). Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence,39(6), 1137–1149. https://doi.org/10.1109/TPAMI.2016.2577031
Sederberg, T. W. (2012). Computer aided geometric design: CAGD course notes (p. 84602). Brigham Young University.
Sederberg, T. W., & Parry, S. R. (1986). Free-form deformation of solid geometric models. ACM SIGGRAPH Computer Graphics,20(4), 151–160. https://doi.org/10.1145/15886.15903
Sheng, X., Jia, L., Xiong, Z., Wang, Z., & Ding, H. (2013). ACF-COG interconnection conductivity inspection system using conductive area. Microelectronics Reliability,53(4), 622–628.
Shorten, C., & Khoshgoftaar, T. M. (2019). A survey on image data augmentation for deep learning. Journal of Big Data,6(1), 1–48.
Sun, C., Shrivastava, A., Singh, S., & Gupta, A. (2017). Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of the IEEE international conference on computer vision (pp. 843-852). https://doi.org/10.1109/ICCV.2017.97
Tao, X., Ma, W., Lu, Z., & Hou, Z. (2021). Conductive particle detection for chip on glass using convolutional neural network. IEEE Transactions on Instrumentation and Measurement,70, 1–10. https://doi.org/10.1109/TIM.2021.3086908
Xia, Y., Luo, C., Zhou, Y., & Jia, L. (2022). A hybrid method of frequency and spatial domain techniques for TFT-LCD circuits defect detection. IEEE Transactions on Semiconductor Manufacturing,36(1), 45–45. https://doi.org/10.1109/TSM.2022.3216289
Yu-ye, C., Ke, X., Zhen-xiong, G., Jun-jie, H., Chang, L., & Song-yan, C. (2017). Detection of conducting particles bonding in the circuit of liquid crystal display. Chinese Journal of Liquid Crystals and Displays,32(7), 553–559. https://doi.org/10.3788/YJYXS20173207.0553
Yue, X., Mu, D., Wang, C., Ren, H., & Ghadimi, P. (2023). Topological structure and COVID-19 related risk propagation in TFT-LCD supply networks. International Journal of Production Research,61(8), 2758–2778. https://doi.org/10.1080/00207543.2022.2027039
Acknowledgements
This study was conducted in collaboration with Shangshi Finevision Co., Ltd who provided equipment for the study and experiment. The company also assisted with the raw data acquisition. This support is gratefully acknowledged.
Funding
This work was supported by the National Natural Science Foundation of China under Grant Nos. 51975119 and 52375487.
Author information
Authors and Affiliations
Contributions
CL contributed to the study conception and design, methodology and projector administration. TF contributed to data collection, data analysis, the deep learning method implementation, and related software development; YX contributed to the experiment. All authors contributed to discussions of the paper.
Corresponding author
Ethics declarations
Competing interests
No competing interest to declare.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Luo, C., Fan, T., Xia, Y. et al. Deep learning-based conductive particle inspection for TFT-LCDs inspired by parametric space envelope. J Intell Manuf 36, 209–219 (2025). https://doi.org/10.1007/s10845-023-02241-x
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10845-023-02241-x