Skip to main content
Springer Nature Link
Log in

Unsupervised anomaly detection via knowledge distillation with non-directly-coupled student block fusion

  • Original Paper
  • Published:
Machine Vision and Applications Aims and scope Submit manuscript

Abstract

Recently, knowledge distillation has achieved excellent results in unsupervised anomaly detection. The representation difference of anomalies between teacher and student model is an essential basis for unsupervised anomaly detection. To fully exploit the diversity of anomaly representations, a novel distillation network is proposed for unsupervised anomaly detection, consisting of a complete teacher network and a set of non-directly-coupled student blocks. Instead of taking a complete network as a student which sequentially inherits the distilled knowledge from the previous layer, the student blocks are specifically designed, which independently take features of each layer of teacher network as their input and target to recover the multi-scale representation of the teacher. For each block, an adaptive weighted multi-branch feature extraction strategy is presented to enable the blocks to better focus on key messages from the teacher model. In addition, a feature reunion technique is given during distillation to make multi-scale features more robust to noisy input. The experimental results indicate that the proposed method achieves an outstanding performance on MVTec AD dataset. Compared with the baseline method, the proposed method improves by 2.21% at ROC-AUC of image level and improves by 1.00 and 2.22% for both ROC-AUC and PRO-AUC at pixel level.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Baur, C., Wiestler, B., Albarqouni, S., Navab, N.: Deep autoencoding models for unsupervised anomaly segmentation in brain MR images. In: International MICCAI Brainlesion Workshop, pp. 161–169. Springer, Berlin (2018)

  2. Bergmann, P., Fauser, M., Sattlegger, D., Steger, C.: MVTec AD—a comprehensive real-world dataset for unsupervised anomaly detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9592–9600 (2019)

  3. Bergmann, P., Fauser, M., Sattlegger, D., Steger, C.: Uninformed students: student–teacher anomaly detection with discriminative latent embeddings. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4183–4192 (2020)

  4. Chalapathy, R., Menon, A.K., Chawla, S.: Anomaly detection using one-class neural networks. arXiv preprint arXiv:1802.06360 (2018)

  5. Chen, L., You, Z., Zhang, N., Xi, J., Le, X.: UTRAD: Anomaly detection and localization with U-transformer. Neural Netw. 147, 53–62 (2022)

    Article  Google Scholar 

  6. Chow, J.K., Su, Z., Wu, J., Tan, P.S., Mao, X., Wang, Y.H.: Anomaly detection of defects on concrete structures with the convolutional autoencoder. Adv. Eng. Inform. 45, 101–105 (2020)

    Article  Google Scholar 

  7. Cohen, N., Hoshen, Y.: Sub-image anomaly detection with deep pyramid correspondences. arXiv preprint arXiv:2005.02357 (2020)

  8. Dai, J., Wang, J., Zhu, Z., Shen, C., Huang, W.: Anomaly detection of mechanical systems based on generative adversarial network and auto-encoder. Chin. J. Sci. Instrum. 40(9), 16–26 (2019)

    Google Scholar 

  9. Defard, T., Setkov, A., Loesch, A., Audigier, R.: Padim: a patch distribution modeling framework for anomaly detection and localization. In: International Conference on Pattern Recognition, pp. 475–489. Springer, Berlin (2021)

  10. Deng, H., Li, X.: Anomaly detection via reverse distillation from one-class embedding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9737–9746 (2022)

  11. Gong, D., Liu, L., Le, V., Saha, B., Mansour, M.R., Venkatesh, S., van den Hengel, A.: Memorizing normality to detect anomaly: memory-augmented deep autoencoder for unsupervised anomaly detection. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 1705–1714 (2019)

  12. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial networks. Commun. ACM 63(11), 139–144 (2020)

    Article  MathSciNet  Google Scholar 

  13. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

  14. Huang, C., Xu, Q., Wang, Y., Wang, Y., Zhang, Y.: Self-supervised masking for unsupervised anomaly detection and localization. IEEE Trans. Multimed. 1(7), 82–92 (2022)

    Google Scholar 

  15. Kawamura, N.: Unsupervised anomaly localization using locally adaptive query-dependent scores. In: ICIAP, pp. 716–737 (2022)

  16. Kingma, D.P., Welling, M.: Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013)

  17. LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)

    Article  Google Scholar 

  18. Li, C.L., Sohn, K., Yoon, J., Pfister, T.: Cutpaste: self-supervised learning for anomaly detection and localization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9664–9674 (2021)

  19. Liu, W., Li, R., Zheng, M., Karanam, S., Wu, Z., Bhanu, B., Radke, R.J., Camps, O.: Towards visually explaining variational autoencoders. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8642–8651 (2020)

  20. Liu, J., Song, K., Feng, M., Yan, Y., Tu, Z., Zhu, L.: Semi-supervised anomaly detection with dual prototypes autoencoder for industrial surface inspection. Opt. Lasers Eng. 136, 106–124 (2021)

    Article  Google Scholar 

  21. Lv, H., Chen, C., Cui, Z., Xu, C., Li, Y., Yang, J.: Learning normal dynamics in videos with meta prototype network. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15425–15434 (2021)

  22. Masana, M., Ruiz, I., Serrat, J., van de Weijer, J., Lopez, A.M.: Metric learning for novelty and anomaly detection. arXiv preprint arXiv:1808.05492 (2018)

  23. Pirnay, J., Chai, K.: Inpainting transformer for anomaly detection. In: International Conference on Image Analysis and Processing, pp. 394–406. Springer, Berlin (2022)

  24. Rudolph, M., Wandt, B., Rosenhahn, B.: Same same but different: semi-supervised defect detection with normalizing flows. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1907–1916 (2021)

  25. Ruff, L., Vandermeulen, R., Goernitz, N., Deecke, L., Siddiqui, S.A., Binder, A., Müller, E., Kloft, M.: Deep one-class classification. In: International Conference on Machine Learning, pp. 4393–4402. PMLR (2018)

  26. Salehi, M., Sadjadi, N., Baselizadeh, S., Rohban, M.H., Rabiee, H.R.: Multiresolution knowledge distillation for anomaly detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14902–14912 (2021)

  27. Schlegl, T., Seeböck, P., Waldstein, S.M., Schmidt-Erfurth, U., Langs, G.: Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. In: International Conference on Information Processing in Medical Imaging, pp. 146–157. Springer, Berlin (2017)

  28. Schlegl, T., Seeböck, P., Waldstein, S.M., Langs, G., Schmidt-Erfurth, U.: F-AnoGAN: fast unsupervised anomaly detection with generative adversarial networks. Med. Image Anal. 54, 30–44 (2019)

    Article  Google Scholar 

  29. Schlüter, H.M., Tan, J., Hou, B., Kainz, B.: Self-supervised out-of-distribution detection and localization with natural synthetic anomalies (NSA). arXiv preprint arXiv:2109.15222 (2021)

  30. Venkataramanan, S., Peng, K.C., Singh, R.V., Mahalanobis, A.: Attention guided anomaly localization in images. In: European Conference on Computer Vision, pp. 485–503. Springer, Berlin (2020)

  31. Wan, Q., Gao, L., Li, X., Wen, L.: Industrial image anomaly localization based on gaussian clustering of pretrained feature. IEEE Trans. Ind. Electron. 69(6), 6182–6192 (2021)

    Article  Google Scholar 

  32. Wang, G., Han, S., Ding, E., Huang, D.: Student-teacher feature pyramid matching for unsupervised anomaly detection. arXiv preprint arXiv:2103.04257 (2021)

  33. Woo, S., Park, J., Lee, J.Y., Kweon, I.S.: Cbam: Convolutional block attention module. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 3–19 (2018)

  34. Yamada, S., Hotta, K.: Reconstruction student with attention for student–teacher pyramid matching. arXiv preprint arXiv:2111.15376 (2021)

  35. Yang, J., Shi, Y., Qi, Z.: DFR: Deep feature reconstruction for unsupervised anomaly segmentation. arXiv preprint arXiv:2012.07122 (2020)

  36. Yi, J., Yoon, S.: Patch SVDD: Patch-level SVDD for anomaly detection and segmentation. In: Proceedings of the Asian Conference on Computer Vision, pp. 1–19 (2020)

  37. Zaheer, M.Z., Lee, J.H., Astrid, M., Lee, S.I.: Old is gold: redefining the adversarially learned one-class classifier training paradigm. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14183–14193 (2020)

  38. Zavrtanik, V., Kristan, M., Skočaj, D.: Reconstruction by inpainting for visual anomaly detection. Pattern Recogn. 112, 107706 (2021)

Download references

Acknowledgements

This work is supported by the National Natural Science Foundation of China (Grant No. 62173160)

Author information

Authors and Affiliations

  1. Key Laboratory of Advanced Process Control for Light Industry (Ministry of Education), Jiangnan University, Wuxi, 214122, China

    Zhiyuan Feng & Ying Chen

  2. Engineering Research Center of Internet of Things Technology Applications (Ministry of Education), Jiangnan University, Wuxi, 214122, China

    Linbo Xie

Authors
  1. Zhiyuan Feng
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Ying Chen
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Linbo Xie
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Ying Chen.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest. The MVTec AD dataset that supports the findings of this study is publicly available from MVTec Software GmbH at https://www.mvtec.com/company/research/datasets.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Feng, Z., Chen, Y. & Xie, L. Unsupervised anomaly detection via knowledge distillation with non-directly-coupled student block fusion. Machine Vision and Applications 34, 104 (2023). https://doi.org/10.1007/s00138-023-01454-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00138-023-01454-7

Keywords