Abstract
Deep neural networks suffer from catastrophic forgetting when continually learning new tasks. Although simply replaying all previous data alleviates the problem, it requires large memory and even worse, often infeasible in real-world applications where access to past data is limited. Therefore, We propose a two-stage framework that dynamically reproduces data features of previous tasks to reduce catastrophic forgetting. Specifically, at each task step, we use a new memory module to learn the data distribution of the new task and reproduce pseudo-data from previous memory modules to learn together. This enables us to integrate new visual concepts with retaining learned knowledge to achieve a better stability-malleability balance. We introduce an N-step model fusion strategy to accelerate the memorization process of the memory module and a screening strategy to control the quantity and quality of generated data, reducing distribution differences. We experimented on CIFAR-100, MNIST, and SVHN datasets to demonstrate the effectiveness of our method.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Grossberg, S.: Adaptive resonance theory: how a brain learns to consciously attend, learn, and recognize a changing world. Neural Netw. 37, 1–47 (2013). https://doi.org/10.1016/j.neunet.2012.09.017
Douillard, A., Ramé, A., Couairon, G., Cord, M.: Dytox: transformers for continual learning with dynamic token expansion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9285–9295 (2022)
Yan, S., Xie, J., He, X.: Der: dynamically expandable representation for class incremental learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3014–3023 (2021)
Yang, L., et al.: Diffusion models: a comprehensive survey of methods and applications. arXiv preprint arXiv:2209.00796. https://doi.org/10.48550/arXiv.2209.00796 (2022)
Shin, H., Lee, J.K., Kim, J., Kim, J.:Continual learning with deep generative replay. In: Advances in Neural Information Processing Systems, vol. 30 (2017)
Zhou, D.W., Wang, F.Y., Ye, H.J., Zhan, D.C.: Pycil: a Python toolbox for class-incremental learning. arXiv preprint arXiv:2112.12533. https://doi.org/10.1007/s11432-022-3600-y (2021)
Rebuffi, S.A., Kolesnikov, A., Sperl, G., Lampert, C.H.: ICARL: incremental classifier and representation learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2001–2010 (2017)
Hou, S., Pan, X., Loy, C.C., Wang, Z., Lin, D.: Learning a unified classifier incrementally via rebalancing. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 831–839 (2019)
Wu, Y., et al.: Large scale incremental learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 374–382 (2019)
Zhao, B., Xiao, X., Gan, G., Zhang, B., Xia, S.T.:Maintaining discrimination and fairness in class incremental learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13208–13217 (2020)
Douillard, A., Cord, M., Ollion, C., Robert, T., Valle, E.: PODNet: pooled outputs distillation for small-tasks incremental learning. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12365, pp. 86–102. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58565-5_6
Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. Adv. Neural. Inf. Process. Syst. 33, 6840–6851 (2020)
Rajasegaran, J., Hayat, M., Khan, S., Khan, F.S., Shao, L., Yang, M.H.: An adaptive random path selection approach for incremental learning. arXiv preprint arXiv:1906.01120 (2019)
Lesort, T., Caselles-Dupré, H., Garcia-Ortiz, M., Stoian, A., Filliat, D.: Generative Models from the perspective of continual learning. In: 2019 International Joint Conference on Neural Networks (IJCNN), Budapest, Hungary, pp. 1–8 (2019). https://doi.org/10.1109/IJCNN.2019.8851986
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny images (2009)
Lecun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998). https://doi.org/10.1109/5.726791
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, A.Y.: Reading digits in natural images with unsupervised feature learning (2011)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Tao, S., Huang, J., Zhang, X., Sun, X., Gu, Y. (2023). Dynamic Memory-Based Continual Learning with Generating and Screening. In: Iliadis, L., Papaleonidas, A., Angelov, P., Jayne, C. (eds) Artificial Neural Networks and Machine Learning – ICANN 2023. ICANN 2023. Lecture Notes in Computer Science, vol 14256. Springer, Cham. https://doi.org/10.1007/978-3-031-44213-1_31
Download citation
DOI: https://doi.org/10.1007/978-3-031-44213-1_31
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-44212-4
Online ISBN: 978-3-031-44213-1
eBook Packages: Computer ScienceComputer Science (R0)