Skip to main content
Springer Nature Link
Log in

Overview of the ImageCLEF 2021: Multimedia Retrieval in Medical, Nature, Internet and Social Media Applications

  • Conference paper
  • First Online:
Experimental IR Meets Multilinguality, Multimodality, and Interaction (CLEF 2021)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 12880))

Abstract

This paper presents an overview of the ImageCLEF 2021 lab that was organized as part of the Conference and Labs of the Evaluation Forum – CLEF Labs 2021. ImageCLEF is an ongoing evaluation initiative (first run in 2003) that promotes the evaluation of technologies for annotation, indexing and retrieval of visual data with the aim of providing information access to large collections of images in various usage scenarios and domains. In 2021, the 19th edition of ImageCLEF runs four main tasks: (i) a medical task that groups three previous tasks, i.e., caption analysis, tuberculosis prediction, and medical visual question answering and question generation, (ii) a nature coral task about segmenting and labeling collections of coral reef images, (iii) an Internet task addressing the problems of identifying hand-drawn and digital user interface components, and (iv) a new social media aware task on estimating potential real-life effects of online image sharing. Despite the current pandemic situation, the benchmark campaign received a strong participation with over 38 groups submitting more than 250 runs.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    http://www.imageclef.org/.

  2. 2.

    https://www.aicrowd.com/.

  3. 3.

    https://scholar.google.com/.

  4. 4.

    https://www.imageclef.org/2021/.

  5. 5.

    https://www.imageclef.org/2021/.

  6. 6.

    https://www.aicrowd.com/.

  7. 7.

    https://www.imageclef.org/2021/medical/vqa.

  8. 8.

    https://medpix.nlm.nih.gov/.

  9. 9.

    https://visualqa.org/evaluation.html.

  10. 10.

    https://github.com/abachaa/VQA-Med-2021/tree/main/EvaluationCode.

  11. 11.

    https://github.com/microsoft/VoTT.

  12. 12.

    https://github.com/cocodataset/cocoapi.

  13. 13.

    http://host.robots.ox.ac.uk/pascal/VOC/.

  14. 14.

    https://www.ncei.noaa.gov/.

  15. 15.

    https://coralnet.ucsd.edu/.

References

  1. Beltramelli, T.: pix2code: generating code from a graphical user interface screenshot. In: Proceedings of the ACM SIGCHI Symposium on Engineering Interactive Computing Systems, pp. 1–9 (2018)

    Google Scholar 

  2. Ben Abacha, A., Datla, V.V., Hasan, S.A., Demner-Fushman, D., Müller, H.: Overview of the VQA-med task at ImageCLEF 2020: visual question answering and generation in the medical domain. In: CLEF 2020 Working Notes. CEUR Workshop Proceedings, CEUR-WS.org, Thessaloniki, Greece, 22–25 September 2020 (2020)

    Google Scholar 

  3. Ben Abacha, A., Hasan, S.A., Datla, V.V., Liu, J., Demner-Fushman, D., Müller, H.: VQA-Med: overview of the medical visual question answering task at ImageCLEF 2019. In: CLEF2019 Working Notes. CEUR Workshop Proceedings, CEUR-WS.org, Lugano, Switzerland, 09–12 September 2019 (2019). http://ceur-ws.org

  4. Ben Abacha, A., Sarrouti, M., Demner-Fushman, D., Hasan, S.A., Müller, H.: Overview of the VQA-med task at ImageCLEF 2021: visual question answering and generation in the medical domain. In: CLEF 2021 Working Notes. CEUR Workshop Proceedings, CEUR-WS.org, Bucharest, Romania, 21–24 September 2021 (2021)

    Google Scholar 

  5. Berari, R., et al.: Overview of ImageCLEFdrawnUI 2021: the detection and recognition of hand drawn and digital website UIs task. In: CLEF2021 Working Notes. CEUR Workshop Proceedings, CEUR-WS.org, Bucharest, Romania, 21–24 September 2021 (2021). http://ceur-ws.org

  6. Bodenreider, O.: The Unified Medical Language System (UMLS): integrating biomedical terminology. Nucleic Acids Res. 32(Database-Issue), 267–270 (2004). https://doi.org/10.1093/nar/gkh061

    Article  Google Scholar 

  7. Chamberlain, J., Campello, A., Wright, J.P., Clift, L.G., Clark, A., García Seco de Herrera, A.: Overview of ImageCLEFcoral 2019 task. In: CLEF2019 Working Notes. CEUR Workshop Proceedings. CEUR-WS.org (2019)

    Google Scholar 

  8. Chamberlain, J., Campello, A., Wright, J.P., Clift, L.G., Clark, A., García Seco de Herrera, A.: Overview of the ImageCLEFcoral 2020 task: automated coral reef image annotation. In: CLEF2020 Working Notes. CEUR Workshop Proceedings. CEUR-WS.org (2020)

    Google Scholar 

  9. Chamberlain, J., García Seco de Herrera, A., Campello, A., Clark, A., Oliver, T.A., Moustahfid, H.: Overview of the ImageCLEFcoral 2021 task: coral reef image annotation of a 3D environment (2021)

    Google Scholar 

  10. Chen, C., Su, T., Meng, G., Xing, Z., Liu, Y.: From UI design image to GUI skeleton: a neural machine translator to bootstrap mobile GUI implementation. In: International Conference on Software Engineering, vol. 6 (2018)

    Google Scholar 

  11. Clough, P., Müller, H., Sanderson, M.: The CLEF 2004 cross-language image retrieval track. In: Peters, C., Clough, P., Gonzalo, J., Jones, G.J.F., Kluck, M., Magnini, B. (eds.) CLEF 2004. LNCS, vol. 3491, pp. 597–613. Springer, Heidelberg (2005). https://doi.org/10.1007/11519645_59

    Chapter  Google Scholar 

  12. Clough, P., Sanderson, M.: The CLEF 2003 cross language image retrieval task. In: Proceedings of the Cross Language Evaluation Forum (CLEF 2003) (2004)

    Google Scholar 

  13. Deka, B., et al.: Rico: a mobile app dataset for building data-driven design applications. In: Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, UIST 2017, pp. 845–854 (2017). https://doi.org/10.1145/3126594.3126651

  14. Dicente Cid, Y., Jimenez-del-Toro, O., Depeursinge, A., Müller, H.: Efficient and fully automatic segmentation of the lungs in CT volumes. In: Goksel, O., Jimenez-del-Toro, O., Foncubierta-Rodriguez, A., Müller, H. (eds.) Proceedings of the VISCERAL Challenge at ISBI. CEUR Workshop Proceedings, vol. 1390, pp. 31–35, April 2015

    Google Scholar 

  15. Dicente Cid, Y., Kalinovsky, A., Liauchuk, V., Kovalev, V., Müller, H.: Overview of ImageCLEFtuberculosis 2017 - predicting tuberculosis type and drug resistances. In: CLEF2017 Working Notes. CEUR Workshop Proceedings, CEUR-WS.org, Dublin, Ireland, 11–14 September 2017 (2017). http://ceur-ws.org

  16. Dicente Cid, Y., Liauchuk, V., Klimuk, D., Tarasau, A., Kovalev, V., Müller, H.: Overview of ImageCLEFtuberculosis 2019 - automatic CT-based report generation and tuberculosis severity assessment. In: CLEF2019 Working Notes. CEUR Workshop Proceedings, CEUR-WS.org, Lugano, Switzerland, 9–12 September 2019 (2019). http://ceur-ws.org

  17. Dicente Cid, Y., Liauchuk, V., Kovalev, V., Müller, H.: Overview of ImageCLEFtuberculosis 2018 - detecting multi-drug resistance, classifying tuberculosis type, and assessing severity score. In: CLEF2018 Working Notes. CEUR Workshop Proceedings, CEUR-WS.org, Avignon, France, 10–14 September 2018 (2018). http://ceur-ws.org

  18. Dosovitskiy, A., et al.: An image is worth \(16 \times 16\) words: transformers for image recognition at scale. In: Proceedings of the 9th International Conference on Learning Representations (ICLR 2021): 03–07 May 2021, Online Event (2021). https://openreview.net/forum?id=YicbFdNTTy

  19. Everingham, M., Gool, L.V., Williams, C.K.I., Winn, J., Zisserman, A.: The PASCAL Visual Object Classes (VOC) challenge. Int. J. Comput. Vis. 88, 303–338 (2010). https://doi.org/10.1007/s11263-009-0275-4

    Article  Google Scholar 

  20. Fichou, D., et al.: Overview of ImageCLEFdrawnUI 2020: the detection and recognition of hand drawn website UIs task. In: CLEF2020 Working Notes. CEUR Workshop Proceedings, CEUR-WS.org, Thessaloniki, Greece, 22–25 September 2020 (2020). http://ceur-ws.org

  21. Hasan, S.A., Ling, Y., Farri, O., Liu, J., Lungren, M., Müller, H.: Overview of the ImageCLEF 2018 medical domain visual question answering task. In: CLEF2018 Working Notes. CEUR Workshop Proceedings, CEUR-WS.org, Avignon, France, 10–14 September 2018 (2018). http://ceur-ws.org

  22. García Seco de Herrera, A., Eickhoff, C., Andrearczyk, V., Müller, H.: Overview of the ImageCLEF 2018 caption prediction tasks. In: CLEF2018 Working Notes. CEUR Workshop Proceedings, CEUR-WS.org, Avignon, France, 10–14 September 2018 (2018). http://ceur-ws.org

  23. García Seco de Herrera, A., Schaer, R., Bromuri, S., Müller, H.: Overview of the ImageCLEF 2016 medical task. In: Working Notes of CLEF 2016 (Cross Language Evaluation Forum), September 2016

    Google Scholar 

  24. Ionescu, B., et al.: ImageCLEF 2019: multimedia retrieval in lifelogging, medical, nature, and security applications. In: Azzopardi, L., Stein, B., Fuhr, N., Mayr, P., Hauff, C., Hiemstra, D. (eds.) ECIR 2019. LNCS, vol. 11438, pp. 301–308. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-15719-7_40

    Chapter  Google Scholar 

  25. Kalpathy-Cramer, J., et al.: Evaluating performance of biomedical image retrieval systems: Overview of the medical image retrieval task at ImageCLEF 2004–2014. Comput. Med. Imaging Graph. 39, 55–61 (2015)

    Article  Google Scholar 

  26. Kozlovski, S., Liauchuk, V., Dicente Cid, Y., Kovalev, V., Müller, H.: Overview of ImageCLEFtuberculosis 2021 - CT-based tuberculosis type classification. In: CLEF2021 Working Notes. CEUR Workshop Proceedings, CEUR-WS.org, Bucharest, Romania, 21–24 September 2021 (2021). http://ceur-ws.org

  27. Kozlovski, S., Liauchuk, V., Dicente Cid, Y., Tarasau, A., Kovalev, V., Müller, H.: Overview of ImageCLEFtuberculosis 2020 - automatic CT-based report generation. In: CLEF2020 Working Notes. CEUR Workshop Proceedings, CEUR-WS.org, Thessaloniki, Greece, 22–25 September 2020 (2020). http://ceur-ws.org

  28. Kuznetsova, A., et al.: The open images dataset V4: unified image classification, object detection, and visual relationship detection at scale. CoRR abs/1811.00982 (2018). arxiv.org/abs/1811.00982

  29. Liauchuk, V., Kovalev, V.: ImageCLEF 2017: supervoxels and co-occurrence for tuberculosis CT image classification. In: CLEF2017 Working Notes. CEUR Workshop Proceedings, CEUR-WS.org, Dublin, Ireland, 11–14 September 2017 (2017). http://ceur-ws.org

  30. Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48

    Chapter  Google Scholar 

  31. Müller, H., Clough, P., Deselaers, T., Caputo, B. (eds.): ImageCLEF - Experimental Evaluation in Visual Information Retrieval, The Springer International Series On Information Retrieval, vol. 32. Springer, Heidelberg (2010)

    MATH  Google Scholar 

  32. Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: BLEU: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, pp. 311–318. Association for Computational Linguistics (2002)

    Google Scholar 

  33. Pelka, O., Ben Abacha, A., García Seco de Herrera, A., Jacutprakart, J., Friedrich, C.M., Müller, H.: Overview of the ImageCLEFmed 2021 concept & caption prediction task. In: CLEF2021 Working Notes. CEUR Workshop Proceedings, CEUR-WS.org, Bucharest, Romania, 21–24 September 2021 (2021)

    Google Scholar 

  34. Pelka, O., Friedrich, C.M., García Seco de Herrera, A., Müller, H.: Overview of the ImageCLEFmed 2019 concept prediction task. In: CLEF2019 Working Notes. CEUR Workshop Proceedings, CEUR-WS.org, Lugano, Switzerland, 09–12 September 2019 (2019). http://ceur-ws.org

  35. Pelka, O., Friedrich, C.M., García Seco de Herrera, A., Müller, H.: Overview of the ImageCLEFmed 2020 concept prediction task: medical image understanding. In: CLEF2020 Working Notes. CEUR Workshop Proceedings, CEUR-WS.org, Thessaloniki, Greece, 22–25 September 2020 (2020)

    Google Scholar 

  36. Pelka, O., Koitka, S., Rückert, J., Nensa, F., Friedrich, C.M.: Radiology Objects in COntext (ROCO): a multimodal image dataset. In: Stoyanov, D., et al. (eds.) LABELS/CVII/STENT -2018. LNCS, vol. 11043, pp. 180–189. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01364-6_20

    Chapter  Google Scholar 

  37. Popescu, A., Deshayes-Chossart, J., Ionescu, B.: Overview of ImageCLEFaware 2021: estimating potential real-life effects of online image sharing task. In: CLEF2021 Working Notes. CEUR Workshop Proceedings, CEUR-WS.org, Bucharest, Romania, 21–24 September 2021 (2021). http://ceur-ws.org

  38. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I.: Language models are unsupervised multitask learners. Technical report, Open-AI (2019)

    Google Scholar 

  39. Russakovsky, O., et al.: ImageNet large scale visual recognition challenge. Int. J. Comput. Vis. 115(3), 211–252 (2015)

    Article  MathSciNet  Google Scholar 

  40. Sarrouti, M., Ben Abacha, A., Demner-Fushman, D.: Visual question generation from radiology images. In: Proceedings of the First Workshop on Advances in Language and Vision Research, pp. 12–18. Association for Computational Linguistics, Online, July 2020. https://www.aclweb.org/anthology/2020.alvr-1.3

  41. Tan, M., Le, Q.: EfficientNet: rethinking model scaling for convolutional neural networks. In: Chaudhuri, K., Salakhutdinov, R. (eds.) Proceedings of the 36th International Conference on Machine Learning (ICML 2019), Long Beach, California, USA, 10–15 June 2019, vol. 97, pp. 6105–6114, June 2019. http://proceedings.mlr.press/v97/tan19a.html

  42. Tan, M., Pang, R., Le, Q.V.: EfficientDet: scalable and efficient object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10781–10790 (2020)

    Google Scholar 

  43. Thomee, B., et al.: YFCC100M: the new data in multimedia research. Commun. ACM 59(2), 64–73 (2016)

    Article  Google Scholar 

  44. Tsikrika, T., de Herrera, A.G.S., Müller, H.: Assessing the scholarly impact of ImageCLEF. In: Forner, P., Gonzalo, J., Kekäläinen, J., Lalmas, M., de Rijke, M. (eds.) CLEF 2011. LNCS, vol. 6941, pp. 95–106. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-23708-9_12

    Chapter  Google Scholar 

  45. Tsikrika, T., Larsen, B., Müller, H., Endrullis, S., Rahm, E.: The scholarly impact of CLEF (2000–2009). In: Forner, P., Müller, H., Paredes, R., Rosso, P., Stein, B. (eds.) CLEF 2013. LNCS, vol. 8138, pp. 1–12. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-40802-1_1

    Chapter  Google Scholar 

  46. Wilkins, K.W., Rosa-Marín, A., Cziesielski, M., Hughes, H., Love, C., Nowakowski, C.: Short and long-term visions for protecting coral reefs. Limnol. Oceanogr. Bull. 30 (2021)

    Google Scholar 

  47. World Health Organization, et al.: Global tuberculosis report 2019 (2019)

    Google Scholar 

Download references

Acknowledgements

Data collection for the Tuberculosis task was supported by the National Institute of Allergy and Infectious Diseases, National Institutes of Health, US Department of Health and Human Services, CRDF project DAA9-19-65987-1.

The aware task was fully supported and the drawnUI was partially supported under project AI4Media, A European Excellence Centre for Media, Society and Democracy, H2020 ICT-48-2020, grant \(\#\)951911.

Author information

Authors and Affiliations

  1. University Politehnica of Bucharest, Bucharest, Romania

    Bogdan Ionescu, Mihai Dogariu, Liviu Daniel Ştefan & Mihai Gabriel Constantin

  2. University of Applied Sciences Western Switzerland (HES-SO), Delémont, Switzerland

    Henning Müller

  3. University of La Rochelle, La Rochelle, France

    Renaud Péteri

  4. National Library of Medicine, Bethesda, USA

    Asma Ben Abacha, Mourad Sarrouti & Dina Demner-Fushman

  5. CVS Health, Wellesley, MA, USA

    Sadid A. Hasan

  6. United Institute of Informatics Problems, Minsk, Belarus

    Serge Kozlovski, Vitali Liauchuk & Vassili Kovalev

  7. University of Warwick, Coventry, UK

    Yashin Dicente Cid

  8. University of Applied Sciences and Arts Dortmund, Dortmund, Germany

    Obioma Pelka & Christoph M. Friedrich

  9. University of Essex, Colchester, UK

    Alba García Seco de Herrera, Janadhip Jacutprakart, Jon Chamberlain & Adrian Clark

  10. teleportHQ, Cluj-Napoca, Romania

    Raul Berari, Andrei Tauteanu, Dimitri Fichou & Paul Brie

  11. Wellcome Trust, London, UK

    Antonio Campello

  12. Pacific Islands Fisheries Science Center, Silver Spring, USA

    Thomas A. Oliver & Hassan Moustahfid

  13. Université Paris-Saclay, CEA, List, Palaiseau, France

    Adrian Popescu & Jérôme Deshayes-Chossart

Authors
  1. Bogdan Ionescu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Henning Müller
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Renaud Péteri
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Asma Ben Abacha
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Mourad Sarrouti
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Dina Demner-Fushman
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Sadid A. Hasan
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Serge Kozlovski
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Vitali Liauchuk
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Yashin Dicente Cid
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Vassili Kovalev
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Obioma Pelka
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Alba García Seco de Herrera
    View author publications

    You can also search for this author in PubMed Google Scholar

  14. Janadhip Jacutprakart
    View author publications

    You can also search for this author in PubMed Google Scholar

  15. Christoph M. Friedrich
    View author publications

    You can also search for this author in PubMed Google Scholar

  16. Raul Berari
    View author publications

    You can also search for this author in PubMed Google Scholar

  17. Andrei Tauteanu
    View author publications

    You can also search for this author in PubMed Google Scholar

  18. Dimitri Fichou
    View author publications

    You can also search for this author in PubMed Google Scholar

  19. Paul Brie
    View author publications

    You can also search for this author in PubMed Google Scholar

  20. Mihai Dogariu
    View author publications

    You can also search for this author in PubMed Google Scholar

  21. Liviu Daniel Ştefan
    View author publications

    You can also search for this author in PubMed Google Scholar

  22. Mihai Gabriel Constantin
    View author publications

    You can also search for this author in PubMed Google Scholar

  23. Jon Chamberlain
    View author publications

    You can also search for this author in PubMed Google Scholar

  24. Antonio Campello
    View author publications

    You can also search for this author in PubMed Google Scholar

  25. Adrian Clark
    View author publications

    You can also search for this author in PubMed Google Scholar

  26. Thomas A. Oliver
    View author publications

    You can also search for this author in PubMed Google Scholar

  27. Hassan Moustahfid
    View author publications

    You can also search for this author in PubMed Google Scholar

  28. Adrian Popescu
    View author publications

    You can also search for this author in PubMed Google Scholar

  29. Jérôme Deshayes-Chossart
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bogdan Ionescu .

Editor information

Editors and Affiliations

  1. Arizona State University, Tempe, AZ, USA

    K. Selçuk Candan

  2. Politehnica University of Bucharest, Bucharest, Romania

    Bogdan Ionescu

  3. Université Grenoble Alpes, Saint-Martin-d’Hères, France

    Lorraine Goeuriot

  4. Aalborg University Copenhagen, Copenhagen, Denmark

    Birger Larsen

  5. HES-SO Valais-Wallis, Sierre, Switzerland

    Henning Müller

  6. University of Montpellier, Montpellier, France

    Alexis Joly

  7. University of Copenhagen, Copenhagen, Denmark

    Maria Maistro

  8. TU Wien, Vienna, Austria

    Florina Piroi

  9. University of Padua, Padova, Italy

    Guglielmo Faggioli

  10. University of Padua, Padova, Italy

    Nicola Ferro

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ionescu, B. et al. (2021). Overview of the ImageCLEF 2021: Multimedia Retrieval in Medical, Nature, Internet and Social Media Applications. In: Candan, K.S., et al. Experimental IR Meets Multilinguality, Multimodality, and Interaction. CLEF 2021. Lecture Notes in Computer Science(), vol 12880. Springer, Cham. https://doi.org/10.1007/978-3-030-85251-1_23

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-85251-1_23

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-85250-4

  • Online ISBN: 978-3-030-85251-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics