Skip to main content
Springer Nature Link
Log in

Methods for automatic and assisted image annotation

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Personal memories composed of digital pictures are very popular at the moment. To retrieve these media items annotation is required. During the last years, several approaches have been proposed in order to overcome the image annotation problem. This paper presents our proposals to address this problem. Automatic and semi-automatic learning methods for semantic concepts are presented. The automatic method is based on semantic concepts estimated using visual content, context metadata and audio information. The semi-automatic method is based on results provided by a computer game. The paper describes our proposals and presents their evaluations.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Baeza-Yates R, Ribeiro-Neto B (1999) Modern information retrieval. Addison Wesley

  2. Barnard K, Forsyth D (2001) Learning the semantics of words and pictures. In: International conference on computer vision, pp 408–415

  3. Belkhatir M (2008) An agent framework based on signal concepts for highlighting the image semantic content. In: DEXA ’08: proceedings of the 19th international conference on database and expert systems applications, pp 465–478

  4. Chang E (2005) EXTENT: fusing context, content, and semantic ontology for photo annotation. In: CVDB ’05: proceedings of the 2nd international workshop on computer vision meets databases, pp 5–11

  5. Choi JY, Seungji R, Yong N, Plataniotis K (2008) Face annotation for personal photos using context-assisted face recognition. In: MIR ’08: proceeding of the 1st ACM international conference on multimedia information retrieval, pp 44–51

  6. Datta R, Joshi D, Li J, Wang J (2008) Image retrieval: ideas, influences, and trends of the new age. ACM Comput Surv 40(2). doi:10.1145/1348246.1348248

    Google Scholar 

  7. Duygulu P, Barnard K, Freitas J, Forsyth D (2002) Object recognition as machine translation: learning a lexicon for a fixed image vocabulary. In: Proceedings of the 7th European conference on computer vision. LNCS, vol 2353. Springer, pp 97–112

  8. Fan J, Gao Y, Luo H (2007) Hierarchical classification for automatic image annotation.In: Proceedings of the 30th annual international ACM SIGIR conference on research and development in information retrieval, pp 111–118

  9. Frohlich D, Kuchinsky A, Pering C, Don A, Ariss S (2002) Requirements for photoware. In: Proceedings of the ACM conference on computer supported cooperative work, pp 166–175

  10. Gonçalves D, Jesus R, Correia N (2008) A gesture based game for image tagging. In: CHI ’08: CHI ’08 extended abstracts on human factors in computing systems. ACM Press, New York, pp 2685–2690

    Google Scholar 

  11. Gonçalves D, Jesus R, Grangeiro F, Romão T, Correia N (2008) Tag around: a 3D gesture game for image annotation. In: ACM SIGCHI international conference on advances in computer entertainment technology (ACE 2008)

  12. Jesus R, Abrantes A, Correia N (2006) Photo retrieval from personal memories using generic concepts. In: Advances in Multimedia Information Processing—PCM 2006. LNCS, vol 4261. Springer, pp 633–640

  13. Jesus R, Dias R, Frias R, Abrantes A, Correia N (2007) Sharing personal experiences while navigating in physical spaces. In: 5th workshop on multimedia information retrieval. ACM SIGIR conference on research and development in information retrieval (SIGIR07)

  14. Jesus R, Dias R, Frias R, Abrantes A, Correia N (2008) Memoria mobile: sharing pictures of a point of interest. In: Proceedings of the working conference on advanced visual interfaces (AVI ’08). ACM, New York

  15. Jesus R, Gonçalves D, Abrantes A, Correia N (2008) Playing games as a way to improve automatic image annotation. In: Proceedings of IEEE international workshop on semantic learning applications in multimedia (SLAM08). In conjuntion with CVPR08

  16. Jiebo L, Boutell M, Brown C (2006) Pictures are not taken in a vacuum—an overview of exploiting context for semantic scene content understanding. IEEE Signal Process Mag 22:101–114

    Article  Google Scholar 

  17. Jing F, Li M, Zhang H, Zhang B (2005) A unified framework for image retrieval using keyword and visual features. IEEE Trans Image Process 14(7):979–989

    Article  Google Scholar 

  18. Kustanowitz J, Shneiderman B. Motivating annotation for personal digital photo libraries: lowering barriers while raising incentives. Technical report, HCIL, Univ. of Maryland

  19. Lavrenko V, Manmatha R, Jeon J (2003) A model for learning the semantics of pictures. In: Neural information processing system conference

  20. Lew M, Sebe N, Djeraba C, Jain R (2006) Content-based multimedia information retrieval: state-of-the-art and challenges. In: ACM transactions on multimedia computing, communication, and applications, pp 1–19

  21. Li J, Wang J (2006) Real-time computerized annotation of pictures. In: ACM international conference on multimedia, pp 911–920

  22. Lu Y, Zhang H, Wenyin L, Hu C (2003) Joint semantics and feature based image retrieval using relevance feedback. IEEE Trans Multimedia 5(3):339–347

    Article  Google Scholar 

  23. Monay F, Gatica-Perez D (2003) On image auto-annotation with latent space models. In: Proceedings of the eleventh ACM international conference on multimedia, pp 275–278

  24. Mori Y, Takahashi H, Oka R (1999) Image-to-word transformation based on dividing and vector quantizing images with words. In: Proceedings of the international workshop on multimedia intelligent storage and retrieval management

  25. Naphade M, Huang T (2001) A probabilistic framework for semantic video indexing, filtering, and retrieval. IEEE Trans Multimedia 3:141–151

    Article  Google Scholar 

  26. Naphade M, Smith J, Tesic J, Chang S, Hsu W, Kennedy L, Hauptmann A, Curtis J (2006) Large-scale concept ontology for multimedia. IEEE Multimed 13(3):86–91

    Article  Google Scholar 

  27. Over P, Ianeva T, Kraaij W, Smeaton A (2006) Trecvid 2006 overview. NIST TRECVID-2006

  28. Platt J (1999) Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In: Advances in large margin classifiers. MIT Press, pp 61–74

  29. Poggio T, Smale S (2003) The mathematics of learning: dealing with data. In: Notice of American Mathematical Society, pp 537–544

  30. Rodden K, Wood K (2003) How do people manage their digital photographs? In: Conference on human factors in computing systems (CHI 2003), pp 409–416

  31. Shneiderman B, Kang H (2000) Direct annotation: a drag-and-drop strategy for labeling photos. In: Proceedings international conference information visualization (IV2000), pp 88–95

  32. Sinnott R (1984) Virtues of the Haversine. Sky Telesc 68:158

    Google Scholar 

  33. Smeulders A, Worring M, Santini S, Gupta A, Jain R (2000) Content-based image retrieval at the end of the early years. IEEE Trans Pattern Anal Mach Intell 22(12):1349–1380

    Article  Google Scholar 

  34. Tulving E (2002) EPISODIC MEMORY: from mind to brain. Annu Rev Psychol 53:1–25

    Article  Google Scholar 

  35. von Ahn L, Dabbish L (2004) Labeling images with a computer game. In: Proceedings of the SIGCHI conference on human factors in computing systems CHI ’04, pp 319–326

  36. Wenyin L, Dumais S, Sun Y, Zhang H, Czerwinski M, Field B (2001) Semi-automatic image annotation. In: Human–computer interaction—interact ’01

  37. Yan R, Natsev A, Campbell M (2007) An efficient manual image annotation approach based on tagging and browsing. In: ACM international workshop on the many faces of multimedia semantics, co-located with ACM multimedia, pp 13–20

  38. http://di205.di.fct.unl.pt/instory-web. Last accessed: 15 Jan 2009

Download references

Author information

Authors and Affiliations

  1. Multimedia and Machine Learning Group, Instituto Superior de Engenharia de Lisboa, Rua Conselheiro Emidio Navarro n 1, Lisboa, Portugal

    Rui Jesus & Arnaldo J. Abrantes

  2. CITI, Departamento de Informatica, Faculdade de Ciencias e Tecnologia, FCT, Universidade Nova de Lisboa, 2829-516, Caparica, Portugal

    Nuno Correia

Authors
  1. Rui Jesus
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Arnaldo J. Abrantes
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Nuno Correia
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Rui Jesus.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Jesus, R., Abrantes, A.J. & Correia, N. Methods for automatic and assisted image annotation. Multimed Tools Appl 55, 7–26 (2011). https://doi.org/10.1007/s11042-010-0586-z

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-010-0586-z

Keywords