Skip to main content
Springer Nature Link
Log in

AQUAdexIM: highly efficient in-memory indexing and querying of astronomy time series images

  • Original Article
  • Published:
Experimental Astronomy Aims and scope Submit manuscript

Abstract

Astronomy has always been, and will continue to be, a data-based science, and astronomers nowadays are faced with increasingly massive datasets, one key problem of which is to efficiently retrieve the desired cup of data from the ocean. AQUAdexIM, an innovative spatial indexing and querying method, performs highly efficient on-the-fly queries under users’ request to search for Time Series Images from existing observation data on the server side and only return the desired FITS images to users, so users no longer need to download entire datasets to their local machines, which will only become more and more impractical as the data size keeps increasing. Moreover, AQUAdexIM manages to keep a very low storage space overhead and its specially designed in-memory index structure enables it to search for Time Series Images of a given area of the sky 10 times faster than using Redis, a state-of-the-art in-memory database.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Aji, A, Wang, F, Saltz, JH: Towards building a high performance spatial query system for large scale medical imaging data. In: Proceedings of the 20th International Conference on Advances in Geographic Information Systems, pp. 309–318. ACM (2012)

  2. Aji, A, Wang, F, Vo, H, Lee, R, Liu, Q, Zhang, X, Saltz, J: Hadoop gis: A high performance spatial data warehousing system over mapreduce. Proc. VLDB Endowm. 6(11), 1009– 1020 (2013)

    Article  Google Scholar 

  3. Alagiannis, I, Borovica, R, Branco, M, Idreos, S, Ailamaki, A: Nodb: Efficient query execution on raw data files. In: Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data, pp 241–252. ACM (2012a)

  4. Alagiannis, I, Borovica, R, Branco, M, Idreos, S, Ailamaki, A: Nodb in action: Adaptive query processing on raw data. Proc. VLDB Endowm. 5(12), 1942–1945 (2012b)

  5. Alam, S, Albareti, F D, Prieto, C A, Anders, F, Anderson, S F, Anderton, T, Andrews, B H, Armengaud, E, Aubourg, É, Bailey, S, et al: The eleventh and twelfth data releases of the sloan digital sky survey: Final data from sdss-iii. Astrophys. J. Supp. Series 219(1), 12 (2015)

    Article  ADS  Google Scholar 

  6. Berriman, G B, Groom, S L: How will astronomy archives survive the data tsunami? Commun. ACM 54(12), 52–56 (2011)

    Article  Google Scholar 

  7. Blanas, S, Wu, K, Byna, S, Dong, B, Shoshani, A: Parallel data analysis directly on scientific file formats. In: Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data, pp. 385–396. ACM (2014)

  8. Brown, P G: Overview of scidb: Large scale array storage, processing and analysis. In: Proceedings of the 2010 ACM SIGMOD International Conference on Management of Data, pp. 963–968. ACM (2010)

  9. Capaccioli, M, Schipani, P: The vlt survey telescope opens to the sky: history of a commissioning. Messenger 146, 2–6 (2011)

    ADS  Google Scholar 

  10. Diaconu, C, Freedman, C, Ismert, E, Larson, P A, Mittal, P, Stonecipher, R, Verma, N, Zwilling, M: Hekaton: Sql server’s memory-optimized oltp engine. In: Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data, pp. 1243–1254. ACM (2013)

  11. Gorski, K M, Hivon, E, Banday, A, Wandelt, B D, Hansen, F K, Reinecke, M, Bartelmann, M: Healpix: A framework for high-resolution discretization and fast analysis of data distributed on the sphere. Astrophys. J. 622 (2), 759 (2005)

    Article  ADS  Google Scholar 

  12. Han, J, Haihong, E, Le, G, Du, J: Survey on nosql database. In: 2011 6th International Conference On Pervasive Computing And Applications (ICPCA), pp. 363–366. IEEE (2011)

  13. He, B, Cui, C, Fan, D, Li, C, Xiao, J, Yu, C, Wang, C, Cao, Z, Chen, J, Yi, W, et al: Astrocloud, a cyber-infrastructure for astronomy research: Data archiving and quality control. In: Astronomical Data Analysis Software an Systems XXIV (ADASS XXIV), vol. 495, pp. 483 (2015)

  14. Hong, Z: Source code of the algorithms in this paper. http://paperdata.china-vo.org/Hong.Zhi/2016/ExpAstron/AQUAdexIM.tar.gz, accessed 2016-04-06 (2016)

  15. Hong, Z, Yu, C, Xia, R, Xiao, J, Wang, J, Sun, J, Cui, C: Aquadex: A highly efficient indexing and retrieving method for astronomical big data of time series images. In: Algorithms and Architectures for Parallel Processing, p.p 92–105. Springer (2015)

  16. Ivanova, M, Kersten, M, Manegold, S: Data vaults: a symbiosis between database technology and scientific file repositories. In: Scientific and Statistical Database Management, pp. 485–494. Springer (2012)

  17. Ivezic, Z, Tyson, J, Abel, B, Acosta, E, Allsman, R, AlSayyad, Y, Anderson, S, Andrew, J, Angel, R, Angeli, G, et al: Lsst: from science drivers to reference design and anticipated data products. arXiv preprint arXiv:08052366 (2008)

  18. Mwebaze, J, Boxhoorn, D, McFarland, J, Valentijn, E A: Sub-image data processing in astro-wise. Exper. Astron. 35(1-2), 245–282 (2013)

    Article  ADS  Google Scholar 

  19. Ng, M K, Huang, Z: Data-mining massive time series astronomical data: challenges, problems and solutions. Inf. Softw. Technol. 41(9), 545–556 (1999)

    Article  Google Scholar 

  20. Planthaber, G, Stonebraker, M, Frew, J: Earthdb: Scalable analysis of modis data using scidb. In: Proceedings of the 1st ACM SIGSPATIAL International Workshop on Analytics for Big Geospatial Data, pp. 11–19. ACM (2012)

  21. Richter, S, Quiané-Ruiz, J A, Schuh, S, Dittrich, J: Towards zero-overhead static and adaptive indexing in hadoop. VLDB J. 23(3), 469–494 (2014)

    Article  Google Scholar 

  22. Ronstrom, M, Thalmann, L: Mysql cluster architecture overview. MySQL Technical White Paper (2004)

  23. Silva, V, de Oliveira, D, Mattoso, M: Exploratory analysis of raw data files through dataflows. In: International Symposium on Computer Architecture and High Performance Computing Workshop (SBAC-PADW) 2014, pp. 114–119. IEEE (2014)

  24. Stonebraker, M, Weisberg, A: The voltdb main memory dbms. IEEE Data Eng. Bull. 36(2), 21–27 (2013)

    Google Scholar 

  25. Stonebraker, M, Brown, P, Poliakov, A, Raman, S: The architecture of scidb. In: Scientific and Statistical Database Management, pp. 1–16. Springer (2011)

  26. Tian, Y, Alagiannis, I, Liarou, E, Ailamaki, A, Michiardi, P, Vukolić, M: Dinodb: Efficient large-scale raw data analytics. In: Proceedings of the First International Workshop on Bringing the Value of Big Data to Users (Data4U 2014), p. 1. ACM (2014)

  27. Tody, D, Plante, R, Harrison, P: Ivoa recommendation: Simple image access specification version 1.0. arXiv preprint arXiv:11100499 (2011)

  28. Waas, FM: Beyond conventional data warehousing—massively parallel data processing with greenplum database. In: International Workshop on Business Intelligence for the Real-Time Enterprise, pp. 89–96. Springer (2008)

  29. van der Wel, A, Noeske, K, Bezanson, R, Pacifici, C, Gallazzi, A, Franx, M, Munoz-Mateos, J, Bell, E, Brammer, G, Charlot, S, et al: The vlt lega-c spectroscopic survey: the physics of galaxies at a lookback time of 7 gyr. Astrophys. J. Supp. Series 223(2), 29 (2016)

    Article  ADS  Google Scholar 

  30. Zhao, Q: Research on high-efficient massive data oriented astronomical cross-match. PhD thesis, Tianjin University (2010)

Download references

Acknowledgments

The authors would like to thank the AST3 team from National Astronomical Observatories, Chinese Academy of Sciences for providing the dataset used in the experiments.

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, Tianjin University, Tianjin, 300350, China

    Ce Yu, Jie Wang & Jizhou Sun

  2. School of Computer Software, Tianjin University, Tianjin, 300350, China

    Zhi Hong & Jian Xiao

  3. National Astronomical Observatories, Chinese Academy of Sciences, Beijing, 100012, China

    Chenzhou Cui

Authors
  1. Zhi Hong
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Ce Yu
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Jie Wang
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Jian Xiao
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Chenzhou Cui
    View author publications

    You can also search for this author inPubMed Google Scholar

  6. Jizhou Sun
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Ce Yu.

Additional information

This work is supported by the Joint Research Fund in Astronomy (U1531111) under cooperative agreement between the National Natural Science Foundation of China (NSFC) and Chinese Academy of Sciences (CAS)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hong, Z., Yu, C., Wang, J. et al. AQUAdexIM: highly efficient in-memory indexing and querying of astronomy time series images. Exp Astron 42, 387–405 (2016). https://doi.org/10.1007/s10686-016-9515-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10686-016-9515-0

Keywords