Abstract
The advanced progress in telescope facilities is continuously generating observation images containing billions of objects. Cross-match is a fundamental operation in astronomical data processing which enables astronomers to identify and correlate objects belonging to different observations in order to make new scientific achievements by studying the temporal evolution of the sources or combining physical properties. Comparing such vast amount of astronomical catalogs with low latency is a serious challenge. In this demonstration, we propose HX-MATCH, a new cross-matching algorithm based on Healpix and showcase an in-memory distributed framework where astronomers can compare large datasets.
This work has made use of data from the European Space Agency (ESA) mission Gaia (https://www.cosmos.esa.int/gaia), processed by the Gaia Data Processing and Analysis Consortium (DPAC, https://www.cosmos.esa.int/web/gaia/dpac/consortium). Funding for the DPAC has been provided by national institutions, in particular the institutions participating in the Gaia Multilateral Agreement.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
ALADIN. http://aladin.u-strasbg.fr/
Armbrust, M., et al.: Spark SQL: relational data processing in spark. In: Proceedings of the 2015 SIGMOD International Conference on Management of Data (2015)
Brahem, M.A., et al.: AstroSpark: towards a distributed data server for big data in astronomy. SIGSPATIAL Ph.D. Symposium (2016)
Eldawy, A., Mokbel, M.F.: Spatialhadoop: a mapreduce framework for spatial data. In: 2015 IEEE 31st International Conference on Data Engineering (ICDE). IEEE (2015)
Gorski, K.M., et al.: HEALPix: a framework for high-resolution discretization and fast analysis of data distributed on the sphere. Astrophys. J. 622(2), 759 (2005)
Xie, D., et al.: Simba: efficient in-memory spatial analytics. In: Proceedings of the 2016 International Conference on Management of Data. ACM (2016)
Zhao, Q., Sun, J., Yu, C., Cui, C., Lv, L., Xiao, J.: A paralleled large-scale astronomical cross-matching function. In: Hua, A., Chang, S.-L. (eds.) ICA3PP 2009. LNCS, vol. 5574, pp. 604–614. Springer, Heidelberg (2009). doi:10.1007/978-3-642-03095-6_57
Acknowledgments
This work is partly founded by the CNES (Centre National d’Etudes Spatiales). We would like to thank Frederic Arenou at Paris Observatory, and Veronique Valette at CNES for their cooperation.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Brahem, M., Zeitouni, K., Yeh, L. (2017). HX-MATCH: In-Memory Cross-Matching Algorithm for Astronomical Big Data. In: Gertz, M., et al. Advances in Spatial and Temporal Databases. SSTD 2017. Lecture Notes in Computer Science(), vol 10411. Springer, Cham. https://doi.org/10.1007/978-3-319-64367-0_26
Download citation
DOI: https://doi.org/10.1007/978-3-319-64367-0_26
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-64366-3
Online ISBN: 978-3-319-64367-0
eBook Packages: Computer ScienceComputer Science (R0)