Abstract
Research on the formation structure and characteristics of top-impact papers can serve as the basis for the construction of social media-based information communication pattern, and guide papers to gain high attention in both the Internet and academia. In this study, we used the analytic hierarchy process (AHP) to assign index weights for comprehensive impact (CI) and uses this to determine high-impact papers (HIP). By analyzing evaluation indexes and text attributes, as well as by comparing the characteristics of high-citation papers (HCPs) and HIPs, we elucidated the mechanism of HIPs formation. For this, we extracted all data (2630 articles) from PLoS Biology (2009–2018) as our objects of analysis. In addition, a structural diagram was used to summarize the results. The study results revealed that academic and social impacts of HIPs are interdependent and complementary. The indicators’ variation frequency and amplitude of HIPs are far higher than those of HCPs. Two indicators are constituting the direct impact source for HIPs. Redundant keyword and title do not significantly contribute to the impact of the paper. Under a moderate number of conditions, the more collaborative the paper, the more it is recommended and communicated. Although citation remains the most widely accepted index of a paper, researchers should not ignore the impact of papers via social media.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Adams, J. (2018). Information and misinformation in bibliometric time-trend analysis. Journal of Informetrics, 12(4), 1063–1071. https://doi.org/10.1016/j.joi.2018.08.009.
Adie, E. (2014). Taking the alternative mainstream. Profesional De La Informacion, 23(4), 349–351. https://doi.org/10.3145/epi.2014.jul.01.
Aksnes, D. W. (2003). Characteristics of highly cited papers. Research Evaluation, 12(3), 159–170. https://doi.org/10.3152/147154403781776645.
Aung, H. H., Zheng, H., Erdt, M., Aw, A. S., Sin, S. C. J., & Theng, Y. L. (2019). Investigating familiarity and usage of traditional metrics and altmetrics. Journal of the Association for Information Science and Technology, 70(8), 872–887. https://doi.org/10.1002/asi.24162.
Balbi, S., Misuraca, M., & Scepi, G. (2018). Combining different evaluation systems on social media for measuring user satisfaction. Information Processing and Management, 54(4), 674–685. https://doi.org/10.1016/j.ipm.2018.04.009.
Baltussen, A., & Kindler, C. H. (2004). Citation classics in critical care medicine. Intensive Care Medicine, 30(5), 902–910. https://doi.org/10.1007/s00134-004-2195-7.
Bornmann, L. (2014). Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics. Journal of Informetrics, 8(4), 895–903. https://doi.org/10.1016/j.joi.2014.09.005.
Bornmann, L., Haunschild, R., & Adams, J. (2019). Do altmetrics assess societal impact in a comparable way to case studies? An empirical test of the convergent validity of altmetrics based on data from the UK research excellence framework (REF). Journal of Informetrics, 13(1), 325–340. https://doi.org/10.1016/j.joi.2019.01.008.
Bouyssou, D., & Marchant, T. (2016). Ranking authors using fractional counting of citations: An axiomatic approach. Journal of Informetrics, 10(1), 183–199. https://doi.org/10.1016/j.joi.2015.12.006.
Bramoullé, Y., & Ductor, L. (2018). Title length. Journal of Economic Behavior & Organization, 150, 311–324. https://doi.org/10.1016/j.jebo.2018.01.014.
Chapa, J., Haq, Z., & Cifu, A. S. (2017). Comparative analysis of the factors associated with citation and media coverage of clinical research. Scientometrics, 112(3), 1271–1283. https://doi.org/10.1007/s11192-017-2428-5.
Chen, S., Arsenault, C., & Larivière, V. (2015). Are top-cited papers more interdisciplinary? Journal of Informetrics, 9(4), 1034–1046. https://doi.org/10.1016/j.joi.2015.09.003.
Chi, P.-S., Gorraiz, J., & Glanzel, W. (2019). Comparing capture, usage and citation indicators: an altmetric analysis of journal papers in chemistry disciplines. Scientometrics, 120(3), 1461–1473. https://doi.org/10.1007/s11192-019-03168-y.
Costas, R., Zahedi, Z., & Wouters, P. (2015). Do “altmetrics” correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective. Journal of the Association for Information Science and Technology, 66(10), 2003–2019. https://doi.org/10.1002/asi.23309.
Dakiche, N., Tayeb, F. B. S., Slimani, Y., & Benatchba, K. (2019). Tracking community evolution in social networks: A survey. Information Processing and Management, 56(3), 1084–1102. https://doi.org/10.1016/j.ipm.2018.03.005.
Ding, Y. (2011). Scientific collaboration and endorsement: Network analysis of coauthorship and citation networks. Journal of Informetrics, 5(1), 187–203. https://doi.org/10.1016/j.joi.2010.10.008.
Erdt, M., Nagarajan, A., Sin, S. C., & Theng, Y. L. (2016). Altmetrics: An analysis of the state-of-the-art in measuring research impact on social media. Scientometrics, 109(2), 1117–1166. https://doi.org/10.1007/s11192-016-2077-0.
Ezema, I. J., & Ugwu, C. I. (2019). Correlating research impact of library and information science journals using citation counts and altmetrics attention. Information Discovery and Delivery, 47(3), 143–153. https://doi.org/10.1108/idd-08-2018-0029.
Fei, S., & Haustein, S. (2017). On the citation advantage of tweeted papers at the journal level. Proceedings of the Association for Information Science and Technology, 54(1), 7. https://doi.org/10.1002/pra2.2017.14505401040.
Fu, H. Z., Chuang, K. Y., Wang, M. H., & Ho, Y. S. (2011). Characteristics of research in China assessed with essential science indicators. Scientometrics, 88(3), 841–862. https://doi.org/10.1007/s11192-011-0416-8.
Hammarfelt, B. (2014). Using altmetrics for assessing research impact in the humanities. Scientometrics, 101(2), 1419–1430. https://doi.org/10.1007/s11192-014-1261-3.
Hassan, S. U., Imran, M., Gillani, U., Aljohani, N. R., Bowman, T. D., & Didegah, F. (2017). Measuring social media activity of scientific literature: an exhaustive comparison of scopus and novel altmetrics big data. Scientometrics, 113(2), 1037–1057. https://doi.org/10.1007/s11192-017-2512-x.
Haustein, S., Costas, R., & Lariviere, V. (2015). Characterizing social media metrics of scholarly papers: The effect of document properties and collaboration patterns. PLoS ONE. https://doi.org/10.1371/journal.pone.0120495.
Heydari, S., Shekofteh, M., & Kazerani, M. (2019). Relationship between altmetrics and citations: A study on the highly cited research papers. Desidoc Journal of Library & Information Technology, 39(4), 169–174. https://doi.org/10.14429/djlit.39.4.14204.
Ho, Y.-S. (2014). A bibliometric analysis of highly cited articles in materials science. Current Science, 107(9), 1565–1572.
Holmberg, K., & Vainio, J. (2018). Why do some research articles receive more online attention and higher altmetrics? Reasons for online success according to the authors. Scientometrics, 116(1), 435–447. https://doi.org/10.1007/s11192-018-2710-1.
Hu, K., Luo, Q., Qi, K. L., Yang, S. L., Mao, J., Fu, X. K., et al. (2019). Understanding the topic evolution of scientific literatures like an evolving city: Using Google Word2Vec model and spatial autocorrelation analysis. Information Processing and Management, 56(4), 1185–1203. https://doi.org/10.1016/j.ipm.2019.02.014.
Hu, Z., Tian, W., Xu, S., Zhang, C., & Wang, X. (2018). Four pitfalls in normalizing citation indicators: An investigation of ESI’s selection of highly cited papers. Journal of Informetrics, 12(4), 1133–1145. https://doi.org/10.1016/j.joi.2018.09.006.
Kim, E. S., Yoon, D. Y., Kim, H. J., Lee, K., Kim, Y., Bae, J. S., et al. (2019). The most mentioned neuroimaging articles in online media: a bibliometric analysis of the top 100 articles with the highest Altmetric Attention Scores. Acta Radiologica, 60(12), 1680–1686. https://doi.org/10.1177/0284185119843226.
Liao, S.-H., & Chang, H.-K. (2016). A rough set-based association rule approach for a recommendation system for online consumers. Information Processing and Management, 52(6), 1142–1160. https://doi.org/10.1016/j.ipm.2016.05.003.
Martínez, M. A., Herrera, M., Contreras, E., Ruíz, A., & Herrera-Viedma, E. (2015). Characterizing highly cited papers in Social Work through H-Classics. Scientometrics, 102(2), 1713–1729. https://doi.org/10.1007/s11192-014-1460-y.
Meier, A., & Tunger, D. (2019). Altmetrics—Usage, impressions and experiences at juelich research center. Information-Wissenschaft Und Praxis, 70(4), 187–191. https://doi.org/10.1515/iwp-2019-2030.
Mohammadi, E., & Thelwall, M. (2014). Mendeley readership altmetrics for the social sciences and humanities: Research evaluation and knowledge flows. Journal of the Association for Information Science and Technology, 65(8), 1627–1638. https://doi.org/10.1002/asi.23071.
Moldwin, M. B., & Liemohn, M. W. (2018). High-citation papers in space physics: Examination of gender, country, and paper characteristics. Journal of Geophysical Research-Space Physics, 123(4), 2557–2565. https://doi.org/10.1002/2018ja025291.
Neylon, C., & Wu, S. (2009). Article-level metrics and the evolution of scientific impact. PLoS Biology. https://doi.org/10.1371/journal.pbio.1000242.
Nuzzolese, A. G., Ciancarini, P., Gangemi, A., Peroni, S., Poggi, F., & Presutti, V. (2019). Do altmetrics work for assessing research quality? Scientometrics, 118(2), 539–562. https://doi.org/10.1007/s11192-018-2988-z.
Ortega, J. L. (2019). Exploratory analysis of Publons metrics and their relationship with bibliometric and altmetric impact. Aslib Journal of Information Management, 71(1), 124–136. https://doi.org/10.1108/ajim-06-2018-0153.
Ouchi, A., Saberi, M. K., Ansari, N., Hashempour, L., & Isfandyari-Moghaddam, A. (2019). Do altmetrics correlate with citations? A study based on the 1,000 most-cited articles. Information Discovery and Delivery, 47(4), 192–202. https://doi.org/10.1108/idd-07-2019-0050.
Parmar, A., Ganesh, R., & Mishra, A. K. (2019). The top 100 cited articles on Obsessive Compulsive Disorder (OCD): A citation analysis. Asian Journal of Psychiatry, 42, 34–41. https://doi.org/10.1016/j.ajp.2019.03.025.
Peoples, B. K., Midway, S. R., Sackett, D., Lynch, A., & Cooney, P. B. (2016). Twitter predicts citation rates of ecological research. PLoS ONE. https://doi.org/10.1371/journal.pone.0166570.
Pislyakov, V., & Shukshina, E. (2014). Measuring excellence in Russia: Highly cited papers, leading institutions, patterns of national and international collaboration. Journal of the Association for Information Science and Technology, 65(11), 2321–2330. https://doi.org/10.1002/asi.23093.
Priem, J., Piwowar, H. A., & Hemminger, B. M. (2012). Altmetrics in the wild: Using social media to explore scholarly impact. arXiv e-prints. https://ui.adsabs.harvard.edu/abs/2012arXiv1203.4745P. Accessed 18 Nov 2019.
Repiso, R., Castillo-Esparcia, A., & Torres-Salinas, D. (2019). Altmetrics, alternative indicators for Web of Science Communication studies journals. Scientometrics, 119(2), 941–958. https://doi.org/10.1007/s11192-019-03070-7.
Ruano, J., Aguilar-Luque, M., Isla-Tejera, B., Alcalde-Mellado, P., Gay-Mimbrera, J., Hernández-Romero, J. L., et al. (2018). Relationships between abstract features and methodological quality explained variations of social media activity derived from systematic reviews about psoriasis interventions. Journal of Clinical Epidemiology, 101, 35–43. https://doi.org/10.1016/j.jclinepi.2018.05.015.
Saberi, M. K., & Ekhtiyari, F. (2019). Usage, captures, mentions, social media and citations of LIS highly cited papers: An altmetrics study. Performance Measurement and Metrics, 20(1), 37–47. https://doi.org/10.1108/pmm-10-2018-0025.
Savov, P., Jatowt, A., & Nielek, R. (2020). Identifying breakthrough scientific papers. Information Processing and Management, 57(2), 102168. https://doi.org/10.1016/j.ipm.2019.102168.
Smith, D. R. (2007). Ten citation classics from the New Zealand Medical Journal. The New Zealand medical journal, 120(1267), U2871–U2871.
Stack, S. (2012). Citation classics in suicide and life threatening behavior: A research note. Suicide and Life-Threatening Behavior, 42(6), 628–639. https://doi.org/10.1111/j.1943-278x.2012.00117.x.
Stamatelatos, G., Gyftopoulos, S., Drosatos, G., & Efraimidis, P. S. (2020). Revealing the political affinity of online entities through their Twitter followers. Information Processing and Management, 57(2), 102172. https://doi.org/10.1016/j.ipm.2019.102172.
Tahamtan, I., Afshar, A. S., & Ahamdzadeh, K. (2016). Factors affecting number of citations: A comprehensive review of the literature. Scientometrics, 107(3), 1195–1225. https://doi.org/10.1007/s11192-016-1889-2.
Tahamtan, I., & Bornmann, L. (2020). Altmetrics and societal impact measurements: Match or mismatch? a literature review. Profesional de la Informacion. https://doi.org/10.3145/epi.2020.ene.02.
Tam, W. W. S., Wong, E. L. Y., Wong, F. C. Y., & Hui, D. S. C. (2013). Citation classics: Top 50 cited articles in ‘respiratory system’. Respirology, 18(1), 71–81. https://doi.org/10.1111/j.1440-1843.2012.02262.x.
Thelwall, M. (2016). Interpreting correlations between citation counts and other indicators. Scientometrics, 108(1), 337–347. https://doi.org/10.1007/s11192-016-1973-7.
Thelwall, M., & Nevill, T. (2018). Could scientists use Altmetric.com scores to predict longer term citation counts? Journal of Informetrics, 12(1), 237–248. https://doi.org/10.1016/j.joi.2018.01.008.
Tuke, J., Nguyen, A., Nasim, M., Mellor, D., Wickramasinghe, A., Bean, N., et al. (2019). Pachinko Prediction: A Bayesian method for event prediction from social media data. Information Processing and Management. https://doi.org/10.1016/j.ipm.2019.102147.
Waltman, L., van Eck, N. J., van Leeuwen, T. N., & Visser, M. S. (2013). Some modifications to the SNIP journal impact indicator. Journal of Informetrics, 7(2), 272–285. https://doi.org/10.1016/j.joi.2012.11.011.
Wang, F., Fan, Y., Zeng, A., & Di, Z. (2019). Can we predict ESI highly cited publications? Scientometrics, 118(1), 109–125. https://doi.org/10.1007/s11192-018-2965-6.
Xie, J., Gong, K., Li, J., Ke, Q., Kang, H., & Cheng, Y. (2019). A probe into 66 factors which are possibly associated with the number of citations an article received. Scientometrics, 119(3), 1429–1454. https://doi.org/10.1007/s11192-019-03094-z.
Zarrinkalam, F., Kahani, M., & Bagheri, E. (2018). Mining user interests over active topics on social networks. Information Processing and Management, 54(2), 339–357. https://doi.org/10.1016/j.ipm.2017.12.003.
Zhang, L., & Wang, J. (2018). Why highly cited articles are not highly tweeted? A biology case. Scientometrics, 117(1), 495–509. https://doi.org/10.1007/s11192-018-2876-6.
Zhou, Y., Yang, S., Li, Y., Chen, Y., Yao, J., & Qazi, A. (2020). Does the review deserve more helpfulness when its title resembles the content? Locating helpful reviews by text mining. Information Processing and Management, 57(2), 102179. https://doi.org/10.1016/j.ipm.2019.102179.
Acknowledgements
This research was supported by the National Social Science Foundation of China under Grant 17BGL031.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Hou, J., Ma, D. How the high-impact papers formed? A study using data from social media and citation. Scientometrics 125, 2597–2615 (2020). https://doi.org/10.1007/s11192-020-03703-2
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11192-020-03703-2