Skip to main content
Springer Nature Link
Log in

The GRAPES evaluation tools based on Python (GetPy)

  • Regular Paper
  • Published:
CCF Transactions on High Performance Computing Aims and scope Submit manuscript

Abstract

This paper describes the GRAPES Evaluation Tools based on Python (GetPy), a community verification and diagnostic tool for the evaluation of numerical models. The traditional statistical verification with confidence level test, the comprehensive scorecard, the precipitation skill score such as TS, ETS, diagnostic score SEEPS and the spatial verification techniques are used as verification modules. The Error tracing techniques conducted on the performance with different scales by wavelet analysis. The diurnal cycle of precipitation can also be calculated by Precipitation frequency-intensity method. Based on simple script architecture GetPy also includes a revised and simplified installation procedure and interactive display system. Users can easily access graphic products and carry out evaluation applications.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  • Ahijevych, D., Gilleland, E., Brown, B.G., et al.: Application of spatial verification methods to idealized and NWP-gridded precipitation forecasts. Wea. Forecast. 24, 1485–1497 (2009)

    Article  Google Scholar 

  • Atger, F.: Verification of intense precipitation forecasts from single models and ensemble prediction systems. Nonlinear Proc. Geophys. 8, 401–417 (2001)

    Article  Google Scholar 

  • Baldwin, M.E., Kain, J.S.: Sensitivity of several performance measures to displacement error, bias, and event frequency. Wea. Forecast. 21, 636–648 (2006)

    Article  Google Scholar 

  • Bannon, P.R.: Atmospheric available energy. J. Atmos. Sci. 69(12), 3745–3762 (2012)

    Article  MathSciNet  Google Scholar 

  • Brady, R.X., Spring, A.: climpred: verification of weather and climate forecasts. J. Open Source Software 6(59), 2781 (2021)

    Article  Google Scholar 

  • Brill, K.F., Mesinger, F.: Applying a general analytic, method for assessing bias sensitivity to bias-adjusted threat, and equitable threat scores. Wea. Forecast. 24, 1748–1754 (2009)

    Article  Google Scholar 

  • Brown, J.D., Demargne, J., Seo, D.-J., Liu, Y.: The ensemble verification system (EVS): a software tool for verifying ensemble forecasts of hydrometeorological and hydrologic variables at discrete locations. Environ. Model. Softw. 25, 854–872 (2010)

    Article  Google Scholar 

  • Davison, A., Hinkley, D.: Bootstrap methods and their application. Cambridge University Press (1997)

    Book  Google Scholar 

  • Demargne, J., Brown, J.D., Y. Liu Y., D-J. Seo, L. Wu, Z. Toth, and Y. Zhu,: Diagnostic verification of hydrometeorological and hydrologic ensembles. Atmos. Sci. Lett. 11, 114–122 (2010)

    Article  Google Scholar 

  • DiCiccio, T., Efron, B.: Bootstrap confidence intervals. Stat. Sci. 11, 189–228 (1996)

    Article  MathSciNet  Google Scholar 

  • Eyring, V., Bock, L., Lauer, A., Righi, M., Schlund, et al.: Earth system model evaluation tool (ESMValTool) v2.0 - an extended set of large-scale diagnostics for quasi-operational and comprehensive evaluation of Earth system models in CMIP. Geosci. Model Dev. 13, 3383–3438 (2020). https://doi.org/10.5194/gmd-13-3383-2020

    Article  Google Scholar 

  • Ferranti, L., Molteni, F., et al.: Diagnosis of extratropical variability in seasonal integrations of the ECMWF model. J. Clim. 7(6), 849–868 (1994)

    Article  Google Scholar 

  • Fraley, C., Raftery, A. E., Gneiting, T., Sloughter J. M.: ensembleBMA: An R package for probabilistic forecasting using ensembles and bayesian model averaging, technical report No. 516R, Department of Statistics, University of Washington (2007)

  • Gilleland, E. Confidence intervals for forecast verification. NCAR Technical Note NCAR/TN-479+STR, 71pp (2010)

  • Haiden, T.M., Rodwell, M.J., Richardson, D.S.: Intercomparison of global model precipitation forecast skill in 2010/11 using the SEEPS score. Mon Wea Rev 140, 2720–2733 (2012)

    Article  Google Scholar 

  • Hall, P., Horowitz, J., Jing, B.: On blocking rules for the bootstrap with dependent data. Biometrika 82, 561–574 (1995)

    Article  MathSciNet  Google Scholar 

  • Klinker, E., Sardeshmukh, P.D.: The diagnosis of mechanical dissipation in the atmosphere from large-scale balance requirements. J. Atm Sci. 49(7), 608–627 (1992)

    Article  Google Scholar 

  • Lahiri, S.: Theoretical comparisons of block bootstrap methods. Ann. Stat. 27, 386–404 (1999)

    Article  MathSciNet  Google Scholar 

  • Laio, F., Tamea, S.: Verification tools for probabilistic forecasts of continuous hydrological variables. Hydrol. Earth Syst. Sci. 11, 1267–1277 (2007)

    Article  Google Scholar 

  • Mass, C.F., Ovens, D., Westrick, K., et al.: Does increasing horizontal resolution produce more skillful forecasts? Bull. Amer. Meteor. Soc. 83, 407–430 (2002)

    Article  Google Scholar 

  • Murphy, A.H.: The Finley affair: a signal event in the history of forecast verification. Wea. Forecast. 11, 3–20 (1996)

    Article  Google Scholar 

  • Murphy, A.H., Winkler, R.L.: A general framework for forecast verification. Mon. Wea. Rev. 115, 1330–1338 (1987)

    Article  Google Scholar 

  • Palmer, T.N., Brankovic, C., et al.: The European centre for medium-range weather forecasts (ECMWF) program on extended-range prediction. Bull. Am. Meteor. Soc. 71(9), 1317–1330 (1990)

    Article  Google Scholar 

  • Paul, K., Mickelson, S., Dennis, J.M.: Light-weight parallel Python tools for earth system modeling workflows. IEEE Int. Confer. Big Data (big Data). 2015, 1985–1994 (2015)

    Article  Google Scholar 

  • Pulkkinen, S., Nerini, D., Pérez Hortal, A.A.: Pysteps: an open-source Python library for probabilistic precipitation nowcasting (v.10). Geosci. Model Develop. Copernic. GmbH 12(10), 4185–4219 (2019)

    Article  Google Scholar 

  • Roberts, N.M., Lean, H.W.: Scale-selective verification of rainfall accumulations from high-resolution forecasts of convective events. Mon. Wea. Rev. 136, 78–97 (2008)

    Article  Google Scholar 

  • Robin, X., Turck, N., Sanchez, J. Muller M.: pROC: Tools for visualizing, smoothing and comparing receiver operating characteristic (ROC curves). R package version 1.3.2 (http://CRAN.R-project.org/package=pROC) (2010)

  • Rocklin, M.” Dask: Parallel computation with blocked algorithms and task scheduling. Proceedings of the 14th python in science conference. 126–132 (2015)

  • Rodwell, M.J., Richardson, D.S., Hewson, T.D., et al.: A new equitable score suitable for verifying precipitation in numerical weather prediction. Quart. J. Roy. Meteor. Soc. 136, 1344–1363 (2010)

    Article  Google Scholar 

  • Roebber, P.J.: Visualizing multiple measures of forecast quality. Wea. Forecast. 24, 601–608 (2009)

    Article  Google Scholar 

  • Skok, G., Roberts, N.: Analysis of fractions skill score properties for random precipitation fields and ECMWF forecasts. Q.J.R. Meteorol. Soc. 142, 2599–2610 (2016)

    Article  Google Scholar 

  • i Ventura, J. F., Lainer, M., Schauwecker, Z.: Pyrad: a real-time weather radar data processing framework based on Py-ART. J. Open Res. Softw. Ubiquity Press. 8(1): 28 (2000)

  • Weisman, M.L., Davis, C., Wang, W., et al.: Experiences with 0–36-h explicit convective forecasts with the WRF-ARW model. Weather Forecast. 23, 407–437 (2008)

    Article  Google Scholar 

  • Zhao, B., Zhang, B.: Assessing hourly precipitation forecast skill with the fractions skill score. J. Meteor. Res. 32(1), 135–145 (2018)

    Article  Google Scholar 

Download references

Funding

This study was funded by the National Key Technologies Research and Development Program of Anhui Province of China grant number (2017YFA0604502).

Author information

Authors and Affiliations

  1. National Key Laboratory of Severe Weather (LaSW), Chinese Academy of Meteorological Science, Beijing, 100081, China

    Bin Zhao, Jiangkai Hu, Dapeng Wang, Fajing Chen, Ziwei Wan & Siyuan Sun

  2. CMA Earth System Modeling and Prediction Centre, China Meteorological Administration, Beijing, 100081, China

    Bin Zhao, Jiangkai Hu, Dapeng Wang, Fajing Chen, Ziwei Wan & Siyuan Sun

  3. National Meteorological Center, China Meteorological Administration, Beijing, 100081, China

    Bo Zhang

Authors
  1. Bin Zhao
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Jiangkai Hu
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Dapeng Wang
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Bo Zhang
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Fajing Chen
    View author publications

    You can also search for this author inPubMed Google Scholar

  6. Ziwei Wan
    View author publications

    You can also search for this author inPubMed Google Scholar

  7. Siyuan Sun
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Bin Zhao.

Ethics declarations

Conflict of interest

On behalf of all authors, the corresponding author states that there is no conflict of interest.

Additional information

Bin Zhao, Ph.D., primarily undertaking research on numerical weather prediction.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhao, B., Hu, J., Wang, D. et al. The GRAPES evaluation tools based on Python (GetPy). CCF Trans. HPC 5, 347–359 (2023). https://doi.org/10.1007/s42514-022-00127-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s42514-022-00127-7

Keywords