Abstract
Sustainable management of groundwater resources under changing climatic conditions require an application of reliable and accurate predictions of groundwater levels. Mechanistic multi-scale, multi-physics simulation models are often too hard to use for this purpose, especially for groundwater managers who do not have access to the complex compute resources and data. Therefore, we analyzed the applicability and performance of four modern deep learning computational models for predictions of groundwater levels. We compare three methods for optimizing the models’ hyperparameters, including two surrogate model-based algorithms and a random sampling method. The models were tested using predictions of the groundwater level in Butte County, California, USA, taking into account the temporal variability of streamflow, precipitation, and ambient temperature. Our numerical study shows that the optimization of the hyperparameters can lead to reasonably accurate performance of all models (root mean squared errors of groundwater predictions of 2 meters or less), but the “simplest” network, namely a multilayer perceptron (MLP) performs overall better for learning and predicting groundwater data than the more advanced long short-term memory or convolutional neural networks in terms of prediction accuracy and time-to-solution, making the MLP a suitable candidate for groundwater prediction.










Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Notes
Note that we usually use a similar approach in continuous optimization where we scale the parameters to the unit hypercube, which improves the surrogate models and eliminates difficulties when sampling by perturbation.
References
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G.S., Davis, A., Dean, J., Devin, M. et al.: Tensorflow: large-scale machine learning on heterogeneous distributed systems (2016). arXiv:1603.04467
Abramson, M.A., Audet, C., Chrissis, J., Walston, J.: Mesh adaptive direct search algorithms for mixed variable optimization. Optim. Lett. 3, 35–47 (2009)
Ali, Z., Hussain, I., Faisal, M., Nazir, H.M., Hussain, T., Shad, M.Y., Shoukry, A.M., Gani, S.H.: Forecasting drought using multilayer perceptron artificial neural network model. Adv. Meteorol., 5681308, 9 pages (2017)
Araujo, P., Astray, G., Ferrerio-Lage, J.A., Mejuto, J.C., Rodriguez-Suarez, J.A., Soto, B.: Multilayer perceptron neural network for flow prediction. J. Environ. Monit. 13(1), 35–41 (2011)
Audet, C., Dennis Jr., J.E.: Mesh adaptive direct search algorithms for constrained optimization. SIAM J. Optim. 17, 188–217 (2006)
Audet, C., Hare, W.: Derivative-Free and Blackbox Optimization. Springer Series in Operations Research and Financial Engineering. Springer, Berlin (2017)
Audet, C., Kokkolaras, M.: Blackbox and derivative-free optimization: theory, algorithms and applications. Optim. Eng. 17(1), 1–2 (2016)
Audet, C., Savard, G., Zghal, W.: A mesh adaptive direct search algorithm for multiobjective optimization. Eur. J. Oper. Res. 204(3), 545–556 (2010)
Balaprakash, P., Salim, M., Uram, T.D., Vishwanath, V., Wild, S.M.: Deephyper: asynchronous hyperparameter search for deep neural networks. In: 2018 IEEE 25th International Conference on High Performance Computing (HiPC), pp. 42–51 (2018)
Bergstra, J., Bengio, Y.: Random search for hyper-parameter optimization. J. Mach. Learn. Res. 13(1), 281–305 (2012)
Bergstra, J., Yamins, D., Cox, D.D.: Making a science of model search: hyperparameter optimization in hundreds of dimensions for vision architectures. In: Proceedings of the 30th International Conference on Machine Learning (2013)
Bishop, C.M., et al.: Neural networks for pattern recognition. Oxford University Press, Oxford (1995)
Booker, A.J., Dennis Jr., J.E., Frank, P.D., Serafini, D.B., Torczon, V., Trosset, M.W.: A rigorous framework for optimization of expensive functions by surrogates. Struct. Multidiscip. Optim. 17, 1–13 (1999)
Borovykh, A., Bohte, S., Oosterlee, C.W.: Conditional Time Series Forecasting with Convolutional Neural Networks (2017). arXiv:1703.04691
Bottou, L.: Large-scale machine learning with stochastic gradient descent. In: 19th International Conference on Computational Statistics, pp. 177–186 (2010)
California Department of Water Resources. SGMA groundwater management. https://www.waterboards.ca.gov/water_issues/programs/gmp/docs/sgma/sgma_20190101.pdf. Accessed 18 May 2020
Chiang, Y.-M., Chang, L.-C., Chang, F.-J.: Comparison of static-feedforward and dynamic-feedback neural networks for rainfall-runoff modeling. J. Hydrol. 290(3–4), 297–311 (2004)
Chollet, F.: keras. GitHub Repository (2015). https://github.com/fchollet/keras. Accessed 18 May 2020
Cook, B.I., Mankin, J.S., Anchukaitis, K.J.: Climate change and drought: from past to future. Curr. Clim. Change Rep. 4(2), 164–179 (2018)
Coulibaly, P., Anctil, F., Aravena, R., Bobée, B.: Artificial neural network modeling of water table depth fluctuations. Water Resour. Res. 37(4), 885–896 (2001)
Cui, Z., Chen, W., Chen, Y.: Multi-scale Convolutional Neural Networks for Time Series Classification (2016). arXiv:1603.06995
Daliakopoulos, I.N., Coulibaly, P., Tsanis, I.K.: Groundwater level forecasting using artificial neural networks. J. Hydrol. 309(1–4), 229–240 (2005)
Datta, R., Regis, R.G.: A surrogate-assisted evolution strategy for constrained multi-objective optimization. Expert Syst. Appl. 57, 270–284 (2016)
Davis, E., Ierapetritou, M.: Kriging based method for the solution of mixed-integer nonlinear programs containing black-box functions. J. Global Optim. 43, 191–205 (2009)
Faunt, C.C.: Groundwater Availability of the Central Valley Aquifer, California. Professional paper 1766, 225 p., U.S. Geological Survey (2009). https://pubs.usgs.gov/pp/1766/PP_1766.pdf. Accessed 18 May 2020
Forrester, A.I.J., Sóbester, A., Keane, A.J.: Multi-fidelity optimization via surrogate modelling. Proc. R. Soc. 463, 3251–3269 (2007)
Fortin, F.-A., De Rainville, F.-M., Gardner, M.-A., Gagné, C., Parizeau, M.: DEAP: evolutionary algorithms made easy. J. Mach. Learn. Res. 13, 2171–2175 (2012)
Gardner, M.W., Dorling, S.R.: Artificial neural networks (the multilayer perceptron): a review of applications in the atmospheric sciences. Atmos. Environ. 32(14–15), 2627–2636 (1998)
Gers, F.A., Schmidhuber, J., Cummins, F.: Learning to forget: continual prediction with LSTM. Neural Comput. 12, 2451–2471 (2000)
Gramacy, R., Le Digabel, S.: The mesh adaptive direct search algorithm with treed Gaussian process surrogates. Pac. J. Optim. 11, 419–447 (2015)
Graves, A., Mohamed, A., Hinton, G.: Speech recognition with deep recurrent neural networks. In: Proceedings of the 2013 International Conference on Acoustics, Speech, and Signal Processing (2013)
Gutmann, H.-M.: A radial basis function method for global optimization. J. Global Optim. 19, 201–227 (2001)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Hinton, G., Srivastava, N., Swersky, K.: Neural Networks for Machine Learning. lecture 6a, Overview of Mini-batch Gradient Descent. Lecture Notes (2012). https://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf. Accessed 18 May 2020
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9, 1735–1780 (1997)
Holmström, K.: An adaptive radial basis algorithm (ARBF) for expensive black-box mixed-integer global optimization. J. Global Optim. 9, 311–339 (2008a)
Holmström, K.: An adaptive radial basis algorithm (ARBF) for expensive black-box global optimization. J. Global Optim. 41, 447–464 (2008b)
Hsu, D.: Multi-period Time Series Modeling with Sparsity Via Bayesian Variational Inference (2018). arXiv:1707.00666v3
Ilievski, I., Akhtar, T., Feng, J., Shoemaker, C.A.: Efficient hyperparameter optimization of deep learning algorithms using deterministic RBF surrogates. In: Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence (2017)
Jin, H., Song, Q., Hu, X.: Auto-Keras: An Efficient Neural Architecture Search System (2019). arXiv:1806.10282 [cs.LG]
Jones, D.R., Schonlau, M., Welch, W.J.: Efficient global optimization of expensive black-box functions. J. Global Optim. 13, 455–492 (1998)
Karandish, F., Šimunek, J.: A comparison of numerical and machine-learning modeling of soil water content with limited input data. J. Hydrol. 543, 892–909 (2016)
Karslıoğlu, O., Gehlmann, M., Müller, J., Nemšàk, S., Sethian, J., Kaduwela, A., Bluhm, H., Fadley, C.: An efficient algorithm for automatic structure optimization in x-ray standing-wave experiments. J. Electron Spectrosc. Relat. Phenom. 230, 10–20 (2019)
Kingma, D.P., Ba, J.L.: ADAM: a method for stochastic optimization. In: ICLR 2015 (2015)
Klein, A., Falkner, S., Bartels, S., Hennig, P., Hutter, F.: Fast Bayesian optimization of machine learning hyperparameters on large datasets. In: Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS) 2017, Fort Lauderdale, Florida, USA, vol. 54 (2017)
Kratzert, F., Klotz, D., Brenner, C., Schulz, K., Herrnegger, M.: Rainfall-runoff modelling using long short-term memory (LSTM) networks. Hydrol. Earth Syst. Sci. 22(11), 6005–6022 (2018)
Kuderer, M., Gulati, S., Burgard, W.: Learning driving styles for autonomous vehicles from demonstration. In: 2015 IEEE International Conference on Robotics and Automation (ICRA), pp. 2641–2646 (2015)
Lakhmiri, D., Digabel, S. Le, Tribes, C.: HyperNOMAD: Hyperparameter Optimization of Deep Neural Networks Using Mesh Adaptive Direct Search (2019). arXiv:1907.01698 [cs.LG]
Langevin, C.D., Hughes, J.D., Banta, E.R., Niswonger, R.G., Panday, S., Provost, A.M.: Documentation for the MODFLOW 6 Groundwater Flow Model. Technical Report, US Geological Survey (2017)
Langhans, W., Müller, J., Collins, W.: Optimization of the Eddy-diffusivity/mass-flux shallow cumulus and boundary-layer parameterization using surrogate models. J. Adv. Model. Earth Syst. 11, 402–416 (2019)
Le Digabel, S.: Algorithm 909: NOMAD–nonlinear optimization with the MADS algorithm. ACM Trans. Math. Softw. 37, 1–15 (2011)
LeCun, Y., Bottou, L., Bengio, Y., Haffner, P., et al.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)
LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521, 436–444 (2015)
Lee, H.K.H., Gramacy, R.B., Linkletter, C., Gray, G.A.: Optimization subject to hidden constraints via statistical emulation. Pac. J. Optim. 7, 467–478 (2011)
Ma, X., Tao, Z., Wang, Y., Yu, H., Wang, Y.: Long short-term memory neural network for traffic speed prediction using remote microwave sensor data. Transp. Res. C Emerg. Technol. 54, 187–197 (2015)
Matheron, G.: Principles of geostatistics. Econ. Geol. 58, 1246–1266 (1963)
Mikolov, T., Karafiát, M., Burget, L., Černockỳ, J., Khudanpur, S.: Recurrent neural network based language model. In: Eleventh Annual Conference of the International Speech Communication Association (2010)
Mitchell, M.: An Introduction to Genetic Algorithms. MIT Press, Cambridge (1996)
Moritz, S., Bartz-Beielstein, T.: imputeTS: Time Series Missing Value Imputation in R. R J. 9, 207–218 (2017)
Müller, J.: MISO: mixed integer surrogate optimization framework. Optim. Eng. 17(1), 177–203 (2015)
Müller, J.: SOCEMO: surrogate optimization of computationally expensive multiobjective problems. INFORMS J. Comput. 29(4), 581–596 (2017)
Müller, J.: An algorithmic framework for the optimization of computationally expensive bi-fidelity black-box problems. INFOR Inf. Syst. Oper. Res. (2019). https://doi.org/10.1080/03155986.2019.1607810
Müller, J., Day, M.: Surrogate optimization of computationally expensive black-box problems with hidden constraints. INFORMS J. Comput. (2019). https://doi.org/10.1287/ijoc.2018.0864
Müller, J., Woodbury, J.: GOSAC: global optimization with surrogate approximation of constraints. J. Glob. Optim. (2017). https://doi.org/10.1007/s10898-017-0496-y
Müller, J., Shoemaker, C.A., Piché, R.: SO-MI: a surrogate model algorithm for computationally expensive nonlinear mixed-integer black-box global optimization problems. Comput. Oper. Res. 40, 1383–1400 (2013a)
Müller, J., Shoemaker, C.A., Piché, R.: SO-I: a surrogate model algorithm for expensive nonlinear integer programming problems including global optimization applications. J. Glob. Optim. 59, 865–889 (2013b)
Müller, J., Paudel, R., Shoemaker, C.A., Woodbury, J., Wang, Y., Mahowald, N.: CH4 parameter estimation in CLM4.5bgc using surrogate global optimization. Geosci. Model Dev. Discus. 8, 141–207 (2015)
Myers, R.H., Montgomery, D.C., Anderson-Cook, C.M.: Response Surface Methodology: Process and Product Optimization Using Designed Experiments, 4th edn. John Wiley & Sons, Inc., Hoboken, NJ (2016)
Najah, A., El-Shafie, A., Karim, O.A., El-Shafie, A.H.: Application of artificial neural networks for water quality prediction. Neural Comput. Appl. 22(1), 187–201 (2013)
Nuñez, L., Regis, R.G., Varela, K.: Accelerated random search for constrained global optimization assisted by radial basis function surrogates. J. Comput. Appl. Math. 340, 276–295 (2018)
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., Duchesnay, E.: Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011)
Powell, M.J.D.: Advances in Numerical Analysis, Vol. 2: Wavelets, Subdivision Algorithms and Radial Basis Functions. Oxford University Press, Oxford, pp. 105–210, Chapter The Theory of Radial Basis Function Approximation in 1990. Oxford University Press, London (1992)
Powell, M.J.D.: Recent Research at Cambridge on Radial Basis Functions New Developments in Approximation Theory, pp. 215–232. Birkhäuser, Basel (1999)
Regis, R.G.: Stochastic radial basis function algorithms for large-scale optimization involving expensive black-box objective and constraint functions. Comput. Oper. Res. 38, 837–853 (2011)
Regis, R.G., Shoemaker, C.A.: A stochastic radial basis function method for the global optimization of expensive functions. INFORMS J. Comput. 19, 497–509 (2007)
Robbins, H., Monro, S.: A stochastic approximation method. Ann. Math. Stat. 22(3), 400–407 (1951)
Rudy, S., Alla, A., Brunton, S.L., Kutz, J.N.: Data-driven identification of parametric partial differential equations. SIAM J. Appl. Dyn. Syst. 18(2), 643–660 (2019)
Rumelhart, D.E., Hinton, G.E., Williams, R.J., et al.: Learning representations by back-propagating errors. Cognit. Model. 5(3), 1 (1988)
Sahoo, S., Russo, T.A., Elliott, J., Foster, I.: Machine learning algorithms for modeling groundwater level changes in agricultural regions of the US. Water Resour. Res. 53(5), 3878–3895 (2017)
Snoek, J., Larochelle, H., Adams, R.P.: Practical Bayesian optimization of machine learning algorithms. In: Advances in Neural Information Processing Systems (2012)
Steefel, C.I., Appelo, C.A.J., Arora, B., Jacques, D., Kalbacher, T., Kolditz, O., Lagneau, V., Lichtner, P.C., Mayer, K.U., Meeussen, J.C.L., et al.: Reactive transport codes for subsurface environmental simulation. Comput. Geosci. 19(3), 445–478 (2015)
Sundermeyer, M., Schluter, R., Ney, H.: LSTM neural networks for language modeling. In: Proceedings of the 12th Annual Conference of the International Speech Communication Association, Portland, Oregon, USA, pp. 601–608 (2012)
Sutskever, I., Martens, J., Hinton, G.E.: Generating text with recurrent neural networks. In: Proceedings of the 28th International Conference on Machine Learning (ICML-11), pp. 1017–1024 (2011)
Tabari, H., Talaee, P.H.: Multilayer perceptron for reference evapotranspiration estimation in a semiarid region. Neural Comput. Appl. 23(2), 341–348 (2013)
Taylor, M.: Liquid Assets: Improving Management of the State’s Groundwater Resources. Legislative Analyst’s Office, Technical Report (2010)
Toal, D., Keane, A.: Efficient multi-point aerodynamic design optimization via co-kriging. J. Aircr. 48(5), 1685–1695 (2011)
Trenn, S.: Multilayer perceptrons: approximation order and necessary number of hidden units. IEEE Trans. Neural Netw. 19(5), 836–844 (2008)
Wild, S.M., Shoemaker, C.A.: Global convergence of radial basis function trust-region algorithms for derivative-free optimization. SIAM Rev. 55, 349–371 (2013)
Xu, T., Spycher, N., Sonnenthal, E., Zhang, G., Zheng, L., Pruess, K.: TOUGHREACT version 2.0: a simulator for subsurface reactive transport under non-isothermal multiphase flow conditions. Comput. Geosci. 37(6), 763–774 (2011)
Young, S.R., Rose, D.C., Karnowski, T.P., Lim, S.-H., Patton, R.M.: Optimizing deep learning hyper-parameters through an evolutionary algorithm. In: MLHPC’15 Proceedings of the Workshop on Machine Learning in High-Performance Computing Environments, Volume Article No. 4 (2015)
Zhang, J., Zhu, Y., Zhang, X., Ye, M., Yang, J.: Developing a long short-term memory (LSTM) based model for predicting water table depth in agricultural areas. J. Hydrol. 561, 918–929 (2018)
Acknowledgements
This work was supported by Laboratory Directed Research and Development (LDRD) funding from Berkeley Lab, provided by the Director, Office of Science, of the U.S. Department of Energy under Contract No. DE-AC02-05CH11231. This research used resources of the National Energy Research Scientific Computing Center (NERSC), a U.S. Department of Energy Office of Science User Facility operated under Contract No. DE-AC02-05CH11231.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
About this article
Cite this article
Müller, J., Park, J., Sahu, R. et al. Surrogate optimization of deep neural networks for groundwater predictions. J Glob Optim 81, 203–231 (2021). https://doi.org/10.1007/s10898-020-00912-0
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10898-020-00912-0