Skip to main content
Springer Nature Link
Log in

Gradient Descent for Hyperparameter Selection in Least Squares Support Vector Regression

  • Conference paper
  • First Online:
Applied Intelligence (ICAI 2024)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 2388))

Included in the following conference series:

  • 183 Accesses

Abstract

Hyperparameter selection, an important and challenging problem in machine learning, is particularly crucial for achieving optimal performance. Least Squares Support Vector Regression (LSSVR) combines the powerful capabilities of Support Vector Machines (SVM) in machine learning with the simplicity of the Least Squares method. In this paper, we propose a new hyperparameter selection method for LSSVR by constructing a novel optimization problem, which is then solved using the gradient descent algorithm. By comparing the performance of the grid algorithm and several heuristic algorithms in terms of mean squared error (MSE), the experimental results demonstrate that our method can select more suitable parameters compared to other approaches, ultimately leading to a smaller MSE value.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Erden, C., Demir, H.I., Kökçam, A.H.: Enhancing Machine Learning Model Performance with Hyper Parameter Optimization: A Comparative Study. arXiv preprint arXiv:2302.11406 (2023)

  2. Van Rijn, J.N., Hutter, F.: Hyperparameter importance across datasets. In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2367–2376. ACM (2018)

    Google Scholar 

  3. Han, S., Qubo, C., Meng, H.: Parameter selection in SVM with RBF Kernel function. In: World Autom. Congr. (WAC), pp. 1–4. IEEE (2012)

    Google Scholar 

  4. Boyd, S., Vandenberghe, L.: Convex Optimization. Cambridge University Press (2004)

    Google Scholar 

  5. Yu, L., Dai, W., Tang, L., Wu, J.: A hybrid grid-GA-based LSSVR learning paradigm for crude oil price forecasting. Neural Comput. Appl. 27, 2193–2215 (2016)

    Article  MATH  Google Scholar 

  6. Zheng, J., Shao, X., Gao, L., Jiang, P., Qiu, H.: A prior-knowledge input LSSVR metamodeling method with tuning based on cellular particle swarm optimization for engineering design. Expert Syst. Appl. 41(5), 2111–2125 (2014)

    Article  MATH  Google Scholar 

  7. Zhou, Z.-H.: Machine Learning. Springer Nature (2021)

    Google Scholar 

  8. Awad, M., Khanna, R.: Support vector regression. In: Efficient Learning Machines: Theories, Concepts, and Applications for Engineers and System Designers, pp. 67–80. Springer (2015)

    Google Scholar 

  9. Liashchynskyi, P., Liashchynskyi, P.: Grid Search, Random Search, Genetic Algorithm: A Big Comparison for NAS. arXiv preprint arXiv:1912.06059 (2019)

  10. Yang, L., Shami, A.: On hyperparameter optimization of machine learning algorithms: theory and practice. Neurocomputing 415, 295–316 (2020)

    Article  MATH  Google Scholar 

  11. Staelin, C.: Parameter Selection for Support Vector Machines. Hewlett-Packard Company, Tech. Rep. HPL-2002–354R1 (2003)

    Google Scholar 

  12. Bergstra, J., Bengio, Y.: Random search for hyper-parameter optimization. J. Mach. Learn. Res. 13(2) (2012)

    Google Scholar 

  13. Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., Deng, S.-H.: Hyperparameter optimization for machine learning models based on bayesian optimization. J. Electron. Sci. Technol. 17(1), 26–40 (2019)

    MATH  Google Scholar 

  14. Holland, J.H.: Genetic algorithms. Sci. Am. 267(1), 66–73 (1992)

    Article  MATH  Google Scholar 

  15. Syarif, I., Prugel-Bennett, A., Wills, G.: SVM parameter optimization using grid search and genetic algorithm to improve classification performance. TELKOMNIKA Telecommun. Comput. Electron. Control 14(4), 1502–1509 (2016)

    MATH  Google Scholar 

  16. Karaboga, D., Basturk, B.: On the performance of artificial bee colony (ABC) algorithm. Appl. Soft Comput. 8(1), 687–697 (2008)

    Article  MATH  Google Scholar 

  17. Karaboga, D., Gorkemli, B., Ozturk, C., Karaboga, N.: A comprehensive survey: artificial bee colony (ABC) algorithm and applications. Artif. Intell. Rev. 42, 21–57 (2014)

    Article  MATH  Google Scholar 

  18. Kennedy, J., Eberhart, R.: Particle swarm optimization. In: Proceedings of ICNN'95-International Conference on Neural Networks, vol. 4, pp. 1942–1948. IEEE (1995)

    Google Scholar 

  19. Mirjalili, S., Mirjalili, S.M., Lewis, A.: Grey wolf optimizer. Adv. Eng. Softw. 69, 46–61 (2014)

    Article  MATH  Google Scholar 

  20. Mirjalili, S., Lewis, A.: The whale optimization algorithm. Adv. Eng. Softw. 95, 51–67 (2016)

    Article  MATH  Google Scholar 

Download references

Acknowledgments

This work is supported in part by National Natural Science Foundation of China (Nos. 62106112, 62366035 and 61966024), and in part by Natural Science Foundation of Inner Mongolia Autonomous Region of China (No. 2023MS01006).

Author information

Authors and Affiliations

  1. Inner Mongolia University, Hohhot, 010021, China

    Ao Le, Bai Lan & Wang Zhen

Authors
  1. Ao Le
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bai Lan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wang Zhen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bai Lan .

Editor information

Editors and Affiliations

  1. Eastern Institute of Technology, Ningbo, China

    De-Shuang Huang

  2. China University of Mining and Technology, Xuzhou, China

    Wei Chen

  3. Tianjin University of Science and Technology, Tianjin, China

    Chuanlei Zhang

  4. Eastern Institute of Technology, Ningbo, China

    Yijie Pan

  5. Eastern Institute of Technology, Ningbo, China

    Qinhu Zhang

  6. Fujian Agriculture and Forestry University, Fuzhou, China

    Xiangzeng Kong

Ethics declarations

Disclosure of Interests

The authors have no competing interests to declare that are relevant to the content of this article.

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Le, A., Lan, B., Zhen, W. (2025). Gradient Descent for Hyperparameter Selection in Least Squares Support Vector Regression. In: Huang, DS., Chen, W., Zhang, C., Pan, Y., Zhang, Q., Kong, X. (eds) Applied Intelligence. ICAI 2024. Communications in Computer and Information Science, vol 2388. Springer, Singapore. https://doi.org/10.1007/978-981-96-1904-7_26

Download citation

  • DOI: https://doi.org/10.1007/978-981-96-1904-7_26

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-96-1903-0

  • Online ISBN: 978-981-96-1904-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics