Reinforcement Learning Data-Driven Optimal Load-Frequency Control for Power Systems
DOI:
https://doi.org/10.4108/ew.7500Keywords:
Power system, reinforcement learning, data-driven, dynamic programming, Load frequency controlAbstract
INTRODUCTION: Power systems are complex due to their time-varying and uncertain parameters, challenging conventional control methods.
OBJECTIVES: This study proposes an adaptive dynamic programming (ADP) controller to address this limitation. The ADP controller eliminates the need for pre-existing knowledge of the system dynamics, a significant advantage in real-world applications.
METHODS: By iteratively solving the Riccati equation using only system state and input data, the controller learns an approximate optimal control strategy. In this study, we use an iterative computational approach with an online adaptive optimal controller designed for unknown power system dynamics.
RESULTS: Utilizing real-time collected system states and input information, even in the absence of knowledge about the power system matrix, we achieve iterative solutions for the algebraic Riccati equation, enabling the computation of an optimal controller. Simulation results demonstrate the ease of implementation of this approach in power system load frequency control (LFC).
CONCLUSION: The proposed ADP controller exhibits good control performance of grid stability, making it a valuable reference for LFC, especially in scenarios with unknown system parameters.
Downloads
References
[1] Wu Jie, Liu Yongqiang, Chen Wei. Application of modern control technology in power system control (1). Chinese Journal of Electrical Engineering, 1998, (06): 2-7.
[2] Cheng Liang. Research on the application of adaptive dynamic programming in power system. Huazhong University of Science and Technology, 2014.
[3] Wu F. F., Moslehi K. and Bose A., "Power System Control Centers: Past, Present, and Future," in Proceedings of the IEEE, vol. 93, no. 11, pp. 1890-1908, Nov. 2005, doi: 10.1109/JPROC.2005.857499.
[4] Kundur P. Power system stability. Power system stability and control, 2007, 10: 7-1.
[5] Kundur P., Paserba J. and Vitet, S. "Overview on definition and classification of power system stability," CIGRE/IEEE PES International Symposium Quality and Security of Electric Power Delivery Systems, 2003. CIGRE/PES 2003., Montreal, QC, Canada, 2003, pp. 1-4, doi: 10.1109/QSEPDS.2003.159786.
[6] Peddakapu K, Mohamed M R, Srinivasarao P, et al. A state-of-the-art review on modern and future developments of AGC/LFC of conventional and renewable energy-based power systems. Renewable energy focus, 2022.
[7] Rangi S, Jain S, Arya Y. SSA-optimized cascade optimal-PIDN controller for multi-area power system with RFB under deregulated environment. Optim Control Appl Meth. 2023; 44(4): 1972–1994.
[8] Sandeep Rangi, Sheilza Jain, Yogendra Arya, Utilization of energy storage devices with optimal controller for multi-area hydro-hydro power system under deregulated environment, Sustainable Energy Technologies and Assessments, Volume 52, Part C,2022,102191, ISSN 2213-1388.
[9] Jagatheesan, Kaliannan et al. “Evolutionary Algorithm Based LFC of Single Area Thermal Power System with Different Steam Configurations and Non-linearity.” Advances in intelligent systems and computing 435 (2016): 185-194.
[10] Khooban M H, Niknam T. A new intelligent online fuzzy tuning approach for multi-area load frequency control: Self Adaptive Modified Bat Algorithm.International Journal of Electrical Power & Energy Systems, 2015, 71(oct.):254-261. DOI:10.1016/j.ijepes.2015.03.017.
[11] Bechert T. E. and Chen Nanming, "Area automatic generation control by multi-pass dynamic programming," in IEEE Transactions on Power Apparatus and Systems, vol. 96, no. 5, pp. 1460-1469, Sept. 1977, doi: 10.1109/T-PAS.1977.32474.
[12] Vrdoljak K., Perić N. and Šepac D., "Optimal distribution of load-frequency control signal to hydro power plants," 2010 IEEE International Symposium on Industrial Electronics, Bari, Italy, 2010, pp. 286-291, doi: 10.1109/ISIE.2010.5637553.
[13] Hain, Y, Kulessky, et al. Identification-Based Power Unit Model for Load-Frequency Control Purposes. IEEE Transactions on Power Systems, 2000.
[14] Feliachi, A. Load frequency control using reduced order models and local observers. Int. J. Energy Syst.
[15] Gao Guige. Development and prospect of optimal control theory. Journal of Shanghai Dianji University, 2005, (03): 33-35 + 39.
[16] Todorov E. Optimal control theory. Bayesian brain: probabilistic approaches to neural coding, 2006: 268-298.
[17] Jordan B W, Polak E. Theory of a class of discrete optimal control systems. International Journal of Electronics, 1964, 17(6): 697-711.
[18] Gholamrezaie V., Dozein M. G., H. Monsef and B. Wu, "An Optimal Frequency Control Method Through a Dynamic Load Frequency Control (LFC) Model Incorporating Wind Farm," in IEEE Systems Journal, vol. 12, no. 1, pp. 392-401, March 2018, doi: 10.1109/JSYST.2016.2563979.
[19] Calovic M., "Linear Regulator Design for a Load and Frequency Control," in IEEE Transactions on Power Apparatus and Systems, vol. PAS-91, no. 6, pp. 2271-2285, Nov. 1972, doi: 10.1109/TPAS.1972.293383.
[20] Jia Y., Meng K., Wu K., Sun C. and Dong Z. Y., "Optimal Load Frequency Control for Networked Power Systems Based on Distributed Economic MPC," in IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 51, no. 4, pp. 2123-2133, April 2021, doi: 10.1109/TSMC.2020.3019444.
[21] Liu F, Song Y H, Ma J, et al. Optimal load-frequency control in restructured power systems. IEE Proceedings-Generation, Transmission and Distribution, 2003, 150(1): 87-95.
[22] Rahmani M, Sadati N. Hierarchical optimal robust load-frequency control for power systems. IET generation, transmission & distribution, 2012, 6(4): 303-312.
[23] Alhelou H H, Hamedani-Golshan M E, Zamani R, et al. Challenges and opportunities of load frequency control in conventional, modern and future smart power systems: A comprehensive review. Energies, 2018, 11(10): 2497.
[24] Bian T, Jiang Z P. Value iteration and adaptive dynamic programming for data-driven adaptive optimal control design. Automatica, 2016, 71: 348-360. DOI: 10.1016/j.automatica.2016.05.003.
[25] Werbos P.J. Advanced forecasting methods for global crisis warning and models of intelligence. General Systems Yearbook, 1977:25~38
[26] Optimal control strategy for a class of discrete-time nonlinear systems with time delay based on adaptive dynamic programming. Journal of Automation, 2010,36 (01): 121-129.
[27] Lin, Zhang, Song et al. Adaptive dynamic programming with ε error bound for nonlinear discrete-time systems. Control and decision, 2011,26 (10): 1586-1590 +1595. DOI: 10.13195 / j.cd.2011.10.149. linxf. 019
[28] Mingming HA. Integrated adaptive dynamic programming intelligent control for discrete-time dynamic systems. Beijing University of Science and Technology, 2023. DOI: 10.26945 / d.cnki.gbjku.2023.000441
[29] Ni Z, He H, Zhong X, et al. Model-Free Dual Heuristic Dynamic Programming. IEEE Transactions on Neural Networks & Learning Systems, 2017, 26(8): 1834-1839. DOI:10.1109/TNNLS.2015.2424971.
[30] Rizvi S. A. A. and Lin Z., "Adaptive Dynamic Programming for Model-Free Global Stabilization of Control Constrained Continuous-Time Systems," in IEEE Transactions on Cybernetics, vol. 52, no. 2, pp. 1048-1060, Feb. 2022, doi: 10.1109/TCYB.2020.2989419.
[31] Jiang Y. and Jiang Z. P., "Global Adaptive Dynamic Programming for Continuous-Time Nonlinear Systems," in IEEE Transactions on Automatic Control, vol. 60, no. 11, pp. 2917-2929, Nov. 2015, doi: 10.1109/TAC.2015.2414811.
[32] Mu C., Ni Z., Sun C. and He H., "Data-Driven Tracking Control with Adaptive Dynamic Programming for a Class of Continuous-Time Nonlinear Systems," in IEEE Transactions on Cybernetics, vol. 47, no. 6, pp. 1460-1470, June 2017, doi: 10.1109/TCYB.2016.2548941.
[33] Wang Y., Zhou R. and Wen C., "Robust controller design for power system load frequency control," [Proceedings 1992] The First IEEE Conference on Control Applications, Dayton, OH, USA, 1992, pp. 642-646 vol.2, doi: 10.1109/CCA.1992.269772.
[34] Kleinman D L. On an iterative technique for Riccati equation computations.IEEE Trans.a.c, 1968, 13(1):114-115.DOI:10.1109/TAC.1968.1098829.
[35] Jiang, Y. Z., et al. Computational adaptive optimal control for continuous-time linear systems with completely unknown dynamics. Automatica Oxford, 2012
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Yi Zhao
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
This is an open-access article distributed under the terms of the Creative Commons Attribution CC BY 4.0 license, which permits unlimited use, distribution, and reproduction in any medium so long as the original work is properly cited.
