Physics-Informed Deep Reinforcement Learning with Adaptive Action Masking for Secure Second-Level Emergency Scheduling in Energy Networks
DOI:
https://doi.org/10.4108/ew.14204Keywords:
real-time market scheduling, situational risk, data-knowledge driven, secure decision-making, physics-informed deep reinforcement learning, adaptive action maskingAbstract
With the integration of a high proportion of volatile renewable energy into modern power grids, systems frequently face severe supply-demand imbalances and escalated situational risks under extreme events. Addressing the high-dimensional risks and stringent response timing requirements in real-time markets of new-type power systems, this paper proposes a Physics-Informed Deep Reinforcement Learning (PI-DRL) framework with adaptive action masking for secure energy network scheduling and emergency control. First, to mitigate the “curse of dimensionality” inherent in massive heterogeneous resources, a situational risk-driven dynamic feature selection and action masking technique is established to exponentially compress the agent’s exploration space, solving the convergence bottleneck of large-scale resource dispatch. Second, targeting low-probability, high-risk scenarios, a knowledge-driven physical security shield is constructed, utilizing local Jacobian sensitivity matrices to achieve near-real-time physical correction of actions. This overcomes the lack of boundary awareness in traditional black-box models. Furthermore, a Security Reward Shaping mechanism is introduced to foster endogenous security awareness within the agent through closed-loop training, ensuring millisecond-level decision safety during online deployment. Validation on a modified IEEE 118-bus system using high-fidelity meteorological big data from China demonstrates that the proposed method accelerates decision-making by 400-fold compared to traditional MILP algorithms, enabling second-level response. Compared to pure data-driven models, it reduces operational costs by 12.4% and ensures absolute compliance with physical constraints while suppressing risk spikes within 3 minutes during an extreme 450 MW power drop event.
Downloads
References
[1] Liu M V, Yuan B, Wang Z, et al. An open source representation for the nys electric grid to support power grid and market transition studies. IEEE Transactions on Power Systems, 2022, 38(4): 3293-3303.
[2] Kadir S U, Majumder S, Srivastava A K, et al. Reinforcement-learning-based proactive control for enabling power grid resilience to wildfire. IEEE transactions on industrial informatics, 2023, 20(1): 795-805.
[3] Shuai Q, Yin Y, Huang S, et al. Deep reinforcement Learning-Based Real-Time energy management for an integrated. Electric–Thermal energy system. Sustainability, 2025, 17(2): 407.
[4] Wang S, Li L, Dehghanian P. Distributed intelligence for online situational awareness in power grids. IEEE Transactions on Power Systems, 2021, 37(4): 2499-2515.
[5] Zhang X, Wang Q, Bi X, et al. Mitigating cascading failure in power grids with deep reinforcement learning-based remedial actions. Reliability Engineering & System Safety, 2024, 250: 110242.
[6] Hossain R, Gautam M, Thapa J, et al. Deep reinforcement learning assisted co-optimization of Volt-VAR grid service in distribution networks. Sustainable Energy, Grids and Networks, 2023, 35: 101086.
[7] Geng Q, Sun H, Zhou X, et al. A storage-based fixed-time frequency synchronization method for improving transient stability and resilience of smart grid. IEEE Transactions on Smart Grid, 2023, 14(6): 4799-4815.
[8] Ngamroo I, Surinkaew T. Control of distributed converter-based resources in a zero-inertia microgrid using robust deep learning neural network. IEEE Transactions on Smart Grid, 2023, 15(1): 49-66.
[9] Weng L, Yang L, Lei Z, et al. Integrated bus voltage control method for DC microgrids based on adaptive virtual inertia control. Journal of Power Electronics, 2024, 24(7): 1163-1176.
[10] Li Y, Yu C, Shahidehpour M, et al. Deep reinforcement learning for smart grid operations: Algorithms, applications, and prospects. Proceedings of the IEEE, 2023, 111(9): 1055-1096.
[11] Prabawa P, Choi D H. Safe deep reinforcement learning-assisted two-stage energy management for active power distribution networks with hydrogen fueling stations. Applied Energy, 2024, 375: 124170.
[12] Heidari A, Girardin L, Dorsaz C, et al. A trustworthy reinforcement learning framework for autonomous control of a large-scale complex heating system: Simulation and field implementation. Applied Energy, 2025, 378: 124815.
[13] Lin W W, Bai X Y, Kong J. Pseudo-Lable based unsupervised anomaly detection framework for energy data. Computer simulation, 2024, 41(2): 131-136. (In Chinese)
[14] Gautam M. Deep Reinforcement learning for resilient power and energy systems: Progress, prospects, and future avenues. Electricity, 2023, 4(4): 336-380.
[15] Hossain R R, Yin T, Du Y, et al. Efficient learning of power grid voltage control strategies via model-based deep reinforcement learning. Machine Learning, 2024, 113(5): 2675-2700.
[16] Cui H, Ye Y, Hu J, et al. Online preventive control for transmission overload relief using safe reinforcement learning with enhanced spatial-temporal awareness. IEEE Transactions on Power Systems, 2023, 39(1): 517-532.
[17] Chen P, Liu S, Wang X, et al. Physics-shielded multi-agent deep reinforcement learning for safe active voltage control with photovoltaic/battery energy storage systems. IEEE Transactions on Smart Grid, 2022, 14(4): 2656-2667.
[18] Liu D, Zang C, Zeng P, et al. Deep reinforcement learning for real-time economic energy management of microgrid system considering uncertainties. Frontiers in Energy Research, 2023, 11: 1163053.
[19] Du P, Huang B, Liu Z, et al. Real-time energy management for net-zero power systems based on shared energy storage. Journal of Modern Power Systems and Clean Energy, 2024, 12(2): 371-380.
[20] Dang Z, Han X, Niu Z, et al. Power System Emergency Control Strategy Based on Dynamic Spatio-Temporal Constrained Graph Reinforcement Learning. 2025 International Conference on Applied Electrical Engineering and Technology (AEET). IEEE, 2025: 12-17.
[21] Jiang C, Yuan W, Tang J, et al. Adaptive look-ahead dispatch of power system based on deep reinforcement learning with knowledge embedded. 2024 IEEE International Conference on Energy Internet (ICEI). IEEE, 2024: 401-406.
[22] Wu Z, Zhang M, Gao S, et al. Physics-informed reinforcement learning for real-time optimal power flow with renewable energy resources. IEEE Transactions on Sustainable Energy, 2024, 16(1): 216-226.
[23] Huang Q, Huang R, Hao W, et al. Adaptive power system emergency control using deep reinforcement learning. IEEE Transactions on Smart Grid, 2019, 11(2): 1171-1182.
[24] Huang B, Wang J. Applications of physics-informed neural networks in power systems-a review. IEEE Transactions on Power Systems, 2022, 38(1): 572-588.
[25] Cheng L, Wang H, Su Y, et al. Response-Driven Optimal Emergency Control of Power Systems via Deep Learning-Based Sensitivity Embedded Optimization. Energies, 2026, 19(5): 1284.
[26] Shi W, Wu X, Zhang L, et al. Data-Driven and Knowledge-Enhanced Approaches for Real-Time Grid Dispatch: A Comprehensive Survey. 2025 4th Asian Conference on Frontiers of Power and Energy (ACFPE). IEEE, 2025: 244-249.
[27] Gu S, Yang L, Du Y, et al. A review of safe reinforcement learning: Methods, theories, and applications. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024, 46(12): 11216-11235.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Bochun Zhan, Zhengbo Shan, Ke Wang, Rong Yan, Shengmin Qiu, Qingbiao Lin, Zhantao Fan, Nan Lou, Xixi Zhang
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
This is an open-access article distributed under the terms of the Creative Commons Attribution CC BY 4.0 license, which permits unlimited use, distribution, and reproduction in any medium so long as the original work is properly cited.
