Grid-Interactive Hyperscale Data Centers: Deep Reinforcement Learning for Joint Workload–Cooling Scheduling to Enable Demand Response and Renewable Integration
DOI:
https://doi.org/10.4108/ew.14440Keywords:
Data Center Flexiblity, Demand Response, Renewable Energy Integration, Grid-Interactive Operation, Deep Reinforcement Learning, Joint Workload-Cooling SchedulingAbstract
Driven by artificial intelligence and cloud computing, hyperscale data centers are becoming one of the fastest-growing electrical loads worldwide and are increasingly recognized as a new class of flexible loads capable of supporting demand response (DR) and the integration of variable renewable energy (VRE). However, their two principal control levers—IT workload scheduling and cooling system operation—have traditionally been managed in a decoupled manner, leaving both energy efficiency and demand-side flexibility under-exploited. This paper proposes a deep reinforcement learning (DRL) framework that jointly co-schedules computing and thermal resources so that a hyperscale data center can operate as a grid-interactive flexible load. We formulate the joint problem as a constrained Markov Decision Process and develop an actor-critic algorithm combining Deep Deterministic Policy Gradient with a safety shield mechanism to guarantee thermal constraint satisfaction during both training and deployment. A high-fidelity digital twin simulation environment enables safe Sim-to-Real training. Extensive experiments demonstrate that the proposed approach reduces total electricity consumption by 18-25% compared to baseline controllers, cuts thermal violations by over 90%, and maintains service level agreement compliance, while broadening the controllable power envelope of the facility to provide a technical basis for participating in DR programs and aligning data-center power profiles with renewable generation. The framework bridges IT-side and facility-side control and supports the evolution of hyperscale data centers from passive electricity consumers toward active, grid-interactive participants in renewable-penetrated power systems.
Downloads
References
[1] MASANET E., SHEHABI A., LEI N., SMITH S. and KOOMEY J. (2020) Recalibrating global data center energy-use estimates. Science 367(6481): 984–986.
[2] DU Y., ZHOU Z., YANG X. et al. (2023) Dynamic thermal environment management technologies for data center: A review. Renewable and Sustainable Energy Reviews 187: 113761.
[3] ZHANG Y., SHAN K., LI X., LI H. and YAN J. (2023) Research and technologies for next-generation high-temperature data centers—State-of-the-arts and future perspectives. Renewable and Sustainable Energy Reviews 171: 112991.
[4] SHEHABI A., SMITH S., SARTOR D., BROWN R., HERRLIN M., KOOMEY J., MASANET E. and HORNER N. (2016) United States Data Center Energy Usage Report. Tech. Rep. LBNL-1005775, Lawrence Berkeley National Laboratory.
[5] KATAL A., DAHIYA S. and CHOUDHURY T. (2022) Energy efficiency in cloud computing data centers: a survey on software technologies. Cluster Computing 26(3): 1845–1875.
[6] MNIH V., KAVUKCUOGLU K., SILVER D. et al. (2015) Humanlevel control through deep reinforcement learning. Nature 518(7540): 529–533.
[7] WAN J., DUAN Y., GUI X., LIU C., LI L. and MA Z. (2023) SafeCool: Safe and Energy-Efficient Cooling Management in Data Centers With Model-Based Reinforcement Learning. IEEE Transactions on Emerging Topics in Computational Intelligence 7(6): 1621–1635.
[8] LI Y., WEN Y., GUAN K. and TAO D. (2019) Transforming Cooling Optimization for Green Data Center via Deep Reinforcement Learning. IEEE Transactions on Cybernetics 50(5): 2002–2013.
[9] RAN Y., HU H., ZHOU X. and WEN Y. (2023) Optimizing data center energy efficiency via event-driven deep reinforcement learning. IEEE Transactions on Services Computing 16(2): 1296–1309.
[10] ZOHDI T.I. (2022) A digital-twin and machine-learning framework for precise heat and energy management of datacenters. Computational Mechanics 69(6): 1501–1516.
[11] LU R., LI X., CHEN R., LEI A. and MA X. (2024) An Alternative Reinforcement Learning (ARL) control strategy for data center air-cooled HVAC systems. Energy 308: 132977.
[12] MOORE J., CHASE J., RANGANATHAN P. and SHARMA R. (2006) Making scheduling “cool”: Temperature-aware workload placement in data centers. In USENIX Annual Technical Conference: 61–74.
[13] LI Z., WANG H., FANG Q. and WANG Y. (2023) A data-driven subspace predictive control method for air-cooled data center thermal modelling and optimization. Journal of the Franklin Institute 360(5): 3657–3676.
[14] MAHBOD R., DHAR N.K. and WARDI Y. (2022) Energy saving evaluation of an energy efficient data center using a model-free reinforcement learning approach. Applied Energy 322: 119392.
[15] BELOGLAZOV A., ABAWAJY J. and BUYYA R. (2012) Energyaware resource allocation heuristics for efficient management of data centers for cloud computing. Future Generation Computer Systems 28(5): 755–768.
[16] ZHANG Y., ZHAO Y., DAI S., PAN X., KARIMIPOUR A. and NI J. (2022) Cooling technologies for data centres and telecommunication base stations—A comprehensive review. Journal of Cleaner Production 334: 130280.
[17] SMITH M., ZHAO L., CORDOVA J., JIANG X. and EBRAHIMI M. (2023) Energy-Efficient GPU-Intensive Workload Scheduling for Data Centers. In 2023 International Conference on Machine Learning and Applications (ICMLA): 1735–1740.
[18] CHI R., ZHANG Q., LU S. and ZHOU W. (2021) Cooperatively Improving Data Center Energy Efficiency Based on Multi- Agent Deep Reinforcement Learning. Energies 14(8): 2071.
[19] WANG Y., VELSWAMY K. and HUANG B. (2024) Green data center cooling control via physics-guided safe reinforcement learning. ACM Transactions on Cyber-Physical Systems 8(2): 1–26.
[20] BIEMANN M., GUNKEL P.A., SCHELLER F., HUANG L. and LIU X. (2023) Data Center HVAC Control Harnessing Flexibility Potential via Real-Time Pricing Cost Optimization Using Reinforcement Learning. IEEE Internet of Things Journal 10(15): 13876–13894.
[21] MU N., HU X., JIA Q.S., ZHU X. and HE X. (2024) Large- Scale Data Center Cooling Control via Sample-Efficient Reinforcement Learning. In 2024 IEEE 20th International Conference on Automation Science and Engineering (CASE): 2780–2785.
[22] KAHIL H., SHARMA S., VALISUO P. and ELMUSRATI M. (2025) Reinforcement learning for data center energy efficiency optimization: A systematic literature review and research roadmap. Applied Energy 389: 125734.
[23] ZHAO Z., FAN L. and HAN Z. (2024) Optimal Data Center Energy Management With Hybrid Quantum-Classical Multi- Cuts Benders’ Decomposition Method. IEEE Transactions on Sustainable Energy 15(2): 847–858.
[24] YANG D., WANG X., SHEN R. et al. (2024) Global optimization strategy of prosumer data center system operation based on multi-agent deep reinforcement learning. Journal of Building Engineering 91: 109519.
[25] ROSTAMI S., DOWN D.G. and KARAKOSTAS G. (2023) Thermalaware Workload Distribution for Data Centers with Demand Variations. In Proceedings of the 14th International Green and Sustainable Computing Conference: 63–66.
[26] HOGADE N., PASRICHA S. and SIEGEL H.J. (2022) Energy and Network Aware Workload Management for Geographically Distributed Data Centers. IEEE Transactions on Sustainable Computing 7(2): 400–413.
[27] WANG R., ZHANG X., ZHOU X., WEN Y. and TAN R. (2022) Toward Physics-Guided Safe Deep Reinforcement Learning for Green Data Center Cooling Control. In 2022 ACM/IEEE 13th International Conference on Cyber-Physical Systems (ICCPS): 159–169.
[28] MU N., HU X. and JIA Q.S. (2023) Integrating Mechanism and Data: Reinforcement Learning Based on Multi-Fidelity Model for Data Center Cooling Control. In 2023 China Automation Congress (CAC): 5283–5288.
[29] JI J., YU D., YANG D. et al. (2023) Predictive Control based on Transformer as Surrogate Model for Cooling System Optimization in Data Center. In 2023 International Conference on Mobile Internet, Cloud Computing and Information Security (MICCIS): 36–42.
[30] ZHU H. and LIN B. (2024) Digital twin-driven energy consumption management of integrated heat pipe cooling system for a data center. Applied Energy 373: 123840.
[31] LIAO H., TANG G., GUO D., WU K. and LUO L. (2024) EVAssisted Computing for Energy Cost Saving at Edge Data Centers. IEEE Transactions on Mobile Computing 23(9): 9029–9041.
[32] LIU W., YAN Y., SUN Y. et al. (2023) Online job scheduling scheme for low-carbon data center operation: An information and energy nexus perspective. Applied Energy 338: 120918.
[33] PATTERSON M.K. (2008) The effect of data center temperature on energy efficiency. In IEEE Intersociety Conference on Thermal and Thermomechanical Phenomena: 1167–1174.
[34] LIU X. (2024) Research on cooperative operation optimization method of internet data center alliance considering diversified flexibility. International Journal of Electrical Power & Energy Systems 162: 110223.
[35] SHARMA R.K., BASH C.E., PATEL C.D. and BEITELMAL M. (2005) Balance of power: Dynamic thermal management for Internet data centers. IEEE Internet Computing 9(1): 42–49.
[36] CRAWLEY D.B., LAWRIE L.K., WINKELMANN F.C. et al. (2001) EnergyPlus: creating a new-generation building energy simulation program. Energy and Buildings 33(4): 319–331.
[37] LILLICRAP T.P., HUNT J.J., PRITZEL A., HEESS N., EREZ T., TASSA Y., SILVER D. and WIERSTRA D. (2016) Continuous control with deep reinforcement learning. In International Conference on Learning Representations: 1–14.
[38] FUJIMOTO S., VAN HOOF H. and MEGER D. (2018) Addressing Function Approximation Error in Actor-Critic Methods. In International Conference on Machine Learning: 1587–1596.
[39] BLAD C., BGH S. and KALLESE C.S. (2022) Data-driven Offline Reinforcement Learning for HVAC-systems. Energy 261: 125290.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Youchun Qiu
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
This is an open-access article distributed under the terms of the Creative Commons Attribution CC BY 4.0 license, which permits unlimited use, distribution, and reproduction in any medium so long as the original work is properly cited.
