Q-обучаемая модель дискретного управления для системы накопления энергии с водородной установкой автономной гибридной электростанции железнодорожной подстанции

В статье рассмотрены перспективы создания автономных гибридных электростанций с использованием возобновляемых источников энергии и водорода в качестве систем хранения энергии, а также аккумуляторных батарей для системы электроснабжения железных дорог. Проведен сравнительный анализ различных систем хранения энергии. Создана имитационная модель, учитывающая характеристики поездов электроподвижного состава, модель контактной сети межстанционной зоны железнодорожного перегона и модель системы накопления энергии. Управление потоком электроэнергии в автономной гибридной электростанции требует учета прогнозов потребления и выработки электроэнергии, а также уровня заряда накопителя энергии. Для решения задачи синтеза алгоритма управления предлагается использование матричного Q-обучения. Новизна исследования заключается в предложенном подходе применения Q-обучения к рассматриваемой задаче на основе дискретизации входных и выходных параметров модели и верификации подхода на имитационной модели, построенной для реального участка железной дороги между Яей и станцией Ижморская (Кемеровская область РФ). Показано, что использование предложенной системы позволяет выровнять напряжение в сети между тяговыми подстанциями и обеспечить значительное увеличение пропускной способности участка железной дороги, а предложенный метод матричного Q-обучения позволяет синтезировать эффективный алгоритм управления системой накопления энергии в составе автономной гибридной электростанции.

1. Konakbaev R. A. Probabilistic analysis of power consumption on the example of railway transport. Intelligent Energy Systems: Proceedings of the IV International Youth Forum, 2016, P. 361-364.

2. Strategy of scientific and technical development of the holding «Russian Railways» for the period up to 2020 and for the future up to 2025. Moscow: JSC «Russian Railways», 2015.

3. Kotin D. A. et. al. Estimation of parameters of electricity storage devices for traction substations. Bulletin of MPEI: theoretical and scientific-practical journal, 2022, no. 5. – P. 11-19.

4. Xuan Liu, Kang Li. Energy Storage Devices in Electrified Railway Systems: a Review. Transportation Safety and Environment, 2020, vol. 2(3). – P. 183-201.

5. Ghaviha N. et. al. Review of Application of Energy Storage Devices in Railway Transportation. Energy Proc. 2017, vol. 105. – P. 4561-456

6. Ratniyomchai T., Hillmansen S., Tricoli P. Recent Developments and Applications of Energy Storage Devices in Electrified Railways. IET Electrical Systems in Transportation, 2014, vol. 4(1). – P. 9-20.

7. Zakaryukin V. P., Kryukov A. V., Cherepanov A. V. Use of energy storage devices in railway power supply systems. Electrical equipment: operation and repair, 2016, no. 8. – P. 20-25.

8. Mitrofanov S. V., Armeev D. V., Domahin E. A. Analysis of the Impact of Autonomous Hybrid Power Plants on the Railways Capacity. 2021 XV International Scientific-Technical Conference on Actual Problems of Electronic Instrument Engineering (APEIE), 2021. – P. 161-165.

9. Feng J., Chu W.Q., Zhang Z., Zhu Z. Q. Power Electronic Transformer-Based Railway Traction Systems: Challenges and Opportunities. IEEE Journal of Emerging and Selected Topics in Power Electronics, 2017, vol. 5(3). – P. 1237-1253.

10. Pouget J., Guo B., Bossoney L., Coppex J., Roggo D., Ellert C. Energetic simulation of DC railway micro-grid interconnecting with PV solar panels, EV charger infrastructures and electrical railway network. 2020 IEEE Vehicle Power and Propulsion Conference (VPPC), 2020.

11. Mitrofanov S. V., Kiryanova N. G., Gorlova A. M. Stationary Hybrid Renewable Energy Systems for Railway Electrification: A Review. Energies. 2021, vol. 14(18), id. 5946.

12. Cheremisin V. T., Nezevak V. L., Shatokhin A. P. Increasing the energy efficiency of the traction power supply system in the conditions of operation of sectioning posts with electrical energy storage devices. News of Tomsk Polytechnic University. Series «Engineering of Georesources», 2015, vol. 326, no. 10. – P. 54-64.

13. Rybalko A. Ya., Dybrin S. V. Selection of energy storage capacity to ensure a reduction in maximum power consumption. Mining Information and Analytical Bulletin, 2008, no. 8. – P. 356-361.

14. Titova T. S., Evstafiev A. M. Increasing the energy efficiency of locomotives with energy storage devices. News of the St. Petersburg University of Transport. – 2017, no. 2. – P. 200-210.

15. Baluev D. Yu., Zyryanov V. M., Kiryanova N. G., Prankevich G. A. Methodology for calculating the main parameters of an energy storage device using experimental load diagrams. Bulletin of the Irkutsk State University. technical university. 2018, vol. 22, no. 5(136). – P. 105-114.

16. Zenith F., Isaac R., Hoffrichter A., Thomassen M. S., Møller-Holst S. Techno-economic analysis of freight railway electrification by overhead line, hydrogen and batteries: Case studies in Norway and USA. Proc. Inst. Mech. Eng. Part F. J. Rail Rapid Transit-IMECHE, 2019, vol. 234, no 7. – P. 791-802.

17. Herwartz S., Pagenkopf J., Streuling C. Sector coupling potential of wind-based hydrogen production and fuel cell train operation in regional rail transport in Berlin and Brandenburg. Int. J. Hydrogen Energy – Pergamon, 2021, vol. 46, no 57. – P. 29597-29615.

18. Dong W., Li Y., Xiang J. Optimal Sizing of a Stand-Alone Hybrid Power System Based on Battery/ Hydrogen with an Improved Ant Colony Optimization. Energies, 2016, vol. 9, no. 10. – P. 785.

19. Dawood F., Shafiullah G. M., Anda M. Stand- alone microgrid with 100% renewable energy: A case study with hybrid solar PV-battery-hydrogen. Sustain, 2020, vol. 12, no. 5, 2047.

20. Li C. H., Zhu X. J., Cao G. Y., Sui S., Hu M. R. Dynamic modeling and sizing optimization of stand-alone photovoltaic power systems using hybrid energy storage technology. Renew. Energy – Pergamon, 2009, vol. 34, no. 3. – P. 815-826.

21. Nordin N. D., Rahman H. A. Comparisonof optimum design, sizing, and economic analysis of standalone photovoltaic/battery without and with hydrogen production systems. Renew. Energy – Pergamon, 2019, vol. 141. – P. 107-123.

22. Marocco P., Ferrero D., Lanzini A., Santarelli M. Optimal design of stand-alone solutions based on RES + hydrogen storage feeding off-grid communities. Energy Convers. Manag. – Pergamon, 2021, vol. 238. – P. 114147.

23. Rusdianasari R., Bow Y., Dewi T., Risma P. Hydrogen Gas Production Using Water Electrolyzer as Hydrogen Power. 2019 International Conference on Electrical Engineering and Computer Science (ICECOS), 2019. – P. 127-131.

24. Koiwa K., Umemura A., Takahashi R., Tamura J. Stand-alone hydrogen production system composed of wind generators and electrolyzer. IECON 2013 – 39th Annual Conference of the IEEE Industrial Electronics Society, 2013. – P. 1873-1879.

25. Yoshida Y., Umemura A., Takahashi R., Tamura J. Experimental study of hydrogen production system with stand-alone wind power generators. 2015 Tenth International Conference on Ecological Vehicles and Renewable Energies (EVER), 2015. – P. 1-5.

26. Koiwa K. et al. A Coordinated Control Method for Integrated System of Wind Farm and Hydrogen Production: Kinetic Energy and Virtual Discharge Controls. IEEE Access, 2022, vol. 10. – P. 28283-28294.

27. Itani N., Moubayed N. Regression Model for a Proton Exchange Membrane Electrolyzer. 2022 International Conference on Smart Systems and Power Management (IC2SPM), 2022. – P. 64-68.

28. Kolhe M., Atlam O. Empirical electrical modeling for a proton exchange membrane electrolyzer. 2011 International Conference on Applied Superconductivity and Electromagnetic Devices, 2011. – P. 131-134.

29. Pirom W., Srisiriwat A. Electrical Energy- Based Hydrogen Production via PEM Water Electrolysis for Sustainable Energy. 2022 International Electrical Engineering Congress (iEECON), 2022. – P. 1-4.

30. Chang W. J., Lee K. -H., Ha J. -I., Nam K. T. Hydrogen Production via Water Electrolysis: The Benefits of a Solar Cell-Powered Process. IEEE Electrification Magazine, 2018, vol. 6, no. 1. – P. 19-25.

31. Costa Lopes F. da, Watanabe E. H. Experimental and theoretical development of a PEM electrolyzer model applied to energy storage systems. 2009 Brazilian Power Electronics Conference, 2009. – P. 775-782.

32. Török L. et al. High efficiency electrolyser power supply for household hydrogen production and storage systems. 2015 17th European Conference on Power Electronics and Applications (EPE’15 ECCE- Europe), 2015. – P. 1-9.

33. Marefatjouikilevaee H., Auger F., Olivier J. -C. Static and Dynamic Electrical Models of Proton Exchange Membrane Electrolysers: A Comprehensive Review. Energies, 2023, vol. 16, no. 18, art. 6503.

34. Maeda T., Nagata Y., Endo N., Ishida M. Effect of water electrolysis temperature of hydrogen production system using direct coupling photovoltaic and water electrolyzer. Journal of International Council on Electrical Engineering, 2016, vol. 6(1). – P. 78-83.

35. Nair A. S. et al. Design and Development of an Efficient Electrolyser for Low-Cost Hydrogen Production. 2023 International Conference on Circuit Power and Computing Technologies (ICCPCT), 2023. – P. 307-311.

36. Buttler A., Spliethoff H. Current status of water electrolysis for energy storage, grid balancing and sector coupling via power-to-gas and power-to-liquids: A review. Renew. Sustain. Energy Rev, 2018, vol. 82. – P. 2440-2454.

37. Santarelli M., Call M., Macagno S. Design and analysis of stand-alone hydrogen energy systems with different renewable sources. Int. J. Hydrogen Energy – Pergamon, 2004, vol. 29, no. 15. – P. 1571-1586.

38. Paul B., Andrews J. Optimal coupling of PV arrays to PEM electrolysers in solar–hydrogen systems for remote area power supply. Int. J. Hydrogen Energy – Pergamon, 2008, vol. 33, no. 2. – P. 490-498.

39. Li X. U., Guang M. A., Peng Sheng. Overview of Hydrogen storage technologies and their application prospects in hydrogen-based energy storage. Smart Grid, 2016, vol. 4, no. 2. – P. 166-171.

40. Wang C. et al. Optimal Configuration of Hydrogen Storage System and Hydrogen Supply Chain Equipment for Regional Integrated Energy System. 2020 IEEE 3rd Student Conference on Electrical Machines and Systems (SCEMS), 2020. – P. 976-981.

41. Zhu Y. From Hydrogen Production to Storage: A Process for Sustainable Development of Underground Hydrogen Storage in Caverns. 2021 3rd International Academic Exchange Conference on Science and Technology Innovation (IAECST), 2021. – P. 1658-1663.

42. Zhang Y. H. et al. Development and application of hydrogen storage. Journal of Iron and Steel Research International, 2015, vol. 22, no. 9. – P. 757-770.

43. Ozarslan A. Large-scale hydrogen energy storage in salt caverns. International journal of hydrogen energy, 2012, vol. 37, no. 19. – P. 14265-14277.

44. Elberry A. M., Thakur J., Santasalo-Aarnio A., Larmi M. Large-scale compressed hydrogen storage as part of renewable electricity storage systems. International Journal of Hydrogen Energy, 2021.

45. Zivar D., Kumar S., Foroozesh J. Underground hydrogen storage: A comprehensive review. International Journal of Hydrogen Energy, 2021, vol. 46, no. 45. – P. 23436-23462.

46. Chvalek R., Moldrik P. Hydrogen storage methods in the fuel cells laboratory. 2011 10thInternational Conference on Environment and Electrical Engineering, 2011. – P. 1-3.

47. Ondrejička K., Putala R., Mikle D. Fuel cells as backup power supply for production processes. 2022 Cybernetics & Informatics (K&I), 2022. – P. 1-6.

48. Taieb A., Mukhopadhyay S., Al-Othman A. Dynamic Model of a Proton-Exchange Membrane Fuel Cell using Equivalent Electrical Circuit. 2019 Advances in Science and Engineering Technology International Conferences (ASET), 2019. – P. 1-4.

49. Mann R. F. et al. Development and application of a generalised steady-state electrochemical model for a PEM fuel cell. J. Power Sources, 2000, vol. 86, no. 1. – P. 173-180.

50. Jiang R., Chu D. Voltage-time behavior of a polymer electrolyte membrane fuel cell stack at constant current discharge. J. Power Sources, 2001, vol. 92, no. 1. – P. 193-198.

51. Yerramalla S., Davari A., Feliachi A., Biswas T. Modeling and simulation of the dynamic behavior of a polymer electrolyte membrane fuel cell. J. Power Sources, 2003, vol. 124, no. 1. – P. 104-113.

52. Yu D., Yuvarajan S. Electronic circuit model for proton exchange membrane fuel cells. J. Power Sources, 2005, vol. 142, no. 1. – P. 238-242.

53. Sujit Sopan Barhate, Rohini Mudhalwadkar. Electrical Equivalent Model for Proton Exchange Membrane Fuel Cell Useful in On-Board Applications. Advances in Clean Energy Technologies, 2021. – P. 649.

54. Sridhar A. N., et al. Solid Oxide Fuel Cell Interfaced with a Three-Phase Power System. 2023 International Conference on Sustainable Communication Networks and Application (ICSCNA), 2023. – P. 697-702.

55. Auld Allie E. et al. Applications of one-cycle control to improve the interconnection of a solid oxide fuel cell and electric power system with a dynamic load. Journal of Power Sources, 2008, vol. 1. – P. 155-163.

56. Barelli Linda, Gianni Bidini, Giovanni Cinti. Operation of a solid oxide fuel cell based power system with ammonia as a fuel: Experimentaltest and system design. Energies, 2020, vol. 23.

57. Salam A. A., Hannan M. A., Mohamed A. Dynamic modeling and simulation of Solid Oxide Fuel Cell system. 2008 IEEE 2nd International Power and Energy Conference, 2008. – P. 813-818.

58. Rechberger J. et al. High Power Density Fuel Cell Systems for Commercial Vehicles. 2023 IEEE International Transportation Electrification Conference (ITEC-India), 2023. – P. 1-4.

59. Chaudhary S., Chauhan Y.K. Studies and performance investigations on fuel cells. 2014 International Conference on Advances in Engineering & Technology Research (ICAETR – 2014), 2014. – P. 1-6.

60. Ali D. M., Salman S. K. A Comprehensive Review of the Fuel Cells Technology and Hydrogen Economy. Proceedings of the 41st International Universities Power Engineering Conference, 2006. – P. 98-102.

61. Salodkar P., Meshram S., Dey S. Review of Hydrogen Fuel Cell as an emergent field in Green Technology. 2022 Second International Conference on Power, Control and Computing Technologies (ICPC2T), 2022. – P. 1-6.

62. Jarosław Milewski et al. Structural Investigation of Orthoborate-Based Electrolytic Materials for Fuel Cell Applications. Energies, 2024, vol. 17, no. 9. – P. 2097.

63. Guaitolini S. V. M. et al. A review of fuel cell and energy cogeneration technologies. 2018 9th International Renewable Energy Congress (IREC), 2018. – P. 1-6.

64. Cigolotti V., Genovese M., Fragiacomo P. Comprehensive Review on Fuel Cell Technology for Stationary Applications as Sustainable and Efficient Poly- Generation Energy Systems. Energies, 2021, vol. 14. – P. 4963.

65. Wang Y. et al. Fundamentals, materials, and machine learning of polymer electrolyte membrane fuel cell technology. Energy AI – Elsevier, 2020, vol. 1. – P. 100014.

66. Bauen A., Hart D., Chase A. Fuel cells for distributed generation in developing countries – an analysis. Int. J. Hydrogen Energy – Pergamon, 2003, vol. 28, no. 7. – P. 695-701.

67. Nallaperuma S., Fletcher D., Harrison R. Optimal control and energy storage for DC electric train systems using evolutionary algorithms. Railway Engineering Science, 2021, vol. 29. – P. 327-335.

68. Xia H., Chen H., Yang Z., Lin F., Wang B. Optimal energy management location and size for stationary energy storage system in a metro line based on genetic algorithm. Energies, 2015, vol. 8(10). – P. 11618-11640.

69. Luo P. et al. Multi-application strategy based on railway static power conditioner with energy storage system. IEEE Transactions on Intelligent Transportation Systems, 2021, vol. 22 (4).

70. Şengör I. et. al. Energy Management of a Smart Railway Station Considering Regenerative Braking and Stochastic Behaviour of ESS and PV Generation. IEEE Transactions on Sustainable Energy, 2018, vol. 9(3), pp. 1041-1050.

71. Deng W., Dai C. A multifunctional energy storage system with fault-tolerance and its hierarchical optimization control in AC-fed railways. IEEE Transactions on power delivery, 2022, vol. 37(4). – P. 2440-2452.

72. Zhu F., Yang Z., Lin F., Xin Y. Decentralized Cooperative Control of Multiple Energy Storage Systems in Urban Railway Based on Multiagent DeepReinforcement Learning. IEEE Transactions on Power Electronics, 2020, vol. 35(9). – P. 9368-9379.

73. Gelfand I. M., Pyatetsky-Shapiro I. I., Tsetlin M. L. About some classes of games and slot machines. Report of the USSR Academy of Sciences, 1963, vol. 152, no. 4. – P. 845-848.

74. Zhang D., Han X., Deng C. Review on the research and practice of deep learning and reinforcement learning in smart grids. CSEE J. Power Energy Syst., 2018, vol. 4(3). – P. 362-370.

75. Nagabandi A., Kahn G., Fearing R. S., Levine S. Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning. In Proc. IEEE Int. Conf. Robot. Automat., 2018. – P. 7559-7566.

76. Jang B., Kim M., Harerimana G., Kim J. W. Q-Learning Algorithms: A Comprehensive Classification and Applications. IEEE Access, 2019, vol. 7. – P. 133653-133667.

77. Manusov V. Z., Khasanzoda N., Matrenin P. V. Application of artificial intelligence methods to energy management in Smart Grids. Novosibirsk: NSTU Publishing House, 2019. 240 p.

78. Yu Y., Cai Z., Huang Y. Energy Storage Arbitrage in Grid-Connected Micro-Grids Under Real- Time Market Price Uncertainty: A Double-Q Learning Approach. IEEE Access, 2020, vol. 8. – P. 54456-54464.

79. Bui V. -H., Hussain A., Kim H. -M. Double Deep Q-Learning-Based Distributed Operation of Battery Energy Storage System Considering Uncertainties. IEEE Transactions on Smart Grid, 2020, vol. 11(1). – P. 457-469.

80. Hu R., Kwasinski A. Q-learning Trained Virtual Inertial Control Strategy to Improve Battery Life in Microgrids Powered by Wind Turbines. 2022 IEEE 13th International Symposium on Power Electronics for Distributed Generation Systems (PEDG), 2022.

81. Matrenin P. V., Rusina A. G., Kyrianova N. G., Sergeev N. N. Short-Term Wind Speed Forecasting for an Autonomous Hybrid Power Plant of a Traction Railway Substation. In Proc of IEEE 23 International Conference of Young Professionals in Electron Devices and Materials (EDM), 2022. – P. 411-415.

82. Matrenin P. V. et. al. Improving of the Generation Accuracy Forecasting of Photovoltaic Plants Based on k-Means and k-Nearest Neighbors Algorithms. Energetika. Proceedings of CIS Higher Education Institutions and Power Engineering Associations, 2023, vol. 66 (4). – P. 305-321.

83. Mitrofanov S. V., Kiryanova N. G., Zyryanov V. M. Analysis of the Renewable Energy Sources Application Efficiency in the Composition of Autonomous Hybrid Power Plants of Railway Transport Electrification Systems. In Proc of IEEE 2nd International Conference Problems of Informatics, Electronics and Radio Engineering (PIERE), 2022.