Journal of Soft Computing and Information Technology

Journal of Soft Computing and Information Technology

A Markov Model-based and Deep Learning-based Electric Vehicle Charging Scheduling in Charge Stations

Document Type : Persian Original Article

Authors
Samane Mohammadi 1
Mojtaba Valinataj 2
Fatemeh Zamani 1
1 Faculty of Electrical and Computer Eng. Babol Noshirvani University of Tech. Babol, Iran.
2 Computer Dept., ECE, Babol Noshirvani Univ. of Tech., Babol, Iran
10.22034/jscit.2026.576052.2229
Abstract
The large-scale deployment and expansion of electric vehicles (EVs) significantly contribute to reducing fossil fuel consumption and environmental pollution. In these vehicles, optimal charging scheduling not only enables more efficient utilization of renewable energy resources, but also reduces charging costs and enhances power grid stability. However, fluctuations in electricity prices, variations in energy demand and charging power, and uncertainties regarding battery charging completion time pose major challenges for accurate prediction and optimal charging scheduling. In this paper, the EV charging scheduling problem is formulated as a Markov Decision Process (MDP), and a model-free Deep Reinforcement Learning (DRL)-based approach is proposed to derive an optimal charging strategy that supports real-time decision-making under uncertainty. The proposed model is trained and evaluated using real-world data from California. The results demonstrate that the proposed approach significantly reduces electricity costs while improving charging performance under uncertain conditions and volatile electricity prices. Compared to the baseline method, the proposed strategy achieves a 41% reduction in electricity costs, and compared to conventional methods, it yields higher cumulative rewards and lower cumulative charging costs, indicating its effectiveness and practical applicability for users at charging stations.
Keywords
Electric Vehicle
Charging Scheduling
Charging Station
Long Short-Term Memory (LSTM)
Deep Reinforcement Learning
 
[1] O. Sadeghian, A. Oshnoei, B. Mohammadi-Ivatloo, V. Vahidinasab, and A. Anvari-Moghaddam, "A comprehensive review on electric vehicles smart charging: Solutions, strategies, technologies, and challenges," Journal of Energy Storage, vol. 54, p. 105241, 2022. 10.1016/j.est.2022.105241.
[2] R. Zai, Y. Guo, Q. Liu, H. Sun, Q. Wu, and L. Xiao, "Knowledge Assisted Deep Reinforcement Learning for Electric Vehicle Charging Control," in 2022 IEEE 6th Conference on Energy Internet and Energy System Integration (EI2), 2022: IEEE, pp. 1882-1887. DOI: 10.1109/EI256261.2022.10116271.
[3] L. Yan, X. Chen, J. Zhou, Y. Chen, and J. Wen, "Deep reinforcement learning for continuous electric vehicles charging control with dynamic user behaviors," IEEE Transactions on Smart Grid, vol. 12, no. 6, pp. 5124-5134, 2021. DOI: 10.1109/TSG.2021.3098298.
[4] W. Kong, Z. Y. Dong, Y. Jia, D. J. Hill, Y. Xu, and Y. Zhang, "Short-term residential load forecasting based on LSTM recurrent neural network," IEEE transactions on smart grid, vol. 10, no. 1, pp. 841-851, 2017. DOI: 10.1109/TSG.2017.2753802.
[5] Z. Wan, H. Li, H. He, and D. Prokhorov, "Model-free real-time EV charging scheduling based on deep reinforcement learning," (in eng), IEEE Transactions on Smart Grid, vol. 10, no. 5, pp. 5246-5257, 2018. DOI: 10.1109/TSG.2018.2879572.
[6] C. Zhang, Y. Liu, F. Wu, B. Tang, and W. Fan, "Effective charging planning based on deep reinforcement learning for electric vehicles," (in eng), IEEE Transactions on Intelligent Transportation Systems, vol. 22, no. 1, pp. 542-554, 2020. DOI: 10.1109/TITS.2020.3002271.
[7] A. Alsabbagh, H. Yin, and C. Ma, "Distributed electric vehicles charging management with social contribution concept," IEEE Transactions on Industrial Informatics, vol. 16, no. 5, pp. 3483-3492, 2019. DOI: 10.1109/TII.2019.2950460.
[8] Q. Xiao et al., "A deep reinforcement learning based charging and discharging scheduling strategy for electric vehicles," Energy Reports, vol. 12, pp. 4854-4863, 2024. 10.1016/j.egyr.2024.10.056.
[9] F. Neumann, "Optimal Scheduling of Electric Vehicle Charging in Distribution Networks," Technische Universität Berlin, Technische Universität Berlin, 2017.   DOI: 10.13140/RG.2.2.13347.96805.
[10] R. Adam, K. Qian, and R. Brehm, "Electric vehicle user behavior prediction using Gaussian mixture models and soft information," in 2021 IEEE PES Innovative Smart Grid Technologies-Asia (ISGT Asia), 2021: IEEE, pp. 1-5. DOI: 10.1109/ISGTAsia49270.2021.9715580.
[11] A. Orzechowski, L. Lugosch, H. Shu, R. Yang, W. Li, and B. H. Meyer, "A data-driven framework for medium-term electric vehicle charging demand forecasting," (in eng), Energy and AI, vol. 14, p. 100267, 2023. 10.1016/j.egyai.2023.100267.
[12] H. Li, Z. Wan, and H. He, "Constrained EV charging scheduling based on safe deep reinforcement learning," (in eng), IEEE Transactions on Smart Grid, vol. 11, no. 3, pp. 2427-2439, 2019. DOI: 10.1109/TSG.2019.2955437.
[13] R. Luo et al., "Deep learning approach for long-term prediction of electric vehicle (ev) charging station availability," in 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), 2021: IEEE, pp. 3334-3339. DOI:  10.1109/ITSC48978.2021.9564633.
[14] F. Mohammad, D.-K. Kang, M. A. Ahmed, and Y.-C. Kim, "Energy demand load forecasting for electric vehicle charging stations network based on convlstm and biconvlstm architectures," IEEE Access, vol. 11, pp. 67350-67369, 2023. DOI:  10.1109/ACCESS.2023.3274657.
[15] S. Wang, A. Chen, P. Wang, and C. Zhuge, "Predicting electric vehicle charging demand using a heterogeneous spatio-temporal graph convolutional network," (in eng), Transportation Research Part C: Emerging Technologies, vol. 153, p. 104205, 2023. DOI: 10.1016/j.trc.2023.104205.
[16] J. Jin, S. Mao, and Y. Xu, "Optimal priority rule-enhanced deep reinforcement learning for charging scheduling in an electric vehicle battery swapping station," IEEE Transactions on Smart Grid, vol. 14, no. 6, pp. 4581-4593, 2023. DOI:  10.1109/TSG.2023.3250505.
[17] M. Boulakhbar, M. Farag, K. Benabdelaziz, M. Zazi, M. Maaroufi, and T. Kousksou, "Electric vehicles arrival and departure time prediction based on deep learning: the case of Morocco," in 2022 2nd International Conference on Innovative Research in Applied Science, Engineering and Technology (IRASET), 2022: IEEE, pp. 1-8. DOI:  10.1109/IRASET52964.2022.9738115.
[18] T. Qi, C. Ye, Y. Zhao, L. Li, and Y. Ding, "Deep reinforcement learning based charging scheduling for household electric vehicles in active distribution network," Journal of Modern Power Systems and Clean Energy, vol. 11, no. 6, pp. 1890-1901, 2023. DOI:  10.35833/MPCE.2022.000456
[19] A. Zhang, Q. Liu, J. Liu, and L. Cheng, "CASA: cost-effective EV charging scheduling based on deep reinforcement learning," Neural Computing and Applications, vol. 36, no. 15, pp. 8355-8370, 2024. 10.1007/s00521-024-09530-3.
[20] S. Li et al., "Electric vehicle charging management based on deep reinforcement learning," (in eng), Journal of Modern Power Systems and Clean Energy, vol. 10, no. 3, pp. 719-730, 2021. DOI:  10.35833/MPCE.2020.000460.
[21] S. Mishra, A. Choubey, S. V. Devarasetty, N. Sharma, and R. Misra, "An innovative multi-head attention model with BiMGRU for real-time electric vehicle charging management through deep reinforcement learning," Cluster Computing, vol. 27, no. 7, pp. 9993-10023, 2024. 10.1007/s10586-024-04494-4.
[22] R. Bellman, Dynamic Programming. Princeton, NJ: Princeton University Press, 1958.
[23] OASIS. California independent system operator: oasis.caiso.com/mrioasis/logon.do.
[24] F. Zhang, Q. Yang, and D. An, "CDDPG: A deep-reinforcement-learning-based approach for electric vehicle charging control," IEEE Internet of Things Journal, vol. 8, no. 5, pp. 3075-3087, 2020.  DOI:  10.1109/JIOT.2020.3015204.
[25] S. Wang, S. Bi, and Y. A. Zhang, "Reinforcement learning for real-time pricing and scheduling control in EV charging stations," (in eng), IEEE Transactions on Industrial Informatics, vol. 17, no. 2, pp. 849-859, 2019. DOI: 10.1109/TII.2019.2950809.
[26] L. Douaidi, S.-M. Senouci, I. El Korbi, and F. Harrou, "Predicting Electric Vehicle Charging Stations Occupancy: A Federated Deep Learning Framework," in 2023 IEEE 97th Vehicular Technology Conference (VTC2023-Spring), 2023: IEEE, pp. 1-5. DOI: 10.1109/VTC2023-Spring57618.2023.10199832.
[27] A. S. Khwaja, B. Venkatesh, and A. Anpalagan, "Short-term individual electric vehicle charging behavior prediction using long short-term memory networks," in 2020 IEEE 25th International Workshop on Computer Aided Modeling and Design of Communication Links and Networks (CAMAD), 2020: IEEE, pp. 1-7. DOI:  10.1109/CAMAD50429.2020.9209296.
[28] Z. Wan, H. Li, H. He, and D. Prokhorov, "A data-driven approach for real-time residential EV charging management," in 2018 IEEE Power & Energy Society General Meeting (PESGM), 2018: IEEE, pp. 1-5. DOI:  10.1109/PESGM.2018.8585945.
 
Journal of Soft Computing and Information Technology
Volume 15, Issue 2
Summer - August
Summer 2026
Pages 78-91