Robust Warehouse AGV Navigation in Dynamic Environments Using Soft Actor-Critic Reinforcement Learning

Autonomous guided vehicles (AGVs) operating in modern warehouse environments face the critical challenge of navigating safely and efficiently amid high-density dynamic obstacles, including human workers and peer vehicles. Classical reactive planners such as the Dynamic Window Approach (DWA) fail under these conditions due to their inability to predict obstacle motion, while conventional deep reinforcement learning methods often overfit to training layouts and lack the behavioral diversity needed for robust generalization. This paper presents a Soft Actor-Critic (SAC) based navigation framework that addresses these limitations through maximum-entropy reinforcement learning combined with a multicomponent reward design and domain-randomized training. The proposed method jointly optimizes goalreaching efficiency, collision avoidance, energy consumption, and trajectory smoothness within a unified learning objective. We evaluate the approach against three strong baselines—DDPG, TD3, and DWA— across four structurally distinct warehouse layouts, three of which are unseen during training. Experimental results demonstrate that SAC achieves the highest mean cumulative reward (99.32 vs. 82.18 for DDPG, 88.58 for TD3, and 55.41 for DWA), the lowest collision rate (1.5 TD3), and the shortest paths (15.54 m average vs. 22.17 m for DDPG and 25.54 m for TD3). Notably, SAC exhibits near-invariant path lengths across all layouts with a cross-layout standard deviation of only 0.22 m, providing compelling evidence of geometry-agnostic navigation. Energy consumption is approximately half that of DDPG and TD3, while trajectory smoothness is improved by a factor of two. These results establish that entropy-regularized deep reinforcement learning, coupled with principled reward shaping and domain randomization, produces warehouse navigation policies that are substantially more robust, efficient, and deployment-ready than both classical planners and deterministic actor-critic alternatives.

Soft Actor-Critic, autonomous guided vehicle navigation, dynamic obstacle avoidance, domain randomization, maximum-entropy reinforcement learning, warehouse robotics, energy-efficient navigation

1. Alahi, Alexandre, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Silvio Savarese. 2016. “Social LSTM: Human Trajectory Prediction in Crowded Spaces.” 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 961–71. https://doi.org/10.1109/CVPR.2016.110. [Google Scholar] [Crossref]

2. Andrychowicz, Marcin, Bowen Baker, Maciek Chociej, et al. 2018. “Learning Dexterous in-Hand Manipulation.” The International Journal of Robotics Research 39: 20–23. https://api.semanticscholar.org/CorpusID:51894399. [Google Scholar] [Crossref]

3. Dijkstra, E. W. 1959. “A Note on Two Problems in Connexion with Graphs.” Numerische Mathematik 1 (1): 269–71. https://doi.org/10.1007/BF01386390. [Google Scholar] [Crossref]

4. Duan, Yan, Xi Chen, Rein Houthooft, John Schulman, and P. Abbeel. 2016. “Benchmarking Deep Reinforcement Learning for Continuous Control.” International Conference on Machine Learning. https://api.semanticscholar.org/CorpusID:12296499. [Google Scholar] [Crossref]

5. Fox, D., W. Burgard, and S. Thrun. 1997. “The Dynamic Window Approach to Collision Avoidance.” IEEE Robotics and Automation Magazine 4 (1): 23–33. https://doi.org/10.1109/100.580977. [Google Scholar] [Crossref]

6. Fragapane, Giuseppe, René de Koster, Fabio Sgarbossa, and Jan Ola Strandhagen. 2021. “Planning and Control of Autonomous Mobile Robots for Intralogistics: Literature Review and Research Agenda.” European Journal of Operational Research 294 (2): 405–26. https://doi.org/https://doi.org/10.1016/j.ejor.2021.01.019. [Google Scholar] [Crossref]

7. Fujimoto, Scott, Herke van Hoof, and David Meger. 2018. Addressing Function Approximation Error in Actor-Critic Methods. https://arxiv.org/abs/1802.09477. [Google Scholar] [Crossref]

8. Guizzo, Eric. 2008. “Three Engineers, Hundreds of Robots, One Warehouse.” IEEE Spectrum 45 (7): 26–34. https://doi.org/10.1109/MSPEC.2008.4547508. [Google Scholar] [Crossref]

9. Haarnoja, Tuomas, Aurick Zhou, Pieter Abbeel, and Sergey Levine. 2018. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. https://arxiv.org/abs/1801.01290. [Google Scholar] [Crossref]

10. Haarnoja, Tuomas, Aurick Zhou, Kristian Hartikainen, et al. 2019. Soft Actor-Critic Algorithms and Applications. https://arxiv.org/abs/1812.05905. [Google Scholar] [Crossref]

11. Hart, Peter E., Nils J. Nilsson, and Bertram Raphael. 1968. “A Formal Basis for the Heuristic Determination of Minimum Cost Paths.” IEEE Transactions on Systems Science and Cybernetics 4 (2): 100–107. https://doi.org/10.1109/TSSC.1968.300136. [Google Scholar] [Crossref]

12. Helbing, Dirk, and Péter Molnár. 1995. “Social Force Model for Pedestrian Dynamics.” Physical Review. E, Statistical Physics, Plasmas, Fluids, and Related Interdisciplinary Topics 51 5: 4282–86. https://api.semanticscholar.org/CorpusID:5771125. [Google Scholar] [Crossref]

13. Henderson, Peter, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. 2017. “Deep Reinforcement Learning That Matters.” AAAI Conference on Artificial Intelligence. https://api.semanticscholar.org/CorpusID:4674781. [Google Scholar] [Crossref]

14. Jian, Zhu, Zihong Yan, Xuanang Lei, et al. 2022. “Dynamic Control Barrier Function-Based Model Predictive Control to Safety-Critical Obstacle-Avoidance of Mobile Robot.” 2023 IEEE International Conference on Robotics and Automation (ICRA), 3679–85. https://api.semanticscholar.org/CorpusID:252367490. [Google Scholar] [Crossref]

15. Kadian, Abhishek, Joanne Truong, Aaron Gokaslan, et al. 2019. “Sim2Real Predictivity: Does Evaluation in Simulation Predict Real-World Performance?” IEEE Robotics and Automation Letters 5: 6670–77. https://api.semanticscholar.org/CorpusID:221082834. [Google Scholar] [Crossref]

16. Karaman, Sertac, and Emilio Frazzoli. 2011. Sampling-Based Algorithms for Optimal Motion Planning. https://arxiv.org/abs/1105.1186. [Google Scholar] [Crossref]

17. Lackner, Thorge, Julian Hermann, Christian Kuhn, and Daniel Palm. 2024. “Review of Autonomous Mobile Robots in Intralogistics: State-of-the-Art, Limitations and Research Gaps.” Procedia CIRP. https://api.semanticscholar.org/CorpusID:274360266. [Google Scholar] [Crossref]

18. LaValle, Steven M. 1998. “Rapidly-Exploring Random Trees : A New Tool for Path Planning.” The Annual Research Report. https://api.semanticscholar.org/CorpusID:14744621. [Google Scholar] [Crossref]

19. LaValle, Steven M. 2006. “Planning Algorithms.” https://api.semanticscholar.org/CorpusID:15371216. [Google Scholar] [Crossref]

20. Lillicrap, Timothy P., Jonathan J. Hunt, Alexander Pritzel, et al. 2015. “Continuous Control with Deep Reinforcement Learning.” arXiv: Learning. https://api.semanticscholar.org/CorpusID:16326763. [Google Scholar] [Crossref]

21. Mnih, Volodymyr, Koray Kavukcuoglu, David Silver, et al. 2015. “Human-Level Control Through Deep Reinforcement Learning.” Nature 518: 529–33. https://api.semanticscholar.org/CorpusID:205242740. [Google Scholar] [Crossref]

22. Ng, A., Daishi Harada, and Stuart J. Russell. 1999. “Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping.” International Conference on Machine Learning. https://api.semanticscholar.org/CorpusID:5730166. [Google Scholar] [Crossref]

23. Peng, Xue Bin, Marcin Andrychowicz, Wojciech Zaremba, and P. Abbeel. 2017. “Sim-to-Real Transfer of Robotic Control with Dynamics Randomization.” 2018 IEEE International Conference on Robotics and Automation (ICRA), 1–8. https://api.semanticscholar.org/CorpusID:3707478. [Google Scholar] [Crossref]

24. Rösmann, Christoph, Wendelin Feiten, Thomas Wösch, Frank Hoffmann, and Torsten Bertram. 2012. “Trajectory Modification Considering Dynamic Constraints of Autonomous Robots.” German Conference on Robotics. https://api.semanticscholar.org/CorpusID:2043153. [Google Scholar] [Crossref]

25. Schulman, John, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal Policy Optimization Algorithms. https://arxiv.org/abs/1707.06347. [Google Scholar] [Crossref]

26. Sutton, Richard S., David A. McAllester, Satinder Singh, and Y. Mansour. 1999. “Policy Gradient Methods for Reinforcement Learning with Function Approximation.” Neural Information Processing Systems. https://api.semanticscholar.org/CorpusID:1211821. [Google Scholar] [Crossref]

27. Tai, Lei, Giuseppe Paolo, and Ming Liu. 2017. “Virtual-to-Real Deep Reinforcement Learning: Continuous Control of Mobile Robots for Mapless Navigation.” 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 31–36. https://doi.org/10.1109/IROS.2017.8202134. [Google Scholar] [Crossref]

28. Tobin, Josh, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. 2017. Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World. https://arxiv.org/abs/1703.06907. [Google Scholar] [Crossref]

29. Waga, Abderrahim, Said Benhlima, Ali Bekri, Jawad Abdouni, and Fatima Zahrae Saber. 2025. “A Survey on Autonomous Navigation for Mobile Robots: From Traditional Techniques to Deep Learning and Large Language Models.” Journal of King Saud University Computer and Information Sciences 37. https://api.semanticscholar.org/CorpusID:280771817. [Google Scholar] [Crossref]

30. Watkins, Christopher, and Peter Dayan. 1992. “Q-Learning.” Machine Learning 8: 279–92. https://api.semanticscholar.org/CorpusID:208910339. [Google Scholar] [Crossref]

31. Zhao, Wenshuai, Jorge Peña Queralta, and Tomi Westerlund. 2020. “Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: A Survey.” 2020 IEEE Symposium Series on Computational Intelligence (SSCI), 737–44. https://doi.org/10.1109/SSCI47803.2020.9308468. [Google Scholar] [Crossref]

32. Zhu, Yingjie, Wan Zuha Wan Hasan, Hafiz Rashidi Harun Ramli, Nor Mohd Haziq Norsahperi, Muhamad Saufi Mohd Kassim, and Yiduo Yao. 2025. “Deep Reinforcement Learning of Mobile Robot Navigation in Dynamic Environment: A Review.” Sensors (Basel, Switzerland) 25. https://api.semanticscholar.org/CorpusID:278981018. [Google Scholar] [Crossref]

• Ayurveda and Skin Health: A Comprehensive Framework for the Management of Dermatological Disorders
• Gender and Political Exclusion: An Analysis of Women's Marginalization in India's Democracy
• Safety Standards Compliance among Universities Employees in Nairobi County, Kenya During the Covid-19 Pandemic
• Cedi Appreciation Relative to Fuel Prices - A Machine Learning and Ancient Geomantic Approach