[1]李康斌,朱齐丹,牟进友,等.基于改进DDQN船舶自动靠泊路径规划方法[J].智能系统学报,2025,20(1):73-80.[doi:10.11992/tis.202401005]
LI Kangbin,ZHU Qidan,MU Jinyou,et al.Automatic ship berthing path-planning method based on improved DDQN[J].CAAI Transactions on Intelligent Systems,2025,20(1):73-80.[doi:10.11992/tis.202401005]
点击复制
LI Kangbin,ZHU Qidan,MU Jinyou,et al.Automatic ship berthing path-planning method based on improved DDQN[J].CAAI Transactions on Intelligent Systems,2025,20(1):73-80.[doi:10.11992/tis.202401005]
基于改进DDQN船舶自动靠泊路径规划方法
《智能系统学报》[ISSN 1673-4785/CN 23-1538/TP] 卷:
20
期数:
2025年第1期
页码:
73-80
栏目:
学术论文—机器学习
出版日期:
2025-01-05
- Title:
- Automatic ship berthing path-planning method based on improved DDQN
- Keywords:
- automatic berthing; path planning; deep reinforcement learning; double deep Q network; reward function; current velocity; state exploration; thrust; time; independent repeated experiments
- 分类号:
- TP273
- 摘要:
- 船舶在自动靠泊过程中会受到风、浪、流和岸壁效应等因素的影响，故需要精确的路径规划方法防止靠泊失败。针对全驱动船舶靠泊过程的基于双深度Q网络（double deep Q network，DDQN）算法，设计了一种船舶自动靠泊路径规划方法。首先建立船舶三自由度模型，然后通过将距离、航向、推力、时间和碰撞作为奖励或惩罚，改进奖励函数。随后引入DDQN来学习动作奖励模型，并使用学习结果来操纵船舶运动。通过追求更高的奖励值，船舶可以自行找到最优的靠泊路径。实验结果表明，在不同水流速度下，船舶都可以在完成靠泊的同时减小时间和推力，并且在相同水流速度下，DDQN算法与Q-learning、SARSA（state action reward state action）、深度Q网络（deep Q network, DQN）等算法相比，靠泊过程推力分别减小了241.940、234.614、80.202 N，且时间仅为252.485 s。
- Abstract:
- During the automatic docking process, ships are influenced by factors such as wind, waves, currents, and quay wall effects, necessitating precise path-planning methods to prevent docking failures. For fully actuated ships, a ship’s automatic docking path-planning method is designed based on the double deep Q network (DDQN) algorithm. Firstly, a three-degree-of-freedom model for the ship is established, and then the reward function is improved by incorporating distance, heading, thrust, time, and collisions as rewards or penalties. DDQN is then introduced to learn the action-reward model and use the learning results to manipulate ship movements. By pursuing higher reward values, the ship can autonomously find the optimal docking path. Experimental results show that under different water flow velocities, ships can reduce both time and thrust while completing docking. Moreover, at the same water flow velocity, compared with Q-learning, SARSA, and deep Q Network (DQN), the DDQN algorithm reduces thrust by 241.940N, 234.614N, and 80.202N respectively during the docking process, with the time being only 252.485 seconds.
- 参考文献/References:
-
[1] 刘志林, 苑守正, 郑林熇, 等. 船舶自动靠泊技术的发展现状和趋势[J]. 中国造船, 2021, 62(4): 293-304.
LIU Zhilin, YUAN Shouzheng, ZHENG Linhe, et al. Development status and trend of ship automatic berthing technology[J]. Shipbuilding of China, 2021, 62(4): 293-304.
[2] 李国帅, 张显库, 张安超. 智能船舶靠泊技术研究热点与趋势[J]. 中国舰船研究, 2024, 19(1): 3-14.
LI Guoshuai, ZHANG Xianku, ZHANG Anchao. Research hotspots and tendency of intelligent ship berthing technology[J]. Chinese journal of ship research, 2024, 19(1): 3-14.
[3] 包政凯, 朱齐丹, 刘永超. 满秩分解最小二乘法船舶航向模型辨识[J]. 智能系统学报, 2022, 17(1): 137-143.
BAO Zhengkai, ZHU Qidan, LIU Yongchao. Ship heading model identification based on full rank decomposition least square method[J]. CAAI transactions on intelligent systems, 2022, 17(1): 137-143.
[4] WANG Shaobo, ZHANG Yingjun, LI Lianbo. A collision avoidance decision-making system for autonomous ship based on modified velocity obstacle method[J]. Ocean engineering, 2020, 215: 107910.
[5] LIU Xinzhao, REN Junsheng, WANG Hui. Optimal energy trajectory planning and control for automatic ship berthing[C]//2020 39th Chinese Control Conference. Shenyang: IEEE, 2020: 1420-1425.
[6] GAO Xuanyu, DONG Yitao, HAN Yi. An optimized path planning method for container ships in Bohai Bay based on improved deep Q-learning[J]. IEEE access, 2023, 11: 91275-91292.
[7] GUO Siyu, ZHANG Xiuguo, ZHENG Yisong, et al. An autonomous path planning model for unmanned ships based on deep reinforcement learning[J]. Sensors, 2020, 20(2): 426.
[8] HE Zhibo, LIU Chenguang, CHU Xiumin, et al. Dynamic anti-collision A-star algorithm for multi-ship encounter situations[J]. Applied ocean research, 2022, 118: 102995.
[9] LIU Chenguang, MAO Qingzhou, CHU Xiumin, et al. An improved A-star algorithm considering water current, traffic separation and berthing for vessel path planning[J]. Applied sciences, 2019, 9(6): 1057.
[10] CHEN Chen, CHEN Xianqiao, MA Feng, et al. A knowledge-free path planning approach for smart ships based on reinforcement learning[J]. Ocean engineering, 2019, 189: 106299.
[11] 周治国, 余思雨, 于家宝, 等. 面向无人艇的T-DQN智能避障算法研究[J]. 自动化学报, 2023, 49(8): 1645-1655.
ZHOU Zhiguo, YU Siyu, YU Jiabao, et al. Research on T-DQN intelligent obstacle avoidance algorithm of unmanned surface vehicle[J]. Acta automatica sinica, 2023, 49(8): 1645-1655.
[12] SILVA JUNIOR A G D, SANTOS D H D, NEGREIROS A P F, et al. High-level path planning for an autonomous sailboat robot using Q-learning[J]. Sensors, 2020, 20(6): 1550.
[13] 邢博闻, 张昭夷, 王世明, 等. 基于深度强化学习的多无人艇协同目标搜索算法[J]. 兵器装备工程学报, 2023, 44(11): 118-125.
XING Bowen, ZHANG Zhaoyi, WANG Shiming, et al. Multi-USV cooperative target search algorithm based on deep reinforcement learning[J]. Journal of ordnance equipment engineering, 2023, 44(11): 118-125.
[14] 曹景祥, 刘其成. 基于深度强化学习的路径规划算法研究[J]. 计算机应用与软件, 2022, 39(11): 231-237.
CAO Jingxiang, LIU Qicheng. Research on path planning algorithm based on deep reinforcement learning[J]. Computer applications and software, 2022, 39(11): 231-237.
[15] 闫皎洁, 张锲石, 胡希平. 基于强化学习的路径规划技术综述[J]. 计算机工程, 2021, 47(10): 16-25.
YAN Jiaojie, ZHANG Qieshi, HU Xiping. Review of path planning techniques based on reinforcement learning[J]. Computer engineering, 2021, 47(10): 16-25.
[16] 邓修朋, 崔建明, 李敏, 等. 深度强化学习在机器人路径规划中的应用[J]. 电子测量技术, 2023, 46(6): 1-8.
DENG Xiupeng, CUI Jianming, LI Min, et al. Application of deep reinforcement learning in robot path planning[J]. Electronic measurement technology, 2023, 46(6): 1-8.
[17] LI Lingyu, WU Defeng, HUANG Youqiang, et al. A path planning strategy unified with a COLREGS collision avoidance function based on deep reinforcement learning and artificial potential field[J]. Applied ocean research, 2021, 113: 102759.
[18] ZHANG Xinyu, WANG Chengbo, LIU Yuanchang, et al. Decision-making for the autonomous navigation of maritime autonomous surface ships based on scene division and deep reinforcement learning[J]. Sensors, 2019, 19(18): 4055.
[19] ZHOU Xinyuan, WU Peng, ZHANG Haifeng, et al. Learn to navigate: cooperative path planning for unmanned surface vehicles using deep reinforcement learning[J]. IEEE access, 2019, 7: 165262-165278.
[20] OZTURK U , AKDAG M , AYABAKAN T . A review of path planning algorithms in maritime autonomous surface ships: navigation safety perspective[J]. Ocean engineering, 2022, 251: 111010.
[21] MCCUE L. Handbook of marine craft hydrodynamics and motion control[bookshelf[J]. IEEE control systems magazine, 2016, 36(1): 78-79.
[22] SKJETNE R . The maneuvering problem[EB/OL]. (2005-03-15)[2024-01-03]. https://www.researchgate.net/publication/236651127_The_Maneuvering_Problem.
[23] SONG Zhaofeng, ZHANG Jinfen, WU Da, et al. A novel path planning algorithm for ships in dynamic current environments[J]. Ocean engineering, 2023, 288: 116091.
[24] GAO Ning, QIN Zhijin, JING Xiaojun, et al. Anti-intelligent UAV jamming strategy via deep Q-networks[J]. IEEE transactions on communications, 2020, 68(1): 569-581.
[25] DEGRIS T, PILARSKI P M, SUTTON R S. Model-Free reinforcement learning with continuous action in practice[C]//2012 American Control Conference. Montreal: IEEE, 2012: 2177-2182.
[26] HIGO Y, SAKANO M, NOBE H, et al. Development of trajectory-tracking maneuvering system for automatic berthing/unberthing based on double deep Q-network and experimental validation with an actual large ferry[J]. Ocean engineering, 2023, 287: 115750.
[27] VAN HASSELT H, GUEZ A, SILVER D. Deep reinforcement learning with double Q-learning[C]//Proceedings of the AAAI conference on artificial intelligence. Arizona: AAAI Press, 2016: 2094-2100.
[28] ZHANG Fei, GU Chaochen, YANG Feng. An improved algorithm of robot path planning in complex environment based on double DQN[M]//Lecture Notes in Electrical Engineering. Singapore: Springer Singapore, 2021: 303-313.
[29] YANG Xiaofei, SHI Yilun, LIU Wei, et al. Global path planning algorithm based on double DQN for multi-tasks amphibious unmanned surface vehicle[J]. Ocean engineering, 2022, 266: 112809.
[30] ZHANG Jiaqi, JIAO Xiaohong, YANG Chao. A double-deep Q-network-based energy management strategy for hybrid electric vehicles under variable driving cycles[J]. Energy technology, 2021, 9(2): 2000770.
- 相似文献/References:
-
[1]黄彦文,曹其新.RoboCup比赛环境下足球机器人路径规划研究[J].智能系统学报,2007,2(4):52.
HUANG Yan-wen,CAO Qin-xin.Path planning for robot soccer in the RoboCup environment[J].CAAI Transactions on Intelligent Systems,2007,2():52.
[2]秦世引,高书征.面向救援任务的地面移动机器人路径规划[J].智能系统学报,2009,4(5):414.[doi:10.3969/j.issn.1673-4785.2009.05.005]
QIN Shi-yin,GAO Shu-zhen.Path planning for mobile rescue robots in disaster areas with complex environments[J].CAAI Transactions on Intelligent Systems,2009,4():414.[doi:10.3969/j.issn.1673-4785.2009.05.005]
[3]曹卫华,吴净斌,吴 敏,等.无路标环境下遥操作机器人SLAM系统[J].智能系统学报,2010,5(3):240.
CAO Wei-hua,WU Jing-bin,WU Min,et al.A system for telerobotics in environments without landmarks[J].CAAI Transactions on Intelligent Systems,2010,5():240.
[4]薛英花,田国会,吴 皓,等.智能空间中的服务机器人路径规划[J].智能系统学报,2010,5(3):260.
XUE Ying-hua,TIAN Guo-hui,WU Hao,et al.Path planning for service robots in an intelligent space[J].CAAI Transactions on Intelligent Systems,2010,5():260.
[5]黄晓丹,王粉花,王志良.情感决策的智能家居虚拟人路径规划[J].智能系统学报,2010,5(4):292.
HUANG Xiao-dan,WANG Fen-hua,WANG Zhi-liang.Using affective decisionmaking for the path planning of virtual humans in a smart home[J].CAAI Transactions on Intelligent Systems,2010,5():292.
[6]唐小勇,于 飞,潘洪悦.改进粒子群算法的潜器导航规划[J].智能系统学报,2010,5(5):443.[doi:10.3969/j.issn.1673-4785.2010.05.011]
TANG Xiao-yong,YU Fei,PAN Hong-yue.Submersible path-planning based on an improved PSO[J].CAAI Transactions on Intelligent Systems,2010,5():443.[doi:10.3969/j.issn.1673-4785.2010.05.011]
[7]夏琳琳,张健沛,初妍.计算智能在移动机器人路径规划中的应用综述[J].智能系统学报,2011,6(2):160.
XIA Linlin,ZHANG Jianpei,CHU Yan.An application survey on computational intelligence for path planning of mobile robots[J].CAAI Transactions on Intelligent Systems,2011,6():160.
[8]蒲兴成,张军,张毅.基于神经网络的改进行为协调控制及其在智能轮椅路径规划中的应用[J].智能系统学报,2011,6(5):456.
PU Xingcheng,ZHANG Jun,ZHANG Yi.Modified behavior coordination for intelligent wheelchair path planning based on a neural network[J].CAAI Transactions on Intelligent Systems,2011,6():456.
[9]肖国宝,严宣辉.一种基于改进Theta *的机器人路径规划算法[J].智能系统学报,2013,8(1):58.[doi:10.3969/j.issn.1673-4785.201208032]
XIAO Guobao,YAN Xuanhui.A path planning algorithm based on improved Theta * for mobile robot[J].CAAI Transactions on Intelligent Systems,2013,8():58.[doi:10.3969/j.issn.1673-4785.201208032]
[10]杨茂,田彦涛.复杂环境下多机器人觅食路径规划与控制[J].智能系统学报,2013,8(2):162.[doi:10.3969/j.issn.1673-4785.201208022]
YANG Mao,TIAN Yantao.Foraging path planning and control for multi-robot in complex environment[J].CAAI Transactions on Intelligent Systems,2013,8():162.[doi:10.3969/j.issn.1673-4785.201208022]
备注/Memo
收稿日期:2024-1-3。
基金项目:国家自然科学基金项目(52171299).
作者简介:李康斌，硕士研究生，主要研究方向为船舶自动靠泊和路径规划。E-mail：422698152@qq.com。;朱齐丹，教授，博士生导师，黑龙江省自动化学会常务理事，主要研究方向为智能机器人技术及应用、智能控制系统设计、图像处理与模式识别。主持国家自然科学基金项目、工信部高技术船舶专项项目、科技部国际合作项目等课题30项。研究成果获国家科技进步二等奖1项、国防科技进步一等奖3项、军队科技进步一等奖1项、黑龙江省科技进步二等奖3项，获发明专利授权20项、软件著作授权5项，出版专著4部，发表学术论文200篇。E-mail：zhuqidan@hrbeu.edu.cn。;牟进友，博士研究生，主要研究方向为船舶自主靠泊和智能船舶。E-mail: mujinyou96@163.com。
通讯作者:朱齐丹. E-mail：zhuqidan@hrbeu.edu.cn
更新日期/Last Update:
2025-01-05