SHAO Xiaoqiang,LI Xin,YANG Tao,et al. Underground personnel detection and tracking based on improved YOLOv5s and DeepSORT[J]. Coal Science and Technology,2023,51(10):291−301
. DOI: 10.13199/j.cnki.cst.2022-1933Citation: |
SHAO Xiaoqiang,LI Xin,YANG Tao,et al. Underground personnel detection and tracking based on improved YOLOv5s and DeepSORT[J]. Coal Science and Technology,2023,51(10):291−301 . DOI: 10.13199/j.cnki.cst.2022-1933 |
The real-time monitoring and tracking system of mine moving targets is an essential part of the construction of smart mines. The appearance of downhole inspection robots can realize the real-time monitoring of operators, but the existence of uneven lighting, coal dust interference and other factors lead to the traditional image detection algorithm can not accurately detect operators. Based on this, this paper proposes an improved YOLOv5s and DeepSORT algorithm for downhole personnel detection and tracking that can be deployed in downhole inspection robots. Firstly, the data set was made by using the video recorded by the surveillance camera and inspection robot, and then the improved YOLOv5s network was used to identify the underground personnel: Considering that the detection and tracking algorithm for downhole personnel contains complex network structure and huge parameter volume, which limits the response speed of the detection model, this paper uses an improved lightweight network ShuffleNetV2 to replace the original YOLOv5s backbone network CSP-Darknet53. Meanwhile, in order to reduce the interference of complex image background and improve the attention of operators, Transformer self-attention module is integrated into the ShuffleNetV2. Secondly, the FPN+PAN structure in Neck is replaced by BiFPN structure in order to effectively fuse multi-scale features and effectively transmit inference information. Then, improved DeepSORT was used to encode and track personnel: considering that the underground environment was dark, with low illumination and no texture, it was difficult for DeepSORT to effectively extract personnel's appearance information, so DeepSORT's small and medium residual network was replaced by deeper convolution to enhance DeepSORT's appearance information extraction ability. Finally, the improved algorithm is verified by open pedestrian data set and self-built underground personnel detection and tracking data set. The results show that compared with the original YOLOv5s model, the average detection accuracy of the improved detection model is increased by 5.2%, the number of parameters is reduced by 41%, and the speed is increased by 21%. The improved YOLOv5s-DeepSORT downhole personnel tracking method has a precision of 89.17% and a speed of 67FPS, which can be effectively deployed in downhole inspection robots to realize real-time detection and tracking of operators.
[1] |
龚 云,颉昕宇. 基于同态滤波方法的煤矿井下图像增强技术研究[J]. 煤炭科学技术,2023,51(3):241−250. doi: 10.13199/j.cnki.cst.2021-0774
GONG Yun,XIE Xinyu. Research on coal mine underground image recognition technology based on homomorphic filtering method[J]. Coal Science and Technology,2023,51(3):241−250. doi: 10.13199/j.cnki.cst.2021-0774
|
[2] |
厉 丹. 视频目标检测与跟踪算法及其在煤矿中应用的研究[D]. 徐州: 中国矿业大学, 2011.
LI Dan. Research on object detection and tracking algorithm and its application in coal mine[D]. Xuzhou: China University of Mining and Technology, 2011.
|
[3] |
刘 丽,赵凌君,郭承玉,等. 图像纹理分类方法研究进展和展望[J]. 自动化学报,2018,44(4):584−607. doi: 10.16383/j.aas.2018.c160452
LIU Li,ZHAO Lingjun,GUO Chengyu,et al. Texture classification: state-of-the-art methods and prospects[J]. Acta Automatica Sinica,2018,44(4):584−607. doi: 10.16383/j.aas.2018.c160452
|
[4] |
ZHU H G. An efficient lane line detection method based on computer vision[J]. Journal of Physics:Conference Series,2021,1802(3):032006. doi: 10.1088/1742-6596/1802/3/032006
|
[5] |
ABBAS Q,LI Y. Cricket video events recognition using HOG, LBP and Multi-class SVM[J]. Journal of Physics:Conference Series,2021,1732(1):012036. doi: 10.1088/1742-6596/1732/1/012036
|
[6] |
阮顺领,李少博,顾清华,等. 基于双向特征融合的露天矿区道路障碍检测[J]. 煤炭学报,2023,48(3):1425−1438. doi: 10.13225/j.cnki.jccs.2022.0198
RUAN Shunling,LI Shaobo,GU Qinghua,et al. Road obstacle detection in open-pit mines based on bidirectional feature fusion[J]. Journal of China Coal Society,2023,48(3):1425−1438. doi: 10.13225/j.cnki.jccs.2022.0198
|
[7] |
李若熙,吕 潇,张元生,等. 改进YOLOv4算法井下人员检测的研究[J]. 矿业研究与开发,2021,41(11):179−185. doi: 10.13827/j.cnki.kyyk.2021.11.029
LI Ruoxi,LYU Xiao,ZHANG Yuansheng,et al. Reserch on Underground Personnel Detection Based on Improved YOLOv4 Algorithm[J]. Mining Research and Development,2021,41(11):179−185. doi: 10.13827/j.cnki.kyyk.2021.11.029
|
[8] |
BOCHKOVSKIY A, WANG C Y, LIAO H Y M. YOLOv4: Optimalspeed and accuracy of object detection [J/OL]. arXiv: 2004.109342020[cs. CV]. (2020-04-23).https://arxiv.org/abs /2004.10934.
|
[9] |
杨世超. 基于Faster-Rcnn的矿井人员识别检测[J]. 信息记录材料,2020,21(12):236−238. doi: 10.16009/j.cnki.cn13-1295/tq.2020.12.154
YANG Shichao. Mine personnel identification and detection based on Faster-RCNN[J]. Information Recording Materials,2020,21(12):236−238. doi: 10.16009/j.cnki.cn13-1295/tq.2020.12.154
|
[10] |
REN S, HE K, GIRSHICK R, et al. Faster RCNN: Towards real-time object detection with region proposal networks[C]//Advances in neural information processing systems, 2015: 91−99.
|
[11] |
董昕宇,师 杰,张国英. 基于参数轻量化的井下人体实时检测算法[J]. 工矿自动化,2021,47(6):71−78. doi: 10.13272/j.issn.1671-251x.2021010035
DONG Xinyu,SHI Jie,ZHANG Guoying. Real-time detection algorithm of underground human body based on lightweight parameters[J]. Industry and Mine Automation,2021,47(6):71−78. doi: 10.13272/j.issn.1671-251x.2021010035
|
[12] |
LIU W, ANGUELOV D, ERHAN D, et al. SSD: Single shot multibox detector[C]//European conference on computer vision. Springer, Cham, 2016: 21 – 37.
|
[13] |
陈 伟,任 鹏,田子建,等. 基于注意力机制的无监督矿井人员跟踪[J]. 煤炭学报,2021,46(S1):601−608. doi: 10.13225/j.cnki.jccs.2020.1348
CHEN Wei,REN Peng,TIAN Zijian,et al. Unsupervised mine personnel tracking based on attention mechanism[J]. Journal of China Coal Society,2021,46(S1):601−608. doi: 10.13225/j.cnki.jccs.2020.1348
|
[14] |
任志玲,朱彦存. 改进CenterNet算法的煤矿皮带运输异物识别研究[J]. 控制工程,2023(4):703−711. doi: 10.14107/j.cnki.kzgc.2020.0792
REN Zhiling,ZHU Yancun. Research on foreign body recognition in coal mine belt transportation based on improved centernet algorithm[J]. Control engineering,2023(4):703−711. doi: 10.14107/j.cnki.kzgc.2020.0792
|
[15] |
葛俏,梁桥康,邹坤霖,等. 基于轻量化网络与嵌入式系统的喷码检测[J]. 控制工程,2022(12):2349−2356. doi: 10.14107/j.cnki.kzgc.2020.0903
GE Qiao,LIANG Qiaokang,ZOU Kunlin,et al. Inkjet quality detection method based on lightweight network and embedded[J]. Control engineering,2022(12):2349−2356. doi: 10.14107/j.cnki.kzgc.2020.0903
|
[16] |
BEWLEY A, GE Z, OTT L, et al. Simple online and realtime tracking[C] //2016 IEEE International Conference on Image Processing(ICIP), 2016: 3464−3468.
|
[17] |
MA NN, ZHANG XY, ZHENG HT, et al. ShuffleNetV2: Practical guidelines for efficient CNN architecture design[C]//Proceedings of the 15th European Conference on Computer Vision (ECCV). Munich: Springer, 2018. : 122–138.
|
[18] |
WANG C Y, LIAO H Y M, WU Y H, et al. CSPNet: A new backbone that can enhance learning capability of CNN[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Seattle: IEEE, 2020: 1571–1580.
|
[19] |
XIA Z, PAN X, SONG S, et al. Vision transformer with deformable attention[J]. ArXiv preprint arXiv: 2201.00520, 2022.
|
[20] |
TAN M, PANG R, LE Q V. EfficientDet: Scalable and efficient object detection[C]//IEEE Conference on Computer Vision and Pattern Recognition, 2020: 10778−10787.
|
[21] |
REDMON J, DIVVALA S, GIRSHICK R, et al. You only look once: unified, real-time object detection[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas: IEEE, 2016: 779−788.
|
[22] |
REDMON J, FARHADI A. YOLO9000: Better, Faster, Stronger[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 21-26, 2017, Honolulu, HI, USA. New York: IEEE, 2017: 6517−6525.
|
[23] |
REDMON J, FARHADI A. YOLOv3: an incremental improvement [EB/OL]. (2018-09-30).https://arxiv.org/abs/1804. 02767.
|
[24] |
张麒麟,林清平,肖 蕾. 改进YOLOv5的航拍图像识别算法[J]. 长江信息通信,2021,34(3):73−76. doi: 10.3969/j.issn.1673-1131.2021.03.021
ZHANG Qilin,LIN Qingping,XIAO Lei. Improved algorithm for aerial image recognition based on yolov5[J]. Changjiang information communication,2021,34(3):73−76. doi: 10.3969/j.issn.1673-1131.2021.03.021
|
[25] |
WANG C Y, LIAO H Y M, WU Y H, et al. CSPNet: A new backbone that can enhance learning capability of CNN[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops. New York: IEEE Press, 2020: 390–391.
|
[26] |
HE K M,ZHANG X Y,REN S Q,et al. Spatial pyramid pooling in deep convolutional networks for visual recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence,2015,37(9):1904−1916. doi: 10.1109/TPAMI.2015.2389824
|
[27] |
ZHANG X, ZHOU X, LIN M, et al. Shufflenet: An extremely efficient convolutional neural network for mobile devices[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. Salt Lake City: IEEE Press, 2018: 6848-6856.
|
[28] |
ZAMIR S W,ARORA A,KHAN S,et al. Restormer: efficient transformer for high-resolution image restoration[J]. ArXiv preprint arXiv: 2111.09881,2022
|
[29] |
PETIT O, Thome N, RAMBOU C, et al. U-net transformer: Self and cross attention for medical image segmentation[C]//International Workshop on Machine Learning in Medical Imaging. Springer, Cham, 2021: 267-276.
|
[30] |
CHU X, TIAN Z, ZHANG B, et al. Conditional positional encodings for vision transformers[J]. arXiv preprint arXiv: 2102.10882, 2021.
|
1. |
董书宁,于树江,董兴玲,张步勤,郭小铭,王晓东,王凯,朱世彬,武博强,刘磊. 煤基固废与高盐废水“固液协同”充填处置关键技术. 煤田地质与勘探. 2025(01): 163-173 .
![]() | |
2. |
孟瑞艳,周晓芳,王新富,张雷,王彦君,章梅,吴求刚. 新汶矿区煤矸石理化特征分析及生态风险评价. 中国煤炭地质. 2025(01): 50-56 .
![]() | |
3. |
王振刚,马杨爱,朱才辉,张佳瑶,曲波. 黄河流域陕西段大宗固废煤矸石综合利用现状与展望. 中国矿业. 2025(03): 305-315 .
![]() | |
4. |
崔昕茹,霍雪萍,周炳杰,胡蕴瑶,杨彦群,杨凤玲,狄子琛. 我国煤矸石空间分布特征与分级分质利用路径. 环境科学. 2025(04): 2281-2291 .
![]() | |
5. |
杨旭,丁子昭,孔令宇,孟凡燃,黄艳利,杨伟,李俊孟. 煤基固废多源信息数据库构建及利用路径评价. 矿业安全与环保. 2025(02): 172-178 .
![]() | |
6. |
王少力,亢福仁,张凯煜,黄苑平,刘馨月,张生. 煤基固废基质对紫穗槐种子萌发及幼苗生长的影响. 现代园艺. 2024(01): 54-58 .
![]() | |
7. |
李恩伟,张世文,周思雨,董祥林,马军,蔡慧珍,胡睿鑫. 基于CiteSpace的煤基固废三化研究热点分析. 辽宁工业大学学报(社会科学版). 2024(01): 32-37 .
![]() | |
8. |
卞正富,张益东,王猛,王长申,韩晓彤,牟守国,任清晨,李国栋. 新疆煤炭资源开发潜力与开发策略. 煤炭学报. 2024(02): 967-977 .
![]() | |
9. |
郭慧超. 化工固体废弃物的资源化利用. 山西化工. 2024(03): 245-246 .
![]() | |
10. |
杨科,何淑欣,何祥,初茉,周伟,袁宁,陈登红,龚鹏,张元春. 煤电化基地大宗固废“三化”协同利用基础与技术. 煤炭科学技术. 2024(04): 69-82 .
![]() | |
11. |
刘浪,罗屹骁,朱梦博,苏臣,吴涛涛,王建友,杭彦龙. 建筑物下特厚煤层镁渣基全固废连采连充开采技术与实践. 煤炭科学技术. 2024(04): 83-92 .
![]() | |
12. |
吴玉意,刘成勇,张鹏,刘文涛,黄剑斌,刘治成,王翰秋. 综采抛矸充填技术原理及抛沙试验研究. 煤炭工程. 2024(05): 69-74 .
![]() | |
13. |
李明,李起行,吴相儒,邬志雄. 粉煤灰膏体充填材料工作性能影响因素研究. 矿业研究与开发. 2024(06): 65-71 .
![]() | |
14. |
赵康,伍俊,马超,朱开成,聂晶磊,胡华龙. 我国煤矿固、液、气“三废”地质封存研究现状与生态环境协同发展的关系. 煤炭学报. 2024(06): 2785-2798 .
![]() | |
15. |
李忠宝. 煤炭洗选废弃物处理与资源化利用研究. 黑龙江环境通报. 2024(08): 102-104 .
![]() | |
16. |
郑向东,李明,刘威,王新富,赵恒,李建,周晓芳. 皖北矿区煤矸石理化特征分析及生态风险评价. 能源与环保. 2024(09): 175-180+186 .
![]() | |
17. |
邓友生,董晨辉,吴阿龙,庄子颖,陈茁. 我国煤矸石山综合治理现状及零碳治理探索. 煤矿安全. 2024(10): 120-128 .
![]() | |
18. |
李新旺,吴荣元,李磊,祁义飞,程立朝,李丽. 粉煤灰/煤矸石离层注浆充填材料性能研究. 煤炭与化工. 2024(10): 1-5+11 .
![]() | |
19. |
单成方,吴来伟,李亚锋,韩创业,龙岩,李俊孟,齐文跃. 新疆地区烧变岩分布特征与资源化利用对策. 矿业安全与环保. 2024(05): 162-167 .
![]() | |
20. |
宋伟,刘成勇,杨军军,王若帆,张新福,刘东升. 煤矸石规模化处置及高附加值利用现状与展望. 矿业研究与开发. 2024(11): 268-282 .
![]() | |
21. |
乔波,杨澜,王红星,邢涛,于利红,王振华,姜晓枫. 新疆煤灰在不同高铬砖表面的侵蚀性研究. 山东化工. 2024(22): 65-69 .
![]() | |
22. |
朱磊,刘治成,刘成勇,赵萌烨,贾金兑,丁自伟. 基于孔隙率和粒径的煤矸石力学性能预测试验研究. 西安科技大学学报. 2024(06): 1083-1094 .
![]() | |
23. |
刘鹏亮. 煤基固废充填材料及充填工艺发展现状与展望. 矿产保护与利用. 2024(06): 15-24 .
![]() | |
24. |
张计林. 煤矸石路基含水率对路基沉降的影响分析及控制措施. 交通世界. 2024(36): 108-110 .
![]() | |
25. |
朱琦,胡振琪,叶春,李春华,李冲. 基于化学-微生物法的煤矸石山酸化污染原位控制技术研究进展. 中国矿业. 2023(01): 52-59 .
![]() | |
26. |
张吉雄,周楠,高峰,闫浩. 煤矿开采嗣后空间矸石注浆充填方法. 煤炭学报. 2023(01): 150-162 .
![]() | |
27. |
范晓平,刘京,康哲,董学光. 煤矸石综合利用与矿山生态修复的战略思考. 环境卫生工程. 2023(01): 8-15 .
![]() | |
28. |
吴长亮,王文龙,王旭江,李敬伟. 煤矸石在建材领域的资源化研究现状及创新利用策略. 能源环境保护. 2023(01): 167-177 .
![]() | |
29. |
王伟,张志义,赵博. 煤矿井下风积沙箱式充填体侧向约束机理数值研究. 新疆大学学报(自然科学版)(中英文). 2023(03): 367-372 .
![]() | |
30. |
肖元庚. 城市固体废弃物处理及资源化利用途径. 皮革制作与环保科技. 2023(09): 131-133 .
![]() | |
31. |
赵学松,刘琦. 煤基固体废弃物制备压裂支撑剂研究进展. 洁净煤技术. 2023(06): 161-172 .
![]() | |
32. |
徐文彬,陈伟. 不同倾角充-岩组合体三轴压缩力学特性及破坏特征. 矿业科学学报. 2023(05): 633-641 .
![]() | |
33. |
张道平. 矿用煤矸石混凝土力学性能试验研究. 煤. 2023(09): 1-5 .
![]() | |
34. |
李宏伟,燕可洲,文朝璐,柳丹丹,郭彦霞,张圆圆. 煤矸石制备活性炭-介孔硅复合材料及其过程物相转变. 煤炭科学技术. 2023(09): 310-319 .
![]() | |
35. |
方屹,杨浪,饶峰,张凯铭,宁旭文. 煤矸石的地质聚合反应资源化利用研究进展. 矿业研究与开发. 2023(12): 206-213 .
![]() | |
36. |
武浩洁. 碳硅复合介孔材料对水体中NO_3~-的吸附机制分析及研究. 山西化工. 2023(12): 13-14+25 .
![]() | |
37. |
原新宇,刘树龙,余青龙,张敏哲,王贻明,许宏图,苏凤波,李敏. 煤矸石基胶结充填体力学性能及微观结构演化研究. 现代矿业. 2023(12): 207-212 .
![]() | |
38. |
韩崇刚. “双碳”目标下煤基固废高值化处理与综合利用研究. 煤炭经济研究. 2023(12): 30-35 .
![]() | |
39. |
李家新,王文峰,陆青锋,车可心,刘恒言. 新疆地区不同岩性煤矸石重金属元素释放规律研究. 煤炭科学技术. 2023(S2): 328-335 .
![]() | |
40. |
李永亮,路彬,杨仁树,林海,徐斌,王树帅,刘晨辉. 煤矿连采连充式胶结充填采煤技术与典型工程案例. 煤炭学报. 2022(03): 1055-1071 .
![]() | |
41. |
桑树勋,袁亮,刘世奇,韩思杰,郑司建,刘统,周效志,王冉. 碳中和地质技术及其煤炭低碳化应用前瞻. 煤炭学报. 2022(04): 1430-1451 .
![]() | |
42. |
徐连兵,卓锦德,张凯. 大型煤电化基地固废分质分类资源化利用研究. 中国煤炭. 2022(07): 131-136 .
![]() | |
43. |
吴昊天,王晨光,黄进,耿可明. 新疆煤渣对高铬砖侵蚀性能的影响. 煤化工. 2022(04): 70-73+100 .
![]() | |
44. |
王艳,左震,文波,高冠一. 煤矸石粗集料理化性质和形状特征对混凝土强度的影响. 矿业科学学报. 2022(05): 554-564 .
![]() | |
45. |
张强,王云搏,张吉雄,左小,杨军辉,张昊,巨峰,孙忠良,魏连河,史磊. 煤矿固体智能充填开采方法研究. 煤炭学报. 2022(07): 2546-2556 .
![]() | |
46. |
吴蒙,秦云虎,李国璋,申建,宋雪娟,朱士飞,韩磊. 煤炭地下气化影响因素及评价方法研究进展. 煤炭科学技术. 2022(08): 259-269 .
![]() | |
47. |
王玉涛. 煤矸石固废无害化处置与资源化综合利用现状与展望. 煤田地质与勘探. 2022(10): 54-66 .
![]() | |
48. |
黄艳利,郭亚超,齐文跃,李俊孟,高华东,欧阳神央,吴来伟,阮泽宇. 西部典型矿区采动地下水流场演变特征与工作面涌水控制技术. 中国科学基金. 2022(06): 1016-1026 .
![]() | |
49. |
杨科,赵新元,何祥,魏祯. 多源煤基固废绿色充填基础理论与技术体系. 煤炭学报. 2022(12): 4201-4216 .
![]() | |
50. |
李衡,刘伦强,周雪军. 新疆润田煤矿矸石充填开采系统设计. 科技创新与应用. 2021(18): 78-79+83 .
![]() | |
51. |
董兴玲,董书宁,王皓,王宝,王强民,周振方,张溪彧. 古土壤层对煤矸石淋滤液中典型污染物的防污性能. 煤炭学报. 2021(06): 1957-1965 .
![]() | |
52. |
于昊辰,陈浮,尹登玉,韩晓彤,牟守国,雷少刚,卞正富. 采矿与气候变化对戈壁矿区土地生态系统的影响——以准东煤炭基地为例. 煤炭学报. 2021(08): 2650-2663 .
![]() |