基于顶点与主体区域同步检测的精准车牌定位

徐光柱; 刘高飞; 匡婉; 万秋波; 马国亮; 雷帮军

doi:10.13700/j.bh.1001-5965.2022.0396

基于顶点与主体区域同步检测的精准车牌定位

doi: 10.13700/j.bh.1001-5965.2022.0396

徐光柱^{1, 2},
刘高飞²,
匡婉²,
万秋波²,
马国亮³,
雷帮军^{1, 2, ,}

1.
湖北省水电工程智能视觉监测重点实验室（三峡大学），宜昌 443002
2.
三峡大学计算机与信息学院，宜昌 443002
3.
宜昌市公安局交通警察支队，宜昌 443002

基金项目: 湖北省中央引导地方科技发展专项(2019ZYYD007)；湖北省水电工程智能视觉监测重点实验室开放基金(2022SDSJ03)

详细信息

通讯作者:
E-mail：Bangjun.Lei@ieee.org

中图分类号: TP391
计量
- 文章访问数: 793
- HTML全文浏览量: 111
- PDF下载量: 33
- 被引次数: 0
出版历程
- 收稿日期: 2022-05-20
- 录用日期: 2022-07-02
- 网络出版日期: 2022-11-02
- 整期出版日期: 2024-02-27

Accurate license plate location based on synchronous vertex and body region detection

XU Guangzhu^{1, 2},
LIU Gaofei²,
KUANG Wan²,
WAN Qiubo²,
MA Guoliang³,
LEI Bangjun^{1, 2
, ,}

1.
Hubei Key Laboratory of Intelligent Vision Monitoring for Hydropower Engineering (China Three Gorges University)，Yichang 443002，China
2.
College of Computer and Information Technology，China Three Gorges University，Yichang 443002，China
3.
Traffic Police Detachment，Public Security Bureau of Yichang City，Yichang 443002，China

Funds: Hubei Provincial Central Leading Local Science and Technology Development Special Project (2019ZYYD007); Funds of Hubei Key Laboratory of Intelligent Vision Monitoring for Hydropower Engineering (2022SDSJ03)

More Information

Corresponding author: E-mail：Bangjun.Lei@ieee.org

摘要

摘要:
为应对非约束环境下的车牌精定位问题，提出一种基于顶点局部区域与主体区域同步检测策略的非约束性车牌定位算法。通过删减YOLOv5网络的输出结构，训练得到可同步检测车牌及顶点区域的车牌检测网络，在兼顾精度与计算速度的前提下，实现车牌顶点和主体区域的同步定位。针对一幅图中存在多个车牌区域及顶点区域存在少量漏检和误检的情况，分别设计了车牌顶点归类和单一缺失顶点预测后处理算法，借助顶点间的空间位置关系进行漏检目标预测和误检目标排查，有效改善了因场景复杂导致的个别顶点目标检测效果差的问题。所提算法在中国城市停车场数据集(CCPD)上的测试结果显示，平均精准率达99.25%，平均召回率达98.70%。所提算法不仅能够准确预测出车牌的4个顶点坐标，而且在中端GPU硬件平台上处理速度可达121帧/s，具有较好的应用价值。
- 深度学习 /
- 卷积网络 /
- 视觉目标检测 /
- 非约束车牌定位 /
- 车牌顶点检测
Abstract:
A novel unconstrained license plate accurate location algorithm is designed by simultaneously detecting the four local vertex regions and the body of a license plate and fusing the results to address the issue that the widely used rectangular bounding boxes in mainstream target detection methods cannot meet the license plate location accuracy requirement in many unconstrained environments where the license plate images are not commonly rectangle. At first, the four local rectangular sub-regions with centers on four vertices of a license plate were annotated as vertex-region objects according to the size of the plate’s contour-rectangle and the vertex coordinates. Then, a multi-class image dataset is built up in which the contour-rectangle region covering the whole license plate body is a class and the four kinds of vertex-region construct other four classes. In order to locate these five object classes efficiently, the output structure of the YOLOv5 network is modified by taking accuracy and efficiency into consideration and trained with the newly constructed multi-class dataset.Finally, vertex region grouping and single missing vertex forecasting are carried out as the post-processing to address the issue that there are multiple candidate license plates in an image and a few vertices region false or missing detection errors will happen in some unique instances.By exploiting the relationship among the vertexes, the post-processing can effectively recognize missing and false detection errors in some special complex scenarios and improve the whole system’s performance greatly. The proposed algorithm is evaluated on the Chinese city parking dataset (CCPD), and reaches an average positioning accuracy of 99.25% and an average recall rate of 98.70%. The performance certificates our method not only can accurately predict the coordinates of the four vertices but also can run at 121 frame/s on a moderate GPU hardware platform, which has great application potential.
- deep learning /
- convolutional network /
- visual target detection /
- unconstrained license plate location /
- license plate vertex detection

HTML全文

图 1 本文采用的双检测头YOLOv5网络结构

Figure 1. Structure of dual-head YOLOv5 network used in this paper

下载: 全尺寸图片幻灯片

图 2 系统示意图

Figure 2. System schematic diagram

下载: 全尺寸图片幻灯片

图 3 车牌顶点区域与车牌区域的空间位置关系

Figure 3. Spatial position relationship between vertex regions and license plate

下载: 全尺寸图片幻灯片

图 4 车牌顶点区域图例

Figure 4. Instances of license plate vertex region

下载: 全尺寸图片幻灯片

图 5 后处理流程

Figure 5. Post-processing flow

下载: 全尺寸图片幻灯片

图 6 车牌顶点归类图例

Figure 6. License plate vertexes grouping instances

下载: 全尺寸图片幻灯片

图 7 车牌顶点归类流程

Figure 7. License plate vertexes grouping flow

下载: 全尺寸图片幻灯片

图 8 单一缺失顶点预测图示

Figure 8. Prediction illustration for single vertex missing cases

下载: 全尺寸图片幻灯片

图 9 图像扩展方式

Figure 9. Image expansion method

下载: 全尺寸图片幻灯片

图 10 车牌标签可视化示例

Figure 10. Visualization instances of license plate labels

下载: 全尺寸图片幻灯片

图 11 精准定位的车牌示例

Figure 11. Accurate license plate location instances

下载: 全尺寸图片幻灯片

图 12 单一缺失顶点时的预测示例

Figure 12. Prediction instances in cases of single vertex missing

下载: 全尺寸图片幻灯片

图 13 交并比计算方式

Figure 13. IOU calculation method

下载: 全尺寸图片幻灯片

图 14 PKU数据集车牌定位示例

Figure 14. License plate location instances in PKU dataset

下载: 全尺寸图片幻灯片

图 15 DB场景中的漏检与误检示例

Figure 15. False and missing detection instances in DB sub-dataset

下载: 全尺寸图片幻灯片

图 16 车牌与背景颜色相似情况的检测示例

Figure 16. Instances of detection of license plate with similar background color

下载: 全尺寸图片幻灯片

图 17 光照不佳情况的检测示例

Figure 17. Instances with poor illumination

下载: 全尺寸图片幻灯片

图 18 车牌模糊情况的检测示例

Figure 18. Instances with fuzzy license plate images

下载: 全尺寸图片幻灯片

图 19 车牌倾斜角度大时的检测示例

Figure 19. Instances with tilted license plate images

下载: 全尺寸图片幻灯片

图 20 极端条件导致的误检情况示例

Figure 20. False detection instances under some extreme scenarios

下载: 全尺寸图片幻灯片

图 21 标签错误导致的误判示例

Figure 21. Misjudgment instances caused by false labels

下载: 全尺寸图片幻灯片

表 1 模型训练参数

Table 1. Model training parameters

参数	数值
批次	16
类别数	5
学习率	0.01
动量	0.937
迭代次数	150
输入宽高	(640,640)
锚框宽高(6个)	(66,64), (123,49), (84,83), (100,99), (176,68), (124,123)

下载: 导出CSV

表 2 不同算法在各子集下的定位准确率

Table 2. Location accuracy of different algorithms under various subset

算法	平均精准率/%	检测速度/(帧·s⁻¹)	准确率/%
算法	平均精准率/%	检测速度/(帧·s⁻¹)	Base	Challenge	DB	FN	Rotate	Tilt	Weather	Blur
YOLO9000^[26]	93.10	42	98.80	88.60	89.60	77.30	93.30	91.80	84.20
RPNet^[5]	94.50	61	99.30	92.80	89.50	85.30	94.70	93.20	84.10
YOLOv3^[16]	98.24	52	99.94	97.72	94.35	91.04	97.87	99.00	99.73	98.60
ResNet-50^[25]	97.80	26	99.70	93.50	92.70	94.50	98.50	96.20	99.10
ILPRNET+YOLOv3^[27]	96.00	52	97.10	90.50	97.20	93.30	91.60	94.60	97.90
LPDR-RSCNet^[23]	98.00	34	99.00	90.40	98.80	94.30	98.90	99.10	99.40
MTLPR^[11]	97.70	65
YOLOv3(结合后处理)	99.20	49	99.99	98.70	96.91	97.44	99.29	99.37	99.96	98.83
本文算法(E)	99.16	100	99.99	98.60	96.61	97.42	99.31	99.36	99.96	98.69
本文算法(F)	99.26	97	99.99	98.78	97.30	97.49	99.32	99.42	99.97	98.96
本文算法(G)	99.10	126	99.98	98.56	96.43	97.29	99.30	99.28	99.96	98.43
本文算法(H)	99.25	121	99.99	98.75	97.10	97.64	99.33	99.39	99.98	98.88

下载: 导出CSV

表 3 不同算法在PKU数据集上的定位准确率

Table 3. Location accuracy of different algorithms on PKU dataset

算法	平均精准率/%	检测速度/(帧·s⁻¹)	准确率/%
算法	平均精准率/%	检测速度/(帧·s⁻¹)	G1	G2	G3	G4	G5
文献[28]	91.52	672	98.89	98.42	95.83	81.17	83.31
文献[29]	97.69	42	98.76	98.42	97.72	96.23	97.32
文献[25]	99.80	21	99.75	100	99.73	99.65	99.65
本文算法	99.84	100	100	100	100	100	99.23

下载: 导出CSV

参考文献(29)

[1]	JAMTSHO Y, RIYAMONGKOL P, WARANUSAST R. Real-time license plate detection for non-helmeted motorcyclist using YOLO[J]. ICT Express, 2021, 7(1): 104-109. doi: 10.1016/j.icte.2020.07.008
[2]	HUANG Q, CAI Z, LAN T. A single neural network for mixed style license plate detection and recognition[J]. IEEE Access, 2021, 9: 21777-21785. doi: 10.1109/ACCESS.2021.3055243
[3]	曾丽华, 李超, 熊璋. 基于边缘与颜色信息的车牌精确定位算法[J]. 北京航空航天大学学报, 2007, 33(9): 1112-1116. doi: 10.3969/j.issn.1001-5965.2007.09.026 ZENG L H, LI C, XIONG Z. Precisely license plate locating algorithm based on edge and color information[J]. Journal of Beijing University of Aeronautics and Astronautics, 2007, 33(9): 1112-1116(in Chinese). doi: 10.3969/j.issn.1001-5965.2007.09.026
[4]	CHEN J I Z, ZONG J I. Automatic vehicle license plate detection using K-means clustering algorithm and CNN[J]. Journal of Electrical Engineering and Automation, 2021, 3(1): 15-23. doi: 10.36548/jeea.2021.1.002
[5]	XU Z B, YANG W, MENG A J, et al. Towards end-to-end license plate detection and recognition: A large dataset and baseline[C]//Proceedings of the European Conference on Computer Vision. Berlin: Springer, 2018: 255-271.
[6]	SILVA S M, JUNG C R. License plate detection and recognition in unconstrained scenarios[C]//Proceedings of the European Conference on Computer Vision. Berlin: Springer, 2018: 580-596.
[7]	SELMI Z, HALIMA M B, PAL U, et al. DELP-DAR system for license plate detection and recognition[J]. Pattern Recognition Letters, 2019, 129: 213-223.
[8]	HE K M, GKIOXARI G, DOLLÁR P, et al. Mask R-CNN[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 42(2): 386-397. doi: 10.1109/TPAMI.2018.2844175
[9]	SZEGEDY C, LIU W, JIA Y Q, et al. Going deeper with convolutions[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE Press, 2015: 1-9.
[10]	CHEN L S, TIAN S, MA W J, et al. End-to-end trainable network for degraded license plate detection via vehicle-plate relation mining[J]. Neurocomputing, 2021, 446: 1-10. doi: 10.1016/j.neucom.2021.03.040
[11]	WANG W, YANG J, CHEN M, et al. A light CNN for end-to-end car license plates detection and recognition[J]. IEEE Access, 2019, 7: 173875-173883. doi: 10.1109/ACCESS.2019.2956357
[12]	YIN X, LIU X. Multi-task convolutional neural network for pose-invariant face recognition[J]. IEEE Transactions on Image Processing, 2017, 27(2): 964-975.
[13]	WANG Y, BIAN Z P, ZHOU Y, et al. Rethinking and designing a high-performing automatic license Plate recognition approach[EB/OL]. (2021-06-17) [2022-05-20]. https://arxiv.org/abs/2011.14936.
[14]	LIN T Y, DOLLÁR P, GIRSHICK R, et al. Feature pyramid networks for object detection[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE Press, 2017: 2117-2125.
[15]	Ultralytics. YOLOv5[EB/OL]. (2022-05-12) [2022-05-20]. https://github.com/ultralytics/yolov5.
[16]	REDMON J, FARHADI A. YOLOv3: An incremental improvement[EB/OL]. (2018-04-08) [2022-05-20]. https://arxiv.org/abs/1804.02767.
[17]	ALEXEY B, WANG C Y, LIAO H. YOLOv4: Optimal speed and accuracy of object detection[EB/OL]. (2020-04-23) [2022-05-20]. https://arxiv.org/abs/2004.10934v1.
[18]	WANG C Y, LIAO H, YEH I H, et al. CSPNet: A new backbone that can enhance learning capability of CNN[EB/OL]. (2019-11-27) [2022-05-20].https://arxiv.org/abs/1911.11929.
[19]	HE K M, ZHANG X, REN S, et al. Spatial pyramid pooling in deep convolutional networks for visual recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 37(9): 1904-1916.
[20]	LIU S, QI L, QIN H, et al. Path aggregation network for instance segmentation[EB/OL]. (2018-03-05) [2022-05-20]. https://arxiv.org/abs/1803.01534.
[21]	LOSHCHILOV I, HUTTER F. SGDR: Stochastic gradient descent with warm restarts[EB/OL]. (2016-08-13) [2022-05-20]. https://arxiv.org/abs/1608.03983.
[22]	徐光柱, 匡婉, 李兴维, 等. YOLOv3与顶点偏移估计相结合的车牌定位[J]. 计算机辅助设计与图形学学报, 2021, 33(4): 569-579. XU G Z, KUANG W, LI X W, et al. License plate location based on YOLOv3 and vertex offset estimation[J]. Journal of Computer-Aided Design & Computer Graphics, 2021, 33(4): 569-579(in Chinese).
[23]	唐倩, 贺伟, 张林江. 基于空间转换网络的端到端车牌检测与识别[J]. 光电子·激光, 2021, 32(5): 524-531. doi: 10.16136/j.joel.2021.05.0423 TANG Q, HE W, ZHANG L J. End to end license plate detection and recognition based on spatial transformer network[J]. Journal of Optoelectronics·Laser, 2021, 32(5): 524-531(in Chinese). doi: 10.16136/j.joel.2021.05.0423
[24]	QIN S S, LIU S J. Towards end-to-end car license plate location and recognition in unconstrained scenarios[EB/OL]. (2020-08-25) [2022-05-20]. https://arxiv.org/abs/2008.10916.
[25]	SILVA S M, JUNG C R. A flexible approach for automatic license plate recognition in unconstrained scenarios[J]. IEEE Transactions on Intelligent Transportation Systems, 2021, 23(6): 5693-5703.
[26]	REDMON J, FARHADI A. YOLO9000: Better, faster, stronger[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE Press, 2017: 7263-7271.
[27]	ZOU Y, ZHANG Y, YAN J, et al. License plate detection and recognition based on YOLOv3 and ILPRNET[J]. Signal Image and Video Processing, 2021, 9: 312-320.
[28]	LI B, TIAN B, LI Y, et al. Component-based license plate detection using conditional random field model[J]. IEEE Transactions on Intelligent Transportation Systems, 2013, 14(4): 1690-1699.
[29]	YUAN Y, ZOU W, ZHAO Y, et al. A robust and efficient approach to license plate detection[J]. IEEE Transactions on Image Processing, 2016, 26(3): 1102-1114.