Before You Start
To install YOLOP dependencies:
pip install -qr https://github.com/hustvl/YOLOP/blob/main/requirements.txt # install dependencies
YOLOP: You Only Look Once for Panoptic driving Perception
Model Description
- YOLOP is an efficient multi-task network that can jointly handle three crucial tasks in autonomous driving: object detection, drivable area segmentation and lane detection. And it is also the first to reach real-time on embedded devices while maintaining state-of-the-art level performance on the BDD100K dataset.
Results
Traffic Object Detection Result
| Model | Recall(%) | mAP50(%) | Speed(fps) |
|---|---|---|---|
Multinet |
81.3 | 60.2 | 8.6 |
DLT-Net |
89.4 | 68.4 | 9.3 |
Faster R-CNN |
77.2 | 55.6 | 5.3 |
YOLOv5s |
86.8 | 77.2 | 82 |
YOLOP(ours) |
89.2 | 76.5 | 41 |
Drivable Area Segmentation Result
| Model | mIOU(%) | Speed(fps) |
|---|---|---|
Multinet |
71.6 | 8.6 |
DLT-Net |
71.3 | 9.3 |
PSPNet |
89.6 | 11.1 |
YOLOP(ours) |
91.5 | 41 |
Lane Detection Result
| Model | mIOU(%) | IOU(%) |
|---|---|---|
ENet |
34.12 | 14.64 |
SCNN |
35.79 | 15.84 |
ENet-SAD |
36.56 | 16.02 |
YOLOP(ours) |
70.50 | 26.20 |
Ablation Studies 1: End-to-end v.s. Step-by-step
| Training_method | Recall(%) | AP(%) | mIoU(%) | Accuracy(%) | IoU(%) |
|---|---|---|---|---|---|
ES-W |
87.0 | 75.3 | 90.4 | 66.8 | 26.2 |
ED-W |
87.3 | 76.0 | 91.6 | 71.2 | 26.1 |
ES-D-W |
87.0 | 75.1 | 91.7 | 68.6 | 27.0 |
ED-S-W |
87.5 | 76.1 | 91.6 | 68.0 | 26.8 |
End-to-end |
89.2 | 76.5 | 91.5 | 70.5 | 26.2 |
Ablation Studies 2: Multi-task v.s. Single task
| Training_method | Recall(%) | AP(%) | mIoU(%) | Accuracy(%) | IoU(%) | Speed(ms/frame) |
|---|---|---|---|---|---|---|
Det(only) |
88.2 | 76.9 | - | - | - | 15.7 |
Da-Seg(only) |
- | - | 92.0 | - | - | 14.8 |
Ll-Seg(only) |
- | - | - | 79.6 | 27.9 | 14.8 |
Multitask |
89.2 | 76.5 | 91.5 | 70.5 | 26.2 | 24.4 |
Notes:
- In table 4, E, D, S and W refer to Encoder, Detect head, two Segment heads and whole network. So the Algorithm (First, we only train Encoder and Detect head. Then we freeze the Encoder and Detect head as well as train two Segmentation heads. Finally, the entire network is trained jointly for all three tasks.) can be marked as ED-S-W, and the same for others.
Visualization
Traffic Object Detection Result
Drivable Area Segmentation Result
Lane Detection Result
Notes:
- The visualization of lane detection result has been post processed by quadratic fitting.
Deployment
Our model can reason in real-time on Jetson Tx2, with Zed Camera to capture image. We use TensorRT tool for speeding up. We provide code for deployment and reasoning of model in github code.
Load From PyTorch Hub
This example loads the pretrained YOLOP model and passes an image for inference.
import torch
# load model
model = torch.hub.load('hustvl/yolop', 'yolop', pretrained=True)
#inference
img = torch.randn(1,3,640,640)
det_out, da_seg_out,ll_seg_out = model(img)
Citation
See for more detail in github code and arxiv paper.
If you find our paper and code useful for your research, please consider giving a star and citation:
