*** Code underlying the publication: Efficient Sequential Neural Network based on Spatial-Temporal Attention and Linear LSTM for Robust Lane Detection Using Multi-Frame Images ***
Authors: Dong, Y., Patil, S., van Arem, B., & Farah, H

TU Delft, Faculty of Civil Engineering and Geosciences, Department of Transport and Planning

Corresponding author: Yongqi Dong
Contact Information:
qiyuandream@gmail.com


***General Introduction***
This is the source code of the paper:

 Patil, S.#, Dong, Y.#,*, Farah, H, & Hellendoorn, J. (2024). Efficient Sequential Neural Network Based on Spatial-Temporal Attention and Linear LSTM for Robust Lane Detection Using Multi-Frame Images (Under Review)



How to use the codes
(1) Download tvtLANE Dataset:

You can download this **dataset** from the link in the '**Dataset-Description-v1.2.pdf**' file. 

BaiduYun：https://pan.baidu.com/s/1lE2CjuFa9OQwLIbi-OomTQ  passcodes：tf9x

Or 

Google Drive: https://drive.google.com/drive/folders/1MI5gMDspzuV44lfwzpK6PX0vKuOHUbb_?usp=sharing   

 

The **pretrained model** is also provided in the "/model" folder, named as 98.48263448061671_RAd_lr0.001_batch70_FocalLoss_poly_alpha0.25_gamma2.0_Attention_UNet_LSTM.pth .

 

(2) Set up

## Requirements

PyTorch 0.4.0 

Python 3.9 

CUDA 8.0 

## Preparation

### Data Preparation

The dataset contains 19383 continuous driving scenes image sequences, and 39460 frames of them are labeled. The size of images is 128*256.

The training set contains 19096 image sequences. Each 13th and 20th frame in a sequence are labeled, and the image and their labels are in “clips_13(_truth)” and “clips_20(_truth)”. All images are contained in “clips_all”. 

Sequences in “0313”, “0531” and “0601” subfolders are constructed on TuSimple lane detection dataset, containing scenes in American highway. The four “weadd” folders are added images in rural road in China. 

The testset has two parts: Testset #1 (270 sequences, each 13th and 20th image is labeled) for testing the overall performance of algorithms. The Testset #2 (12 kinds of hard scenes, all frames are labeled) for testing the robustness of algorithms.  

To input the data, we provide three index files(train_index, val_index, and test_index). Each row in the index represents for a sequence and its label, including the former 5 input images and the last ground truth (corresponding to the last frame of 5 inputs).

The dataset needs to be put into a folder with regards to the location in index files, (i.e., txt files in "./data/". The index files should also be modified add cording to your local computer settings. If you want to use your own data, please refer to the format of our dataset and indexes.

 

(3) Training

Before training, change the paths including "train_path"(for train_index.txt), "val_path"(for val_index.txt), "pretrained_path" in config_Att.py to adapt to your environment. 

Choose the models (UNet_ConvLSTM | SCNN_UNet_ConvLSTM | SCNN_UNet_Attention) as the default one which is also indicated by default='UNet-ConvLSTM' thus you do not need to make change for this. And adjust the arguments such as class weights (now the weights are set to fit the tvtLANE dataset), batch size, learning rate, and epochs in config_Att.py. You can also adjust other settings, e.g., optimizer, check in the codes for details.

Then simply run: train.py. If running successfully, there will be model files saved in the "./model" folder. The validating results will also be printed.

 

(4) Test

To evaluate the performance of a trained model, please select the trained model or put your own models into the "./model/" folder and change "pretrained_path" in test.py according to the local setting, then change "test_path" to the location of test_index.txt, and "save_path" for the saved results.

Choose the right model that would be evaluated, and then simply run: test.py.


The quantitative evaluations of Accuracy, Precision, Recall, and F1 measure would be printed, and the lane detection segmented results will be saved in the "./save/" folder as pictures.



Datasets description
The original dataset of the TuSimple Lane Detection Challenge consists of 3,626 training and 2,782 testing one-second clips that are collected under different driving conditions. Each clip is extracted into 20 continuous frames, and only the last frame, i.e., the 20th frame, is labelled as the ground truth. Additionally, Zou et al. (2020) added the label of the 13th frame and augmented the TuSimple dataset with 1,148 additional clips (with also the 13th and 20th frames labelled) regarding rural road driving scenes collected in China. Moreover, rotation, flip, and crop operations are employed for data augmentation, and finally, a total number of (3,626 + 1,148) × 4 = 19,096 sequences were produced, among which 38,192 frames are labelled with ground truth.
For testing, there are 2,782 testing clips in the original TuSimple dataset. While in tvtLANE, there are two different testing sets, namely, Testset #1 which is based on the original TuSimple test set for normal driving scene testing, as well as Testset #2 which contains 12 challenging driving scenarios for testing challenging scenes and assessing the model robustness.
In the training phase, three different sampling strides, with an interval of 1, 2, and 3 frames respectively, were adopted to adapt to different driving speeds which also augment the training samples by three times, whereas in the test phase, the sampling stride was set as a fixed interval of 1 frame.
Detailed descriptions of the two datasets and sampling settings can be found in (Dong et al., 2023; Zou et al., 2020).

Reference:
Dong, Y., Patil, S., van Arem, B., Farah, H., 2022. A hybrid spatial–temporal deep learning architecture for lane detection. Comput. Civ. Infrastruct. Eng. 1–20. https://doi.org/10.1111/mice.12829
Zou, Q., Jiang, H., Dai, Q., Yue, Y., Chen, L., Wang, Q., 2020. Robust lane detection from continuous driving scenes using deep neural networks. IEEE Trans. Veh. Technol. 69, 41–54. https://doi.org/10.1109/TVT.2019.2949603

 
Authors
Yongqi Dong (y.dong-4@tudelft.nl), Sandeep Patil, Haneen Farah, Hans Hellendoorn


