DPIF-Net: a dual path network for rural road extraction based on the fusion of global and local information

Dataset

Although the currently available public road datasets cover a wide range of road categoriesin cities, suburbs and rural areas in many countries worldwide, they contain many normal roads and few rural roads. Therefore, they are not suitable for analysis with a special focus on rural roads, but they can be used as test data for model generalization performance.

In this study, a rural road dataset was constructed based on the GaoFen-2 (GF-2) satellite. The GF-2 satellite carries a range of sensors, including a Panchromatic and Multispectral sensor (PMS), a wide-field-of-view sensor (WFV), and a hyperspectral sensor (HSI), which provide high-resolution imagery with spatial resolutions ranging from 0.8 m to 16 m. The images for the PMS sensor (450 to 900 nm) at 0.8 m were utilized in this article. The images include three regions covering an area of over 2,300 square kilometers: the junction between Jiancaoping District and Gujiao City in Shanxi characterized by imagery measuring 36,500 × 34,258 pixels, covering an approximate area of 752 square kilometers, the junction between Anyang and Shijiazhuang cities in Hebei characterized by imagery measuring 39,695 × 31,311 pixels, covering an approximate area of 795 square kilometers, and the junction area of Guangzhou and Foshan in Guangdong characterized by imagery measuring 36,020 × 32,431 pixels, covering an approximate area of 747 square kilometers. For example, the study area at the junction between Taiyuan and Jiancaoping District and mask samples, as illustrated in Fig. 1. All GF-2 data in our study is sourced from the China Centre for Resources Satellite Data and Application.

Nevertheless, the considerable size of the satellite images poses challenges in terms of efficient data loading, potentially leading to a substantial increase in training duration. Moreover, the absence of masks within the original dataset introduces complexities in conducting effective supervised training. To overcome these challenges, the images underwent cropping to achieve dimensions of 512 × 512 pixels initially. Subsequently, manual annotation based on image texture was executed to generate corresponding masks. In the end, we generated a dataset similar to the examples shown in Fig. 2. To address the potential sample imbalance, our dataset excluded the images containing abundant higher levelroads, consequently encompassing intricate details of rural road attributes, such as tree coverage, agricultural field irrigation channels, and road incompleteness. These adjustments were intended to enhance the model’s performance in extracting low-grade rural roads. After preprocessing the data, 5,501 samples remained, with 5,421 as training samples, 40 as validation samples, and 40 as test samples.

In addition, experiments were carried out on two public datasets, DeepGlobe (Demir et al., 2018) and Massachusetts (Mnih, 2013). However, it is imperative to emphasize that these two datasets contain a limited quantity of low-grade roads in comparison to a substantial volume of regular highways. They are specifically employed to enhance our model’s extraction performance and validate findings. The images in the DeepGlobe road dataset come from three countries, namely, India, Thailand, and Indonesia, and include multiple imaged scenes covering an area of over 2,220 square kilometers, such as cities, villages, wilderness, suburbs, seashores, and tropical rainforests. The ground resolution of the images is 0.5 m per pixel, and the image size is 1,024 × 1,024 pixels. There are a total of 6,226 images, of which 4,976 are designated for training and 1,250 are designated for testing. In this study, following Zhu et al. (2021), the original images were cropped to a resolution of 512 × 512 pixels with an overlap of 256 pixels. Finally, a total of 5,000 images for training, 40 images for validation and 4,500 images for testing were obtained.

The Massachusetts road dataset consists of 1,171 aerial images of the Massachusetts region, which cover a wide variety of urban, suburban, and rural regions and an area of over 2,600 square kilometers. With a spatial resolution of 1 m per pixel, the images in this dataset have a size of 1,500 × 1,500 pixels and are composed of red, green, and blue channels. Similarly, the original data were cropped into nonoverlapping images with a resolution of 512 × 512 pixels. Finally, 3,744 images were used for training, 30 images were used for validation, and 196 images were used for testing. Moreover, all images with large white blank areas were removed manually.

Details of the model structure

We propose Dual Path Information Fusion Network (DPIF-Net) to improve the performance of rural road extraction by exploring the potential of combining the capabilities of transformers and CNNs for road segmentation. The schematic structure of DPIF-Net is displayed in Fig. 3. First, the top encoder branch uses a transformer to model global road information in the input remote sensing image, while the other encoder branch uses convolution operations to extract local details of roads and process spatial and channel information. Second, the feature information of the two branches is effectively fused. Finally, each layer of the decoder fuses high-level features from the previous layer with low-level features from the convolutional branch and gradually upsamples the image to the original resolution to obtain a binary image containing only roads.

Local detail information encoder based on a CNN

In DPIF-Net, we propose a convolution module called the local detail feature extraction (LDFE) block as shown in Fig. 4. This block is composed of three parts to efficiently extract road features while keeping the number of network parameters low.

In part A, a traditional 3 × 3 convolution is applied to the input feature map to extract preliminary feature information without altering its resolution, resulting in a feature map that is four times larger than the input in the channel dimension. Then, the obtained features are spliced and fused using a skip connection.

In part B, each channel of the feature map is processed separately using a 3 × 3 convolution to extract semantic feature information, and a leaky-ReLU (Maas, Hannun & Ng, 2013) activation layer is used to obtain nonlinear features, as expressed in Eq. (1). (1)${\displaystyle Leak-ReLU\left(x\right)=\left\{\begin{array}{cc}x\hfill & \left(x>0\right)\hfill \\ \alpha x\hfill & \left(x<=0\right)\hfill \end{array}\right.}$ α is a small gradient value, which is set to 0.2 in this article.

In part C, the feature information from part B is combined, cross-channel interaction is achieved using a 1 × 1 convolution, and then another leaky-ReLU activation layer is applied.

Global information encoder based on a Transformer module

The ASPP module was originally introduced to increase the receptive field without sacrificing too much resolution, allowing for the preservation of image details as much as possible (Chen et al., 2018a). To process multiscale road feature information extraction, we propose improvements to the ASPP module. Specifically, there are three different dilated convolution modules and a global adaptive pooling module, which can extract more feature information, as shown in Fig. 5. Moreover, different dilation ratios are used to obtain abstract feature information at different scales, which provides a basis for the subsequent integration of features and accurate road segmentation. In this way, the modified ASPP module obtains more abstract semantic information and road topology information and achieves better robustness; it can accept input feature images of any size and finally produce output of a fixed size.

In addition to the improvements to the CNN module proposed above, a Trans block is proposed to employ the Transformer architecture at the beginning of the model to capture global context information while avoiding interference with the CNN branch. Specifically, the Trans block can compensate for the shortcomings of the CNN in capturing global context information.

To process image blocks with a transformer, we convert them into one-dimensional data by using a fully connected (FC) layer. Due to the expansion into one-dimensional data, the position information is lost; therefore, the input feature data are converted from index numbers into a one-hot encoding matrix to maintain the position relationships of the sequence. Moreover, a random weight matrix is right-multiplied to complete the input position embedding. This method ensures that each token in the sequence has a unique position representation, which is crucial for the transformer to capture long-range dependencies.

The schematic structure of an individual transformer block is shown in Fig. 6 (Dosovitskiy et al., 2020). The Transformer encoder mainly consists of alternating multihead self-attention (MSA) layers and multilayer perceptron (MLP) blocks. Before these two modules, layer normalization (LN) is applied for normalization, and a residual connection structure is used. The MLP block consists of two FC layers, in which the Gaussian error linear unit (GeLU) function is applied for nonlinear activation to obtain nonlinear features. The definition of the GeLU (Hendrycks & Gimpel, 2016) activation function is shown in Eq. (2). (2)${\displaystyle GeLU\left(x\right)=xP\left(X\le x\right)=x\times \varphi \left(x\right)X\sim N\left(0,1\right)}$ x is the input, and X is a random variable that follows a Gaussian distribution with mean 0 and variance 1. $P\left(X\le x\right)\varphi \left(x\right)$ is the cumulative distribution of the Gaussian normal distribution of x, for which there is no analytical expression; instead, its approximate calculation method is shown in Eq. (3). (3)${\displaystyle GeLU\left(x\right)=\frac{1}{2}x\left(1+tanh\left(\sqrt{\frac{2}{\pi }}\left(x+0.044715x^3\right)\right)\right).}$

In our work, the number of attention heads is set to 16, assuming that the input image is x (x ∈ i^H×W×C), where H, W and C are the height, width and channels, respectively, of x. The original image block feature x_p (x_p ∈ i^N×P2×C) is obtained by expansion into one-dimensional data, x_p ∈ i^N×P2×C, where N is the number of small P ×P patches into which the original input image is cropped. N and P satisfy the relationship shown in Eq. (4). (4)${\displaystyle N=\frac{H\times W}{P^2}.}$

Then, after dimension embedding and nonlinear mapping of the GeLU function, the i-th image block feature z ⁱ can be expressed as shown in Eq. (5). (5)${\displaystyle z^i=GeLU\left(x_p^{i}E\right)}$ E ∈ i^P2×C×D. After the corresponding position encoding, the vector z input into the transformer (TF) can be expressed as shown in Eq. (6). (6)${\displaystyle z=\left[GeLU\left(x_p^{1}E\right),GeLU\left(x_p^{2}E\right),\dots ,GeLU\left(x_p^{N}E\right)\right]+E_{pos}}$ E_pos ∈ i^N×D. Finally, the input vector z is subjected to the calculations shown in Eq. (7) and Eq. (8). (7)${\displaystyle z^{\prime }=MSA\left(LN\left(z\right)\right)+z}$ (8)${\displaystyle z_1=MLP\left(LN\left(z^{\prime }\right)\right)+z^{\prime }.}$

Multiple TF blocks are used to build the Trans block, as shown in Fig. 7, and the generated output of global road information from the original remote sensing image, z _out, is expressed as shown in Eq. (9). (9)${\displaystyle z_{out}=z_1+z_2+z_3}$

Decoder based on context information fusion

After the two different kinds of information discussed above are obtained, the final branch combines the two different kinds of road semantic information to fuse the global and local road features, as shown in part A of Fig. 8. The high-level feature information is extracted using two 3 × 3 convolutional layers without changing the feature map resolution. Group normalization (GN) is used to normalize the features, and Leaky-ReLU is used as the nonlinear activation function to capture the nonlinear characteristics of the roads. This process fully integrates the semantic information from the two different sources, resulting in more comprehensive road semantic feature information that covers the feature information of various road categories.

To solve the problem that the spatial location information of the roads is not obvious due to multiple convolutions, this article adopts the design concept of the U-Net network structure and employs skip connections in the decoder to transfer the low-level feature information from the output of the LDFE module to the corresponding decoder port for splicing, as depicted in part B of Fig. 8. Subsequently, the features are fused layer by layer via 3 × 3 convolutional layers, and the feature map’s resolution is incrementally restored via bilinear interpolation until it finally reaches the original image resolution.

Finally, Wu & He (2018) proposed group normalization (GN) to improve the efficiency of training rather than batch normalization (BN) BN works effectively for a relatively large batch size. However, a small batch size leads to inaccurate estimation of the batch statistics, and reducing the batch size for BN dramatically increases the model error. GN can achieve approximately the same accuracy performance as BN for a moderate batch size and outperforms other normalization variants because it can still achieve a small error rate even when the batch size undergoes large fluctuations. The GN calculation is expressed as shown in Eq. (10): (Wu & He, 2018). (10)${\displaystyle y_i=\frac{\gamma }{{\sigma }_i}\left(x_i-{\mu }_i\right)+\beta }$ γ and β are trainable scale and shift parameters, respectively, and μ_i and σ_i in Eq. (10) are the mean and standard deviation computed as shown in Eq. (11): (Wu & He, 2018) (11)${\displaystyle {\mu }_i=\frac{1}{m}\sum _{k\in S_i}{x}_k,{\sigma }_i=\sqrt{\frac{1}{m}\sum _{k\in S_i}{\left(x_k-{\mu }_i\right)}^2+\varepsilon }}$ ɛ is a small nonzero constant. S_i is the set of pixels, and m is the size of S_i.

Experimental design description

DPIF-Net was trained on an RTX 3090 GPU. At the beginning of training, we utilized common data augmentation techniques, including translation, rotation, flipping, scaling, and random color jitter as shown in Fig. 9, which contributed to improving the model’s robustness and performance. The model implementation was based on PyTorch, the optimizer for all structures was Adam, the initial learning rate was set to 0.0002, and the batch size during training was set to 2. The MSELoss was used to calculate the loss.

To assess the performance of DPIF-Net in rural road extraction, we conducted three primary comparative experiments across distinct datasets. Each of these comparative experiments entailed a juxtaposition between mainstream models and ours. In addition, the performance of DPIF-Net was compared with that of the U-Net, SegNet, D-LinkNet, and DeepLabv3+ models on the above three road datasets. In these experiments, our goal was to observe its capabilities for extracting roads in complex scenarios through evaluation metrics, assessing both the completeness and accuracy of road extraction. In the discussion, we assessed the parameter sizes among the models and the visual assessments of both completeness and accuracy. Furthermore, the ablation experiments are conducted to observe the contributions of its two branches, which are designed to identify which modules are most crucial for DPIF-Net’s performance.

Experimental evaluation metrics

To evaluate the effectiveness of our model, four common metrics are selected: intersection over union (IoU), precision, recall, and F₁ score (F₁). High IoU indicates the model’s accuracy in predicting the location and shape of roads, which is particularly crucial for assessing the model’s ability to recognize roads (Lian et al., 2020). The IoU is calculated as shown in Eq. (12). (12)${\displaystyle IoU=\frac{TP}{TP+FP+FN}}$ TP, FP, TN, and FN denote true positives, false positives, true negatives, and false negatives, respectively.

In the context of rural roads, where non-road areas often constitute a significant proportion, leading to an imbalance between positive and negative samples in the dataset, F₁ score becomes particularly crucial for assessing performance under such conditions, which are calculated as shown in Eq. (13). (13)${\displaystyle F_1=\frac{2\times precision\times recall}{precision+recall}}$ The precision and recall are employed to evaluate the model’s capability to correctly identify roads, which are calculated as shown in Eq. (14) and Eq. (15), respectively. (14)${\displaystyle precision=\frac{TP}{TP+FP}}$ (15)${\displaystyle recall=\frac{TP}{TP+FN}.}$

Road extraction experiment based on our road dataset

The results on our dataset are shown in Fig. 10. Through detailed comparisons, we find that in Examples 1–2 and Examples 4–7, DPIF-Net generates fewer broken road segments (misidentified connected vectors) than the other four models. At the same time, the occluded parts caused by trees can be completely extracted to ensure the connectivity of the roads. In Example 3, DPIF-Net extracts the least erroneous information, yielding results almost consistent with the ground truth, whereas the other four models incorrectly extract road information in this example.

More detailed comparison results are presented in Table 1. DPIF-Net achieves the best performance in two metrics, namely, IoU and F₁ score, reaching 61.40% and 76.08%, respectively. In addition, DPIF-Net reaches 77.27% precision, which is 1.05% lower than the highest score and 0.6% lower than U-Net’s score. DPIF-Net also reaches 74.94% recall, which is 0.08% lower than D-LinkNet’s highest recall score of 75.02%, representing a small, almost negligible fluctuation.

Compared with U-Net, the IoU, recall and F₁ values of DPIF-Net are increased by 3.34%, 5.41%, and 2.61%, respectively. Compared with DeepLabv3+, the IoU, recall and F1 values are increased by 4.8%, 7.83% and 3.8%, respectively. Compared with D-LinkNet, the IoU, precision and F1 values increased by 0.35%, 0.64%, and 0.26%, respectively. Compared with SegNet, the IoU, precision, recall and F1 values are increased by 7.52%, 1.52%, 9.83%, and 6.05%, respectively. The largest increases in IoU, precision, recall, and F1 score reach 7.52%, 1.52%, 9.83%, and 6.05%, respectively. In the extraction of rural roads, the crucial aspects lie in correctly identifying roads and preserving their completeness, both reflected in the IoU and F1 metrics. DPIF-Net achieves the highest IoU and F1 on the lower level roads, highlighting its superiority in extracting rural roads, which underscores its proficiency in correctly extracting rural roads and recognizing road shapes.

Road extraction experiment based on the DeepGlobe road dataset

A second experiment was conducted on the DeepGlobe road dataset, and the results of the visual assessment comparison are presented in Fig. 11. Figure 11 shows the detailed effects of rural road segment extraction for 8 examples. In the first example, compared with U-Net and SegNet, DPIF-Net extracts more complete road information. In contrast, D-LinkNet produces more mistakenly extracted road segments and more fractures than DPIF-Net. In the 2nd to 6th examples, the extraction results of U-Net and SegNet show more broken segments, while DPIF-Net achieves almost the same road integrity as D-LinkNet, whereas D-LinkNet misextracts more road segments than DPIF-Net. In the 7th and 8th examples, the integrity of the roads extracted by DPIF-Net is higher than that of the other three methods.

The detailed evaluation index comparisons are shown in Table 2. From the indicator data in Table 2, it can be seen that the model with the worst comprehensive performance is DeepLabv3+, which has the lowest score in all metrics. Its recall is lower by more than 30%, and the other three metrics are lower by nearly 20%. DPIF-Net achieves the best results in terms of IoU, precision and F₁ score, reaching values of 57%, 74.76% and 72.61%, respectively. However, its recall is lower than those of D-LinkNet and SegNet at only 70.58%. D-LinkNet has the highest recall score of 89.62%.

In summary, DPIF-Net improves the IoU by 0.6–20.08%, the precision by 0.08–20.48%, and the F₁ score by 0.49–18.68% on the DeepGlobe road dataset. Its recall is weaker than those of SegNet and D-LinkNet but 6.96% higher than that of U-Net and 16.99% higher than that of DeepLabv3+.

Road extraction experiment based on the Massachusetts road dataset

To further test the generalization ability of the proposed DPIF-Net, a similar experiment was carried out on the Massachusetts road dataset. The visual assessment of the compared methods on this dataset is shown in Fig. 12. All the models almost completely extract the road information, but from Fig. 12, it can be seen that the other models do not extract some road details completely enough, resulting in various fractures. Comprehensive comparisons show that the proposed model is better than the others in extracting many details.

Table 3 displays the detailed road extraction results on the Massachusetts road dataset, revealing that DPIF-Net surpasses the other models in terms of IoU, precision, and F1 score, with values of 53.82%, 82.48%, and 70%, respectively. Although DeepLabv3+ achieves the highest recall score of 63.92%, the IoU, F1 score and precision of DeepLabv3+ are significantly lower than those of all other models, with differences of 10–40%.

Compared to other models, DPIF-Net integrates global and local information more extensively, enabling it to capture more features and thereby enhance its recognition capabilities. DPIF-Net achieves superior performance in predicting road accuracy and completeness, demonstrated by attaining maximum values in IoU, precision, and F1 on the DeepGlobe and Massachusetts datasets, showcasing its advantages in overall road extraction.

Evaluation of the generalization performance and road representation ability of the proposed model

First, we evaluate the generalization performance and road representation ability of the proposed model. To gain deeper insight into the inner workings of DPIF-Net, feature heatmaps for five different stages in the decoder are displayed in Fig. 13 to illustrate how the model extracts roads. The results from the three different datasets indicate that DPIF-Net effectively learns road features with clear boundaries. The feature maps show that the encoder learns various levels of feature representations of the input image, and the decoder combines these representations to generate accurate road masks. As the decoder performs upsampling four times, the semantic information of the extracted roads becomes increasingly abstract, which highlights the ability of DPIF-Net to learn high-level features. These results demonstrate that DPIF-Net not only has excellent road extraction performance but also possesses good robustness and feature representation capabilities; thus, it shows promising potential for various applications in remote sensing image analysis.

Discussion on the quantity of model parameters

Furthermore, a comparison of the parameters used in the experiments for each model provides insight into their respective strengths and weaknesses. Table 4 presents a comprehensive overview of the parameters for each model, indicating that DPIF-Net has the fewest parameters, with a data volume of only 63.9 MB, which is similar to that of U-Net. While D-LinkNet achieves better feature extraction in some cases, it also has a significantly larger number of parameters due to its use of ResNet101 as the encoder and deeper network layers. On the other hand, DeepLabv3+ also has a high number of parameters due to the use of Xception as the encoding network, but it performs poorly in road extraction experiments. SegNet, with approximately twice as many parameters as DPIF-Net, also yields inferior experimental results for road segmentation. These comparisons highlight the trade-off between the number of parameters and the effectiveness of a model for road extraction tasks. While having more parameters may improve feature extraction, it also increases a model’s complexity and computational cost. DPIF-Net, with its simple yet effective structure and relatively few parameters, proves to be a promising model for road extraction from remote sensing images.

Ablation study

To better understand the contribution of each branch in the encoder part of DPIF-Net to the road extraction results, an ablation study was conducted using our dataset. A comparison of the results reveals that the two branches make different levels of contributions to the road extraction results, as presented in Table 5. Specifically, the branch that incorporates the CNN-based feature extractor makes a stronger contribution to the final road extraction results than the branch that employs the transformer-based feature extractor.

Notably, due to the strict parameter limitations of the final model presented in this article, only three transformer blocks were used in the transformer-based feature extractor. This resulted in suboptimal performance in comparison to the CNN-based feature extractor. However, the performance of the transformer-based feature extractor could be improved by stacking more transformer blocks, albeit at the cost of dramatically increasing the number of parameters.

While DPIF-Net has demonstrated impressive performance by effectively combining a CNN and a transformer, there are several potential areas for improvement in future research on road extraction using DPIF-Net. First, more advanced architectures for the transformer block could be explored to further enhance the model’s performance without significantly increasing the number of parameters. Second, modified attention mechanisms or other forms of spatial information modeling might improve the model’s ability to capture fine details and connectivity information in the road network. Third, methods to improve the accuracy of road boundary delineation and reduce false positive rates could further increase the model’s utility in practical applications. Overall, these potential areas for improvement could help advance the state of the art in road extraction using deep learning models.