[DRAFT] FLAIR#2 Dataset and Datamodule Integration

[MathiasBaumgartinger]

FLAIR#2 dataset

The FLAIR #2 <https://github.com/IGNF/FLAIR-2> dataset is an extensive dataset from the French National Institute of Geographical and Forest Information (IGN) that provides a unique and rich resource for large-scale geospatial analysis.
The dataset is sampled countrywide and is composed of over 20 billion annotated pixels of very high resolution aerial imagery at 0.2 m spatial resolution, acquired over three years and different months (spatio-temporal domains).

The FLAIR2 dataset is a dataset for semantic segmentation of aerial images. It contains aerial images, sentinel-2 images and masks for 13 classes.
The dataset is split into a training and test set.

Dataset features:

* over 20 billion annotated pixels
* aerial imagery
    * 5x512x512
    * 0.2m spatial resolution
    * 5 channels (RGB-NIR-Elevation)
* Sentinel-2 imagery
    * 10-20m spatial resolution
    * 10 spectral bands
    * snow/cloud masks (with 0-100 probability)
    * multiple time steps (T)
    * Tx10xWxH, T, W, H are variable
* label (masks)
    * 512x512
    * 13 classes

Dataset classes:

0: "building",
1: "pervious surface",
2: "impervious surface",
3: "bare soil",
4: "water",
5: "coniferous",
6: "deciduous",
7: "brushwood",
8: "vineyard",
9: "herbaceous vegetation",
10: "agricultural land",
11: "plowed land",
12: "other"  

If you use this dataset in your research, please cite the following paper:

* https://doi.org/10.48550/arXiv.2310.13336

Implementation Details

NonGeoDataset, __init()__

After discussions following #2303, we decided that at least until faulty mask data are fixed the flair2 ds will be of type NonGeoDataset. Other than with common NonGeoDatasets, FLAIR2 exposes a use_toy and use_sentinel argument. The use_toy-flag will instead use the toy data which is a small subset of data. The use_sentinel argument on the other hand decides whether a sample includes the augmented sentinel data provided by the maintainers of FLAIR2.

_verify, _download, _extract

As each of the splits/sample-types (i.e. [train, test], [aerial, sentinel, labels] are contained in a individual zip download, download and extraction has to happen multiple times. On the other hand, the toy dataset is contained in a singular zip. Furthermore, to map the super-patches of the sentinel data to the actual input image, a flair-2_centroids_sp_to_patch.json is required, which has to be equally has to be downloaded as an individual zip.

_load_image, _load_sentinel, _load_target

For storage reasons, the elevation (5th band) of the image is stored as a uint. The original height thus is multiplied by 5. We decided to divide the height by 5 to get the original height, to make the trained model more usable for other data. See Questions please.

As mentioned previously, additional metadata has to be used to get from the sentinel.npy to the actual area. Initially for debugging reasons, we implemented to return not the cropped image but the original data and the cropping-slices (i.e. indices). Consequently, the images can be plot in a more meaningful matter. Otherwise, the resolution is so low that one can hardly recognize features. This was crucial for debugging to find the correct logic (classic y, x instead of x, y ordering mistake). We do not know if this is smart for "production code". See Questions please.
Moreover, the dimensions of the sentinel data $T \times C=10 \times W \times H$ vary both $T$ and $W$, $H$. This is problematic for the datamodule. We have not done extensive research, but the varying dimensions seem to bug the module. Disabling the use_sentinel-flag will make the module work.

The labels include values from 1 to 19. The datapaper clearly mentions grouping classes $> 13$ into one class other due to underrepresentation. We followed this suggestion. Furthermore, rescaling from 0 to 12 was applied. See Questions please.

Questions

• Do you consider the Elevation rescaling as distortion of the dataset? Shall I exclude it? The argument for it would be easier re-usability on new datasets.

For storage optimization reasons, this elevation information is multiplied by a factor of 5 and encoded as a 8bit unsigned integer datatype.

• How shall we load/provide sentinel data? As cropped data or any other way. I do not see the current implementation as fit for production.

  • Also, how do we want to plot it? The small red rectangle in the example plot above is the actual region. The low resolution is quite observable there.

• Shall we rescale the Classes to start from 0? Shall we group the classes as suggested in the datapaper?

• Check integrity in download_url does not seem to work (in unit-tests), why?

  • I have to call an own check_integrity call otherwise it passes, even if md5s do not match.

• The github actions on the forked repo produce a magic ruff error (https://github.com/MathiasBaumgartinger/torchgeo/actions/runs/11687694109/job/32556175383#step:7:1265). Can you help me resolve this mystery?

TODOs/FIXMEs

• Extend tests for toy datasets and apply md5 check
• Find correct band for plotting sentinel
• Datamodule cannot handle sentinel data yet

[nilsleh]

Thanks a lot for the PR, here are just some answers at the top of my head, others might have additional opinions.

Questions

* Do you consider the Elevation rescaling as distortion of the dataset? Shall I exclude it? The argument for it would be easier re-usability on new datasets.

Not sure about this one, but your proposition seems reasonable and as long as this is clearly documented, I would think it is fine.

* How shall we load/provide sentinel data? As cropped data or any other way. I do not see the current implementation as fit for production.

I think that depends on how, the large sentinel tile is expected to be used/useful for a model. The very low resolution area does not seem useful intuitively. I think having the flag you implemented is nice.

If you provide the entire tile, there will have to be a collate or cropping function in the datamodule, otherwise the dataloader cannot return consistent batches, as by default it tries to batch individual samples into a tensor by dict key.

  * Also, how do we want to plot it? The small red rectangle in the example plot above is the actual region. The low resolution is quite observable there.

I actually like that approach.

* Shall we rescale the Classes to start from 0? Shall we group the classes as suggested in the datapaper?

Classes should definitely be ordinally mapped to start from 0, otherwise there will be issues with the CrossEntropyLoss etc. Grouping is also okay since it follows the paper.

[nilsleh]

something seems off here, since the entire csv file changed, and not just the FLAIR2 addition.

[adamjstewart]

Probably line endings

[MathiasBaumgartinger]

Should I try to change to the original line endings, or does it not really matter?

[nilsleh]

For the toy data we usually use smaller image sizes like 32 just to speed up tests.

[nilsleh]

The definition of patch_size is useful if there is an additional kornia Resize augmentation under self.aug, however, we would also have to consider how this plays with the larger sentinel tile. I am still not sure what purpose the additional Sentinel 2 should provide in terms of model performance if the high res image only covers such a tiny area.

[MathiasBaumgartinger]

I am by no means an expert, but have been reading some papers in the domain the last year. I suppose, the increased feature dimensions provided by the hyperspectral bands of sentinel-2 have shown to increase model performance.

Furthermore, in the datapaper [p. 6] of FLAIR2 the authors argue:

The interest of the extended spatial information provided by the Sentinel-2 super-patches is particularly visible in the last two rows of Figure 4. Indeed, the location on a beach or on a lake is difficult to determine from the aerial image alone, and could easily be confused with the sea for example in the last row.

The model leveraging the additional information outperform a standard baseline model:

Model	mIoU	Building	Pervious Surface	Impervious Surface	Bare Soil	Water	Coniferous	Deciduous	Brushwood	Vineyard	Herbaceous Vegetation	Agricultural Land	Plowed Land
U-Net	0.547	0.8009	0.4727	0.6988	0.3076	0.7985	0.5758	0.7014	0.2392	0.6012	0.4653	0.5449	0.3583
U-T&T	0.5594	0.8285	0.4980	0.7204	0.2982	0.8009	0.6041	0.7189	0.2541	0.6580	0.4684	0.5478	0.3157
U-T&T best	0.5758	0.8368	0.4995	0.7446	0.3959	0.7952	0.6339	0.7239	0.2485	0.6678	0.4750	0.5513	0.3381

[nilsleh]

Thanks for pointing that out, I should have looked at the paper myself... In that case, if we want to support a resize, it would be more involved, if we wont to resize the sentinel image as well but keeping the resolution difference. Or still just provide the Sentinel 2 scene as context.

[MathiasBaumgartinger]

My suggestion would be to initialize a sentinel scaling of 0 to 1 in the model. This in turn could be injected into the loading function.

• 0 would resize the image to the exact extent of the aerial images.
• 1 would take the entire image
• anything between would linearly scale from aerial to original extent

This way, both a straight-forward streamlined model (i.e. uses sentinel as aerial extent as just another source of information) and more sophisticated models that fuse classification at a later point in time could be used. Any suggestions for improvement?

[nilsleh]

change to FLAIR2

[nilsleh]

add versionadded for 0.7

[nilsleh]

versionadded missing

[nilsleh]

this can go above the init in class attributes

[MathiasBaumgartinger]

Hi! Sorry for the late response.

First of all, thanks for the review. I will get to work on the proposed changes ;-)