Support finding mutliple data layers during dataset exploration for simpler dataset formats without a metadata file

[MichaelBuessemeyer]

Detailed Description

The upcoming changes of #7912 allow the exploration to work recursively and to look for a dataset within subpaths of the given path by the user. This works well for dataset formats that have a metadata file describing the whole dataset and the location of all its layers (e.g. like a datasource-properties.json file.) However, the exploration of simpler formats with such a metadata file usually only finds some mag of one single layer during the exploration.

If a dataset would look like this:

• dataset
  • layer 1
    • mag 1
      • shard 1
        • actual data files
      • shard 2
        • actual data files
      • ...
    • mag 2
      • shard 1
        • actual data files
      • shard 2
        • actual data files
      • ...
  • layer 2
    • mag 1
      • shard 1
        • actual data files
      • shard 2
        • actual data files
      • ...
    • mag 2
      • shard 1
        • actual data files
      • shard 2
        • actual data files
      • ...

The exploration current code would only find layer 1 & mag 1 as a layer. The other mags and layers would not be found in case of a simpler dataset format.

Thus, the exploration could be extended to look for additional layers and mags and group them correctly.

For reference:

• Recursive exploration of remote datasets #7912 (comment)
• Recursive exploration of remote datasets #7912 (comment)
• https://scm.slack.com/archives/C5AKLAV0B/p1721129347974129