Bug: New version (2023.10.14) doesn not handle Sentinel 2 dataset correctly.

[SOutmani]

s1 = hf.Sentinel1(aoi, start_time, end_time)
s1.n_images
The sentinel 1 data search works fine.

s2 = hf.Sentinel2(aoi, start_time, end_time)
s2.n_images
However the Sentinel 2 data search as written above, always gives a 0 number of images, even though we know for a fact that there are (and the previous version finds them, too).

[KMarkert]

Can you please share a minimal reproducible example that includes what the aoi, start_time , and end_time are so I can investigate?

[KMarkert]

@SOutmani Gentle ping to see if you can please provide a minimal working example of the issue you are seeing so I can replicate and investigate.

[SOutmani]

Hi @KMarkert , thanks for the reminder.
I am trying out different dates and aoi's and the function 's2.n_images' now does work as expected, so the initial issue seems to be solved!

However, I am encountering a new problem with this specific case only:

geom = ee.Geometry.Rectangle([10.81,43.60,11.57,44.01])
startTime = datetime.datetime(2023,10,29)
endTime = datetime.datetime(2023,11,14)
s2 = hf.Sentinel2(geom, startTime, endTime)
s2.n_images
3
s2.dates

EEException: Collection.reduceColumns: Error in map(ID=20231109T101231_20231109T101230_T32TPP):
Image.select: Parameter 'input' is required.

Note that the error seems to be related to the image acquired on nov 9th at 10:18:42 UTC. Perhaps it's not an issue related to hydrafloods but to the google earth dataset itself.

[biplovbhandari]

I couldn't setup the local installation because I was getting the following issue with python 3.8 or 3.9

Python 3.9.18 | packaged by conda-forge | (main, Aug 30 2023, 03:53:08)
[Clang 15.0.7 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ee
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/bbhandar/miniforge3/envs/hydrafloods-dev/lib/python3.9/site-packages/earthengine_api-0.1.378-py3.9.egg/ee/__init__.py", line 16, in <module>
    from ee import _cloud_api_utils
  File "/Users/bbhandar/miniforge3/envs/hydrafloods-dev/lib/python3.9/site-packages/earthengine_api-0.1.378-py3.9.egg/ee/_cloud_api_utils.py", line 18, in <module>
    from google import auth
ImportError: cannot import name 'auth' from 'google' (/Users/bbhandar/miniforge3/envs/hydrafloods-dev/lib/python3.9/site-packages/google_cloud_storage-2.13.0-py3.9.egg/google/__init__.py)

But I think the issue is that not all s2 image have the associated probability mask with it (e.g. https://code.earthengine.google.com/638145f913d6628e45b4554a2bab7462). Again haven't been able to test it locally because of the above issue, but this maybe able to solve it.

[KMarkert]

@biplovbhandari I am pretty sure you are correct about the s2 cloud probabilities not having full coverage of the s2 collection.

I am thinking we should transition from using s2 cloud probabilities to start using the CloudScore+ data for QA masking the S2 collections.

[biplovbhandari]

+1

[kyle-woodward]

I am experiencing the original issue that @SOutmani mentions:

import hydrafloods as hf
aoi = ee.Geometry.Rectangle([10.81,43.60,11.57,44.01])

s2composite = hf.Sentinel2(aoi,'2019-01-01','2019-12-31',)
print('hf.Sentinel2 n_images',s2composite.n_images)

s2_test = (ee.ImageCollection("COPERNICUS/S2_SR_HARMONIZED")
.filterDate('2019-01-01','2019-12-31')
.filterBounds(aoi))
print('COPERNICUS/S2_SR_HARMONZIED n_images',s2_test.size().getInfo())

Output:

hf.Sentinel2 n_images 0
COPERNICUS/S2_SR_HARMONZIED n_images 144

It looks like its the use_qa() fn for Sentinel2 is the issue?

s2composite = hf.Sentinel2(aoi,'2019-01-01','2019-12-31',use_qa=False)
print('hf.Sentinel2 NOQA n_images',s2_test.size().getInfo())

Output:
hf.Sentinel2 NOQA n_images 144

I know @biplovbhandari mentions his PR #47 above, not sure if that was intended to resolve this case?

If we fully replace the use_qa fn with the Cloud Score + method I think it would solve it. but I think we'd need to provide qa band (cs or cdf) and threshold kwargs with sensible defaults..