FastAPI for Landsat vessel detection

[pbeukema]

• Add api entry point to Dockerfile
• integration tests built CI not a dummy for now (add github workflows tests in CI in different PR)
• ...

[yawenzzzz]

After force push, the log history is cleaned up.

[Hgherzog]

• integration tests built CI not a dummy for now (add github workflows tests in CI in different PR)

@pbeukema I am doing github actions in a seperate PR

[yawenzzzz]

@pbeukema I couldn’t add you as a reviewer, but could you take a look at this PR?​

[yawenzzzz]

@mikerjacobi Here's the fastapi for landsat.

[favyen2]

I added a test and it looks like everything is working well but originally we were thinking the json_path would be optional since the JSON is also returned in the response body, but right now it got changed to be required in class LandsatRequest (needs to be str | None = None to be optional). Same for scratch_path and crop_path but those are less important to be optional.

[favyen2]

json_path should be optional since Skylight can read the JSON from the response body.

[yawenzzzz]

Yes, made json_path optional now.

[favyen2]

I think it would be nice for crop_path to be optional as well. If you don't make it optional then in the test I added, crop_path should be changed to be a subdirectory of the scratch_path, otherwise it writes the crops in the current directory.

[yawenzzzz]

Fixed. Also leave it as None in the test, so that we can query API just with a scene_id.

[favyen2]

I think scratch_path is fine to be required.

[favyen2]

[yawenzzzz]

I added a test and it looks like everything is working well but originally we were thinking the json_path would be optional since the JSON is also returned in the response body, but right now it got changed to be required in class LandsatRequest (needs to be str | None = None to be optional). Same for scratch_path and crop_path but those are less important to be optional.

OK, I made all three arguments Optional for api, we may just need the returned json response. And if specified, they can be saved as well.

[favyen2]

The test was failing for me with two issues:

• tmp_dir.cleanup only exists when doing tmp_dir = tempfile.TemporaryDirectory(), but it was using with tempfile.TemporaryDirectory() as tmp_dir, in the latter case it is just a string while in the former it is object where the directory name (string) can be accessed with tmp_dir.name.
• b8_fname/rgb_fname were used but not defined if crop_path is not passed.

It seems to be working now. I excluded it from test because it needs GPU, so I put it in tests/integration_slow/.

[yawenzzzz]

The test was failing for me with two issues:

• tmp_dir.cleanup only exists when doing tmp_dir = tempfile.TemporaryDirectory(), but it was using with tempfile.TemporaryDirectory() as tmp_dir, in the latter case it is just a string while in the former it is object where the directory name (string) can be accessed with tmp_dir.name.
• b8_fname/rgb_fname were used but not defined if crop_path is not passed.

It seems to be working now. I excluded it from test because it needs GPU, so I put it in tests/integration_slow/.

Thanks for the fix! I still got the following error, and it seems the prediction pipeline took very long (> 20 minutes) to run on the test scene. The materializing ran fast, but the preparing windows for classifier took long to complete (more than 1K windows here).

Loaded model weights from the checkpoint at gs://rslearn-eai/projects/rslearn-landsat-recheck/phase123_20240919_01_copy/checkpoints/epoch=27-step=864.ckpt
Predicting: |                                 | 0/? [00:00<?, ?it/s]ERROR:api_main:Unexpected error during prediction pipeline: received 0 items of ancdata

[favyen2]

Could it be not enough shared memory?

[yawenzzzz]

The test was failing for me with two issues:

• tmp_dir.cleanup only exists when doing tmp_dir = tempfile.TemporaryDirectory(), but it was using with tempfile.TemporaryDirectory() as tmp_dir, in the latter case it is just a string while in the former it is object where the directory name (string) can be accessed with tmp_dir.name.
• b8_fname/rgb_fname were used but not defined if crop_path is not passed.

It seems to be working now. I excluded it from test because it needs GPU, so I put it in tests/integration_slow/.

Thanks for the fix! I still got the following error, and it seems the prediction pipeline took very long (> 20 minutes) to run on the test scene. The materializing ran fast, but the preparing windows for classifier took long to complete (more than 1K windows here).

Loaded model weights from the checkpoint at gs://rslearn-eai/projects/rslearn-landsat-recheck/phase123_20240919_01_copy/checkpoints/epoch=27-step=864.ckpt
Predicting: |                                 | 0/? [00:00<?, ?it/s]ERROR:api_main:Unexpected error during prediction pipeline: received 0 items of ancdata

After reducing batch_size, the issue resolved.