Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pass preprocess_image function to generators in evaluate.py #1290

Merged
merged 2 commits into from
Mar 2, 2020

Conversation

foghegehog
Copy link
Contributor

@foghegehog foghegehog commented Feb 25, 2020

I've encountered a problem similar to this issue: #647 (with custom backbone, mobilenetv3, not in the "zoo"). Here #1055 (comment) @mariaculman18 offered a solution that worked for me and seems to be reasonable. The author is not going to create a pull request, but I decided to do so, as other users with non-resnet50 backbones can be affected.

@codecov-io
Copy link

Codecov Report

Merging #1290 into master will increase coverage by <1%.
The diff coverage is 33%.

@@           Coverage Diff           @@
##           master   #1290    +/-   ##
=======================================
+ Coverage      23%     23%   +<1%     
=======================================
  Files          43      43            
  Lines        2535    2539     +4     
=======================================
+ Hits          576     577     +1     
- Misses       1959    1962     +3

@@ -141,6 +150,7 @@ def main(args=None):
# load the model
print('Loading model, this may take a second...')
model = models.load_model(args.model, backbone_name=args.backbone)
generator.compute_shapes = make_shapes_callback(model)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can also be passed to the constructor of generator. It does mean that the order of creating generators followed by loading of the model has to be the other way around, but I don't think this is an issue.

@hgaiser
Copy link
Contributor

hgaiser commented Feb 27, 2020

Thank you for the PR! I haven't gotten around to do this so it is much appreciated. Does the callback during training work properly? I could imagine it needs a similar treatment.

@foghegehog
Copy link
Contributor Author

@hgaiser

Thank you for the PR! I haven't gotten around to do this so it is much appreciated. Does the callback during training work properly? I could imagine it needs a similar treatment.

Do you mean the make_shapes_callback and that it should be passed directly to generators' constructors for all models except resnet50, instead of this:

# this lets the generator compute backbone layer shapes using the actual backbone model
if 'vgg' in args.backbone or 'densenet' in args.backbone:
train_generator.compute_shapes = make_shapes_callback(model)
if validation_generator:
validation_generator.compute_shapes = train_generator.compute_shapes

?

I'm not getting high mAP with my backbone during training, so it can indeed be an issue. But not for sure, as I'm just started and there can be other problems.

@hgaiser
Copy link
Contributor

hgaiser commented Mar 2, 2020

Ah it looks like preprocess_image is passed correctly. Are you getting different results with the evaluation callback versus running evaluate.py?

@foghegehog
Copy link
Contributor Author

Ah it looks like preprocess_image is passed correctly. Are you getting different results with the evaluation callback versus running evaluate.py?

Yes I do, when running without the fix. I'm getting 20-30-40% mAP in the evaluation of the first epochs during training, but in the evaluate.py restored snapshots always give 0.

It looks reasonable to apply the same image preprocessing during evaluation as in the training process, however I can't get why resnet50 works without it. May be the 'tf' preprocessing mode for mobilenet backbone instead of 'caffe' for resnet causes that?

@hgaiser
Copy link
Contributor

hgaiser commented Mar 2, 2020

It looks reasonable to apply the same image preprocessing during evaluation as in the training process

Oh yeah definitely, was just an oversight that this wasn't handled properly.

I can't get why resnet50 works without it.

It's because the default value follows resnet50 style preprocessing.

I was asking because I thought you meant that, even with this fix, you get different results whether you evaluate during training or evaluate after training. I get now that this isn't the case for you. Then everything looks fine to me, thanks again for the PR.

@hgaiser hgaiser merged commit a81b313 into fizyr:master Mar 2, 2020
@foghegehog
Copy link
Contributor Author

@hgaiser
Could you please also explain about make_shapes_callback and code snippet from train.py I've mentioned above?
if 'vgg' in args.backbone or 'densenet' in args.backbone:

Should there be vgg and densenet only or other backbones as well, except the resnet50?

@hgaiser
Copy link
Contributor

hgaiser commented Mar 5, 2020

The shapes of the feature pyramid for resnet50 and probably some others are well defined, ie. you can predict how they will end up. For others, like vgg and densenet they were more difficult to predict so the decision was made to write a function that, given a model, extracts the sizes of the feature pyramid. I recall this from memory, so I could be wrong on some details.

@foghegehog
Copy link
Contributor Author

The shapes of the feature pyramid for resnet50 and probably some others are well defined, ie. you can predict how they will end up. For others, like vgg and densenet they were more difficult to predict so the decision was made to write a function that, given a model, extracts the sizes of the feature pyramid. I recall this from memory, so I could be wrong on some details.

Wouldn't it be better to change the line:
if 'vgg' in args.backbone or 'densenet' in args.backbone:
with
if not args.backbone.startswith('resnet50):
?

As I see, there is only one guess_shape function. Is it designed specially for resnet50?

def guess_shapes(image_shape, pyramid_levels):
"""Guess shapes based on pyramid levels.
Args
image_shape: The shape of the image.
pyramid_levels: A list of what pyramid levels are used.
Returns
A list of image shapes at each pyramid level.
"""
image_shape = np.array(image_shape[:2])
image_shapes = [(image_shape + 2 ** x - 1) // (2 ** x) for x in pyramid_levels]
return image_shapes

@hgaiser
Copy link
Contributor

hgaiser commented Mar 5, 2020

I'm pretty sure other backbones (like mobilenet and efficientnet) work similarly as resnet50, so you would get some list of backbones there either way.

@foghegehog
Copy link
Contributor Author

I'm pretty sure other backbones (like mobilenet and efficientnet) work similarly as resnet50, so you would get some list of backbones there either way.

Ok, thank you!

kazushi-fa pushed a commit to kazushi-fa/keras-retinanet_rareplanes that referenced this pull request Aug 17, 2021
Pass preprocess_image function to generators in evaluate.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants