-
Notifications
You must be signed in to change notification settings - Fork 276
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Learning custom data always 0% #321
Comments
Let it train to epoch 100. And also check the output on in the tensorboard.
Then you open chrome/firefox to the localhost and check the image tab. Check some other issues here to see what sort of output you should get. |
lower the learning rate a tad. The 0% is about the data it loads, not the perf. Sorry. I should update this. Can you try on a single image? Normally I test this first. |
The train belief guess should look like the gt above it, can you run it for longer. run it for like 1000 epochs. |
I have changed the background of the input image and added symmetry information and trained on the following image: I run it for 1000 epochs and in the end the result was the following: It seems much better! Do you think that now I can train on a bigger dataset with more instances of objects/distractors? |
Are you aware of the symmetries in your object? Check the generating data with symmetries. But yeah this looks good now. DOPE takes a while to train, so you will have to patient, like on a 60k image dataset I train for ~30 epochs. |
I will adjust everything and try to run with more images if the PC allows me. Thank you! |
I have defined an object, as the Ketchup one. I have a generated 1000 images with the following command:
python single_video_pybullet.py --nb_frames 1000 --scale 0.015 --path_single_obj ~/Deep_Object_Pose/scripts/nvisii_data_gen/models/iros_block/google_16k/textured_simple.obj --nb_distractors 0 --nb_object 5
And I was able to obtain the 1000 images with n object like this:
![00962](https://private-user-images.githubusercontent.com/37381508/271665214-693b55fd-5b72-4ed8-9ea2-89b9736e4d1c.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk5Nzg5OTEsIm5iZiI6MTcxOTk3ODY5MSwicGF0aCI6Ii8zNzM4MTUwOC8yNzE2NjUyMTQtNjkzYjU1ZmQtNWI3Mi00ZWQ4LTllYTItODliOTczNmU0ZDFjLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA3MDMlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNzAzVDAzNTEzMVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWE1NGMyYWQzYWU1YjVhNzgyYTFmM2ZhOGM0ZDE0OWE4MWU0YTRlYzQ5N2ZjNDEzZTQ3MjBhMGI2NjA0YzViOWMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.0P1uBUQRC-yOTNMBL_ngYaBQiaLEuP3Gm9Ahk9N-hfg)
Then, I tried to use the train with the aforementioned set of images with:
python -m torch.distributed.launch --nproc_per_node=1 train.py --network dope --epochs 25 --batchsize 10 --outf tmp/ --data ../nvisii_data_gen/output/output_example/
I tried with different epochs, batchsize and generating more times the set of images, howver I obtain always 0% for each epoch:
I am kind of new with learning so I do not know in deep the details, what I am doing wrong ?
In the csv files inside
output
folder, there are no data, only the header. In addition, I add the flag--save
I have no results.Thank you !
The text was updated successfully, but these errors were encountered: