nova examples fail with EndpointNotFound error

[mchlumsky]

The nova examples (compute-versions, list-servers-paginated and list-servers) fail with EndpointNotFound
It happens on some of our clouds (from liberty to stein) but not all of them.

I suspect it's because the nova endpoint URLs contains the project_id because that's the only difference I see between the successes and failures. Version 2 or 2.1 doesn't make any difference, I see it fail in both cases when there is a project_id in the url.

The python openstack sdk works with all these clouds which is why I think it's ok to have the project_id in the endpoint URLs. In newer versions of openstack, the project_id is no longer necessary in the endpoint URLs but it's still supported AFAIK.

Here is the error with debug logging (with redacted hostnames, etc...). The endpoint url as seen in the catalog is https://host.domain.tld:8774/v2/.

    Finished dev [unoptimized + debuginfo] target(s) in 0.12s
     Running `target/debug/examples/compute-versions`
[2020-07-07T01:32:28Z DEBUG osauth::session] No cached information for service compute, fetching
[2020-07-07T01:32:28Z DEBUG osauth::identity] Requesting a catalog endpoint for service 'compute', filters EndpointFilters { interfaces: ValidInterfaces { items: [Public, Public, Public], len: 1 }, region: None }
[2020-07-07T01:32:28Z DEBUG reqwest::connect] starting new connection: https://host.domain.tld:5000/
[2020-07-07T01:32:28Z DEBUG hyper::client::connect::dns] resolving host="host.domain.tld"
[2020-07-07T01:32:28Z DEBUG hyper::client::connect::http] connecting to x.x.x.x:5000
[2020-07-07T01:32:28Z DEBUG hyper::client::connect::http] connected to x.x.x.x:5000
[2020-07-07T01:32:28Z DEBUG hyper::proto::h1::io] flushed 170 bytes
[2020-07-07T01:32:28Z DEBUG hyper::proto::h1::io] flushed 206 bytes
[2020-07-07T01:32:28Z DEBUG hyper::proto::h1::io] read 8192 bytes
[2020-07-07T01:32:28Z DEBUG hyper::proto::h1::io] parsed 8 headers
[2020-07-07T01:32:28Z DEBUG hyper::proto::h1::conn] incoming body is content-length (70698 bytes)
[2020-07-07T01:32:28Z DEBUG reqwest::async_impl::client] response '201 Created' for https://host.domain.tld:5000/v3/auth/tokens
[2020-07-07T01:32:28Z DEBUG hyper::proto::h1::io] read 8192 bytes
[2020-07-07T01:32:28Z DEBUG hyper::proto::h1::io] read 7706 bytes
[2020-07-07T01:32:28Z DEBUG hyper::proto::h1::io] read 16000 bytes
[2020-07-07T01:32:28Z DEBUG hyper::proto::h1::io] read 16384 bytes
[2020-07-07T01:32:28Z DEBUG hyper::proto::h1::io] read 14680 bytes
[2020-07-07T01:32:28Z DEBUG hyper::proto::h1::conn] incoming body completed
[2020-07-07T01:32:28Z DEBUG hyper::client::pool] pooling idle connection for ("https", host.domain.tld:5000)
[2020-07-07T01:32:28Z DEBUG osauth::identity] Received a token expiring at 2020-07-07 09:32:28 +00:00
[2020-07-07T01:32:28Z DEBUG osauth::endpointfilters] Received Endpoint { interface: "public", region: "RegionOne", url: "https://host.domain.tld:8774/v2/<project-id>" } for compute
[2020-07-07T01:32:28Z DEBUG osauth::protocol] Fetching compute service info from https://host.domain.tld:8774/v2/<project-id>
[2020-07-07T01:32:28Z DEBUG reqwest::connect] starting new connection: https://host.domain.tld:8774/
[2020-07-07T01:32:28Z DEBUG hyper::client::connect::dns] resolving host="host.domain.tld"
[2020-07-07T01:32:28Z DEBUG hyper::client::connect::http] connecting to x.x.x.x:8774
[2020-07-07T01:32:28Z DEBUG hyper::client::connect::http] connected to x.x.x.x:8774
[2020-07-07T01:32:28Z DEBUG hyper::proto::h1::io] flushed 336 bytes
[2020-07-07T01:32:30Z DEBUG hyper::proto::h1::io] read 240 bytes
[2020-07-07T01:32:30Z DEBUG hyper::proto::h1::io] parsed 4 headers
[2020-07-07T01:32:30Z DEBUG hyper::proto::h1::conn] incoming body is content-length (52 bytes)
[2020-07-07T01:32:30Z DEBUG hyper::proto::h1::conn] incoming body completed
[2020-07-07T01:32:30Z DEBUG hyper::client::pool] pooling idle connection for ("https", host.domain.tld:8774)
[2020-07-07T01:32:30Z DEBUG reqwest::async_impl::client] response '404 Not Found' for https://host.domain.tld:8774/v2/<project-id>
thread 'main' panicked at 'Failed to get major version: Error { kind: EndpointNotFound, message: "Endpoint for service compute was not found", status: None }', examples/compute-versions.rs:21:25
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

[dtantsur]

Ouch, I honestly thought we handled this case :( Apparently, not enough.

[dtantsur]

@mchlumsky could you test 1cb0926 and see if it helps?

[mchlumsky]

I tested with the 0.3.1 release, and it works on all the clouds I have access to (from Newton all the way up to Stein) except one very old Liberty cloud.

$ cargo run --example list-servers
    Finished dev [unoptimized + debuginfo] target(s) in 0.06s
     Running `target/debug/examples/list-servers`
thread 'main' panicked at 'Failed to list servers: Error { kind: ResourceNotFound, message: "<html>\n <head>\n  <title>404 Not Found</title>\n </head>\n <body>\n  <h1>404 Not Found</h1>\n  The resource could not be found.<br /><br />\n\n\n\n </body>\n</html>", status: Some(404) }', examples/list-servers.rs:34:32
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

I won't hold it against you if you don't care about such an old release ;)

Thanks for the fix!

[dtantsur]

I won't hold it against you if you don't care about such an old release ;)

Could you file a separate bug with full logging? I cannot promise fixing it (it's been a while since I last saw Liberty), but maybe it will be obvious.

[mchlumsky]

Bug opened: #20