VBET Performance Issues

[MattReimer]

This is mainly for @KellyMWhitehead and @philipbaileynar but @joewheaton might find it interesting too.

EDIT: I was really wrong about this. I didn't realize that VBET had crashed immediately after the layer path calcs so these numbers don't include all the rest of VBET (which is significant). I'll update the ticket when this finally completes.

TL;DR:

• This was for the smallest HUC8 I could find. I'm running a larger one now but that will take time.
• This discussion is about performance. the next discussion will be about errors.

The problem

• Calculating level paths of VBET is taking way too long!
  • For huc 17060304 processing time was 154 minutes for 352 level paths (by comparison our old VBET took 8 minutes for the same HUC).
  • This doesn't include the work @KellyMWhitehead is doing to push the processing down into raster space but that may not affect the time to calculate centerlines because:
• The chief offenders are:
  • The centerline calculation: unsurprising. It's all vector ops and voronoi polygons
  • raster_merge(): I find this surprising. More investigation is needed.
• As the HUC grows VBET calculation time rises exponentially, making even modestly large HUC8s prohibitively expensive to compute.
• The minimum possible calculation time per level path is 20.3 seconds. This is probably too long since a modestly-sized huc like 05010003 with 6674 level paths the shortest amount of time this could take is 37 hours and that's if each level path is less than 100m long (which is a ridiculous assumption)

Possible solutions

• Do nothing. VBET will take days (maybe a week for big HUC8s). We just need to wait it out.
• Don't calculate small level paths.
• Find a faster way to do centerlines
• Move more of the process from vector down to raster space. Diminishing returns here. There's a hard limit around the minimum time to calculate a level path (see minimum possible calculation time reference above)
• That's all I have off-hand. Ideas?

Deep dive:

Why is it taking so long? Where is the time being spent?

Generally: Calculating individual level paths

Inside the level path calculation it breaks down like this:

When we chart processing time against ascending path length we get an exponential curve. This is the part that has me worried. You can see that calculating centerlines is the chief offender. raster_merge is still a problem for the minimum time but it doesn't scale with path length so that's hopeful.

Here's a rough histogram of how the level path lengths are distributed

There's some good news though. The next chart shows that there's nothing increasing from one iteration to the next. If this chart were ticking up over time it would indicate a memory leak or something that gets recursively harder as we do more of them

GIS Stuff

I also imported my debug stats into QGIS and married it to the flowlines so we can see where the problems exist (For 17060304 there aren't that many problems but in my next HUC this will be more revalatory)

Here are all the VBET errors labeled by type. Green is pass, red is fail. The reason you don't see any red lines are that they are tiny. The error label is there just to show you how many there are and that they exist.

Edit: Talking with @philipbaileynar I realized the red isn't necessarily an error. The red COULD represent level paths we don't need to calculate and thus discard.

Here are all the level paths assigned random colors so we can see where they are.

Here are the calculation times heatmap

Here is a level path length heatmap

---

Notes and resources:

1. All of this is using 17060304 because anything else took too long to get any data. I'll start a new ticket for a bigger HUC but it will just represent whatever I can get done overnight.

2. For the QGIS images above You can download the Riverscapes project that produced these graphics. There's a QGS project you can open with all of this symbolized to explore it further (or test my assumptions).

3. The charting and spreadsheeting is here if you want to dive in.

[MattReimer]

Just talked to Philip and we discussed strategy:

• Remove centerline calculation for now
• @KellyMWhitehead to investigate whether or not all the little level paths need to be considered or if we can filter them out more aggressively. Is there anything else we can filter out with fcode etc...
• We sort implicitly by length but I think we need to actually sort explicitly by length.
• @MattReimer to investigate why the raster_merge is taking so long.

[joewheaton]

@MattReimer I love this ticket. Super, super interesting and informative.

I am following the basic logic of lets do everything we can in raster space and I love it...

The chief offenders are:

• The centerline calculation: unsurprising. It's all vector ops and voronoi polygons

I am really confused though by the centerline thing. You say this is taking forever because it is using vornoi polygons, but @philipbaileynar came up with a really cool raster based way of deriving centerlines for each level path, based on i) a Euclidian distance (i.e. raster) from the edge clipped to inside the valley bottom mask (also know reason that mask can't stay as a raster mask instead of being converted to vector one), and ii) a least-cost path analysis (i.e. also raster), and then only converting the path from raster to polyline at the end. If this isn't what we're doing, why aren't we? It made a ton of sense back then and I don't know why we wouldn't do something different.

[joewheaton]

I am also a little confused on how many level paths you are running things for. The test HUC might not be a great one for big enough, but can someone confirm what I thought we were doing (or where I have it wrong):

1. Start with longest level path...
2. run VBET
3. Threshold VBET liklihood to get mask (now raster instead of polygon) representing valley bottom
4. Identify ALL levelpaths entirely contained within the valley bottom mask and remove from analysis.
5. Go to next longest level path, and repeat steps 2-4 until no more level paths.

If that is the case, for a bunch of watersheds it seems like we should be getting rid of a lot of levelpaths representing the side channels and canals and such (BTW... can't we just remove ALL canals and perhaps connectors from consideration)? So it seems like you might start with a lot (e.g. 6674 like you pointed out), but then for those with bigger valley bottoms, after the first pass through, it decreases off a lot (maybe down to 50%? I don' t know) until we start getting to the smaller level paths that don't really have much? What am I missing?

I think we should be reporting for ourselves some diagnostics (much like your graphs above), but also things like:

• How many level paths to start?
• How many level paths actually end up getting VBET run for them?
• Maybe a plot showing how number of level paths processed tapers off as we find ones we do not need to consider in the analyses.
• Level path lengths
• Level path computation time
  These will be really interesting (especially as we start running LiDAR datasets).

[MattReimer]

The test HUC might not be a great one for big enough,

Agreed and I'm not stopping there. It's just the first one I could get to finish. I've got a bunch more running in CYbercastor as I type this. Results as they appear. I'm also uploading projects with the QGS files I use to produce these maps so you can download them yourself and play with the data.

I think we should be reporting for ourselves some diagnostics

I'm working on getting all those metrics. Level path length, computation time and the number that fall off for various reasons. The rest are easy to implement and we really should be doing it.

For now I produce a quick and dirty CSV file but it would take almost no effort to bake it into a geopackage that we can link to a map layer.

[MattReimer]

Here is an update now that 4 HUC8's have completed (you can download them all with qgs projects and symbolized debug data)

• VBET for HUC 17060304 (Warehouse) (WebRAVE)
• VBET for HUC 10190004 (Warehouse) (WebRAVE)
• VBET for HUC 10190010 (Warehouse) (WebRAVE)
• VBET for HUC 10190005 (Warehouse) (WebRAVE)

The first part of the discussion only talked about level paths (in this legend I've labeled it for(level_path). Here is the rest of the breakdown for where VBET is spending its time.

NOTE: 05010003 is still running.

[MattReimer]

An update:

Yesterday's fix to write rasters inside-out instead of outside-in had some positive results

A nice benefit here is that it seems to affect larger HUCs more than smaller ones

This is because we've optimized the cost of raster writing and tied it to the area of the level path bounds

So now the breakdown looks a little like this (apples/apples comparison isn't totally possible here because I changed the labels and granularity of the pie slices)

---

Oh, and here are the download links. The projects have the debug tables in the geopackage but I didn't add the QGS projects this time.

• VBET for HUC 10190010: (Warehouse) (WebRAVE)
• VBET for HUC 17060304: (Warehouse) (WebRAVE)
• VBET for HUC 10190004: (Warehouse) (WebRAVE)

[MattReimer]

NOTE: Two regressions here. The first one is why WebRAVE looks so bad. in the above examples

• VBET NODATA problems #634
• VBET: Level path fail on HUC 10190002 #635

[MattReimer]

Ok. I'm marking this discussion "answered" because we've gotten VBET as fast as possible given the week we had to work on it.

Note that the warehouse links above may stop working as we delete these projects and replace them with new ones. I'm creating a new discussion for the results of this work.

[joewheaton]

Thank you so much @MattReimer for your efforts on this. I'm very excited to see where things go. @shelbysawyer and @lauren-herbine and @jtgilbert, FYI!

[MattReimer]

They go here: 😎
#637

[MattReimer]

I'll just leave this here. It's not complete but it indicates that we've gotten most of the smaller HUCs down to a manageable (but still very long) processing time:

[MattReimer]

Finally, here's a chart of RAM and CPU usage on one of the larger HUCs we ran

What I'm looking for here:

• The graphs are totally flat. Nothing is trending upwards over time 👍
• The memory consumption stays well below the 16Gb we allocated 👍
• The CPU spikes aren't that concerning. Could be some weirdness with the way AWS provisions shared CPU usage.