[13pt] Look into OverClocking for some FIM pipeline py scripts that have multi-processing

[RobHanna-NOAA]

Note: Now part of 1377 EPIC: FIM Sys Admin Tasks (and a few related FIM tasks)

A good handful of our tools have multi processing in them. Do some tests on over-clocking which is letting the job number exceed the number of CPU's available. Most of our products make very little usage of CPU usage and Memory usage. With such poor usage of almost all CPU's in multi-processing, it will be pretty safe to over-clock and add more jobs than CPUs. Each product that uses multi-processing, will need testing and small code adjustments for this to happen. The amount of jobs for overclocking on each product will need to be figured our or at least a safe range. Keeping an eye on the EC2's CPU and Memory counter tools now included on many of our ECS's can help. Also watching docker stats can help figure this out.

Gain? Potentially very significant.

We have already tested this to some degree on the latest Catfim which saw performance gains of 20% but CatFIM has multi-proc in multi-proc so it was hard to be definitive on it, but overclocking clearly had a major impact.

I added multi-proc and overclocking to test_case_by_hydro_id.py and safely had it process a full batch in 2 hrs and 40 mins. Various testing without mp and without overclocking in different enviros showed durations ranging from 32 hrs, 24, 20 adn 12 hrs all pre-mp and overclocking. Overclocking tests against the tool after adding mp also showed major improvements in the range of 50% improvements.