In recent CONUS baseline testing, it is realized that a specific sub-directory is needed to store the ngen output data files, which count to over a million. Current form of realization files writes these files to project directory, which makes working in the directory extrmely incovennient. For example, a simple ls command takes many minutes to complete. Then you cannot spot the file you are looking for. In general, such situation would arise for any relatively large basin.
Code exists that accommodate this. But requires the user to create the sub-directory in advance, otherwise, it default to project directory. A updated version of the code added a safety feature for case the user forget to create the needed sub-directory.
To alleviate the problem, all current realization files need to add the following line:
"output_root": "./output_dir/"
at the top level of the realization json tree. That will make ngen to write all the output files to ./output_dir. You are free to choose your own favorite name instead of "output_dir". At the time of this writing, if you forget to create ./output_dir/ in advance before running, the default output directory is the project directory.
The updated code (to be merged) throws an error if you do not created the output_root before starting ngen to remind you to create the subdirectory before things get messy. It may require all updated realization to be merged first.
Current behavior
Large amount of output files when running ngen for CONUS or any large basin.
Expected behavior
Output files are written to a user specified directory to keep the project directory clean.
Steps to replicate behavior (include URLs)
- Running
ngen in framework for a relatively large basin
Screenshots
In recent CONUS baseline testing, it is realized that a specific sub-directory is needed to store the ngen output data files, which count to over a million. Current form of realization files writes these files to project directory, which makes working in the directory extrmely incovennient. For example, a simple
lscommand takes many minutes to complete. Then you cannot spot the file you are looking for. In general, such situation would arise for any relatively large basin.Code exists that accommodate this. But requires the user to create the sub-directory in advance, otherwise, it default to project directory. A updated version of the code added a safety feature for case the user forget to create the needed sub-directory.
To alleviate the problem, all current realization files need to add the following line:
"output_root": "./output_dir/"at the top level of the realization json tree. That will make ngen to write all the output files to ./output_dir. You are free to choose your own favorite name instead of "output_dir". At the time of this writing, if you forget to create
./output_dir/in advance before running, the default output directory is the project directory.The updated code (to be merged) throws an error if you do not created the output_root before starting ngen to remind you to create the subdirectory before things get messy. It may require all updated realization to be merged first.
Current behavior
Large amount of output files when running ngen for CONUS or any large basin.
Expected behavior
Output files are written to a user specified directory to keep the project directory clean.
Steps to replicate behavior (include URLs)
ngenin framework for a relatively large basinScreenshots