These dirs, /proc and /sys (or subsets we collect) are the biggest contributor to disk space for our crucible results. Individually compressing the files is not effective because there are so many which are tiny. Using tar Jcf reduces this by ~95%.
Tar/compress should probably happen at tool-stop, then during postprocess, we will likely need to un-tar, then discard the untarred files (unless there is a way to read files directly from a tar + compressed archive in perl or python).
These dirs, /proc and /sys (or subsets we collect) are the biggest contributor to disk space for our crucible results. Individually compressing the files is not effective because there are so many which are tiny. Using tar Jcf reduces this by ~95%.
Tar/compress should probably happen at tool-stop, then during postprocess, we will likely need to un-tar, then discard the untarred files (unless there is a way to read files directly from a tar + compressed archive in perl or python).