optimize compressed storage of downloaded data into GTIFF

[ivanhanigan]

from @ahsparks

writeRaster(raster, file = "out location", format = "GTiff", dataType = "INT2S",

options = c("COMPRESS=LZW"))

to save space with GTiffs. Check all the options with dataType to see what works best. http://artax.karlin.mff.cuni.cz/r-help/library/raster/html/dataType.html

[ivanhanigan]

@ahsparks
I added your code as function compress_gtifs to the develop branch, and added you as author to package.

Started demo of the workflow in development
https://github.com/swish-climate-impact-assessment/awaptools/blob/develop/README.md

Cheers!

[adamhsparks]

I completely missed this, @ivanhanigan.

Thanks! 👍

[ivanhanigan]

@adamhsparks I wonder what you think about

1. not creating Gtifs subdir
2. removing the .grid file after

Today this was what I wanted to do so I hacked the following, my thought though is it is better to make your compress_gtifs function do the unzip, compress and clean up.
swish-climate-impact-assessment/AWAP_GRIDS@6411403

require(devtools)
install_github("swish-climate-impact-assessment/awaptools", ref = "develop")
require(awaptools)
require(rgdal)
workdir <- "data"
setwd(workdir)
startdate <- "2014-01-01"
enddate <- "2014-02-28"
load_monthly(start_date = startdate, end_date = enddate)
filelist <- dir(pattern = "grid.Z$")
for(fname in filelist){unzip_monthly(fname, aggregation_factor = 1)}
compress_gtifs(indir = getwd())
system("rm *.grid")
system("mv GTif/* ./")

[adamhsparks]

@ivanhanigan is there any reason to add the extra step to generate the GTiff files rather than just going straight to that step during the download? You're not giving the user an option to keep the .grid file if the first step only downloads and the second step/function does the unzipping and writes the GTiff file onto disk.

If you do that we could download the .Z files to a tempdirectory , ?tempdir, unzip them in that tempdirectory, import them using the raster package and then write a GTiff file out to disk. Once the session is done, the .grid.Z and .grid file disappear, no need to unlink them.

[ivanhanigan]

@adamhsparks Thanks, yes - no need to keep grid file so I'll try to
implement tempdir as you suggest. Cheers.
On 28/08/2016 6:34 PM, "Adam H. Sparks" [email protected] wrote:

@ivanhanigan https://github.com/ivanhanigan is there any reason to keep
the .grid files around and add the extra step to generate the GTiff files
rather than just going straight to that step during the download? You're
not giving the user an option to keep the .grid file if the first step only
downloads and the second step/function does the unzipping and writes the
GTiff file onto disk.

If you do that we could download the .Z files to a tempdirectory ,
?tempdir, unzip them in that tempdirectory, import them using the raster
package and then write a GTiff file out to disk. Once the session is done,
the .grid.Z and .grid file disappear, no need to unlink them.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#1 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABPsp6dqOFXQC7Idk4rlhawEbqna8vvUks5qkUgvgaJpZM4IP5Vu
.