compare_to_field function error

[perovskite13]

I had trouble using compare_to_field function from glmtools package.
Referring to the sample code, I have gave "nc_file" a string path to "output.nc" and "field_file" a string path to "my_field_data.tsv". I had also formatted "my_field_data.tsv" to proper column headings accordingly : DateTime, Depth, wTemp.

Nevertheless, I still have trouble returning the overall RMSE value of water temperature data and a time series of obs vs mod data with the following codes respectively :

could not return RMSE value

It returned "NaN"

temp_rmse <- compare_to_field(nc_file, field_file, metric = 'water.temperature', as_value = FALSE)

did not return obs vs modeled values

It returned "No data available in table" instead

temp_rmse <- compare_to_field(nc_file, field_file, metric = 'water.temperature', as_value = TRUE)

I wonder if you could enlighten me on this issue and let me know what did I miss.
Your advices and tips are much appreciated! Thank you!

[jordansread]

Thanks @perovskite13 . This function doesn't do much validation on the files coming in, so the error isn't very helpful. Would you mind emailing me at [email protected] and sending the .tsv so I can try it out? I will document progress on this issue, but email is probably best for the file if you are willing to share.

[jordansread]

Hi @perovskite13 I am not sure if this is the only issue, but it definitely is a start:

The file has the row index, and it should not be there. So instead, format the file as:

If you wrote this file in R w/ write.table I think you just need to

write.table(data, filename, ..., row.names = FALSE)

[perovskite13]

Thank you very much for your reply. Unfortunately, I think that is not the only issue. I have done changes and wrote :
write.table(uniqueLBG,"LBG507_temp_depth_field.tsv", quote = FALSE, sep='\t', row.names = FALSE)

Nevertheless, I received warnings below, which I'm not sure whether they are the reasons that affect the function. I also wonder why I kept receiving error messages for "timezone" even though I had made sure to set "tz=UTC" for all input files i.e. inflow, outflow & meteorology file.

Warning messages:
1: In strptime(xx, f <- "%Y-%m-%d %H:%M:%OS", tz = tz) :
unknown timezone 'Etc/GMT--10'
2: In as.POSIXct.POSIXlt(x) : unknown timezone 'Etc/GMT--10'
3: In strptime(x, f, tz = tz) : unknown timezone 'Etc/GMT--10'
4: In as.POSIXct.POSIXlt(as.POSIXlt(x, tz, ...), tz, ...) :
unknown timezone 'Etc/GMT--10'
5: In double(totvarsize) :
Reached total allocation of 3977Mb: see help(memory.size)
6: In double(totvarsize) :
Reached total allocation of 3977Mb: see help(memory.size)
7: In strptime(xx, f <- "%Y-%m-%d %H:%M:%OS", tz = tz) :
unknown timezone 'Etc/GMT--10'
8: In as.POSIXct.POSIXlt(x) : unknown timezone 'Etc/GMT--10'
9: In strptime(x, f, tz = tz) : unknown timezone 'Etc/GMT--10'
10: In as.POSIXct.POSIXlt(as.POSIXlt(x, tz, ...), tz, ...) :
unknown timezone 'Etc/GMT--10'
11: In as.POSIXlt.POSIXct(x) : unknown timezone 'Etc/GMT--10'
12: In as.POSIXct.POSIXlt(e2) : unknown timezone 'Etc/GMT--10'
13: In as.POSIXlt.POSIXct(x) : unknown timezone 'Etc/GMT--10'
14: In as.POSIXct.POSIXlt(e2) : unknown timezone 'Etc/GMT--10'
15: In as.POSIXlt.POSIXct(x, tz) : unknown timezone 'Etc/GMT--10'
16: In double(totvarsize) :
Reached total allocation of 3977Mb: see help(memory.size)
17: In double(totvarsize) :
Reached total allocation of 3977Mb: see help(memory.size)
18: In get.var.ncdf(glm_nc, var_name) :
Reached total allocation of 3977Mb: see help(memory.size)
19: In get.var.ncdf(glm_nc, var_name) :
Reached total allocation of 3977Mb: see help(memory.size)
20: In rv$data[abs(rv$data - mv) < tol] <- NA :
Reached total allocation of 3977Mb: see help(memory.size)
21: In rv$data[abs(rv$data - mv) < tol] <- NA :
Reached total allocation of 3977Mb: see help(memory.size)
22: In strptime(xx, f <- "%Y-%m-%d %H:%M:%OS", tz = tz) :
unknown timezone 'Etc/GMT--10'
23: In as.POSIXct.POSIXlt(x) : unknown timezone 'Etc/GMT--10'
24: In strptime(x, f, tz = tz) : unknown timezone 'Etc/GMT--10'
25: In as.POSIXct.POSIXlt(as.POSIXlt(x, tz, ...), tz, ...) :
unknown timezone 'Etc/GMT--10'
26: In double(totvarsize) :
Reached total allocation of 3977Mb: see help(memory.size)
27: In double(totvarsize) :
Reached total allocation of 3977Mb: see help(memory.size)
28: In get.var.ncdf(nc, forcevarid = d$dimvarid, verbose = verbose) :
Reached total allocation of 3977Mb: see help(memory.size)
29: In get.var.ncdf(nc, forcevarid = d$dimvarid, verbose = verbose) :
Reached total allocation of 3977Mb: see help(memory.size)
30: In get.var.ncdf(nc, forcevarid = d$dimvarid, verbose = verbose) :
Reached total allocation of 3977Mb: see help(memory.size)
31: In get.var.ncdf(nc, forcevarid = d$dimvarid, verbose = verbose) :
Reached total allocation of 3977Mb: see help(memory.size)

Your advices and help are much appreciated! Thank you!

[perovskite13]

For your information, if this is relevant, I'm using 64-bit R 3.2.2 platform on 64-bit Windows, and downloaded 32-bit GLM.

Thanks!

[perovskite13]

UPDATE : The memory allocation error turns out to be the lack of RAM. I had re-run the similar steps with a new R project and with minimal program running to allocate more memory spaces. The error message about memory allocation error disappears, but there are still error messages about "timezone error". The function still did not return any value as well.

I hope you can enlighten me on this issue. Your advices are much appreciated! Thank you so much!

[jordansread]

Thanks @perovskite13 I will look into this further. I was sidetracked by some travel and dealing with this #188 issue.

[jordansread]

I think there are two issues here, assuming this is the same issue as was emailed about today. The issue with the timezone, and then an error coming from pivot_match() for "duplicate date match". I am going to create a separate issue for the latter.

[jordansread]

This timezone issue is odd, because it does seem that "Etc/GMT-10" is valid:

OlsonNames()[386:390]
[1] "Etc/GMT-10" "Etc/GMT-11" "Etc/GMT-12" "Etc/GMT-13" "Etc/GMT-14"

but I also don't think this warning causes any actual failure. I set my timezone to it, and see the same warning, but plot_compar_var and others work fine, they just have the tz warnings:

sim_folder <- run_example_sim(verbose = FALSE)
nc_file <- file.path(sim_folder, 'output.nc')
nml_file <- file.path(sim_folder, 'glm2.nml')
field_file <- file.path(sim_folder, 'field_data.tsv')

run_glm(sim_folder)

plot_var_compare(nc_file, field_file, 'temp', resample=FALSE) ##makes a plot!

I get the same warning for other timezones in AUS that should be valid:
e.g., Sys.setenv(TZ="Australia/Queensland")

[lawinslow]

Stupid question, is the timezone just printed wrong in the warning message? Or are there really two hyphens? Might point to the culprit.

[jordansread]

yeah, it seems to just be printed wrong in the warning message.

[lawinslow]

Hmmm, do you have a example that reproduces this?

[jordansread]

@perovskite13 can you let us know what version of R you are using for this?

[AnaAlicia]

Hi, good evening! I'm trying to using "compare_to_field" function but it's showing this warning message:
In time_precision(df$DateTime, precision) :
days precision resulted in duplicate date values

Could you help me solve this, please?

Thank you so much in advance!
Have a nice day.

[lawinslow]

Does your field data field data file have duplicate date/time values?

[AnaAlicia]

Yes, but different depths. Is it ok?

[lawinslow]

Yes. It's unclear what is wrong then. Please submit an example here that re-creates the problem you are having. That way we can re-create it on our and and determine the problem.