Can you help me understand how to fix the ValueError: Unable to calculate Pearson Type III parameters due to invalid L-moments #512
Comments
|
Please post a link to your datasets so I can attempt to reproduce and understand the error. For example |
|
I have now an output for spi but with masked array, when I visualize the nc
file there are a warning in my data masked array (warning valid max and
min).
I'll send you my output spi to check if my output is correct or not.
Are you recieve my output drive link ?
…On Mon, May 22, 2023, 15:58 James Adams ***@***.***> wrote:
Please post a link to your datasets so I can attempt to reproduce and
understand the error. For example sliced_pre_1901_18.nc There have been
many issues regarding L-moments in the past, and all seem to have been
resolved now if using the latest scipy etc. Please install the
climate_indices package from the master branch (which is ahead of what's
now on PyPI) to see if that will fix your issue. My apologies, I haven't
had time to update PyPI with the latest/greatest code yet.
—
Reply to this email directly, view it on GitHub
<#512 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AJ5OUUYMEMIQXI6JBVTJS7LXHN5INANCNFSM6AAAAAAXVXSRYU>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
|
Please post a link to your data here. Are you using the latest code from the master branch of this repository, or are you using the version from PyPI which is a little behind that (sorry for out of sync, tech debt)? Please use the latest code from this repository which should have the L-moments issue worked out. There have been many problems with the L-moments calculations over the years but seems like they've been rectified now. |
|
I'm having a similar issue, which I've managed to trace to a violation of theorem 1 in Hosking (1990), which bounds tau3 (or t3 in the code here) to -1 < t3 < 1. In my case, I get t3 = 1. Below is a summary of how I found it. I have precipitation and evapotranspiration data with shape (lat, lon, time) = (8520, 7320, 180), in monthly time steps from Jan 2004 to Dec 2018. I'm trying to calculate SPEI. I added some print messages to the main loop in main._apply_along_axis_double() to get the lat, lon indexes of where the L-moments error were happening. Then I proceeded with a step-by-step calculation using the modules. Below is an example of what is causing the problem. For the month of August, we have: When the calculation calls lmoments._estimate_pearson3_parameters(), we get And hence, the error. Using scipy.stats.pearson3.fit() on the data with both maximum likelihood or method of moments (MM) we get valid output: Just for comparison, for the month of February, which doesn't trigger an error, we get estimates that are pretty close between scipy with "MM" and the estimate_pearson3_parameters routine: Looking at other points this seems to happen when all P = 0 for a certain month (where seasonality is high). Maximum likelihood has been used before to calculate SPI/SPEI, with L-moments providing initial values (as in Stagge et al. 2015, https://doi.org/10.1002/joc.4267), so this might be an option in the future for similar cases. |
|
@caiomattos' analysis looks really solid here. Thanks for the input! If there's a way to fix our code to accommodate this then a PR will be welcome. |
|
Hi, I installed climate-indices into a fresh Python3.10 virtual env with the latest scipy version 1.10.1 and climate indices 1.10.13 The above exception was the direct cause of the following exception: Traceback (most recent call last): The above exception was the direct cause of the following exception: Traceback (most recent call last): On running: Link to rainfall and PET datasets: Not sure if related or not but also get: I have to run : export NUMBA_DISABLE_JIT=1 to get the job to even start to process to get to the L-moments error I got the above command to run and worked a couple of weeks ago using version 1.10.12 but it wouldnt run this time so I updated to 1.10.13 and still not working. conda list: |
|
There is an imminent update with code that I've tested against these Australian datasets, looking good now using the development branch named |
|
Hi...thanks for that. I did an uninstall and reinstall from the issue_522_pyproject_poetry tree. The line 10 import typing in main.py needs to import Dict and Any as well. When I fixed that, it ran, but again threw the L-moments error on Pearson typeIII :( Is there a way to just run the Gamma distribution by the way? And, from your answer, I assume the package is okay with cftime conventions? |
|
Thanks @Darren-Ray I appreciate the beta testing! On my copy of that branch, I am having good results running the processing command. I've created a fresh conda environment and installed from source, like so: I have added We used to have an option that allowed users to specify a single distribution but looking at the docs it seems that that was removed at some point, or maybe I am misremembering. Anyway, it shouldn't be too hard to re-implement that by adding a |
|
Hello every one, I tested the package "Another one" with fixing some
problems and it's working fine.
I would thank all of you, specially Mr. James Adams. All my good wishes to
you.
Le sam. 1 juil. 2023 à 15:06, James Adams ***@***.***> a
écrit :
… Thanks @Darren-Ray <https://github.com/Darren-Ray> I appreciate the beta
testing!
On my copy of that branch, I am having good results running the processing
command. I've created a fresh conda environment and installed from source,
like so:
conda create -n myvenv poetry pytest
conda activate myvenv
python -m poetry install
python -m poetry run pytest
cd ~/tmp/data
process_climate_indices --index spei --periodicity monthly --netcdf_precip LME_monthlyRAIN_AUSTNZ.nc --var_name_precip RAIN --netcdf_pet LME_monthlyEto_AUSTNZ.nc --var_name_pet PM_FAO_56 --output_file_base nclimgrid_SPEI12 --scales 12 --calibration_start_year 1986 --calibration_end_year 2005 --multiprocessing all
I have added cftime as a dependency which got me past an error I saw
around that. So we should be OK for cftime conventions, but I'm not
well-versed in that and can't promise too much there.
We used to have an option that allowed users to specify a single
distribution but looking at the docs it seems that that was removed at some
point, or maybe I am misremembering. Anyway, it shouldn't be too hard to
re-implement that by adding a --distfit option or something like that.
—
Reply to this email directly, view it on GitHub
<#512 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AJ5OUU6RAFMIARDVT2YKSPDXOAVEVANCNFSM6AAAAAAXVXSRYU>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
|
Hi! @monocongo The example data from nclimgrid and cmorph is working well. |
|
Thanks, @chaowats Please confirm that you get the same result if installing this package into a fresh virtual environment using poetry. Please install from PyPI rather than github. |
|
@chaowats You can tell me how you installed the climate indices package because I'm traveling with the CHIRPS data. |
|
@monocongo Yes I did install to a new environment with the 2.0 version and master branch using pip install climate-indices from PyPI (but with conda environment, is that okay?) |
|
@ben29med I install using pip install climate-indices from PyPI but with the CHIRPS data I change precipitation units from mm/month to mm, longitude to lon, latitude to lat, and reorder to lat,lon,time before process_climate_indices --index spi. |
|
@chaowats please, send me the nclimgrid dataset example. Thank's ! |
|
我在计算月尺度SPI Pearson 也碰到了同样的问题，可能是由于时间序列数据的问题，我将其加上很小的随机数就可计算。 |
Hello, I came across the same problem with CRU data but ERA5 works fine. Did you find the solution for CRU data? |
|
I also experience the L-moments error. I use the UCSB CHIRPS data. This follows a number of runtime warnings "divide by zero encountered in divide" and " invalid value encountered in multiply", which may be related to the L-moments error? Full log below. The data can be found here: https://tubcloud.tu-berlin.de/s/4YcpA9RXLBXzka6 I am running the following commands: Am I doing this right? |
|
@rabons I see the same error when I run on my laptop. Looking at the data with I was only recently made aware of this time step period but apparently, it's used by some researchers, so I'd like to support it as a new |
|
@monocongo thank you for lookig into this! time_step = "dekad" is actually a labeling error in this specific file, I think. Will double-check tomorrow. Would this label affect the SPI calculations in any way? I can re-produce it with a correct label. EDIT: I have changed the file with correct labeling, it doesnt change the error with climate-indices. I have created this file with this script (slightly modified): https://bennyistanto.github.io/spi/chirpstif/ I actually do need a "dekadal" time step, but more important is that I make the monthly calculations work. |
|
OK then this isn't caused by the dekad time step since it's just a label. Thanks for the clarification @rabons This error is raised in the lmoments.py code, here: We can add some debugging around that to see if there are anomalies in the data etc. causing this to happen. |
|
Thank you. EDIT: SPI calculations seem to be working for a file clipped to a shapefile (in this case, Uganda boundaries). There may have been an unrecognized problem with my previous input file. The working input file: https://tubcloud.tu-berlin.de/s/omStM55dAmC2XAw The issue is hence solved for me when first clipping the input file (CHIRPS africa_monthly tiffs). Not sure what this is about. A few 'division by zero' errors persist, but I can still produce valid SPI output. |
I had a problem with L-moments when i excute my climate indices command:
rocess_climate_indices --index spi --periodicity monthly --netcdf_precip sliced_pre_1901_18.nc --var_name_precip pre --output_file_base sliced_pr.nc --scales 12 --calibration_start_year 1901 --calibration_end_year 2018
What's the problem ! @monocongo @monocongo @WeatherGod @ScottWales @dawiedotcom
The text was updated successfully, but these errors were encountered: