NC production parser down

[electricitymapsbot]

Description

This is an automatic error report generated for New Caledonia (NC).

Issues:

• No recent data found for production parser

Suggestions

• Try running the parser locally using the command poetry run test_parser NC production
• Explore the runtime logs

You can see an overview of all parser issues here.

[VIKTORVAV99]

Ugh, I just created this parser and it's already down... 😅

[t-couery]

After investigation, the wordpress api is still up and running (the parser use https://www.enercal.nc/wp-content/themes/enercal/ajax-e-co2.php), but it's only returning empty values.
The same problem occurs on the Enercal portal : https://www.enercal.nc/

As far as I can see, there's no alternative data at the moment, and it's possible that this will be solved on its own.

[consideRatio]

The webpage is now presenting relevant data, but I'm not sure what to probe to acquire information programmatically as the previous URL is still responding but not returning any data.

Can we find another URL to inspect?

This dashboard is rendered as static HTML, it wasn't populated by an AJAX request or similar:

[consideRatio]

Aha! I did some googling etc and found https://www.enercal.nc/les-energies-caledoniennes/mix-energetique-emissions-de-co2/, and from that page there was an AJAX request looking like this - and that served data!

I'll see if I can get this running again with a small fix!

https://www.enercal.nc/wp-content/themes/enercal/ajax-e-co2.php?date_year=2024

Oh... Nevermind, that was already something used - the issue is that they don't present data via that API or webpage from 2024-06-30 and later currently.

[t-couery]

Yes, there has been no production data since July 2024, but it will certainly come back one day.
The consumption section that you saw on the website displays the data again. It is technically recoverable from the web page, but there are no details of the different sources of production. I don't know if it's worth working on that.

[VIKTORVAV99]

FYI if it was possible to just get the parser working with historical data that would also be very valuable!

[t-couery]

FYI if it was possible to just get the parser working with historical data that would also be very valuable!

Are you talking about the line you added? In this case, it's possible to move it to a later date, as the old data is still accessible.

electricitymaps-contrib/parsers/ENERCAL.py

Line 27 in 9e8130f

sixteen_weeks_ago = datetime.now(tz=TZ) - timedelta(weeks=16)

Exemple with one year :

https://www.enercal.nc/wp-content/themes/enercal/ajax-e-co2.php?date_day=2024-01-03

[VIKTORVAV99]

FYI if it was possible to just get the parser working with historical data that would also be very valuable!

Are you talking about the line you added? In this case, it's possible to move it to a later date, as the old data is still accessible.

electricitymaps-contrib/parsers/ENERCAL.py

Line 27 in 9e8130f

sixteen_weeks_ago = datetime.now(tz=TZ) - timedelta(weeks=16)

Exemple with one year :

https://www.enercal.nc/wp-content/themes/enercal/ajax-e-co2.php?date_day=2024-01-03

Yeah I think I might just have messed up the check here:

    target_datetime = (
        target_datetime
        if target_datetime and target_datetime > sixteen_weeks_ago
        else sixteen_weeks_ago
    )

It should check that the date is less than not greater than... 🤦🏻‍♂️

Might be good to add something similar to this before the above check as well:

    if target_datetime and target_datetime.tzinfo is None:
        target_datetime = target_datetime.replace(tzinfo=TZ)

[t-couery]

@VIKTORVAV99 Can you give your opinion for NC parser, let me sum up the situation:

• The parser works with the json https://www.enercal.nc/wp-content/themes/enercal/ajax-e-co2.php?date_day=YYYY-MM-DD, but there is no more production data after 2024-06-30

• There is consumption data in the HTML of the https://www.enercal.nc/ page (with a regexp => series: \[\n\t\t{\n\t\t\t data: \[([^\]]+)), the parser does not implement consumption.

What should be done?

[VIKTORVAV99]

@VIKTORVAV99 Can you give your opinion for NC parser, let me sum up the situation:

• The parser works with the json https://www.enercal.nc/wp-content/themes/enercal/ajax-e-co2.php?date_day=YYYY-MM-DD, but there is no more production data after 2024-06-30

• There is consumption data in the HTML of the https://www.enercal.nc/ page (with a regexp => series: \[\n\t\t{\n\t\t\t data: \[([^\]]+)), the parser does not implement consumption.

What should be done?

For the first part id say fix the issue with the comparison I mentioned above so we can backfill all the data that is available at least.

For the second part that would be highly optional but you could create a parser for consumption. Currently however consumption is not used in the app but might be helpful in the future if we determine there is a need to create a estimation model for NC. But given its location and low population I'll be honest and say the priority for that would be very low.

[t-couery]

For the first part id say fix the issue with the comparison I mentioned above so we can backfill all the data that is available at least.

Thank you for your prompt reply. As far as I can see, the production data can be accessed from 2022-02-01 to 2024-06-30.
If we modify the parser by changing the code above, will the history > 3 months be refetched or you meean to change the "sixteen_weeks_ago" for something bigger to get an estimatation ?

[VIKTORVAV99]

I think this:

    target_datetime = (
        target_datetime
        if target_datetime and target_datetime > sixteen_weeks_ago
        else sixteen_weeks_ago
    )

should be changed to this:

    target_datetime = (
        target_datetime
        if target_datetime and target_datetime < sixteen_weeks_ago
        else sixteen_weeks_ago
    )

It was added as when the parser was working the data was delayed by roughly 4 months (maybe they just stopped uploading data in retrospect) but I messed up the sign in the process.

Might be safer to remove this whole check actually incase they start publishing data again.

[VIKTORVAV99]

There should be a check to make sure target_datetime is defined else set it to now though.