Details about the bot creating/updating "parser down" issues

[consideRatio]

I saw that the "SV parser down" issue #5158 is still open, even though its functional now, so I'm asking a few questions that I can't figure out myself as I think its closed source.

1. At what schedule is the bot updating the issues?
2. What steps need's to happen behind the curtains after a PR is merged in this repo fixing a parser before data starts flowing?
3. Is staging.electricitymaps.com and app.electricitymaps.com using different data or the same - is the staging website mostly a UI staging rather than UI and other related backend also?

Knowledge about this can help me debug and understand "parser down" issues in general, and I'm interested in working on resolving them.

[VIKTORVAV99]

At what schedule is the bot updating the issues?

I think the bot is running once per day, so it should close the issue by tomorrow since I made a new release today.

What steps need's to happen behind the curtains after a PR is merged in this repo fixing a parser before data starts flowing?

When PRs are merged to master in this repository a automated PR is created against our monorepo that contains all changes since the last release and then run all integration tests in the backend against it and ensure all our dependencies are in sync.

This PR requires manual approval and merging to create a new automated release.

Is staging.electricitymaps.com and app.electricitymaps.com using different data or the same - is the staging website mostly a UI staging rather than UI and other related backend also?

This is for the frontend only, at least for now, when we do backend changes we usually just version them and run them in parallel with the old version.

I hope this makes some sense. BTW part of the reason the bot only runs once per day is to avoid some flakiness and the fact that it's looking at the incoming data instead of parser errors, so sometimes (like for Italy) it means the data is unavailable but the parser is technically working.

[consideRatio]

@VIKTORVAV99 is the automation doing this something I could look at or be extracted to be made public, and something I could contribute to? I'm not sure how the automation is made, but it could be made very easily using a github workflow - and if its already a github workflow, I have a lot of know-how about working it.

Top of the head improvement ideas:

• make it update the original post so that information stays relevant instead of just saying data is flowing/not flowing in a new comment
  • this would also fix a broken link in MY-WM production parser down #3341 (comment) linking to kibana instead of google storage
• clarify other details asked in this issue

[VIKTORVAV99]

It's in our internal monorepo now so it's not a decision I can make alone, but I'll ask and see what the others say. It's currently running in a Google Cloud instance with access to our database so not sure how easy it would be to migrate.

[consideRatio]

I think improving that automation is quite essential to drive this project with regards to parser development.

I think the key things I look to have contributing to the parsers currently are:

• Updated info
  The title and the original posts isn't getting updated, so they are currently misleading.
  Instead of opening/closing the issue and leaving a comment about "data is flowing", it should update the title and top post to reflect the current state, and possibly also make a comment to help interested parties get a notification.
• Last failure report
  Retaining a report from the last failure helps handle intermittent errors.

I'm happy to help with this in general in any way I can, I've freed up time in my life to work on things I think matters. I have gained some relevant know-how for this kind of work via maintenance of cloud infra in the public on github (https://github.com/jupyterhub/mybinder.org-deploy, https://github.com/2i2c-org/infrastructure), and have a lot of cloud know how in general.