Event Listener - Trigger Github action

[franTarkenton]

Having:

1. created an automated build process to create the container image
2. created an automated deployment for the container

This ticket will figure out:

1. how to call a gha remotely via rest call or webhook etc...
2. ammend existing listener code to emit the web call above ^ when all the regional_grib and global_grib data that we are interested in has been emitted.

[franTarkenton]

completed so far

• have determined how to define a remote job... example: https://github.com/bcgov/nr-rfc-grib-copy/actions/runs/5273197792/jobs/9536328668
• made modifications to the code to be able to detect all the expected events, and then emit an action

remaining

• create a modified CMC job that receives the date to process as the payload

[franTarkenton]

I deployed and seems to be working. Picks up residual events that are defined in the cache and emits the associated events. Still not seeing the script emit events based on regular event loop.

In other words events get cached and written to the database. At startup the cached events are checked to see if their are associated events to be emitted. When there are these events are working. After this step the script starts to monitor the topics it has subscribed to. For each event the script should check to see if all the data is now there, and if so emit an event. This final step is not currently working.

Changes in latest pr... will montior to see if they have resolved this issue

[franTarkenton]

Once we see the jobs being triggered, will circle back and remove the cron trigger that is currently described in the workflow.

[franTarkenton]

The events are not getting emitted because the token used to trigger the actions is not getting passed to the helm chart secret. Turns out this is no longer possible, and you need to create a PAT, then copy the pat into the repo. It doesn't look like there is a way to update the PAT in any kind of automated way! That said all that is required is to update the token, then update the secret in the repo with the token, then re-deploy the helm chart.

• should create an action that detects change of specific secret, and then triggers re-deploy.

[franTarkenton]

Merging this code. Looking through the kibana logs believe that its working correctly considering deficiencies identified in #56

Will continue to monitor the deployed version associated with PR-51 and branch: feat/16-add-debugging-endpoint.

If prod succeeds this weekend will close this ticket

Pivotting onto issue: #56

[franTarkenton]

As the prod database grows with events that are stale the query to the database is starting to use up a lot of memory. Going to modify how the events are cached in the following way:

• eliminate logging of non pertinent events to the database, and only log them in the logs. Can search for them in kibana for debugging purposes.
• Create a process that checks for stale logged events. Stale events will be defined by the number of days old they are. This parameter will be defined in an environment variable with a default value of 3 days.

[franTarkenton]

Changes recently pushed should improve pod stability. Will check in the morning to verify that the messages are being recieved and the appropriate events are being emitted

[franTarkenton]

Looking at the logs for the gha... it looks like the following improvements are now functional:

• only caching events that are of interest
• events are getting flushed preventing ongoing growth of the database
• The downstream actions to do the actual task of downloading / processing the data is now working.

[franTarkenton]

Been monitoring for 1 week without incident. Events are being emitted, and the CMC download / process pipeline is now working