feat: add Airflow provider integration and documentation

[tuantran0910]

Closes #4089

📑 Description

This PR is to add a feature for integration between Apache Airflow and Keep.

✅ Checks

• My pull request adheres to the code style of this project
• My code requires changes to the documentation
• I have updated the documentation as required
• All the tests have passed

[vercel]

@tuantran0910 is attempting to deploy a commit to the KeepHQ Team on Vercel.

A member of the Team first needs to authorize it.

[CLAassistant]

All committers have signed the CLA.

[talboren]

❤️ will review soon

[shahargl]

👑

[tuantran0910]

Hi everyone,

I'm facing the following issues:

1. I don't see the Airflow provider listed on the Provider page in the UI.

2. I cannot send requests to the /alerts/event/airflow endpoint using the following command:

   curl --location 'http://0.0.0.0:8080/alerts/event/airflow' \
     --header 'Content-Type: application/json' \
     --header 'Accept: application/json' \
     --header 'X-API-KEY: 0a73d419-89dc-4d0f-9b7d-376577a76823' \
     --data '{
     "name": "Airflow Task Failure: keep_task",
     "status": "firing",
     "service": "pipeline",
     "message": "Task keep-task failed in DAG keep_dag at 2025-03-18",
     "description": "Bash command failed. The command returned a non-zero exit code",
     "severity": "critical"
   }'

   Instead, I have to use the generic /alerts/event endpoint:

   curl --location 'http://0.0.0.0:8080/alerts/event' \
     --header 'Content-Type: application/json' \
     --header 'Accept: application/json' \
     --header 'X-API-KEY: 0a73d419-89dc-4d0f-9b7d-376577a76823' \
     --data '{
     "name": "Airflow Task Failure: keep_task",
     "status": "firing",
     "service": "pipeline",
     "source": ["airflow"],
     "message": "Task keep-task failed in DAG keep_dag at 2025-03-18",
     "description": "Bash command failed. The command returned a non-zero exit code",
     "severity": "critical"
   }'

Is this expected behavior, or am I missing something?

Thanks in advance!

[talboren]

Hi everyone,

I'm facing the following issues:

1. I don't see the Airflow provider listed on the Provider page in the UI.

2. I cannot send requests to the /alerts/event/airflow endpoint using the following command:

   curl --location 'http://0.0.0.0:8080/alerts/event/airflow' \
     --header 'Content-Type: application/json' \
     --header 'Accept: application/json' \
     --header 'X-API-KEY: 0a73d419-89dc-4d0f-9b7d-376577a76823' \
     --data '{
     "name": "Airflow Task Failure: keep_task",
     "status": "firing",
     "service": "pipeline",
     "message": "Task keep-task failed in DAG keep_dag at 2025-03-18",
     "description": "Bash command failed. The command returned a non-zero exit code",
     "severity": "critical"
   }'

   Instead, I have to use the generic /alerts/event endpoint:

   curl --location 'http://0.0.0.0:8080/alerts/event' \
     --header 'Content-Type: application/json' \
     --header 'Accept: application/json' \
     --header 'X-API-KEY: 0a73d419-89dc-4d0f-9b7d-376577a76823' \
     --data '{
     "name": "Airflow Task Failure: keep_task",
     "status": "firing",
     "service": "pipeline",
     "source": ["airflow"],
     "message": "Task keep-task failed in DAG keep_dag at 2025-03-18",
     "description": "Bash command failed. The command returned a non-zero exit code",
     "severity": "critical"
   }'

Is this expected behavior, or am I missing something?

Thanks in advance!

Hey @tuantran0910, could be that there is some problem with the provider and it's not loading for some reason.
Are you seeing any "Cannot import provider Airflow" log in the backend logs?

[tuantran0910]

Hi @talboren, I have figured out the reason that cannot making requests to the /alerts/event/airflow endpoint is that the environment must be set and cannot be None. So I will change the code so that the default environment will be undefined if it is not set like Keep.

The only concern right now is that there is no Airflow provider shown in the Providers page:

There is not error log Cannot import provider Airflow in the backend logs.

[talboren]

Hi @talboren, I have figured out the reason that cannot making requests to the /alerts/event/airflow endpoint is that the environment must be set and cannot be None. So I will change the code so that the default environment will be undefined if it is not set like Keep.

The only concern right now is that there is no Airflow provider shown in the Providers page:

There is not error log `Cannot import provider Airflow` in the backend logs.

That's it, you found the reason :)

Regarding not being able to see the provider in the list, it's because the provider doesn't really have any reason to be "installed" and we do not present it, because it has no configuration.

I can actually see that the backend sends it to the frontend:

But we do not present it because the provider has no configuration

^ this is an example from Pagerduty

[tuantran0910]

Hi @talboren, I have figured out the reason that cannot making requests to the /alerts/event/airflow endpoint is that the environment must be set and cannot be None. So I will change the code so that the default environment will be undefined if it is not set like Keep.
The only concern right now is that there is no Airflow provider shown in the Providers page:

There is not error log Cannot import provider Airflow in the backend logs.

That's it, you found the reason :)

Regarding not being able to see the provider in the list, it's because the provider doesn't really have any reason to be "installed" and we do not present it, because it has no configuration.

I can actually see that the backend sends it to the frontend:

But we do not present it because the provider has no configuration

^ this is an example from Pagerduty

Thanks @talboren, I understand it. So because Airflow does not need to configure or install anything from the Keep side, so yeah, there is no reason to display it in the Providers page.

[shahargl]

Hi @talboren, I have figured out the reason that cannot making requests to the /alerts/event/airflow endpoint is that the environment must be set and cannot be None. So I will change the code so that the default environment will be undefined if it is not set like Keep.
The only concern right now is that there is no Airflow provider shown in the Providers page:

There is not error log Cannot import provider Airflow in the backend logs.

That's it, you found the reason :)
Regarding not being able to see the provider in the list, it's because the provider doesn't really have any reason to be "installed" and we do not present it, because it has no configuration.
I can actually see that the backend sends it to the frontend:

But we do not present it because the provider has no configuration

^ this is an example from Pagerduty

Thanks @talboren, I understand it. So because Airflow does not need to configure or install anything from the Keep side, so yeah, there is no reason to display it in the Providers page.

Actually I think its cool to have it so people will know we have integration with them.

There are couple of other providers that do not need any configuration but have provider tiles.

For example, you can see checkmk provider - it only have webhook_markdown to explain how to install the webhook:

[tuantran0910]

I agree with @shahargl, we can put the instruction of how to setup the callback for Airflow that connect with Keep in here :D Maybe by adding three class variable webhook_description, webhook_template and webhook_markdown like the CheckmkProvider provider ?

[talboren]

I agree with @shahargl, we can put the instruction of how to setup the callback for Airflow that connect with Keep in here :D Maybe by adding three class variable webhook_description, webhook_template and webhook_markdown like the CheckmkProvider provider ?

That will be more than awesome

[tuantran0910]

So one more thing to concern here is that I have setup the fingerprint field fingerprint for the AirflowProvider:

class AirflowProvider(BaseProvider):
    """Enrich alerts with data sent from Airflow."""

    PROVIDER_DISPLAY_NAME = "Airflow"
    PROVIDER_CATEGORY = ["Orchestration"]
    FINGERPRINT_FIELDS = ["fingerprint"]

So for example, if I make the request with the same fingerprint but different in other fields (for example the description contains the timestamp). Will those alerts actually get deduplicated ?

But I see that I got this instead:

[tuantran0910]

I agree with @shahargl, we can put the instruction of how to setup the callback for Airflow that connect with Keep in here :D Maybe by adding three class variable webhook_description, webhook_template and webhook_markdown like the CheckmkProvider provider ?

That will be more than awesome

Yeah, this can help the users to reduce the gap on exploring provider. I have configure this for the Airflow Provider :D

Checkout this commit 7165b7c.

[shahargl]

@tuantran0910 regarding the fingerprint - yea if the description contains timestamp than deduplication won't deduplicate it.

but it can be configured later on "deduplication" rules, so let's start simple and continue from there?

[tuantran0910]

@tuantran0910 regarding the fingerprint - yea if the description contains timestamp than deduplication won't deduplicate it.

but it can be configured later on "deduplication" rules, so let's start simple and continue from there?

Totally agree @shahargl !

[talboren]

@tuantran0910 can you add a screenshot from provider screen to show how it looks like when you open Airflow? 🙏🏼

[tuantran0910]

Hi @talboren, here is the screenshot from the provider screen:

Actually, I cannot click the block under the Linked Providers section, I have do search the Airflow on the search bar:

Click on it under the Available Providers, then I got the information of the Airflow Provider:

From my opinion, I think it would be ideal to let the user click on the block under Linked Providers :D Or maybe this is the default behavior in Keep.

[shahargl]

yea, currently "linkedin providers" aren't clickable (because they do not have any information/state). how would u change it?

and apart from it - its ready for PR?

[tuantran0910]

yea, currently "linkedin providers" aren't clickable (because they do not have any information/state). how would u change it?

and apart from it - its ready for PR?

Yes, it's ready for PR.

[shahargl]

lgtm

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
keep	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Mar 19, 2025 0:15am