add crowdstrike falcon connector example #439
Conversation
fivetran-surabhisingh
added
the
hackathon
🧹 Python Code Quality Check📎 Download full report from workflow artifacts. 📌 Only Python files changed in this PR were checked. This comment is auto-updated with every commit. |
fivetran-sahilkhirwal
requested review from
Copilot,
fivetran-ameysharma and
fivetran-sahilkhirwal
There was a problem hiding this comment.
Pull Request Overview
This PR adds a new CrowdStrike Falcon connector to pull endpoint detection and response data. However, the implementation uses an API pattern that is incompatible with the Fivetran Connector SDK.
- Uses decorator-based connector pattern with
@connectordecorator - Implements configuration and schema using
config.Configandschema.Schemaobjects - Attempts to write records using
records.write()function
| """ | ||
|
|
||
| import requests | ||
| from fivetran_connector_sdk import connector, config, state, records, log, schema |
There was a problem hiding this comment.
BLOCKER: Wrong import statements for Fivetran Connector SDK. The SDK does not support connector, config, state, records, or schema imports. Use the correct imports: from fivetran_connector_sdk import Connector, from fivetran_connector_sdk import Logging as log, and from fivetran_connector_sdk import Operations as op.
| Pulls endpoint detection and response data. | ||
| """ | ||
|
|
||
| import requests |
There was a problem hiding this comment.
Missing inline comment explaining the purpose of this import. Add a comment like # For making HTTP API requests (NOTE: provided by SDK runtime) to follow SDK conventions.
| CONFIG = config.Config( | ||
| base_url=config.StringField(default="https://api.crowdstrike.com"), | ||
| client_id=config.SecretField(), | ||
| client_secret=config.SecretField() | ||
| ) |
There was a problem hiding this comment.
BLOCKER: The Fivetran Connector SDK does not use config.Config objects. Configuration is provided via a configuration.json file and accessed as a dictionary parameter in the update() and schema() functions. Remove this configuration object and create a proper configuration.json file instead.
| SCHEMA = schema.Schema( | ||
| name="falcon_detections", | ||
| columns={ | ||
| "id": schema.StringColumn(), | ||
| "created_timestamp": schema.StringColumn(), | ||
| "status": schema.StringColumn(), | ||
| "severity": schema.StringColumn(), | ||
| "behavior": schema.JSONColumn(), | ||
| } | ||
| ) |
There was a problem hiding this comment.
BLOCKER: The Fivetran Connector SDK does not use schema.Schema objects. Define a schema(configuration: dict) function that returns a list of table schema dictionaries. See the template for the correct format.
| @connector( | ||
| name="CrowdStrikeFalconConnector", | ||
| version="0.1.0", | ||
| config=CONFIG, | ||
| schema=SCHEMA, | ||
| ) | ||
| def run_connector(ctx: state.Context): |
There was a problem hiding this comment.
BLOCKER: The Fivetran Connector SDK does not use decorator-based patterns. Replace this with a standard update(configuration: dict, state: dict) function. The connector is initialized using connector = Connector(update=update, schema=schema) at the module level.
| for item in response.json().get("resources", []): | ||
| records.write("falcon_detections", {"id": item}) |
There was a problem hiding this comment.
BLOCKER: The SDK does not have a records.write() function. Use op.upsert(table='falcon_detections', data=record) instead. Additionally, this line should be preceded by the required upsert comment explaining the operation.
| for item in response.json().get("resources", []): | ||
| records.write("falcon_detections", {"id": item}) | ||
|
|
||
| return ctx.update_state({"last_sync": "now"}) |
There was a problem hiding this comment.
BLOCKER: The SDK does not use ctx.update_state(). State management is done via op.checkpoint(state). The update() function should not return anything (returns None). Add the required checkpoint comment before calling op.checkpoint(state).
| """ | ||
| CrowdStrike Falcon connector for Fivetran Connector SDK. | ||
| Pulls endpoint detection and response data. | ||
| """ | ||
|
|
||
| import requests | ||
| from fivetran_connector_sdk import connector, config, state, records, log, schema | ||
|
|
||
| CONFIG = config.Config( | ||
| base_url=config.StringField(default="https://api.crowdstrike.com"), | ||
| client_id=config.SecretField(), | ||
| client_secret=config.SecretField() | ||
| ) | ||
|
|
||
| SCHEMA = schema.Schema( | ||
| name="falcon_detections", | ||
| columns={ | ||
| "id": schema.StringColumn(), | ||
| "created_timestamp": schema.StringColumn(), | ||
| "status": schema.StringColumn(), | ||
| "severity": schema.StringColumn(), | ||
| "behavior": schema.JSONColumn(), | ||
| } | ||
| ) | ||
|
|
||
| @connector( | ||
| name="CrowdStrikeFalconConnector", | ||
| version="0.1.0", | ||
| config=CONFIG, | ||
| schema=SCHEMA, | ||
| ) | ||
| def run_connector(ctx: state.Context): | ||
| # Normally you'd authenticate via OAuth2 first | ||
| headers = {"Authorization": f"Bearer {ctx.config.client_secret}"} | ||
| url = f"{ctx.config.base_url}/detects/queries/detects/v1" | ||
|
|
||
| response = requests.get(url, headers=headers) | ||
| response.raise_for_status() | ||
|
|
||
| for item in response.json().get("resources", []): | ||
| records.write("falcon_detections", {"id": item}) | ||
|
|
||
| return ctx.update_state({"last_sync": "now"}) |
There was a problem hiding this comment.
BLOCKER: Missing required structure elements. The connector must include: (1) A schema(configuration: dict) function with the exact docstring from the template, (2) An update(configuration: dict, state: dict) function with the exact docstring from the template, (3) Connector initialization: connector = Connector(update=update, schema=schema), (4) Main block for debugging with if __name__ == '__main__': section. Refer to the template at template_example_connector/connector.py for the complete required structure.
| # Normally you'd authenticate via OAuth2 first | ||
| headers = {"Authorization": f"Bearer {ctx.config.client_secret}"} |
There was a problem hiding this comment.
The comment states 'Normally you'd authenticate via OAuth2 first' but the code directly uses client_secret as a bearer token. This is incorrect for OAuth2 authentication. Proper OAuth2 requires exchanging client credentials for an access token first. Implement proper token acquisition using the OAuth2 token endpoint before making API calls.
| # Normally you'd authenticate via OAuth2 first | |
| headers = {"Authorization": f"Bearer {ctx.config.client_secret}"} | |
| # Authenticate via OAuth2 client credentials flow to obtain an access token | |
| token_url = f"{ctx.config.base_url}/oauth2/token" | |
| token_data = { | |
| "client_id": ctx.config.client_id, | |
| "client_secret": ctx.config.client_secret | |
| } | |
| token_headers = {"Content-Type": "application/x-www-form-urlencoded"} | |
| token_response = requests.post(token_url, data=token_data, headers=token_headers) | |
| token_response.raise_for_status() | |
| access_token = token_response.json().get("access_token") | |
| if not access_token: | |
| raise Exception("Failed to obtain access token from CrowdStrike Falcon API.") | |
| headers = {"Authorization": f"Bearer {access_token}"} |
| """ | ||
|
|
||
| import requests | ||
| from fivetran_connector_sdk import connector, config, state, records, log, schema |
There was a problem hiding this comment.
Import of 'log' is not used.
| from fivetran_connector_sdk import connector, config, state, records, log, schema | |
| from fivetran_connector_sdk import connector, config, state, records, schema |
1 participant
Jira ticket
Closes
<ADD TICKET LINK HERE, EACH PR MUST BE LINKED TO A JIRA TICKET>Description of Change
<MENTION A SHORT DESCRIPTION OF YOUR CHANGES HERE>Testing
<MENTION ABOUT YOUR TESTING DETAILS HERE, ATTACH SCREENSHOTS IF NEEDED (WITHOUT PII)>Checklist
Some tips and links to help validate your PR:
fivetran debugcommand.