Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Check datasource validity on loading. #966

Open
1 of 7 tasks
xCodeMuse opened this issue Feb 23, 2024 · 1 comment · Fixed by #975
Open
1 of 7 tasks

Check datasource validity on loading. #966

xCodeMuse opened this issue Feb 23, 2024 · 1 comment · Fixed by #975
Assignees
Labels
bug Something isn't working current current sprint gs_service gs_service in progress team is currently working on this feature P0 Highest priority

Comments

@xCodeMuse
Copy link
Contributor

xCodeMuse commented Feb 23, 2024

The Problem

Currently the dynamic fn [in case of datasource as fn] script of task is not evaluated/compiled beforehand, when it uses inputs or outputs like references which are available during a workflow call (runtime only).
But for other cases where datasource name is statically given or the dynamic fn uses mappings or config we can evaluate and check validaity of those in the load time itself. generating the fn in functionloader to check if datasource is available or not.

The Solution

Evaluate script beforehand and check if datasource is present in fn as datasource of tasks of a workflow.

How will we solve

  • Make changes in the core functionloader.ts file and make required changes to evaluate the fn script before checking the datasource validation success. [ commented the code for now ]
  • Ensure that loaded datasource is a valid datasource (load time checks). The cases to cover are
    • fn: <js% datasource.<% config.hello %>.dynamodb.listTables %>
    • fn: datasource.<% config.hello %>.dynamodb.listTables
    • fn: <% mappings.datasourcename.for.dbcall %> fn: <% config.datasourcename.for.dbcall %>
    • fn: datasource.<%mappings[outputs.create_lender_api_request.data.request_data.headers['pl-product']][outputs.create_lender_api_request.data.request_data.headers['pl-lender']].lender_api_suit[outputs.create_lender_api_request.data.request_data.params?.wrapper_event].datasource%>.get.<%outputs.path_params_url.data.updated_url%>
  • In expandVariables during loadTime, when an error occurs in script evaluation, check whether the script was using outputs. or inputs. don't log this - because this will be evaluated in runtime. Otherwise process.exit(1) with a fatal log message.
@xCodeMuse xCodeMuse added bug Something isn't working gs_service gs_service P0 Highest priority current current sprint in progress team is currently working on this feature labels Feb 23, 2024
@xCodeMuse xCodeMuse self-assigned this Feb 23, 2024
@xCodeMuse xCodeMuse changed the title Dynamic datasource [script] in task fn not working. Evaluating Dynamic datasource [script] in task fn not working. Feb 24, 2024
@xCodeMuse xCodeMuse linked a pull request Feb 28, 2024 that will close this issue
@ghost
Copy link

ghost commented Feb 28, 2024

@xCodeMuse Please follow this design to check these cases:

Step-1: check if task.fn starts with 'datasource.'
    step 1.1: if <% %> is present in task.fn then do expandVariable
    step 1.2: check if <% %> is still present in the expanded variable. 
        step 1.2.1: If yes, then skip checking datasource name, as the expanded variable might have dynamic variables like inputs/outputs.
        step 1.2.2: else check datasource name and return false if datasource name not found in the datasources.
Step-2: else
    step 2.1: if <% %> is present task.fn, then do expandVariable
    step 2.2: After expandVariable, check if task.fn starts with 'datasource.' 
        step 2.2.1: If yes, then repeat Step-1 for this case.
return true

@mastersilv3r mastersilv3r changed the title Evaluating Dynamic datasource [script] in task fn not working. Check datasource validity on loading. Mar 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working current current sprint gs_service gs_service in progress team is currently working on this feature P0 Highest priority
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant