Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ObjectStoragePath should be template-able #40361

Open
1 of 2 tasks
fritz-astronomer opened this issue Jun 21, 2024 · 1 comment
Open
1 of 2 tasks

ObjectStoragePath should be template-able #40361

fritz-astronomer opened this issue Jun 21, 2024 · 1 comment
Labels

Comments

@fritz-astronomer
Copy link
Contributor

Description

ObjectStoragePath should be able to handle Jinja Templates

Use case/motivation

If I have an operator, that has an ObjectStoragePath argument

class MyOperator(BaseOperator):
  template_fields = ('path',)

  def __init__(path, **kwargs):
    self.path = path
    super().__init__(**kwargs)

and am using it in a DAG, with templating

MyOperator(
  ...,
  path=ObjectStoragePath(
    "s3://my_bucket/foo/{{ ds_nodash }}/file.csv",
    conn_id="s3"
  )
)

Templating didn't seem to work

Related issues

No response

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct

@fritz-astronomer fritz-astronomer added kind:feature Feature Requests needs-triage label for new issues that we didn't triage yet labels Jun 21, 2024
@fritz-astronomer
Copy link
Contributor Author

acc'd to this https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/operators.html#jinja-templating
it looks like ObjectStoragePath doesn't have template_fields =

I tried a few things:

ObjectStoragePath.template_fields = (...)
doesn't work, because the init field is *args (I tried with args)

I can't do render_template(...) in my operator, I think, because ObjectStoragePath is read-only

@shahar1 shahar1 added area:core and removed needs-triage label for new issues that we didn't triage yet labels Jun 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants