OPAL topics and configuring data sources

[crystalrage777]

Hi,

Trying to understand how I should design my opal data sources, taking into account what types of queries should they support. making sure they don't limit the querying ability of OPAL.

My current direction is having a http server (and a mongo server under it).

From my understanding, when the client needs data it queries it based on the topic of the data. given the topic, it knows what query it should run (and what data source it should run it against). and that the query is a bulk query, meaning - say I have a topic of users&roles that holds documents each holding tuples of "user", "role", "resource" (user x has role y regarding resource z) and my query to OPA references that topic then it will send a single query for getting all these user-role-resource tuples (not a sub group of them).

If I'm correct in my understanding my http client needs to contain one endpoint per topic that returns a json that is a list of the documents in the topic.

Is my understanding correct?

thanks

[idan-ben-ami]

Hello crystalrage777,

You almost got it right 🙂.
It is true that OPAL client fetches data from a configured endpoint based on the topic. OPAL client is triggered by the OPAL server to fetch data - and the triggering information contains the topic telling the OPAL client which data to fetch.

Your endpoint should return whichever JSON data you’d like to locate on the OPA hierarchy tree. (The location of the data on the OPA tree is also configurable in the same place you configured the endpoint fetcher for a topic).

A clarification: you can configure different fetching behaviors for different topics:

• From where to fetch the data (url)
• How to fetch (optional)
• Where to place it in the OPA hierarchy tree. (dst_path - optional, default to ‘/’)

So two different http server implementation strategies for different topics would be:

• A single endpoint per topic.
• A single endpoint for all topics, with query parameters in the url.
  Both cases are done using the “url” parameter in the DataSourceParameter

For more information on when OPAL server trigger data fetching, please see:
https://docs.opal.ac/tutorials/trigger_data_updates

For more information on configuring OPAL_DATA_CONFIG_SOURCES, please see:
https://docs.opal.ac/getting-started/get_started_with_opal_using_docker#step-5-server-config---data-sources

Don’t hesitate to reach out if something is unclear!