Missing data in DATADUMP-FED API action

[geoffroya]

Defect/Feature description

As a NRO, we are tring to see how we can make use of the existing admin API.
Our first goal is to get data from CAT about our institution, to identify misconfigured IdPs and next guide them to fix issues.

For this, we tested the DATADUMP-FED action:
POST /admin/API.php
Body:

{
    "ACTION": "DATADUMP-FED",
    "APIKEY": "*******",
    "PARAMETERS": []
}

Unfortunately, we found out that:
1- useful data is missing, like the instid provided by eduroam database we expose as a NRO - that could alow us to join data from CAT API and data from our national registry
2- the eap certificates are sometimes present, sometimes not. We failed understanding why it's present for some institutions and missing for others. Of course, we checked that for the missing ones, they are present in CAT UI.

As a NRO, I would expect to have complete information about what's in CAT for the institutions we manage.

Nevertheless, maybe we used the API the wrong way. If so please tell us what we did wrong.

Best regards

[restena-sw]

Hi,

ad 1) CAT 2.0.x still uses the eduroam DB v1 format for correlating a CAT inst with a eduroam-DB-inst. This means that while the eduroam DB has a unique identifier for the inst, that is an internally assigned ID which does not correlate to the identifiers you send in your XML/JSON file (with DB version 2). Since the DB identifier is (still) the internal one from DBv1 times, there is little point in sharing this with anyone.
If there are any other data points you find useful but missing, please elaborate which data points you are thinking of.

ad 2) It would really help me understand what's going on there if you could give me a few examples - the CAT inst_id of organisations with missing CA data would be enough.

[geoffroya]

Hi

1. I will need some time to elaborate on this. Basically, I would love to get everything about each info declared by IdP admins => I will come back soon with more precise needs.

2. Some working internal IDs : 30, 84, 218
   Some non working internal IDs : 36, 50, 217

[restena-sw]

Can you execute the DATADUMP-FED call and send me the resulting file to my personal inbox? Thanks!

[restena-sw]

This was informative. The API call dumps all the information present on the IdP level. If the IdP admin specifies the CAs to use as IdP-wide attributes, they are returned by the API call.

CAs defined on a per-profile basis are not considered because the API call doesn't dive that "deep" into the data set.

E.g. the inst with ID 36 does not have IdP-level CAs defined, but has two profiles, each with their own CA info ( profile_id 42, "eduroam_permanents_..." and profile_id 7852 "tests_sectigo").

Since different profiles can have different CAs, it is not a good idea to join them all together in a mish-mash on the IdP level. The most proper thing to do would be to change the API call to send all nested info about all the profiles inside the IdP data strucuture.

I note however that the data set you sent me was already 42 MB with just IdP-level information; I fear it may become difficult to handle if we were to cram "everything" into it. Please reflect on this and let me know if this really is what you want.

[geoffroya]

Given the size of all this, I can think of a different approach:

Call to DATADUMP-FED provides something that could be used for example to build a table with 1 row per institution:

{
    "result": "SUCCESS",
    "details": {
        "<CAT internal ID>": [
          {
            "name": "general:inst_id",
            "value": "<Institution ID provided by NRO>",
            ... // whatever other fields the legacy code requires to fill
          },
          // Other "level 0" information - no info from profile(s)
          // - general:* / exclude logo
          // - support:* 
        ]
    }
}

A new action, DATADUMP-IDP (or DATADUMP-FED when ATTRIB-CAT-INSTID is provided), could bring back detailed info:

POST:

{
    "ACTION": "DATADUMP-IDP", // or DATADUMP-FED if behaviour vhanges based on ATTRIB-CAT-INSTID presence
    "APIKEY": "*******",
    "PARAMETERS": [
        {
            "NAME": "ATTRIB-CAT-INSTID",
            "VALUE": "30"
        }
    ]
}

Result would include:

• all "level 0" data
• all data from all profiles - but this would require nesting and adding data:

{
    "result": "SUCCESS",
    "details": {
        "<CAT internal ID>": [ 
          {
            "name": "general:inst_id",
            "value": "<Institution ID provided by NRO>",
            ... // whatever other fields the legacy code requires to fill
          },
          // ...
          {
            "profiles": {
              "<profile #1 name>": [
                // Certificates: BTW: how do you distinguish CA and Intermediate(s)?
                {
                  "eap:ca_file",
                  // ...
                },
                {
                  "eap:ca_file",
                  // ...
                },
                // Supported EAP Methods
                // Profile properties (desc, name, ready for production, realm, outer identity...)
                // Local overriden data (support, network)
              ],
              //...
              "<profile #N name>": [
                
              ]
            }
          }
      ]
  }
}

I understand that it could be a lot of work.

From our NRO point of view, the needs are:

• Retreiving the CAT INST ID from our local inst ID, so we can use API for other purposes (decreasing priority):
  • Analyse CAT profiles (identify Sectigo AC issues, missing outer identity...) and provide advices to institutions
  • Automate CAT tokens granting, through ACTION_ADMIN_ADD
  • Automate User management in hosted.eduroam.org

Best regards