Thoughts on using KDL for Hipcheck configuration under the new plugin system.

[alilleybrinker]

I want to explore two things with Hipcheck configuration:

• Possibly using KDL instead of TOML as the configuration language.
• How to handle configuration under the new plugin system we're building.

The following is a sample KDL file that represents one idea of how Hipcheck config could look with plugins and in KDL (you can enter the full text below into the KDL Playground to see what it would look like):

Version 1

/*
 * The following is a sample configuration for Hipcheck
 * using a possible KDL syntax. Note that for any "analysis"
 * block, there are only three parameters permitted:
 *
 * - A string denoting the name of the analysis, of the form
 *   "<producer>:<plugin>:<analysis>:<version>".
 * - A weight (optional, defaults to 1)
 * - A child block.
 *
 * The contents of the child block are passed to the plugin
 * identified by the string, which validates them and uses
 * them to configure itself.
 *
 * Plugins are expected to use provided functions to enable
 * Hipcheck to report useful errors back to the user if a
 * plugin has been given invalid configuration.
 *
 * Plugins are encouraged to use defaults in their configuration
 * so users can employ the plugin easily.
 *
 * Note that Hipcheck will report scores in the context of a
 * hash produced from this configuration with all non-specified
 * values filled-in with defaults. This is intended to show that
 * in Hipcheck, scores are _only_ comparable with other scores
 * produced under the same configuration.
 */

// Use this analysis configuration when targeting source
// repositories or things that resolve to them like packages
// pulled from package hosts like NPM, Maven, and PyPI.
analyze target="source_repository" {
    investigate_above score=0.5

    category "practices" weight=1 {
        // Maybe we really care if a project is inactive.
        analysis "mitre:std:activity:1" weight=3 {
            flag_if_inactive_more_than weeks=52
        }

        analysis "mitre:std:binary:1" weight=1 {
            flag_if_find_more_than binaries=0
        }

        analysis "mitre:std:fuzz:1"

        analysis "mitre:std:review:1" {
            check_github_prs {
                flag_if_unreviewed_above percent=0.05
            }

            check_gitlab_prs {
                flag_if_unreviewed_above percent=0.05
            }

            check_committer_identity {
                flag_if_same_identity_above percent=0.05
            }
        }
    }

    category "attacks" weight=1 {
        analysis "mitre:std:typo:1" weight=1 {
            typo_file "./config/typo.kdl"
        }

        category "commit" weight=1 {
            analysis "mitre:std:affiliation:1" weight=1 {
                flag_if_found_more_than count=0
                orgs_file "./config/orgs.kdl"
            }

            analysis "mitre:std:entropy:1" weight=1 {
                flag_if_high_entropy_commits_above count=0
                commit_is_high_entropy_if_above score=8.0
            }

            analysis "mitre:std:churn:1" weight=1 {
                flag_if_high_churn_commit_above count=0
                commit_is_high_churn_if_above score=3.0
            }
        }
    }

    // Anything listed here causes an automatic "investigate"
    // recommendation, regardless of score, if the analysis
    // identified results in a flag.
    //
    // Note that any analysis identified here must be present
    // as an "analysis" block elsewhere in the same "analyze"
    // block, otherwise Hipcheck will produce an error.
    auto_investigate_if_flagged "mitre:std:affiliation:1" \
                                "mitre:std:binary:1"
}

// Use this analysis configuration when targeting change
// requests like GitHub Pull Requests, GitLab Merge Requests,
// or plain Git patch files.
analyze target="change_request" {
    investigate_above score=0.3

    category "commits" {
        analysis "mitre:std:commit_trust:1"

        analysis "mitre:std:change_affiliation:1" {
            flag_if_found_more_than count=0
            orgs_file "./config/orgs.kdl"
        }

        analysis "mitre:std:change_module_contributors:1" weight=5 {
            flag_if_found_more_than percent=0.3
        }
    }

    auto_investigate_if_flagged "mitre:std:change_affiliation:1"
}

Version 2

/*
 * The following is a sample configuration for Hipcheck
 * using a possible KDL syntax. Note that for any "analysis"
 * block, there are only four parameters permitted:
 *
 * - A string denoting the name of the analysis, of the form
 *   "<producer>:<plugin>:<analysis>:<version>".
 * - A weight (optional, defaults to 1)
 * - If the failure should cause auto-investigate (defaults to false)
 * - A child block.
 *
 * The contents of the child block are passed to the plugin
 * identified by the string, which validates them and uses
 * them to configure itself.
 *
 * Plugins are expected to use provided functions to enable
 * Hipcheck to report useful errors back to the user if a
 * plugin has been given invalid configuration.
 *
 * Plugins are encouraged to use defaults in their configuration
 * so users can employ the plugin easily.
 *
 * Note that Hipcheck will report scores in the context of a
 * hash produced from this configuration with all non-specified
 * values filled-in with defaults. This is intended to show that
 * in Hipcheck, scores are _only_ comparable with other scores
 * produced under the same configuration.
 */

// Use this analysis configuration when targeting source
// repositories or things that resolve to them like packages
// pulled from package hosts like NPM, Maven, and PyPI.
analyze target="source_repository" {
    investigate_above score=0.5

    category "practices" weight=1 {
        // Maybe we really care if a project is inactive.
        analysis "mitre:std:activity:1" weight=3 {
            flag_if_inactive_more_than weeks=52
        }

        analysis "mitre:std:binary:1" \
                 weight=1 \
                 investigate_if_flagged=true \
        {
            flag_if_find_more_than binaries=0
        }

        analysis "mitre:std:fuzz:1"

        analysis "mitre:std:review:1" {
            check_github_prs {
                flag_if_unreviewed_above percent=0.05
            }

            check_gitlab_prs {
                flag_if_unreviewed_above percent=0.05
            }

            check_committer_identity {
                flag_if_same_identity_above percent=0.05
            }
        }
    }

    category "attacks" weight=1 {
        analysis "mitre:std:typo:1" weight=1 {
            typo_file "./config/typo.kdl"
        }

        category "commit" weight=1 {
            analysis "mitre:std:affiliation:1" \
                     weight=1 \
                     investigate_if_flagged=true \
            {
                flag_if_found_more_than count=0
                orgs_file "./config/orgs.kdl"
            }

            analysis "mitre:std:entropy:1" weight=1 {
                flag_if_high_entropy_commits_above count=0
                commit_is_high_entropy_if_above score=8.0
            }

            analysis "mitre:std:churn:1" weight=1 {
                flag_if_high_churn_commit_above count=0
                commit_is_high_churn_if_above score=3.0
            }
        }
    }
}

// Use this analysis configuration when targeting change
// requests like GitHub Pull Requests, GitLab Merge Requests,
// or plain Git patch files.
analyze target="change_request" {
    investigate_above score=0.3

    category "commits" {
        analysis "mitre:std:commit_trust:1"

        analysis "mitre:std:change_affiliation:1" \
                 investigate_if_flagged=true \
        {
            flag_if_found_more_than count=0
            orgs_file "./config/orgs.kdl"
        }

        analysis "mitre:std:change_module_contributors:1" weight=5 {
            flag_if_found_more_than percent=0.3
        }
    }
}

Version 3

/*
 * The following is a sample configuration for Hipcheck
 * using a possible KDL syntax. Note that for any "analysis"
 * block, there are only four parameters permitted:
 *
 * - A string denoting the name of the analysis, of the form
 *   "<producer>:<plugin>:<analysis>:<version>".
 * - A weight (optional, defaults to 1)
 * - If the failure should cause auto-investigate (defaults to false)
 * - A child block.
 *
 * The contents of the child block are passed to the plugin
 * identified by the string, which validates them and uses
 * them to configure itself.
 *
 * Plugins are expected to use provided functions to enable
 * Hipcheck to report useful errors back to the user if a
 * plugin has been given invalid configuration.
 *
 * Plugins are encouraged to use defaults in their configuration
 * so users can employ the plugin easily.
 *
 * Note that Hipcheck will report scores in the context of a
 * hash produced from this configuration with all non-specified
 * values filled-in with defaults. This is intended to show that
 * in Hipcheck, scores are _only_ comparable with other scores
 * produced under the same configuration.
 */

plugins {
    plugin "mitre:std:git:1" type="data" {
        fetch-from git="https://github.com/mitre/hipcheck" \
                   path="plugins/std/git" \
                   hash="c7b7db6f46e70efe9c4293ee71d0bdb1a329520b"
    }
    
    plugin "mitre:std:github:1" type="data" {
        fetch-from git="https://github.com/mitre/hipcheck" \
                   path="plugins/std/github" \
                   hash="c7b7db6f46e70efe9c4293ee71d0bdb1a329520b"
    }
    
    plugin "mitre:std:npm:1" type="data" {
        fetch-from git="https://github.com/mitre/hipcheck" \
                   path="plugins/std/npm" \
                   hash="c7b7db6f46e70efe9c4293ee71d0bdb1a329520b"
    }
    
    plugin "mitre:std:maven:1" type="data" {
        fetch-from git="https://github.com/mitre/hipcheck" \
                   path="plugins/std/maven" \
                   hash="c7b7db6f46e70efe9c4293ee71d0bdb1a329520b"
    }
    
    plugin "mitre:std:pypi:1" type="data" {
        fetch-from git="https://github.com/mitre/hipcheck" \
                   path="plugins/std/pypi" \
                   hash="c7b7db6f46e70efe9c4293ee71d0bdb1a329520b"
    }

    plugin "mitre:std:activity:1" type="analysis" {
        fetch-from git="https://github.com/mitre/hipcheck" \
                   path="plugins/std/activity" \
                   hash="c7b7db6f46e70efe9c4293ee71d0bdb1a329520b"
    }

    plugin "mitre:std:binary:1" type="analysis" {
        fetch-from git="https://github.com/mitre/hipcheck" \
                   path="plugins/std/binary" \
                   hash="c7b7db6f46e70efe9c4293ee71d0bdb1a329520b"
    }

    plugin "mitre:std:fuzz:1" type="analysis" {
        fetch-from git="https://github.com/mitre/hipcheck" \
                   path="plugins/std/fuzz" \
                   hash="c7b7db6f46e70efe9c4293ee71d0bdb1a329520b"
    }

    plugin "mitre:std:review:1" type="analysis" {
        fetch-from git="https://github.com/mitre/hipcheck" \
                   path="plugins/std/review" \
                   hash="c7b7db6f46e70efe9c4293ee71d0bdb1a329520b"
    }

    plugin "mitre:std:typo:1" type="analysis" {
        fetch-from git="https://github.com/mitre/hipcheck" \
                   path="plugins/std/typo" \
                   hash="c7b7db6f46e70efe9c4293ee71d0bdb1a329520b"
    }

    plugin "mitre:std:affiliation:1" type="analysis" {
        fetch-from git="https://github.com/mitre/hipcheck" \
                   path="plugins/std/affiliation" \
                   hash="c7b7db6f46e70efe9c4293ee71d0bdb1a329520b"
    }

    plugin "mitre:std:entropy:1" type="analysis" {
        fetch-from git="https://github.com/mitre/hipcheck" \
                   path="plugins/std/entropy" \
                   hash="c7b7db6f46e70efe9c4293ee71d0bdb1a329520b"
    }

    plugin "mitre:std:churn:1" type="analysis" {
        fetch-from git="https://github.com/mitre/hipcheck" \
                   path="plugins/std/churn" \
                   hash="c7b7db6f46e70efe9c4293ee71d0bdb1a329520b"
    }

    plugin "mitre:std:commit-trust:1" type="analysis" {
        fetch-from git="https://github.com/mitre/hipcheck" \
                   path="plugins/std/commit-trust" \
                   hash="c7b7db6f46e70efe9c4293ee71d0bdb1a329520b"
    }

    plugin "mitre:std:change-affiliation:1" type="analysis" {
        fetch-from git="https://github.com/mitre/hipcheck" \
                   path="plugins/std/change-affiliation" \
                   hash="c7b7db6f46e70efe9c4293ee71d0bdb1a329520b"
    }

    plugin "mitre:std:change-module-contributors:1" type="analysis" {
        fetch-from git="https://github.com/mitre/hipcheck" \
                   path="plugins/std/change-module-contributors" \
                   hash="c7b7db6f46e70efe9c4293ee71d0bdb1a329520b"
    }
}

// Use this analysis configuration when targeting source
// repositories or things that resolve to them like packages
// pulled from package hosts like NPM, Maven, and PyPI.
analyze target="source-repository" {
    investigate-above score=0.5

    category "practices" weight=1 {
        // Maybe we really care if a project is inactive.
        analysis "mitre:std:activity:1" weight=3 {
            flag-if-inactive-more-than weeks=52
        }

        analysis "mitre:std:binary:1" \
                 weight=1 \
                 investigate-if-flagged=true \
        {
            flag-if-find-more-than binaries=0
        }

        analysis "mitre:std:fuzz:1"

        analysis "mitre:std:review:1" {
            check-github-prs {
                flag-if-unreviewed-above percent=0.05
            }

            check-gitlab-prs {
                flag-if-unreviewed-above percent=0.05
            }

            check-committer-identity {
                flag-if-same-identity-above percent=0.05
            }
        }
    }

    category "attacks" weight=1 {
        analysis "mitre:std:typo:1" weight=1 {
            typo-file "./config/typo.kdl"
        }

        category "commit" weight=1 {
            analysis "mitre:std:affiliation:1" \
                     weight=1 \
                     investigate-if-flagged=true \
            {
                flag-if-found-more-than count=0
                orgs-file "./config/orgs.kdl"
            }

            analysis "mitre:std:entropy:1" weight=1 {
                flag-if-high-entropy-commits-above count=0
                commit-is-high-entropy-if-above score=8.0
            }

            analysis "mitre:std:churn:1" weight=1 {
                flag-if-high-churn-commit-above count=0
                commit-is-high-churn-if-above score=3.0
            }
        }
    }
}

// Use this analysis configuration when targeting change
// requests like GitHub Pull Requests, GitLab Merge Requests,
// or plain Git patch files.
analyze target="change-request" {
    investigate-above score=0.3

    category "commits" {
        analysis "mitre:std:commit-trust:1"

        analysis "mitre:std:change-affiliation:1" \
                 investigate-if-flagged=true \
        {
            flag-if-found-more-than count=0
            orgs-file "./config/orgs.kdl"
        }

        analysis "mitre:std:change-module-contributors:1" weight=5 {
            flag-if-found-more-than percent=0.3
        }
    }
}

Version 4

/*
 * The following is a sample configuration for Hipcheck
 * using a possible KDL syntax.
 */

plugins {
    plugin "mitre/git" version="1.0.0"
    plugin "mitre/github" version="1.0.0"
    plugin "mitre/npm" version="1.0.0"
    plugin "mitre/maven" version="1.0.0"
    plugin "mitre/pypi" version="1.0.0"
    plugin "mitre/activity" version="1.0.0"
    plugin "mitre/binary" version="1.0.0"
    plugin "mitre/fuzz" version="1.0.0"
    plugin "mitre/review" version="1.0.0"
    plugin "mitre/typo" version="1.0.0"
    plugin "mitre/affiliation" version="1.0.0"
    plugin "mitre/entropy" version="1.0.0"
    plugin "mitre/churn" version="1.0.0"

    // The following represents a download-based flow for getting
    // a plugin that isn't part of the Hipcheck plugin registry.
    //
    // In this case, the URL provided is to a manifest that
    // specifies how to get the plugin artifacts for any of the
    // architectures supported by the plugin.
    //
    // plugin "mitre/git" \
    //        version="^1.0.0" \
    //        manifest="https://github.com/mitre/hipcheck/releases/download/hipcheck-v3.4.0/hipcheck-git-manifest.kdl"
}

analyze {
    investigate policy="(gt 0.5)"
    investigate-if-fail "mitre/binary" "mitre/affiliation"

    category "practices" {
        analysis "mitre/activity" policy="(lte 52 $)" weight=3
        analysis "mitre/binary" policy="(eq 0 $)"
        analysis "mitre/fuzz" policy="(eq #t $)"
        analysis "mitre/review" policy="(lte 0.05 $)"
    }

    category "attacks" {
        analysis "mitre/typo" policy="(eq 0 $)" {
            typo-file "./config/typo.kdl"
        }

        category "commit" {
            analysis "mitre/affiliation" poilicy="(eq 0 $)" {
                orgs-file "./config/orgs.kdl"
            }

            analysis "mitre/entropy" policy="(eq 0 (count (filter (gt 8.0) $)))"
            analysis "mitre/churn" policy="(eq 0 (count (filter (gt 8.0) $)))"
        }
    }
}

/*
- `gt :: (Cmp T) => T -> T -> Bool`: Greater Than
- `lt :: (Cmp T) => T -> T -> Bool`: Less Than
- `gte :: (Cmp t) => T -> T -> Bool`: Greater Than or Equal To
- `lte :: (Cmp t) => T -> T -> Bool`: Less Than or Equal To
- `eq :: (Eq T) => T -> T -> Bool`: Equal To
- `neq :: (Eq T) => T -> T -> Bool`: Not Equal To
- `and :: (Eq T) => T -> T -> Bool`: And
- `or :: (Eq T) => T -> T -> Bool`: Or
- `xor :: (Eq T) => T -> T -> Bool`: Xor
- `not :: (Eq T) => T -> T -> Bool`: Not
- `all`: All
- `nall`: Not All
- `some`: Some
- `none`: None
- `max`: Max
- `min`: Min
- `avg`: Avg
- `median`: Median
- `filter`: Filter
- `count`: Count
*/

[alilleybrinker]

In terms of the following for how this configuration file is used, here is what I am imagining:

1. Hipcheck first checks if the configuration file itself meets the schema, and reports errors if not.
2. Then it checks if the analyses requested can all be found (can find the plugin by that producer, with that analysis and that version number), and reports errors if not.
3. Then it checks if any child block configuration is correct by passing it to the plugin for validation and reports errors if not.
4. Then it finds out the data needed from data plugins for all the requested analyses, as reported by them based on their configurations, and reports errors if those data sources can't be found.
5. Then it parallelizes the data collection and queues up analyses to run once their dependent data is ready.
6. Finally, when the final score is reported, it's reported alongside a hash produced from the configuration to limit the degree to which people try to compare scores made from different configurations.

[alilleybrinker]

Note that (unsurprisingly) plugins are central to the user experience of configuration in Hipcheck. Plugins themselves will need to be configurable, and we want this to be done in a way that still provides the best possible experience to the user. This will mean designing the APIs for plugins in a way that lets plugins report fine-grained errors, ideally with spans, so we can use something like miette to report very pretty and helpful error messages back to users!

The key thing to look at here is the Diagnostic trait in miette. This trait includes the following info, all of which is optional:

• code: A unique diagnostic code for the error, just has to be something that implements Display. For us, if we go with permitting error codes, we'd just need to make clear the difference between Hipcheck error codes and plugin error codes. I think this is doable.
• severity: Can be "Advice", "Warning", or "Error". Easy to expose.
• help: An explanatory help string that suggests what to do about the error.
• url: A URL to get more info about the error.
• source_code: Basically a representation of the config source code, used to print nicely labeled error messages.
• labels: A set of labeled spans into the source code (the config file) which is used to print nice error messages inline with a copy of the file itself.
• related: Any additional diagnostics related to the first one.
• diagnostic_source: A "parent" diagnostic related to the current one.

So basically you have 5 pieces of information which plugins might want to set for regular diagnostic metadata — code, severity, help, url, and labels — along with two relationship pieces of metadata — diagnostic_source and related — and Hipcheck itself can provide source_code.

This isn't too bad, and exposing an API to set all of this (while hiding the underlying implementation with miette from plugins) would mean plugins could report very rich and pretty error messages for end users to keep the quality level of the user experience high!

[alilleybrinker]

Another thing came to mind last night about this: the config file could include instructions for fetching plugins, which would mean the plugin state could be part of the score hash as well, and that an analysis is fully defined by its config (no ambient configuration outside of it).

I'll update the first post with a version 3 to reflect this.

[alilleybrinker]

One additional thing to consider is whether we want the config to include target-specific configuration. For example, if config for something specific to Maven needs to be set, it could be set in the config file.

[alilleybrinker]

One thing that occurs to me with the policy files proposed here is that we'd want to distinguish between the configuration of the individual plugins and the expression of the predicates used for scoring the results of an analysis.

[alilleybrinker]

A version of this design was merged as part of RFD 4!