Explicitly error when an agent is requested without an API key

[joellabes]

I don't have a Codex API configured. ade run airbnb001 --db snowflake --agent codex --project-type dbt-fusion results in

AGENT RESPONSE: 
Reading API key from stdin...
No API key provided via stdin.

but the whole thing continues such that the job fails due to a failing dbt test. It'd be better to throw a specific error.

[bstancil]

@RobertIsmo this is probably a good thing to check in the pre-flight stuff (or just as a thing that happens when the harness runs with a given agent).

That said, we should probably deal with it in the runs too. @joellabes, when this happens, it just keeps plowing ahead with the test, and doesn't log a harness error and exit?

[joellabes]

@joellabes, when this happens, it just keeps plowing ahead with the test, and doesn't log a harness error and exit?

That's what I saw with the codex agent, yes. idk if it varies from agent to agent depending on their CLI (I assume it's consistent though)

[RobertIsmo]

Actually just a minute ago I just added a state checker to the CLI that checks your environment to see if you have a key set or not and gives you information about agents you want to use. @bstancil was wondering if we want to put some of these checks into the harness as well just to stop odd behavior like running with an agent you don't have a key for etc, I am actually okay with keeping the checker as seperate though and allow the user to do with that info what they will.

@joellabes pull, uv reinstall, and run ade check all right now it doesn't do anything too crazy, but the goal is for this to check a variety of stuff with someones environment to resolve #26 if it's not really hitting your goals lmk, major goal is for it not just check state, but to also give suggestions and solutions on how to get it to the point someone wants it to be at, and give a summary of current capabilities, basically tutorialware.

[RobertIsmo]

Other interesting note is that atm the checker does not check if the key is valid, give me some time to think about it properly, but I think I will setup a container to try the keys that are available to see if they are valid, because goal would be for the user to not actually have to install any coding assistants on the host itself.

[joellabes]

ade check all is cool - feels in line with the debug workflow Doug describes in 26.

I think what I'm looking for here is basically the same as I just did in #75, where if the Snowflake configuration fails, then it early-exits. When invoking the LLM, if something goes wrong (invalid or missing API key), then the outcome shouldn't be a dbt compilation error.

Current:

(ade-bench) joel@Joel-Labes ade-bench % ade run airbnb001 --db snowflake --agent codex --project-type dbt-fusion
12:17:29 | system                           | START        | STARTING HARNESS RUN
12:17:29 | system                           | START        | Run ID: 2025-12-09__12-17-29
12:17:29 | system                           | START        | Running with snowflake + dbt-fusion. Found 1 tasks of 1 requested tasks.
┏━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Time     ┃ Task                             ┃ Stage        ┃ Message                                                                                              ┃
┡━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ 12:18:05 │ airbnb001                        │ ✓ ✓ ✓  DONE  │ FAIL - dbt compilation error found                                                                   │
│ ──────── │ ──────────────────────────────── │ ──────────── │ ──────────────────────────────────────────────────────────────────────────────────────────────────── │
│ 12:18:05 │ SUMMARY                          │              │ Run 1 of 1 tasks, 0 successful (0.0%)                                                                │
└──────────┴──────────────────────────────────┴──────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────────┘
12:18:05 | system                           | FINISH       | Generated HTML dashboard.
12:18:05 | system                           | FINISH       | Harness run completed: 0 of 1 tasks successful

Ideal:

...
┏━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Time     ┃ Task                             ┃ Stage        ┃ Message                                                                                              ┃
┡━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ 12:18:05 │ airbnb001                        │ ✓ ✓ ✓  DONE  │ FAIL - No API key provided via stdin                                                                   │
└──────────┴──────────────────────────────────┴──────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────────┘
...

[bstancil]

Yeah, agreed. The main thing we want to make sure on these things is that the errors that are harness errors do not get recorded as eval errors. I'm mixed on if we should halt the whole harness (and in general, lean no), but we definitely have to be careful about having harness errors get misinterpreted, because that breaks the results.

[bstancil]

I added a way to handle rate limit stuff today, and we can probably fix this issue in the same way. It's a bit annoying, because you have to basically parse text (and do it by agent, since they aren't consistent), but the framework is there in all the agents now:

eg, codex: https://github.com/dbt-labs/ade-bench/blob/main/ade_bench/parsers/codex_parser.py#L63-L74
gemini: https://github.com/dbt-labs/ade-bench/blob/main/ade_bench/parsers/gemini_parser.py#L81-L89

[joellabes]

Yeah, agreed. The main thing we want to make sure on these things is that the errors that are harness errors do not get recorded as eval errors [...] we definitely have to be careful about having harness errors get misinterpreted, because that breaks the results.

100%. Is there a mechanism for that today which we can use to catch (and suppress from analysis) these problems? It doesn't look like it today but I haven't dug deep to be sure. As another example, I'm setting up quickbooks to work with Snowflake and something is going wrong during setup, but it's still continuing all the way to test execution and resulting in an eval_error. If it had been classified as a harness error, would it have been excluded from the aggregate result summaries? Trying to work out if the problem is only the harness error was misclassified, or both that the harness error was misclassified and even if it wasn't harness errors are considered in result aggregation.

I'm mixed on if we should halt the whole harness (and in general, lean no)

Agreed - I don't think we need to halt the whole thing.