Skip to content

Conversation

leondz
Copy link
Collaborator

@leondz leondz commented May 7, 2025

Support construction-time loading of optional modules. Includes

  • many generators now using this pattern
  • pattern for loading modules at run-time and failing if absent
  • optional requirements moved to pyproject.toml options and pruned from requirements.txt
  • _load_deps and _clear_deps pattern, used in generator constructor and _load_client / _clear_client
  • tests to check that optional deps are in the right place and not in the wrong place

Todo / in scope:

  • skipping tests requiring absent optional imports
  • take a stance on whether to skip or stop on missing module, per plugin type (likely skip for probe, stop for main generator, beyond that tba)
  • Consider moving load/clear dep function defs up and out for all classes (where - garak._plugins? How - assign func member? create mixin?)
  • validation: exception throws correctly / doesn't throw if dependency present; module still runs
    • generators.cohere
    • generators.langchain
    • generators.litellm
    • generators.mistral
    • generators.nemoguardrails
    • generators.nemollm
    • generators.ollama
    • generators.optimum (huggingface.OptimumPipeline) - exception fires, pkg install seems borked
    • generators.replicate
    • probes.audio

Not done:

  • load/clear deps for plugin types other than generators
  • gh actions for testing optional components
  • deeper validation

Out of scope:

  • handling of versioning outside of pyproject.toml

Resolves #101

@leondz leondz requested a review from jmartin-tech May 7, 2025 11:00
@leondz leondz added the architecture Architectural upgrades label May 7, 2025
@leondz leondz requested review from jmartin-tech and removed request for jmartin-tech May 7, 2025 11:04
@leondz leondz added this to the 0.11.0 milestone May 8, 2025
@leondz leondz self-assigned this May 8, 2025
@leondz leondz removed this from the release 0.11.0 milestone May 8, 2025
@leondz
Copy link
Collaborator Author

leondz commented Jul 3, 2025

need to solve how optional/deferred testing works

@leondz leondz mentioned this pull request Aug 5, 2025
@erickgalinkin erickgalinkin linked an issue Aug 12, 2025 that may be closed by this pull request
@leondz leondz mentioned this pull request Aug 28, 2025
1 task
@leondz leondz marked this pull request as ready for review September 26, 2025 09:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
architecture Architectural upgrades
Projects
None yet
Development

Successfully merging this pull request may close these issues.

tests: test garak's loading performance improved / consistent plugin lazy loading
3 participants