Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

generators: add option to skip output btw given delimiters; add hook for postprocessing #1097

Merged
merged 14 commits into from
Feb 20, 2025

Conversation

leondz
Copy link
Collaborator

@leondz leondz commented Feb 3, 2025

  1. add generators.base.Generator._prune_skip_sequences() to cut content btw Generator.skip_seq_start and Generator.skip_seq_end
  2. add generators.base.Generator._post_generate_hook() to allow postprocessing of output in generators
  3. factor "test" generators out into their own module, for consistency w other plugin types
  4. handle openai exceptions caused by interrupted thought (e.g. when hitting max_tokens)
  5. add configurability to OpenAICompatible

Verification

  • python -m pytest tests/generators/test_generators_base.py
  • python -m pytest tests/generators/test_generators.py
  • Verify Run something with test.Repeat, it should work as normal (output==prompt)
  • Verify Configure skip_seq_start and skip_seq_end, content int he middle should be pruned
  • Docs updated in docs/source/garak.generators.base.rst

@leondz leondz requested a review from jmartin-tech February 3, 2025 10:16
@leondz leondz added the generators Interfaces with LLMs label Feb 3, 2025
@leondz leondz marked this pull request as ready for review February 3, 2025 14:11
@jmartin-tech jmartin-tech self-assigned this Feb 3, 2025
Copy link
Collaborator

@jmartin-tech jmartin-tech left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Testing shows this works however the long length of responses often truncates before the </think> output, often leaving the full response that started with <think>... in the result.

Also noted that I do not believe timeout should be suppressed by default for OpenAICompatible.

@jmartin-tech jmartin-tech merged commit 1139fce into NVIDIA:main Feb 20, 2025
9 checks passed
@github-actions github-actions bot locked and limited conversation to collaborators Feb 20, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
generators Interfaces with LLMs
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants