Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Simple Baselines (CoT, Vanilla, Retries, Warming), Renaming Agent Classes, Adding Prompting component & core module #261

Merged
merged 58 commits into from
Sep 1, 2024

Conversation

alckasoc
Copy link
Member

@alckasoc alckasoc commented Aug 28, 2024

🤔 Reasoning

1 - Zero-shot/Standard prompt (generate n-samples)
2 - Chain-of-Thought (generate n-samples)
3 - Retries (up to 5)
4 - Warming (gradually increase temperature from 0 to 0.5)

Generate N-samples
Generate N-samples with Retries/Warming/Rejection Sampling
Escalation (cheap to expensive models)
Chain-of-Thought Self-Consistency
Rejection Sampling

-> agential.cog.baselines.cot X
-> agential.cog.cot
-> nothing, you just write it yourself X

Retries -> Zero-shot or CoT?
Warming -> Zero-shot or CoT?
Retries Warming
Zero-shot/Standard X X
Chain-of-Thought

🚧 Changes

Describe the changes made...

✅ PR Checklist

  • Using this PR template?
  • Linked issue?
  • Added feature?
    • Added/updated docs?
    • Added/updated tests?

@alckasoc alckasoc added this to the Baselines milestone Aug 28, 2024
@alckasoc alckasoc self-assigned this Aug 28, 2024
@alckasoc alckasoc linked an issue Aug 28, 2024 that may be closed by this pull request
Copy link

codecov bot commented Aug 29, 2024

@alckasoc alckasoc added enhancement New feature or request add-benchmark Adding support for a benchmark labels Aug 31, 2024
@alckasoc alckasoc changed the title Simple Baselines Simple Baselines (CoT, Vanilla, Retries, Warming), Renaming Agent Classes, Adding Prompting component & core module Aug 31, 2024
@alckasoc alckasoc merged commit 6e03f4c into main Sep 1, 2024
5 checks passed
@alckasoc alckasoc deleted the vincent/baseline branch September 1, 2024 11:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
add-benchmark Adding support for a benchmark enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature Request]: Add simple baselines
2 participants