Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tune AD and alerting tool descriptions #154

Merged
merged 1 commit into from
Jan 29, 2024

Conversation

ohltyler
Copy link
Member

Description

This PR enhances the default descriptions of the AD and alerting tools in several ways:

  • more specific on the expected input types and returned output types (e.g., timestamps in epoch milliseconds format) to help guide the LLM to have valid suggestions when the tool generates the relevant search query / API calls.
  • adds options for params that are not clearly defined or interfaced (e.g., monitor types, alert state, alert severity level)
  • minor wording changes

It also fixes the parsing of the SearchMonitorsTool to parse monitor_type instead of type. type always is monitor, but monitor_type provides more detail, such as if the monitor is a query-level, bucket-level, or document-level monitor, which is more useful / user-expected

Variety of questions were tested on a test cluster with a provisioned LLM, and this PR is a direct result of where gaps were seen to help increase the effectiveness and accuracy of the tool itself.

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Copy link

codecov bot commented Jan 29, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (722bfd2) 81.73% compared to head (0b51d31) 81.73%.

Additional details and impacted files
@@            Coverage Diff            @@
##               main     #154   +/-   ##
=========================================
  Coverage     81.73%   81.73%           
  Complexity      196      196           
=========================================
  Files            13       13           
  Lines          1002     1002           
  Branches        132      132           
=========================================
  Hits            819      819           
  Misses          133      133           
  Partials         50       50           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@ohltyler
Copy link
Member Author

All tests are passing besides known issue with neural sparse search tool:

30 tests completed, 2 failed
Tests with failures:
 - org.opensearch.integTest.NeuralSparseSearchToolIT.testNeuralSparseSearchToolInFlowAgent_withIllegalSourceField_thenGetEmptySource
 - org.opensearch.integTest.NeuralSparseSearchToolIT.testNeuralSparseSearchToolInFlowAgent_withIllegalEmbeddingField_thenThrowException

@ohltyler ohltyler merged commit 1626af1 into opensearch-project:main Jan 29, 2024
12 of 15 checks passed
@ohltyler ohltyler deleted the tuning branch January 29, 2024 21:31
opensearch-trigger-bot bot pushed a commit that referenced this pull request Jan 29, 2024
Signed-off-by: Tyler Ohlsen <[email protected]>
(cherry picked from commit 1626af1)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
ohltyler pushed a commit that referenced this pull request Jan 29, 2024
(cherry picked from commit 1626af1)

Signed-off-by: Tyler Ohlsen <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
zhichao-aws pushed a commit to zhichao-aws/skills that referenced this pull request Jan 30, 2024
yuye-aws pushed a commit to yuye-aws/skills that referenced this pull request Apr 26, 2024
…nsearch-project#155)

(cherry picked from commit 1626af1)

Signed-off-by: Tyler Ohlsen <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Signed-off-by: yuye-aws <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants