tests: add LLM based test generation tool #59626
Conversation
ti-chi-bot
bot
added
release-note-none
YangKeao
force-pushed
the
add-llm-test
branch
3 times, most recently
from
b167ea9 to
edd17d4
Compare
|
/retest |
YangKeao
force-pushed
the
add-llm-test
branch
2 times, most recently
from
cca5df7 to
f20e3ba
Compare
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #59626 +/- ##
================================================
+ Coverage 73.0044% 73.7579% +0.7535%
================================================
Files 1697 1697
Lines 468869 471868 +2999
================================================
+ Hits 342295 348040 +5745
+ Misses 105521 102878 -2643
+ Partials 21053 20950 -103
Flags with carried forward coverage won't be shown. Click here to find out more.
|
|
|
||
| func createGenerateCmd() *cobra.Command { | ||
| var ( | ||
| openaiToken string |
There was a problem hiding this comment.
Can I use the other LLM such as deepseek?
There was a problem hiding this comment.
Sure. The API of deepseek is compatible with OpenAI. Actually, I personally used deepseek to run this tool on my machine.
I've ran this tool with OpenAI gpt-4o, claude sonnet and deepseek-r1. Deepseek R1 gave the best result. Nearly every SQLs generated by it are valid, while the SQLs generated by other models will have some grammar errors 🤦 and cannot handle the escape character well. (But I haven't tested with gpt-o1 or newer models).
YangKeao
force-pushed
the
add-llm-test
branch
from
f20e3ba to
20a72c8
Compare
| @@ -0,0 +1,218 @@ | |||
| { | |||
| "delete": [ | |||
There was a problem hiding this comment.
Actually I'm not sure whether it's good to submit all generated test cases to the repo 🤔 .
There was a problem hiding this comment.
You can add some documents about it.
|
|
||
| func createGenerateCmd() *cobra.Command { | ||
| var ( | ||
| openaiToken string |
There was a problem hiding this comment.
Sure. The API of deepseek is compatible with OpenAI. Actually, I personally used deepseek to run this tool on my machine.
I've ran this tool with OpenAI gpt-4o, claude sonnet and deepseek-r1. Deepseek R1 gave the best result. Nearly every SQLs generated by it are valid, while the SQLs generated by other models will have some grammar errors 🤦 and cannot handle the escape character well. (But I haven't tested with gpt-o1 or newer models).
YangKeao
force-pushed
the
add-llm-test
branch
3 times, most recently
from
a03170b to
0d0045e
Compare
ti-chi-bot
bot
added
approved
needs-1-more-lgtm
|
/retest |
1 similar comment
|
/retest |
YangKeao
force-pushed
the
add-llm-test
branch
2 times, most recently
from
26a1b2c to
9d6cce0
Compare
|
/retest |
Signed-off-by: Yang Keao <[email protected]>
YangKeao
force-pushed
the
add-llm-test
branch
from
9d6cce0 to
f7349e7
Compare
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: bb7133, hawkingrei The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
ti-chi-bot
bot
added
lgtm
and removed
needs-1-more-lgtm
[LGTM Timeline notifier]Timeline:
|
What problem does this PR solve?
Issue Number: close #59625
Problem Summary:
What changed and how does it work?
expression,dml, andctefeature. However, it only found bugs forexpressionfor now (it's kind of expected from my perspective. TiDB is good!).Reference to https://github.com/YangKeao/tidb/blob/add-llm-test/tests/llmtest/README.md if you want to know how to use it.
Check List
Tests
Side effects
Documentation
Release note
Please refer to Release Notes Language Style Guide to write a quality release note.