feat: add "stop" keywords as alternative to eot token #769
Conversation
| for (auto id : last_n_tokens) { | ||
| last_output += llama_token_to_str(ctx, id); | ||
| } | ||
| } | ||
|
|
||
| // Check for stop keywords, a configurable alternative to the end-of-text token |
There was a problem hiding this comment.
we should check the token id instead of the string for stop.
There was a problem hiding this comment.
Indeed, this should work in the same way as "antiprompts" to provide a better UX to users. Users should be able to add to the CLI parameters --stop "### Assistant" (for example, in the spirit of the trained vicuna model) or --stop PAUSE (for example, to implement Simon Willinson's ReAct Python example), even though these are multi-token markers.
There was a problem hiding this comment.
do we know why? It will be a good learning for me to understand why token will not work? Because one stop string can be generated by the different tokens??
There was a problem hiding this comment.
I also noticed that the stop words weren't always consistently formatted. Sometimes it would do STOP instead of stop. I'd appreciate a normalize function to down case before comparing.
|
Yes, I got busy and no longer have time to maintain that PR. Thanks for rewriting! |
|
While #863 appears to be more polished, I'd merge these changes since it's uncertain when that one will be reviewed and integrated. Plus, it's likely that any significant use of this code would involve utilizing the library rather than relying on this example binary. |
|
@longregen |
#863 is a superset of this PR. Closed this one as that other got merged. Thank you for your awesome work! |
|
@longregen The PR you mentioned has been reverted (c85e03d). Is the plan to wait for that to be tested more or should we reopen this PR? |
Rewrite of #365 (addresses #57) by @joshmackwilliams, updated to the new folder/file structure as the PR might have been abandoned.
From the original author: