Add split option to split large input into smaller parts instead of truncating #2
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit introduces a new option (-s, --split) to the command-line interface
of the token counting script. The new option allows users to split their input
text into multiple files, each containing a specific number of tokens.
When the split option is used in conjunction with the truncate option (-t, --truncate),
the script splits the tokens into groups of the specified size and writes each group
into a separate file. The output files are named using the provided filename and are
appended with '.part' followed by the chunk index.
A new helper function 'grouper' has been added to divide the iterable (tokens) into
fixed-size chunks.
The split function ensures that each chunk is decoded before being written to its
respective output file.
|
By the way, thank you for the great handy tools (llm / ttok / strip-tags). |
|
I opened an issue to discuss this here: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Sometimes, I don't want to discard the remaining output, might be useful :)
So, this pull request adds a new feature to the token counting command-line interface. The new -s or --split option allows users to split their input text into multiple files, each containing a specific number of tokens.
Changes
click.command()decorator in theclifunction.clifunction has been updated to handle this new option. When thesplitandtruncateoptions are used together, the script splits the input tokens into groups of the specified size and writes each group to a separate file.grouperhas been added to split an iterable into fixed-length chunks.