Skip to content

Commit

Permalink
potential out of bounds read
Browse files Browse the repository at this point in the history
  • Loading branch information
eiz committed Mar 18, 2023
1 parent 5ae5e2c commit c67e5ef
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion utils.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -324,7 +324,7 @@ struct llama_tokenizer {
while (!text.empty()) {
llama_sp_symbol sym;
size_t char_len = utf8_len(text.data()[0]);
sym.text = std::string_view(text.data(), char_len);
sym.text = std::string_view(text.data(), std::min(text.size(), char_len));
sym.prev = index - 1;
text.remove_prefix(char_len);
sym.next = text.empty() ? -1 : index + 1;
Expand Down

0 comments on commit c67e5ef

Please sign in to comment.