Code

Add tokenizer test + revert to C++11#355

Code

Merged

ggerganov merged 2 commits intomasterggml-org/llama.cpp:masterfrom tokenizer-testggml-org/llama.cpp:tokenizer-testCopy head branch name to clipboardMar 21, 2023

+249-148

Commits

Commits on Mar 21, 2023

• Add tokenizer unit test + vocab-only data for tests

  ggerganov

  committed
  ecd982d

  Copy full SHA for ecd982d

• Revert back to C++11, avoid std::string_view in the tokenizer

  ggerganov

  committed
  a0d00bd

  Copy full SHA for a0d00bd