Code

Add Support For Two String Inputs#422

Code

Draft

pavel-esir wants to merge 6 commits intoopenvinotoolkit:masteropenvinotoolkit/openvino_tokenizers:masterfrom pavel-esir:add_several_inputspavel-esir/openvino_tokenizers:add_several_inputsCopy head branch name to clipboard

+577-26

Commits

Commits on Feb 20, 2025

• solution with concat

  pavel-esir

  committed
  42cee5f

  Copy full SHA for 42cee5f

• with Reshape

  pavel-esir

  committed
  dc7ad0a

  Copy full SHA for dc7ad0a

Commits on Feb 21, 2025

• successfully works on tokenizers

  pavel-esir

  committed
  8a99e98

  Copy full SHA for 8a99e98

Commits on Feb 25, 2025

• add max_length truncation for pair input

  pavel-esir

  committed
  9a84022

  Copy full SHA for 9a84022

• add check_tokenizer.ipynb

  pavel-esir

  committed
  b3d5c70

  Copy full SHA for b3d5c70

• corrected division by half

  pavel-esir

  committed
  78c5097

  Copy full SHA for 78c5097