Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

gaz tokenize问题 #38

Open
DSXiangLi opened this issue May 11, 2021 · 2 comments
Open

gaz tokenize问题 #38

DSXiangLi opened this issue May 11, 2021 · 2 comments

Comments

@DSXiangLi
Copy link

你好我看bert tokenizer只对text进行了tokenize，如果碰到tokenizer把例如1994分成了19和##94, 但是gaz是针对每个character 1/9/9/4识别的BMES word，不会发生输入mismatch的问题么？
@v-mipeng
Copy link
Owner

很好的问题！这种情况可以将1/9/9/4得BMES标签进行合并处理，如原始的1/9/9/4标签为BMME，分割成19，##94后变成（19）B，（##94）E。这在一步的标签转换可以在tokenizer阶段进行。

@DSXiangLi
Copy link
Author

哦哦好的十分感谢～

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@v-mipeng @DSXiangLi