Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

syncer(dm): support GBK replication #3990

Merged
merged 17 commits into from
Dec 29, 2021
Merged

syncer(dm): support GBK replication #3990

merged 17 commits into from
Dec 29, 2021

Conversation

lance6716
Copy link
Contributor

@lance6716 lance6716 commented Dec 20, 2021 •
edited
Loading

What problem does this PR solve?

close #3774

What is changed and how it works?

  • use _binary'x' to insert GBK data from binlog, and add simple test.
  • For DDL, decode the query when it's GBK

Also move some function logic together

Check List

Tests

  • Integration test

Code changes

Side effects

  • Breaking backward compatibility

Related changes

Release note 

Support replicate GBK tables as-is

@ti-chi-bot
Copy link
Member

ti-chi-bot commented Dec 20, 2021 •
edited
Loading

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • Ehco1996
  • lichunzhu

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Dec 20, 2021
@ti-chi-bot ti-chi-bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Dec 20, 2021
@lance6716 lance6716 changed the title [WIP] parser(dm): support GBK replication [WIP] syncer(dm): support GBK replication Dec 20, 2021
@lance6716

This comment has been minimized.

Sign in to view
@lance6716 lance6716 changed the title [WIP] syncer(dm): support GBK replication syncer(dm): support GBK replication Dec 20, 2021
@ti-chi-bot ti-chi-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 20, 2021
@lance6716 lance6716 changed the title syncer(dm): support GBK replication [WIP] syncer(dm): support GBK replication Dec 20, 2021
@ti-chi-bot ti-chi-bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 20, 2021
@lance6716 lance6716 added the area/dm Issues or PRs related to DM. label Dec 20, 2021
@lance6716
Copy link
Contributor Author

/run-all-tests

@lance6716
Copy link
Contributor Author

/run-dm-integration-test

2 similar comments
@lance6716
Copy link
Contributor Author

/run-dm-integration-test

@lance6716
Copy link
Contributor Author

/run-dm-integration-test

@lance6716 lance6716 changed the title [WIP] syncer(dm): support GBK replication syncer(dm): support basic GBK replication Dec 23, 2021
@ti-chi-bot ti-chi-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 23, 2021
@lance6716
Copy link
Contributor Author

/run-dm-integration-test

1 similar comment
@lance6716
Copy link
Contributor Author

/run-dm-integration-test

@lance6716
Copy link
Contributor Author

/run-dm-integration-test

@lance6716 lance6716 changed the title syncer(dm): support basic GBK replication syncer(dm): support GBK replication Dec 24, 2021
@lance6716
Copy link
Contributor Author

/run-dm-integration-test

2 similar comments
@lance6716
Copy link
Contributor Author

/run-dm-integration-test

@lance6716
Copy link
Contributor Author

/run-dm-integration-test

@lance6716 lance6716 added this to the v5.4.0 milestone Dec 28, 2021
lichunzhu
lichunzhu approved these changes Dec 28, 2021
View reviewed changes
Copy link
Contributor

@lichunzhu lichunzhu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rest LGTM

@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Dec 28, 2021
Ehco1996
Ehco1996 reviewed Dec 29, 2021
View reviewed changes
Copy link
Contributor

@Ehco1996 Ehco1996 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rest LGTM

dm/pkg/binlog/event/util.go
@@ -376,6 +379,35 @@ func GetServerCollationByStatusVars(statusVars []byte, idAndCollationMap map[int
return idAndCollationMap[int(v)], err
}

// GetCharsetCodecByStatusVars returns an encoding.Encoding to encode and decode original query if needed.
func GetCharsetCodecByStatusVars(statusVars []byte) (encoding.Encoding, error) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this func called GetCharsetCodecByStatusVars but only return chartset code when chartset is gbk , this looks litte strange, maybe change a name?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the comment says "returns ... if needed", we can return more codec if user wants

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok i see

@codecov-commenter
Copy link

codecov-commenter commented Dec 29, 2021 •
edited
Loading

Codecov Report

Merging #3990 (7bb62af) into master (3873d39) will decrease coverage by 1.6848%.
The diff coverage is 68.3383%.

Flag Coverage Δ
cdc 59.0046% <68.3383%> (+0.7680%) ⬆️
dm 52.2936% <ø> (-3.7410%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

@@               Coverage Diff                @@
##             master      #3990        +/-   ##
================================================
- Coverage   57.0741%   55.3893%   -1.6849%     
================================================
  Files           478        489        +11     
  Lines         56551      60416      +3865     
================================================
+ Hits          32276      33464      +1188     
- Misses        20978      23572      +2594     
- Partials       3297       3380        +83

@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Dec 29, 2021
@lance6716
Copy link
Contributor Author

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: fc8f585

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Dec 29, 2021
@lance6716
Copy link
Contributor Author

/run-verify
/run-leak-test

@lance6716
Copy link
Contributor Author

/run-verify

@ti-chi-bot ti-chi-bot merged commit 5f40465 into pingcap:master Dec 29, 2021
@lance6716 lance6716 deleted the gbk branch October 13, 2022 08:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lichunzhu lichunzhu lichunzhu approved these changes

@Ehco1996 Ehco1996 Ehco1996 approved these changes

@WizardXiao WizardXiao Awaiting requested review from WizardXiao

@glorv glorv Awaiting requested review from glorv

@GMHDBJD GMHDBJD Awaiting requested review from GMHDBJD

Assignees
No one assigned
Labels
area/dm Issues or PRs related to DM. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2.
Projects
None yet
Milestone
v5.4.0
Development

Successfully merging this pull request may close these issues.

Support migrating GBK charset upstream as-is
5 participants
@lance6716 @ti-chi-bot @codecov-commenter @lichunzhu @Ehco1996