Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

decompressing Snappy encoded RecordBatch fails #79

Open
pdeva opened this issue Sep 14, 2024 · 2 comments
Open

decompressing Snappy encoded RecordBatch fails #79

pdeva opened this issue Sep 14, 2024 · 2 comments

Comments

@pdeva
Copy link
Contributor

pdeva commented Sep 14, 2024

getting error:

 snappy: corrupt input (expected valid offset but got offset 20545; dst position: 0)

changing to another compression like lz4 works fine.

@pdeva pdeva changed the title decompressing snappy encoded RecordBatch fails decompressing Snappy encoded RecordBatch fails Sep 14, 2024
@pdeva
Copy link
Contributor Author

pdeva commented Sep 14, 2024

seems the bug occurs when the batch is compressed with snappy 'framing' turned on.

this is the code from kafka-go client.
when x.framed is set to true, it will cause the decoder in kafka-protocol-rs to crash.
seems its set to true by default for all popular kafka clients including the cannonical java client.

func (c *Codec) NewWriter(w io.Writer) io.WriteCloser {
	x, _ := writerPool.Get().(*xerialWriter)
	if x != nil {
		x.Reset(w)
	} else {
		x = &xerialWriter{writer: w}
	}
	x.framed = c.Framing == Framed
	switch c.Compression {
	case FasterCompression:
		x.encode = s2.EncodeSnappy
	case BetterCompression:
		x.encode = s2.EncodeSnappyBetter
	case BestCompression:
		x.encode = s2.EncodeSnappyBest
	default:
		x.encode = snappy.Encode // aka. s2.EncodeSnappyBetter
	}
	return &writer{xerialWriter: x}
}

@tychedelia
Copy link
Owner

For compression options, we want to match the official Java client wherever possible, so this would certainly be a bug on our end.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants