sarama broker connection can block forever if there is no response on the socket

[dnwe]

Versions

Sarama	Kafka	Go
96122a6	2.2 apache/kafka@6694636	1.13

Problem Description

The broker.go connection doesn't consistently set Read and Write deadlines before performing read/write operations on the net.Conn. In various scenarios if the broker stops responding that can cause the given connection in the client to block indefinitely

Example: sarama goroutine blocked for 20h (click to expand)

goroutine 186110 [IO wait, 1136 minutes]:
internal/poll.runtime_pollWait(0x7fc35450bc90, 0x72, 0xc4205be740)
    /usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc4208aac18, 0x72, 0xffffffffffffff00, 0x1d48c80, 0x2a7d240)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9b
internal/poll.(*pollDesc).waitRead(0xc4208aac18, 0xc4221a4400, 0x400, 0x400)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc4208aac00, 0xc4221a4400, 0x400, 0x400, 0x0, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_unix.go:157 +0x17d
net.(*netFD).Read(0xc4208aac00, 0xc4221a4400, 0x400, 0x400, 0xc4206e1e00, 0x300000002, 0xc4206e1e00)
    /usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc4200164f8, 0xc4221a4400, 0x400, 0x400, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:176 +0x6a
crypto/tls.(*block).readFromUntil(0xc420cc3bc0, 0x1d46f60, 0xc4200164f8, 0x5, 0xc4200164f8, 0x44e904)
    /usr/local/go/src/crypto/tls/conn.go:493 +0x96
crypto/tls.(*Conn).readRecord(0xc420c24000, 0x1c61a17, 0xc420c24120, 0xc4214728e0)
    /usr/local/go/src/crypto/tls/conn.go:595 +0xe0
crypto/tls.(*Conn).Read(0xc420c24000, 0xc42104d000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:1156 +0x100
bufio.(*Reader).Read(0xc42107b140, 0xc420c07888, 0x4, 0x4, 0xc420056570, 0x46de18, 0x30d088a401dda7)
    /usr/local/go/src/bufio/bufio.go:216 +0x238
github.com/Shopify/sarama.(*bufConn).Read(0xc4213226a0, 0xc420c07888, 0x4, 0x4, 0x1a2ba40, 0x7fc35444bdb8, 0x0)
    /usr/src/github.com/Shopify/sarama/utils.go:109 +0x4d
io.ReadAtLeast(0x7fc35444bdb8, 0xc4213226a0, 0xc420c07888, 0x4, 0x4, 0x4, 0x1a2ba40, 0xc420943900, 0x7fc35444bdb8)
    /usr/local/go/src/io/io.go:309 +0x86
io.ReadFull(0x7fc35444bdb8, 0xc4213226a0, 0xc420c07888, 0x4, 0x4, 0x4, 0x0, 0xc420165e60)
    /usr/local/go/src/io/io.go:327 +0x58
github.com/Shopify/sarama.(*Broker).sendAndReceiveV0SASLPlainAuth(0xc42026f180, 0x1bab5da, 0x5)
    /usr/src/github.com/Shopify/sarama/broker.go:986 +0x606
github.com/Shopify/sarama.(*Broker).sendAndReceiveSASLPlainAuth(0xc42026f180, 0x1166c20, 0xc42026f180)
    /usr/src/github.com/Shopify/sarama/broker.go:960 +0x6e
github.com/Shopify/sarama.(*Broker).authenticateViaSASL(0xc42026f180, 0xd, 0x1d7ad00)
    /usr/src/github.com/Shopify/sarama/broker.go:853 +0x97
github.com/Shopify/sarama.(*Broker).Open.func1()
    /usr/src/github.com/Shopify/sarama/broker.go:193 +0x9d3
github.com/Shopify/sarama.withRecover(0xc420ae4e20)
    /usr/src/github.com/Shopify/sarama/utils.go:45 +0x43
created by github.com/Shopify/sarama.(*Broker).Open
    /usr/src/github.com/Shopify/sarama/broker.go:151 +0x12b