Unsupported framed Thrift transports results in connection resets

[teepark]

happybase 0.3 with hbase 0.94.0 is getting its connection closed consistently:

>>> import happybase
>>> conn = happybase.Connection("localhost")
>>> t = conn.table("test")
>>> list(t.scan())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "happybase/api.py", line 535, in scan
    scan_id = client.scannerOpenWithScan(self.name, scan)
  File "happybase/hbase/Hbase.py", line 1716, in scannerOpenWithScan
    return self.recv_scannerOpenWithScan()
  File "happybase/hbase/Hbase.py", line 1728, in recv_scannerOpenWithScan
    (fname, mtype, rseqid) = self._iprot.readMessageBegin()
  File "thrift/protocol/TBinaryProtocol.py", line 126, in readMessageBegin
    sz = self.readI32()
  File "thrift/protocol/TBinaryProtocol.py", line 203, in readI32
    buff = self.trans.readAll(4)
  File "thrift/transport/TTransport.py", line 58, in readAll
    chunk = self.read(sz-have)
  File "thrift/transport/TTransport.py", line 160, in read
    self.__rbuf = StringIO(self.__trans.read(max(sz, self.__rbuf_size)))
  File "thrift/transport/TSocket.py", line 94, in read
    buff = self.handle.recv(sz)
socket.error: [Errno 104] Connection reset by peer

and in the thrift server log:

2012-06-29 13:06:01,330 ERROR org.apache.thrift.server.TNonblockingServer: Read an invalid frame size of -2147418111. Are you using TFramedTransport on the client side?

[teepark]

not sure about discovering the thrift version, but in hbase/lib I have a libthrift-0.8.0.jar if that helps

[teepark]

the same with happybase git master

[wbolster]

HappyBase does not use the framed transport for the Thrift connection. Do you have any particular reason to use that transport instead if the other ones?

— Wouter

Travis Parker [email protected] schreef:

happybase 0.3 with hbase 0.94.0 is getting its connection closed
consistently:

import happybase
conn = happybase.Connection("localhost")
t = conn.table("test")
list(t.scan())
Traceback (most recent call last):
File "", line 1, in
File "happybase/api.py", line 535, in scan
scan_id = client.scannerOpenWithScan(self.name, scan)
File "happybase/hbase/Hbase.py", line 1716, in scannerOpenWithScan
return self.recv_scannerOpenWithScan()
File "happybase/hbase/Hbase.py", line 1728, in recv_scannerOpenWithScan
(fname, mtype, rseqid) = self._iprot.readMessageBegin()
File "thrift/protocol/TBinaryProtocol.py", line 126, in
readMessageBegin
sz = self.readI32()
File "thrift/protocol/TBinaryProtocol.py", line 203, in readI32
buff = self.trans.readAll(4)
File "thrift/transport/TTransport.py", line 58, in readAll
chunk = self.read(sz-have)
File "thrift/transport/TTransport.py", line 160, in read
self.__rbuf = StringIO(self.__trans.read(max(sz, self.__rbuf_size)))
File "thrift/transport/TSocket.py", line 94, in read
buff = self.handle.recv(sz)
socket.error: [Errno 104] Connection reset by peer

and in the thrift server log:

2012-06-29 13:06:01,330 ERROR
org.apache.thrift.server.TNonblockingServer: Read an invalid frame size
of -2147418111. Are you using TFramedTransport on the client side?

---

Reply to this email directly or view it on GitHub:
#6

[teepark]

Do you have any particular reason to use that transport instead if the other ones?

No, I'm running exactly the python code I posted above, not doing anything special. If happybase isn't using the framed transport then I guess the thrift server's log message is mis-diagnosing the issue.

[wbolster]

Maybe the default server side transport type has changed in 0.94. I guess it's a flag to the start-daemon thrift script.

— Wouter

Travis Parker [email protected] schreef:

Do you have any particular reason to use that transport instead if
the other ones?

No, I'm running exactly the python code I posted above, not doing
anything special. If happybase isn't using the framed transport then I
guess the thrift server's log message is mis-diagnosing the issue.

---

Reply to this email directly or view it on GitHub:
#6 (comment)

[teepark]

starting the thrift server in the foreground with bin/hbase thrift start -c -threadpool I see this error on the console running the server:

ERROR thrift.TBoundedThreadPoolServer: Thrift error occurred during processing of message.
org.apache.thrift.protocol.TProtocolException: Expected protocol id ffffff82 but got ffffff80
    at org.apache.thrift.protocol.TCompactProtocol.readMessageBegin(TCompactProtocol.java:445)
    at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:22)
    at org.apache.hadoop.hbase.thrift.TBoundedThreadPoolServer$ClientConnnection.run(TBoundedThreadPoolServer.java:287)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:722)

and get this in the python client:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "happybase/api.py", line 546, in scan
  File "happybase/hbase/Hbase.py", line 1716, in scannerOpenWithScan
  File "happybase/hbase/Hbase.py", line 1728, in recv_scannerOpenWithScan
  File "thrift/protocol/TBinaryProtocol.py", line 126, in readMessageBegin
    sz = self.readI32()
  File "thrift/protocol/TBinaryProtocol.py", line 203, in readI32
    buff = self.trans.readAll(4)
  File "thrift/transport/TTransport.py", line 58, in readAll
    chunk = self.read(sz-have)
  File "thrift/transport/TTransport.py", line 160, in read
    self.__rbuf = StringIO(self.__trans.read(max(sz, self.__rbuf_size)))
  File "thrift/transport/TSocket.py", line 108, in read
    raise TTransportException(type=TTransportException.END_OF_FILE, message='TSocket read 0 bytes')
thrift.transport.TTransport.TTransportException: TSocket read 0 bytes

for -hsha, -nonblocking, and -threadedselector the help output says "this implies the framed transport" and indeed using any of those produces the error in the OP.

[teepark]

ah HAH. the -c was wrong (I had thought that was a negation of -f for framed transport).

the thrift server must be started with bin/hbase thrift start -threadpool. The help output says of -threadpool that "this is the default", but it requires that one of [-hsha, -nonblocking, -threadpool, -threadedselector] be explicitly used.

starting it with bin/hbase-daemon.sh start thrift was picking one of the framed transport modes.

this is worth documenting imho

[wbolster]

Ah, this should definitely end up in the docs. Maybe HappyBase can work with other Thrift transports as well. Reopening so that I don't forget about this.

[wbolster]

Okay, this issues seems fixed now. I've also updated the tutorial and the API docs. Could you please have a look at http://happybase.readthedocs.org/en/latest/tutorial.html and tell me what you think? Thanks!

[teepark]

looks great, thanks!

[bilkulbekar]

I still see these errors on using latest code.

[wbolster]

@bilkulbekar Please provide more details. This usually only happens when the Thrift server speaks a different protocol than the Thrift client (HappyBase).