Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix pickling of std(in, out, err) streams #45

Merged
merged 1 commit into from
May 28, 2014

Conversation

matsjoyce
Copy link
Contributor

Fixes one of the problems in pickling sys as encounted in #41. As you cannot file.tell() a standard stream, this pr stops dill from trying and failing with an IOError.

@@ -551,12 +551,15 @@ def save_file(pickler, obj):
if obj.closed:
position = None
else:
position = obj.tell()
if obj in (sys.__stdout__, sys.__stderr__, sys.__stdin__):
position = -1
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think that files ever have a negative tell()

@mmckerns
Copy link
Member

Not sure setting the position = -1 is the right thing to do.

For sys.stdin, it seems the position rolls forward with time, on it's own (starting from 0?). It also seems that stdin retains a queue -- as long as you don't seek back to a different position.

>>> f = sys.stdin
>>> f.read(3)
koaksdcpoaskovkas
'koa'
>>> f.read(3)
'ksd'
>>> f.read(3)
'cpo'
>>> f.tell()
1842
>>> f.tell()
1869
>>> f.seek(1800)
>>> f.tell()
1841
>>> f.read(3)
fff
'fff'

So, it's probably the right thing to do… and probably safe to do. Probably.

@matsjoyce
Copy link
Contributor Author

Which version of python is that?
I get

Python 3.4.1 (default, May 19 2014, 17:23:49) 
[GCC 4.9.0 20140507 (prerelease)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> f=sys.stdin
>>> f.read(3)
asdfghjkl
'asd'
>>> f.tell()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
io.UnsupportedOperation: underlying stream is not seekable
Python 2.7.6 (default, Feb 26 2014, 12:07:17) 
[GCC 4.8.2 20140206 (prerelease)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> f=sys.stdin
>>> f.read(3)
asdfghjkl
'asd'
>>> f.tell()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
IOError: [Errno 29] Illegal seek

@@ -340,7 +340,7 @@ def _create_filehandle(name, mode, position, closed, open=open): # buffering=0
raise UnpicklingError(err)
#XXX: python default is closed '<uninitialized file>' file/mode
if closed: f.close()
else: f.seek(position)
elif position >= 0: f.seek(position)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The position is only set if it is non negative

@mmckerns
Copy link
Member

Sorry, I clipped that off…

Python 2.7.6 (default, Nov 12 2013, 13:26:39) 
[GCC 4.2.1 Compatible Apple Clang 4.1 ((tags/Apple/clang-421.11.66))] on darwin
Type "help", "copyright", "credits" or "license" for more information.

@matsjoyce
Copy link
Contributor Author

Maybe its a mac - linux difference?

@mmckerns
Copy link
Member

Probably not. More likely a new "feature" since yours is Feb 2014 and mine is a Nov 2013 build

@mmckerns
Copy link
Member

Nope. I'm wrong, you are right. http://www.gossamer-threads.com/lists/python/python/99983
It seems to be an old issue.

@mmckerns
Copy link
Member

It seems like a fine thing to do, as if you are pickling within a single session, the stdin queue will be preserved. If you dump_session (or go off-processor), I'd assume the queue getting cleaned out might be ok.

>>> f.read(3)
asd;vkadpsvjkapjvkpqejvpqejvqerv
'asd'
>>> 
>>> _f = dill.copy(f)
>>> _f
<open file '<stdin>', mode 'r' at 0x100e7a0c0>
>>> 
>>> _f.read(3)
';vk'
>>> f.read(3)
'adp'

This is indeed what happens by not setting the position.

>>> dill.dump_session('foo.pkl')
>>> 
dude@hilbert>$ python
Python 2.7.6 (default, Nov 12 2013, 13:26:39) 
[GCC 4.2.1 Compatible Apple Clang 4.1 ((tags/Apple/clang-421.11.66))] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import dill
>>> dill.load_session('foo.pkl')
>>> f
<open file '<stdin>', mode 'r' at 0x1076900c0>
>>> f.read(3)
ddd
'ddd'

But it also seems like that would be the case for setting the position as well. Since, the queue seems to be maintained as long as you don't seek.

mmckerns added a commit that referenced this pull request May 28, 2014
Fix pickling of std(in, out, err) streams
@mmckerns mmckerns merged commit 65d88d5 into uqfoundation:master May 28, 2014
@mmckerns mmckerns added the bug label May 28, 2014
@matsjoyce matsjoyce deleted the std_stream_fix branch May 28, 2014 20:47
@mmckerns
Copy link
Member

@matsjoyce: I'm sending you an email about a new release, and some etc.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants