Saving a modules dict attribute #41

matsjoyce · 2014-05-27T11:30:56Z

Possibly a follow up for #35, but saving a session doesn't save the state of the modules:

Python 3.4.1 (default, May 19 2014, 17:23:49) 
[GCC 4.9.0 20140507 (prerelease)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import dill
>>> import test
>>> test.a = 2
>>> dill.dump_session()
>>> 
==========================Restart==========================
Python 3.4.1 (default, May 19 2014, 17:23:49) 
[GCC 4.9.0 20140507 (prerelease)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import dill
>>> dill.load_session()
>>> test
<module 'test' from 'test.py'>
>>> test.a
1
>>>

test.py:

def f():
    print(x)
a = 1

I think this code does the job, but it may need a few adjustments, depending which modules need to be pickled.

mmckerns · 2014-05-27T13:45:09Z

Ok, I'm taking a look at your code. It would be nice if your test failed with an AssertionError instead of an AttributeError.

dude@hilbert>$ python test_module.py 
Traceback (most recent call last):
  File "test_module.py", line 23, in <module>
    assert module.a == 1234
AttributeError: 'module' object has no attribute 'a'

matsjoyce · 2014-05-27T13:54:51Z

Oops...
I'll change it from

assert module.a == 1234

to

assert hasattr(module, "a") and module.a == 1234

I'll wait to see if there is anything else, though.

mmckerns · 2014-05-27T13:57:27Z

should probably clean up the test_mixins.pyc file after the test is run.

I'm poking around on the different python versions and different tests now.

matsjoyce · 2014-05-27T14:06:35Z

It would be nice if your test failed with an AssertionError instead of an AttributeError.
should probably clean up the test_mixins.pyc file after the test is run.

Done

mmckerns · 2014-05-27T14:24:24Z

Python 3.3.5 (default, Mar 10 2014, 21:37:38) 
[GCC 4.2.1 Compatible Apple Clang 4.1 ((tags/Apple/clang-421.11.66))] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import dill
>>> import test_mixins
>>> test_mixins.a = 2
>>> dill.dump_session()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/mmckerns/lib/python3.3/site-packages/dill-0.2.1.dev-py3.3.egg/dill/dill.py", line 182, in dump_session
    pickler.dump(main_module)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 235, in dump
    self.save(obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 297, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/mmckerns/lib/python3.3/site-packages/dill-0.2.1.dev-py3.3.egg/dill/dill.py", line 772, in save_module
    state=_main_dict)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 430, in save_reduce
    save(state)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 297, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/mmckerns/lib/python3.3/site-packages/dill-0.2.1.dev-py3.3.egg/dill/dill.py", line 504, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 633, in save_dict
    self._batch_setitems(obj.items())
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 666, in _batch_setitems
    save(v)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 297, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/mmckerns/lib/python3.3/site-packages/dill-0.2.1.dev-py3.3.egg/dill/dill.py", line 772, in save_module
    state=_main_dict)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 430, in save_reduce
    save(state)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 297, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/mmckerns/lib/python3.3/site-packages/dill-0.2.1.dev-py3.3.egg/dill/dill.py", line 504, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 633, in save_dict
    self._batch_setitems(obj.items())
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 666, in _batch_setitems
    save(v)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 297, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/mmckerns/lib/python3.3/site-packages/dill-0.2.1.dev-py3.3.egg/dill/dill.py", line 467, in save_function
    obj.__dict__), obj=obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 412, in save_reduce
    save(args)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 297, in save
    f(self, obj) # Call unbound method with explicit self
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 553, in save_tuple
    save(element)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 297, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/mmckerns/lib/python3.3/site-packages/dill-0.2.1.dev-py3.3.egg/dill/dill.py", line 456, in save_code
    pickler.save_reduce(_unmarshal, (marshal.dumps(obj),), obj=obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 412, in save_reduce
    save(args)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 297, in save
    f(self, obj) # Call unbound method with explicit self
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 539, in save_tuple
    save(element)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 297, in save
    f(self, obj) # Call unbound method with explicit self
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 500, in save_bytes
    (str(obj, 'latin1'), 'latin1'), obj=obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 411, in save_reduce
    save(func)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 297, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/mmckerns/lib/python3.3/site-packages/dill-0.2.1.dev-py3.3.egg/dill/dill.py", line 618, in save_builtin_method
    pickler.save_reduce(_get_attr, (obj.__self__, obj.__name__), obj=obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 416, in save_reduce
    self.memoize(obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/pickle.py", line 255, in memoize
    assert id(obj) not in self.memo
AssertionError

Possibly an issue of getting rid of if is_dill(pickler)?

matsjoyce · 2014-05-27T14:28:55Z

😩 Sorry, mistake in condition.

matsjoyce · 2014-05-27T14:33:17Z

Do you want me to squash the commits?

mmckerns · 2014-05-27T16:37:41Z

It still doesn't seem to work for me with dump_session.

Python 3.3.5 (default, Mar 10 2014, 21:37:38) 
[GCC 4.2.1 Compatible Apple Clang 4.1 ((tags/Apple/clang-421.11.66))] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import dill
>>> import numpy as np
>>> np.sin = np.cos
>>> import test_mixins as module
>>> module.a = 2
>>> dill.dump_session('foo.pik')
>>> ^D
dude@hilbert>$ python
Python 3.3.5 (default, Mar 10 2014, 21:37:38) 
[GCC 4.2.1 Compatible Apple Clang 4.1 ((tags/Apple/clang-421.11.66))] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import dill
>>> dill.load_session('foo.pik')
>>> np
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'np' is not defined
>>> module
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'module' is not defined

matsjoyce · 2014-05-27T16:46:53Z

😩 Still the condition. I've combined it with the old condition, so the __main__ module ought to pickle correctly.

matsjoyce · 2014-05-27T17:24:34Z

should probably clean up the test_mixins.pyc file after the test is run.

tests/test_nested.py doesn't clean up when run by python 3. Shall I file a separate issue?

mmckerns · 2014-05-27T17:55:27Z

Looks like matsjoyce@7bcf040 works both in a file and with dump_session. It looks to work nicely in most cases I can think of.

I could accept it, as is… because it adds a feature and doesn't remove any… however, I'd like to ask your opinion on proceeding. Should the following be the correct behavior? It currently is, b/c "standard" modules are stored by reference… but it is definitely possible to add something to a standard module's dict.

>>> import dill
>>> import numpy as np
>>> np.min = np.max
>>> dill.dump_session()
>>> ^D
############ restart ############
>>> import dill
>>> dill.load_session()
>>> np.min([1,2,3,4,5])
1

mmckerns · 2014-05-27T17:57:24Z

Note that this is also the current behavior, so maybe it makes sense? I'm not sure.

Python 2.7.6 (default, Nov 12 2013, 13:26:39) 
[GCC 4.2.1 Compatible Apple Clang 4.1 ((tags/Apple/clang-421.11.66))] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import dill        
>>> import abc
>>> abc.ABCMeta.zzz = 1
>>> p = dill.copy(abc.ABCMeta)
>>> p
<class 'abc.ABCMeta'>
>>> p.zzz
1
>>> dill.dump_session('foo.pik')
>>> ^D
dude@hilbert>$ python
Python 2.7.6 (default, Nov 12 2013, 13:26:39) 
[GCC 4.2.1 Compatible Apple Clang 4.1 ((tags/Apple/clang-421.11.66))] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import dill
>>> dill.load_session('foo.pik')
>>> p
<class 'abc.ABCMeta'>
>>> p.zzz
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: type object 'ABCMeta' has no attribute 'zzz'

matsjoyce · 2014-05-27T18:17:48Z

The reason for excluding standard modules is that most of them don't pickle (sys, re, etc). If a list of picklable or non-picklable modules could be created, that behaviour could be changed to be more logical. Could a list be made though dill._objects?

from dill import _objects
{getattr(i.__class__, '__module__', "")  for i in _objects.x.values()}

matsjoyce · 2014-05-27T19:35:27Z

Using a function like:

def _can_pickle_module(mod, _cache={}, _recursion_protection=[None]):
    if _recursion_protection[0] is not None:
        return mod is _recursion_protection[0]
    if mod not in _cache:
        _recursion_protection[0] = mod
        try:
            dumps(mod)
        except:
            _cache[mod] = False
        else:
            _cache[mod] = True
        finally:
            _recursion_protection[0] = None
    return _cache[mod]

and the condition in save_module being:

if _can_pickle_module(obj) or is_dill(pickler) and obj is pickler._main_module:

it'll work. However, this approach is less efficient.

matsjoyce · 2014-05-27T20:51:28Z

The issue with abc.ABCMeta is that is it dumped like:

>>> dill.dumps(p)
T4: <class 'abc.ABCMeta'>
b'\x80\x03cabc\nABCMeta\nq\x00.'

The type is handed to pickle, which does not preserve the __dict__ attribute. An edit to dill.dill.save_type would be needed to change that.

mmckerns · 2014-05-27T21:02:00Z

Right. What I'm saying, is what do you think the expected behavior should be?

My solution to abc.ABCMeta, is exactly the same as yours… but do you think it's the proper thing to do? I'm tending to think it might be.

My solution to an edited __dict__ on a builtin or "standard module" would be to pickle the object by reference like it is now, but also pickle it's existing __dict__… then reconstitute in unpicking. Make a _save_module that serializes the module as is done now, and also serializes the __dict__… and in unpicking, deserializes the module, then deserializes the associated __dict__ and applies it to the module before returning it to the user. Then it doesn't matter which modules serialize and which don't. The issue is serializing the associated __dict__.

matsjoyce · 2014-05-28T08:08:49Z

My solution to abc.ABCMeta, is exactly the same as yours… but do you think it's the proper thing to do? I'm tending to think it might be.

Yes, as if dill.dump_session is supposed to restore the session exactly, the more that is saved, the better.

My solution to an edited __dict__ on a builtin or "standard module" would be to pickle the object by reference like it is now, but also pickle it's existing __dict__… then reconstitute in unpicking. Make a _save_module that serializes the module as is done now, and also serializes the __dict__… and in unpicking, deserializes the module, then deserializes the associated __dict__ and applies it to the module before returning it to the user. Then it doesn't matter which modules serialize and which don't. The issue is serializing the associated __dict__.

All the builtin modules (why do I keep calling them standard...) serialize by reference, but the __dicts__ don't:

>>> import dill, sys
>>> dill.dumps(sys.__dict__)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/matthew/GitHub/dill/dill/dill.py", line 143, in dumps
    dump(obj, file, protocol, byref)
  File "/home/matthew/GitHub/dill/dill/dill.py", line 136, in dump
    pik.dump(obj)
  File "/usr/lib/python3.4/pickle.py", line 410, in dump
    self.save(obj)
  File "/usr/lib/python3.4/pickle.py", line 477, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/matthew/GitHub/dill/dill/dill.py", line 504, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/usr/lib/python3.4/pickle.py", line 812, in save_dict
    self._batch_setitems(obj.items())
  File "/usr/lib/python3.4/pickle.py", line 838, in _batch_setitems
    save(v)
  File "/usr/lib/python3.4/pickle.py", line 522, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python3.4/pickle.py", line 600, in save_reduce
    save(func)
  File "/usr/lib/python3.4/pickle.py", line 477, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/matthew/GitHub/dill/dill/dill.py", line 827, in save_type
    StockPickler.save_global(pickler, obj)
  File "/usr/lib/python3.4/pickle.py", line 918, in save_global
    (obj, module_name, name))
_pickle.PicklingError: Can't pickle <class 'sys.flags'>: it's not the same object as sys.flags

If we have a way to test whether a module's __dict__ can be serialised, then it'll all be fine. I'm not sure if there is a cleverer way of checking that than _can_pickle_module above.

Saving a modules __dict__ attribute

mmckerns · 2014-05-28T11:03:07Z

This issues should stay open, or be a new issue.

mmckerns · 2014-05-28T11:06:36Z

And, with regard to your test above… you have to be extra careful with a __dict__… in many cases, it might actually be a weakref… so it will break unless you grab it the right way. I haven't investigated what kind of beast it is, however.

matsjoyce · 2014-05-28T11:08:39Z

Shall I file new issues for:

type's __dict__s not being pickled
tests/test_nested.py doesn't clean up when run by python 3
builtin modules __dict__s are not pickled

mmckerns · 2014-05-28T11:09:08Z

I was just typing the same. Sure.

mmckerns · 2014-05-28T11:09:39Z

ref this issue

Allows saving of a modules __dict__ attribute

8e241dd

Add code to clean up

5e57dce

Fixes error with dealing with modules without a file

5b459b0

Check for __main__ module when saving

7bcf040

mmckerns added a commit that referenced this pull request May 28, 2014

Merge pull request #41 from matsjoyce/module_dict

86e0bb8

Saving a modules __dict__ attribute

mmckerns merged commit 86e0bb8 into uqfoundation:master May 28, 2014

matsjoyce deleted the module_dict branch May 28, 2014 11:04

mmckerns added enhancement and removed enhancement labels May 28, 2014

This was referenced May 28, 2014

Type's __dict__s are not pickled #42

Open

Builtin modules do not have the __dict__ pickled #43

Open

Clean up after test_nested.py is run with python 3 #44

Merged

Fix pickling of std(in, out, err) streams #45

Merged

matsjoyce mentioned this pull request Jun 4, 2014

[WIP] Allow module level type's __dicts__ to be saved #47

Open

matsjoyce mentioned this pull request Jul 13, 2014

dumping functions does not store dependencies #52

Closed

matsjoyce mentioned this pull request Aug 20, 2014

Implement changes in #57: improve behaviour of dumped files #59

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Saving a modules dict attribute #41

Saving a modules dict attribute #41

matsjoyce commented May 27, 2014

mmckerns commented May 27, 2014

matsjoyce commented May 27, 2014

mmckerns commented May 27, 2014

matsjoyce commented May 27, 2014

mmckerns commented May 27, 2014

matsjoyce commented May 27, 2014

matsjoyce commented May 27, 2014

mmckerns commented May 27, 2014

matsjoyce commented May 27, 2014

matsjoyce commented May 27, 2014

mmckerns commented May 27, 2014

mmckerns commented May 27, 2014

matsjoyce commented May 27, 2014

matsjoyce commented May 27, 2014

matsjoyce commented May 27, 2014

mmckerns commented May 27, 2014

matsjoyce commented May 28, 2014

mmckerns commented May 28, 2014

mmckerns commented May 28, 2014

matsjoyce commented May 28, 2014

mmckerns commented May 28, 2014

mmckerns commented May 28, 2014

Saving a modules __dict__ attribute #41

Saving a modules __dict__ attribute #41

Conversation

matsjoyce commented May 27, 2014

mmckerns commented May 27, 2014

matsjoyce commented May 27, 2014

mmckerns commented May 27, 2014

matsjoyce commented May 27, 2014

mmckerns commented May 27, 2014

matsjoyce commented May 27, 2014

matsjoyce commented May 27, 2014

mmckerns commented May 27, 2014

matsjoyce commented May 27, 2014

matsjoyce commented May 27, 2014

mmckerns commented May 27, 2014

mmckerns commented May 27, 2014

matsjoyce commented May 27, 2014

matsjoyce commented May 27, 2014

matsjoyce commented May 27, 2014

mmckerns commented May 27, 2014

matsjoyce commented May 28, 2014

mmckerns commented May 28, 2014

mmckerns commented May 28, 2014

matsjoyce commented May 28, 2014

mmckerns commented May 28, 2014

mmckerns commented May 28, 2014

Saving a modules dict attribute #41

Saving a modules dict attribute #41