Archipelagos and the mp_island namespace

[mishpat]

Hi,
I'm attempting to parallelize some maximization problems I'm working on with pygmo, and I'm running into a bit of an issue. I'm on Win64, Python 3.6.1, Pygmo 2.2. The simplest example I can get to fail that replicated my problem is below.

import pygmo as pg
import numpy as np


class toy_problem:

     def fitness(self, x):
         return [np.sum(np.sin((x-.2)**2))]

     def get_bounds(self):
         return (np.array([-1] * 3), np.array([1] * 3))


if __name__=='__main__':
    algo = pg.algorithm(pg.de(gen=1000, seed=126))
    prob = pg.problem(toy_problem())
    pop = pg.population(prob=prob, size=10)
    print(pop.champion_f)
    pop = algo.evolve(pop)
    print(pop.champion_f)
    # fine up to this point

    archi = pg.archipelago(n=6, algo=algo, prob=prob, pop_size=70)
    archi.evolve()
    archi.wait_check()

This fails at wait_check() with the message

Traceback (most recent call last):
  File "C:\Python36\lib\site-packages\pygmo\_py_islands.py", line 128, in run_evolve
    return res.get()
  File "C:\Python36\lib\multiprocessing\pool.py", line 608, in get
    raise self._value
NameError: name 'np' is not defined

Apparently when mp_island creates a pool object, the imports from __main__ never get imported by the children, not entirely sure why. I can fix this by either 1. not using numpy, or 2. changing fitness to

 def fitness(self, x):
    import numpy as np
    return [np.sum(np.sin((x-.2)**2))]

but obviously this is not a long-term solution, and this problem prevents me from even calling other functions in __main__ as well as importing any outside modules. Any help is appreciated, thanks.

[bluescarni]

This is due to the behaviour of the serialization library we are using (dill). There's a few possible fixes:

1. do the imports within the methods (as you mentioned already)

2. do the imports as part of the class definition, and then employ this slightly kludgy way of using them:

class toy_problem:
     import numpy as _np

     def fitness(self, x):
         return [type(self)._np.sum(type(self)._np.sin((x-.2)**2))]

     def get_bounds(self):
         return (type(self)._np.array([-1] * 3), type(self)._np.array([1] * 3))

3. Set the "recurse" option for dill:

import dill
dill.settings['recurse'] = True

import numpy as np

class toy_problem:
    ...

[mishpat]

Oh thanks, I had been thinking it was an issue with multiprocessing itself. Is this a significant enough issue that I could try to work on a quick documentation update, perhaps under Python Tutorials/Use of the class archipelago?

[bluescarni]

It would be good to have a section in the documentation illustrating these types of pitfalls, so feel free to go ahead if you like :)

For context, we decided to go with an external serialization library (dill) because, contrary to the standard pickle module, it allows also to serialize classes/functions/etc. defined interactively (note that pagmo also supports standard pickling, it just uses dill instead of pickle when it needs to distribute data to external processes). A library similar to dill is cloudpickle, and we could consider eventually adding support for more serialization protocols if the need arises. For the moment, though, we need more experience before taking a decision like this.

[mishpat]

I'm back with a similar issue already!

import pygmo as pg
import numpy as np
import dill

dill.settings['recurse'] = True

class toy_problem:

    def fitness(self, x):
     return [np.sin(x[0] + x[1] - x[2])]

    def gradient(self, x):
     return pg.estimate_gradient(lambda x: self.fitness(x), x)

    def get_bounds(self):
     return (np.array([-1] * 3), np.array([1] * 3))


if __name__=='__main__':
    mma = pg.algorithm(pg.nlopt("mma"))
    p_toy = pg.problem(toy_problem())
    archi = pg.archipelago(n=6, algo=mma, prob=p_toy, pop_size=1)
    archi.evolve()
    archi.wait_check()

dies with

KeyError: '__builtins__

in worker->_evolve_func->estimate_gradient
It's a nlopt problem with estimate_gradient, and the gradient-less nlopt algorithms run fine. Presumably also a serialization problem?
Thanks for the help, I promise to update the docs when I get everything fully working :)

[darioizzo]

Issue confirmed on my system.

[darioizzo]

here is the full stack

RuntimeError                              Traceback (most recent call last)
<ipython-input-1-38e735f44774> in <module>()
     22     archi = pg.archipelago(n=6, algo=mma, prob=p_toy, pop_size=1)
     23     archi.evolve()
---> 24     archi.wait_check()
     25 

RuntimeError: The asynchronous evolution of a Pythonic island of type 'Multiprocessing island' raised an error:
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/home/dario/.local/lib/python3.6/site-packages/pygmo/_py_islands.py", line 40, in _evolve_func
    return algo.evolve(pop)
  File "<ipython-input-1-38e735f44774>", line 13, in gradient
KeyError: '__builtins__'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/dario/.local/lib/python3.6/site-packages/pygmo/_py_islands.py", line 128, in run_evolve
    return res.get()
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 608, in get
    raise self._value
KeyError: '__builtins__'

[darioizzo]

The error is somehow linked to the use of:

    def gradient(self, x):
     return pg.estimate_gradient(lambda x: self.fitness(x), x)

as it disappears trying for example pygmo.de() instead of the nlopt

[bluescarni]

It is connected to the use of the lambda function in the estimate_gradient call. For some reason, the recurse option of dill causes this issue. If you don't use recurse, and do the imports within the methods, it works as expected. Testing things out right now to see if there are better workarounds.

[bluescarni]

@mishpat We are testing out a cloudpickle as an alternative to dill in #108. My local tests indicate that it handles both the examples in this report without any extra setup. If it works fine in the CI, we'll merge the PR and do a new pagmo release shortly afterwards.

[bluescarni]

@mishpat we have released version 2.3 which includes the change to cloudpickle. Packages should be up in a few hours. Let us know if it works for you.

[mishpat]

Everything seems to work great now, thanks!