Revision 1880
Added by Aaron Marcuse-Kubitza over 12 years ago
| parallel.py | ||
|---|---|---|
| 25 | 25 |
|
| 26 | 26 |
def prepickle(value, vars_id_dict_): |
| 27 | 27 |
def filter_(value): |
| 28 |
id_ = id(value) |
|
| 29 |
if id_ in vars_id_dict_: value = id_ |
|
| 28 | 30 |
# Try pickling the value. If it fails, we'll get a full traceback here, |
| 29 | 31 |
# which is not provided with pickling errors in multiprocessing's Pool. |
| 30 |
try: try_pickle(value) |
|
| 31 |
except Exception, e: |
|
| 32 |
id_ = id(value) |
|
| 33 |
if id_ in vars_id_dict_: value = id_ |
|
| 34 |
else: raise e |
|
| 32 |
else: try_pickle(value) |
|
| 35 | 33 |
return value |
| 36 | 34 |
return collection.rmap(filter_, value) |
| 37 | 35 |
|
Also available in: Unified diff
parallel.py: prepickle(): Pickle all objects in vars_id_dict_ by ID, not just unpicklable ones. This ensures that a DB connection created in the main process will be shared with subprocesses by reference (id()) instead of by value, so that each process can take advantage of e.g. shared caches in the connection object. Note that this may require some synchronization.