I think this is a really really bad bug, unless I'm totally misunderstanding something here.
What happened:
Inside the futures function, the value of a dict parameter is sometimes replaced. (I'm guessing when it collides with a value of a scattered dataset.)
Output:
{'anything': 'a', 'anything2': {'a': <Future: finished, type: builtins.str, key: a>, 'b': <Future: finished, type: builtins.str, key: b>}}
{'anything': 'a_df', 'anything2': {'a': 'a_df', 'b': 'b_df'}}
{'anything': 'b', 'anything2': {'a': <Future: finished, type: builtins.str, key: a>, 'b': <Future: finished, type: builtins.str, key: b>}}
{'anything': 'b_df', 'anything2': {'a': 'a_df', 'b': 'b_df'}}
{'anything': 'c', 'anything2': {'a': <Future: finished, type: builtins.str, key: a>, 'b': <Future: finished, type: builtins.str, key: b>}}
{'anything': 'c', 'anything2': {'a': 'a_df', 'b': 'b_df'}}
What you expected to happen:
{'anything': 'a', 'anything2': {'a': <Future: finished, type: builtins.str, key: a>, 'b': <Future: finished, type: builtins.str, key: b>}}
{'anything': 'a', 'anything2': {'a': 'a_df', 'b': 'b_df'}}
{'anything': 'b', 'anything2': {'a': <Future: finished, type: builtins.str, key: a>, 'b': <Future: finished, type: builtins.str, key: b>}}
{'anything': 'b', 'anything2': {'a': 'a_df', 'b': 'b_df'}}
{'anything': 'c', 'anything2': {'a': <Future: finished, type: builtins.str, key: a>, 'b': <Future: finished, type: builtins.str, key: b>}}
{'anything': 'c', 'anything2': {'a': 'a_df', 'b': 'b_df'}}
Minimal Complete Verifiable Example:
import dask
from dask.distributed import Client, LocalCluster, Future, as_completed
cluster = dask.distributed.LocalCluster()
dask = Client(cluster)
remote_data = dask.scatter(dict(a="a_df", b="b_df"))
def remote_fn(data_fn):
return data_fn
for any_value in ["a", "b", "c"]:
data = dict(
anything=any_value,
anything2=remote_data,
)
print(data)
print(dask.submit(remote_fn, data).result())Anything else we need to know?:
Environment:
- Dask version: 2.19.0
- Python version: 3.6.4
- Operating System: Mac OS X (Mojave 10.14.6)
- Install method (conda, pip, source): pip
I think this is a really really bad bug, unless I'm totally misunderstanding something here.
What happened:
Inside the futures function, the value of a dict parameter is sometimes replaced. (I'm guessing when it collides with a value of a scattered dataset.)
Output:
What you expected to happen:
Minimal Complete Verifiable Example:
Anything else we need to know?:
Environment: