Add keyword to allow disabling config forwarding in SSHCluster by jacobtomlinson · Pull Request #8994 · dask/distributed

Closes #8138

#8138 shows that in Windows there is a limit to environment variable size, and SSHCluster serialises your Dask config to forward to the remote processes. If your Dask config is really big then this will fail. There isn't a clear solution here, in #4465 we explored compressing config, but that isn't a perfect solution so hasn't been merged.

A reasonable workaround is to be able to just disable this functionality if you don't need it. Users in #8138 (comment) have reported just modifying distributed to disable this. This PR adds a kwarg to officially support disabling config forwarding.

  • Tests added / passed
  • Passes pre-commit run --all-files