Add keyword to allow disabling config forwarding in SSHCluster by jacobtomlinson · Pull Request #8994 · dask/distributed
Closes #8138
#8138 shows that in Windows there is a limit to environment variable size, and SSHCluster serialises your Dask config to forward to the remote processes. If your Dask config is really big then this will fail. There isn't a clear solution here, in #4465 we explored compressing config, but that isn't a perfect solution so hasn't been merged.
A reasonable workaround is to be able to just disable this functionality if you don't need it. Users in #8138 (comment) have reported just modifying distributed to disable this. This PR adds a kwarg to officially support disabling config forwarding.
- Tests added / passed
- Passes
pre-commit run --all-files