Compress serialized config data with zlib by jacobtomlinson · Pull Request #4465 · dask/distributed
Coming from here (thanks @quasiben), I can confirm that this PR solved my issue, at least on this side of things there is no more error. However my machine is stuck at Waiting for scheduler to run at ip:port, while on aws it is "shutting down" / terminated, I believe I also have to pull this change into my packer build, but am somewhat at a loss on how to do this.
edit: So I did a bad thing; I launched the AMI created by packer, edited all files /distributed/utils.py in the conda env's as in the merge changes, and created a new AMI from this image. I still get the same behavior as above unfortunately.