feat: use datasets.IterableDataset shard if possible by SunMarc · Pull Request #3635 · huggingface/accelerate

and others added 4 commits

May 22, 2025 16:59
When `accelerator.prepare` is called on a
`datasets.IterableDataset`, use the `shard` method to
split the dataset across the available processes. This
allows for more efficient data loading and processing.
Without load and slice overhead of `IterableDatasetShard`