Document that random_shuffle shuffles only locally - #6491

Category:

Other (e.g. Documentation, Tests, Configuration)

Description:

Closes #3088.

random_shuffle draws samples from a buffer of initial_fill samples that is filled sequentially, so it only shuffles locally. Readers of container formats (TFRecord, MXNet, Caffe, Caffe2, WebDataset) read the samples in the order in which they are stored, which means a dataset sorted by label comes out mostly sorted with the default buffer. #3088 ran into this when switching from the file reader to TFRecord, and the reply there suggested proposing wording for loader.cc.

This adds a note to the random_shuffle and initial_fill arguments, and two paragraphs to the Shuffling section added in #6484. They also explain why the file-based readers are not affected: they shuffle the whole file list once when random_shuffle is set.

To check the wording against real behavior, I read a TFRecord with 20 classes of 1000 samples each, stored class after class, using DALI 2.3.0 on CPU with a batch size of 256 (loader.h has not changed since that release). With the default initial_fill=1024, the first batch contained 2 of the 20 classes, on average 80% of a batch came from just two classes, and no sample was returned more than 1023 positions ahead of its stored position. With initial_fill equal to the dataset size, every batch contained all 20 classes.

Additional information:

Affected modules and functionalities:

Documentation only: the random_shuffle and initial_fill argument docs in the LoaderBase schema, which all readers share, and docs/advanced_topics_sharding.rst.

Key points relevant for the review:

The list of readers that read in the stored order. I went through the loaders: every list-based one (file, numpy, fits, coco, sequence, nemo_asr, video) shuffles its whole list once under if (shuffle_), while the indexed-file (TFRecord), RecordIO, LMDB and WebDataset loaders do not.

Tests:

• Existing tests apply
• New tests added
  • Python tests
  • GTests
  • Benchmark
  • Other
• N/A

Checklist

Documentation

• Existing documentation applies
• Documentation updated
  • Docstring
  • Doxygen
  • RST
  • Jupyter
  • Other
• N/A

DALI team only

Requirements

• Implements new requirements
• Affects existing requirements
• N/A

REQ IDs: N/A

JIRA TASK: N/A