MXNet Python Data Loading API

This topic introduces the data input method for MXNet. MXNet uses an iterator to provide data to the neural network. Iterators do some preprocessing and generate batches for the neural network.

MXNet provides basic iterators for MNIST and Recordio images. To hide the cost of I/O, MXNet uses a prefetch strategy that enables parallelism for the learning process and data fetching. Data is automatically fetched by an independent thread.

Topics:

Data Iterator Parameters

To create a data iterator, you typically need to provide five parameters:

  • Dataset Param provides basic information about the dataset, e.g., file path, input shape.
  • Batch Param provides information required to form a batch, e.g., batch size.
  • Augmentation Param tells MXNet which augmentation operations (e.g., crop or mirror) to perform on an input image.
  • Backend Param controls the behavior of the back-end threads to hide the cost of data loading.
  • Auxiliary Param provides options for checking and debugging.

You must provide the Dataset Param and Batch Param, otherwise MXNet can’t create the data batch. Provide other parameters as required by your algorithm and performance needs. We provide a detailed explanation and examples of the options later.

Create a Data Iterator

The IO API provides a simple way to create a data iterator in Python. The following example code shows how to create a CIFAR data iterator.

    >>>dataiter = mx.io.ImageRecordIter(
    >>>        # Utility Parameter
    >>>        # Optional
    >>>        # Name of the data, should match the name of the data input of the network
    >>>        # data_name='data',
    >>>        # Utility Parameter
    >>>        # Optional
    >>>        # Name of the label, should match the name of the label parameter of the network.
    >>>        # Usually, if the loss layer is named 'foo', then the label input has the name
    >>>        # 'foo_label', unless overwritten
    >>>        # label_name='softmax_label',
    >>>        # Dataset Parameter
    >>>        # Impulsary
    >>>        # indicating the data file, please check the data is already there
    >>>        path_imgrec="data/cifar/train.rec",
    >>>        # Dataset Parameter
    >>>        # Impulsary
    >>>        # indicating the image size after preprocessing
    >>>        data_shape=(3,28,28),
    >>>        # Batch Parameter
    >>>        # Impulsary
    >>>        # tells how many images in a batch
    >>>        batch_size=100,
    >>>        # Augmentation Parameter
    >>>        # Optional
    >>>        # when offers mean_img, each image will subtract the mean value at each pixel
    >>>        mean_img="data/cifar/cifar10_mean.bin",
    >>>        # Augmentation Parameter
    >>>        # Optional
    >>>        # randomly crop a patch of the data_shape from the original image
    >>>        rand_crop=True,
    >>>        # Augmentation Parameter
    >>>        # Optional
    >>>        # randomly mirror the image horizontally
    >>>        rand_mirror=True,
    >>>        # Augmentation Parameter
    >>>        # Optional
    >>>        # randomly shuffle the data
    >>>        shuffle=False,
    >>>        # Backend Parameter
    >>>        # Optional
    >>>        # Preprocessing thread number
    >>>        preprocess_threads=4,
    >>>        # Backend Parameter
    >>>        # Optional
    >>>        # Prefetch buffer size
    >>>        prefetch_buffer=1)

First, explicitly specify the kind of data (MNIST, ImageRecord, etc.) to fetch. Then, provide the options for the dataset, batching, image augmentation, multi-tread processing, and prefetching operations. The code automatically validates the parameters. If a required parameter is missing, MXNet returns an error.

How to Get Data

We provide a script to download MNIST data and CIFAR10 ImageRecord data. If you want to create your own dataset, we recommend using the Image Recordio data format.

Create a Dataset Using Recordio

Recordio implements a file format for a sequence of records. We recommend storing images as records and packing them together. The benefits include:

  • Storing images in a compact format–e.g., JPEG, for records–greatly reduces the size of the dataset on the disk.
  • Packing data together allows continuous reading on the disk.
  • Recordio has a simple way to partition, simplifying distributed setting. We provide an example later.

We provide the im2rec tool so you can create an Image Recordio dataset by yourself. The following walkthrough shows you how.

Prerequisites

Download the data. You don’t need to resize the images manually. You can use im2rec to resize them automatically. For details, see the “Extension” topic.

Step 1. Make an Image List File

After you download the data, you need to make an image list file. The format is:

    integer_image_index \t label_index \t path_to_image

Typically, the program takes the list of names of all of the images, shuffles them, then separates them into two lists: a training files name list and a testing file name list. Write the list in the right format.

This is example file:

    95099  464     n04467665_17283.JPEG
    10025081        412     ILSVRC2010_val_00025082.JPEG
    74181   789     n01915811_2739.JPEG
    10035553        859     ILSVRC2010_val_00035554.JPEG
    10048727        929     ILSVRC2010_val_00048728.JPEG
    94028   924     n01980166_4956.JPEG
    1080682 650     n11807979_571.JPEG
    972457  633     n07723039_1627.JPEG
    7534    11      n01630670_4486.JPEG
    1191261 249     n12407079_5106.JPEG

Step 2. Create the Binary File

To generate a binary image, use im2rec in the tool folder. im2rec takes the path of the image list file you generated, the root path of the images, and the output file path as input. This process usually takes several hours, so be patient.

A sample command:

    ./bin/im2rec image.lst image_root_dir output.bin resize=256

For more details, run ./bin/im2rec.

Extension: Multiple Labels for a Single Image

The im2rec tool and mx.io.ImageRecordIter have multi-label support for a single image. For example, if you have four labels for a single image, you can use the following procedure to use the Recordio tools.

  1. Write the the image list files as follows:

        integer_image_index \t label_1 \t label_2 \t   label_3 \t label_4 \t path_to_image
    
  2. Run im2rec, adding a ‘label_width=4’ to the command argument, for example:

        ./bin/im2rec image.lst image_root_dir output.bin resize=256 label_width=4
    
  3. In the iterator generation code, set label_width=4 and path_imglist=<<The PATH TO YOUR image.lst>>, for example:

        dataiter = mx.io.ImageRecordIter(
          path_imgrec="data/cifar/train.rec",
          data_shape=(3,28,28),
          path_imglist="data/cifar/image.lst",
          label_width=4
        )
    
  1. Run the multi-label image iterator:

IO API Reference

NDArray interface of mxnet

class mxnet.io.DataDesc

Named data desc description contains name, shape, type and other extended attributes.

static get_batch_axis(layout)

Get the dimension that corresponds to the batch size.

Parameters:layout (str) – layout string. For example, “NCHW”.
Returns:
  • An axis indicating the batch_size dimension. When data-parallelism is
  • used, the data will be automatically split and concatenate along the batch_size
  • dimension. Axis can be -1, which means the whole array will be copied for each
  • data-parallelism device.
static get_list(shapes, types)

Get DataDesc list from attribute lists.

Parameters:
  • shapes (shape tuple list with (name, shape) tuples) –
  • types (type tuple list with (name, type) tuples) –
class mxnet.io.DataBatch(data, label, pad=None, index=None, bucket_key=None, provide_data=None, provide_label=None)

Default object for holding a mini-batch of data and related information.

class mxnet.io.DataIter

DataIter object in mxnet.

reset()

Reset the iterator.

next()

Get next data batch from iterator. Equivalent to self.iter_next() DataBatch(self.getdata(), self.getlabel(), self.getpad(), None)

Returns:data – The data of next batch.
Return type:DataBatch
iter_next()

Iterate to next batch.

Returns:has_next – Whether the move is successful.
Return type:boolean
getdata()

Get data of current batch.

Returns:data – The data of current batch.
Return type:NDArray
getlabel()

Get label of current batch.

Returns:label – The label of current batch.
Return type:NDArray
getindex()

Get index of the current batch.

Returns:index – The index of current batch
Return type:numpy.array
getpad()

Get the number of padding examples in current batch.

Returns:pad – Number of padding examples in current batch
Return type:int
class mxnet.io.ResizeIter(data_iter, size, reset_internal=True)

Resize a DataIter to given number of batches per epoch. May produce incomplete batch in the middle of an epoch due to padding from internal iterator.

Parameters:
  • data_iter (DataIter) – Internal data iterator.
  • size (number of batches per epoch to resize to.) –
  • reset_internal (whether to reset internal iterator on ResizeIter.reset) –
class mxnet.io.PrefetchingIter(iters, rename_data=None, rename_label=None)

Base class for prefetching iterators. Takes one or more DataIters ( or any class with “reset” and “next” methods) and combine them with prefetching. For example:

Parameters:
  • iters (DataIter or list of DataIter) – one or more DataIters (or any class with “reset” and “next” methods)
  • rename_data (None or list of dict) – i-th element is a renaming map for i-th iter, in the form of {‘original_name’ : ‘new_name’}. Should have one entry for each entry in iter[i].provide_data
  • rename_label (None or list of dict) – Similar to rename_data

Examples

iter = PrefetchingIter([NDArrayIter({‘data’: X1}), NDArrayIter({‘data’: X2})],
rename_data=[{‘data’: ‘data1’}, {‘data’: ‘data2’}])
provide_data

The name and shape of data provided by this iterator

provide_label

The name and shape of label provided by this iterator

class mxnet.io.NDArrayIter(data, label=None, batch_size=1, shuffle=False, last_batch_handle='pad', label_name='softmax_label')

NDArrayIter object in mxnet. Taking NDArray or numpy array to get dataiter.

Parameters:
  • data (NDArray or numpy.ndarray, a list of them, or a dict of string to them.) – NDArrayIter supports single or multiple data and label.
  • label (NDArray or numpy.ndarray, a list of them, or a dict of them.) – Same as data, but is not fed to the model during testing.
  • batch_size (int) – Batch Size
  • shuffle (bool) – Whether to shuffle the data
  • last_batch_handle ('pad', 'discard' or 'roll_over') – How to handle the last batch

Note

This iterator will pad, discard or roll over the last batch if the size of data does not match batch_size. Roll over is intended for training and can cause problems if used for prediction.

provide_data

The name and shape of data provided by this iterator

provide_label

The name and shape of label provided by this iterator

hard_reset()

Igore roll over data and set to start

class mxnet.io.MXDataIter(handle, data_name='data', label_name='softmax_label', **_)

DataIter built in MXNet. List all the needed functions here.

Parameters:handle (DataIterHandle) – the handle to the underlying C++ Data Iterator
debug_skip_load()

Set the iterator to simply return always first batch. .. rubric:: Notes

This can be used to test the speed of network without taking the loading delay into account.

mxnet.io.CSVIter(*args, **kwargs)

Create iterator for dataset in csv.

Parameters:
  • data_csv (string, required) – Dataset Param: Data csv path.
  • data_shape (Shape(tuple), required) – Dataset Param: Shape of the data.
  • label_csv (string, optional, default='NULL') – Dataset Param: Label csv path. If is NULL, all labels will be returned as 0
  • label_shape (Shape(tuple), optional, default=(1,)) – Dataset Param: Shape of the label.
  • name (string, required.) – Name of the resulting data iterator.
Returns:

iterator – The result iterator.

Return type:

DataIter

mxnet.io.ImageRecordIter(*args, **kwargs)

Create iterator for dataset packed in recordio.

Parameters:
  • path_imglist (string, optional, default='') – Dataset Param: Path to image list.
  • path_imgrec (string, optional, default='./data/imgrec.rec') – Dataset Param: Path to image record file.
  • aug_seq (string, optional, default='aug_default') – Augmentation Param: the augmenter names to represent sequence of augmenters to be applied, seperated by comma. Additional keyword parameters will be seen by these augmenters.
  • label_width (int, optional, default='1') – Dataset Param: How many labels for an image.
  • data_shape (Shape(tuple), required) – Dataset Param: Shape of each instance generated by the DataIter.
  • preprocess_threads (int, optional, default='4') – Backend Param: Number of thread to do preprocessing.
  • verbose (boolean, optional, default=True) – Auxiliary Param: Whether to output parser information.
  • num_parts (int, optional, default='1') – partition the data into multiple parts
  • part_index (int, optional, default='0') – the index of the part will read
  • shuffle_chunk_size (long (non-negative), optional, default=0) – the size(MB) of the shuffle chunk, used with shuffle=True, it can enable global shuffling
  • shuffle_chunk_seed (int, optional, default='0') – the seed for chunk shuffling
  • shuffle (boolean, optional, default=False) – Augmentation Param: Whether to shuffle data.
  • seed (int, optional, default='0') – Augmentation Param: Random Seed.
  • batch_size (int (non-negative), required) – Batch Param: Batch size.
  • round_batch (boolean, optional, default=True) – Batch Param: Use round robin to handle overflow batch.
  • prefetch_buffer (long (non-negative), optional, default=4) – Backend Param: Number of prefetched parameters
  • dtype ({None, 'float16', 'float32', 'float64', 'int32', 'uint8'},optional, default='None') – Output data type. Leave as None to useinternal data iterator’s output type
  • resize (int, optional, default='-1') – Augmentation Param: scale shorter edge to size before applying other augmentations.
  • rand_crop (boolean, optional, default=False) – Augmentation Param: Whether to random crop on the image
  • crop_y_start (int, optional, default='-1') – Augmentation Param: Where to nonrandom crop on y.
  • crop_x_start (int, optional, default='-1') – Augmentation Param: Where to nonrandom crop on x.
  • max_rotate_angle (int, optional, default='0') – Augmentation Param: rotated randomly in [-max_rotate_angle, max_rotate_angle].
  • max_aspect_ratio (float, optional, default=0) – Augmentation Param: denotes the max ratio of random aspect ratio augmentation.
  • max_shear_ratio (float, optional, default=0) – Augmentation Param: denotes the max random shearing ratio.
  • max_crop_size (int, optional, default='-1') – Augmentation Param: Maximum crop size.
  • min_crop_size (int, optional, default='-1') – Augmentation Param: Minimum crop size.
  • max_random_scale (float, optional, default=1) – Augmentation Param: Maximum scale ratio.
  • min_random_scale (float, optional, default=1) – Augmentation Param: Minimum scale ratio.
  • max_img_size (float, optional, default=1e+10) – Augmentation Param: Maximum image size after resizing.
  • min_img_size (float, optional, default=0) – Augmentation Param: Minimum image size after resizing.
  • random_h (int, optional, default='0') – Augmentation Param: Maximum random value of H channel in HSL color space.
  • random_s (int, optional, default='0') – Augmentation Param: Maximum random value of S channel in HSL color space.
  • random_l (int, optional, default='0') – Augmentation Param: Maximum random value of L channel in HSL color space.
  • rotate (int, optional, default='-1') – Augmentation Param: Rotate angle.
  • fill_value (int, optional, default='255') – Augmentation Param: Filled color value while padding.
  • inter_method (int, optional, default='1') – Augmentation Param: 0-NN 1-bilinear 2-cubic 3-area 4-lanczos4 9-auto 10-rand.
  • pad (int, optional, default='0') – Augmentation Param: Padding size.
  • mirror (boolean, optional, default=False) – Augmentation Param: Whether to mirror the image.
  • rand_mirror (boolean, optional, default=False) – Augmentation Param: Whether to mirror the image randomly.
  • mean_img (string, optional, default='') – Augmentation Param: Mean Image to be subtracted.
  • mean_r (float, optional, default=0) – Augmentation Param: Mean value on R channel.
  • mean_g (float, optional, default=0) – Augmentation Param: Mean value on G channel.
  • mean_b (float, optional, default=0) – Augmentation Param: Mean value on B channel.
  • mean_a (float, optional, default=0) – Augmentation Param: Mean value on Alpha channel.
  • scale (float, optional, default=1) – Augmentation Param: Scale in color space.
  • max_random_contrast (float, optional, default=0) – Augmentation Param: Maximum ratio of contrast variation.
  • max_random_illumination (float, optional, default=0) – Augmentation Param: Maximum value of illumination variation.
  • name (string, required.) – Name of the resulting data iterator.
Returns:

iterator – The result iterator.

Return type:

DataIter

mxnet.io.ImageRecordUInt8Iter(*args, **kwargs)

Create iterator for dataset packed in recordio.

Parameters:
  • path_imglist (string, optional, default='') – Dataset Param: Path to image list.
  • path_imgrec (string, optional, default='./data/imgrec.rec') – Dataset Param: Path to image record file.
  • aug_seq (string, optional, default='aug_default') – Augmentation Param: the augmenter names to represent sequence of augmenters to be applied, seperated by comma. Additional keyword parameters will be seen by these augmenters.
  • label_width (int, optional, default='1') – Dataset Param: How many labels for an image.
  • data_shape (Shape(tuple), required) – Dataset Param: Shape of each instance generated by the DataIter.
  • preprocess_threads (int, optional, default='4') – Backend Param: Number of thread to do preprocessing.
  • verbose (boolean, optional, default=True) – Auxiliary Param: Whether to output parser information.
  • num_parts (int, optional, default='1') – partition the data into multiple parts
  • part_index (int, optional, default='0') – the index of the part will read
  • shuffle_chunk_size (long (non-negative), optional, default=0) – the size(MB) of the shuffle chunk, used with shuffle=True, it can enable global shuffling
  • shuffle_chunk_seed (int, optional, default='0') – the seed for chunk shuffling
  • shuffle (boolean, optional, default=False) – Augmentation Param: Whether to shuffle data.
  • seed (int, optional, default='0') – Augmentation Param: Random Seed.
  • batch_size (int (non-negative), required) – Batch Param: Batch size.
  • round_batch (boolean, optional, default=True) – Batch Param: Use round robin to handle overflow batch.
  • prefetch_buffer (long (non-negative), optional, default=4) – Backend Param: Number of prefetched parameters
  • dtype ({None, 'float16', 'float32', 'float64', 'int32', 'uint8'},optional, default='None') – Output data type. Leave as None to useinternal data iterator’s output type
  • resize (int, optional, default='-1') – Augmentation Param: scale shorter edge to size before applying other augmentations.
  • rand_crop (boolean, optional, default=False) – Augmentation Param: Whether to random crop on the image
  • crop_y_start (int, optional, default='-1') – Augmentation Param: Where to nonrandom crop on y.
  • crop_x_start (int, optional, default='-1') – Augmentation Param: Where to nonrandom crop on x.
  • max_rotate_angle (int, optional, default='0') – Augmentation Param: rotated randomly in [-max_rotate_angle, max_rotate_angle].
  • max_aspect_ratio (float, optional, default=0) – Augmentation Param: denotes the max ratio of random aspect ratio augmentation.
  • max_shear_ratio (float, optional, default=0) – Augmentation Param: denotes the max random shearing ratio.
  • max_crop_size (int, optional, default='-1') – Augmentation Param: Maximum crop size.
  • min_crop_size (int, optional, default='-1') – Augmentation Param: Minimum crop size.
  • max_random_scale (float, optional, default=1) – Augmentation Param: Maximum scale ratio.
  • min_random_scale (float, optional, default=1) – Augmentation Param: Minimum scale ratio.
  • max_img_size (float, optional, default=1e+10) – Augmentation Param: Maximum image size after resizing.
  • min_img_size (float, optional, default=0) – Augmentation Param: Minimum image size after resizing.
  • random_h (int, optional, default='0') – Augmentation Param: Maximum random value of H channel in HSL color space.
  • random_s (int, optional, default='0') – Augmentation Param: Maximum random value of S channel in HSL color space.
  • random_l (int, optional, default='0') – Augmentation Param: Maximum random value of L channel in HSL color space.
  • rotate (int, optional, default='-1') – Augmentation Param: Rotate angle.
  • fill_value (int, optional, default='255') – Augmentation Param: Filled color value while padding.
  • inter_method (int, optional, default='1') – Augmentation Param: 0-NN 1-bilinear 2-cubic 3-area 4-lanczos4 9-auto 10-rand.
  • pad (int, optional, default='0') – Augmentation Param: Padding size.
  • name (string, required.) – Name of the resulting data iterator.
Returns:

iterator – The result iterator.

Return type:

DataIter

mxnet.io.MNISTIter(*args, **kwargs)

Create iterator for MNIST hand-written digit number recognition dataset.

Parameters:
  • image (string, optional, default='./train-images-idx3-ubyte') – Dataset Param: Mnist image path.
  • label (string, optional, default='./train-labels-idx1-ubyte') – Dataset Param: Mnist label path.
  • batch_size (int, optional, default='128') – Batch Param: Batch Size.
  • shuffle (boolean, optional, default=True) – Augmentation Param: Whether to shuffle data.
  • flat (boolean, optional, default=False) – Augmentation Param: Whether to flat the data into 1D.
  • seed (int, optional, default='0') – Augmentation Param: Random Seed.
  • silent (boolean, optional, default=False) – Auxiliary Param: Whether to print out data info.
  • num_parts (int, optional, default='1') – partition the data into multiple parts
  • part_index (int, optional, default='0') – the index of the part will read
  • prefetch_buffer (long (non-negative), optional, default=4) – Backend Param: Number of prefetched parameters
  • dtype ({None, 'float16', 'float32', 'float64', 'int32', 'uint8'},optional, default='None') – Output data type. Leave as None to useinternal data iterator’s output type
  • name (string, required.) – Name of the resulting data iterator.
Returns:

iterator – The result iterator.

Return type:

DataIter

Next Steps

  • NDArray API for vector/matrix/tensor operations
  • KVStore API for multi-GPU and multi-host distributed training