Data Loading API

Overview

This document summeries supported data formats and iterator APIs to read the data including

mxnet.io Data iterators for common data formats.
mxnet.recordio Read and write for the RecrodIO data format
mxnet.image Read invidual image files and perform augmentations.

It will also show how to write an iterator for a new data format.

A data iterator reads data batch by batch.

>>> data = mx.nd.ones((100,10))
>>> nd_iter = mx.io.NDArrayIter(data, batch_size=25)
>>> for batch in nd_iter:
...     print(batch.data)
[<NDArray 25x10 @cpu(0)>]
[<NDArray 25x10 @cpu(0)>]
[<NDArray 25x10 @cpu(0)>]
[<NDArray 25x10 @cpu(0)>]

If nd_iter.reset() is called, then reads the data again from beginning.

In addition, an iterator provides information about the batch, including the shapes and name.

>>> nd_iter = mx.io.NDArrayIter(data={'data':mx.nd.ones((100,10))},
...                             label={'softmax_label':mx.nd.ones((100,))},
...                             batch_size=25)
>>> print(nd_iter.provide_data)
[DataDesc[data,(25, 10L),<type 'numpy.float32'>,NCHW]]
>>> print(nd_iter.provide_label)
[DataDesc[softmax_label,(25,),<type 'numpy.float32'>,NCHW]]

So this iterator can be used to train a symbol whose input data variable has name data and input label variable has name softmax_label.

>>> data = mx.sym.Variable('data')
>>> label = mx.sym.Variable('softmax_label')
>>> fullc = mx.sym.FullyConnected(data=data, num_hidden=1)
>>> loss = mx.sym.SoftmaxOutput(data=data, label=label)
>>> mod = mx.mod.Module(loss)
>>> print(mod.data_names)
['data']
>>> print(mod.label_names)
['softmax_label']
>>> mod.bind(data_shapes=nd_iter.provide_data, label_shapes=nd_iter.provide_label)

Then we can call mod.fit(nd_iter, num_epoch=2) to train loss by 2 epochs.

Data iterators

io.NDArrayIter Iterating on either mx.nd.NDArray or numpy.ndarray.
io.CSVIter Iterating on CSV files
io.ImageRecordIter Iterating on image RecordIO files
io.ImageRecordUInt8Iter Create iterator for dataset packed in recordio.
io.MNISTIter Iterating on the MNIST dataset.
recordio.MXRecordIO Read/write RecordIO formmat data.
recordio.MXIndexedRecordIO Read/write RecordIO formmat data supporting random access.
image.ImageIter Image data iterator with a large number of augumentation choices.

Helper classes and functions

Data structures and other iterators provided in the mxnet.io packages.

io.DataDesc Data description
io.DataBatch A data batch.
io.DataIter The base class of a data iterator.
io.ResizeIter Resize a data iterator to a given number of batches.
io.PrefetchingIter Performs pre-fetch for other data iterators.
io.MXDataIter A python wrapper a C++ data iterator.

A list of image modification functions provided by mxnet.image.

image.imdecode Decode an image from string.
image.scale_down Scale down crop size if it’s bigger than image size.
image.resize_short Resize shorter edge to size.
image.fixed_crop Crop src at fixed location, and (optionally) resize it to size.
image.random_crop Randomly crop src with size.
image.center_crop Randomly crop src with size.
image.color_normalize Normalize src with mean and std.
image.random_size_crop Randomly crop src with size.
image.ResizeAug Make resize shorter edge to size augumenter.
image.RandomCropAug Make random crop augumenter
image.RandomSizedCropAug Make random crop with random resizing and random aspect ratio jitter augumenter.
image.CenterCropAug Make center crop augmenter.
image.RandomOrderAug Apply list of augmenters in random order
image.ColorJitterAug Apply random brightness, contrast and saturation jitter in random order.
image.LightingAug Add PCA based noise.
image.ColorNormalizeAug Mean and std normalization.
image.HorizontalFlipAug Random horizontal flipping.
image.CastAug Cast to float32
image.CreateAugmenter Create augumenter list.

Functions to read and write RecordIO files.

recordio.pack Pack an string into MXImageRecord.
recordio.unpack Unpack a MXImageRecord to string.
recordio.unpack_img Unpack a MXImageRecord to image.
recordio.pack_img Pack an image into MXImageRecord.

Develop a new iterator

Writing a new data iterator in Python is straightforward. Most MXNet training/inference program accepts an iteratable object with provide_data and provide_label properties. This tutorial how to write an iterator from scratch.

The following example demonstrates how to combine multiple data iterators into a single one. It can be used for multiple modality training such as image captioning, in which images can be read by ImageRecordIter while documents by CSVIter

class MultiIter:
    def __init__(self, iter_list):
        self.iters = iter_list
    def next(self):
        batches = [i.next() for i in self.iters]
        return DataBatch(data=[*b.data for b in batches],
                         label=[*b.label for b in batches])
    def reset(self):
        for i in self.iters:
            i.reset()
    @property
    def provide_data(self):
        return [*i.provide_data for i in self.iters]
    @property
    def provide_label(self):
        return [*i.provide_label for i in self.iters]

iter = MultiIter([mx.io.ImageRecordIter('image.rec'), mx.io.CSVIter('txt.csv')])

Parsing and another pre-processing such as augmentation may be expensive. If performance is critical, we can implement a data iterator in C++. Refer to src/io for examples.

API Reference

Data iterators for common data formats.

class mxnet.io.DataDesc

Data description

Parameters:
  • cls (DataDesc) – The class.
  • name (str) – Data name.
  • shape (tuple of int) – Data shape.
  • dtype (np.dtype, optional) – Data type.
  • layout (str, optional) – Data layout.
static get_batch_axis(layout)

Get the dimension that corresponds to the batch size.

When data parallelism is used, the data will be automatically split and concatenated along the batch-size dimension. Axis can be -1, which means the whole array will be copied for each data-parallelism device.

Parameters:layout (str) – layout string. For example, “NCHW”.
Returns:An axis indicating the batch_size dimension.
Return type:int
static get_list(shapes, types)

Get DataDesc list from attribute lists.

Parameters:
class mxnet.io.DataBatch(data, label, pad=None, index=None, bucket_key=None, provide_data=None, provide_label=None)

A data batch.

Parameters:
  • data (list of NDArray) – A list of input data.
  • label (list of NDArray) – A list of input labels.
  • pad (int, optional) – The number of examples padded at the batch end. It is used when the examples read is less than the batch size.
  • index (numpy.array, optional) – The example indices in this batch.
  • bucket_key (int, optional) – The key of the bucket, used for bucket IO.
  • provide_data (list of (name, shape), optional) – The i-th elements describes the name and shape of data[i].
  • provide_label (list of (name, shape), optional) – The i-th elements describes the name and shape of label[i].
class mxnet.io.DataIter(batch_size=0)

The base class of a data iterator.

Parameters:batch_size (int, optional) – The batch size, namely the number of examples in a batch.
reset()

Reset the iterator to the begin of the data.

next()

Get next data batch from iterator.

Returns:The data of next batch.
Return type:DataBatch
Raises:StopIteration – If the end of the data is reached.
iter_next()

Move to the next batch.

Returns:Whether the move is successful.
Return type:boolean
getdata()

Get data of current batch.

Returns:The data of the current batch.
Return type:list of NDArray
getlabel()

Get label of the current batch.

Returns:The label of the current batch.
Return type:list of NDArray
getindex()

Get index of the current batch.

Returns:index – The indices of examples in the current batch.
Return type:numpy.array
getpad()

Get the number of padding examples in the current batch.

Returns:Number of padding examples in the current batch.
Return type:int
class mxnet.io.ResizeIter(data_iter, size, reset_internal=True)

Resize a data iterator to a given number of batches.

Parameters:
  • data_iter (DataIter) – The data iterator to be resized.
  • size (int) – The number of batches per epoch to resize to.
  • reset_internal (bool) – Whether to reset internal iterator on ResizeIter.reset.

Examples

>>> nd_iter = mx.io.NDArrayIter(mx.nd.ones((100,10)), batch_size=25)
>>> resize_iter = mx.io.ResizeIter(nd_iter, 2)
>>> for batch in resize_iter:
...     print(batch.data)
[<NDArray 25x10 @cpu(0)>]
[<NDArray 25x10 @cpu(0)>]
class mxnet.io.PrefetchingIter(iters, rename_data=None, rename_label=None)

Performs pre-fetch for other data iterators.

This iterator will create another thread to perform iter_next and then store the data in memory. It potentially accelerates the data read, at the cost of more memory usage.

Parameters:
  • iters (DataIter or list of DataIter) – The data iterators to be pre-fetched.
  • rename_data (None or list of dict) – The i-th element is a renaming map for the i-th iter, in the form of {‘original_name’ : ‘new_name’}. Should have one entry for each entry in iter[i].provide_data.
  • rename_label (None or list of dict) – Similar to rename_data.

Examples

>>> iter1 = mx.io.NDArrayIter({'data':mx.nd.ones((100,10))}, batch_size=25)
>>> iter2 = mx.io.NDArrayIter({'data':mx.nd.ones((100,10))}, batch_size=25)
>>> piter = mx.io.PrefetchingIter([iter1, iter2],
...                               rename_data=[{'data': 'data_1'}, {'data': 'data_2'}])
>>> print(piter.provide_data)
[DataDesc[data_1,(25, 10L),<type 'numpy.float32'>,NCHW],
 DataDesc[data_2,(25, 10L),<type 'numpy.float32'>,NCHW]]
class mxnet.io.NDArrayIter(data, label=None, batch_size=1, shuffle=False, last_batch_handle='pad', data_name='data', label_name='softmax_label')

Iterating on either mx.nd.NDArray or numpy.ndarray.

Parameters:
  • data (array or list of array or dict of string to array) – Input data
  • label (array or list of array or dict of string to array, optional) – Input label
  • batch_size (int) – Batch Size
  • shuffle (bool, optional) – Whether to shuffle the data
  • last_batch_handle (str, optional) – How to handle the last batch, can be ‘pad’, ‘discard’ or ‘roll_over’. ‘roll_over’ is intended for training and can cause problems if used for prediction.
  • data_name (str, optional) – The data name
  • label_name (str, optional) – The label name
provide_data

The name and shape of data provided by this iterator.

provide_label

The name and shape of label provided by this iterator.

hard_reset()

Ignore roll over data and set to start.

class mxnet.io.MXDataIter(handle, data_name='data', label_name='softmax_label', **_)

A python wrapper a C++ data iterator.

Parameters:handle (DataIterHandle) – The handle to the underlying C++ Data Iterator.
mxnet.io.CSVIter(*args, **kwargs)

Iterating on CSV files

Assume there is CSV file at data/data.csv with content:

1,2,3
2,3,4
3,4,5
4,5,6

If we set:

data_csv = 'data/data.csv'
data_shape = (3,)
batch_size = 2

Then this iterator will reads two batches:

[[ 1.  2.  3.]
 [ 2.  3.  4.]]
[[ 3.  4.  5.]
 [ 4.  5.  6.]]

If set data_csv = 'data/', then all files in this directory will be read.

Defined in src/io/iter_csv.cc:L157

Parameters:
  • data_csv (string, required) – The filename of a CSV file or a directory path
  • data_shape (Shape(tuple), required) – The shape of one example
  • label_csv (string, optional, default='NULL') – The filename of a CSV file or a directory path. If NULL, all labels will be returned as 0
  • label_shape (Shape(tuple), optional, default=(1,)) – The shape of one label.
  • batch_size (int (non-negative), required) – Batch size.
  • round_batch (boolean, optional, default=True) – If or not use round robin to handle overflow batch.
  • prefetch_buffer (long (non-negative), optional, default=4) – Maximal Number of batches to prefetch
  • dtype ({None, 'float16', 'float32', 'float64', 'int32', 'uint8'},optional, default='None') – Output data type. None means no change
Returns:

The result iterator.

Return type:

MXDataIter

mxnet.io.ImageRecordIter(*args, **kwargs)

Iterating on image RecordIO files

Read images batches from RecordIO files with a rich of data augmentation options.

One can use tools/im2rec.py to pack individual image files into RecordIO files.

Defined in src/io/iter_image_recordio.cc:L467

Parameters:
  • path_imglist (string, optional, default='') – Path to the image list file
  • path_imgrec (string, optional, default='') – Filename of the image RecordIO file or a directory path.
  • aug_seq (string, optional, default='aug_default') – The augmenter names to represent sequence of augmenters to be applied, seperated by comma. Additional keyword parameters will be seen by these augmenters.
  • label_width (int, optional, default='1') – The number of labels per image.
  • data_shape (Shape(tuple), required) – The shape of one output image.
  • preprocess_threads (int, optional, default='4') – The number of threads.
  • verbose (boolean, optional, default=True) – If or not output verbose information.
  • num_parts (int, optional, default='1') – Virtual partition data into n parts
  • part_index (int, optional, default='0') – The i-th virtual partition will read
  • shuffle_chunk_size (long (non-negative), optional, default=0) – The data shuffle buffer size in MB. Only valid if shuffle is true
  • shuffle_chunk_seed (int, optional, default='0') – The random seed for shuffling
  • shuffle (boolean, optional, default=False) – If or not randomly shuffle data.
  • seed (int, optional, default='0') – The random seed.
  • batch_size (int (non-negative), required) – Batch size.
  • round_batch (boolean, optional, default=True) – If or not use round robin to handle overflow batch.
  • prefetch_buffer (long (non-negative), optional, default=4) – Maximal Number of batches to prefetch
  • dtype ({None, 'float16', 'float32', 'float64', 'int32', 'uint8'},optional, default='None') – Output data type. None means no change
  • resize (int, optional, default='-1') – Down scale the shorter edge to a new size before applying other augmentations.
  • rand_crop (boolean, optional, default=False) – If or not randomly crop the image
  • max_rotate_angle (int, optional, default='0') – Rotate by a random degree in [-v, v]
  • max_aspect_ratio (float, optional, default=0) – Change the aspect (namely width/height) to a random value in [1 - max_aspect_ratio, 1 + max_aspect_ratio]
  • max_shear_ratio (float, optional, default=0) – Apply a shear transformation (namely (x,y)->(x+my,y)) with m randomly chose from [-max_shear_ratio, max_shear_ratio]
  • max_crop_size (int, optional, default='-1') – Crop both width and height into a random size in [min_crop_size, max_crop_size]
  • min_crop_size (int, optional, default='-1') – Crop both width and height into a random size in [min_crop_size, max_crop_size]
  • max_random_scale (float, optional, default=1) – Resize into [width*s, height*s] with s randsomly chosen from [min_random_scale, max_random_scale]
  • min_random_scale (float, optional, default=1) – Resize into [width*s, height*s] with s randsomly chosen from [min_random_scale, max_random_scale]
  • max_img_size (float, optional, default=1e+10) – Set the maximal width and height after all resize and rotate argumentation are applied
  • min_img_size (float, optional, default=0) – Set the minimal width and height after all resize and rotate argumentation are applied
  • random_h (int, optional, default='0') – Add a random value in [-random_h, random_h] to the H channel in HSL color space.
  • random_s (int, optional, default='0') – Add a random value in [-random_s, random_s] to the S channel in HSL color space.
  • random_l (int, optional, default='0') – Add a random value in [-random_l, random_l] to the L channel in HSL color space.
  • rotate (int, optional, default='-1') – Rotate by an angle. If set, it overrites the max_rotate_angle option.
  • fill_value (int, optional, default='255') – Set the padding pixes value into fill_value.
  • inter_method (int, optional, default='1') – The interpolation method: 0-NN 1-bilinear 2-cubic 3-area 4-lanczos4 9-auto 10-rand.
  • pad (int, optional, default='0') – Change size from [width, height] into [pad + width + pad, pad + height + pad] by padding pixes
  • mirror (boolean, optional, default=False) – If or not mirror the image.
  • rand_mirror (boolean, optional, default=False) – If or not randomly the image.
  • mean_img (string, optional, default='') – Filename of the The mean image.
  • mean_r (float, optional, default=0) – The mean value to be subtracted on the R channel
  • mean_g (float, optional, default=0) – The mean value to be subtracted on the G channel
  • mean_b (float, optional, default=0) – The mean value to be subtracted on the B channel
  • mean_a (float, optional, default=0) – The mean value to be subtracted on the alpha channel
  • scale (float, optional, default=1) – Multiply the image with a scale value.
  • max_random_contrast (float, optional, default=0) – Change the contrast with a value randomly chosen from [-max_random_contrast, max_random_contrast]
  • max_random_illumination (float, optional, default=0) – Change the illumination with a value randomly chosen from [-max_random_illumination, max_random_illumination]
Returns:

The result iterator.

Return type:

MXDataIter

mxnet.io.ImageRecordUInt8Iter(*args, **kwargs)

Create iterator for dataset packed in recordio.

Parameters:
  • path_imglist (string, optional, default='') – Path to the image list file
  • path_imgrec (string, optional, default='') – Filename of the image RecordIO file or a directory path.
  • aug_seq (string, optional, default='aug_default') – The augmenter names to represent sequence of augmenters to be applied, seperated by comma. Additional keyword parameters will be seen by these augmenters.
  • label_width (int, optional, default='1') – The number of labels per image.
  • data_shape (Shape(tuple), required) – The shape of one output image.
  • preprocess_threads (int, optional, default='4') – The number of threads.
  • verbose (boolean, optional, default=True) – If or not output verbose information.
  • num_parts (int, optional, default='1') – Virtual partition data into n parts
  • part_index (int, optional, default='0') – The i-th virtual partition will read
  • shuffle_chunk_size (long (non-negative), optional, default=0) – The data shuffle buffer size in MB. Only valid if shuffle is true
  • shuffle_chunk_seed (int, optional, default='0') – The random seed for shuffling
  • shuffle (boolean, optional, default=False) – If or not randomly shuffle data.
  • seed (int, optional, default='0') – The random seed.
  • batch_size (int (non-negative), required) – Batch size.
  • round_batch (boolean, optional, default=True) – If or not use round robin to handle overflow batch.
  • prefetch_buffer (long (non-negative), optional, default=4) – Maximal Number of batches to prefetch
  • dtype ({None, 'float16', 'float32', 'float64', 'int32', 'uint8'},optional, default='None') – Output data type. None means no change
  • resize (int, optional, default='-1') – Down scale the shorter edge to a new size before applying other augmentations.
  • rand_crop (boolean, optional, default=False) – If or not randomly crop the image
  • max_rotate_angle (int, optional, default='0') – Rotate by a random degree in [-v, v]
  • max_aspect_ratio (float, optional, default=0) – Change the aspect (namely width/height) to a random value in [1 - max_aspect_ratio, 1 + max_aspect_ratio]
  • max_shear_ratio (float, optional, default=0) – Apply a shear transformation (namely (x,y)->(x+my,y)) with m randomly chose from [-max_shear_ratio, max_shear_ratio]
  • max_crop_size (int, optional, default='-1') – Crop both width and height into a random size in [min_crop_size, max_crop_size]
  • min_crop_size (int, optional, default='-1') – Crop both width and height into a random size in [min_crop_size, max_crop_size]
  • max_random_scale (float, optional, default=1) – Resize into [width*s, height*s] with s randsomly chosen from [min_random_scale, max_random_scale]
  • min_random_scale (float, optional, default=1) – Resize into [width*s, height*s] with s randsomly chosen from [min_random_scale, max_random_scale]
  • max_img_size (float, optional, default=1e+10) – Set the maximal width and height after all resize and rotate argumentation are applied
  • min_img_size (float, optional, default=0) – Set the minimal width and height after all resize and rotate argumentation are applied
  • random_h (int, optional, default='0') – Add a random value in [-random_h, random_h] to the H channel in HSL color space.
  • random_s (int, optional, default='0') – Add a random value in [-random_s, random_s] to the S channel in HSL color space.
  • random_l (int, optional, default='0') – Add a random value in [-random_l, random_l] to the L channel in HSL color space.
  • rotate (int, optional, default='-1') – Rotate by an angle. If set, it overrites the max_rotate_angle option.
  • fill_value (int, optional, default='255') – Set the padding pixes value into fill_value.
  • inter_method (int, optional, default='1') – The interpolation method: 0-NN 1-bilinear 2-cubic 3-area 4-lanczos4 9-auto 10-rand.
  • pad (int, optional, default='0') – Change size from [width, height] into [pad + width + pad, pad + height + pad] by padding pixes
Returns:

The result iterator.

Return type:

MXDataIter

mxnet.io.MNISTIter(*args, **kwargs)

Iterating on the MNIST dataset.

One can download the dataset from http://yann.lecun.com/exdb/mnist/

Defined in src/io/iter_mnist.cc:L246

Parameters:
  • image (string, optional, default='./train-images-idx3-ubyte') – Dataset Param: Mnist image path.
  • label (string, optional, default='./train-labels-idx1-ubyte') – Dataset Param: Mnist label path.
  • batch_size (int, optional, default='128') – Batch Param: Batch Size.
  • shuffle (boolean, optional, default=True) – Augmentation Param: Whether to shuffle data.
  • flat (boolean, optional, default=False) – Augmentation Param: Whether to flat the data into 1D.
  • seed (int, optional, default='0') – Augmentation Param: Random Seed.
  • silent (boolean, optional, default=False) – Auxiliary Param: Whether to print out data info.
  • num_parts (int, optional, default='1') – partition the data into multiple parts
  • part_index (int, optional, default='0') – the index of the part will read
  • prefetch_buffer (long (non-negative), optional, default=4) – Maximal Number of batches to prefetch
  • dtype ({None, 'float16', 'float32', 'float64', 'int32', 'uint8'},optional, default='None') – Output data type. None means no change
Returns:

The result iterator.

Return type:

MXDataIter

Read invidual image files and perform augmentations.

mxnet.image.imdecode(buf, **kwargs)

Decode an image from string. Requires OpenCV to work.

Parameters:
  • buf (str/bytes, or numpy.ndarray) – Binary image data.
  • flag (int) – 0 for grayscale. 1 for colored.
  • to_rgb (int) – 0 for BGR format (OpenCV default). 1 for RGB format (MXNet default).
  • out (NDArray) – Output buffer. Use None for automatic allocation.
mxnet.image.scale_down(src_size, size)

Scale down crop size if it’s bigger than image size.

mxnet.image.resize_short(src, size, interp=2)

Resize shorter edge to size.

mxnet.image.fixed_crop(src, x0, y0, w, h, size=None, interp=2)

Crop src at fixed location, and (optionally) resize it to size.

mxnet.image.random_crop(src, size, interp=2)

Randomly crop src with size. Upsample result if src is smaller than size.

mxnet.image.center_crop(src, size, interp=2)

Randomly crop src with size. Upsample result if src is smaller than size.

mxnet.image.color_normalize(src, mean, std=None)

Normalize src with mean and std.

mxnet.image.random_size_crop(src, size, min_area, ratio, interp=2)

Randomly crop src with size. Randomize area and aspect ratio.

mxnet.image.ResizeAug(size, interp=2)

Make resize shorter edge to size augumenter.

mxnet.image.RandomCropAug(size, interp=2)

Make random crop augumenter

mxnet.image.RandomSizedCropAug(size, min_area, ratio, interp=2)

Make random crop with random resizing and random aspect ratio jitter augumenter.

mxnet.image.CenterCropAug(size, interp=2)

Make center crop augmenter.

mxnet.image.RandomOrderAug(ts)

Apply list of augmenters in random order

mxnet.image.ColorJitterAug(brightness, contrast, saturation)

Apply random brightness, contrast and saturation jitter in random order.

mxnet.image.LightingAug(alphastd, eigval, eigvec)

Add PCA based noise.

mxnet.image.ColorNormalizeAug(mean, std)

Mean and std normalization.

mxnet.image.HorizontalFlipAug(p)

Random horizontal flipping.

mxnet.image.CastAug()

Cast to float32

mxnet.image.CreateAugmenter(data_shape, resize=0, rand_crop=False, rand_resize=False, rand_mirror=False, mean=None, std=None, brightness=0, contrast=0, saturation=0, pca_noise=0, inter_method=2)

Create augumenter list.

class mxnet.image.ImageIter(batch_size, data_shape, label_width=1, path_imgrec=None, path_imglist=None, path_root=None, path_imgidx=None, shuffle=False, part_index=0, num_parts=1, aug_list=None, imglist=None, **kwargs)

Image data iterator with a large number of augumentation choices. Supports reading from both .rec files and raw image files with image list.

To load from .rec files, please specify path_imgrec. Also specify path_imgidx to use data partition (for distributed training) or shuffling.

To load from raw image files, specify path_imglist and path_root.

Parameters:
  • batch_size (int) – Number of examples per batch.
  • data_shape (tuple) – Data shape in (channels, height, width). For now, only RGB image with 3 channels is supported.
  • label_width (int) – dimension of label
  • path_imgrec (str) – path to image record file (.rec). Created with tools/im2rec.py or bin/im2rec
  • path_imglist (str) – path to image list (.lst) Created with tools/im2rec.py or with custom script. Format: index [one or more label separated by ] relative_path_from_root.
  • imglist (list) – a list of image with the label(s) each item is a list [imagelabel: float or list of float, imgpath].
  • path_root (str) – Root folder of image files
  • path_imgidx (str) – Path to image index file. Needed for partition and shuffling when using .rec source.
  • shuffle (bool) – Whether to shuffle all images at the start of each iteration. Can be slow for HDD.
  • part_index (int) – Partition index
  • num_parts (int) – Total number of partitions.
  • kwargs – More arguments for creating augumenter. See mx.image.CreateAugmenter.
next_sample()

Helper function for reading in next sample.

Read and write for the RecrodIO data format

class mxnet.recordio.MXRecordIO(uri, flag)

Read/write RecordIO formmat data.

Parameters:
  • uri (string) – uri path to recordIO file.
  • flag (string) – “r” for reading or “w” writing.
open()

Open record file.

close()

Close record file.

reset()

Reset pointer to first item. If record is opened with ‘w’, this will truncate the file to empty.

write(buf)

Write a string buffer as a record.

Parameters:buf (string (python2), bytes (python3)) – Buffer to write.
read()

Read a record as string.

Returns:buf – Buffer read.
Return type:string
class mxnet.recordio.MXIndexedRecordIO(idx_path, uri, flag, key_type=<type 'int'>)

Read/write RecordIO formmat data supporting random access.

Parameters:
  • idx_path (str) – Path to index file.
  • uri (str) – Path to record file. Only support file types that are seekable.
  • flag (str) – ‘w’ for write or ‘r’ for read
  • key_type (type) – Data type for keys.
seek(idx)

Query current read head position.

tell()

Query current write head position.

read_idx(idx)

Read record with index.

write_idx(idx, buf)

Write record with index.

mxnet.recordio.IRHeader

alias of HEADER

mxnet.recordio.pack(header, s)

Pack an string into MXImageRecord.

Parameters:
  • header (IRHeader) – Header of the image record. header.label can be a number or an array.
  • s (str) – string to pack
mxnet.recordio.unpack(s)

Unpack a MXImageRecord to string.

Parameters:s (str) – String buffer from MXRecordIO.read.
Returns:
  • header (IRHeader) – Header of the image record.
  • s (str) – Unpacked string.
mxnet.recordio.unpack_img(s, iscolor=-1)

Unpack a MXImageRecord to image.

Parameters:
  • s (str) – String buffer from MXRecordIO.read.
  • iscolor (int) – image format option for cv2.imdecode.
Returns:

  • header (IRHeader) – Header of the image record.
  • img (numpy.ndarray) – Unpacked image.

mxnet.recordio.pack_img(header, img, quality=95, img_fmt='.jpg')

Pack an image into MXImageRecord.

Parameters:
  • header (IRHeader) – Header of the image record. header.label can be a number or an array.
  • img (numpy.ndarray) – image to pack
  • quality (int) – Quality for JPEG encoding in range 1-100, or compression for PNG encoding in range 1-9.
  • img_fmt (str) – Encoding of the image (.jpg for JPEG, .png for PNG).
Returns:

s – The packed string.

Return type:

str