Matrix Factorization

In a recommendation system, there is a group of users and a set of items. Given that each users have rated some items in the system, we would like to predict how the users would rate the items that they have not yet rated, such that we can make recommendations to the users.

Matrix factorization is one of the mainly used algorithm in recommendation systems. It can be used to discover latent features underlying the interactions between two different kinds of entities.

Assume we assign a k-dimensional vector to each user and a k-dimensional vector to each item such that the dot product of these two vectors gives the user’s rating of that item. We can learn the user and item vectors directly, which is essentially performing SVD on the user-item matrix. We can also try to learn the latent features using multi-layer neural networks.

In this tutorial, we will work though the steps to implement these ideas in MXNet.

Prepare Data

We use the MovieLens data here, but it can apply to other datasets as well. Each row of this dataset contains a tuple of user id, movie id, rating, and time stamp, we will only use the first three items. We first define the a batch which contains n tuples. It also provides name and shape information to MXNet about the data and label.

class Batch(object):
    def __init__(self, data_names, data, label_names, label):
        self.data = data
        self.label = label
        self.data_names = data_names
        self.label_names = label_names
        
    @property
    def provide_data(self):
        return [(n, x.shape) for n, x in zip(self.data_names, self.data)]
    
    @property
    def provide_label(self):
        return [(n, x.shape) for n, x in zip(self.label_names, self.label)]

Then we define a data iterator, which returns a batch of tuples each time.

import mxnet as mx
import random

class Batch(object):
    def __init__(self, data_names, data, label_names, label):
        self.data = data
        self.label = label
        self.data_names = data_names
        self.label_names = label_names

    @property
    def provide_data(self):
        return [(n, x.shape) for n, x in zip(self.data_names, self.data)]

    @property
    def provide_label(self):
        return [(n, x.shape) for n, x in zip(self.label_names, self.label)]

class DataIter(mx.io.DataIter):
    def __init__(self, fname, batch_size):
        super(DataIter, self).__init__()
        self.batch_size = batch_size
        self.data = []
        for line in file(fname):
            tks = line.strip().split('\t')
            if len(tks) != 4:
                continue
            self.data.append((int(tks[0]), int(tks[1]), float(tks[2])))
        self.provide_data = [('user', (batch_size, )), ('item', (batch_size, ))]
        self.provide_label = [('score', (self.batch_size, ))]

    def __iter__(self):
        for k in range(len(self.data) / self.batch_size):
            users = []
            items = []
            scores = []
            for i in range(self.batch_size):
                j = k * self.batch_size + i
                user, item, score = self.data[j]
                users.append(user)
                items.append(item)
                scores.append(score)

            data_all = [mx.nd.array(users), mx.nd.array(items)]
            label_all = [mx.nd.array(scores)]
            data_names = ['user', 'item']
            label_names = ['score']

            data_batch = Batch(data_names, data_all, label_names, label_all)
            yield data_batch

    def reset(self):
        random.shuffle(self.data)

Now we download the data and provide a function to obtain the data iterator:

import os
import urllib
import zipfile
if not os.path.exists('ml-100k.zip'):
    urllib.urlretrieve('http://files.grouplens.org/datasets/movielens/ml-100k.zip', 'ml-100k.zip')
with zipfile.ZipFile("ml-100k.zip","r") as f:
    f.extractall("./")
def get_data(batch_size):
    return (DataIter('./ml-100k/u1.base', batch_size), DataIter('./ml-100k/u1.test', batch_size))

Finally we calculate the numbers of users and items for later use.

def max_id(fname):
    mu = 0
    mi = 0
    for line in file(fname):
        tks = line.strip().split('\t')
        if len(tks) != 4:
            continue
        mu = max(mu, int(tks[0]))
        mi = max(mi, int(tks[1]))
    return mu + 1, mi + 1
max_user, max_item = max_id('./ml-100k/u.data')
(max_user, max_item)
(944, 1683)

Optimization

We first implement the RMSE (root-mean-square error) measurement, which is commonly used by matrix factorization.

import math
def RMSE(label, pred):
    ret = 0.0
    n = 0.0
    pred = pred.flatten()
    for i in range(len(label)):
        ret += (label[i] - pred[i]) * (label[i] - pred[i])
        n += 1.0
    return math.sqrt(ret / n)
"""test_RMSM"""
def test_RMSE(label, pred, expect_ret):
    ret = 0.0
    n = 0.0
    for i in range(len(label)):
        ret += (label[i] - pred[i]) * (label[i] - pred[i])
        n += 1.0
    assert math.sqrt(ret / n) == expect_ret, "RMSE function failed."

test_cases = [{"label" : [1, 2, 3, 4, 5], "pred" : [2, 1, 3, 5, 6]},
             {"label" : [10, 10, 9, 9, 8], "pred" : [10, 10, 9, 9, 8]},
             {"label" : [1, 5, 10, 15, 20], "pred" : [6, 10, 5, 20, 25]}]
expect_res = [math.sqrt(0.8), math.sqrt(0), math.sqrt(25)]
for ts_case, epc_res in zip(test_cases, expect_res):
    test_RMSE(ts_case["label"], ts_case["pred"], epc_res)

Then we define a general training module, which is borrowed from the image classification application.

def train(network, batch_size, num_epoch, learning_rate):
    model = mx.model.FeedForward(
        ctx = mx.gpu(0),  
        symbol = network,
        num_epoch = num_epoch,
        learning_rate = learning_rate,
        wd = 0.0001,
        momentum = 0.9)

    batch_size = 64
    train, test = get_data(batch_size)

    import logging
    head = '%(asctime)-15s %(message)s'
    logging.basicConfig(level=logging.DEBUG)

    model.fit(X = train, 
              eval_data = test,
              eval_metric = RMSE,
              batch_end_callback=mx.callback.Speedometer(batch_size, 20000/batch_size),)

Networks

Now we try various networks. We first learn the latent vectors directly.

# Output may vary
def plain_net(k):
    # input
    user = mx.symbol.Variable('user')
    item = mx.symbol.Variable('item')
    score = mx.symbol.Variable('score')
    # user feature lookup
    user = mx.symbol.Embedding(data = user, input_dim = max_user, output_dim = k) 
    # item feature lookup
    item = mx.symbol.Embedding(data = item, input_dim = max_item, output_dim = k)
    # predict by the inner product, which is elementwise product and then sum
    pred = user * item
    pred = mx.symbol.sum_axis(data = pred, axis = 1)
    pred = mx.symbol.Flatten(data = pred)
    # loss layer
    pred = mx.symbol.LinearRegressionOutput(data = pred, label = score)
    return pred

train(plain_net(64), batch_size=64, num_epoch=10, learning_rate=.05)
INFO:root:Start training with [gpu(0)]
INFO:root:Epoch[0] Batch [312]  Speed: 9629.94 samples/sec  Train-RMSE=3.708882
INFO:root:Epoch[0] Batch [624]  Speed: 9096.20 samples/sec  Train-RMSE=3.696499
INFO:root:Epoch[0] Batch [936]  Speed: 8944.45 samples/sec  Train-RMSE=3.699572
INFO:root:Epoch[0] Batch [1248] Speed: 9526.14 samples/sec  Train-RMSE=3.689985
INFO:root:Epoch[0] Resetting Data Iterator
INFO:root:Epoch[0] Time cost=8.992
INFO:root:Epoch[0] Validation-RMSE=3.714671
INFO:root:Epoch[1] Batch [312]  Speed: 9528.79 samples/sec  Train-RMSE=3.688951
INFO:root:Epoch[1] Batch [624]  Speed: 9012.53 samples/sec  Train-RMSE=3.622500
INFO:root:Epoch[1] Batch [936]  Speed: 9294.79 samples/sec  Train-RMSE=3.284029
INFO:root:Epoch[1] Batch [1248] Speed: 9308.24 samples/sec  Train-RMSE=2.616937
INFO:root:Epoch[1] Resetting Data Iterator
INFO:root:Epoch[1] Time cost=8.664
INFO:root:Epoch[1] Validation-RMSE=2.461273
INFO:root:Epoch[2] Batch [312]  Speed: 9515.72 samples/sec  Train-RMSE=2.030215
INFO:root:Epoch[2] Batch [624]  Speed: 8979.66 samples/sec  Train-RMSE=1.712967
INFO:root:Epoch[2] Batch [936]  Speed: 9189.99 samples/sec  Train-RMSE=1.520069
INFO:root:Epoch[2] Batch [1248] Speed: 9395.22 samples/sec  Train-RMSE=1.378079
INFO:root:Epoch[2] Resetting Data Iterator
INFO:root:Epoch[2] Time cost=8.678
INFO:root:Epoch[2] Validation-RMSE=1.433565
INFO:root:Epoch[3] Batch [312]  Speed: 9463.78 samples/sec  Train-RMSE=1.266224
INFO:root:Epoch[3] Batch [624]  Speed: 9038.83 samples/sec  Train-RMSE=1.216932
INFO:root:Epoch[3] Batch [936]  Speed: 8904.51 samples/sec  Train-RMSE=1.174758
INFO:root:Epoch[3] Batch [1248] Speed: 9103.14 samples/sec  Train-RMSE=1.141839
INFO:root:Epoch[3] Resetting Data Iterator
INFO:root:Epoch[3] Time cost=8.810
INFO:root:Epoch[3] Validation-RMSE=1.202290
INFO:root:Epoch[4] Batch [312]  Speed: 8911.64 samples/sec  Train-RMSE=1.102462
INFO:root:Epoch[4] Batch [624]  Speed: 9559.90 samples/sec  Train-RMSE=1.085300
INFO:root:Epoch[4] Batch [936]  Speed: 8952.48 samples/sec  Train-RMSE=1.064377
INFO:root:Epoch[4] Batch [1248] Speed: 8839.09 samples/sec  Train-RMSE=1.055865
INFO:root:Epoch[4] Resetting Data Iterator
INFO:root:Epoch[4] Time cost=8.867
INFO:root:Epoch[4] Validation-RMSE=1.116540
INFO:root:Epoch[5] Batch [312]  Speed: 8889.74 samples/sec  Train-RMSE=1.038979
INFO:root:Epoch[5] Batch [624]  Speed: 8871.94 samples/sec  Train-RMSE=1.018975
INFO:root:Epoch[5] Batch [936]  Speed: 8951.88 samples/sec  Train-RMSE=1.023208
INFO:root:Epoch[5] Batch [1248] Speed: 9379.24 samples/sec  Train-RMSE=1.032250
INFO:root:Epoch[5] Resetting Data Iterator
INFO:root:Epoch[5] Time cost=8.909
INFO:root:Epoch[5] Validation-RMSE=1.073104
INFO:root:Epoch[6] Batch [312]  Speed: 9332.61 samples/sec  Train-RMSE=1.008186
INFO:root:Epoch[6] Batch [624]  Speed: 8802.00 samples/sec  Train-RMSE=1.008806
INFO:root:Epoch[6] Batch [936]  Speed: 8913.93 samples/sec  Train-RMSE=1.005332
INFO:root:Epoch[6] Batch [1248] Speed: 9095.58 samples/sec  Train-RMSE=0.994927
INFO:root:Epoch[6] Resetting Data Iterator
INFO:root:Epoch[6] Time cost=8.914
INFO:root:Epoch[6] Validation-RMSE=1.051047
INFO:root:Epoch[7] Batch [312]  Speed: 8937.14 samples/sec  Train-RMSE=0.982865
INFO:root:Epoch[7] Batch [624]  Speed: 8962.56 samples/sec  Train-RMSE=0.995960
INFO:root:Epoch[7] Batch [936]  Speed: 8899.06 samples/sec  Train-RMSE=0.992290
INFO:root:Epoch[7] Batch [1248] Speed: 8936.41 samples/sec  Train-RMSE=0.991053
INFO:root:Epoch[7] Resetting Data Iterator
INFO:root:Epoch[7] Time cost=8.987
INFO:root:Epoch[7] Validation-RMSE=1.036044
INFO:root:Epoch[8] Batch [312]  Speed: 9201.62 samples/sec  Train-RMSE=0.975995
INFO:root:Epoch[8] Batch [624]  Speed: 9043.05 samples/sec  Train-RMSE=0.982983
INFO:root:Epoch[8] Batch [936]  Speed: 8869.47 samples/sec  Train-RMSE=0.980471
INFO:root:Epoch[8] Batch [1248] Speed: 8994.31 samples/sec  Train-RMSE=0.983206
INFO:root:Epoch[8] Resetting Data Iterator
INFO:root:Epoch[8] Time cost=8.913
INFO:root:Epoch[8] Validation-RMSE=1.025197
INFO:root:Epoch[9] Batch [312]  Speed: 9027.00 samples/sec  Train-RMSE=0.971992
INFO:root:Epoch[9] Batch [624]  Speed: 9178.34 samples/sec  Train-RMSE=0.983564
INFO:root:Epoch[9] Batch [936]  Speed: 8927.31 samples/sec  Train-RMSE=0.961615
INFO:root:Epoch[9] Batch [1248] Speed: 8870.87 samples/sec  Train-RMSE=0.979395
INFO:root:Epoch[9] Resetting Data Iterator
INFO:root:Epoch[9] Time cost=8.925
INFO:root:Epoch[9] Validation-RMSE=1.017218

Next we try to use 2 layers neural network to learn the latent variables, which stack a fully connected layer above the embedding layers:

# Output may vary
def get_one_layer_mlp(hidden, k):
    # input
    user = mx.symbol.Variable('user')
    item = mx.symbol.Variable('item')
    score = mx.symbol.Variable('score')
    # user latent features
    user = mx.symbol.Embedding(data = user, input_dim = max_user, output_dim = k)
    user = mx.symbol.Activation(data = user, act_type="relu")
    user = mx.symbol.FullyConnected(data = user, num_hidden = hidden)
    # item latent features
    item = mx.symbol.Embedding(data = item, input_dim = max_item, output_dim = k)
    item = mx.symbol.Activation(data = item, act_type="relu")
    item = mx.symbol.FullyConnected(data = item, num_hidden = hidden)
    # predict by the inner product
    pred = user * item
    pred = mx.symbol.sum_axis(data = pred, axis = 1)
    pred = mx.symbol.Flatten(data = pred)
    # loss layer
    pred = mx.symbol.LinearRegressionOutput(data = pred, label = score)
    return pred

train(get_one_layer_mlp(64, 64), batch_size=64, num_epoch=10, learning_rate=.05)
INFO:root:Start training with [gpu(0)]
INFO:root:Epoch[0] Batch [312]  Speed: 5128.80 samples/sec  Train-RMSE=1.336099
INFO:root:Epoch[0] Batch [624]  Speed: 5181.87 samples/sec  Train-RMSE=1.035280
INFO:root:Epoch[0] Batch [936]  Speed: 5283.08 samples/sec  Train-RMSE=1.014390
INFO:root:Epoch[0] Batch [1248] Speed: 5155.22 samples/sec  Train-RMSE=0.989979
INFO:root:Epoch[0] Resetting Data Iterator
INFO:root:Epoch[0] Time cost=15.474
INFO:root:Epoch[0] Validation-RMSE=1.000422
INFO:root:Epoch[1] Batch [312]  Speed: 5247.33 samples/sec  Train-RMSE=0.972317
INFO:root:Epoch[1] Batch [624]  Speed: 5721.52 samples/sec  Train-RMSE=0.961227
INFO:root:Epoch[1] Batch [936]  Speed: 28217.22 samples/sec Train-RMSE=0.965804
INFO:root:Epoch[1] Batch [1248] Speed: 28448.51 samples/sec Train-RMSE=0.964275
INFO:root:Epoch[1] Resetting Data Iterator
INFO:root:Epoch[1] Time cost=8.776
INFO:root:Epoch[1] Validation-RMSE=0.990981
INFO:root:Epoch[2] Batch [312]  Speed: 28208.21 samples/sec Train-RMSE=0.950532
INFO:root:Epoch[2] Batch [624]  Speed: 28778.35 samples/sec Train-RMSE=0.959189
INFO:root:Epoch[2] Batch [936]  Speed: 28823.88 samples/sec Train-RMSE=0.958293
INFO:root:Epoch[2] Batch [1248] Speed: 27988.46 samples/sec Train-RMSE=0.959046
INFO:root:Epoch[2] Resetting Data Iterator
INFO:root:Epoch[2] Time cost=2.870
INFO:root:Epoch[2] Validation-RMSE=0.988801
INFO:root:Epoch[3] Batch [312]  Speed: 27580.76 samples/sec Train-RMSE=0.947317
INFO:root:Epoch[3] Batch [624]  Speed: 29759.86 samples/sec Train-RMSE=0.945208
INFO:root:Epoch[3] Batch [936]  Speed: 29877.50 samples/sec Train-RMSE=0.957878
INFO:root:Epoch[3] Batch [1248] Speed: 29878.52 samples/sec Train-RMSE=0.955912
INFO:root:Epoch[3] Resetting Data Iterator
INFO:root:Epoch[3] Time cost=2.768
INFO:root:Epoch[3] Validation-RMSE=0.970326
INFO:root:Epoch[4] Batch [312]  Speed: 29720.52 samples/sec Train-RMSE=0.942919
INFO:root:Epoch[4] Batch [624]  Speed: 29622.91 samples/sec Train-RMSE=0.944339
INFO:root:Epoch[4] Batch [936]  Speed: 29856.27 samples/sec Train-RMSE=0.955653
INFO:root:Epoch[4] Batch [1248] Speed: 29784.05 samples/sec Train-RMSE=0.949717
INFO:root:Epoch[4] Resetting Data Iterator
INFO:root:Epoch[4] Time cost=2.722
INFO:root:Epoch[4] Validation-RMSE=0.996474
INFO:root:Epoch[5] Batch [312]  Speed: 29795.78 samples/sec Train-RMSE=0.944096
INFO:root:Epoch[5] Batch [624]  Speed: 31356.88 samples/sec Train-RMSE=0.951511
INFO:root:Epoch[5] Batch [936]  Speed: 30231.73 samples/sec Train-RMSE=0.945889
INFO:root:Epoch[5] Batch [1248] Speed: 29484.21 samples/sec Train-RMSE=0.953087
INFO:root:Epoch[5] Resetting Data Iterator
INFO:root:Epoch[5] Time cost=2.682
INFO:root:Epoch[5] Validation-RMSE=0.972851
INFO:root:Epoch[6] Batch [312]  Speed: 30993.66 samples/sec Train-RMSE=0.944349
INFO:root:Epoch[6] Batch [624]  Speed: 30989.71 samples/sec Train-RMSE=0.941070
INFO:root:Epoch[6] Batch [936]  Speed: 31810.05 samples/sec Train-RMSE=0.956245
INFO:root:Epoch[6] Batch [1248] Speed: 29624.67 samples/sec Train-RMSE=0.949010
INFO:root:Epoch[6] Resetting Data Iterator
INFO:root:Epoch[6] Time cost=2.628
INFO:root:Epoch[6] Validation-RMSE=0.982445
INFO:root:Epoch[7] Batch [312]  Speed: 29768.86 samples/sec Train-RMSE=0.929764
INFO:root:Epoch[7] Batch [624]  Speed: 29914.88 samples/sec Train-RMSE=0.942125
INFO:root:Epoch[7] Batch [936]  Speed: 29518.73 samples/sec Train-RMSE=0.950328
INFO:root:Epoch[7] Batch [1248] Speed: 30352.83 samples/sec Train-RMSE=0.959071
INFO:root:Epoch[7] Resetting Data Iterator
INFO:root:Epoch[7] Time cost=2.709
INFO:root:Epoch[7] Validation-RMSE=0.964528
INFO:root:Epoch[8] Batch [312]  Speed: 31856.75 samples/sec Train-RMSE=0.941097
INFO:root:Epoch[8] Batch [624]  Speed: 31219.11 samples/sec Train-RMSE=0.945597
INFO:root:Epoch[8] Batch [936]  Speed: 29146.11 samples/sec Train-RMSE=0.943886
INFO:root:Epoch[8] Batch [1248] Speed: 29576.31 samples/sec Train-RMSE=0.938769
INFO:root:Epoch[8] Resetting Data Iterator
INFO:root:Epoch[8] Time cost=2.664
INFO:root:Epoch[8] Validation-RMSE=1.004047
INFO:root:Epoch[9] Batch [312]  Speed: 29893.42 samples/sec Train-RMSE=0.941613
INFO:root:Epoch[9] Batch [624]  Speed: 29769.79 samples/sec Train-RMSE=0.947198
INFO:root:Epoch[9] Batch [936]  Speed: 29687.21 samples/sec Train-RMSE=0.940494
INFO:root:Epoch[9] Batch [1248] Speed: 29535.93 samples/sec Train-RMSE=0.950362
INFO:root:Epoch[9] Resetting Data Iterator
INFO:root:Epoch[9] Time cost=2.727
INFO:root:Epoch[9] Validation-RMSE=0.963068

Adding dropout layers to relief the over-fitting.

# Output may vary
def get_one_layer_dropout_mlp(hidden, k):
    # input
    user = mx.symbol.Variable('user')
    item = mx.symbol.Variable('item')
    score = mx.symbol.Variable('score')
    # user latent features
    user = mx.symbol.Embedding(data = user, input_dim = max_user, output_dim = k)
    user = mx.symbol.Activation(data = user, act_type="relu")
    user = mx.symbol.FullyConnected(data = user, num_hidden = hidden)
    user = mx.symbol.Dropout(data=user, p=0.5)
    # item latent features
    item = mx.symbol.Embedding(data = item, input_dim = max_item, output_dim = k)
    item = mx.symbol.Activation(data = item, act_type="relu")
    item = mx.symbol.FullyConnected(data = item, num_hidden = hidden)
    item = mx.symbol.Dropout(data=item, p=0.5)    
    # predict by the inner product
    pred = user * item
    pred = mx.symbol.sum_axis(data = pred, axis = 1)
    pred = mx.symbol.Flatten(data = pred)
    # loss layer
    pred = mx.symbol.LinearRegressionOutput(data = pred, label = score)
    return pred
train(get_one_layer_mlp(256, 512), batch_size=64, num_epoch=10, learning_rate=.05)
INFO:root:Start training with [gpu(0)]
INFO:root:Epoch[0] Batch [312]  Speed: 31054.29 samples/sec Train-RMSE=1.292729
INFO:root:Epoch[0] Batch [624]  Speed: 32702.31 samples/sec Train-RMSE=1.003246
INFO:root:Epoch[0] Batch [936]  Speed: 34531.11 samples/sec Train-RMSE=0.988894
INFO:root:Epoch[0] Batch [1248] Speed: 36607.22 samples/sec Train-RMSE=0.976663
INFO:root:Epoch[0] Resetting Data Iterator
INFO:root:Epoch[0] Time cost=2.416
INFO:root:Epoch[0] Validation-RMSE=0.994245
INFO:root:Epoch[1] Batch [312]  Speed: 31948.34 samples/sec Train-RMSE=0.944435
INFO:root:Epoch[1] Batch [624]  Speed: 31777.51 samples/sec Train-RMSE=0.957313
INFO:root:Epoch[1] Batch [936]  Speed: 32294.43 samples/sec Train-RMSE=0.967431
INFO:root:Epoch[1] Batch [1248] Speed: 31945.88 samples/sec Train-RMSE=0.944283
INFO:root:Epoch[1] Resetting Data Iterator
INFO:root:Epoch[1] Time cost=2.536
INFO:root:Epoch[1] Validation-RMSE=0.967241
INFO:root:Epoch[2] Batch [312]  Speed: 32686.94 samples/sec Train-RMSE=0.936268
INFO:root:Epoch[2] Batch [624]  Speed: 36517.05 samples/sec Train-RMSE=0.943588
INFO:root:Epoch[2] Batch [936]  Speed: 34505.16 samples/sec Train-RMSE=0.939588
INFO:root:Epoch[2] Batch [1248] Speed: 32621.01 samples/sec Train-RMSE=0.956108
INFO:root:Epoch[2] Resetting Data Iterator
INFO:root:Epoch[2] Time cost=2.388
INFO:root:Epoch[2] Validation-RMSE=0.960393
INFO:root:Epoch[3] Batch [312]  Speed: 33524.17 samples/sec Train-RMSE=0.935396
INFO:root:Epoch[3] Batch [624]  Speed: 33045.71 samples/sec Train-RMSE=0.948004
INFO:root:Epoch[3] Batch [936]  Speed: 32517.41 samples/sec Train-RMSE=0.930857
INFO:root:Epoch[3] Batch [1248] Speed: 36528.62 samples/sec Train-RMSE=0.951217
INFO:root:Epoch[3] Resetting Data Iterator
INFO:root:Epoch[3] Time cost=2.397
INFO:root:Epoch[3] Validation-RMSE=0.974277
INFO:root:Epoch[4] Batch [312]  Speed: 36791.90 samples/sec Train-RMSE=0.939337
INFO:root:Epoch[4] Batch [624]  Speed: 31957.96 samples/sec Train-RMSE=0.932733
INFO:root:Epoch[4] Batch [936]  Speed: 31928.97 samples/sec Train-RMSE=0.944499
INFO:root:Epoch[4] Batch [1248] Speed: 31924.08 samples/sec Train-RMSE=0.947778
INFO:root:Epoch[4] Resetting Data Iterator
INFO:root:Epoch[4] Time cost=2.458
INFO:root:Epoch[4] Validation-RMSE=1.000416
INFO:root:Epoch[5] Batch [312]  Speed: 31895.47 samples/sec Train-RMSE=0.939974
INFO:root:Epoch[5] Batch [624]  Speed: 31741.68 samples/sec Train-RMSE=0.930035
INFO:root:Epoch[5] Batch [936]  Speed: 31669.50 samples/sec Train-RMSE=0.932193
INFO:root:Epoch[5] Batch [1248] Speed: 31695.69 samples/sec Train-RMSE=0.940311
INFO:root:Epoch[5] Resetting Data Iterator
INFO:root:Epoch[5] Time cost=2.554
INFO:root:Epoch[5] Validation-RMSE=0.958969
INFO:root:Epoch[6] Batch [312]  Speed: 31972.02 samples/sec Train-RMSE=0.921066
INFO:root:Epoch[6] Batch [624]  Speed: 35340.98 samples/sec Train-RMSE=0.945087
INFO:root:Epoch[6] Batch [936]  Speed: 32366.93 samples/sec Train-RMSE=0.943553
INFO:root:Epoch[6] Batch [1248] Speed: 32365.36 samples/sec Train-RMSE=0.933524
INFO:root:Epoch[6] Resetting Data Iterator
INFO:root:Epoch[6] Time cost=2.462
INFO:root:Epoch[6] Validation-RMSE=0.954820
INFO:root:Epoch[7] Batch [312]  Speed: 31849.23 samples/sec Train-RMSE=0.928088
INFO:root:Epoch[7] Batch [624]  Speed: 31838.36 samples/sec Train-RMSE=0.938085
INFO:root:Epoch[7] Batch [936]  Speed: 32740.23 samples/sec Train-RMSE=0.923663
INFO:root:Epoch[7] Batch [1248] Speed: 32602.77 samples/sec Train-RMSE=0.949745
INFO:root:Epoch[7] Resetting Data Iterator
INFO:root:Epoch[7] Time cost=2.514
INFO:root:Epoch[7] Validation-RMSE=0.961318
INFO:root:Epoch[8] Batch [312]  Speed: 34563.87 samples/sec Train-RMSE=0.933115
INFO:root:Epoch[8] Batch [624]  Speed: 32887.82 samples/sec Train-RMSE=0.925201
INFO:root:Epoch[8] Batch [936]  Speed: 32030.49 samples/sec Train-RMSE=0.925663
INFO:root:Epoch[8] Batch [1248] Speed: 31951.20 samples/sec Train-RMSE=0.938921
INFO:root:Epoch[8] Resetting Data Iterator
INFO:root:Epoch[8] Time cost=2.470
INFO:root:Epoch[8] Validation-RMSE=0.954113
INFO:root:Epoch[9] Batch [312]  Speed: 32616.63 samples/sec Train-RMSE=0.933334
INFO:root:Epoch[9] Batch [624]  Speed: 32516.19 samples/sec Train-RMSE=0.921008
INFO:root:Epoch[9] Batch [936]  Speed: 31847.39 samples/sec Train-RMSE=0.926811
INFO:root:Epoch[9] Batch [1248] Speed: 34600.90 samples/sec Train-RMSE=0.935853
INFO:root:Epoch[9] Resetting Data Iterator
INFO:root:Epoch[9] Time cost=2.467
INFO:root:Epoch[9] Validation-RMSE=0.972423

Acknowledgement

This tutorial is based on examples from xlvector/github.