Skip to main content

Incremental collaborative filtering algorithms for recommender systems

Project description

CF STEP - Incremental Collaborative Filtering

Incremental learning for recommender systems

CF STEP is an open-source library, written in python, that enables fast implementation of incremental learning recommender systems. The library is a by-product of the research project CloudDBAppliance.


Run pip install cf-step to install the library in your environment.

How to use

For this example, we will use the popular movielens dataset. The dataset has collected and made available rating data sets from the MovieLens web site. The data sets were collected over various periods of time, depending on the size of the set.

First let us load the data in a pandas DataFrame. We assume that the reader has downloaded the 1m movielense dataset and have unziped it in the /tmp folder.

To avoid creating a user and movie vocabularies we turn each user and movie to a categorical feature and use the pandas convenient cat attribute to get the codes

# local

# load the data
col_names = ['user_id', 'movie_id', 'rating', 'timestamp']
ratings_df = pd.read_csv('/tmp/ratings.dat', delimiter='::', names=col_names, engine='python')

# transform users and movies to categorical features
ratings_df['user_id'] = ratings_df['user_id'].astype('category')
ratings_df['movie_id'] = ratings_df['movie_id'].astype('category')

# use the codes to avoid creating separate vocabularies
ratings_df['user_code'] = ratings_df['user_id']
ratings_df['movie_code'] = ratings_df['movie_id']

.dataframe tbody tr th {
    vertical-align: top;

.dataframe thead th {
    text-align: right;

Using the codes we can see how many users and movies are in the dataset.

# local
n_users = ratings_df['user_code'].max() + 1
n_movies = ratings_df['movie_code'].max() + 1

print(f'There are {n_users} unique users and {n_movies} unique movies in the movielens dataset.')
There are 6040 unique users and 3706 unique movies in the movielens dataset.

We will sort the data by Timestamp so as to simulate streaming events.

# local
data_df = ratings_df.sort_values(by='timestamp')

The Step model supports only positive feedback. Thus, we will consider a rating of 5 as positive feedback and discard any other. We want to identify likes with 1 and dislikes with 0.

# local
# more than 4 -> 1, less than 5 -> 0
data_df['preference'] = np.where(data_df['rating'] > 4, 1, 0)
# keep only ones and discard the others
data_df_cleaned = data_df.loc[data_df['preference'] == 1]

.dataframe tbody tr th {
    vertical-align: top;

.dataframe thead th {
    text-align: right;

Following, let us initialize our model.

# local
net = SimpleCF(n_users, n_movies, factors=128, mean=0., std=.1)
objective = lambda pred, targ: targ - pred
optimizer = SGD(net.parameters(), lr=0.06)
device = 'cuda' if torch.cuda.is_available() else 'cpu'

model = Step(net, objective, optimizer, device=device)

Finally, let us get 20% of the data to fit the model for bootstrapping and create the Pytorch Dataset that we will use.

# local
pct = int(data_df_cleaned.shape[0] * .2)
bootstrapping_data = data_df_cleaned[:pct]

We will create a dataset from our Dataframe. We extract four elements:

  • The user code
  • The movie code
  • The rating
  • The preference
# local
features = ['user_code', 'movie_code', 'rating']
target = ['preference']

data_set = TensorDataset(torch.tensor(bootstrapping_data[features].values), 

Create the Pytorch DataLoader that we will use. Batch size should always be 1 for online training.

# local
data_loader = DataLoader(data_set, batch_size=512, shuffle=False)

Let us now use the batch_fit() method of the Step trainer to bootstrap our model.

# local
100%|██████████| 89/89 [00:01<00:00, 81.00it/s]

Then, to simulate streaming we get the remaining data and create a different data set.

# local
data_df_step = data_df_cleaned.drop(bootstrapping_data.index)
data_df_step = data_df_step.reset_index(drop=True)

# create the DataLoader
stream_data_set = TensorDataset(torch.tensor(data_df_step[features].values), 
stream_data_loader = DataLoader(stream_data_set, batch_size=1, shuffle=False)

Simulate the stream...

# local
k = 10 # we keep only the top 10 recommendations
recalls = []
known_users = []

with tqdm(total=len(stream_data_loader)) as pbar:
    for idx, (features, preferences) in enumerate(stream_data_loader):
        itr = idx + 1

        user = features[:, 0]
        item = features[:, 1]
        rtng = features[:, 2]
        pref = preferences

        if user.item() in known_users:
            predictions = model.predict(user, k)
            recall = recall_at_k(predictions.tolist(), item.tolist(), k)
            model.step(user, item, rtng, pref)
            model.step(user, item, rtng, pref)

100%|██████████| 181048/181048 [15:23<00:00, 195.94it/s]

Last but not least, we visualize the results of the recall@10 metric, using a moving average window of 5k elements.

# local
avgs = moving_avg(recalls, 5000)

plt.ylim(0., .1)


Finally, save the model's weights.

# local'artefacts', ''))


  1. Vinagre, J., Jorge, A. M., & Gama, J. (2014, July). Fast incremental matrix factorization for recommendation with positive-only feedback. In International Conference on User Modeling, Adaptation, and Personalization (pp. 459-470). Springer, Cham.
  2. Hu, Y., Koren, Y., & Volinsky, C. (2008, December). Collaborative filtering for implicit feedback datasets. In 2008 Eighth IEEE International Conference on Data Mining (pp. 263-272). Ieee.

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for cf-step, version 0.2.3
Filename, size File type Python version Upload date Hashes
Filename, size cf_step-0.2.3-py3-none-any.whl (13.4 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size cf_step-0.2.3.tar.gz (17.9 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page