HashDL 1.0.2

Hash-based Deep Learning

Table of Contents

1. Overview
2. Install
   1. Requirement
   2. Install from PyPI
   3. Install from Source
3. Features

Overview

This repository is non-official third-paty re-implementation of SLIDE¹.

We provide

• Python package
• Hash based Deep Learning
• Parallel computing based on C++17 parallel STL

We don't provide

• Explicit CPU optimized code like AVX (We just rely on compiler optimization)
• Compiled binary (You need to compile by yourself)

Install

There are two options, "Install from PyPI" and "Install from Source". For ordinary user, "Install from PyPI" is recommended.

For both case, sufficient C++ compiler is neccessary.

Requirement

• Recent C++ compiler with parallel STL algorithm support
  • GCC 9.1 or newer together with Intel TBB
• Python 3

Requirements can be installed on Docker image gcc:10.

# On local machine
docker run -it gcc:10 bash

# On gcc:10 image
apt update && apt install -y python3-pip libtbb-dev

Install from PyPI

pip install HashDL

Install from Source

git clone https://gitlab.com/ymd_h/hashdl.git HashDL
cd HashDL
pip install .

Features

• Neural Network
  • hash-based sparse dense layer
• Activation
  • ReLU
  • linear (no activation)
  • sigmoid
• Optimizer
  • SGD
  • Adam²
• Weight Initializer
  • constant
  • Gauss distribution
• Hash for similarity
  • WTA
  • DWTA³
• Scheduler for hash update
  • constant
  • exponential decay

In the current architecture, CNN is impossible.

Footnotes

¹ B. Chen et al., "SLIDE : In Defense of Smart Algorithms over Hardware Acceleration for Large-Scale Deep Learning Systems", MLSys 2020 (arXiv, code)

² D. P. Kingma and J. Ba, "Adam: A Method for Stochastic Optimization", ICLR (2015) (arXiv)

³ B. Chen et al., "Densified Winner Take All (WTA) Hashing for Sparse Datasets", Uncertainty in artificial intelligence (2018)