Deep Learning Model Lightening Library Distiller

Introduction

What is Distiller

Distiller is a PyTorch-based library created by intel with algorithms to reduce the weight of Deep Learning models. The main examples of weight reduction of the model are Quantization, Pruning, Distillation, etc., and Distiller is easy to use. In addition, the tutorial even included a function that allows you to check the learning status in solidarity with TensorBoard (thanks).

Click here for a detailed site about model weight reduction https://laboro.ai/column/%E3%83%87%E3%82%A3%E3%83%BC%E3%83%97%E3%83%A9%E3%83%BC%E3%83%8B%E3%83%B3%E3%82%B0%E3%82%92%E8%BB%BD%E9%87%8F%E5%8C%96%E3%81%99%E3%82%8B%E3%83%A2%E3%83%87%E3%83%AB%E5%9C%A7%E7%B8%AE/

Environmental development

$ git clone https://github.com/NervanaSystems/distiller.git
$ cd distiller
$ pip install -r requirements.txt
$ pip install -e .
$ python
>>> import distiller
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/mnt/PytorchIntro/distiller/distiller/__init__.py", line 20, in <module>
    from .config import file_config, dict_config, config_component_from_file_by_class

...

  File "/root/local/python-3.7.1/lib/python3.7/site-packages/git/exc.py", line 9, in <module>
    from git.compat import UnicodeMixin, safe_decode, string_types
  File "/root/local/python-3.7.1/lib/python3.7/site-packages/git/compat.py", line 16, in <module>
    from gitdb.utils.compat import (
ModuleNotFoundError: No module named 'gitdb.utils.compat'

In my case, when I tried to import the Distiller added to the library, I got an error related to the git library, so I downgraded the bad gitdb2 and it was fixed. (My installed version is 4.0.2)

$ pip uninstall gitdb2
$ pip install gitdb2==2.0.6

Confirmation

$ cd distiller/examples/classifier_compression/
$ python3 compress_classifier.py --arch simplenet_cifar ../../../data.cifar10 -p 30 -j=1 --lr=0.01

--------------------------------------------------------
Logging to TensorBoard - remember to execute the server:
> tensorboard --logdir='./logs'

=> created a simplenet_cifar model with the cifar10 dataset
Downloading https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz to ../../../data.cifar10/cifar-10-python.tar.gz
 99%|█████████████████████████████████████████████████████████████████████████████▌| 169582592/170498071 [00:18<00:00, 11451969.71it/s]Extracting ../../../data.cifar10/cifar-10-python.tar.gz to ../../../data.cifar10
Files already downloaded and verified
Dataset sizes:
        training=45000
        validation=5000
        test=10000


Training epoch: 45000 samples (256 per mini-batch)
170500096it [00:30, 11451969.71it/s]                                                                                                   Epoch: [0][   30/  176]    Overall Loss 2.303411    Objective Loss 2.303411    Top1 10.299479    Top5 50.104167    LR 0.010000    Time 0.038285
Epoch: [0][   60/  176]    Overall Loss 2.301507    Objective Loss 2.301507    Top1 10.774740    Top5 51.328125    LR 0.010000    Time 0.037495
Epoch: [0][   90/  176]    Overall Loss 2.299031    Objective Loss 2.299031    Top1 12.335069    Top5 54.973958    LR 0.010000    Time 0.037465
Epoch: [0][  120/  176]    Overall Loss 2.293749    Objective Loss 2.293749    Top1 13.424479    Top5 57.542318    LR 0.010000    Time 0.037429
Epoch: [0][  150/  176]    Overall Loss 2.278429    Objective Loss 2.278429    Top1 14.692708    Top5 59.864583    LR 0.010000    Time 0.037407

Parameters:
+----+---------------------+---------------+---------------+----------------+------------+------------+----------+----------+----------+------------+---------+----------+------------+
|    | Name                | Shape         |   NNZ (dense) |   NNZ (sparse) |   Cols (%) |   Rows (%) |   Ch (%) |   2D (%) |   3D (%) |   Fine (%) |     Std |     Mean |   Abs-Mean |
|----+---------------------+---------------+---------------+----------------+------------+------------+----------+----------+----------+------------+---------+----------+------------|
|  0 | module.conv1.weight | (6, 3, 5, 5)  |           450 |            450 |    0.00000 |    0.00000 |  0.00000 |  0.00000 |  0.00000 |    0.00000 | 0.07800 | -0.01404 |    0.06724 |
|  1 | module.conv2.weight | (16, 6, 5, 5) |          2400 |           2400 |    0.00000 |    0.00000 |  0.00000 |  0.00000 |  0.00000 |    0.00000 | 0.04952 |  0.00678 |    0.04246 |
|  2 | module.fc1.weight   | (120, 400)    |         48000 |          48000 |    0.00000 |    0.00000 |  0.00000 |  0.00000 |  0.00000 |    0.00000 | 0.02906 |  0.00082 |    0.02511 |
|  3 | module.fc2.weight   | (84, 120)     |         10080 |          10080 |    0.00000 |    0.00000 |  0.00000 |  0.00000 |  0.00000 |    0.00000 | 0.05328 |  0.00084 |    0.04607 |
|  4 | module.fc3.weight   | (10, 84)      |           840 |            840 |    0.00000 |    0.00000 |  0.00000 |  0.00000 |  0.00000 |    0.00000 | 0.06967 | -0.00275 |    0.06040 |
|  5 | Total sparsity:     | -             |         61770 |          61770 |    0.00000 |    0.00000 |  0.00000 |  0.00000 |  0.00000 |    0.00000 | 0.00000 |  0.00000 |    0.00000 |
+----+---------------------+---------------+---------------+----------------+------------+------------+----------+----------+----------+------------+---------+----------+------------+
Total sparsity: 0.00

--- validate (epoch=0)-----------
5000 samples (256 per mini-batch)
==> Top1: 25.240    Top5: 75.520    Loss: 2.060

==> Best [Top1: 25.240   Top5: 75.520   Sparsity:0.00   NNZ-Params: 61770 on epoch: 0]
Saving checkpoint to: logs/2020.05.02-235616/checkpoint.pth.tar

...

For the time being, I'm relieved because it moved ε- (´∀ ` *) Hot I will add it as soon as I find something.

Reference site

https://github.com/NervanaSystems/distiller

Recommended Posts

Deep Learning Model Lightening Library Distiller
Microsoft's Deep Learning Library "CNTK" Tutorial
Deep learning image recognition 2 model implementation
Deep Learning
(python) Deep Learning Library Chainer Basics Basics
Image recognition model using deep learning in 2016
Deep learning image recognition 3 after model creation
Deep Learning Memorandum
Start Deep learning
I tried using the trained model VGG16 of the deep learning library Keras
Python Deep Learning
Deep learning × Python
Implementation of Deep Learning model for image recognition
Why what? Deep Learning Scientific Calculation Library Numpy Edition
First Deep Learning ~ Struggle ~
I installed and used the Deep Learning library Chainer
Python: Deep Learning Practices
Deep learning / activation functions
Deep Learning from scratch
Count the number of parameters in the deep learning model
Deep learning 1 Practice of deep learning
Deep learning / cross entropy
First Deep Learning ~ Preparation ~
First Deep Learning ~ Solution ~
[AI] Deep Metric Learning
I tried deep learning
Machine learning library dlib
DNN (Deep Learning) Library: Comparison of chainer and TensorFlow (1)
Python: Deep Learning Tuning
Deep learning large-scale technology
Machine learning library Shogun
DEEP PROBABILISTIC PROGRAMMING --- "Deep Learning + Bayes" Library --- Introduction of Edward
Deep learning / softmax function
Sine wave prediction using RNN in deep learning library Keras
I tried hosting a TensorFlow deep learning model using TensorFlow Serving
I tried to divide with a deep learning language model
Machine learning model considering maintainability
Deep Learning from scratch 1-3 chapters
Try deep learning with TensorFlow
Deep Learning Gaiden ~ GPU Programming ~
<Course> Deep Learning: Day2 CNN
Deep learning image recognition 1 theory
Deep running 2 Tuning of deep learning
Deep learning / LSTM scratch code
Rabbit Challenge Deep Learning 1Day
<Course> Deep Learning: Day1 NN
Deep Kernel Learning with Pyro
Try Deep Learning with FPGA
Deep learning for compound formation?
Introducing Udacity Deep Learning Nanodegree
Subjects> Deep Learning: Day3 RNN
Introduction to Deep Learning ~ Learning Rules ~
Rabbit Challenge Deep Learning 2Day
Learning model creation, learning and reasoning
Deep Reinforcement Learning 1 Introduction to Reinforcement Learning
Deep reinforcement learning 2 Implementation of reinforcement learning
Generate Pokemon with Deep Learning
Introduction to Deep Learning ~ Backpropagation ~
[Windows] Library Keras course where you can try Deep Learning immediately-Part 1