Lua version Deep Learning from scratch Part 5.5 [Making pkl files available in Lua Torch]

Past articles

Lua version Deep Learning from scratch Part 1 [Implementation of Perceptron] Lua version Deep Learning from scratch 2 [Activation function] Lua version Deep Learning from scratch Part 3 [Implementation of 3-layer neural network] [Lua version Deep Learning from scratch 4 [Implementation of softmax function]] (http://qiita.com/Kazuki-Nakamae/items/20e53a02a8b759583d31) Lua version Deep Learning from scratch Part 5 [Display MNIST image]

Introduction

This time it has nothing to do with the main story, but I will show you how to make the data stored in the pkl file available in Lua.   STEP1: Convert pkl file to npz file

You can do it with the following script.

pkl2npz.py


#!/usr/local/bin/python3
# coding: utf-8

"""
Output the contents of the pkl file to the npz file.
"""
__author__ = "Kazuki Nakamae <[email protected]>"
__version__ = "0.00"
__date__ = "22 Jun 2017"

import sys
import numpy as np
import pickle

def pkl2npz(infn, outfn):
    """
    @function   pkl2npz();
Output the contents of the pkl file to the npz file.
    @param  {string} infn :Input file name
    @param  {string} outfn :Output file name
    """

    with open(infn, 'rb') as f:
            ndarr = pickle.load(f)
            np.savez(outfn, W1=ndarr['W1'],W2=ndarr['W2'],W3=ndarr['W3'],b1=ndarr['b1'],b2=ndarr['b2'],b3=ndarr['b3'])

if __name__ == '__main__':
    argvs = sys.argv
    argc = len(argvs)

    if (argc != 3):   # Checking input
        print("USAGE : python3 pkl2npz.py <INPUT_PKLFILE> <OUTPUT_NPZFILE>")
        quit()

    pkl2npz(str(argvs[1]),str(argvs[2]))
quit()

pkl2npz.Run py


$ python3 pkl2npz.py sample_weight.pkl sample_weight.npz

One caveat is that you need to know the elements in advance. In this case, it is a pkl file that stores the weights (W1, W2, W3) and biases (b1, b2, b3) of the three-layer NN.   STEP2: Read the npz file

Now let's load the created sample_weight.npz on Lua. No hassle. The following people have created a package (npy4th) for that purpose. htwaijry/npy4th

npy4th installation


$ git clone https://github.com/htwaijry/npy4th.git
$ cd npy4th
$ luarocks make

It's easy to use and you can load it just by inserting loadnpz ([filename]).

loadnpz()How to use


npy4th = require 'npy4th'

-- read a .npz file into a table
tbl = npy4th.loadnpz('sample_weight.npz')

print(tbl["W1"])

Output result


Columns 1 to 6
-7.4125e-03 -7.9044e-03 -1.3075e-02  1.8526e-02 -1.5346e-03 -8.7649e-03
-1.0297e-02 -1.6167e-02 -1.2284e-02 -1.7926e-02  3.3988e-03 -7.0708e-02
-1.3092e-02 -2.4475e-03 -1.7722e-02 -2.4240e-02 -2.2041e-02 -5.0149e-03
-1.0008e-02  1.9586e-02 -5.6170e-03  3.8307e-02 -5.2507e-02 -2.3568e-02
(Omitted)
 1.1210e-02  1.0272e-02
-1.2299e-02  2.4070e-02
 7.4309e-03 -4.0211e-02
[torch.FloatTensor of size 784x50]

It has been properly converted to Tensor type. I didn't introduce it this time, but it is possible to read npy files in the same way. However, although it does load, please note that there is no guarantee that it will be in the form of a matrix that can be used as is. What it looks like depends on what it was on the numpy side. If necessary, you will need to resize () etc. after this.   in conclusion

Torch is not popular in Japan, but I think it would be easier to use if numpy resources could be used in this way. that's all. Thank you very much.

Recommended Posts

Lua version Deep Learning from scratch Part 5.5 [Making pkl files available in Lua Torch]
Lua version Deep Learning from scratch Part 6 [Neural network inference processing]
"Deep Learning from scratch" Self-study memo (Part 12) Deep learning
Deep Learning from scratch
Deep Learning from scratch 1-3 chapters
An amateur stumbled in Deep Learning from scratch Note: Chapter 1
Making from scratch Deep Learning ❷ An amateur stumbled Note: Chapter 5
Making from scratch Deep Learning ❷ An amateur stumbled Note: Chapter 2
An amateur stumbled in Deep Learning from scratch Note: Chapter 3
An amateur stumbled in Deep Learning from scratch Note: Chapter 7
An amateur stumbled in Deep Learning from scratch Note: Chapter 5
Making from scratch Deep Learning ❷ An amateur stumbled Note: Chapter 7
Making from scratch Deep Learning ❷ An amateur stumbled Note: Chapter 1
Making from scratch Deep Learning ❷ An amateur stumbled Note: Chapter 4
An amateur stumbled in Deep Learning from scratch Note: Chapter 4
An amateur stumbled in Deep Learning from scratch Note: Chapter 2
I tried to implement Perceptron Part 1 [Deep Learning from scratch]
Making from scratch Deep Learning ❷ An amateur stumbled Note: Chapter 6
"Deep Learning from scratch" Self-study memo (Part 8) I drew the graph in Chapter 6 with matplotlib
Deep Learning memos made from scratch
Why ModuleNotFoundError: No module named'dataset.mnist' appears in "Deep Learning from scratch".
[Learning memo] Deep Learning made from scratch [Chapter 7]
Deep learning from scratch (forward propagation edition)
Deep learning / Deep learning from scratch 2-Try moving GRU
Deep learning / Deep learning made from scratch Chapter 6 Memo
[Learning memo] Deep Learning made from scratch [Chapter 5]
Deep learning / Deep learning made from scratch Chapter 7 Memo
[Windows 10] "Deep Learning from scratch" environment construction
Learning record of reading "Deep Learning from scratch"
[Deep Learning from scratch] About hyperparameter optimization
[Learning memo] Deep Learning made from scratch [~ Chapter 4]
[Python] [Natural language processing] I tried Deep Learning ❷ made from scratch in Japanese ①
[For beginners] After all, what is written in Deep Learning made from scratch?
"Deep Learning from scratch" self-study memo (unreadable glossary)
"Deep Learning from scratch" Self-study memo (9) MultiLayerNet class
Good book "Deep Learning from scratch" on GitHub
Deep Learning from scratch Chapter 2 Perceptron (reading memo)
[Learning memo] Deep Learning from scratch ~ Implementation of Dropout ~
Python vs Ruby "Deep Learning from scratch" Summary
"Deep Learning from scratch" Self-study memo (10) MultiLayerNet class
"Deep Learning from scratch" Self-study memo (No. 11) CNN
Dare to learn with Ruby "Deep Learning from scratch" Importing pickle files from forbidden PyCall
"Deep Learning from scratch" Self-study memo (No. 19) Data Augmentation
"Deep Learning from scratch 2" Self-study memo (No. 21) Chapters 3 and 4
Application of Deep Learning 2 made from scratch Spam filter
Deep Learning from the mathematical basics Part 2 (during attendance)
Study method for learning machine learning from scratch (March 2020 version)
[Deep Learning from scratch] I tried to explain Dropout
[Deep Learning from scratch] I tried to explain the gradient confirmation in an easy-to-understand manner.
"Deep Learning from scratch" Self-study memo (No. 14) Run the program in Chapter 4 on Google Colaboratory
[Deep Learning from scratch] Implement backpropagation processing in neural network by error back propagation method
[Deep Learning from scratch] Implementation of Momentum method and AdaGrad method
[Part 4] Use Deep Learning to forecast the weather from weather images
[Part 3] Use Deep Learning to forecast the weather from weather images
[Part 2] Use Deep Learning to forecast the weather from weather images
"Deep Learning from scratch" self-study memo (No. 19-2) Data Augmentation continued
"Deep Learning from scratch" self-study memo (No. 15) TensorFlow beginner tutorial
[Deep Learning from scratch] About the layers required to implement backpropagation processing in a neural network