Small speedup when using pytorch

Introduction

When training a model using pytorch etc., it may take several hours to train, or even days if anything. In such a case, the effect of a constant multiple of the execution time becomes large. For example, even if the constant multiple of the amount of calculation changes from 2 to 1, the learning that took 2 days can be shortened to 1 day. Competitive programming is mainly about the order of computational complexity, so the atmosphere is different in that respect. This time, I will introduce a simple example that can be used often.

Code example

You'll often see the code that converts a list of np.arrays to a Tensor type. This time, it will be faster to use in such situations.

import time
import numpy as np
import torch

a = [np.ones([100]) for _ in range(100)]

# 8.44s
start = time.time()
for _ in range(10000):
    b = torch.tensor(a)
print(time.time()-start)

# 0.44s
start = time.time()
for _ in range(10000):
    b = torch.tensor(np.array(a))
print(time.time()-start)

# 0,32s
start = time.time()
for _ in range(10000):
    b = torch.from_numpy(np.array(a))
print(time.time()-start)

There is almost no difference between the above three codes, but there is a big difference in execution time. I don't know why, but when converting the list of np.array to Tensor type, rather than directly using torch.tensor (), use np.array () once to convert the whole np. The execution time converted to array will be shorter.

I think that you can notice the difference relatively visibly when learning about images, which often converts a large list to Tensor type. By the way, even if you use torch.as_tensor, you can do the same processing as if you use torch.tensor, and it is a little slow if you use it alone.

I personally use it with the following code

def convert2tensor(x, device):
    return torch.tensor(np.array(x, dtype=np.float32), device=device)

in conclusion

Life is finite, so don't spend too much time learning the model.

Recommended Posts

Small speedup when using pytorch
Summary when using Fabric
Precautions when using Chainer
I got an error when using Tensorboard with Pytorch
(Personal) points when using ctypes
Environment variables when using Tkinter
When using optparse with iPython
DEBUG settings when using Django
Notes on optimization using Pytorch
When using if and when using while
File structure when using serverless-python-requirements
Use configparser when using API
Variable scope when using internal functions
Proxy measures when using WEB API
Precautions when using pit in Python
Precautions when using codecs and pandas
Precautions when using the urllib.parse.quote function
[Python] Be careful when using print
Precautions when using phantomjs from python
[PyTorch] Japanese sentence generation using Transformer
When using MeCab with virtualenv python
[VS Code] ~ Tips when using python ~
When using regular expressions in Python