How to write faster when using numpy like deque

Thing you want to do

If you want to do something like a deque with a numpy.ndarray, is it faster to simply use np.append (x, y) and np.delete (x, 0), or use a deque and then a deque? I will verify that.

Suspicious code that seems to be late


x = np.append(x, 1)
x = np.delete(x, 0)

I added it at the end and deleted the beginning.

numpy seems to secure a fixed length area when creating an array, so changing the length seems to be somewhat slow.

This verification seems to be about FIFO [First-In First-Out / First-In First-Out / First-in First-Out], but it is assumed that the whole is read without reading the beginning. In other words, it is a verification when used like a buffer.

This time I want to use numpy.mean () and numpy.std () every time I add it, so I limited the conditions so that it will be numpy.ndarray at the end. If you just want to create an array, it's faster to use the deque as it is, but then you can't use numpy's mean () or std () as it is.

This time, assuming that you use the numpy function for each loop, I would like to compare it including the conversion cost.

Alternative candidate

Use numpy.roll ()

x = np.roll(x, -1)
x[-1] = 1

The simplest way to do this is to shift it by one like a deque and substitute it at the back.

append and delete

x = np.append(x, 1)
x = np.delete(x, 0)

Just what you want to do. This seems to be late, so I am verifying this time.

Copy one before and substitute at the end

x[0:-1] = x[1:]
x[-1] = 1

Slicing seems to be fast, so it looks like it's doing something similar to roll, but it's subtly different.

Manipulate with deque and then convert to numpy.ndarray

d.append(1)
x = np.array(d)

Where d is the deque generated by d = deque (maxlen). If maxlen is specified, the beginning will disappear without permission. Deque is overwhelmingly fast if you simply append, but this time you need to convert it to numpy.array each time.

verification code

Then it is verification which is the fastest. Click here for the code used for verification

import time
import numpy as np
from collections import deque

xlen = 1000

n = 100000

x = np.zeros(xlen)

s = time.time()
for i in range(n):
    x = np.roll(x, -1)
    x[-1] = 1

print(time.time() - s)

s = time.time()
for i in range(n):
    x = np.append(x, 1)
    x = np.delete(x, 0)

print(time.time() - s)

s = time.time()
for i in range(n):
    x[0:-1] = x[1:]
    x[-1] = 1

print(time.time() - s)

#Declaration and initialization of deque
d = deque(maxlen=xlen)
for i in range(xlen):
	d.append(1)

s = time.time()
for i in range(n):
    d.append(1)
    x = np.array(d)

print(time.time() - s)

inspection result

I ran it with python3.8.

Method xlen=In the case of 100 xlen=In the case of 10000
numpy.roll()use 2.58s 3.28s
append and delete 1.79s 2.78s
Copy one before and substitute at the end 0.100s 0.366s
Manipulate with deque and then numpy.Convert to ndarray 1.52s 88.0s

The fastest result was ** copying in slices and assigning at the end **. The alleged code is also unlikely to be the wrong choice. The deque may also be good because np.array () is not so slow when the array is short. If it gets longer, it will take a tremendous amount of time for np.array (), which is dangerous.

I hope it will be helpful for you.

Recommended Posts

How to write faster when using numpy like deque
How to speed up scikit-learn like conda Numpy
How to deal with SessionNotCreatedException when using Selenium
How to make Python faster for beginners [numpy]
How to use numpy
How to write a GUI using the maya command
How to exit when using Python in Terminal (Mac)
[Introduction to Python] How to write repetitive statements using for statements
XPath Basics (2) -How to write XPath
[Introduction to Python] How to write conditional branches using if statements
How to resolve CSRF Protection when using AngularJS with Django
How doi may be useful when asking how to write code?
How to install python using anaconda
How to write a Python class
How to write soberly in pandas
How to display formulas in latex when using sympy (> = 1.4) in Google Colaboratory
Flask reuse How to write html
How to add sudo when debugging
How to write Docker base image
How to write Django1.9 environment-independent wsgi.py
Notes on how to write requirements.txt
How to deal with OAuth2 error when using Google APIs from Python
How to write a string when there are multiple lines in python
Qiita (1) How to write a code name
How to set optuna (how to write search space)
How to draw a graph using Matplotlib
How to get IP when Tornado + nginx
How to set up SVM using Optuna
How to write Python document comments (Docstrings)
How to install a package using a repository
How to set xg boost using Optuna
How to use "deque" for Python data
How to install NumPy on Raspberry Pi
How to write this process in Perl?
How to write Ruby to_s in Python
Summary of how to write AWS Lambda
How to write pydoc and multi-line comments
How to take multiple arguments when doing parallel processing using multiprocessing in python
How to write urlfetch unittest on GAE / P
How to start Python (Flask) when EC2 starts
How to deal with errors when hitting pip ②
How to code a drone using image recognition
How to set up Random forest using Optuna
[Django] How to resolve errors when installing mysqlclient
How to install pip, numpy in Autodesk MAYA
Answer to "Offline real-time how to write F02 problem"
How to write regular expression patterns in Linux
How to use Tweepy ~ Part 2 ~ [Follow, like, etc.]
How to fix when Terminal input becomes abnormal
How to get article data using Qiita API
How to write a ShellScript Bash for statement
Answer to "Offline Real-time How to Write F01 Problem"
How to set up Random forest using Optuna
How to search HTML data using Beautiful Soup
Answer to "Offline Real-time How to Write E13 Problem"
How to write async and await in Vue.js
How to upload to a shared drive using pydrive
How to uninstall a module installed using setup.py
How to write a ShellScript bash case statement
How to deal with "Type Error: No matching signature found" error when using pandas fillna