How to create large files at high speed

Introduction

What command do you use to create an empty file? In my case, I used the touch command if the file size was 0, and the dd command if I needed a file of a certain size, as shown below.

However, apart from the dd command, I learned that there is a command called fallocate as a way to create a file of a certain size, so I compared the performance of both.

[nkojima@akagi sample_files]$ touch sample.txt
[nkojima@akagi sample_files]$ dd if=/dev/zero of=./sample2.txt bs=1K count=1024
1024+0 Record input
1024+0 record output
1048576 bytes(1.0 MB)Copied, 0.00580267 seconds, 181 MB/Seconds
[nkojima@akagi sample_files]$ ls -lah
Total 1.0M
drwxrwxr-x 2 nkojima nkojima 43 June 21 12:18 .
drwx------.5 nkojima nkojima 175 June 21 12:17 ..
-rw-rw-r--1 nkojima nkojima 0 June 21 12:17 sample.txt
-rw-rw-r--  1 nkojima nkojima 1.0M June 21 12:18 sample2.txt

Environment used for testing

Test method

[root@akagi ~]# date; dd if=/dev/zero of=dd.txt bs=1M count=100; date;
Sunday, June 21, 2020 12:48:28 JST
100+0 Record input
100+0 record output
104857600 bytes(105 MB)Copied, 0.0362178 seconds, 2.9 GB/Seconds
Sunday, June 21, 2020 12:48:29 JST
[root@akagi ~]# date; fallocate -l 100m ./fallocate.txt; date;
Sunday, June 21, 2020 12:48:52 JST
Sunday, June 21, 2020 12:48:52 JST

test results

file size Processing time of dd command(Seconds) Processing time of fallocate command(Seconds)
100MB <1 <1
500MB <1 <1
1GB 1 <1
2GB 2 <1
4GB 4 <1
8GB 7 <1
16GB 15 <1

Related article

Recommended Posts

How to create large files at high speed
How to create * .spec files for pyinstaller.
How to find large files on Linux
[Python] How to get divisors of natural numbers at high speed
One-liner to create a large number of test files at once on Linux
[Flask + Keras] How to infer multiple models at high speed on the server
PostgreSQL-For those who want to INSERT at high speed
How to scrape at speed per second with Python Selenium
How to create a Conda package
How to create your own Transform
How to create an email user
How to create a virtual bridge
How to create / delete symbolic links
How to create a Dockerfile (basic)
How to speed up Python calculations
How to create a config file
How to create a large amount of test data in MySQL? ??
A script that downloads AWS RDS log files at high speed
How to create a clone from Github
[Linux] How to subdivide files and folders
How to speed up instantiation of BeautifulSoup
How to create a git clone folder
How to create an NVIDIA Docker environment
How to create a repository from media
How to read CSV files in Pandas
[Python] How to open two or more files at the same time
How to automatically upload .gpx files to Strava
How to save all Instagram photos at once
How to create a Python virtual environment (venv)
How to hold a competition at Coda Lab
How to create a function object from a string
Summary of how to import files in Python 3
How to speed up scikit-learn like conda Numpy
How to create explanatory variables and objective functions
How to create a JSON file in Python
How to create an OCF compliant resource agent
How to check / extract files in RPM package
How to create data to put in CNN (Chainer)
How to measure line speed from the terminal
How to create a shortcut command for LINUX
How to get the files in the [Python] folder
[Note] How to create a Ruby development environment
How to create a Kivy 1-line input box
How to create a multi-platform app with kivy
How to create a Rest Api in Django
[Note] How to create a Mac development environment
How to create random numbers with NumPy's random module
Tutorial to infer the model learned in Tensorflow with C ++/OpenVINO at high speed
Example of how to aggregate a large amount of time series data using Python at a reasonable speed in a small memory environment
Read the Python-Markdown source: How to create a parser
How to create an article from the command line
Beginners try to convert Word files to PDF at once
[Blender x Python] How to create an original object
How to create a submenu with the [Blender] plugin
Create machine learning projects at explosive speed using templates
How to quickly create array sample data during coding
[Go] How to create a custom error for Sentry
How to load files in Google Drive with Google Colaboratory
How to use variables in systemd Unit definition files
How to download files from Selenium in Python in Chrome
How to create a local repository for Linux OS