Combine multiple csv files into one csv file with python (assuming only one line of header)

Introduction

This is my first post. I'm new to python. I would like to combine multiple csv files of the common format discharged from the equipment etc. into one file, and I will introduce the code I made by researching various things.

2020/04/06 postscript Effective when the header is one line and all the data is clean. Note that an error may occur if there is a row with no data.

environment

code

It is assumed that there are a lot of csv with the same format in the test folder in the execution location.

import pandas as pd
import glob

#Get the list of files specified by the path in list format.(Here, below the test file one level below)
csv_files = glob.glob('test/*.csv')

#Show list of files to read
for a in csv_files:
    print(a)

#Prepare a list to add the contents of the csv file
data_list = []

#Scan the list of files to read
for file in csv_files:
    data_list.append(pd.read_csv(file))

#Combine all lists in row direction
#axis=0:Join in row direction, sort
df = pd.concat(data_list, axis=0, sort=True)

df.to_csv("test/total1.csv",index=False)

What you are doing

--Get the list of files specified by glob --Read the file with pd.read_csv () (read with header = 0) --Add the contents of all lists with a for loop --Merge everything in the list with pd.concat --Finally, write to csv file with df.to_csv ()

Impressions

I was concerned about the processing of the header, but if you use pandas, it is convenient because it will join things with the same column only with data without worrying about it, and I was able to understand the behavior of pandas.

reference

Postscript (2020/04/06)

--In this case, the file with a seemingly different number of columns could not be read well. --Solved by reading line by line. (Separate article)

Recommended Posts

Combine multiple csv files into one csv file with python (assuming only one line of header)
Combine multiple python files into one python file
[Python & Unix] Combine multiple PDF files into one.
[Python] Combine multiple Excel sheets into one
Combine multiple Excel files loaded using pandas into one
Remove headings from multiple format CSV files with python
Consolidate a large number of CSV files in folders with python (data without header)
Download csv file with python
Create another version of Python conda environment with one command line
[Python] Write to csv file with Python
How to import CSV and TSV files into SQLite with Python
Output to csv file with Python
Handle Excel CSV files with Python
Read CSV file with python (Download & parse CSV file)
Combine python function arguments into one variable
Various ways to read the last line of a csv file in Python
Draw a line / scatter plot on the CSV file (2 columns) with python matplotlib
How to read a CSV file with Python 2/3
Python hand play (one line notation of if)
Use multiple versions of python environment with pyenv
[Python] Summary of S3 file operations with boto3
Convert multiple jpg files to one PDF file
Convert the character code of the file with Python3
Read line by line from a file with Python
Make JSON into CSV with Python from Splunk
Convert multiple proto files at once with python
Read wav files with only Python standard packages
Speed evaluation of CSV file output in Python
Example of reading and writing CSV with Python
Extract the table of image files with OneDrive & Python
Python text reading for multiple lines and one line
I tried to touch the CSV file with Python
Python that merges a lot of excel into one excel
How to output CSV of multi-line header with pandas
How to convert JSON file to CSV file with Python Pandas
[Memo] Load csv of s3 into pandas with boto3
Generate multiple HTML files at once by pouring JSON data into an HTML template with Python