Make apache log csv with python

install apache_log_parser

pip install apache_log_parser

Create the code below

It's almost below, but some bugfixes https://github.com/isonet/apache-log-to-csv

alog2csv.py


#!/usr/bin/env python
# coding: utf-8

import csv
import apache_log_parser
import argparse

class bcolors:
    HEADER = '\033[95m'
    OKBLUE = '\033[94m'
    OKGREEN = '\033[92m'
    WARNING = '\033[93m'
    FAIL = '\033[91m'
    ENDC = '\033[0m'


def main(**kwargs):

    print('Converting, please wait...')

    line_parser = apache_log_parser.make_parser(kwargs['format'])
    header = True

    with open(kwargs['input'], 'rb') as inFile, open(kwargs['output'], 'w') as outFile:

        lines = inFile.readlines()
        writer = csv.writer(outFile, delimiter=',')

        for line in lines:
            try:
                log_line_data = line_parser(line)
            except apache_log_parser.LineDoesntMatchException as ex:
                print(bcolors.FAIL + 'The format specified does not match the log file. Aborting...' + bcolors.ENDC)
                print('Line: ' + ex.log_line + 'RegEx: ' + ex.regex)
                exit()

            if header:
                writer.writerow(list(log_line_data.keys()))
                header = False
            else:
                writer.writerow(list(log_line_data.values()))

    print(bcolors.OKGREEN + 'Conversion finished.' + bcolors.ENDC)


if __name__ == '__main__':
    parser = argparse.ArgumentParser(description='Convert Apache logs to csv', version='%(prog)s 1.0')
    parser.add_argument('format', type=str, help='Apache log format (see http://httpd.apache.org/docs/2.2/logs.html)')
    parser.add_argument('input', type=str, help='Input log file ex. /var/log/apache/access.log')
    parser.add_argument('output', type=str, help='Output csv file ex. ~/accesslog.csv')
    args = parser.parse_args()
    main(**vars(args))

Execute

ʻAlog2csv.py format source output`

python alog2csv.py "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" access_log_20200430 output.csv

Recommended Posts

Make apache log csv with python
Csv tinkering with python
Make JSON into CSV with Python from Splunk
Python3 + Django ~ Mac ~ with Apache
Make Puyo Puyo AI with Python
Make a fortune with Python
Write to csv with Python
Download csv file with python
Make a CSV formatting tool with Python Pandas PyInstaller
Make Echolalia LINEbot with Python + heroku
[Python] Write to csv file with Python
Let's make a GUI with python.
Output to csv file with Python
python log
Unit test log output with python
Make a recommender system with python
Handle Excel CSV files with Python
Let's make a graph with python! !!
Reading and writing CSV with Python
Until you run python with apache
Read CSV file with python (Download & parse CSV file)
Output Python log to console with GAE
Fractal to make and play with Python
Let's make a voice slowly with Python
Make pypy submission easier with atcoder-cli (python)
CentOS 6.4 with Python 2.7.3 with Apache with mod_wsgi and Django
[Python] Let's make matplotlib compatible with Japanese
[Part1] Scraping with Python → Organize to csv!
Let's make a web framework with Python! (1)
Make a desktop app with Python with Electron
Let's make a Twitter Bot with Python!
Let's make a web framework with Python! (2)
FizzBuzz with Python3
Scraping with Python
Statistics with python
Scraping with Python
Python with Go
Twilio with Python
Integrate with Python
Play with 2016-Python
AES256 with python
Tested with Python
python starts with ()
with syntax (Python)
Bingo with python
Zundokokiyoshi with python
Csv in python
Excel with Python
Microcomputer with Python
Cast with python
Create "operation log" CSV formatting tool in 5 days with Python Pandas PyInstaller
How to read a CSV file with Python 2/3
Csv output from Google search with [Python]! 【Easy】
Scraping tabelog with python and outputting to CSV
Make Python scripts into Windows-executable .exes with Pyinstaller
Make a Twitter trend bot with heroku + Python
[Python] Make a game with Pyxel-Use an editor-
Read Python csv data with Pandas ⇒ Graph with Matplotlib
Read JSON with Python and output as CSV
Python log is not output with docker-compose up
I want to make a game with Python