Import audit.log into Splunk and check the behavior when Splunk is started for the first time

Introduction

When you start Splunk Enterprise for the first time, various processing messages are output to the screen.

Example of first startup message (click to expand)
# /opt/splunk/bin/splunk start --accept-license --seed-passwd password

This appears to be your first time running this version of Splunk.
Copying '/opt/splunk/etc/openldap/ldap.conf.default' to '/opt/splunk/etc/openldap/ldap.conf'.
Generating RSA private key, 2048 bit long modulus
.................................+++++
...........................+++++
e is 65537 (0x10001)
writing RSA key

Generating RSA private key, 2048 bit long modulus
.............................................................................+++++
.................................................................+++++
e is 65537 (0x10001)
writing RSA key

Moving '/opt/splunk/share/splunk/search_mrsparkle/modules.new' to '/opt/splunk/share/splunk/search_mrsparkle/modules'.

Splunk> See your world.  Maybe wish you hadn't.

Checking prerequisites...
	Checking http port [8000]: open
	Checking mgmt port [8089]: open
	Checking appserver port [127.0.0.1:8065]: open
	Checking kvstore port [8191]: open
	Checking configuration... Done.
		Creating: /opt/splunk/var/lib/splunk
		Creating: /opt/splunk/var/run/splunk
		Creating: /opt/splunk/var/run/splunk/appserver/i18n
		Creating: /opt/splunk/var/run/splunk/appserver/modules/static/css
		Creating: /opt/splunk/var/run/splunk/upload
		Creating: /opt/splunk/var/run/splunk/search_telemetry
		Creating: /opt/splunk/var/spool/splunk
		Creating: /opt/splunk/var/spool/dirmoncache
		Creating: /opt/splunk/var/lib/splunk/authDb
		Creating: /opt/splunk/var/lib/splunk/hashDb
New certs have been generated in '/opt/splunk/etc/auth'.
	Checking critical directories...	Done
	Checking indexes...
		Validated: _audit _internal _introspection _metrics _telemetry _thefishbucket history main summary
	Done
	Checking filesystem compatibility...  Done
	Checking conf files for problems...
	Done
	Checking default conf files for edits...
	Validating installed files against hashes from '/opt/splunk/splunk-8.0.2.1-f002026bad55-linux-2.6-x86_64-manifest'
	All installed files intact.
	Done
All preliminary checks passed.

Starting splunk server daemon (splunkd)...  
Generating a RSA private key
...............................................+++++
...............+++++
writing new private key to 'privKeySecure.pem'
-----
Signature ok
subject=/CN=aio/O=SplunkUser
Getting CA Private Key
writing RSA key
Done
 [  OK  ]

Waiting for web server at http://127.0.0.1:8000 to be available.... Done


If you get stuck, we're here to help.  
Look for answers here: http://docs.splunk.com

The Splunk web interface is at http://aio:8000

Judging from the content of the message, it seems that you are creating some file. So what will be created this time at the first startup? I investigated that with Splunk.

Environment at the time of survey

OS: Amazon Linux 2 (4.14.171-136.231.amzn2.x86_64) Splunk version: 8.0.2.1 Splunk installation path ($ SPLUNK_HOME): / opt / splunk

1. Splunk Enterprise installation

Install Splunk. I will omit the detailed procedure, but this time I will install it with the tgz file.

Splunk Enterprise/opt/Install in directory


tar fvxz /tmp/splunk-8.0.2.1-f002026bad55-Linux-x86_64.tgz -C /opt/

2. Audit log output settings

Change the OS settings to output the change information of the Splunk setting directory to the audit log.

Set up an audit log(Temporary change)


#Add an audited directory
auditctl -w /opt/splunk/etc/ -p wa -k etc_changes
#Confirm that the settings have been added
auditctl -l
#OK if the command result is output."No rules"Is displayed, please check if the settings are correct.

3. Start Splunk Enterprise

Start Splunk Enterprise.

Start Splunk Enterprise


/opt/splunk/bin/splunk start --accept-license --seed-passwd password

4. Check the audit log output

Check if the change information of the audited directory set in step 2 is output to the log.

Check if the change information of the audited directory is output to the log


grep /opt/splunk/etc/ /var/log/audit/audit.log | tail
#It is OK if one or more logs are output.

5. Audit log capture

Set audit logs to be permanently populated in Splunk.

Set to capture audit logs permanently


/opt/splunk/bin/splunk add monitor /var/log/audit/audit.log -auth admin:password

6. Confirmation of import result

After completing the above settings, the audit log should have been imported to Splunk, so log in to Splunk Enterprise and check the import result with the search statement.

Search audit logs


sourcetype="linux_audit" type="PATH" nametype!="PARENT"
| rex field=name "^(?<directory>.*\/)?(?<file>.*)"
| eval directory=if(len(mode)==6,directory+file,directory),file=if(len(mode)==6,"",file)
| table _time,msg,nametype,mode,directory,file
| sort msg

Explanation of search text

Line 1 ... Searching for ** events in the audit log where the type field is PATH and the nametype field is not PARENT **. Lines 2 and 3 ... The directory name and file name are set in the new fields (directory, file) from the path information recorded in the name field. 4th line: Only the fields required for visualization are formatted and displayed from the search results. 5th line: Since the time stamp and ID information are recorded in the msg field, they are sorted and displayed in the order of the msg field.

7. Result

Here are some excerpts of the results of the search in this environment.

Click to expand and display. ![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/170600/82fcdf32-b546-403d-597d-60a2544235c8.png) ![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/170600/9fce088f-930a-1539-eabb-1e0326bb064f.png) ![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/170600/1c040d09-7e04-7ccc-93b7-178f2b2733e4.png) ![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/170600/e5f649dd-6183-aabf-d278-d7ec10c424fa.png)

8. Analysis

Analyzing the search results, we found that Splunk was running the file / directory creation process in the / opt / splunk / etc / directory in the following order when it was first started.

Seq File Remarks
1 /opt/splunk/etc/splunk-launch.conf splunk-lanch.conf.Created by copying from default.
2 /opt/splunk/etc/licenses/download-trial/enttrial.lic Trial license file./opt/splunk/splunk-enttrial.Created by copying from lic.
3 /opt/splunk/etc/auth/splunk.secret Confidential information stored in Splunk's configuration file(Password etc.)Encryption/The key file used for decryption.
4 /opt/splunk/etc/system/local/user-seed.conf Initial admin user and password configuration file(Version 7.1 or later)In subsequent processing/opt/splunk/etc/It will be deleted when the passwd is created.
5 /opt/splunk/etc/system/local/inputs.conf The default value is set when the host field is not specified in the data capture settings.
6 /opt/splunk/etc/system/local/server.conf
7 /opt/splunk/etc/system/local/migration.conf
8 /opt/splunk/etc/openldap/ldap.conf ldap.conf.Created by copying from default.
9 /opt/splunk/etc/auth/audit/private.pem
10 /opt/splunk/etc/auth/audit/public.pem
11 /opt/splunk/etc/auth/distServerKeys/private.pem A private key used for distributed search communication.
12 /opt/splunk/etc/auth/distServerKeys/trusted.pem Public key used for distributed search communication.
13 /opt/splunk/etc/auth/ca.pem Root CA certificate. ca.pem.Created by copying from default.
14 /opt/splunk/etc/auth/cacert.pem Root CA certificate. cacert.pem.Created by copying from default. The content of the certificate is ca.Same as pem, but the private key with passphrase is added in the file.
15 /opt/splunk/etc/myinstall/splunkd.xml
16 /opt/splunk/etc/users/users.ini
17 /opt/splunk/etc/auth/serverkey.pem Of SplunkInternal communication encryptionThe private key required to create the server certificate used in.Deleted during the first initial startup process.
18 /opt/splunk/etc/auth/serverreq.pem Of SplunkInternal communication encryptionCSR that asks the root CA to create a server certificate to be used in.Deleted during the initial startup process.
19 /opt/splunk/etc/auth/servercert.pem Of SplunkInternal communication encryptionServer certificate used in. Issued by Splunk's own root CA.Deleted during the initial startup process.
20 /opt/splunk/etc/auth/server.pem Of SplunkInternal communication encryptionServer certificate used in.**This server certificate is valid for 3 years from the first startup, and the CN is always set to SplunkServerDefaultCert.**In addition to the server certificate, the server private key and root CA certificate information are added to this file.
21 privKeySecure.pem Of Splunk**Web screen encrypted communication(https)**The private key required to create the server certificate used in.Deleted during the initial startup process.
22 req.pem Of Splunk**Web screen encrypted communication(https)**CSR that asks the CA to create a server certificate to be used in.Deleted during the initial startup process.
23 /opt/splunk/etc/auth/splunkweb/cert.pem Of Splunk**Web screen encrypted communication(https)**Server certificate used in. Issued by Splunk's own root CA.The expiration date of this server certificate is 3 years from the first startup, and the host name of the server is set for CN.
24 ca.srl A file containing the serial number of the server certificate issued by the root CA.
25 /opt/splunk/etc/auth/splunkweb/privkey.pem Of Splunk**Web screen encrypted communication(https)**The private key of the server certificate used in.
26 /opt/splunk/etc/instance.cfg Splunk unique ID(GUID)The file in which is recorded.
27 /opt/splunk/etc/passwd User information. At the first startup, only the administrator user is recorded.
28 /opt/splunk/etc/apps/learned/metadata/local.meta
29 /opt/splunk/etc/apps/learned/local/props.conf

Digression

The reason for this time was to see "How the server certificate used by default in Splunk's encrypted communication process is created when Splunk is started for the first time", but as a result, other configuration files I was able to understand the flow of the creation process. I would also like to organize and post the survey on the default certificate.

Thank you for reading this far.

Sequel

I wrote a sequel. Import audit.log into Splunk and check the behavior when logging in to Splunkweb for the first time

reference

6.6. About AUDIT log files This is the official Redhat manual, but it contains the specification information of the Audit log. However, although Amazon Linux2 is based on Rhel7, it seems that the log format (field) is slightly tuned.

Detect Linux file tampering with auditd

What is the splunk.secret file, and is it possible to change it? What is splunk.secret? This is the QA of Spunk's official community site regarding the question.

Recommended Posts

Import audit.log into Splunk and check the behavior when Splunk is started for the first time
Impressions and memorandums when working with VS code for the first time
For the time being, import them into jupyter
What I got into Python for the first time
Kaggle for the first time (kaggle ①)
Kaguru for the first time
What is a dog? Django--Getting Started with Form for the First Time POST Transmission Volume
Check the behavior when assigning Python
Behavior when 0, 1, False, True is used for the dictionary (dict) key
Differences C # engineers felt when learning python for the first time
[For self-learning] Go2 for the first time
See python for the first time
Start Django for the first time
A useful note when using Python for the first time in a while
I tried tensorflow for the first time
MongoDB for the first time in Python
Let's try Linux for the first time
[For beginners] Unexpected behavior if "\" is included when setting the path in Python
Let's display a simple template that is ideal for Django for the first time
How to use MkDocs for the first time
[Note] Deploying Azure Functions for the first time
I tried python programming for the first time.
I tried Mind Meld for the first time
virtualenv For the time being, this is all!
Try posting to Qiita for the first time
Check the processing time and the number of calls for each process in python (cProfile)
I tried Python on Mac for the first time.
[Python] Measures and displays the time required for processing
Register a task in cron for the first time
I tried python on heroku for the first time
For the first time, I learned about Unix (Linux).
Embedding in datetime when only the time is known
Creating a list when the nomenclature is a fixed time
AI Gaming I tried it for the first time
Check what the character code is for all files under the directory that is Python and output