Import audit.log into Splunk and check the behavior when Splunk is started for the first time

Introduction

When you start Splunk Enterprise for the first time, various processing messages are output to the screen.

# /opt/splunk/bin/splunk start --accept-license --seed-passwd password

This appears to be your first time running this version of Splunk.
Copying '/opt/splunk/etc/openldap/ldap.conf.default' to '/opt/splunk/etc/openldap/ldap.conf'.
Generating RSA private key, 2048 bit long modulus
.................................+++++
...........................+++++
e is 65537 (0x10001)
writing RSA key

Generating RSA private key, 2048 bit long modulus
.............................................................................+++++
.................................................................+++++
e is 65537 (0x10001)
writing RSA key

Moving '/opt/splunk/share/splunk/search_mrsparkle/modules.new' to '/opt/splunk/share/splunk/search_mrsparkle/modules'.

Splunk> See your world.  Maybe wish you hadn't.

Checking prerequisites...
	Checking http port [8000]: open
	Checking mgmt port [8089]: open
	Checking appserver port [127.0.0.1:8065]: open
	Checking kvstore port [8191]: open
	Checking configuration... Done.
		Creating: /opt/splunk/var/lib/splunk
		Creating: /opt/splunk/var/run/splunk
		Creating: /opt/splunk/var/run/splunk/appserver/i18n
		Creating: /opt/splunk/var/run/splunk/appserver/modules/static/css
		Creating: /opt/splunk/var/run/splunk/upload
		Creating: /opt/splunk/var/run/splunk/search_telemetry
		Creating: /opt/splunk/var/spool/splunk
		Creating: /opt/splunk/var/spool/dirmoncache
		Creating: /opt/splunk/var/lib/splunk/authDb
		Creating: /opt/splunk/var/lib/splunk/hashDb
New certs have been generated in '/opt/splunk/etc/auth'.
	Checking critical directories...	Done
	Checking indexes...
		Validated: _audit _internal _introspection _metrics _telemetry _thefishbucket history main summary
	Done
	Checking filesystem compatibility...  Done
	Checking conf files for problems...
	Done
	Checking default conf files for edits...
	Validating installed files against hashes from '/opt/splunk/splunk-8.0.2.1-f002026bad55-linux-2.6-x86_64-manifest'
	All installed files intact.
	Done
All preliminary checks passed.

Starting splunk server daemon (splunkd)...  
Generating a RSA private key
...............................................+++++
...............+++++
writing new private key to 'privKeySecure.pem'
-----
Signature ok
subject=/CN=aio/O=SplunkUser
Getting CA Private Key
writing RSA key
Done
 [  OK  ]

Waiting for web server at http://127.0.0.1:8000 to be available.... Done


If you get stuck, we're here to help.  
Look for answers here: http://docs.splunk.com

The Splunk web interface is at http://aio:8000

Judging from the content of the message, it seems that you are creating some file. So what will be created this time at the first startup? I investigated that with Splunk.

• In this survey, the audit settings of the OS will be changed. Please note that the setting method and log format may differ depending on the distribution.

Environment at the time of survey

OS: Amazon Linux 2 (4.14.171-136.231.amzn2.x86_64) Splunk version: 8.0.2.1 Splunk installation path ($ SPLUNK_HOME): / opt / splunk

1. Splunk Enterprise installation

Install Splunk. I will omit the detailed procedure, but this time I will install it with the tgz file.

Splunk Enterprise/opt/Install in directory

tar fvxz /tmp/splunk-8.0.2.1-f002026bad55-Linux-x86_64.tgz -C /opt/

2. Audit log output settings

Change the OS settings to output the change information of the Splunk setting directory to the audit log.

Set up an audit log(Temporary change)

#Add an audited directory
auditctl -w /opt/splunk/etc/ -p wa -k etc_changes
#Confirm that the settings have been added
auditctl -l
#OK if the command result is output."No rules"Is displayed, please check if the settings are correct.

3. Start Splunk Enterprise

Start Splunk Enterprise.

• The following command example is not very secure because the administrator password is set to password. Please specify any password if necessary.

Start Splunk Enterprise

/opt/splunk/bin/splunk start --accept-license --seed-passwd password

4. Check the audit log output

Check if the change information of the audited directory set in step 2 is output to the log.

Check if the change information of the audited directory is output to the log

grep /opt/splunk/etc/ /var/log/audit/audit.log | tail
#It is OK if one or more logs are output.

5. Audit log capture

Set audit logs to be permanently populated in Splunk.

• For research purposes, it is set to be included in the main index. If you want to import to another index, specify -index <index name> as an argument.
• Since the import settings are made using the CLI, the setting information is recorded in /opt/splunk/etc/apps/search/local/inputs.conf.

Set to capture audit logs permanently

/opt/splunk/bin/splunk add monitor /var/log/audit/audit.log -auth admin:password

6. Confirmation of import result

After completing the above settings, the audit log should have been imported to Splunk, so log in to Splunk Enterprise and check the import result with the search statement.

Search audit logs

sourcetype="linux_audit" type="PATH" nametype!="PARENT"
| rex field=name "^(?<directory>.*\/)?(?<file>.*)"
| eval directory=if(len(mode)==6,directory+file,directory),file=if(len(mode)==6,"",file)
| table _time,msg,nametype,mode,directory,file
| sort msg

Explanation of search text

Line 1 ... Searching for ** events in the audit log where the type field is PATH and the nametype field is not PARENT **. Lines 2 and 3 ... The directory name and file name are set in the new fields (directory, file) from the path information recorded in the name field. 4th line: Only the fields required for visualization are formatted and displayed from the search results. 5th line: Since the time stamp and ID information are recorded in the msg field, they are sorted and displayed in the order of the msg field.

7. Result

Here are some excerpts of the results of the search in this environment.

8. Analysis

Analyzing the search results, we found that Splunk was running the file / directory creation process in the / opt / splunk / etc / directory in the following order when it was first started.

• It seems that some files are updated several times during the processing process, but we will omit the update process.

Digression

The reason for this time was to see "How the server certificate used by default in Splunk's encrypted communication process is created when Splunk is started for the first time", but as a result, other configuration files I was able to understand the flow of the creation process. I would also like to organize and post the survey on the default certificate.

Thank you for reading this far.

Sequel

I wrote a sequel. Import audit.log into Splunk and check the behavior when logging in to Splunkweb for the first time

reference

6.6. About AUDIT log files This is the official Redhat manual, but it contains the specification information of the Audit log. However, although Amazon Linux2 is based on Rhel7, it seems that the log format (field) is slightly tuned.

Detect Linux file tampering with auditd

What is the splunk.secret file, and is it possible to change it? What is splunk.secret? This is the QA of Spunk's official community site regarding the question.