Creating custom search commands for Splunk EventingCommand

Create a custom search command of type EventingCommand for Splunk. In the article below, we created a GeneratingCommand.

This time, create an EventingCommand.

What is "EventingCommand"? For those who say, "[Four types of custom search commands](https://qiita.com/msi/items/02d029d655e1f9285806#%E3%82%AB%E3%82%B9%E3%82%BF] % E3% 83% A0% E3% 82% B5% E3% 83% BC% E3% 83% 81% E3% 82% B3% E3% 83% 9E% E3% 83% B3% E3% 83% 89% E3 % 81% AE4% E3% 81% A4% E3% 81% AE% E7% A8% AE% E9% A1% 9E) ". Regarding the development environment, the explanation of SDK etc. is omitted.

goal

Let's create a command toupper that takes the previous output" hello, world" and converts all characters to uppercase.

SPL Example


| generatehello2 | toupper greeting     ← "greeting"Is the field name

The equivalent of eval's upper () is converted by field specification.

Difference between Version 1 protocol and Version 2 protocol

Before we get started, let's write about the search command protocols Version 1 and Version 2.

There are various introductory articles about creating custom search commands for Python, but I mainly use the following techniques.

  1. Execute the created script (command) using the run command.
  2. Use the splunk.InterSplunk module internally instead of using run.

There are differences in how they handle inputs and outputs, but there are no fundamental differences. The first can be any form that can be called from splunk. The second uses splunk.InterSplunk for input and output.

These uses are called in protocol Version 1 or in the traditional way called the legacy protocol.

text:commands.conf.spec


chunked = [true|false]
* If set to "true", this command supports the new "chunked" custom
  search command protocol.
* If set to "true", the only other commands.conf settings supported are
  'is_risky', 'maxwait', 'maxchunksize', 'filename', 'command.arg.<N>', and
  'run_in_preview'.
* If set to "false", this command uses the legacy custom search command
  protocol supported by Intersplunk.py.
* Default: false

Quoted from "commands.conf --Splunk Documentation"

If you use Version 1, set chuncked = false in the commands.conf setting, or leave it to the default and leave it as false.

On the other hand, if you want to use the new version 2 custom search command, use the class ʻEventingCommand TransformingComannd` `GeneratingCommand StreamingCommand` instead of using splunk.InterSplunk.

The differences between the Version 1 protocol and the Version 2 protocol are as follows:

About the protocols

Version 2 protocol

There are significant advantages to using the Version 2 of the Custom Search Command protocol.

  • With the Version 2 protocol, external programs process through an entire set of Splunk search results in one invocation. The external program is invoked once for the entire search, greatly reducing runtime overhead.
  • The Version 2 protocol requires fewer configuration attributes than the Version 1 protocol.
  • Supports non-Python custom search commands, for example C++, Go, Java and JavaScript (Node.js).
  • Support for platform-specific executable files and binaries. You can write custom search commands in compiled languages, such as C++, on multiple platforms.

Version 1 protocol

The Version 1 of the Custom Search Command protocol processes events in groups, using 50,000 events for each group. The external program is invoked and terminated multiple times for large result sets.

Quoted from "[About writing custom search commands --Splunk Documentation](https://docs.splunk.com/Documentation/Splunk/latest/Search/Aboutcustomsearchcommands)"

With v2, more languages will be supported besides Python, but the biggest difference is that the program will not be called every 50,000 events. Conversely, ** v1 launches a custom search command every 50,000 events **, which makes a difference in performance. It makes no difference if you handle less than 50,000 events, but Splunk will soon have more than 50,000 events, so if you're going to create one, don't use splunk.InterSPlunk and v2. You should use.

In this article, we will create a custom search command using the Version 2 protocol.

Preparation

  1. If there is an environment of Last time, continue to use it as it is.
  2. To start with this article, download step1-2 from GitHub and change the HelloWorld-work directory to$ SPLUNK_HOME. Copy it to / etc / apps / HelloWorld /and restart Splunk.
  3. Create $ SPLUNK_HOME / etc / apps / HelloWorld / lib / and under it splunklib in the SDK Copy. (You can also install the SDK library with pip3 install -t lib / splunk-sdk.)

The structure of directories and files is as follows.

${SPLUNK_HOME}/etc/apps/


HelloWorld/
├─ bin
│   ├─ HelloWorld.py
│   ├─ HelloWorld2.py
│   ├─ filter.py
│   ├─ generate.py
│   ├─ report.py
│   └─ stream.py
├─ default
│   ├─ app.conf
│   ├─ commands.conf
│   └─ data
│       └─ ui
│           └─ nav
│               └─ default.xml
├─ lib
│   ├─ splunk_sdk-1.6.12.dist-info
│   │   (This directory exists if you installed with pip.)
│   └─ splunklib
│       ├─ __init__.py
│       ... (abridgement)
└─ metadata
    └─ default.meta

development of

File creation

Create ToUpper.py by copying filter.py under bin /.

cd bin/
cp filter.py ToUpper.py

The original filter.py is a template for EventingCommand, but it doesn't seem to be maintained and can't be used as is. (SDK v1.6.12)

ToUpper.py


#!/usr/bin/env python

import sys
import os

sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "lib"))
from splunklib.searchcommands import \
    dispatch, StreamingCommand, Configuration, Option, validators


@Configuration()
class %(command.title())Command(EventingCommand):
    """ %(synopsis)

    ##Syntax

    %(syntax)

    ##Description

    %(description)

    """
    def transform(self, events):
       # Put your event transformation code here
       pass

dispatch(%(command.title())Command, sys.argv, sys.stdin, sys.stdout, __name__)

Modify the part that is StreamingCommand in import to ʻEventingCommand`.

diff


--- filter.py
+++ ToUpper.py
@@ -5,8 +5,8 @@

 sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "lib"))
 from splunklib.searchcommands import \
-    dispatch, StreamingCommand, Configuration, Option, validators
+    dispatch, EventingCommand, Configuration, Option, validators

ToUpper.py&nbsp;Revised


#!/usr/bin/env python

import sys
import os

sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "lib"))
from splunklib.searchcommands import \
    dispatch, EventingCommand, Configuration, Option, validators


@Configuration()
class %(command.title())Command(EventingCommand):
    """ %(synopsis)

    ##Syntax

    %(syntax)

    ##Description

    %(description)

    """
    def transform(self, events):
       # Put your event transformation code here
       pass

dispatch(%(command.title())Command, sys.argv, sys.stdin, sys.stdout, __name__)

Edit

  1. Modify all the % (command.title ()) parts to ToUpper.
  2. transform () is a required body, so add processing to this part. (Explanation will be described later)

diff


--- ToUpper.py.orig
+++ ToUpper.py
@@ -9,7 +9,7 @@


 @Configuration()
-class %(command.title())Command(EventingCommand):
+class ToUpperCommand(EventingCommand):
     """ %(synopsis)

     ##Syntax
@@ -23,6 +23,11 @@
     """
     def transform(self, events):
        # Put your event transformation code here
-       pass
+        for event in events:
+            for field in self.fieldnames:
+                if field in event:
+                    event[field] = event[field].upper()
+            yield event
+        return

-dispatch(%(command.title())Command, sys.argv, sys.stdin, sys.stdout, __name__)
+dispatch(ToUpperCommand, sys.argv, sys.stdin, sys.stdout, __name__)

ToUpper.py&nbsp;After update


#!/usr/bin/env python

import sys
import os

sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "lib"))
from splunklib.searchcommands import \
    dispatch, EventingCommand, Configuration, Option, validators


@Configuration()
class ToUpperCommand(EventingCommand):
    """ %(synopsis)

    ##Syntax

    %(syntax)

    ##Description

    %(description)

    """
    def transform(self, events):
       # Put your event transformation code here
        for event in events:
            for field in self.fieldnames:
                if field in event:
                    event[field] = event[field].upper()
            yield event
        return

dispatch(ToUpperCommand, sys.argv, sys.stdin, sys.stdout, __name__)

That is the element of EventingCommand.

In the above example, for each event (for event in events:), for each specified field (for field in self.fieldnames:), convert it to uppercase and reassign it (ʻevent). [field] = event [field] .upper () `).

Operation check

Operation check by command line

Check the operation on the command line separated from Splunk.

To run it locally, specify __EXECUTE__ and run it.

python3 ToUpper.py __EXECUTE__ </dev/nul

If there are no errors, it is successful.

Since / dev / null is specified for the input, there is no output, but in fact, nothing is displayed when I input.

Try it with the previously created HelloWorld2.py.

$ python3 HelloWorld2.py __EXECUTE__ count=3 < /dev/null  # HelloWorld2.Operation check with py alone

_time,__mv__time,greeting,__mv_greeting,_raw,__mv__raw
1589160289.7283704,,"hello, world",,"hello, world",
1589160289.7284214,,"hello, world",,"hello, world",
1589160289.7284305,,"hello, world",,"hello, world",
$ python3 HelloWorld2.py __EXECUTE__ count=3 < /dev/null | python3 ToUpper.py __EXECUTE__ greeting
$← Return to the command prompt without displaying anything.

Note: I couldn't find a way to display it on the command line. If anyone knows about it, I would appreciate it if you could teach it.

For the time being, you can check if there is an error.

Operation check by search

Check the operation as a Splunk search command.

Setting

If you want to modify what is provided, create a file under the local / directory and set it, but this time, from the standpoint of "provider", change the file under default / ..

commands.conf Edit default / commands.conf.

--- commands.conf.orig
+++ commands.conf
@@ -10,3 +10,8 @@
 filename = HelloWorld2.py
 chunked = true
 python.version = python3
+
+[toupper]
+filename = ToUpper.py
+chunked = true
+python.version = python3

default/commands.conf


# [commands.conf]($SPLUNK_HOME/etc/system/README/commands.conf.spec)
# Configuration for Search Commands Protocol version 2

[generatehello]
filename = HelloWorld.py
chunked = true
python.version = python3

[generatehello2]
filename = HelloWorld2.py
chunked = true
python.version = python3

[toupper]
filename = ToUpper.py
chunked = true
python.version = python3
searchbnf.conf and default.meta

This time, set searchbnf.conf and default.meta as Tips.

Create a new searchbnf.conf.

searchbnf.conf


[toupper-command]
syntax = toupper <field-list>
description = toupper command to transfer into upper cases by the specify fields.\
        This is a long description.
shortdesc = make fields contents to upper cases.
usage = public

See also: searchbnf.conf --Splunk Documentation

[^ 1]: As an aside, UNIX roff convention is to write variable parts in italics. The bad news about MarkDown notation and HTML is that you can't write italics in \ quote \ to indicate variable parts. It's hard to tell which is a variable character and which is a fixed character, so I have to explain it every time. On the contrary, it may be difficult to understand from the document alone without explanation.

Make additional settings for searchbnf to metadata / default.meta.

--- default.meta.orig
+++ default.meta
@@ -1,2 +1,5 @@
 []
 access = read: [ * ], write : [ admin ]
+
+[searchbnf]
+export = system

metadata/default.meta


[]
access = read: [ * ], write : [ admin ]

[searchbnf]
export = system

Reboot

Restart splunkd [^ 2].

[^ 2]: Not sure if it will be enabled by debug / refresh.

sudo ${SPLUNK_HOME}/bin/splunk restart

Or, if set with systemctl

sudo systemctl restart Splunkd

Operation check

Check the operation in the search window.

1. Confirm generatehello2.

SPL statement


| generatehello2 count=3
| table greeting

Execution result


greeting
hello, world
hello, world
hello, world
2. Confirm toupper.

SPL statement


| generatehello2 count=3
| table greeting
| toupper greeting

Execution result


greeting
HELLO, WORLD
HELLO, WORLD
HELLO, WORLD

The lowercase letters have been successfully converted to uppercase.

Summary

Recommended Posts

Creating custom search commands for Splunk EventingCommand
Create Splunk custom search command Part 2
Commands for creating SNS with Django
Commands for creating a new django project
Create a Splunk custom search command-the first step-
Create a custom search command in Splunk (Streaming Command)
Commands for creating a python3 environment with virtualenv
About creating and modifying custom themes for Python IDLE
Jupyter Docker image summary
Make Jupyter Lab accessible remotely
Snippet settings for python jupyter notebook
Build Jupyter Lab (Python) environment with Docker
Jupyter Notebook extension, nbextensions settings for myself
Creating amateur python environment settings (for MAC)
Let's make jupyter lab easy to use
Creating custom search commands for Splunk EventingCommand