The road to updating Splunkbase with your own Splunk app for Python v2 / v3

This article is one of the articles in Splunk Advent Calendar 2019. Please also enjoy Other Articles.

This time, it's good to publish your own App on Splunkbase, It's about time (?!) I have to support Python 3 ... Splunk also Migrate to Python 3!

This is for Splunk App developers who are busy with the deadline.

To briefly introduce myself, In my daily work, I'm mainly in charge of building and troubleshooting Splunk. My favorite feature is the Metrics type index.

Briefly about the background of the article.

What is Python 3 migration in the first place?

Python, an interpreted programming language, The v2 and v3 series have been used for a long time, Maintenance of v2 series (v2.7) by the development community Ended on 2020/1/1 It has been announced 12 years ago (2008).

Now, 2019 is finally over a little. It's time to raise your heavy hips.

By the way, just after the maintenance is finished After 2020/1, it doesn't stop suddenly. Just to be sure.

In the first place, it should be written for Python 3.

Yes. You're right.

However,

I think that there are many cases where it is postponed for some reason.

Splunk's Python affairs.

What's more, there are some special circumstances with Splunk.

Software called Splunk ** Python runtime environment is included **. Therefore, without installing Python separately on the OS side, It was available as a plugin (App, Add-on).

This bundled Python will be available until just the other day (October 22, 2019) I was in a situation where I could only choose Python 2.7.

Python 3.7, a v3 family, began to be included in Splunk v8.0.0.

Now I can't use the environment as an excuse (it's a misleading nuance).

By the way, v8.0.0 includes Python 2.7 and Python 3.7. Both environments are included and For each function used in App and Add-on You can choose which environment to run in. By default, it works with the traditional v2.7 series (However, with some exceptions such as web application functions).

As a future roadmap, By mid-2020, the default Python will change from v2.7 to v3.7 (v2.7 can be selected). In the second half of 2020, it seems that v2.7 will be deprecated and only v3.7 will be included.

Just rewrite to v3.7?

As a point to note when migrating ** v2 / v3 compatible notation is highly recommended **.

Well, you said that v2 will not be maintained by EOL, The opinion is reasonable, but ...

If you're familiar with Splunk, you might be familiar with it, There is a function called distributed search.

Splunk can be used all-in-one in a single instance, or it can be used. It is also possible to separate instances for each major function to improve search efficiency and availability. As the main actor,

It can be divided into. When this distributed configuration is adopted, The search command received by the search head, To multiple indexers (also known as peers), Request distributed search processing.

At this time, the App on the search head also It is passed to the indexer as part of the instruction and executed on the indexer.

With a custom search command, etc. If you have written a process using Python, Searchhead and indexer settings (specifying Python environment) It would be nice if they were the same Search head is v3.7 compatible, If you specify that the indexer works with v2.7, v3 compliant, v2 incompatible notation Distributed processing on the indexer fails The scenario is waiting.

For such a situation, the notation for both Python v2 / v3 is Recommended.

What is the difference between Python v2 / v3?

It seems that it will gradually deviate from the main subject I'll wrap up the details of the differences related to the Python version, Please refer to other articles for details.

In my experience

I remember being addicted to it.

When rewriting to compatible notation, I would like to introduce two, six and tox. In a nutshell, six is for absorbing differences such as function names, tox is a unit test for both v2 / v3.

Differences between Python v2 and v3 Efforts to absorb in the library six is It's also included in the library for Splunk App development.

Also, Python 2 and Python 3 respectively To run unit tests in your environment tox A framework is also available. Unit test of the same code, It can be run in multiple interpreter environments (such as Python v2 and v3).

Until the check that there is no grammatical problem To some extent, it can be handled mechanically with a framework or the like.

Rewrite your own Splunk App.

By the way, the pillow of the story has become quite long, but it's about time for the main subject.

For app compatibility check, The Splunk Platform Upgrade Readiness App (https://splunkbase.splunk.com/app/4698/) is also available, You may need a Splunk Enterprise environment, so First, let's take a quick look at AppInspect's REST API. Later, we'll talk about the Splunk Platform Upgrade Readiness App later.

Compatibility Check Part 1: AppInspect REST API

Obtaining an authentication token

$ curl -X GET \
       -u {splunk.com username} \
       --url "https://api.splunk.com/2.0/rest/login/splunk"

You will be asked for your password Enter the password for your splunk.com user account.

{
    "data": {
        "token": "{Token string. Less than,$See in TOKEN}",
        "user": {
            "email": "{Registered email address}",
            "groups": [
                "Beta Users"
            ],
            "name": "{Registered name}",
            "username": "{splunk.com username}"
        }
    },
    "msg": "Successfully authenticated user and assigned a token",
    "status": "success",
    "status_code": 200
}

Here, we will check the token string in the response using the following steps. (The token string is quite long, so it may be better to put it in a shell variable etc.)

In the following, it is assumed that the token obtained here is set in the environment variable TOKEN. (Refer to $ TOKEN)

Issuance of App inspection request

$ curl -X POST
       -H "Authorization: bearer $TOKEN"
       -H "Cache-Control: no-cache"
       -F "app_package=@\"line-alert-for-splunk_100.tgz\""
       -F "included_tags=py3_migration"
       --url "https://appinspect.splunk.com/v1/app/validate"
{
    "request_id": "a32e91f8-7767-400f-a33f-xxxxxxxxxxxx",
    "message": "Validation request submitted.",
    "links": [
        {
            "rel": "status",
            "href": "/v1/app/validate/status/a32e91f8-7767-400f-a33f-xxxxxxxxxxxx"
        },
        {
            "rel": "report",
            "href": "/v1/app/report/a32e91f8-7767-400f-a33f-xxxxxxxxxxxx"
        },
        {
            "rel": "package",
            "href": "/v1/app/package/a32e91f8-7767-400f-a33f-xxxxxxxxxxxx"
        }
    ]
}

Use the request ID included in the response to query the results.

Check App inspection status and results

To check the progress of the inspection request Query the status endpoint.

$ curl -X GET
       -H "Authorization: bearer $TOKEN"
       -H "Cache-Control: no-cache"
       --url https://appinspect.splunk.com/v1/app/validate/status/a32e91f8-7767-400f-a33f-xxxxxxxxxxxx
{
    "request_id": "a32e91f8-7767-400f-a33f-xxxxxxxxxxxx",
    "status": "SUCCESS",
    "info": {
        "error": 0,
        "failure": 0,
        "skipped": 0,
        "manual_check": 0,
        "not_applicable": 3,
        "warning": 0,
        "success": 10
    },
    "links": [
        {
            "rel": "self",
            "href": "/v1/app/validate/status/a32e91f8-7767-400f-a33f-xxxxxxxxxxxx"
        },
        {
            "rel": "report",
            "href": "/v1/app/report/a32e91f8-7767-400f-a33f-xxxxxxxxxxxx"
        }
    ]
}

If the status is SUCCESS, you are done.

Use the report endpoint to get the results.

$ curl -X GET
       -H "Authorization: bearer $TOKEN"
       -H "Cache-Control: no-cache"
       --url https://appinspect.splunk.com/v1/app/report/a32e91f8-7767-400f-a33f-xxxxxxxxxxxx

If not specified, json will be returned. You can also get the result in HTML by specifying Content-Type as appropriate.

$ curl -X GET
       -H "Authorization: bearer $TOKEN"
       -H "Content-Type: text/html"
       -H "Cache-Control: no-cache"
       --url https://appinspect.splunk.com/v1/app/report/a32e91f8-7767-400f-a33f-xxxxxxxxxxxx

191215-appinspect-MSTeams-1.png

This is easier to see. If Failures and Error in the Totals column are zero, it's a relief.

Compatibility Check # 2: Splunk Platform Upgrade Readiness App

Next, in the Splunk Platform Upgrade Readiness App, I will introduce the check method.

Originally, when upgrading from Splunk v7.x to Splunk v8.0 It's an app to check, It is convenient because you can also check for Python 3 compatibility.

However, to do Upgrade source Splunk Enterprise v7.1, v7.2, v7.3 Either environment is required. In addition, it should be noted ** Doesn't work with upgraded Splunk 8! **

If you are lucky enough to have the target v7.x series, install the App and From the App list in the Splunk Web UI Open the Splunk Platform Upgrade Readiness App.

Select the Run New Scan button in the upper right From the Scan Settings pull-down Select Scan custom selection of apps.

191215-upgrade_readiness_app-MSTeams-3a.png

A list of installed apps will be displayed on the right side of the screen. Select the app you want to check from the list.

191215-upgrade_readiness_app-MSTeams-3b.png

Select the Scan button.

Wait for a while and you should see the result. It depends on the files included in the app, but it can take a few minutes to a few tens of minutes, so Is it good to wait patiently?

When completed, the GUI will display Scan completed You can see the scan results.

191215-upgrade_readiness_app-MSTeams-4.png

By the way, the operation log of Upgrade Readiness App is In $ SPLUNK_HOME / var / log / upgrade_readiness_app / upgrade_readiness.log It will be output.

2019-12-11 16:57:44,671 INFO 140663382782080 - Scan initiated
2019-12-11 16:57:44,671 INFO 140663382782080 - Retrieving key to write progress
2019-12-11 16:57:44,771 INFO 140663382782080 - Found key for existing entry: 5df0a1788f02502eb0569f21
2019-12-11 16:57:44,829 INFO 140663382782080 - Total 1 apps found for user: admin
2019-12-11 16:57:44,905 INFO 140663382782080 - 0 apps out of 1 scanned. Scanning App: Microsoft Teams alert for Splunk
2019-12-11 16:57:59,288 INFO 140084004323456 - Handling a request
2019-12-11 16:57:59,289 INFO 140084004323456 - Executing function, name=get_read_progress

(Omission)

2019-12-11 17:19:17,115 INFO 140663382782080 - Deployment scanned successfully for user: admin

In Public Documents (https://docs.splunk.com/Documentation/UpgradeReadiness/2.0.0/Use/Use),

Some Splunk apps are too large to scan. If you cannot scan a Splunk app, follow the app's documentation for updates on Python 3 readiness. I also find a proviso that says, so I pray not to fail.

It's not productive to wait while squeezing, Let's take a look at the refurbishment points for the Splunk App. If you look at the public document Act on scan results (https://docs.splunk.com/Documentation/UpgradeReadiness/2.0.0/Use/Use#Act_on_scan_results), The following items are described.

It is also a good idea to carry out the parts that can be dealt with in advance.

When the scan is complete, a list of results will be displayed.

191215-upgrade_readiness_app-MSTeams-1.png

Check 7: You can find that the Status: of Python scripts is ** Warning **.

This app is modeled with the slightly older Add-on Builder, so It seems that the files are completely stuck.

It is displayed on the right side of each item in the Issues list. Click the See Issues link The correction points are displayed (this is useful).

191215-upgrade_readiness_app-MSTeams-2.png

Scan for Upgrade Readiness App It looks like you're looking at a Python file independently (my imagination) Even when v2 / v3 is conditional branched and processed in another file, etc. It seems that there is a possibility that it will be a candidate for repair. I'm scared of the number of scan results for a moment, Check it out and fix it if necessary.

(Extra edition) I found a problem with the compatibility of third-party apps other than my own.

If it was published on Splunkbase, Let's look for an updated version.

If support continues by the author and maintainer It may be a good idea to ask for repairs.

If you have the courage, it may be a good idea to repair it yourself, After confirming that there is no problem with the license, while considering the operation impact on others, I think it would be good to be supported. When it's sunny and the corresponding version is completed You can also contact the author and have them merge It may be the real thrill of open source.

Upload updated version to Splunkbase.

Finally, upload the updated version and reapply.

Log in to Splunkbase (https://splunkbase.splunk.com) and Go to My Account --My Profile from the top email.

191211-splunkbase-1.png

Select the target app from Your Apps and select From the Administrator Tools menu at the top Select Manage App.

191211-splunkbase-2.png

Select New Version at the top right.

191211-splunkbase-3.png

The Splunkbase Developer Distribution Agreement will be displayed. Agree

A drop area called Version: New Release will be displayed. Upload a new version of the app.

You will be taken to a page where you can write Release Notes, etc., so please write them as appropriate. There is also a Splunk Version Compatibility column, so I also checked 8.0.

If you save, the application is complete. After that, let's pray that the examination will pass safely.

Finally.

In this article, how to use Splunk such as search, Without touching on a convenient Add-on introduction or how to use a nifty configuration, I've put the spotlight on how to develop and publish Splunk App.

I would appreciate it if you could find something useful.

Of course, the updated version of the app It's also important to run it in a Splunk v8 environment and check its operation.

Not limited to Splunk App, when modifying applications It may also be required to check the part that there is no difference in the business logic. It may be an ideal theory, but I prepared a unit test properly and prepared it properly. Checking that the same result can be obtained with Python 2 and Python 3 with tox etc. I think it's a way to prevent regression.

Regarding the new release of your own App, I have written an article called My First Splunk App on Splunkbase, so if you are interested, please have a look.

Well then, Happy Merry Christmas !!

Recommended Posts

The road to updating Splunkbase with your own Splunk app for Python v2 / v3
The road to compiling to Python 3 with Thrift
How to access data with object ['key'] for your own Python class
Memo to create your own Box with Pepper's Python
[Road to intermediate Python] Define in in your own class
Run the intellisense of your own python library with VScode.
Try docker: Create your own container image for your Python web app
I tried to refer to the fun rock-paper-scissors poi for beginners with Python
Memo to ask for KPI with python
[Introduction to Udemy Python3 + Application] 47. Process the dictionary with a for statement
To import your own module with jupyter
Publish your own Python library with Homebrew
[TensorFlow / Keras] The road to assembling an RNN with your favorite structure
[Introduction to Python] How to get the index of data with a for statement
Try to make your own AWS-SDK with bash
The fastest way for beginners to master Python
[Python] matplotlib: Format the diagram for your dissertation
Specify the Python executable to use with virtualenv
Use logger with Python for the time being
Say hello to the world with Python with IntelliJ
Make your own module quickly with setuptools (python)
The easiest way to use OpenCV with python
Introduction to Python with Atom (on the way)
python: Use your own class for numpy ndarray
Steps to install your own library with pip
Get an Access Token for your service account with the Firebase Admin Python SDK
The first step of machine learning ~ For those who want to implement with python ~
[For beginners] Web scraping with Python "Access the URL in the page to get the contents"
How to set the development environment for each project with VSCode + Python extension + Miniconda
Try to solve the programming challenge book with python3
Make your python CUI application an app for mac
[Introduction to Udemy Python 3 + Application] 66. Creating your own exceptions
[Introduction to Python] How to iterate with the range function?
Run with CentOS7 + Apache2.4 + Python3.6 for the time being
Try to improve your own intro quiz in Python
Fleet provisioning with AWS IoT SDK for Python v2
Try to solve the internship assignment problem with Python
The first algorithm to learn with Python: FizzBuzz problem
I tried to touch the CSV file with Python
Steps to install the latest Python on your Mac
I tried to solve the soma cube with python
[Python] How to specify the download location with youtube-dl
Convert the image in .zip to PDF with Python
Use the CASA Toolkit in your own Python environment
Information for controlling the motor with Python on RaspberryPi
PIL with Python on Windows 8 (for Google App Engine)
Getting Started with Google App Engine for Python & PHP
Specify MinGW as the compiler to use with Python
For those who want to write Python with vim
Call your own python module from the ROS package
I tried to solve the problem with Python Vol.1
[Python] How to rewrite the table style with python-pptx [python-pptx]
[Python] Road to a snake charmer (5) Play with Matplotlib
Programming with your smartphone anywhere! (Recommended for C / Python)
[Python] When you want to import and use your own package in the upper directory
The road to Pythonista
The road to Djangoist
I tried to find the entropy of the image with python
I tried to simulate how the infection spreads with Python
I was hooked for 2 minutes with the Python debugger pdb
Update Python for Raspberry Pi to 3.7 or later with pyenv