Track baseball balls with Python + OpenCV

When I saw the OpenCV site for the first time in a while, it was new, so I touched it for a moment.

https://opencv.org/

I used to track baseball balls in Java a long time ago, so this time I tried it in Python. Tracking is a combination of background subtraction and template matching.

Environment

The environment is as follows.

I am creating an environment with venv. It doesn't matter if it's not venv.

$ python -m venv pythonOpevCV
$ cd pythonOpenCV
$ Script\activate
$ python -m pip install --upgrade pip
$ pip install opencv-python
Successfully installed numpy-1.18.2 opencv-python-4.2.0.34

numpy will also be installed.

What to prepare

Please prepare the following items.

--Video showing a baseball ball --Cut image of baseball ball (template image)

The image of the cut out ball should look like the one below. Below is a sample.

template.jpeg

procedure

Follow the steps below to track.

  1. Save as an image for each frame of the video
  2. Grayscale frame and template images
  3. Binarize frame image and template image
  4. Perform background subtraction between the previous and next frame images
  5. Perform template matching from the image resulting from background subtraction.
  6. Draw tracking results

Even if template matching is performed on the original frame image suddenly, clouds and buildings in the background will be detected by mistake as a ball, and the accuracy is not good, so grayscale → binarization will make a black and white image. If background subtraction is performed on the previous and next frames in the black-and-white image state, the clouds and background will hardly move in about 0.1 seconds, so the moving ball can be clearly detected.

The grayscale image looks like this:

gray.jpeg

Detects balls in the area surrounded by the red frame. This image is binarized into a black and white image.

binary.jpeg

The ball can be detected in white, but only the ball cannot be detected because the background and the ball are the same white. Here, the background subtraction between the previous and next frame images is as follows.

sub.jpeg

The white background is not working and is not detected by background subtraction, only the ball and other noise are detected in white. In this state, template matching is performed with the binarized image of the following template image.

templatebi.jpeg

This will allow the ball to be detected without being affected by luminosity or background.

Source code

The source code is as follows. Place the target video in VIDEOPATH and the template image in TEMPLATEPATH.

main.py


import glob
import re
import cv2


VIDEOPATH = "media/video/video.mp4"
IMAGEPATH = "media/image/"
TEMPLATEPATH = "template.jpeg "


def save_frames(video_path, image_dir):
    """
Extract frame images from videos
    """
    cap = cv2.VideoCapture(video_path)
    digit = len(str(int(cap.get(cv2.CAP_PROP_FRAME_COUNT))))
    n = 0
    while True:
        ret, frame = cap.read()
        if ret:
            cv2.imwrite("{}original/frame_{}.{}".format(IMAGEPATH, n, "jpeg"), frame)
            n += 1
        else:
            return


def do_grayscale(image_path):
    """
Grayscale the image
    """
    img = cv2.imread(image_path)
    gray = cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)
    save_image(image_path, "gray", gray)


def do_binarization(image_path):
    """
Image binarized
    """
    img = cv2.imread(image_path)
    ret, img_thresh = cv2.threshold(img, 100, 255, cv2.THRESH_BINARY)
    save_image(image_path, "binary", img_thresh)


def do_backgroundsub():
    """
Perform background subtraction
    """
    img_list = glob.glob(IMAGEPATH + "binary/frame*.jpeg ")
    num = lambda val: int(re.sub("\D","",val))
    sorted(img_list,key=(num))
    source = img_list[0]
    for path in img_list:
        diff = cv2.absdiff(cv2.imread(source),cv2.imread(path))
        source = path
        save_image(path, "bgsub", diff)


def do_template_matching():
    """
Perform template matching between template image and frame image
    """
    template_img = cv2.imread(IMAGEPATH + "binary/" + TEMPLATEPATH)
    img_list = glob.glob(IMAGEPATH + "bgsub/frame*.jpeg ")
    num = lambda val: int(re.sub("\D","",val))
    sorted(img_list,key=(num))
    location_list = []
    for path in img_list:
        result = cv2.matchTemplate(cv2.imread(path), template_img, cv2.TM_CCOEFF)
        minVal, maxVal, minLoc, maxLoc = cv2.minMaxLoc(result)
        location_list.append(maxLoc)
    return location_list

def draw_rectangle(location_list):
    """
Draw the matching result on the image
    """
    source = cv2.imread(IMAGEPATH + "original/frame_0.jpeg ")
    cv2.imwrite(IMAGEPATH + "result.jpeg ",source)
    source = cv2.imread(IMAGEPATH + "result.jpeg ")
    for loc in location_list:
        lx, ly, rx, ry = loc[0] - 10, loc[1] - 10, loc[0] + 10, loc[1] + 10
        img = cv2.rectangle(source, (lx, ly), (rx, ry), (0, 255, 0), 3)
        cv2.imwrite(IMAGEPATH + "result.jpeg ",img)

def save_image(img_path, dir, img):
    """
Save the image
    img_path :Image path
    dir :Directory name
    img :image data
    """
    file_name = img_path.replace("\\","/").split(".")[0].split("/")[-1]
    cv2.imwrite("{}{}/{}.{}".format(IMAGEPATH, dir, file_name,"jpeg"), img)


if __name__=="__main__":
    #① Divide the video into frames
    save_frames(VIDEOPATH,IMAGEPATH)
    #(2) Grayscale the template image and frame image
    do_grayscale(IMAGEPATH + TEMPLATEPATH)
    for path in glob.glob(IMAGEPATH + "original/*.jpeg "):
        do_grayscale(path)
    #③ Binarization of template image and frame image
    for path in glob.glob(IMAGEPATH + "gray/*.jpeg "):
        do_binarization(path)
    #④ Perform background subtraction
    do_backgroundsub()
    #⑤ Perform template matching
    location_list = do_template_matching()
    #⑥ Project the matched coordinates
    draw_rectangle(location_list)

result

I tried to detect the ball in the pitching video. Although it can be detected in general, it also detects parts other than the trajectory of the ball.

result.jpeg

There was a way to correct noise and outliers, but I forgot, so I remember it again.

Summary

Baseball ball tracking was performed using background subtraction and template matching in Python + OpenCV. If you use YOLO, it seems that you can detect a baseball ball in the image, so I would like to try this area as well.

Recommended Posts

Track baseball balls with Python + OpenCV
Binarization with OpenCV / Python
"Apple processing" with OpenCV3 + Python3
Image editing with python OpenCV
Camera capture with Python + OpenCV
[Python] Using OpenCV with Python (Basic)
Face detection with Python + OpenCV
Using OpenCV with Python @Mac
Shining life with Python and OpenCV
[Python] Using OpenCV with Python (Image Filtering)
Neural network with OpenCV 3 and Python 3
[Python] Using OpenCV with Python (Image transformation)
[Python] Using OpenCV with Python (Edge Detection)
Easy Python + OpenCV programming with Canopy
Cut out face with Python + OpenCV
Face recognition with camera with opencv3 + python2.7
Load gif images with Python + OpenCV
Find image similarity with Python + OpenCV
Graph Based Segmentation with Python + OpenCV
Draw arrows (vectors) with opencv / python
Basic study of OpenCV with Python
Face detection with Python + OpenCV (rotation invariant)
Save video frame by frame with Python OpenCV
Capturing images with Pupil, python and OpenCV
I tried non-photorealistic rendering with Python + opencv
Image processing with Python & OpenCV [Tone Curve]
Image acquisition from camera with Python + OpenCV
[python, openCV] base64 Face recognition with images
Create miscellaneous Photoshop videos with Python + OpenCV ③ Create miscellaneous Photoshop videos
[Python] Read images with OpenCV (for beginners)
Until you can use opencv with python
Light image processing with Python x OpenCV
Smoothing edge-saved with python + OpenCV (BilateralFilter, NLMeansFilter)
FizzBuzz with Python3
Scraping with Python
Statistics with python
Scraping with Python
Twilio with Python
Integrate with Python
Play with 2016-Python
AES256 with python
python starts with ()
OpenCV Samples (Python)
Bingo with python
Zundokokiyoshi with python
[Note] openCV + python
Track Python programs
Excel with Python
Microcomputer with Python
Cast with python
Hello World and face detection with OpenCV 4.3 + Python
I tried "smoothing" the image with Python + OpenCV
Track objects in your video with OpenCV Tracker
Performance comparison of face detector with Python + OpenCV
I tried "differentiating" the image with Python + OpenCV
Edge extraction with python + OpenCV (Sobel filter, Laplacian filter)
How to crop an image with Python + OpenCV
Install OpenCV 4.0 and Python 3.7 on Windows 10 with Anaconda
Track green objects with python + numpy (particle filter)
Make OpenCV3 available from python3 installed with pyenv
I tried "binarizing" the image with Python + OpenCV