How to draw OpenCV images in Pygame

If you want to run OpenCV in Python and display the results of analysis and processing, it is easiest to use the cv2.imshow () method. But if you want to do more than just display, OpenCV windows aren't enough. One way to solve this problem is to use Pygame, a Python GUI framework [^ Pygame reason for choosing]. In this post, I will write about how to convert OpenCV images to images for Pygame.

[Reason for choosing ^ Pygame]: The reason for Pygame is that it can also be drawn on Raspbian Lite (super important). Most GUI frameworks (including OpenCV windowing) rely on the X Window System and cannot be viewed on Raspbian Lite without a built-in GUI. However, Pygame also has the ability to draw using SDL, so it also works with Raspbian Lite.

environment

hardware
Raspberry Pi 3 Model B+
OS
Raspbian Buster Lite / 2019-09-26
Python
3.7.3
OpenCV
3.2.0
Pygame
1.9.4.post1

Install command

#After writing the OS, log in with ssh

#Update package list
sudo apt update

#Install OpenCV for Python 3
sudo apt install python3-opencv

#Install Pygame for Python 3
sudo apt install python3-pygame

#Check Python version
python3 --version

#Check the version of OpenCV
python3 -c 'import cv2; print(cv2.__version__)'

#Check the version of Pygame
python3 -c 'import pygame; print(pygame.version.ver)'

How to find it on the web

According to "Draw an image created with opencv with pygame. --BlankTar", it is as follows. It seems that it can be converted by the method.

opencv_image = opencv_image[:,:,::-1]  #Since OpenCV is BGR and pygame is RGB, it is necessary to convert it.
shape = opencv_image.shape[1::-1]  #OpenCV(height,width,Number of colors), Pygame(width, height)So this is also converted.
pygame_image = pygame.image.frombuffer(opencv_image.tostring(), shape, 'RGB')

Let's write the code to convert this way.

show-image.py


import time

import cv2
import pygame


def get_opencv_img_res(opencv_image):
    height, width = opencv_image.shape[:2]
    return width, height

def convert_opencv_img_to_pygame(opencv_image):
    """
Convert OpenCV images for Pygame.

    see https://blanktar.jp/blog/2016/01/pygame-draw-opencv-image.html
    """
    opencv_image = opencv_image[:,:,::-1]  #Since OpenCV is BGR and pygame is RGB, it is necessary to convert it.
    shape = opencv_image.shape[1::-1]  #OpenCV(height,width,Number of colors), Pygame(width, height)So this is also converted.
    pygame_image = pygame.image.frombuffer(opencv_image.tostring(), shape, 'RGB')

    return pygame_image

def main():
    #Load images with OpenCV
    image_path = '/usr/share/info/gnupg-module-overview.png'  #The path of the image file that was originally included in Raspbian Buster Lite
    opencv_image = cv2.imread(image_path)

    #Initialize Pygame
    pygame.init()
    width, height = get_opencv_img_res(opencv_image)
    screen = pygame.display.set_mode((width, height))

    #Convert OpenCV images for Pygame
    pygame_image = convert_opencv_img_to_pygame(opencv_image)

    #Draw image
    screen.blit(pygame_image, (0, 0))
    pygame.display.update()  #Update screen

    #Wait 5 seconds to finish
    time.sleep(5)
    pygame.quit()


if __name__ == '__main__':
    main()

Start command


sudo python3 show-image.py
# Note:You need to add "sudo" and run it with root privileges in order to view it using the SDL library.
#"With desktop" version of Raspbian that is not Lite, and "ssh command"-For example, if you are connecting with the "X" option added,
#No "sudo" command is required as it can be viewed using the X Window System.
#       see https://www.subthread.co.jp/blog/20181206/

When you start it, the image will be displayed for 5 seconds and it will end automatically. The conversion works correctly.

Conversion speed

Now, let's consider the case of displaying the analyzed video in real time by the above method. Such applications are often used, such as when analyzing and displaying images read from a camera. In this case, the important thing is the conversion speed. Let's measure.

show-image.py


import time

import cv2
import pygame


def get_opencv_img_res(opencv_image):
    height, width = opencv_image.shape[:2]
    return width, height

def convert_opencv_img_to_pygame(opencv_image):
    """
Convert OpenCV images for Pygame.

    see https://blanktar.jp/blog/2016/01/pygame-draw-opencv-image.html
    """
    opencv_image = opencv_image[:,:,::-1]  #Since OpenCV is BGR and pygame is RGB, it is necessary to convert it.
    shape = opencv_image.shape[1::-1]  #OpenCV(height,width,Number of colors), Pygame(width, height)So this is also converted.
    pygame_image = pygame.image.frombuffer(opencv_image.tostring(), shape, 'RGB')

    return pygame_image

def main():
    #Load images with OpenCV
    image_path = '/usr/share/info/gnupg-module-overview.png'  #The path of the image file that was originally included in Raspbian Buster Lite
    opencv_image = cv2.imread(image_path)

    #Initialize Pygame
    pygame.init()
    width, height = get_opencv_img_res(opencv_image)
    screen = pygame.display.set_mode((width, height))

    #Convert OpenCV images for Pygame
    time_start = time.perf_counter()  #Start measurement
    pygame_image = convert_opencv_img_to_pygame(opencv_image)
    time_end = time.perf_counter()  #End of measurement
    print(f'Conversion time: {time_end - time_start}Seconds/ {1/(time_end - time_start)}fps')

    #Draw image
    screen.blit(pygame_image, (0, 0))
    pygame.display.update()  #Update screen

    #Wait 5 seconds to finish
    time.sleep(5)
    pygame.quit()


if __name__ == '__main__':
    main()
Conversion time: 0.14485926300017127 seconds/ 6.903252020540914fps

**slow! !! !! ** **

0.14 seconds with a code that just displays. If you write 7fps in terms of frame rate, you can see this slowness.

Is this the limit of OpenCV + Pygame? No, it's not. I'm just doing it wrong. The reason why it is so slow is that the image data is purposely converted to a character string by the tostring () method. As you can see by measuring, 90% of this weight is due to the tostring () method.

Faster way

The actual image data of OpenCV for Python is NumPy's ndarray. Pygame has a function that reads image data from an array of NumPy. How to make full use of this is described in "OpenCV VideoCapture running on PyGame".

frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
frame = np.rot90(frame)
frame = pygame.surfarray.make_surface(frame)

Also, in Comment section of "OpenCV VideoCapture running on PyGame", the solution to the image inversion problem caused by this method + further speedup Is also described.

 frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
-frame = np.rot90(frame)
+frame = frame.swapaxes(0,1)
 frame = pygame.surfarray.make_surface(frame)

Change to this method and measure again.

show-image.py


import time

import cv2
import pygame


def get_opencv_img_res(opencv_image):
    height, width = opencv_image.shape[:2]
    return width, height

def convert_opencv_img_to_pygame(opencv_image):
    """
Convert OpenCV images for Pygame.

    see https://gist.github.com/radames/1e7c794842755683162b
    """
    rgb_image = cv2.cvtColor(opencv_image, cv2.COLOR_BGR2RGB).swapaxes(0, 1)
    #Generate a Surface for drawing images with Pygame based on OpenCV images
    pygame_image = pygame.surfarray.make_surface(rgb_image)

    return pygame_image

def main():
    #Load images with OpenCV
    image_path = '/usr/share/info/gnupg-module-overview.png'  #The path of the image file that was originally included in Raspbian Buster Lite
    opencv_image = cv2.imread(image_path)

    #Initialize Pygame
    pygame.init()
    width, height = get_opencv_img_res(opencv_image)
    screen = pygame.display.set_mode((width, height))

    #Convert OpenCV images for Pygame
    time_start = time.perf_counter()  #Start measurement
    pygame_image = convert_opencv_img_to_pygame(opencv_image)
    time_end = time.perf_counter()  #End of measurement
    print(f'Conversion time: {time_end - time_start}Seconds/ {1/(time_end - time_start)}fps')

    #Draw image
    screen.blit(pygame_image, (0, 0))
    pygame.display.update()  #Update screen

    #Wait 5 seconds to finish
    time.sleep(5)
    pygame.quit()


if __name__ == '__main__':
    main()
Conversion time: 0.030075492999912967 seconds/ 33.24966277370395fps

It has improved a lot.

Drawing grayscale images

Now that the speed issue has been improved, let's view the result of doing something with OpenCV. For example, suppose you want to display the binarized result.

show-image.py


import time

import cv2
import pygame


def get_opencv_img_res(opencv_image):
    height, width = opencv_image.shape[:2]
    return width, height

def convert_opencv_img_to_pygame(opencv_image):
    """
Convert OpenCV images for Pygame.

    see https://gist.github.com/radames/1e7c794842755683162b
    """
    rgb_image = cv2.cvtColor(opencv_image, cv2.COLOR_BGR2RGB).swapaxes(0, 1)
    #Generate a Surface for drawing images with Pygame based on OpenCV images
    pygame_image = pygame.surfarray.make_surface(rgb_image)

    return pygame_image

def main():
    #Load images with OpenCV
    image_path = '/usr/share/info/gnupg-module-overview.png'  #The path of the image file that was originally included in Raspbian Buster Lite
    opencv_image = cv2.imread(image_path)

    #Process image
    opencv_gray_image = cv2.cvtColor(opencv_image, cv2.COLOR_BGR2GRAY)
    ret, opencv_threshold_image = cv2.threshold(opencv_gray_image, 128, 255, cv2.THRESH_BINARY)
    opencv_image = opencv_threshold_image

    #Initialize Pygame
    pygame.init()
    width, height = get_opencv_img_res(opencv_image)
    screen = pygame.display.set_mode((width, height))

    #Convert OpenCV images for Pygame
    pygame_image = convert_opencv_img_to_pygame(opencv_image)

    #Draw image
    screen.blit(pygame_image, (0, 0))
    pygame.display.update()  #Update screen

    #Wait 5 seconds to finish
    time.sleep(5)
    pygame.quit()


if __name__ == '__main__':
    main()

However, the execution fails with the following error:

OpenCV Error: Assertion failed (scn == 3 || scn == 4) in cvtColor, file /build/opencv-L65chJ/opencv-3.2.0+dfsg/modules/imgproc/src/color.cpp, line 9716
Traceback (most recent call last):
  File "show-image.py", line 51, in <module>
    main()
  File "show-image.py", line 39, in main
    pygame_image = convert_opencv_img_to_pygame(opencv_image)
  File "show-image.py", line 17, in convert_opencv_img_to_pygame
    rgb_image = cv2.cvtColor(opencv_image, cv2.COLOR_BGR2RGB).swapaxes(0, 1)
cv2.error: /build/opencv-L65chJ/opencv-3.2.0+dfsg/modules/imgproc/src/color.cpp:9716: error: (-215) scn == 3 || scn == 4 in function cvtColor

The binarized image is a grayscale image with no color components. However, in the process of converting to an image for Pygame, there is a code that converts the color of the input image from BGR to RGB. Therefore, an error will occur in this part.

To solve this problem, for grayscale images, a branch process is required to change the second argument of the cv2.cvtColor () method to cv2.COLOR_GRAY2RGB.

show-image.py


import time

import cv2
import pygame


def get_opencv_img_res(opencv_image):
    height, width = opencv_image.shape[:2]
    return width, height

def convert_opencv_img_to_pygame(opencv_image):
    """
Convert OpenCV images for Pygame.

    see https://gist.github.com/radames/1e7c794842755683162b
    see https://github.com/atinfinity/lab/wiki/%5BOpenCV-Python%5D%E7%94%BB%E5%83%8F%E3%81%AE%E5%B9%85%E3%80%81%E9%AB%98%E3%81%95%E3%80%81%E3%83%81%E3%83%A3%E3%83%B3%E3%83%8D%E3%83%AB%E6%95%B0%E3%80%81depth%E5%8F%96%E5%BE%97
    """
    if len(opencv_image.shape) == 2:
        #For grayscale images
        cvt_code = cv2.COLOR_GRAY2RGB
    else:
        #In other cases:
        cvt_code = cv2.COLOR_BGR2RGB
    rgb_image = cv2.cvtColor(opencv_image, cvt_code).swapaxes(0, 1)
    #Generate a Surface for drawing images with Pygame based on OpenCV images
    pygame_image = pygame.surfarray.make_surface(rgb_image)

    return pygame_image

def main():
    #Load images with OpenCV
    image_path = '/usr/share/info/gnupg-module-overview.png'  #The path of the image file that was originally included in Raspbian Buster Lite
    opencv_image = cv2.imread(image_path)

    #Process image
    opencv_gray_image = cv2.cvtColor(opencv_image, cv2.COLOR_BGR2GRAY)
    ret, opencv_threshold_image = cv2.threshold(opencv_gray_image, 128, 255, cv2.THRESH_BINARY)
    opencv_image = opencv_threshold_image

    #Initialize Pygame
    pygame.init()
    width, height = get_opencv_img_res(opencv_image)
    screen = pygame.display.set_mode((width, height))

    #Convert OpenCV images for Pygame
    pygame_image = convert_opencv_img_to_pygame(opencv_image)

    #Draw image
    screen.blit(pygame_image, (0, 0))
    pygame.display.update()  #Update screen

    #Wait 5 seconds to finish
    time.sleep(5)
    pygame.quit()


if __name__ == '__main__':
    main()

You should now be able to see a binarized grayscale image as well.

Further speedup

When converting images of the same size and color many times, such as when processing camera images continuously, pygame.surfarray.blit_array () instead of the pygame.surfarray.make_surface () method You can speed it up using the method.

First, let's measure the traditional way of using the pygame.surfarray.make_surface () method for each conversion.

show-image.py


import statistics
import time

import cv2
import pygame


def get_opencv_img_res(opencv_image):
    height, width = opencv_image.shape[:2]
    return width, height

def convert_opencv_img_to_pygame(opencv_image):
    """
Convert OpenCV images for Pygame.

    see https://gist.github.com/radames/1e7c794842755683162b
    see https://github.com/atinfinity/lab/wiki/%5BOpenCV-Python%5D%E7%94%BB%E5%83%8F%E3%81%AE%E5%B9%85%E3%80%81%E9%AB%98%E3%81%95%E3%80%81%E3%83%81%E3%83%A3%E3%83%B3%E3%83%8D%E3%83%AB%E6%95%B0%E3%80%81depth%E5%8F%96%E5%BE%97
    """
    if len(opencv_image.shape) == 2:
        #For grayscale images
        cvt_code = cv2.COLOR_GRAY2RGB
    else:
        #In other cases:
        cvt_code = cv2.COLOR_BGR2RGB
    rgb_image = cv2.cvtColor(opencv_image, cvt_code).swapaxes(0, 1)
    #Generate a Surface for drawing images with Pygame based on OpenCV images
    pygame_image = pygame.surfarray.make_surface(rgb_image)

    return pygame_image

def main():
    #Load images with OpenCV
    image_path = '/usr/share/info/gnupg-module-overview.png'  #The path of the image file that was originally included in Raspbian Buster Lite
    opencv_image = cv2.imread(image_path)

    #Initialize Pygame
    pygame.init()
    width, height = get_opencv_img_res(opencv_image)
    screen = pygame.display.set_mode((width, height))

    #Convert OpenCV images for Pygame
    time_diff_list = []
    for _ in range(500):
      time_start = time.perf_counter()  #Start measurement
      pygame_image = convert_opencv_img_to_pygame(opencv_image)
      time_end = time.perf_counter()  #End of measurement
      time_diff_list.append(time_end - time_start)
    time_diff = statistics.mean(time_diff_list)
    print(f'Average conversion time: {time_diff}Seconds/ {1/time_diff}fps')

    #Draw image
    screen.blit(pygame_image, (0, 0))
    pygame.display.update()  #Update screen

    #Wait 1 second and finish
    time.sleep(1)
    pygame.quit()


if __name__ == '__main__':
    main()
Average conversion time: 0.01808052065800075 seconds/ 55.30814177950641fps

It is as fast as approaching 60fps. This is still fast enough, but it's a bit annoying when you think you're just converting.

Next, let's measure the speedup method using the pygame.surfarray.blit_array () method.

show-image.py


import statistics
import time

import cv2
import pygame


def get_opencv_img_res(opencv_image):
    height, width = opencv_image.shape[:2]
    return width, height

pygame_surface_cache = {}

def convert_opencv_img_to_pygame(opencv_image):
    """
Convert OpenCV images for Pygame.

    see https://gist.github.com/radames/1e7c794842755683162b
    see https://github.com/atinfinity/lab/wiki/%5BOpenCV-Python%5D%E7%94%BB%E5%83%8F%E3%81%AE%E5%B9%85%E3%80%81%E9%AB%98%E3%81%95%E3%80%81%E3%83%81%E3%83%A3%E3%83%B3%E3%83%8D%E3%83%AB%E6%95%B0%E3%80%81depth%E5%8F%96%E5%BE%97
    see https://stackoverflow.com/a/42589544/4907315
    """
    if len(opencv_image.shape) == 2:
        #For grayscale images
        cvt_code = cv2.COLOR_GRAY2RGB
    else:
        #In other cases:
        cvt_code = cv2.COLOR_BGR2RGB
    rgb_image = cv2.cvtColor(opencv_image, cvt_code).swapaxes(0, 1)

    #Get a generated Surface with the same image size from the cache
    cache_key = rgb_image.shape
    cached_surface = pygame_surface_cache.get(cache_key)

    if cached_surface is None:
        #Generate a Surface for drawing images with Pygame based on OpenCV images
        cached_surface = pygame.surfarray.make_surface(rgb_image)
        #Add Surface to cache
        pygame_surface_cache[cache_key] = cached_surface
    else:
        #If you find a Surface with the same image size, reuse the already generated Surface.
        pygame.surfarray.blit_array(cached_surface, rgb_image)

    return cached_surface

def main():
    #Load images with OpenCV
    image_path = '/usr/share/info/gnupg-module-overview.png'  #The path of the image file that was originally included in Raspbian Buster Lite
    opencv_image = cv2.imread(image_path)

    #Initialize Pygame
    pygame.init()
    width, height = get_opencv_img_res(opencv_image)
    screen = pygame.display.set_mode((width, height))

    #Convert OpenCV images for Pygame
    time_diff_list = []
    for _ in range(500):
      time_start = time.perf_counter()  #Start measurement
      pygame_image = convert_opencv_img_to_pygame(opencv_image)
      time_end = time.perf_counter()  #End of measurement
      time_diff_list.append(time_end - time_start)
    time_diff = statistics.mean(time_diff_list)
    print(f'Average conversion time: {time_diff}Seconds/ {1/time_diff}fps')

    #Draw image
    screen.blit(pygame_image, (0, 0))
    pygame.display.update()  #Update screen

    #Wait 1 second and finish
    time.sleep(1)
    pygame.quit()


if __name__ == '__main__':
    main()
Average conversion time: 0.013679669161995207 seconds/ 73.1011830884182fps

It has exceeded 60fps! If it is so fast, it will be practical enough.

How to convert OpenCV images for Pygame

def convert_opencv_img_to_pygame(opencv_image):
    """
Convert OpenCV images for Pygame.

    see https://gist.github.com/radames/1e7c794842755683162b
    see https://github.com/atinfinity/lab/wiki/%5BOpenCV-Python%5D%E7%94%BB%E5%83%8F%E3%81%AE%E5%B9%85%E3%80%81%E9%AB%98%E3%81%95%E3%80%81%E3%83%81%E3%83%A3%E3%83%B3%E3%83%8D%E3%83%AB%E6%95%B0%E3%80%81depth%E5%8F%96%E5%BE%97
    """
    if len(opencv_image.shape) == 2:
        #For grayscale images
        cvt_code = cv2.COLOR_GRAY2RGB
    else:
        #In other cases:
        cvt_code = cv2.COLOR_BGR2RGB
    rgb_image = cv2.cvtColor(opencv_image, cvt_code).swapaxes(0, 1)
    #Generate a Surface for drawing images with Pygame based on OpenCV images
    pygame_image = pygame.surfarray.make_surface(rgb_image)

    return pygame_image

If you want to convert images of the same size many times:

pygame_surface_cache = {}

def convert_opencv_img_to_pygame(opencv_image):
    """
Convert OpenCV images for Pygame.

    see https://gist.github.com/radames/1e7c794842755683162b
    see https://github.com/atinfinity/lab/wiki/%5BOpenCV-Python%5D%E7%94%BB%E5%83%8F%E3%81%AE%E5%B9%85%E3%80%81%E9%AB%98%E3%81%95%E3%80%81%E3%83%81%E3%83%A3%E3%83%B3%E3%83%8D%E3%83%AB%E6%95%B0%E3%80%81depth%E5%8F%96%E5%BE%97
    see https://stackoverflow.com/a/42589544/4907315
    """
    if len(opencv_image.shape) == 2:
        #For grayscale images
        cvt_code = cv2.COLOR_GRAY2RGB
    else:
        #In other cases:
        cvt_code = cv2.COLOR_BGR2RGB
    rgb_image = cv2.cvtColor(opencv_image, cvt_code).swapaxes(0, 1)

    #Get a generated Surface with the same image size from the cache
    cache_key = rgb_image.shape
    cached_surface = pygame_surface_cache.get(cache_key)

    if cached_surface is None:
        #Generate a Surface for drawing images with Pygame based on OpenCV images
        cached_surface = pygame.surfarray.make_surface(rgb_image)
        #Add Surface to cache
        pygame_surface_cache[cache_key] = cached_surface
    else:
        #If you find a Surface with the same image size, reuse the already generated Surface.
        pygame.surfarray.blit_array(cached_surface, rgb_image)

    return cached_surface

reference

Recommended Posts

How to draw OpenCV images in Pygame
How to collect images in Python
How to view images in Django's Admin
[Python] How to draw a histogram in Matplotlib
How to do zero-padding in one line with OpenCV
How to display multiple images of galaxies in tiles
How to get RGB and HSV histograms in OpenCV
[Python] How to do PCA in Python
How to handle session in SQLAlchemy
How to use classes in Theano
How to write soberly in pandas
How to update Spyder in Anaconda
How to use SQLite in Python
How to install OpenCV on Mac
How to convert 0.5 to 1056964608 in one shot
How to reflect CSS in Django
How to use Mysql in python
How to wrap C in Python
How to use ChemSpider in Python
How to use PubChem in Python
How to run TensorFlow 1.0 code in 2.0
How to call PyTorch in Julia
How to install OpenCV on Cloud9 and run it in Python
How to use calculated columns in CASTable
[Introduction to Python] How to use class in Python?
How to suppress display error in matplotlib
How to access environment variables in Python
How to dynamically define variables in Python
How to do R chartr () in Python
Mosaic images in various shapes (Python, OpenCV)
How to draw a graph using Matplotlib
How to convert csv to tsv in CLI
How to delete expired sessions in Django
[Itertools.permutations] How to put permutations in Python
How to implement nested serializer in drf-flex-fields
How to work with BigQuery in Python
How to execute commands in jupyter notebook
How to do'git fetch --tags' in GitPython
How to get a stacktrace in python
How to display multiplication table in python
How to reassign index in pandas dataframe
How to use Anaconda interpreter in PyCharm
How to handle consecutive values in MySQL
How to switch python versions in cloud9
How to adjust image contrast in Python
How to use __slots__ in Python class
How to collect face images relatively easily
How to dynamically zero pad in Python
How to use regular expressions in Python
How to implement Scroll View in pythonista 1
How to convert DateTimeField format in Django
How to use Map in Android ViewPager
How to display Hello world in python
How to read CSV files in Pandas
How to change editor color in PyCharm
How to write this process in Perl?
How to use is and == in Python
How to revive cells in iPython notebook
How to move BufferImageStim object in PsychoPy
Understand how to display images on Jupyter (utilization of imshow / matplotlib of OpenCV)
How to put OpenCV in Raspberry Pi and easily collect images of face detection results with Python