I made an app that warns me if I mess with my smartphone while studying with OpenCV

Introduction

I started studying Python and OpenCV. At that time, the information on Qiita was very helpful, so I will post it as a summary of what I learned. It's my first time to post, so I think there is something wrong with it, but I hope it will be helpful to those who read this.

What was made

It is an application that recognizes the smartphone placed on the desk by template matching, judges whether the power is on from the brightness near the center, and warns if the power is on for a certain period of time.

What I used

--WEB camera UCAM-C520FEBK

• Python 3.7.5
• pip 20.0.1
  • numpy 1.18.1
  • opencv-python 4.1.2.30
  • PyInstaller 3.6

Each module can be easily installed using pip. I referred to the following site for how to do it. https://qiita.com/fiftystorm36/items/1a285b5fbf99f8ac82eb (opencv-python, numpy) https://techacademy.jp/magazine/18963 (PyInstaller)

Development flow

1. Prepare an image to be used for warning and an image to be used for template matching.
2. Perform template matching between the image acquired by the WEB camera and the prepared image so that the position of the smartphone can be acquired.
3. When the brightness near the center of the smartphone is above a certain threshold, it is judged that the power is on, and when it counts a certain number of times, a warning image is displayed.
4. Make the created file into an exe

Image preparation

First, I downloaded the warning image online.

https://ameblo.jp/ufcrekishi/entry-10967280575.html

In addition, the template image was acquired using the following program.

temp.py

import numpy as np
import cv2

cap = cv2.VideoCapture(0)

while(True):
	#Read webcam image and rotate
	ret, frame = cap.read()
	img1 = cv2.flip(frame, -1)
	cv2.imshow("img", img1)

	#Press Esc to save and exit
	k = cv2.waitKey(0)

	if k == 27:
		cv2.imwrite('model.png',img1)
		break

cap.release()
cv2.destroyAllWindows()

This should give you model.png. From now on, if you trim only the part of the smartphone, you will get something like the following.

This completes the image preparation.

Get the position of your smartphone

The position of the smartphone was acquired by the following program.

caution.py

import numpy as np
import cv2

cap = cv2.VideoCapture(0)

while(True):
	#Read webcam image and rotate
	ret, frame = cap.read()
	img = cv2.flip(frame, -1)
	
	#Read the image of the smartphone taken in advance
	temp = cv2.imread('model.png')
	
	#Run matching template
	#The comparison method is cv2.TM_CCOEFF_Select NORMED
	result = cv2.matchTemplate(img, temp, cv2.TM_CCOEFF_NORMED)

	#Get the position and value of the detection area from the detection result
	#The upper left position of the area with high similarity is max_loc
	min_val, max_val, min_loc, max_loc = cv2.minMaxLoc(result)

	#Get the lower right and center positions of areas with high similarity
	#Use the previous loop position if the match is not successful
	if max_loc[0] < img.shape[1] - temp.shape[1] and \
	max_loc[1] < img.shape[0] - temp.shape[0]:
		topleft_x, topleft_y = max_loc
		h, w = temp.shape[:2]
		bottom_right = (topleft_x + w, topleft_y + h)
		center = (topleft_x + w / 2, topleft_y + h / 2)
		
	print(center)

	#Display the detection area surrounded by a blue rectangle
	result = img
	cv2.rectangle(result,(topleft_x, topleft_y), bottom_right, (255, 0, 0), 2)
	cv2.imshow("img", result)

	#Wait for 1 second and finish when there is an Esc key input
	k = cv2.waitKey(1000)
	if k == 27:
		cv2.imwrite("temp.png ", result)
		break

cap.release()
cv2.destroyAllWindows()

If template matching is not possible first, center cannot be printed and an error will occur, so it is necessary to turn off the power first. If template matching is done well, it should be as follows.

Determine if the power is on from the brightness

The following program determines whether the power is on and displays an image warning that the power is on for a long time.

caution.py

import numpy as np
import cv2

cap = cv2.VideoCapture(0)
count = 0

while(True):
	#Read webcam image and rotate
	ret, frame = cap.read()
	img = cv2.flip(frame, -1)
	
	#Read the image of the smartphone taken in advance
	temp = cv2.imread('model.png')
	
	#Run matching template
	#The comparison method is cv2.TM_CCOEFF_Select NORMED
	result = cv2.matchTemplate(img, temp, cv2.TM_CCOEFF_NORMED)

	#Get the position and value of the detection area from the detection result
	#The upper left position of the area with high similarity is max_loc
	min_val, max_val, min_loc, max_loc = cv2.minMaxLoc(result)

	#Get the lower right and center positions of areas with high similarity
	#Use the previous loop position if the match is not successful
	if max_loc[0] < img.shape[1] - temp.shape[1] and \
	max_loc[1] < img.shape[0] - temp.shape[0]:
		topleft_x, topleft_y = max_loc
		h, w = temp.shape[:2]
		bottom_right = (topleft_x + w, topleft_y + h)
		center = (topleft_x + w / 2, topleft_y + h / 2)
		
	print(center)

	#HSV conversion
	hsv_image = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)

	#Determine if you are using a smartphone based on the brightness in the center of the acquired points, and count when using it.
	#Surrounding brightness is also used to prevent false detection
	value = 0

	for i in range(-15, 16):
		for j in range(-15, 16):
			value = value + hsv_image[int(center[0]) + i][int(center[1]) + j][2]
	v_ave = value / 961

	if v_ave > 200:
		print("ON,brightness:", v_ave)
		count = count + 5
	else:
		print("OFF,brightness:", v_ave)

	#Display the detection area surrounded by a blue rectangle
	#The center position is indicated by a red circle
	result = img
	cv2.rectangle(result,(topleft_x, topleft_y), bottom_right, (255, 0, 0), 2)
	cv2.circle(result, (int(center[0]), int(center[1])), 5, (0, 0, 255))
	cv2.imshow("img", result)

	#An image is displayed when the count exceeds a certain level
	if count > 50:
		img4 = cv2.imread('studyordie.jpg')
		cv2.imshow('studyordie', img4)
		cv2.moveWindow('studyordie', 555, 0)
		cv2.waitKey(0)
		count = 0
		break

	print("count = ", count)
	count = count - 1
	if count < 0:
		count = 0

	#Wait for 1 second and finish when there is an Esc key input
	k = cv2.waitKey(1000)
	if k == 27:
		break

cap.release()
cv2.destroyAllWindows()

If you do this, you should see the following console at the same time as the matching image. A warning image is displayed when the value of this count exceeds 50.

(157.0, 193.0)
OFF,Brightness: 159.49739854318418
count =  0

(225.0, 260.0)
ON,Brightness: 203.90426638917793
count =  15

Here, when calculating the brightness, the average is taken for a 31 x 31 square centered on the center of matching, but if this is only one point, is it the reflection of the fluorescent lamp? Because a large value was returned. I think that the threshold value will change depending on the environment in which it is used.

exe conversion

I wanted to be able to use it just by double-clicking the shortcut on the desktop, so I made it into an exe. With Pyinstaller installed

pyinstaller caution.py --onefile

It can be converted to an exe with the command of. Please note that you need to put the prepared image in the dist folder after converting it to an exe. Create a shortcut and paste it on your desktop and you're done.