Visualize the appreciation status of art works with OpenCV

Overview

• There was a request from an artist in Fukuoka to create a system that visualizes the value of art works.
• We examined how many people and how long the art work could be viewed with the cheapest system possible.
• It was decided to install Raspberry Pi in front of the art work and accumulate the time when face was detected by OpenCV.
• Ambient (https://ambidata.io) was used as the data visualization service.
• Data was uploaded via Wifi on Raspberry Pi.
• I decided not to send camera images to the Internet, but only to send numerical data.

It seems that it was installed and operated at the startup support facility FGN (https://growth-next.com) in Fukuoka City.

What to prepare

• Raspberry Pi3 Model B (4 is probably okay)
• Case for Raspberry Pi
• Heat sink for Raspberry Pi
• Raspberry Pi3 Model B B + compatible power supply set (5V 3.0A)
• Camera for raspberry Pi
• Monitor or TV with HDMI input
• 100-yen shop tripod

Preparation

• Set up the Raspberry Pi so that you can connect to Wifi. https://www.raspberrypi.org/downloads/
• Make the OpenCV and Ambient (IoT visualization service) libraries available.
• Create a dashboard in Ambient and get the channel ID and light key. https://ambidata.io/refs/python/

programming

• Uses OpenCV and Ambient (IoT visualization service) libraries
• Face detection frame, number of visits every 5 minutes, total stay time overlaid on camera image
• Number of visits every 5 minutes, total stay time Send to Ambient for visualization
• The number of visits is roughly estimated from the time when the face was detected.
• Since it was an exhibition for about 3 weeks, we also acquired the CPU temperature and sent it to Ambient.

detect_face_camera.py

import sys

sys.path.append('/home/pi/.local/lib/python2.7/site-packages')

import numpy as np

import cv2

import time

import datetime

import ambient

args = sys.argv

face_cascade = cv2.CascadeClassifier('haarcascade_frontalface_default.xml')

eye_cascade = cv2.CascadeClassifier('haarcascade_eye.xml')

cap = cv2.VideoCapture(0)

flg = 0

stime =  0

am = ambient.Ambient(xxxx, 'xxxxxxxxxxxxxxxx')  #Ambient channel ID/Light key(Number of visits / stay time)

am2 = ambient.Ambient(xxxx, 'xxxxxxxxxxxxxxxx')  #Ambient channel ID/Light key(CPU temperature)

acnt = 0

while(True):

    t3 = time.time()

    cnt = 0

    ret, frame = cap.read()

    #Get the height and width of the loaded image

    height = frame.shape[0]

    width = frame.shape[1]

    import sys

    gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

    faces = face_cascade.detectMultiScale(gray, 1.3, 5)

    for (x,y,w,h) in faces:

        t1 = time.time()

        cv2.rectangle(frame,(x,y),(x+w,y+h),(152,145,234),2)

        roi_gray = gray[y:y+h, x:x+w]

        roi_color = frame[y:y+h, x:x+w]

        eyes = eye_cascade.detectMultiScale(roi_gray)

        cnt += 1

        for (ex,ey,ew,eh) in eyes:

            cv2.rectangle(roi_color,(ex,ey),(ex+ew,ey+eh),(108,97,232),2)

        t2 = time.time()

        stime += (t2 - t1) * 1000 / 60

        acnt = acnt + cnt

    cv2.putText(frame, 'count: ' + str(acnt / 100), (10, 420), cv2.FONT_HERSHEY_DUPLEX, 1.0, (0,255,0), thickness=2)

    cv2.putText(frame, 'stay[sec]: ' + str('{:.1f}'.format(stime)), (10, 460), cv2.FONT_HERSHEY_DUPLEX, 1.0, (0,255,0), thickness=2)

    cv2.putText(frame, 'senseart v.0.0.1b', (410, 20), cv2.FONT_HERSHEY_PLAIN, 1.5, (0,255,0), thickness=2)

    resized_img = cv2.resize(frame,(width*2, height*2))

    cv2.imshow('senceart (v.0.0.1 beta)', resized_img)

    if cv2.waitKey(1) & 0xFF == ord('q'):

        break

    t4 = time.time()

    f = open("/sys/class/thermal/thermal_zone0/temp","r")  #CPU temperature

    for t in f:

        tmp = t[:2]+"."+t[2:5]

    f.close()

    fps = cap.get(cv2.CAP_PROP_FPS)

    now = datetime.datetime.now() 

    minute = '{0:%M}'.format(now)

    second = '{0:%S}'.format(now)

    print('count', acnt / 100)

    print('stay', stime)

    print("minute", minute)

    print("second", second)

    print("temp", tmp)

    print("\n")

    if int(minute) % 5 == 0 and int(second) == 0:

        r = am.send({'d1': acnt / 100, 'd2': stime})  #Send visits and length of stay to Ambient

        r = am2.send({'d1': tmp})  #Send CPU temperature to Ambient

        r.close()
        acnt = 0

        cnt = 0

        print("******************sended*****************\n")

        continue

    if int(minute) % 10 == 0 and int(second) == 0:

        r = am.send({'d1': acnt / 100, 'd2': stime})  #Send visits and length of stay to Ambient

        r = am2.send({'d1': tmp})  #Send CPU temperature to Ambient

        r.close()

        acnt = 0

        cnt = 0

        print("******************sended*****************\n")

        continue

cap.release()

• Place a shell script on the desktop of Raspberry Pi so that the Python program can be started.
• As an on-site operation, double-click the icon to start it.

senceart.sh

#! /bin/bash
cd /home/pi/opencv_programs/
sudo python3 detect_face_camera.py

important point

• Initially, I was connected to Fukuoka City's public Free Wifi (no time limit), but I switched to a Wifi router because it was disconnected in a few days. (Cause unknown…)
• Remote operation software such as VNC is required for remote maintenance (Hiroshima → Fukuoka this time).
• A mouse, keyboard, and manual are required so that local facility staff can handle problems to some extent.

Try using

• It went down only once in 3 weeks, but it moved almost stably.
• Since it was experimental, the logic for counting the number of visits is not complete. It is necessary to carry out field verification many times.
• I couldn't say that the operation was light, so I'm curious about how much performance will be improved with the Raspberry Pi 4.