Realize a super IoT house by acquiring sensor data in the house with Raspberry Pi

Introduction

Data management by the supercity method is a hot topic in the streets, As an indoor sect, let's ride the wave of the times at the house in front of the city! I thought, ** We have created a mechanism to display a large amount of sensor data on the dashboard in real time **.

** From the conclusion, I feel that we have built a system that can visualize sensor data in a stable manner **.

I would like to try to create an article without a leap of explanation so that even beginners can easily understand it. ** I would appreciate it if you could point out any jumps or mistakes! ** **

Things necessary

** ・ RaspberryPi ** (Pi3Model B is used this time) **-Python execution environment ** (preset Python 3.7.3 is used this time)

• As a result of trial and error, the Python development environment on Raspberry Pi settled down to here. ** ・ Google account ** (required to use spreadsheet) ** ・ Smartphone ** (for BLE Scanner setting) ** ・ Various sensors → 6 types of sensors are used. Details below **
• It is not necessary to prepare all 6 types, so please mount as appropriate according to the purchased sensor.

List of sensors used

・ Omron Environment Sensor BAG Type (2JCIE-BL01) ・ Omron Environment Sensor USB Type (2JCIE-BU01) ・ [Inkbird IBS-TH1 mini](https://www.amazon.co.jp/Inkbird-%E3%83%96%E3%83%AB%E3%83%BC%E3%83%88%E3% 82% A5% E3% 83% BC% E3% 82% B9-% E6% B8% A9% E6% B9% BF% E5% BA% A6% E3% 82% BB% E3% 83% B3% E3% 82 % B5% E3% 83% BC-Android-IBS-TH1 / dp / B07D74CQLJ) ・ [Inkbird IBS-TH1](https://www.amazon.co.jp/Inkbird-IBS-TH1-Bluetooth-%E5%A4%96%E9%83%A8%E3%83%97%E3%83 % AD% E3% 83% BC% E3% 83% 96% E4% BB% 98% E3% 81% 8D-% E3% 82% B9% E3% 83% 9E% E3% 83% 9B% E3% 81% A8% E9% 80% A3% E6% 8E% A5% E3% 81% A7% E3% 81% 8D / dp / B07D75H734 / ref = pd_aw_sbs_201_3 / 356-7389076-5979439?_encoding=UTF8&pd_rd_i=B07D75H734&pd_rd_r=184 4e16-8524-c984893af9bd & pd_rd_w = Ip0MD & pd_rd_wg = lcDuD & pf_rd_p = bff3a3a6-0f6e-4187-bd60-25e75d4c1c8f & pf_rd_r = 939W116VQ9B4YV5DQ69A & ps ・ SwitchBot Thermo-Hygrometer ・ Nature Remo

System configuration

Overall configuration diagram

The overall structure is as shown in the figure below. The arrows represent the flow of data.

Feature

Periodic execution

Since long-term operation is difficult with manual script execution, we created a mechanism for automatic periodic execution. The script sensors_to_spreadsheet.py that controls the whole is periodically executed by the cron of Raspberry Pi to accumulate data.

Wireless communication between sensor and Raspberry Pi

Basically, it communicates with Bluetooth Low Energy (BLE). BLE is a standard that is becoming the de facto standard for home IoT devices, and the points to note are summarized in this article in an easy-to-understand manner. [Details below](https://qiita.com/c60evaporator/items/283d0569eba58830f86e#%E5%90%84%E7%A8%AE%E3%82%A8%E3%83%A9%E3%83%BC % E5% AF% BE% E5% BF% 9C), but stabilization of BLE was the biggest barrier to this development. In addition, only Nature Remo is connected by WiFi instead of BLE.

Data storage destination

The data storage destination is divided into two systems, local CSV storage and Google Spreadsheets. CSV: later used for analysis Google Spreadsheets: Used for visualization on the data portal (I wanted to visualize on the cloud) I divided it on the assumption that it will be used, but I think it is redundant, so I think it is okay to unify it into one system. When unifying, I think that it is ideal to be able to save in a DB that can be accessed on the cloud.

procedure

The above system was built by the following procedure ** ① Initial setting of sensor and operation check ** ** ② Create sensor data acquisition class ** ** ③ Create main script ** ** ④ Hit the GAS API from Python to write data to the spreadsheet ** ** ⑤ Script execution regularly ** ** ⑥ Visualization by Google Data Portal **

Details will be explained in the next chapter

Details

① Initial setting of sensor and operation check

Since I created the article individually, please perform the initial setting and operation check for each sensor.

・ Omron environment sensor BAG type (2JCIE-BL01) ・ Omron Environmental Sensor USB Type (2JCIE-BU01) ・ Inkbird IBS-TH1 mini ・ Inkbird IBS-TH1 ・ SwitchBot Thermo-Hygrometer ・ Nature Remo

(2) Create a sensor data acquisition class

Create a Python class to get sensor data for each sensor type. (Inkbird IBS-TH1 and IBS-TH1 mini are acquired in the same class. ")

In addition, modules (.py files) are organized by manufacturer. Below, we will explain each module (by manufacturer) in separate sections.

Omron environment sensor

Create a data acquisition class for Omron 2JCIE-BL01 & 2JCIE-BU01. The content is generally [BAG type](https://qiita.com/c60evaporator/items/ed2ffde4c87001111c12#%E3%82%BB%E3%83%B3%E3%82%B5%E6%B8%AC%E5%AE%9A% E5% 80% A4% E5% 8F% 96% E5% BE% 97% E7% 94% A8% E3% 82% AF% E3% 83% A9% E3% 82% B9% E3% 82% 92% E4% BD% 9C% E6% 88% 90) [USB type](https://qiita.com/c60evaporator/items/ced01c241031c6c9c379#%E3%82%BB%E3%83%B3%E3%82%B5%E5%80%A4%E5%8F%96% E5% BE% 97% E3% 82% B9% E3% 82% AF% E3% 83% AA% E3% 83% 97% E3% 83% 88% E3% 81% AE% E4% BD% 9C% E6% 88% 90) Is a combination of

** ・ Calling method from the outside BAG type: ** Create an instance of OmniBroadcastScanDelegate class and

Instance name.scan(timeout)

You can get the sensor data with. For the argument "timeout", specify the data acquisition timeout value in seconds (roughly, 3 to 5 seconds seems to be good).

** USB type: ** You can get the sensor data by the following method

GetOmronConnectModeData().get_env_usb_data(macaddr)

For the argument "macaddr", specify the Mac address of the sensor.

**-Actual script **

omron_env.py

# coding: UTF-8
from bluepy import btle
import struct

#Delegate for acquiring Broadcast data (BAG type)
class OmronBroadcastScanDelegate(btle.DefaultDelegate):
    #constructor
    def __init__(self):
        btle.DefaultDelegate.__init__(self)
        #Variables for holding sensor data
        self.sensorValue = None

    #Scan handler
    def handleDiscovery(self, dev, isNewDev, isNewData):  
        #When a new device is found
        if isNewDev or isNewData:  
            #Extract advertisement data
            for (adtype, desc, value) in dev.getScanData():  
                #Data retrieval is executed when the environment sensor is used.
                if desc == 'Manufacturer' and value[0:4] == 'd502':
                    #Take out the sensor type (EP or IM)
                    sensorType = dev.scanData[dev.SHORT_LOCAL_NAME].decode(encoding='utf-8')
                    #Extraction of sensor data for EP
                    if sensorType == 'EP':
                        self._decodeSensorData_EP(value)
                    #Extraction of sensor data during IM
                    if sensorType == 'IM':
                        self._decodeSensorData_IM(value)

    #Take out sensor data and convert it to dict format (in EP mode)
    def _decodeSensorData_EP(self, valueStr):
        #Sensor data from character strings(6th and subsequent characters)Only take out and convert to binary
        valueBinary = bytes.fromhex(valueStr[6:])
        #Convert binary sensor data to integer Tuple
        (temp, humid, light, uv, press, noise, discomf, wbgt, rfu, batt) = struct.unpack('<hhhhhhhhhB', valueBinary)
        #Store in dict type after unit conversion
        self.sensorValue = {
            'SensorType': 'Omron_BAG_EP',
            'Temperature': temp / 100,
            'Humidity': humid / 100,
            'Light': light,
            'UV': uv / 100,
            'Pressure': press / 10,
            'Noise': noise / 100,
            'Discomfort': discomf / 100,
            'WBGT': wbgt / 100,
            'BatteryVoltage': (batt + 100) / 100
        }
    
    #Take out sensor data and convert it to dict format (in IM mode)
    def _decodeSensorData_IM(self, valueStr):
        #Sensor data from character strings(6th and subsequent characters)Only take out and convert to binary
        valueBinary = bytes.fromhex(valueStr[6:])
        #Convert binary sensor data to integer Tuple
        (temp, humid, light, uv, press, noise, accelX, accelY, accelZ, batt) = struct.unpack('<hhhhhhhhhB', valueBinary)
        #Store in dict type after unit conversion
        self.sensorValue = {
            'SensorType': 'Omron_BAG_IM',
            'Temperature': temp / 100,
            'Humidity': humid / 100,
            'Light': light,
            'UV': uv / 100,
            'Pressure': press / 10,
            'Noise': noise / 100,
            'AccelerationX': accelX / 10,
            'AccelerationY': accelY / 10,
            'AccelerationZ': accelZ / 10,
            'BatteryVoltage': (batt + 100) / 100
        }

#Connect mode (USB type) data acquisition class
class GetOmronConnectModeData():
    def get_env_usb_data(self, macaddr):
        peripheral = btle.Peripheral(macaddr, addrType=btle.ADDR_TYPE_RANDOM)
        characteristic = peripheral.readCharacteristic(0x0059)
        return self._decodeSensorData_EP(characteristic)

    def _decodeSensorData_EP(self, valueBinary):
        (seq, temp, humid, light, press, noise, eTVOC, eCO2) = struct.unpack('<Bhhhlhhh', valueBinary)
        sensorValue = {
                'SensorType': 'Omron_USB_EP',
                'Temperature': temp / 100,
                'Humidity': humid / 100,
                'Light': light,
                'Pressure': press / 1000,
                'Noise': noise / 100,
                'eTVOC': eTVOC,
                'eCO2': eCO2
            }
        return sensorValue

Inkbird IBS-TH1 & IBS-TH1 mini Create a data acquisition class for Inkbird IBS-TH1 & IBS-TH1 mini. The content is generally IBS-TH1 IBS-TH1 mini Is a combination of

** ・ Calling method from the outside ** Inkbird_IBSTH1 and Inkbird_IBSTH1mini can both get sensor data by the following methods.

GetIBSTH1Data().get_ibsth1_data(macaddr, sensortype)

For the argument "macaddr", specify the Mac address of the sensor, and for "sensortype", specify'Inkbird_IBSTH1mini'or'Inkbird_IBSTH1' according to the sensor type.

**-Actual script **

inkbird_ibsth1.py

from bluepy import btle
import struct

#Inkbird IBS-TH1 data acquisition class
class GetIBSTH1Data():
    def get_ibsth1_data(self, macaddr, sensortype):
        #Connect to device
        peripheral = btle.Peripheral(macaddr)
        #IBS-For TH1 mini
        if sensortype == 'Inkbird_IBSTH1mini':
            characteristic = peripheral.readCharacteristic(0x002d)
            return self._decodeSensorData_mini(characteristic)
        #IBS-At TH1
        elif sensortype == 'Inkbird_IBSTH1':
            characteristic = peripheral.readCharacteristic(0x28)
            return self._decodeSensorData(characteristic)
        else:
            return None
    
    #IBS-TH1 mini
    def _decodeSensorData_mini(self, valueBinary):
        (temp, humid, unknown1, unknown2, unknown3) = struct.unpack('<hhBBB', valueBinary)
        sensorValue = {
                'SensorType': 'Inkbird_IBSTH1mini',
                'Temperature': temp / 100,
                'Humidity': humid / 100,
                'unknown1': unknown1,
                'unknown2': unknown2,
                'unknown3': unknown3,
            }
        return sensorValue

    #IBS-TH1
    def _decodeSensorData(self, valueBinary):
        (temp, humid, unknown1, unknown2, unknown3) = struct.unpack('<hhBBB', valueBinary)
        sensorValue = {
                'SensorType': 'Inkbird_IBSTH1',
                'Temperature': temp / 100,
                'Humidity': humid / 100,
                'unknown1': unknown1,
                'unknown2': unknown2,
                'unknown3': unknown3,
            }
        return sensorValue

SwitchBot Thermo-Hygrometer

Create a data acquisition class for the SwitchBot thermo-hygrometer. The contents are [here](https://qiita.com/c60evaporator/items/7c3156a6bbb7c6c59052#%E3%82%BB%E3%83%B3%E3%82%B5%E5%80%A4%E5%8F%96 % E5% BE% 97% E3% 82% B9% E3% 82% AF% E3% 83% AA% E3% 83% 97% E3% 83% 88% E3% 81% AE% E4% BD% 9C% E6 Same as% 88% 90)

** ・ Calling method from the outside ** Create an instance of the SwitchbotScanDelegate class

Instance name.scan(timeout)

You can get the sensor data with. For the argument "timeout", specify the data acquisition timeout value in seconds (roughly, about 5 seconds seems to be good).

**-Actual script **

switchbot.py

from bluepy import btle
import struct

#Broadcast data acquisition delegate
class SwitchbotScanDelegate(btle.DefaultDelegate):
    #constructor
    def __init__(self, macaddr):
        btle.DefaultDelegate.__init__(self)
        #Variables for holding sensor data
        self.sensorValue = None
        self.macaddr = macaddr

    #Scan handler
    def handleDiscovery(self, dev, isNewDev, isNewData):
        #If you find a device with the target Mac address
        if dev.addr == self.macaddr:
            #Extract advertisement data
            for (adtype, desc, value) in dev.getScanData():  
                #Data retrieval is executed when the environment sensor is used.
                if desc == '16b Service Data':
                    #Extract sensor data
                    self._decodeSensorData(value)

    #Extract sensor data and convert to dict format
    def _decodeSensorData(self, valueStr):
        #Sensor data from character strings(4th and subsequent characters)Only take out and convert to binary
        valueBinary = bytes.fromhex(valueStr[4:])
        #Convert binary sensor data to numbers
        batt = valueBinary[2] & 0b01111111
        isTemperatureAboveFreezing = valueBinary[4] & 0b10000000
        temp = ( valueBinary[3] & 0b00001111 ) / 10 + ( valueBinary[4] & 0b01111111 )
        if not isTemperatureAboveFreezing:
            temp = -temp
        humid = valueBinary[5] & 0b01111111
        #Store in dict type
        self.sensorValue = {
            'SensorType': 'SwitchBot',
            'Temperature': temp,
            'Humidity': humid,
            'BatteryVoltage': batt
        }

Nature Remo In addition to sensor data, air conditioner data is also acquired through the remote control.

** ・ Calling method from the outside Sensor data acquisition: ** You can get the sensor data by the following method

get_sensor_data(Token, API_URL)

The argument "Token" is the Remo access token (reference), and the "API_URL" is "https://api.nature.global/" (fixed). Value)

** Air conditioner data acquisition **: You can get the sensor data by the following method

get_aircon_data(Token, API_URL)

The arguments are the same as when acquiring sensor data.

**-Actual script **

remo.py

import json
import requests
import glob
import pandas as pd

#Remo data acquisition class
class GetRemoData():
    def get_sensor_data(self, Token, API_URL):
        headers = {
            'accept': 'application/json',
            'Authorization': 'Bearer ' + Token,
        }
        response = requests.get(f"{API_URL}/1/devices", headers=headers)
        rjson = response.json()
        return self._decodeSensorData(rjson)

    def get_aircon_data(self, Token, API_URL):
        headers = {
            'accept': 'application/json',
            'Authorization': 'Bearer ' + Token,
        }
        response = requests.get(f"{API_URL}/1/appliances", headers=headers)
        rjson = response.json()
        return self._decodeAirconData(rjson)

    def calc_human_motion(self, Human_last, csvdir):
        filelist = glob.glob(f"{csvdir}/*/*.csv")
        if len(filelist) == 0:
            return 0
        filelist.sort()
        df = pd.read_csv(filelist[-1])
        if df.Human_last[len(df) - 1] != Human_last:
            return 1
        else:
            return 0

    #Extract sensor data and convert to dict format
    def _decodeSensorData(self, rjson):
        for device in rjson:
            #Select Remo (to avoid selecting Remo E by mistake)
            if device['firmware_version'].split('/')[0] == 'Remo':
                sensorValue = {
                    'SensorType': 'Remo_Sensor',
                    'Temperature': device['newest_events']['te']['val'],
                    'Humidity': device['newest_events']['hu']['val'],
                    'Light': device['newest_events']['il']['val'],
                    'Human_last': device['newest_events']['mo']['created_at']
                }
        return sensorValue

    #Take out air conditioner data and convert it to dict format
    def _decodeAirconData(self, rjson):
        for appliance in rjson:
            if appliance['type'] == 'AC':
                Value = {
                    'TempSetting': appliance['settings']['temp'],
                    'Mode': appliance['settings']['mode'],
                    'AirVolume': appliance['settings']['vol'],
                    'AirDirection': appliance['settings']['dir'],
                    'Power': appliance['settings']['button']
                }
                break
        return Value

③ Create main script

Overall structure

Call the class (2) and create a main script to acquire sensor data. The structure is divided into the following 5 types of methods and structures.

** 1) Method to execute class ② ** getdata_omron_bag (device): Omron environment sensor BAG type sensor value acquisition getdata_omron_usb (device): Omron environment sensor USB type sensor value acquisition getdata_ibsth1 (device): Get sensor value of Inkbird IBS-TH1 & IBS-TH1 mini getdata_switchbot_thermo (device): Get sensor value of SwitchBot thermo-hygrometer getdata_remo (device, csvpath): Get Nature Remo sensor value & air conditioner data The meaning of the argument is as follows device: Specify the line read from DeviceList.csv. csvpath: CSV storage path (Remo motion sensor needs to be compared with CSV saved past data)

** 2) CSV output method ** output_csv (data, csvpath): Outputs the measured value data for each sensor in CSV format. The meaning of the argument is as follows data: Data acquired from the sensor by the method of 1 csvpath: CSV save path

** 3) SpreadSheet output method ** output_spreadsheet (all_values_dict): Posts the measured data for all sensors to a spreadsheet. The meaning of the argument is as follows all_values_dict: Data of all sensors held in dict format (assuming the structure below)

all_values_dict = {
Sensor name 1:{
Column name 1:Value 1,
Column name 2:Value 2,
          ：
    },
Sensor name 2:{
Column name 1:Value 1,
Column name 2:Value 2,
          ：
    },
    ：
}

** 4) Bluetooth adapter restart method ** restart_hci0 (devicename): This is the "Bug 2" countermeasure method described at the bottom. The meaning of the argument is as follows devicename: device name (used for log output)

** 5) Main processing part if name == after'main': Executes a series of processes as shown in the flow below.

setting file

As the number of sensors increased and I felt that there was a limit to management with in-code, I created the following two types of configuration files. ** ・ DeviceList.csv: Describe necessary information for each sensor **

DeviceList.csv

DeviceName,SensorType,MacAddress,Timeout,Retry,Offset_Temp,Offset_Humid,API_URL,Token
SwitchBot_Thermo1,SwitchBot_Thermo,[SwitchBot Mac address],4,3,0,0,,
Inkbird_IBSTH1_Mini1,Inkbird_IBSTH1mini,[IBS-TH1 mini Mac address],0,2,0,0,,
Inkbird_IBSTH1_1,Inkbird_IBSTH1,[IBS-TH1 Mac address],0,2,0,0,,
Remo1,Nature_Remo,,0,2,0,0,https://api.nature.global/,[Nature Remo access token]
Omron_USB1,Omron_USB_EP,[Omron USB type Mac address],0,2,0,0,,
Omron_BAG1,Omron_BAG_EP,[Omron BAG type Mac address],3,2,0,0,,

The meaning of the columns is as follows DeviceName: Manages device names and identifies when there are multiple sensors of the same type SensorType: The type of sensor. Separate the sensor data acquisition classes to be executed according to this value MacAddress: MAC address of the sensor Timeout: Timeout value during scanning (broadcast mode sensor only) Retry: Maximum number of retries [Details](https://qiita.com/c60evaporator/items/283d0569eba58830f86e#%E4%B8%8D%E5%85%B7%E5%90%881peripheral%E3%81%AE%E5 % 88% 9D% E6% 9C% 9F% E5% 8C% 96% E6% 99% 82% E3% 81% AB% E3% 82% A8% E3% 83% A9% E3% 83% BC% E3% 81 % 8C% E5% 87% BA% E3% 82% 8B) Offset_Temp: Temperature offset value (currently unused) Offset_Humid: Humidity offset value (currently unused) API_URL: API URL (only used by Nature Remo) Token: Access token (only used by Nature Remo)

** ・ config.ini: Specify CSV and log output directory **

config.ini

[Path]
CSVOutput = /share/Data/Sensor
LogOutput = /share/Log/Sensor

If you output both in the shared folder created by samba, it is convenient to access from outside the Raspberry Pi.

Actual script

sensors_to_spreadsheet.py

from bluepy import btle
from omron_env import OmronBroadcastScanDelegate, GetOmronConnectModeData
from inkbird_ibsth1 import GetIBSTH1Data
from switchbot import SwitchbotScanDelegate
from remo import GetRemoData
from datetime import datetime, timedelta
import os
import csv
import configparser
import pandas as pd
import requests
import logging
import subprocess

#Global variables
global masterdate

######Acquisition of value of OMRON environment sensor (BAG type)######
def getdata_omron_bag(device):
    #Maximum device when no value is available.Retry Repeat scan
    for i in range(device.Retry):
        #omron_Set the sensor value acquisition delegate of env to execute at scan time
        scanner = btle.Scanner().withDelegate(OmronBroadcastScanDelegate())
        #Scan to get sensor value
        try:
            scanner.scan(device.Timeout)
        #If an error occurs in the scan, restart the Bluetooth adapter
        except:
            restart_hci0(device.DeviceName)
        #End the loop when the value can be obtained
        if scanner.delegate.sensorValue is not None:
            break
        #If the value cannot be obtained, write it to the log
        else:
            logging.warning(f'retry to get data [loop{str(i)}, date{str(masterdate)}, device{device.DeviceName}, timeout{device.Timeout}]')
    
    #If the value can be obtained, store the POST data in dict
    if scanner.delegate.sensorValue is not None:
        #Data to POST
        data = {        
            'DeviceName': device.DeviceName,        
            'Date_Master': str(masterdate),
            'Date': str(datetime.today()),
            'Temperature': str(scanner.delegate.sensorValue['Temperature']),
            'Humidity': str(scanner.delegate.sensorValue['Humidity']),
            'Light': str(scanner.delegate.sensorValue['Light']),
            'UV': str(scanner.delegate.sensorValue['UV']),
            'Pressure': str(scanner.delegate.sensorValue['Pressure']),
            'Noise': str(scanner.delegate.sensorValue['Noise']),
            'BatteryVoltage': str(scanner.delegate.sensorValue['BatteryVoltage'])
        }
        return data
    #If the value could not be obtained, output the log and restart the Bluetooth adapter.
    else:
        logging.error(f'cannot get data [date{str(masterdate)}, device{device.DeviceName}, timeout{device.Timeout}]')
        restart_hci0(device.DeviceName)
        return None

######Data acquisition of OMRON environmental sensor (USB type)######
def getdata_omron_usb(device):
    #Maximum device when no value is available.Retry Repeat scan
    for i in range(device.Retry):
        try:
            sensorValue = GetOmronConnectModeData().get_env_usb_data(device.MacAddress)
        #Log output if an error occurs
        except:
            logging.warning(f'retry to get data [loop{str(i)}, date{str(masterdate)}, device{device.DeviceName}]')
            sensorValue = None
            continue
        else:
            break
    
    #If the value can be obtained, store the POST data in dict
    if sensorValue is not None:
        #Data to POST
        data = {        
            'DeviceName': device.DeviceName,        
            'Date_Master': str(masterdate),
            'Date': str(datetime.today()),
            'Temperature': str(sensorValue['Temperature']),
            'Humidity': str(sensorValue['Humidity']),
            'Light': str(sensorValue['Light']),
            'Pressure': str(sensorValue['Pressure']),
            'Noise': str(sensorValue['Noise']),
            'eTVOC': str(sensorValue['eTVOC']),
            'eCO2': str(sensorValue['eCO2'])
        }
        return data
    #If the value could not be obtained, output the log and restart the Bluetooth adapter.
    else:
        logging.error(f'cannot get data [loop{str(device.Retry)}, date{str(masterdate)}, device{device.DeviceName}]')
        restart_hci0(device.DeviceName)
        return None

######Inkbird IBS-TH1 data acquisition######
def getdata_ibsth1(device):
    #Maximum device when no value is available.Retry Repeat scan
    for i in range(device.Retry):
        try:
            sensorValue = GetIBSTH1Data().get_ibsth1_data(device.MacAddress, device.SensorType)
        #Log output if an error occurs
        except:
            logging.warning(f'retry to get data [loop{str(i)}, date{str(masterdate)}, device{device.DeviceName}]')
            sensorValue = None
            continue
        else:
            break

    if sensorValue is not None:
        #Data to POST
        data = {        
            'DeviceName': device.DeviceName,        
            'Date_Master': str(masterdate),
            'Date': str(datetime.today()),
            'Temperature': str(sensorValue['Temperature']),
            'Humidity': str(sensorValue['Humidity']),
        }
        return data
    #If the value could not be obtained, output the log and restart the Bluetooth adapter.
    else:
        logging.error(f'cannot get data [loop{str(device.Retry)}, date{str(masterdate)}, device{device.DeviceName}]')
        restart_hci0(device.DeviceName)
        return None

######SwitchBot thermo-hygrometer data acquisition######
def getdata_switchbot_thermo(device):
    #Maximum device when no value is available.Retry Repeat scan
    for i in range(device.Retry):
        #Set the switchbot sensor value acquisition delegate
        scanner = btle.Scanner().withDelegate(SwitchbotScanDelegate(str.lower(device.MacAddress)))
        #Scan to get sensor value
        try:
            scanner.scan(device.Timeout)
        #If an error occurs in the scan, restart the Bluetooth adapter
        except:
            restart_hci0(device.DeviceName)
        #End the loop when the value can be obtained
        if scanner.delegate.sensorValue is not None:
            break
        #If the value cannot be obtained, write it to the log
        else:
            logging.warning(f'retry to get data [loop{str(i)}, date{str(masterdate)}, device{device.DeviceName}, timeout{device.Timeout}]')
    
    #If the value can be obtained, store the POST data in dict
    if scanner.delegate.sensorValue is not None:
        #Data to POST
        data = {        
            'DeviceName': device.DeviceName,
            'Date_Master': str(masterdate),
            'Date': str(datetime.today()),
            'Temperature': str(scanner.delegate.sensorValue['Temperature']),
            'Humidity': str(scanner.delegate.sensorValue['Humidity']),
            'BatteryVoltage': str(scanner.delegate.sensorValue['BatteryVoltage'])
        }
        return data
    #If it could not be obtained, output the log and restart the Bluetooth adapter
    else:
        logging.error(f'cannot get data [loop{str(device.Retry)}, date{str(masterdate)}, device{device.DeviceName}, timeout{device.Timeout}]')
        restart_hci0(device.DeviceName)
        return None

######Nature Remo data acquisition######
def getdata_remo(device, csvpath):
    #Maximum device when sensor data value is not available.Retry Repeat scan
    for i in range(device.Retry):
        try:
            sensorValue = GetRemoData().get_sensor_data(device.Token, device.API_URL)
        #Log output if an error occurs
        except:
            logging.warning(f'retry to get data [loop{str(i)}, date{str(masterdate)}, device{device.DeviceName}, sensor]')
            sensorValue = None
            continue
        else:
            break
    #Maximum device when air conditioner data value is not available.Retry Repeat scan
    for i in range(device.Retry):
        try:
            airconValue = GetRemoData().get_aircon_data(device.Token, device.API_URL)
        #Log output if an error occurs
        except:
            logging.warning(f'retry to get data [loop{str(i)}, date{str(masterdate)}, device{device.DeviceName}, aircon]')
            sensorValue = None
            continue
        else:
            break
        
    #If the value can be obtained, store the POST data in dict
    if sensorValue is not None:
        #Sensor data
        data = {        
            'DeviceName': device.DeviceName,        
            'Date_Master': str(masterdate),
            'Date': str(datetime.today()),
            'Temperature': str(sensorValue['Temperature']),
            'Humidity': str(sensorValue['Humidity']),
            'Light': str(sensorValue['Light']),
            'Human_last': str(sensorValue['Human_last']),
            'HumanMotion': GetRemoData().calc_human_motion(sensorValue['Human_last'], f'{csvpath}/{device.DeviceName}')
        }
        #Air conditioner data
        if airconValue is not None:
            data['TempSetting'] = airconValue['TempSetting']
            data['AirconMode'] = airconValue['Mode']
            data['AirVolume'] = airconValue['AirVolume']
            data['AirDirection'] = airconValue['AirDirection']
            data['AirconPower'] = airconValue['Power']
        return data
    #If you can not get it, log output (Because it is via WiFi, do not restart the Bluetooth adapter)
    else:
        logging.error(f'cannot get data [loop{str(device.Retry)}, date{str(masterdate)}, device{device.DeviceName}]')
        return None

######CSV output of data######
def output_csv(data, csvpath):
    dvname = data['DeviceName']
    monthstr = masterdate.strftime('%Y%m')
    #Output destination folder name
    outdir = f'{csvpath}/{dvname}/{masterdate.year}'
    #When the output destination folder does not exist, create a new one
    os.makedirs(outdir, exist_ok=True)
    #Output file path
    outpath = f'{outdir}/{dvname}_{monthstr}.csv'
    
    #Create a new output file when it does not exist
    if not os.path.exists(outpath):        
        with open(outpath, 'w') as f:
            writer = csv.DictWriter(f, data.keys())
            writer.writeheader()
            writer.writerow(data)
    #Add one line when the output file exists
    else:
        with open(outpath, 'a') as f:
            writer = csv.DictWriter(f, data.keys())
            writer.writerow(data)

######Processing to upload to Google Spreadsheet######
def output_spreadsheet(all_values_dict):
    #API URL
    url = 'The URL of the GAS API is listed here'
    #POST data to API
    response = requests.post(url, json=all_values_dict)
    print(response.text)

######Bluetooth adapter restart######
def restart_hci0(devicename):
    passwd = 'Enter your Raspberry Pi password'
    subprocess.run(('sudo','-S','hciconfig','hci0','down'), input=passwd, check=True)
    subprocess.run(('sudo','-S','hciconfig','hci0','up'), input=passwd, check=True)
    logging.error(f'restart bluetooth adapter [date{str(masterdate)}, device{devicename}]')


######Main######
if __name__ == '__main__':    
    #Get start time
    startdate = datetime.today()
    #Round the start time in minutes
    masterdate = startdate.replace(second=0, microsecond=0)   
    if startdate.second >= 30:
        masterdate += timedelta(minutes=1)

    #Read configuration file and device list
    cfg = configparser.ConfigParser()
    cfg.read('./config.ini', encoding='utf-8')
    df_devicelist = pd.read_csv('./DeviceList.csv')
    #Total number of sensors and successful data acquisition
    sensor_num = len(df_devicelist)
    success_num = 0

    #Log initialization
    logname = f"/sensorlog_{str(masterdate.strftime('%y%m%d'))}.log"
    logging.basicConfig(filename=cfg['Path']['LogOutput'] + logname, level=logging.INFO)

    #Dict for holding all acquired data
    all_values_dict = None

    ######Data acquisition for each device######
    for device in df_devicelist.itertuples():
        #Omron environment sensor BAG type (BroadCast connection)
        if device.SensorType in ['Omron_BAG_EP','Omron_BAG_IM']:
            data = getdata_omron_bag(device)
        #Omron environment sensor USB type (Connect mode connection)
        elif device.SensorType in ['Omron_USB_EP','Omron_USB_IM']:
            data = getdata_omron_usb(device)
        #Inkbird IBS-TH1
        elif device.SensorType in ['Inkbird_IBSTH1mini','Inkbird_IBSTH1']:
            data = getdata_ibsth1(device)
        #SwitchBot Thermo-Hygrometer
        elif device.SensorType == 'SwitchBot_Thermo':
            data = getdata_switchbot_thermo(device)
        #remo
        elif device.SensorType == 'Nature_Remo':
            data = getdata_remo(device, cfg['Path']['CSVOutput'])
        #Other than those above
        else:
            data = None        

        #When data exists, add it to Dict for holding all data and output CSV
        if data is not None:
            #all_values_Create a new dictionary when dict is None
            if all_values_dict is None:
                all_values_dict = {data['DeviceName']: data}
            #all_values_Add to existing dictionary when dict is not None
            else:
                all_values_dict[data['DeviceName']] = data

            #CSV output
            output_csv(data, cfg['Path']['CSVOutput'])
            #Success number plus
            success_num+=1

    ######Processing to upload to Google Spreadsheet######
    output_spreadsheet(all_values_dict)

    #Log output of processing end
    logging.info(f'[masterdate{str(masterdate)} startdate{str(startdate)} enddate{str(datetime.today())} success{str(success_num)}/{str(sensor_num)}]')

• I use try & except processing a lot, but it is due to the unstable Bluetooth connection and frequent errors. For more information [see below](https://qiita.com/c60evaporator/items/283d0569eba58830f86e#%E4%B8%8D%E5%85%B7%E5%90%881peripheral%E3%81%AE%E5%88%9D % E6% 9C% 9F% E5% 8C% 96% E6% 99% 82% E3% 81% AB% E3% 82% A8% E3% 83% A9% E3% 83% BC% E3% 81% 8C% E5 % 87% BA% E3% 82% 8B)

④ Hit the GAS API from Python to write data to the spreadsheet

Write data from Raspberry Pi to Google Sheets.

On the RaspberryPi side, use Python's API execution class to post data and On the spreadsheet side, use the Google Apps Script script to receive the above data

• Articles that I referred to https://max999blog.com/python-requests-get-post-json-data/ https://qiita.com/shirakiya/items/db22de49f00710478cfc https://www.weed.nagoya/entry/2016/05/11/105145 https://hibiki-press.tech/python/requests_module/1882#toc7

Data POST processing from Python side (RaspberryPi side)

It seems that only the function named doPost can be used to publish the API, so you need to post all the data from Python to GAS at once. For this purpose, I made a script to post in JSON format.

sensors_to_spreadsheet.Part of py

######Processing to upload to Google Spreadsheet######
def output_spreadsheet(all_values_dict):
    #API URL
    url = 'The URL of the GAS API is listed here'
    #POST data to API
    response = requests.post(url, json=all_values_dict)
    print(response.text)

The point is that the value passed after "requests.post (url, json =" is dict. (Do not convert the character string with json.dumps (). It is a addictive point to pass it without converting it to json even though the name is json = ...)

Create a destination spreadsheet

Go to Google Spreadsheets and go to Create a spreadsheet like the one below.

The sheet name is "SensorData". In the next section, we will create a script that outputs data to all sensors at once on this sheet.

GAS script creation (data reception and spreadsheet writing process)

Select "Tools"-> "Script Editor" on the spreadsheet and select Create a GAS script like the one below

postSensorData.gs

var spreadsheetId = '******'//← Enter the spreadsheet ID

//Method to receive posted data and write it to a spreadsheet
function doPost(e){
  //Get spreadsheet information (sheet instance, header name list, number of rows)
  var sheet = SpreadsheetApp.openById(spreadsheetId).getSheetByName('SensorData');
  var headerList = sheet.getRange(1, 1, 1, sheet.getLastColumn()).getValues()[0];
  var lastRow = sheet.getLastRow();

  //Convert received data to JSON
  var json = JSON.parse(e.postData.getDataAsString());

  //Hold the number of devices
  var devCnt = 0;
  for(var device in json) {
    //Add 1 device
    devCnt++;
    //Scan all sensor measurements
    for(var colName in json[device]){
      //The column name is"DeviceName"Ignore the value when"device"(Acquired as)
      if(colName == "DeviceName"){
      //The column name is"Date_Master"When, ignore the value except for the first device (prevent duplicate description)
      }else if(colName == "Date_Master" && devCnt >= 2){
      //Other than the above, write data to the spreadsheet
      }else{        
        headerList = addData(sheet, lastRow, headerList, device, colName, json[device][colName]);
      }
    }
  }
  //Return success to API execution source
  return ContentService.createTextOutput(JSON.stringify({content:"post ok", endtime:new Date()})).setMimeType(ContentService.MimeType.JSON);
}

//Write data to a spreadsheet
function addData(sheet, lastRow, headerList, deviceName, colName, cell){
  //Column name is Date_When Master, write data & write data divided only into date part
  if (colName == 'Date_Master'){
    sheet.getRange(lastRow + 1, 1).setValue(cell);
    sheet.getRange(lastRow + 1, 2).setValue(cell.substring(0, 10));
  //Column name is Date_When other than Master
  }else{
    //Column name (combines device name and original column name)
    var newColName = deviceName + "_" + colName;
    //Scan existing header names
    for (i=0; i<headerList.length; i++) {
      //Write sensor data to this column if there is an existing header name equal to the column name
      if(newColName == headerList[i]){
        sheet.getRange(lastRow + 1, i + 1).setValue(cell);
        break;
      }
      //If the column name is not included in the existing header name, add a new header and write the sensor data.
      if(i == headerList.length - 1){
        sheet.getRange(1, i + 2).setValue(newColName);
        sheet.getRange(lastRow + 1, i + 2).setValue(cell);
        headerList.push(newColName);
      }
    }
  }
  return headerList
}

** * Spreadsheet ID ** The spreadsheet URL is "https://docs.google.com/spreadsheets/d/AAAAAA/edit#gid=0" If so, 「AAAAAA」 The part of is the ID of the spreadsheet.

GAS script API release

[Here](https://qiita.com/c60evaporator/items/ed2ffde4c87001111c12#python%E3%81%8B%E3%82%89gas%E3%81%AEapi%E3%82%92%E5%8F%A9% E3% 81% 84% E3% 81% A6% E3% 82% B9% E3% 83% 97% E3% 83% AC% E3% 83% 83% E3% 83% 89% E3% 82% B7% E3% 83% BC% E3% 83% 88% E3% 81% AB% E3% 83% 87% E3% 83% BC% E3% 82% BF% E6% 9B% B8% E3% 81% 8D% E8% BE% Please publish the GAS script API by referring to BC% E3% 81% BF). If you paste the URL that appears at the end into the following part of the main script sensor_to_spreadsheet.py,

sensors_to_spreadsheet.Part of py

def output_spreadsheet(all_values_dict):
    #API URL
    url = 'The URL of the GAS API is listed here'

GAS script is called when sensors_to_spreadsheet.py is executed Values are automatically written to the spreadsheet (headers are also added automatically, as in the next section)

Description of the output spreadsheet

As shown in the figure below Header name: [Device name_meaning of value] → Automatically added if it does not exist Data: Add one line to the end of the measured value corresponding to the above header I made a GAS script so that headers can be generated and data can be written automatically.

⑤ Periodic execution of script

It's a hassle to run the script every time to get the sensor value, so Automate using the periodic package "cron". It looks like a Windows task scheduler.

Enable cron

It may be disabled by default, so enable it by referring to here.

**-Checkpoint 1: Check rsyslog.conf ** It will not work if "cron" in /etc/rsyslog.conf is commented out. In my case, it was commented out as follows, so

###############
#### RULES ####
###############

#
# First some standard log files.  Log by facility.
#
auth,authpriv.*                 /var/log/auth.log
*.*;auth,authpriv.none          -/var/log/syslog
#cron.*                         /var/log/cron.log
daemon.*                        -/var/log/daemon.log

After uncommenting as below,

cron.*                         /var/log/cron.log

Restart rsyslog with the following command.

sudo /etc/init.d/rsyslog restart

** ・ Checkpoint 2: Change log level ** In / etc / default / cron, specify the items to be described in the log when cron is executed. By default, it seems that the log is not output as shown below, so

# For quick reference, the currently available log levels are:
#   0   no logging (errors are logged regardless)
#   1   log start of jobs
#   2   log end of jobs
#   4   log jobs with exit status != 0
#   8   log the process identifier of child process (in all logs)
#
#EXTRA_OPTS=""

Uncomment EXTRA_OPTS as shown below

EXTRA_OPTS='-L 15'

will do. It means that all will be output with 1 + 2 + 4 + 8 = 15.

Restart cron with the following command.

sudo /etc/init.d/cron restart

If cron.log is generated in / var / log, it is successful. (Please check here if you want to see the execution history of cron)

Schedule execution with cron

Register periodic execution with cron

** ・ Editing crontab ** Open crontab with the following command

crontab -e

• "Sudo crontab -e" will open the root execution file, so please do not add sudo.

When asked which editor to open, select the editor you like (nano recommended for beginners)

# Edit this file to introduce tasks to be run by cron.
# 
# Each task to run has to be defined through a single line
# indicating with different fields when the task will be run
: Various things continue

There are various comments written as above, At the very end, describe the timing and command you want to execute. The timing format is [here](http://tech.junkpot.net/archives/721/crontab-%E3%81%A7%E7%B4%B0%E3%81%8B%E3%81%84%E3 % 82% B9% E3% 82% B1% E3% 82% B8% E3% 83% A5% E3% 83% BC% E3% 83% AB% E3% 81% AE% E8% A8% AD% E5% AE Please refer to% 9A.html)

This time, since it is executed every 5 minutes, the following contents are described.

*/5 * * * * [Python full path] [sensors_to_spreadSheet.full path of py] >/dev/null 2>&1

• Python requires full path specification as shown in here. Below is an example of a full path -Default Python3: / usr / bin / python3 -Pyenv environment: /home/[username]/.pyenv/shims/python
• The last "> / dev / null 2> & 1" is a process to prevent sending an email. [Here](https://orebibou.com/2014/12/cron%E3%81%A7%E3%82%B9%E3%82%AF%E3%83%AA%E3%83%97%E3% 83% 88% E3% 82% 92% E5% AE% 9A% E6% 9C% 9F% E5% AE% 9F% E8% A1% 8C% E3% 81% 95% E3% 81% 9B% E3% 82% See 8B% E3% 81% A8% E3% 81% 8D% E3% 81% AB% E6% B3% A8% E6% 84% 8F% E3% 81% 97% E3% 81% 9F /)

** ・ Start cron ** Start cron with the following command

sudo /etc/init.d/cron start

Wait for a while above and you're done if the spreadsheet is updated every 5 minutes!

• If it doesn't work, see the cron log in / var / log / syslog and deal with it.

⑥ Visualization by Google Data Portal

Google Data Portal is a dashboard that can be edited and viewed on the cloud. In this section, we will explain how to create the temperature transition graph part of the dashboard at the beginning.

Creating a new report

** Specify Google Spreadsheet as the data connection destination ** ** If you are asked for approval, press the approval button ** ** You will be asked for the referenced sheet, so specify the spreadsheet created in ④ ** ** Press "Add to Report" ** ** Rename the report **

Creating a daily average graph

Create a graph to see long-term changes

** Click Resources → Manage Added Data Sources ** ** Click "Edit" for the target data source ** ** Change the date and time to a recognizable format ** (The data portal has a strict date and time recognition format) If you want to get the correct statistics such as average: YYYYMMDD If you want to display the measured value for each line: YYYYMMDDhhmm ** Change to time series graph ** ** Specify the dimension (horizontal axis = Date_Master_Day) and index (vertical axis = temperature of various sensors) ** ** Change temperature statistics from total to average ** ** Change to hide missing values **

Creating a graph for each measurement (every 5 minutes)

Create a graph to see short-term changes

** Add a time series graph ** ** Specify the dimension (horizontal axis = Date_Master_Day) and index (vertical axis = temperature of various sensors) ** ** Change temperature statistics from total to average ** ** Change missing values to linearly interpolate ** This completes the temperature transition graph!

To complete the dashboard

At this point, only the part of the opening dashboard surrounded by the red frame below is completed.

In addition to the above, it is necessary to make a "humidity transition graph" and a "pie chart" to complete it. I will omit it because it will be long if I explain everything.

The humidity transition graph only changes the "index" of the temperature transition graph from temperature to humidity, so I think it can be created without any particular clogging.

For information on how to create a pie chart, click here [https://qiita.com/t-chi/items/01b9a9b98fbccef880c3#%E6%B9%BF%E5%BA%A6%E3%81%AE%E5%86% 86% E3% 82% B0% E3% 83% A9% E3% 83% 95) is summarized in an easy-to-understand manner.

I'm sorry I did a cooking-like way of flying for 3 minutes while saying "I won't leap!" At first. If you do your best to complete the graph and arrange the layout, you can make the following dashboard.

** That's it! !! ** ** There may be people who want to rush into "Where is the supermarket!" Without any strange parts, but Personally, I was impressed by the combination of low-priced gadgets and free services.

There were new discoveries, such as the humidity rising after cooking and the temperature drawing a beautiful rising curve with the seasons, so I think that the finish will not get tired of watching!

Corresponding to various errors

As with any system, continuous operation reveals a wide variety of problems that were not noticed in a single shot. In this case, I was particularly annoyed by ** Bluetooth reception instability **.

Each problem that occurs is dealt with individually and reflected in various scripts. Before the countermeasures, innumerable stops and missing values occurred, but with the following countermeasures, although missing values rarely appear, it has been improved to ** a level where continuous operation can be performed with almost no maintenance **!

Bug 1: An error occurs when initializing the peripheral

Phenomenon that occurs

inkbird_ibsth1.Part of py

    def get_ibsth1_data(self, macaddr, sensortype):
        #Connect to device
        peripheral = btle.Peripheral(macaddr)
        #IBS-For TH1 mini
        if sensortype == 'Inkbird_IBSTH1mini':
            characteristic = peripheral.readCharacteristic(0x002d)

above peripheral = btle.Peripheral(macaddr) By the way

BTLEException: failed to connect to peripheral addr type: public

Error appears or does not appear (it is troublesome that there is no reproducibility)

Coping

If you re-execute, it often works as if nothing happened, so The caller picks up the exception with try except and re-executes it up to a certain number of times.

sensors_to_spreadsheet.py getdata_Part of the ibsth1 method

def getdata_ibsth1(device):
    #Maximum device when no value is available.Retry Repeat scan
    for i in range(device.Retry):
        try:
            sensorValue = GetIBSTH1Data().get_ibsth1_data(device.MacAddress, device.SensorType)
        #Log output if an error occurs
        except:
            logging.warning(f'retry to get data [loop{str(i)}, date{str(masterdate)}, device{device.DeviceName}]')
            sensorValue = None
            continue
        else:
            break

The number of redoes is now specified for each device in the "Retry" column of the configuration file "DeviceList.csv".

Bug 2: Error when scanning in broadcast mode

Phenomenon that occurs

getdata_switchbot_Part of the thermo method

        #Scan to get sensor value
        try:
            scanner.scan(device.Timeout)
        #If an error occurs in the scan, restart the Bluetooth adapter
        except:
            restart_hci0(masterdate, device.DeviceName)

If you repeat the above scanner.scan (), it will be executed about 20 times.

bluepy.btle.BTLEManagementError: Failed to execute management command 'scan' (code: 11, error: Rejected)

Error occurs, and no matter how many times you re-execute, the error will make it impossible to acquire data. It is very troublesome because there is no reproducibility of the number of times, such as the number of executions is 10 times and 34 times. (The cause is unknown, but it looks like a section-like behavior is accumulated)

Coping

Refer to here and use the following command (restart the Bluetooth adapter)

sudo hciconfig hci0 down
sudo hciconfig hci0 up

I found that it recovers from the error when I execute, so refer to here

python

def restart_hci0(masterdate, devicename):
    passwd = 'Enter password'
    subprocess.run(('sudo','-S','hciconfig','hci0','down'), input=passwd, check=True)
    subprocess.run(('sudo','-S','hciconfig','hci0','up'), input=passwd, check=True)
    logging.error(f'restart bluetooth adapter [date{str(masterdate)}, device{devicename}]')

I made a method called.

• If you leave the above as it is, there is a security risk by directly typing the password, so please hide it as appropriate for reference here

this,

getdata_switchbot_Part of the thermo method

        #Scan to get sensor value
        try:
            scanner.scan(device.Timeout)
        #If an error occurs in the scan, restart the Bluetooth adapter
        except:
            restart_hci0(masterdate, device.DeviceName)

It is a mechanism that can be executed when an error occurs and automatically restarted and recovered.

The above command seems to be effective in improving bug 1, so I tried to execute it even if the value could not be obtained within the number of retries. ** Reboot in case of trouble ** The versatility of the hat is off!

Bug 3: When the value cannot be obtained, the cause cannot be tracked.

• This is an operational bottleneck rather than a malfunction.

Phenomenon that occurs

Even if you repeat the re-execution for bug 1, you may not be able to get the value within the number of retries.

Also, in broadcast mode like SwitchBot, Peripheral related error does not occur, but bug 2 occurs, the value is not returned within the timeout value, and the value may not be acquired within the retry count.

If you do nothing, you will not realize that you could not get the value, and if you can not get the value all the time due to a failure, you can not track when the failure started.

approach

Record the value could not be obtained and output the following 4 patterns of logs by referring to here so that you can follow the cause later. I made it. ** Pattern 1: Log output part when defect 1 occurs **

sensors_to_spreadsheet.py getdata_omron_Part of usb method

        #Log output if an error occurs
        except:
            logging.warning(f'retry to get data [loop{str(i)}, date{str(masterdate)}, device{device.DeviceName}]')

** Pattern 2: Log output part when defect 2 occurs **

sensors_to_spreadsheet.py restart_Part of the hci0 method

    logging.error(f'restart bluetooth adapter [date{str(masterdate)}, device{devicename}]')

** Pattern 3: Log output part when the value could not be acquired even after retrying the specified number of times **

sensors_to_spreadsheet.py getdata_omron_Part of usb method

    #If the value could not be obtained, output the log and restart the Bluetooth adapter.
    else:
        logging.error(f'cannot get data [loop{str(device.Retry)}, date{str(masterdate)}, device{device.DeviceName}]')

** Pattern 4: Log output part when a series of processing is completed **

sensors_to_spreadsheet.The end of the main code of py

    #Log output of processing end
    logging.info(f'[masterdate{str(masterdate)} startdate{str(startdate)} enddate{str(datetime.today())} success{str(success_num)}/{str(sensor_num)}]')

The log output destination is specified in "Log Output" of the configuration file "config.ini".

What I felt when I tried it

Difficulty of long-term stable operation

It was more difficult to secure long-term stable operation than to acquire data. In particular, the Bluetooth reception area is unstable, and irreproducible errors mercilessly break my heart (laughs). The try-except process is very useful for solving the above problem, but if you abuse it, it will cause you to not notice the problem, so you should use it with caution.

Importance of extensibility

By listing device management in configuration files & automating spreadsheet column generation It has become much easier to respond when increasing the number of sensors. The extensibility requirement is often put off, but I realized that it wouldn't be a pain later if you thought about it carefully in the early stages of design!

The depth of the roots of security issues

If you try to ensure security such as password hiding, implementation and operation will be complicated, so I found it difficult to balance the trade-off relationship with convenience. There is also privacy concerns because the sensor data can tell to some extent the absence of people and what they are doing. (This is exactly the part that is a problem even in the SuperCity law)

Impressions for each sensor

• Please note that only one model has been tested, so it may be an individual difference rather than a sensor difference. ** Omron: ** Made in Japan only, both USB type and BAG type are stable (it is natural because it is more than double the price of other sensors) ** Inkbird IBS-TH1 mini: ** There are problems such as the value cannot be obtained and the battery does not last for a month. I can't recommend it honestly ** Not a mini IBS-TH1: ** Not as stable as Omron. I also have a battery, so I personally recommend ** ** SwitchBot **: This is also unstable like the IBS-TH1 mini, but I thought it was a good option considering that there are no negative outliers, battery life, and low price. ** Nature Remo: ** Stable with Wifi connection, but the bottleneck is the coarse particle size of the measurement. This device is mainly a remote control, not a sensor, so I think it can't be helped.

A sign of running out of battery

The Inkbird IBS-TH1 mini showed a lot of slack, with the battery running out in less than a month, but About 2 days before the battery ran out, abnormal values (temperature = -46.2 ° C, humidity = 0%) like the pink line in the figure below began to occur frequently.

If this is a reproducible sign of a dead battery, it's very useful information as it can be replaced before it's completely dead to prevent missing values. Even if the battery runs out next time, I will carefully observe the situation immediately before.

By the way, the remaining battery power acquired by the Inkbird app was about 40% remaining at this point, so I found that it was completely unreliable (laughs).

I would like to utilize the above knowledge for future development!