Using "Nature Remo E lite" that can acquire the amount of electricity used at home in real time, I made a dashboard that visualizes power consumption By displaying the usage status of the air conditioner acquired by Nature Remo, ** You can see the relationship between power consumption and air conditioner On-Off at a glance! ** ** I feel that we have taken a step forward toward the realization of an energy-saving house.
It's been a long time since the word ** IoT ** became a boom, At the same time, the difficulty of creating value with IoT is becoming more widespread, and I feel that the world's eyes on IoT are becoming stricter.
The figure below is the 2019 version of the "Hype Cycle Diagram" announced by Gartner. The IoT has entered a period of disillusionment.
Especially for home-use IoT products, except for smart speakers and smart remote controls, which are predominant in the world, I feel that there are many things that do not seem to have much merit even if they are made smarter.
Under these circumstances, the IoT product that electric power companies are steadily introducing is ** smart meters **. If we can visualize and efficiently use power consumption, which is a major cost for homes and society, It is based on the idea that ** "value" ** can be created.
The above smart meter transmits power consumption data using a wireless standard called WiSUN, This data can also be received and used by third-party products (B-root service).
Of these third-party products The ** power consumption data receiving device ** developed by Nature, which is famous for its smart remote controller "Nature Remo",
** Nature Remo E lite **.
With Nature Remo E lite, you can:
With the official app, you can check the current power consumption and a simple history graph
Power consumption data can be handled programmatically using the API. Basically, it can be obtained from API common to Nature Remo.
GET /1/appliances
With the command, you can get the power consumption data of Remo E lite as well as the device data of Nature Remo.
Type the following command on Linux ([Click here for access token issuance method](https://qiita.com/t-chi/items/01b9a9b98fbccef880c3#natureremo%E3%81%AE%E3%82%A2%E3%82%AF % E3% 82% BB% E3% 82% B9% E3% 83% 88% E3% 83% BC% E3% 82% AF% E3% 83% B3% E3% 82% 92% E7% 99% BA% E8 % A1% 8C% E3% 81% 99% E3% 82% 8B))
curl -X GET "https://api.nature.global/1/appliances" -k --header "Authorization:Bearer access token"
The following JSON will be returned
[
{
:
Other device information
:
},
{
"id": "****",
"device": {
"name": "Remo E lite",
"id": "****",
"created_at": "2020-06-09T17:16:06Z",
"updated_at": "2020-06-17T13:55:50Z",
"mac_address": "**:**:**:**:**:**",
"bt_mac_address": "**:**:**:**:**:**",
"serial_number": "****",
"firmware_version": "Remo-E-lite/1.1.3",
"temperature_offset": 0,
"humidity_offset": 0
},
"model": {
"id": "****",
"manufacturer": "",
"name": "Smart Meter",
"image": "ico_smartmeter"
},
"type": "EL_SMART_METER",
"nickname": "Smart meter",
"image": "ico_smartmeter",
"settings": null,
"aircon": null,
"signals": [],
"smart_meter": {
"echonetlite_properties": [
{
"name": "cumulative_electric_energy_effective_digits",
"epc": 215,
"val": "7",
"updated_at": "2020-08-12T10:09:14Z"
},
{
"name": "normal_direction_cumulative_electric_energy",
"epc": 224,
"val": "294263",
"updated_at": "2020-08-12T10:09:14Z"
},
{
"name": "cumulative_electric_energy_unit",
"epc": 225,
"val": "2",
"updated_at": "2020-08-12T10:09:14Z"
},
{
"name": "measured_instantaneous",
"epc": 231,
"val": "464",
"updated_at": "2020-08-12T10:09:14Z"
}
]
}
},
{
:
Other device information
:
}
]
Information on smart meters "smart_meter": You can find it in the "val" in the "echonetlite_properties" section. The two sections below are of particular importance.
normal_direction_cumulative_electric_energy: Total power consumption so far (unit: 0.01kWh)
measured_instantaneous: Current power consumption (unit: Watt)
In the case of the above JSON, the total power consumption is 2942.63kWh and the current power consumption is 464Watt.
** This time, we will use this API to display the relationship between power consumption and air conditioner On-Off in a graph as shown in the top image **
** ・ Nature Remo E lite ** ** ・ Normal Nature Remo ** ‥ Required to obtain on-off information for air conditioners ** ・ Raspberry Pi ** (Raspberry Pi3 Model B is used this time) **-Python execution environment ** (preset Python 3.7.3 is used this time) ** ・ Google account ** (required to use spreadsheet & data portal)
Follow the steps below to create a power consumption dashboard.
** 1. Apply for B route service to the electric power company 2. Nature Remo E lite initial settings 3. Building a logging system with Raspberry Pi 4. Creating a power consumption dashboard **
In order to obtain smart meter data, you need to apply for B route service to your electric power company. (Even if you are using a service like au Denki for electricity liberalization, you need to apply to an area power company like TEPCO.)
In my case, I applied from the Kansai Electric Power Group website below. https://www.kansai-td.co.jp/application/smart-meter/low-pressure/index.html
For other power company areas, please refer to the following https://chasuke.com/nremoe/
When I applied, I received a phone call and said that simple construction (free of charge) was required, so the flow was to adjust the schedule and then carry out the construction.
Whether or not construction is necessary depends on the smart meter installation status of the house. As of 2020, about two-thirds of households nationwide are becoming smart meters. ・ For households that are not smart meters, you can apply for replacement with priority when applying above. ・ Even if it is a smart meter, additional work may be required like me. It seems.
When the above application and construction are completed, the electric power company will send you the B route service ID and password by mail. At TEPCO, the password is sent by e-mail, and the sending method seems to differ depending on the area (in any case, I think it is either e-mail or mail).
Plug the Nature Remo E lite into an electrical outlet. I felt that plugging it into an outlet close to the switchboard would often work better when synchronizing later.
Download the app from the site below to your smartphone and install it. android iOS
Basically, proceed with the initial settings according to the instructions of the application, If you are asked for an ID and password on the way, enter the contents mailed from the electric power company.
Sometimes it fails when syncing with meter or WiFi, but I In the case of, it often worked if I tried again several times.
The previously developed sensor data logging system will be used. First of all, please carry out the contents of this article (it is long, but I would appreciate it if you could associate with me)
This time, in order to make the above system compatible with Nature Remo E lite, You need to rewrite remo.py and sensors_to_spreadsheet.py. Please rewrite each as follows. (It corresponds to the operation to incorporate the JSON returned in the above [API execution example] into the logging target)
remo.py
import json
import requests
import glob
import pandas as pd
#Remo data acquisition class
class GetRemoData():
def get_sensor_data(self, Token, API_URL):
headers = {
'accept': 'application/json',
'Authorization': 'Bearer ' + Token,
}
response = requests.get(f"{API_URL}/1/devices", headers=headers)
rjson = response.json()
return self._decodeSensorData(rjson)
def get_aircon_power_data(self, Token, API_URL):
headers = {
'accept': 'application/json',
'Authorization': 'Bearer ' + Token,
}
response = requests.get(f"{API_URL}/1/appliances", headers=headers)
rjson = response.json()
return self._decodeAirconPowerData(rjson)
def calc_human_motion(self, Human_last, csvdir):
filelist = glob.glob(f"{csvdir}/*/*.csv")
if len(filelist) == 0:
return 0
filelist.sort()
df = pd.read_csv(filelist[-1])
if df.Human_last[len(df) - 1] != Human_last:
return 1
else:
return 0
#Extract sensor data and convert to dict format
def _decodeSensorData(self, rjson):
for device in rjson:
#Remo data
if device['firmware_version'].split('/')[0] == 'Remo':
sensorValue = {
'SensorType': 'Remo_Sensor',
'Temperature': device['newest_events']['te']['val'],
'Humidity': device['newest_events']['hu']['val'],
'Light': device['newest_events']['il']['val'],
'Human_last': device['newest_events']['mo']['created_at']
}
return sensorValue
#Extract air conditioner and power data and convert to dict format
def _decodeAirconPowerData(self, rjson):
Value = {}
for appliance in rjson:
#Air conditioner
if appliance['type'] == 'AC':
Value['TempSetting'] = appliance['settings']['temp']
Value['Mode'] = appliance['settings']['mode']
Value['AirVolume'] = appliance['settings']['vol']
Value['AirDirection'] = appliance['settings']['dir']
Value['Power'] = appliance['settings']['button']
#Smart meter power data
elif appliance['type'] == 'EL_SMART_METER':
for meterValue in appliance['smart_meter']['echonetlite_properties']:
if meterValue['name'] == 'normal_direction_cumulative_electric_energy':
Value['CumulativeEnergy'] = float(meterValue['val'])/100
elif meterValue['name'] == 'measured_instantaneous':
Value['Watt'] = int(meterValue['val'])
#If the value cannot be obtained, set it to None.
if len(Value) == 0:
Value = None
return Value
The power consumption data is acquired by the above get_aircon_power_data
and _decodeAirconPowerData
methods.
sensors_to_spreadsheet.py
from bluepy import btle
from omron_env import OmronBroadcastScanDelegate, GetOmronConnectModeData
from inkbird_ibsth1 import GetIBSTH1Data
from switchbot import SwitchbotScanDelegate
from remo import GetRemoData
from mesh import GetMeshFromSpreadsheet
from datetime import datetime, timedelta
import os
import csv
import configparser
import pandas as pd
import requests
import logging
import subprocess
import pymongo
from pit import Pit
#Global variables(Acquisition time)
global masterdate
######Acquisition of value of OMRON environment sensor (BAG type)######
def getdata_omron_bag(device):
#Maximum device when no value is available.Retry Repeat scan
for i in range(device.Retry):
#omron_Set the sensor value acquisition delegate of env to execute at scan time
scanner = btle.Scanner().withDelegate(OmronBroadcastScanDelegate())
#Scan to get sensor value
try:
scanner.scan(device.Timeout)
#If an error occurs in the scan, restart the Bluetooth adapter
except:
restart_hci0(device.DeviceName)
#End the loop when the value can be obtained
if scanner.delegate.sensorValue is not None:
break
#If the value cannot be obtained, write it to the log
else:
logging.warning(f'retry to get data [loop{str(i)}, date{str(masterdate)}, device{device.DeviceName}, timeout{device.Timeout}]')
#If the value can be obtained, store the POST data in dict
if scanner.delegate.sensorValue is not None:
#Data to POST
data = {
'DeviceName': device.DeviceName,
'Date_Master': masterdate,
'Date': datetime.today(),
'Temperature': scanner.delegate.sensorValue['Temperature'],
'Humidity': scanner.delegate.sensorValue['Humidity'],
'Light': scanner.delegate.sensorValue['Light'],
'UV': scanner.delegate.sensorValue['UV'],
'Pressure': scanner.delegate.sensorValue['Pressure'],
'Noise': scanner.delegate.sensorValue['Noise'],
'BatteryVoltage': scanner.delegate.sensorValue['BatteryVoltage']
}
return data
#If the value could not be obtained, output the log and restart the Bluetooth adapter.
else:
logging.error(f'cannot get data [date{str(masterdate)}, device{device.DeviceName}, timeout{device.Timeout}]')
restart_hci0(device.DeviceName)
return None
######Data acquisition of OMRON environmental sensor (USB type)######
def getdata_omron_usb(device):
#Maximum device when no value is available.Retry Repeat scan
for i in range(device.Retry):
try:
sensorValue = GetOmronConnectModeData().get_env_usb_data(device.MacAddress)
#Log output if an error occurs
except:
logging.warning(f'retry to get data [loop{str(i)}, date{str(masterdate)}, device{device.DeviceName}]')
sensorValue = None
continue
else:
break
#If the value can be obtained, store the POST data in dict
if sensorValue is not None:
#Data to POST
data = {
'DeviceName': device.DeviceName,
'Date_Master': masterdate,
'Date': datetime.today(),
'Temperature': sensorValue['Temperature'],
'Humidity': sensorValue['Humidity'],
'Light': sensorValue['Light'],
'Pressure': sensorValue['Pressure'],
'Noise': sensorValue['Noise'],
'eTVOC': sensorValue['eTVOC'],
'eCO2': sensorValue['eCO2']
}
return data
#If the value could not be obtained, output the log and restart the Bluetooth adapter.
else:
logging.error(f'cannot get data [loop{str(device.Retry)}, date{str(masterdate)}, device{device.DeviceName}]')
restart_hci0(device.DeviceName)
return None
######Inkbird IBS-TH1 data acquisition######
def getdata_ibsth1(device):
#Maximum device when no value is available.Retry Repeat scan
for i in range(device.Retry):
try:
sensorValue = GetIBSTH1Data().get_ibsth1_data(device.MacAddress, device.SensorType)
#Log output if an error occurs
except:
logging.warning(f'retry to get data [loop{str(i)}, date{str(masterdate)}, device{device.DeviceName}]')
sensorValue = None
continue
else:
break
if sensorValue is not None:
#Data to POST
data = {
'DeviceName': device.DeviceName,
'Date_Master': masterdate,
'Date': datetime.today(),
'Temperature': sensorValue['Temperature'],
'Humidity': sensorValue['Humidity']
}
return data
#If the value could not be obtained, output the log and restart the Bluetooth adapter.
else:
logging.error(f'cannot get data [loop{str(device.Retry)}, date{str(masterdate)}, device{device.DeviceName}]')
restart_hci0(device.DeviceName)
return None
######SwitchBot thermo-hygrometer data acquisition######
def getdata_switchbot_thermo(device):
#Maximum device when no value is available.Retry Repeat scan
for i in range(device.Retry):
#Set the switchbot sensor value acquisition delegate
scanner = btle.Scanner().withDelegate(SwitchbotScanDelegate(str.lower(device.MacAddress)))
#Scan to get sensor value
try:
scanner.scan(device.Timeout)
#If an error occurs in the scan, restart the Bluetooth adapter
except:
restart_hci0(device.DeviceName)
#End the loop when the value can be obtained
if scanner.delegate.sensorValue is not None:
break
#If the value cannot be obtained, write it to the log
else:
logging.warning(f'retry to get data [loop{str(i)}, date{str(masterdate)}, device{device.DeviceName}, timeout{device.Timeout}]')
#If the value can be obtained, store the POST data in dict
if scanner.delegate.sensorValue is not None:
#Data to POST
data = {
'DeviceName': device.DeviceName,
'Date_Master': masterdate,
'Date': datetime.today(),
'Temperature': scanner.delegate.sensorValue['Temperature'],
'Humidity': float(scanner.delegate.sensorValue['Humidity']),
'BatteryVoltage': scanner.delegate.sensorValue['BatteryVoltage']
}
return data
#If it could not be obtained, output the log and restart the Bluetooth adapter
else:
logging.error(f'cannot get data [loop{str(device.Retry)}, date{str(masterdate)}, device{device.DeviceName}, timeout{device.Timeout}]')
restart_hci0(device.DeviceName)
return None
######Nature Remo data acquisition######
def getdata_remo(device, csvpath):
#Maximum device when sensor data value is not available.Retry Repeat scan
for i in range(device.Retry):
try:
sensorValue = GetRemoData().get_sensor_data(device.Token, device.API_URL)
#Log output if an error occurs
except:
logging.warning(f'retry to get data [loop{str(i)}, date{str(masterdate)}, device{device.DeviceName}, sensor]')
sensorValue = None
continue
else:
break
#Maximum device when air conditioner and power data values are not available.Retry Repeat scan
for i in range(device.Retry):
try:
airconPowerValue = GetRemoData().get_aircon_power_data(device.Token, device.API_URL)
#Log output if an error occurs
except:
logging.warning(f'retry to get data [loop{str(i)}, date{str(masterdate)}, device{device.DeviceName}, aircon]')
sensorValue = None
continue
else:
break
#If the value can be obtained, store the POST data in dict
if sensorValue is not None:
#Sensor data
data = {
'DeviceName': device.DeviceName,
'Date_Master': masterdate,
'Date': datetime.today(),
'Temperature': sensorValue['Temperature'],
'Humidity': float(sensorValue['Humidity']),
'Light': sensorValue['Light'],
'Human_last': sensorValue['Human_last'],
'HumanMotion': GetRemoData().calc_human_motion(sensorValue['Human_last'], f'{csvpath}/{device.DeviceName}')
}
#Air conditioner & power data
if airconPowerValue is not None:
data['TempSetting'] = int(airconPowerValue['TempSetting'])
data['AirconMode'] = airconPowerValue['Mode']
data['AirVolume'] = airconPowerValue['AirVolume']
data['AirDirection'] = airconPowerValue['AirDirection']
data['AirconPower'] = airconPowerValue['Power']
if data['AirconPower'] == "":
data['AirconPower'] = 'power-on_maybe'
#Electric power
if 'CumulativeEnergy' in airconPowerValue:
data['CumulativeEnergy'] = float(airconPowerValue['CumulativeEnergy'])
if 'Watt' in airconPowerValue:
data['Watt'] = int(airconPowerValue['Watt'])
return data
#If you can not get it, log output (Because it is via WiFi, do not restart the Bluetooth adapter)
else:
logging.error(f'cannot get data [loop{str(device.Retry)}, date{str(masterdate)}, device{device.DeviceName}]')
return None
######CSV output of data######
def output_csv(data, csvpath):
dvname = data['DeviceName']
monthstr = masterdate.strftime('%Y%m')
#Output destination folder name
outdir = f'{csvpath}/{dvname}/{masterdate.year}'
#When the output destination folder does not exist, create a new one
os.makedirs(outdir, exist_ok=True)
#Output file path
outpath = f'{outdir}/{dvname}_{monthstr}.csv'
#Create a new output file when it does not exist
if not os.path.exists(outpath):
with open(outpath, 'w') as f:
writer = csv.DictWriter(f, data.keys())
writer.writeheader()
writer.writerow(data)
#Add one line when the output file exists
else:
with open(outpath, 'a') as f:
writer = csv.DictWriter(f, data.keys())
writer.writerow(data)
######Processing to upload to Google Spreadsheet######
def output_spreadsheet(all_values_dict_str):
#API URL
url = 'The URL of the GAS API is listed here'
#POST data to API
response = requests.post(url, json=all_values_dict_str)
print(response.text)
######Bluetooth adapter restart######
def restart_hci0(devicename):
passwd = 'Enter your Raspberry Pi password'#Add concealment if necessary
subprocess.run(('sudo','-S','hciconfig','hci0','down'), input=passwd, check=True)
subprocess.run(('sudo','-S','hciconfig','hci0','up'), input=passwd, check=True)
logging.error(f'restart bluetooth adapter [date{str(masterdate)}, device{devicename}]')
######Main######
if __name__ == '__main__':
#Get start time
startdate = datetime.today()
#Round the start time in minutes
masterdate = startdate.replace(second=0, microsecond=0)
if startdate.second >= 30:
masterdate += timedelta(minutes=1)
#Read configuration file and device list
cfg = configparser.ConfigParser()
cfg.read('./config.ini', encoding='utf-8')
df_devicelist = pd.read_csv('./DeviceList.csv')
#Total number of sensors and successful data acquisition
sensor_num = len(df_devicelist)
success_num = 0
#Log initialization
logname = f"/sensorlog_{str(masterdate.strftime('%y%m%d'))}.log"
logging.basicConfig(filename=cfg['Path']['LogOutput'] + logname, level=logging.INFO)
#Dict for holding all acquired data
all_values_dict = None
#String version of the above dict (for GAS Post, because the datetime type cannot be converted to JSON)
all_values_dict_str = None
#Data acquisition start time
scan_start_date = datetime.today()
######Data acquisition for each device######
for device in df_devicelist.itertuples():
#Omron environment sensor BAG type (BroadCast connection)
if device.SensorType in ['Omron_BAG_EP','Omron_BAG_IM']:
data = getdata_omron_bag(device)
#Omron environment sensor USB type (Connect mode connection)
elif device.SensorType in ['Omron_USB_EP','Omron_USB_IM']:
data = getdata_omron_usb(device)
#Inkbird IBS-TH1
elif device.SensorType in ['Inkbird_IBSTH1mini','Inkbird_IBSTH1']:
data = getdata_ibsth1(device)
#SwitchBot Thermo-Hygrometer
elif device.SensorType == 'SwitchBot_Thermo':
data = getdata_switchbot_thermo(device)
#remo
elif device.SensorType == 'Nature_Remo':
data = getdata_remo(device, cfg['Path']['CSVOutput'])
#mesh
elif device.SensorType == 'Sony_MeshHuman':
data = getdata_mesh_human(device)
#Other than those above
else:
data = None
#When data exists, add it to Dict for holding all data and output CSV
if data is not None:
#all_values_Create a new dictionary when dict is None
if all_values_dict is None:
#all_values_Create a dict (Because it's the first, Date_Master and Date_ScanStart is also added)
all_values_dict = {'Date_Master':data['Date_Master'], 'Date_ScanStart':scan_start_date}
all_values_dict.update(dict([(data['DeviceName']+'_'+k, v) for k,v in data.items() if k != 'Date_Master']))
#Convert data to string and all_values_dict_create str(Date because it's the first_Added ScanStart)
data_str = dict([(k, str(v)) for k,v in data.items()])
data_str['Date_ScanStart'] = str(scan_start_date)
all_values_dict_str = {data_str['DeviceName']: data_str}
#all_values_Add to existing dictionary when dict is not None
else:
#all_values_Added to dict (Date because it's not the first_Master excluded)
all_values_dict.update(dict([(data['DeviceName']+'_'+k, v) for k,v in data.items() if k != 'Date_Master']))
#all_values_dict_Add to str
data_str = dict([(k, str(v)) for k,v in data.items()])
all_values_dict_str[data_str['DeviceName']] = data_str
#CSV output
output_csv(data_str, cfg['Path']['CSVOutput'])
#Success number plus
success_num+=1
######Processing to upload to Google Spreadsheet######
output_spreadsheet(all_values_dict_str)
#Log output of processing end
logging.info(f'[masterdate{str(masterdate)} startdate{str(startdate)} enddate{str(datetime.today())} success{str(success_num)}/{str(sensor_num)}]')
Other than making it possible to acquire Remo E lite data, we have made some improvements, but I think that it is a level that you do not have to worry about in actual operation.
Also, if the configuration file DeviceList.csv contains NatureRemo's (API_URL and Token must be entered), No new additions are required for Remo E lite.
When you start logging with cron, as shown in the red frame below,
Device name_CumulativeEnergy: Total power consumption (unit: kWh)
Device name_Watt: Current power consumption (unit: Watt)
Two types of fields are added.
You can see that the power consumption increases at once when the air conditioner is turned on. In the next chapter, let's graph this situation.
Google Data Portal is a dashboard that can be edited and viewed on the cloud. Use this tool to create a dashboard that shows the relationship between power consumption and air conditioner on-off.
** Specify Google Spreadsheet as the data connection destination ** ** If you are asked for approval, press the approval button ** ** You will be asked for the referenced sheet, so specify the spreadsheet created in ④ ** ** Press "Add to Report" ** ** Rename the report **
Create a graph to see long-term changes
** Click Resources → Manage Added Data Sources **
** Click "Edit" for the target data source **
** Change the date and time to a recognizable format ** (The data portal has a strict date and time recognition format) If you want to get the correct statistics such as average: YYYYMMDD If you want to display the measured value for each line: YYYYMMDDhhmm
** Add a field (field for power consumption calculation) **
** Set the fields as shown below (power consumption maximum value-minimum value ≒ power consumption of the day) **
** Return to the main screen and click on the graph to change to a time series graph **
** Specify the dimension (horizontal axis = Date_Master_Day) and index (vertical axis = Diff_CumlativeEnergy created above) **
** Change the index name to "Daily power consumption (kWh)" **
** Change to hide missing values **
Create a graph comparing the power consumption (Watt) measured every 5 minutes with the air conditioner On-Off
** Specify the dimension (horizontal axis = Date_Master) and index (vertical axis = device name_Watt) **
** Specify the date range to be displayed (in the figure below, from 0:00 two days ago to 24:00 on the day) **
** Click Resources → Manage Added Data Sources **
** Click "Edit" for the target data source **
** Create a field "Aircon_On" that represents the on-off of the air conditioner as shown in the figure below **
** Add "Aircon_On" to the same graph as Instantaneous Power Consumption **
** Change missing values to linearly interpolate **
** Adjust the display color of the graph **
Adjust the overall layout from "Themes and Layouts" Add a graph to your liking → Display the required numbers on the scorecard, It looks better
** That's it! ** **
As you can see, the power consumption of the air conditioner is tremendous (when it is turned on, the power consumption more than doubles ...) In addition, it seems that the power consumption is the highest immediately after it is turned on, and when the power consumption gradually decreases and drops to a certain level, it becomes sick.
** TV and light bulb ON-OFF information ** can also be obtained with API, so After adding these, let's look at the relationship with power consumption.
I want to create "value" by reducing electricity bills without ending with visualization! Aim! ** Energy saving house! ** **
Recommended Posts