I tried to uncover our darkness with Chatwork API

The application of the article is available on github

I've put together a story about creating an application that looks up silent retirees from Chatwork that the company doesn't announce. (* We are not responsible for any troubles with the company to which the user belongs due to operation)

Slack, scraping, Excel documents should be able to be diverted by arranging the data acquisition method

Origin

――We have more than 200 employees, but I was very uncomfortable because there was no such thing as an announcement of retirees because some of the employees I knew had quit before I knew it. ――You are an engineer only if you solve problems with technology! ――When I noticed my previous job, people had quit, so there should be people who have the same problems → The desire for approval to be satisfied ―― ~~ I have no plans for consecutive holidays ~~

Functional goals

--Monitor daily enrollment changes based on your Chatwork contacts --Post the monitoring results to My Channel -Allow to see summary reports for a certain period such as yearly or monthly --Be careful when handling personal information and API keys

Technology used

--Host environment: Windows10 + Ubuntu20.4 on WSL2 (If you can use Docker, the host OS does not matter)

Outcome

I was able to display new employees / retirees within the period and grasp daily changes in internal members.

↓ Report screen

report.png

↓ Chatwork notification result screen

myroom.png

Work process

Environment

--If you can use the docker-compose command, you can find other necessary items in Project published on github.

--Please prepare your own Chatwork API token (+ room_id of notification destination if you use batch notification) required for application operation. How to get API token (official)

Docker environment construction

Dockerfile


FROM python:3
ENV PYTHONUNBUFFERED 1
ENV PYTHONIOENCODING utf-8
RUN mkdir /script /app
WORKDIR /script
COPY requirements.txt /script/
RUN apt update && apt install -y cron vim
RUN pip install -r requirements.txt
WORKDIR /app

requirements.txt


Django>=3.0,<4.0
psycopg2-binary>=2.8
django-environ
requests
python-dateutil

docker-compose.yml


version: '3'

services:
  app:
    build: .
    command: python manage.py runserver 0.0.0.0:8000
    environment:
      - CHATWORK_API_TOKEN=${CHATWORK_API_TOKEN}
      - ROOM_ID=${ROOM_ID}
      - TZ=Asia/Tokyo
    volumes:
      - ./app:/app
      - ./export_environment.sh:/script/export_environment.sh
      - ./crontab:/script/crontab
    ports:
      - "8000:8000"

Main processing explanation

--Get a list of contact members using your account token --Clarify the joining / leaving members by comparing the list data with different acquisition times. --Since past data cannot be referenced from API, the data to be compared is stored in the DB. --Designed to return the difference between contact members when given two dates to get_diff () --show (), which has a function to display a report, displays a member change report for the past six months on a monthly basis. ――In batch processing, the same method as for reports is used to post members who have changed day by day to Chatwork as breaking news.

Report screen processing + Chatwork contact list acquisition and saving, data comparison method

app/chatwork/views.py


from django.shortcuts import render
from django.http import HttpResponse
from chatwork.models import Account
import requests
from datetime import date
from dateutil.relativedelta import relativedelta
from django.db.models import Count
import environ
env = environ.Env(DEBUG=(bool, False))

# Create your views here.

def show(request):
    diff_list = list()
    for i in range(6):
        end = (date.today() - relativedelta(months=i)).isoformat()
        start = (date.today() - relativedelta(months=(i+1))).isoformat()
        diff_list.append(get_diff(start, end))

    params = dict(d1=diff_list[0], d2=diff_list[1], d3=diff_list[2], d4=diff_list[3], d5=diff_list[4], d6=diff_list[5])
    return render(request, 'chatwork/show.html', params)

def get_diff(start, end):
    if not Account.objects.filter(date=date.today().isoformat()):
        base = 'https://api.chatwork.com/v2/'
        end_point = 'contacts'
        api_token = env('CHATWORK_API_TOKEN')
        headers = {'X-ChatWorkToken': api_token, 'Content-Type': 'application/x-www-form-urlencoded'}
        res = requests.get(base + end_point, headers=headers)
        for contact in res.json():
            data = dict(account_id=contact['account_id'], name=contact['name'][:2], department=contact['department'], date=date.today().isoformat())
            Account.objects.update_or_create(**data)
    query = Account.objects.filter(date__gte=start, date__lte=end).values('date').annotate(Count('date'))
    if len(query) < 2:
        return dict(period='no comparable data found during ' + start + ' ~ ' + end, added=list(), dropped=list())
    latest = query.order_by('-date')[0]['date'].isoformat()
    oldest = query.order_by('date')[0]['date'].isoformat()
    period = oldest + '~' + latest

    data_latest = Account.objects.filter(date=latest) or list()
    data_oldest = Account.objects.filter(date=oldest) or list()

    ids_latest = data_latest.values_list('account_id', flat=True) if data_latest else list()
    ids_oldest = data_oldest.values_list('account_id', flat=True) if data_oldest else list()

    added = Account.objects.filter(date=latest).filter(account_id__in=ids_latest).exclude(account_id__in=ids_oldest)
    dropped = Account.objects.filter(date=oldest).filter(account_id__in=ids_oldest).exclude(account_id__in=ids_latest)

    return dict(period=period, added=added, dropped=dropped)
Batch processing (send the difference between the current day and the previous day data to your own My Chat)

app/chatwork/management/commands/contact_daily.py


from django.core.management.base import BaseCommand

from chatwork.models import Account
from chatwork.views import get_diff

from datetime import date
from dateutil.relativedelta import relativedelta
import environ
import requests
env = environ.Env(DEBUG=(bool, False))

class Command(BaseCommand):
    def handle(self, *args, **options):
        today = date.today().isoformat()
        yesterday = (date.today() - relativedelta(days=1)).isoformat()
        data = get_diff(yesterday, today)
        report_title = data['period']
        report_added = 'added: ' + '(' + str(len(data['added'])) + ')' + ' / '.join(list(d.name for d in data['added']))
        report_dropped = 'dropped: ' + '(' + str(len(data['dropped'])) + ')' + ' / '.join(list(d.name for d in data['dropped']))
        report = """
{report_title}
{report_added}
{report_dropped}
        """.format(report_title=report_title, report_added=report_added, report_dropped=report_dropped).strip()

        base = 'https://api.chatwork.com/v2/'
        room_id = env('ROOM_ID')
        end_point = 'rooms/' + room_id + '/messages'
        api_token = env('CHATWORK_API_TOKEN')
        headers = {'X-ChatWorkToken': api_token, 'Content-Type': 'application/x-www-form-urlencoded'}
        payload = dict(body=report, self_unread=1)
        res = requests.post(base + end_point, headers=headers, params=payload)

Other

--Unit test is simple but implemented --Since leakage of personal information is not fashionable, the specification is such that only the first two characters of the name obtained from the API are saved in the DB. --Since it is absolutely NG to leak the API token of the company account, we decided to pass it as an environment variable. --Batch processing is executed every day, so it is via cron, but since the environment variables are independent of the user, it was necessary to take measures against it. --There were many inconveniences such as DB acquisition using Django's Models and handling of iterators. Recognize that Laravel's QueryBuilder and Collection classes are apt, including the fact that they are easy to understand even in the code of other companies.

Finally

It's a rough app that I made on holidays, but it's very encouraging to hear your opinions, so please leave a comment.

--Detailed commentary addition request --Guidance on improvement plans --Other technical questions, etc.

Recommended Posts

I tried to uncover our darkness with Chatwork API
I tried to create Quip API
I tried to touch Tesla's API
I tried to automatically post to ChatWork at the time of deployment with fabric and ChatWork Api
I tried to implement Autoencoder with TensorFlow
I tried to visualize AutoEncoder with TensorFlow
I tried to get started with Hy
I tried to touch the COTOHA API
I tried to implement CVAE with PyTorch
I tried to make a Web API
I tried to solve TSP with QAOA
I tried to introduce a serverless chatbot linked with Rakuten API to Teams
I tried to delete bad tweets regularly with AWS Lambda + Twitter API
I tried to make "Sakurai-san" a LINE BOT with API Gateway + Lambda
I tried to get the authentication code of Qiita API with Python.
I tried to get the movie information of TMDb API with Python
I tried to detect Mario with pytorch + yolov3
I tried to implement reading Dataset with PyTorch
I tried to use lightGBM, xgboost with Boruta
I tried to learn logical operations with TF Learn
I tried to move GAN (mnist) with keras
I want to analyze songs with Spotify API 2
I tried "License OCR" with Google Vision API
I tried to save the data with discord
I tried to detect motion quickly with OpenCV
I tried to integrate with Keras in TFv1.1
I tried to touch the API of ebay
I tried to get CloudWatch data with Python
I tried to output LLVM IR with Python
I tried to detect an object with M2Det!
I tried to automate sushi making with python
I tried to predict Titanic survival with PyCaret
I tried "Receipt OCR" with Google Vision API
I tried to operate Linux with Discord Bot
I tried to study DP with Fibonacci sequence
I tried to start Jupyter with Amazon lightsail
I tried to judge Tsundere with Naive Bayes
I tried to paste
I tried to automate internal operations with Docker, Python and Twitter API + bonus
I tried to make a simple image recognition API with Fast API and Tensorflow
I tried to learn the sin function with chainer
I tried to move machine learning (ObjectDetection) with TouchDesigner
I tried to create a table only with Django
I tried to extract features with SIFT of OpenCV
I tried to move Faster R-CNN quickly with pytorch
I tried to read and save automatically with VOICEROID2 2
I tried to implement and learn DCGAN with PyTorch
I tried to get started with blender python script_Part 01
I tried to touch the CSV file with Python
I tried to draw a route map with Python
I tried to automatically read and save with VOICEROID2
I tried to get started with blender python script_Part 02
I tried to generate ObjectId (primary key) with pymongo
I tried to implement an artificial perceptron with python
I tried to build ML Pipeline with Cloud Composer
I tried to implement time series prediction with GBDT
I tried to automatically generate a password with Python3
[Introduction to Pytorch] I tried categorizing Cifar10 with VGG16 ♬
I tried to solve the problem with Python Vol.1
I tried to analyze J League data with Python
I tried to implement Grad-CAM with keras and tensorflow