I made a system that allows you to tweet just by making a phone call

I want to tweet, but I don't have a smartphone or computer at hand. But there are landlines. What's the matter?

Even in such a situation, we have created a system that allows you to tweet with a single phone call.

How it works

スライド1.JPG

Technology to use

Preparation

Do it in a Linux environment. This time I used Debian with VPS.

There is no need to install a special library, just build a LAMP environment. Please change the directory in the source code as appropriate.

It seems that Twilio can also recognize voice, but since it does not support Japanese, this time we adopted the Bing Speech API for voice recognition.

Preparing Twilio

Just get an account. There are paid accounts and free accounts (trial is limited), but in this case we were able to implement it with a free account.

This area will be helpful for account acquisition and mechanism. Twilio First Steps

Preparing the Bing Speech API

You can get it from the here page. azure.JPG

You can get the key immediately by agreeing to the terms of use from "Getting API Key" and logging in. You can also log in with your Github account.

--Key endpoint (https://api.cognitive.microsoft.com/sts/v1.0) --Key 1 (32-digit alphanumeric characters) --Key 2 (32-digit alphanumeric characters)

Two types of keys will be issued, but it seems that either one can be used.

program

Twilio's processing is written in TwiML and PHP, and other processing is written in Python.

Write TwiML

Place the following xml in a location that can be accessed from the Internet, such as a web server. Then set the URL from your twilio account page.

record.xml


<?xml version="1.0" encoding="UTF-8"?>
<Response>
<Say language="ja-JP">Record the content of the tweet. After recording, press Sharp.</Say>
<Record method="GET" timeout="60" maxLength="30" finishOnKey="#" action="./get.php" />
</Response>

These few lines will automatically speak and even record. TwiML It's good because it's easy to understand and easy. See Official Reference for more information.

Write PHP

This is the part of ʻaction ="./get.php "written inof TwiML. The process is passed toget.php` after the call is closed.

get.php


<?php

//Get the recording file URL
$recordingUrl = $_GET['RecordingUrl'];
$recordingUrl .= ".wav";

//Download recording file
exec("wget -N -O ./record.wav $recordingUrl");

//Run python
exec("python ./tweet.py");

?>

As you can see in the reference, RecordingUrl is sent as a parameter from the ʻactionattribute. You can receive it by GET or POST. It's the same as HTML and other forms. This time, I specified GET withmethod` of TwiML, so it is GET.

Note that parameters may not be obtained or audio may not be saved depending on the permission settings of the file or save destination directory. I was worried for a few days because I couldn't solve the super-super rudimentary permission-related problems.

After downloading the recorded voice file, execute the python program to perform voice recognition → tweet processing.

Write python

This article helped me create a Python program. How to convert Twilio's voice into text using Microsoft's Bing Speech API

First, create settings.py to refer to the API key for tweets.

settings.py


CONSUMER_KEY = "xxxxxxxxxxxxxxxxxxxxxxxxx"
CONSUMER_SECRET = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
ACCESS_TOKEN = "xxxxxxxxxxxxxxxxxx-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
ACCESS_TOKEN_SECRET = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

Next is a program that performs everything from voice recognition to tweets.

tweet.py


#coding:utf-8
from requests_oauthlib import OAuth1Session
import json
import settings
import requests
import urllib

#Get token from API
def authorize():

    url = "https://api.cognitive.microsoft.com/sts/v1.0/issueToken"

    headers = {
        "Content-type": "application/x-www-form-urlencoded",
        "Ocp-Apim-Subscription-Key": "Obtained key"
    }

    response = requests.post(url, headers=headers)

    if response.ok:
        _body = response.text
        return _body
    else:
        response.raise_for_status()

#Store the acquired token
token = authorize()

#File selection
infile = open("record.wav","r")
data = infile.read()

#Throw to API and get results
def speech_to_text( raw_data, token, lang="ja-JP", samplerate=8000, scenarios="ulm"):
    data = raw_data
    params = {
        "version": "3.0",
        "requestid": "b2c95ede-97eb-4c88-81e4-80f32d6aee54",
        "appid": "D4D52672-91D7-4C74-8AD8-42B1D98141A5",
        "format": "json",
        "locale": lang,
        "device.os": "Windows",
        "scenarios": scenarios,
        "instanceid": "565D69FF-E928-4B7E-87DA-9A750B96D9E3"
    }

    url = "https://speech.platform.bing.com/recognize?" + urllib.urlencode(params)
    headers = {"Content-type": "audio/wav; samplerate={0}".format(samplerate),
               "Authorization": "Bearer " + token }

    response = requests.post(url, data=data, headers=headers)

    if response.ok:
        result = response.json()["results"][0]
        return result["lexical"]
    else:
        raise response.raise_for_status()

#Store the data acquired by API
message = speech_to_text(data,token,lang="ja-JP", samplerate=8000, scenarios="ulm")

#settings.Get Token from py
twitter = OAuth1Session(settings.CONSUMER_KEY, settings.CONSUMER_SECRET, settings.ACCESS_TOKEN, settings.ACCESS_TOKEN_SECRET)

#Tweet
params = {"status":"%s"%message}
req = twitter.post("https://api.twitter.com/1.1/statuses/update.json",params = params

Try to call

When you call your Twilio number, you'll be told "Record your tweet. Press Sharp after recording." Tell them what you want to tweet. It will be tweeted after a while.

It doesn't seem to work if it's too long or too short.

Challenges point

It worked for the time being, but there are some issues.

1. Somehow an error occurs in Twilio (Added on October 26, 2017)

At the end of the call, it says "An application error has occurred". Twilio's debugger also shows an error, but it's not resolved.

→ I solved it, so I wrote an article. About system errors that allow you to tweet just by calling

2. Setting permissions

This time I was quite addicted to the rudimentary permission settings. There are some parts that I haven't grasped yet, so study is required.

3. Security story

I feel that it is not security-friendly, such as forcibly executing it with ʻexec ()` during the program. (Command injection) After that, it is necessary to devise PHP data transfer as a countermeasure for CSRF.

I want to restrict access to TwiML, but I wonder if it can be done.

4. Recorded files remain on Twilio (Added on October 27, 2017)

Recording files will remain on Twilio and will accumulate each time you tweet. It seems that it can be deleted with the API, so it seems that this problem can be solved.

→ I wrote a program for deletion, so I made it an article. Delete all recorded data with Twilio

Recommended Posts

I made a system that allows you to tweet just by making a phone call
I made a program to notify you by LINE when switches arrive
I made a fucking app that won't let you skip
I made you to execute a command from a web browser
[Python] I made a function that decrypts AES encryption just by throwing it with pycrypto.
I tried to make a system that fetches only deleted tweets
Don't you want to say that you made a face recognition program?
I made a Twitter bot that mutters Pokemon caught by #PokemonGO
I made a library that adds docstring to a Python stub file.
Created a service that allows you to search J League data
I wrote a book that allows you to learn machine learning implementations and algorithms in a well-balanced manner.
I made a web application in Python that converts Markdown to HTML
I made a library konoha that switches the tokenizer to a nice feeling
I made a script to display emoji
I made a system that automatically decides whether to run tomorrow with Python and adds it to Google Calendar.
I made a Line Bot that uses Python to retrieve unread Gmail emails!
I made a tool to easily display data as a graph by GUI operation.
I made a Docker image that can call FBX SDK Python from Node.js
I made a module in C language to filter images loaded by Python
A story that I was addicted to when I made SFTP communication with python
I made a tool to compile Hy natively
I made a tool to get new articles
[Python] I made a system to introduce "recipes I really want" from the recipe site!
I forgot to operate VIM, so I made a video for memorization. 3 videos by level
I made a library to separate Japanese sentences nicely
I made a VM that runs OpenCV for Python
I made a script to put a snippet in README.md
I made a Python module to translate comment outs
I tried to implement a recommendation system (content-based filtering)
I made a code to convert illustration2vec to keras model
A story that I was addicted to at np.where
I made a command to markdown the table clipboard
Allows you to select by name from Django User Name
I made a python library to do rolling rank
I made a toolsver that spits out OS, Python, modules and tool versions to Markdown
I made a tool that makes it a little easier to create and install a public key.
I tried to automatically generate OGP of a blog made with Hugo with tcardgen made by Go
A learning roadmap that allows you to develop and publish services from scratch with Python
I made a class to get the analysis result by MeCab in ndarray with python
I made a program to collect images in tweets that I liked on twitter with Python
I wrote a miscellaneous Ansible module that enables Virtualenv to be used by installing Pythonz.
A program that failed when trying to create a linebot with reference to "Dialogue system made with python"
Introducing "Sherlock", a CLI tool that allows you to search for user names across SNS
I made a tool that makes it convenient to set parameters for machine learning models.
A Python script that allows you to check the status of the server from your browser