I'm creating a portfolio of web apps in Ruby on Rails. I'm a Rails beginner. This time, I implemented a simple image analysis work at the time of image preview before posting using Google Cloud Vision API, so I will leave it as a memorandum.
The contents implemented this time are as follows.
The finished product looks like this.
I am creating a web service called "is your sky", an SNS that shares the sky I took. https://isyoursky-web.herokuapp.com/
Since the sky is the main service, I would like users to post ** images showing the sky **! I often see the function that prevents you from posting radical images, but the purpose is a little different. This time, as a policy, we had to ** "prompt users to post empty images" **. So, once the image on the posting screen is not an empty image at the time of preview, I thought about issuing an alert and letting the user notice it voluntarily. That is the function of 1.
In addition, the posting screen also has a tag function, but it would be convenient if tags matching the photos were automatically added! I also created the function of 2. __ * However, as I found out after implementing this function, the text on the label does not function as a tag very much. Words such as Sky and red come out relatively, so it doesn't make much sense as a tag ... For the time being, I will leave it as a memorandum, but it is a policy not to adopt it in my web application. __
It is one of the services of Vision AI, an image analysis service that utilizes machine learning provided by Google.
Google Cloud's Vision API provides a powerful pre-trained machine learning model via REST API and RPC API. Assigning labels to images allows you to quickly classify images into millions of predefined categories. Detects objects and faces, reads printed text and handwriting, and creates useful metadata in image catalogs -Quoted from the official website
In addition to Google, IBM, AWS, Window, etc. provide such image analysis APIs, Google has high accuracy of object extraction and labeling functions I chose it because I thought it would be the best match for what I wanted to achieve this time. Another big factor is that I was originally using Google Cloud Platform.
reference) Comparison of image analysis API services Google's image recognition API is the strongest! !! Image recognition API thorough comparison result
Log in to Google Cloud Platform with your google account and create a project, You have already got the json file for the service account key. Please refer to various reference sites for this area. reference) Let's analyze images with Google Cloud Vision API [Introduction] Introducing Google Cloud Vision API to Rails to easily detect radical images
I usually create containers with Docker for Mac and develop Rails.
ruby 2.4.5
Ruby on rails 5.0.7.2
Install gem to use google cloud vision API
Gemfile
gem 'google-cloud-vision'
For the time being bundle install
terminal
bundle install
Gemfile.lock
google-cloud-vision (1.0.0)
In this case, I will write it in the jQuery code that fires at the timing of the image preview. After the user selects an image, the flow is as follows.
It's a little confusing when it's written, but since it's mainly the js file and Controller that process it, We will focus on the processing in that area.
View
First is the View code.
A general form with code for image preview.
Added a span class with an id of ʻimage_isnot_sky at the bottom of the ʻimage-preview
class. If it is determined that the image is not empty, this part is hidden and displayed as characters on the screen.
For the tag function, tag-it is used. Reference) tag-it
ruby:_form.html.slim
= form_with model: post, local: true do |f|
:
(Omission)
.form-group
= f.label :image, "image"
= f.file_field :image, class:"form-control", id: 'post_img'
.image-preview
img[id="image_img_prev" src="#" class='hidden']
- if post.persisted? && post.image?
= image_tag post.image.url, class: 'image_present_img'
label
= f.check_box :remove_image
|Delete image
- else
= f.hidden_field :avatar_cache
span[id="image_isnot_sky" class="hidden"]
|It may not be an empty image. Please post an empty photo
#post-tags-field
= f.label "tag"
ul[id="post-tags"]
= hidden_field_tag :tag_hidden, tag_list
:
javascript(jQuery)
As we saw earlier, jQuery fires when the file is loaded into the View's file_field.
In the part of $ .ajax
, asynchronous communication is performed, and if it succeeds, it enters done
, and if it fails, it enters fail
.
Put controller and method name
in url, POST
in type, and put the data you want to communicate this time (formData in this case) in data.
I haven't caught up with the parts of processData: false
and contentType: false
, but if I remove them, the data will be processed and sent, so it will not be able to be read properly. ..
(I didn't write here and got an error for a while ...)
post.js
:
function readURL(input) {
if (input.files && input.files[0]) {
var reader = new FileReader();
reader.onload = function (e) {
//Open the file uploaded with formData
var formData = new FormData();
formData.append("image_cache", input.files[0]);
//ajax communication processing
$.ajax({
url: "/posts/check_cache_image",
type: "POST",
data: formData,
dataType: "json",
processData: false,
contentType: false,
}).done(function(data){
var tag_list = data.data.tag_list
var image_flag = data.data.image_flag
if(image_flag == true){
//Show alerts
$('#image_isnot_sky').removeClass('hidden');
//Delete tag
$("#post-tags").tagit("removeAll");
}else{
//Hide alerts if they have already been displayed
$('#image_isnot_sky').addClass('hidden');
//Delete tag
$("#post-tags").tagit("removeAll");
//Create a tag based on the label
$.each(tag_list, function(index, tag){
$('#post-tags').tagit('createTag', tag);
})
}
}).fail(function(){
//In case of error, issue an alert.
alert('Failed to load the image');
})
//Perform image preview
$('#image_img_prev').attr('src', e.target.result);
}
reader.readAsDataURL(input.files[0]);
}
}
:
Also, routes are essential for asynchronous communication. I will add it.
route/.rb
Rails.application.routes.draw do
:
post "posts/check_cache_image"
:
end
The variable data
used in done
stores the data returned from the controller using jBuilder in json format.
jBuilder can be used by creating a controller name.json.jbuilder
in the same folder in view.
See below for details
Reference) https://pikawaka.com/rails/jbuilder
:check_cache_image.json.jbuilder
json.data do |data|
json.image_flag @data[:image_flag]
json.tag_list @data[:tag_list]
end
Controller
From here, let's take a closer look. The first is the link with the Vision API. This time, we will use the function ** "Label detection" ** in the API.
For the environment variable, specify the file path of the json file of the service account mentioned at the beginning.
post_controller.rb
:
def check_cache_image
require "google/cloud/vision"
#Vision API settings
image_annotator = Google::Cloud::Vision.image_annotator do | config |
config.credentials = ENV["GOOGLE_APPLICATION_CREDENTIALS"]
end
:
Get the image_cache brought by formData and pass it to the Vision API.
The code here is based on the Official Code almost as it is.
I am getting the label name with label.description
.
post_controller.rb
:
#Get cached information
image = params[:image_cache].path
#Pass the cache to the Vision API as a response.
response = image_annotator.label_detection image: image
#Put label in an array
label_list = []
response.responses.each do |res|
res.label_annotations.each do |label|
label_list.push(label.description)
end
end
:
After that, various necessary information is acquired based on the detected label.
post_controller.rb
:
#1.Determine if it is an empty photo
if label_list.include?("Sky")
@image_flag = false
else
@image_flag = true
end
#2.Bring the first four labels as tags
if label_list.length >= 4
@tag_list = label_list[0..3]
else
@tag_list = label_list
end
:
The last is JSON processing.
post_controller.rb
:
#Ready to send to jBuilder
@data = { tag_list: @tag_list, image_flag: @image_flag }
respond_to do |format|
format.html
format.json
end
end
:
With the above, at the timing of the image preview, you can "determine whether it is an empty image" and "add a tag automatically". It may be quite impossible, but I was able to implement it for the time being ... I would appreciate it if you could point out any mistakes.
I was able to implement simple image analysis using the Google Cloud Vision API. I have sent some photos at random in the test, but if there is even a little sky, "Sky" will be labeled properly. It's too amazing ...! It's huge that you can use this for free. (If you go a certain number, you will be charged ...) It was easier to implement than I expected, so I would like to use other functions as well :)
Thank you very much! Asynchronously read text from food package image using Google Cloud Vision API and fill in form Rails: How to hit the action of the controller with jquery and return the variable to jquery Introducing Google Cloud Vision API to Rails to easily detect radical images
Recommended Posts