Get data labels by linking with Google Cloud Vision API when previewing images with Rails

Introduction

I'm creating a portfolio of web apps in Ruby on Rails. I'm a Rails beginner. This time, I implemented a simple image analysis work at the time of image preview before posting using Google Cloud Vision API, so I will leave it as a memorandum.

Overview

The contents implemented this time are as follows.

1. When uploading an image on the posting screen, "is it an empty image" is determined at the same time as the image preview, and if it is not an empty image, "not an empty image" is output.
2. If it is determined to be an empty image, the top four labels of the analysis result are automatically output as tags.

The finished product looks like this.

The reason I decided to do it

I am creating a web service called "is your sky", an SNS that shares the sky I took. https://isyoursky-web.herokuapp.com/

Since the sky is the main service, I would like users to post ** images showing the sky **! I often see the function that prevents you from posting radical images, but the purpose is a little different. This time, as a policy, we had to ** "prompt users to post empty images" **. So, once the image on the posting screen is not an empty image at the time of preview, I thought about issuing an alert and letting the user notice it voluntarily. That is the function of 1.

In addition, the posting screen also has a tag function, but it would be convenient if tags matching the photos were automatically added! I also created the function of 2. __ * However, as I found out after implementing this function, the text on the label does not function as a tag very much. Words such as Sky and red come out relatively, so it doesn't make much sense as a tag ... For the time being, I will leave it as a memorandum, but it is a policy not to adopt it in my web application. __

What is the Google Cloud Vision API?

It is one of the services of Vision AI, an image analysis service that utilizes machine learning provided by Google.

Google Cloud's Vision API provides a powerful pre-trained machine learning model via REST API and RPC API. Assigning labels to images allows you to quickly classify images into millions of predefined categories. Detects objects and faces, reads printed text and handwriting, and creates useful metadata in image catalogs -Quoted from the official website

In addition to Google, IBM, AWS, Window, etc. provide such image analysis APIs, Google has high accuracy of object extraction and labeling functions I chose it because I thought it would be the best match for what I wanted to achieve this time. Another big factor is that I was originally using Google Cloud Platform.

reference) Comparison of image analysis API services Google's image recognition API is the strongest! !! Image recognition API thorough comparison result

Premise

Log in to Google Cloud Platform with your google account and create a project, You have already got the json file for the service account key. Please refer to various reference sites for this area. reference) Let's analyze images with Google Cloud Vision API [Introduction] Introducing Google Cloud Vision API to Rails to easily detect radical images

environment

I usually create containers with Docker for Mac and develop Rails.

ruby 2.4.5
Ruby on rails 5.0.7.2

Preparation

Install gem to use google cloud vision API

Gemfile

gem 'google-cloud-vision'

For the time being bundle install

terminal

bundle install

• As of October 2020, the installed vers are as follows. This time, I will write in this 1.0.0 way of writing.

Gemfile.lock

google-cloud-vision (1.0.0)

• Google-cloud-vision seems to have been significantly updated with the 1.0.0 release. There are some parts that are different from the previous version, so for your reference. Details are listed in Doc, so please see here for details ...

How to implement

In this case, I will write it in the jQuery code that fires at the timing of the image preview. After the user selects an image, the flow is as follows.

1. jQuery fires
2. Send the image uploaded by the user to the controller by asynchronous communication (Ajax)
3. Link to Vision API in the controller and set various information in instance variables
4. Use jBuilder to convert the instance variables into an array and return it to jQuery.
5. Process on jQuery side and display on screen

It's a little confusing when it's written, but since it's mainly the js file and Controller that process it, We will focus on the processing in that area.

code

View First is the View code. A general form with code for image preview. Added a span class with an id of ʻimage_isnot_sky at the bottom of the ʻimage-preview class. If it is determined that the image is not empty, this part is hidden and displayed as characters on the screen.

For the tag function, tag-it is used. Reference) tag-it

ruby:_form.html.slim

= form_with model: post, local: true do |f|
:
(Omission)
  .form-group
    = f.label :image, "image"
    = f.file_field :image, class:"form-control", id: 'post_img'
  .image-preview
    img[id="image_img_prev" src="#" class='hidden']
    - if post.persisted? && post.image?
      = image_tag post.image.url, class: 'image_present_img'
      label
        = f.check_box :remove_image
        |Delete image
    - else
      = f.hidden_field :avatar_cache
    span[id="image_isnot_sky" class="hidden"]
      |It may not be an empty image. Please post an empty photo


  #post-tags-field
    = f.label "tag"
    ul[id="post-tags"]
    = hidden_field_tag :tag_hidden, tag_list
:

javascript(jQuery)

As we saw earlier, jQuery fires when the file is loaded into the View's file_field. In the part of $ .ajax, asynchronous communication is performed, and if it succeeds, it enters done, and if it fails, it enters fail. Put controller and method name in url, POST in type, and put the data you want to communicate this time (formData in this case) in data.

I haven't caught up with the parts of processData: false and contentType: false, but if I remove them, the data will be processed and sent, so it will not be able to be read properly. .. (I didn't write here and got an error for a while ...)

post.js

:
  function readURL(input) {
    if (input.files && input.files[0]) {
      var reader = new FileReader();

      reader.onload = function (e) {
        //Open the file uploaded with formData
        var formData = new FormData();
        formData.append("image_cache", input.files[0]);
        //ajax communication processing
        $.ajax({
          url: "/posts/check_cache_image",
          type: "POST",
          data: formData,
          dataType: "json",
          processData: false,
          contentType: false,
        }).done(function(data){
           var tag_list = data.data.tag_list
           var image_flag = data.data.image_flag

           if(image_flag == true){
             //Show alerts
             $('#image_isnot_sky').removeClass('hidden');
             //Delete tag
             $("#post-tags").tagit("removeAll");
           }else{
             //Hide alerts if they have already been displayed
             $('#image_isnot_sky').addClass('hidden');
             //Delete tag
             $("#post-tags").tagit("removeAll");

             //Create a tag based on the label
             $.each(tag_list, function(index, tag){
               $('#post-tags').tagit('createTag', tag);
             })
           }

        }).fail(function(){
          //In case of error, issue an alert.
          alert('Failed to load the image');
        })
        //Perform image preview
        $('#image_img_prev').attr('src', e.target.result);
      }
      reader.readAsDataURL(input.files[0]);
    }
  }
:

Also, routes are essential for asynchronous communication. I will add it.

route/.rb

Rails.application.routes.draw do
:
  post "posts/check_cache_image"
:
end

JSON (using jbuilder)

The variable data used in done stores the data returned from the controller using jBuilder in json format. jBuilder can be used by creating a controller name.json.jbuilder in the same folder in view. See below for details Reference) https://pikawaka.com/rails/jbuilder

:check_cache_image.json.jbuilder

json.data do |data|
  json.image_flag @data[:image_flag]
  json.tag_list @data[:tag_list]
end

Controller

From here, let's take a closer look. The first is the link with the Vision API. This time, we will use the function ** "Label detection" ** in the API.

For the environment variable, specify the file path of the json file of the service account mentioned at the beginning.

• The way to write config has been changed as follows from 1.0.0.

post_controller.rb

:
  def check_cache_image
    require "google/cloud/vision"

    #Vision API settings
    image_annotator  = Google::Cloud::Vision.image_annotator do | config |
      config.credentials = ENV["GOOGLE_APPLICATION_CREDENTIALS"]
    end
:

Get the image_cache brought by formData and pass it to the Vision API. The code here is based on the Official Code almost as it is. I am getting the label name with label.description.

post_controller.rb

:
    #Get cached information
    image = params[:image_cache].path
    #Pass the cache to the Vision API as a response.
    response = image_annotator.label_detection image: image

    #Put label in an array
    label_list = []
    response.responses.each do |res|
      res.label_annotations.each do |label|
        label_list.push(label.description)
      end
    end
:

After that, various necessary information is acquired based on the detected label.

1. If there is "Sky" in the label, it is judged as "empty photo"
2. Get the top 4 items on the label and display them as tags

post_controller.rb

:
    #1.Determine if it is an empty photo
    if label_list.include?("Sky")
      @image_flag = false
    else
      @image_flag = true
    end

    #2.Bring the first four labels as tags
    if label_list.length >= 4
      @tag_list = label_list[0..3]
    else
      @tag_list = label_list
    end
:

The last is JSON processing.

post_controller.rb

:
    #Ready to send to jBuilder
    @data = { tag_list: @tag_list, image_flag: @image_flag }
    
    respond_to do |format|
      format.html
      format.json
    end
  end
:

With the above, at the timing of the image preview, you can "determine whether it is an empty image" and "add a tag automatically". It may be quite impossible, but I was able to implement it for the time being ... I would appreciate it if you could point out any mistakes.

At the end

I was able to implement simple image analysis using the Google Cloud Vision API. I have sent some photos at random in the test, but if there is even a little sky, "Sky" will be labeled properly. It's too amazing ...! It's huge that you can use this for free. (If you go a certain number, you will be charged ...) It was easier to implement than I expected, so I would like to use other functions as well :)

Helpful site

Thank you very much! Asynchronously read text from food package image using Google Cloud Vision API and fill in form Rails: How to hit the action of the controller with jquery and return the variable to jquery Introducing Google Cloud Vision API to Rails to easily detect radical images