I tried using OpenCV with Java + Tomcat

This article is the second day article of WebCrew Advent Calendar 2017. Yesterday was @ kouares's "Implement CRUD with PlayFramework 2.6.7, play-slick 3.0.2 (Slick 3.2)".

** Introduction **

@Hahegawa will be in charge of the second day. Thank you.


――In my work, I had the opportunity to want to detect an object from an image, so I tried OpenCV. --As for OpenCV, there are many implementation examples and explanations in Python and C ++ in the streets, but considering maintenance for business use, I thought it would be nice if it could be done in Java, so I tried it in Java. --Also, there were few articles that mentioned the "overall flow" from installation to execution when using OpenCV with Java, so I tried to summarize it with that in mind.

** Premise **

--This time, I tried "feature detection" and "feature matching". --For the mechanism and implementation image of OpenCV itself, I referred to the following. --Mechanism: http://labs.eecs.tottori-u.ac.jp/sd/Member/oyamada/OpenCV/html/py_tutorials/py_feature2d/py_table_of_contents_feature2d/py_table_of_contents_feature2d.html#py-table-of-content-feature2d --How to implement: https://qiita.com/hmichu/items/f5f1c778a155c7c414fd


It's a little old, but I tried it in the following environment. ・ Windows 10 ・ Eclipse Oxygen ・ OpenCV3.1 -Apache Tomcat 7.0

** Project construction **

    1. Eclipse installation and project creation Omitted in this article
  1. Download OpenCV library From the following page, download and unzip the library of the Ver you want to use to an appropriate location      https://opencv.org/releases.html アドベントカレンダー1-min.png

    1. Added Windows system environment variables Add the folder under "\ build \ java " in the unzipped folder to Path アドベントカレンダー2-min.png

4. Right-click the Eclipse project → Set build path in Properties アドベントカレンダー3-min.png

  1. Add user library When adding, set the same path as 3 to the location of the native library アドベントカレンダー4-min.png

6. Add runtime configuration When running OpenCV assuming a web application, set the classpath of the runtime configuration to 5 Add the same user library アドベントカレンダー5-min.png

  1. Source code creation By following the processing procedure below, feature points were detected and matched from the image. (1) Read the image and store it in the BufferedImage class (2) Convert from BufferedImage class to Mat class → When processing an image with the method of OpenCV library Basically using this Mat class (3) Feature detector generation → This time, use AKAZE (4) Feature detection execution + cross check → The detection accuracy will be higher if the cross check process is added. Processing time and load will increase, so it is a point of concern (5) Draw the detection result of the feature point

(2)~(5)Source code

  //Mat conversion of images sent from the screen
  Mat src = new Mat(bufImg.getHeight(), bufImg.getWidth(), CvType.CV_8UC3);
  byte[] bufImgBinary = ((DataBufferByte)bufImg.getRaster().getDataBuffer()).getData();
  src.put(0, 0, bufImgBinary);

  //Mat conversion of template image used for feature detection
  Mat tempSrc = new Mat(tmpImg.getHeight(), tmpImg.getWidth(), CvType.CV_8UC3);
  byte[] tempImgBinary = ((DataBufferByte)tmpImg.getRaster().getDataBuffer()).getData();
  tempSrc.put(0, 0, tempImgBinary);

  //Grayscale conversion for feature detection
  Mat srcGray = new Mat(bufImg.getHeight(), bufImg.getWidth(), CvType.CV_8UC1);
  Imgproc.cvtColor(src, srcGray, Imgproc.COLOR_BGR2GRAY);
  Core.normalize(srcGray, srcGray, 0, 255, Core.NORM_MINMAX);
  Mat tempSrcGray = new Mat(tmpImg.getHeight(), tmpImg.getWidth(), CvType.CV_8UC1);
  Imgproc.cvtColor(tempSrc, tempSrcGray, Imgproc.COLOR_BGR2GRAY);
  Core.normalize(tempSrcGray, tempSrcGray, 0, 255, Core.NORM_MINMAX);

  MatOfKeyPoint keyPointSrc = new MatOfKeyPoint();
  MatOfKeyPoint keyPointTemp = new MatOfKeyPoint();
  Mat descriptersSrc = new Mat(src.rows(), src.cols(), src.type());
  Mat descriptersBase = new Mat(tempSrc.rows(), tempSrc.cols(), tempSrc.type());

  //Feature detector generation 1
  FeatureDetector akazeDetector = FeatureDetector.create(FeatureDetector.AKAZE);
  DescriptorExtractor akazeExtractor = DescriptorExtractor.create(DescriptorExtractor.AKAZE);
  akazeDetector.detect(srcGray, keyPointSrc);  
  akazeDetector.detect(tempSrcGray, keyPointTemp);
  akazeExtractor.compute(srcGray, keyPointSrc, descriptersSrc);  
  akazeExtractor.compute(tempSrcGray, keyPointTemp, descriptersBase);

  //Feature detector generation 2
  DescriptorMatcher matcher = DescriptorMatcher.create(DescriptorMatcher.BRUTEFORCE);

  MatOfDMatch matche = new MatOfDMatch();
  List<MatOfDMatch> matches = new ArrayList<MatOfDMatch>();
  List<DMatch> dmatch = new ArrayList<DMatch>();

  //Features Cross check available

    MatOfDMatch srcToBase = new MatOfDMatch();
    MatOfDMatch baseToSrc = new MatOfDMatch();
    matcher.match(descriptersSrc, descriptersBase, srcToBase);
    matcher.match(descriptersBase, descriptersSrc, baseToSrc);

    List<DMatch> ldm_srcToBase = srcToBase.toList();
    List<DMatch> ldm_baseToSrc = baseToSrc.toList();

    for(int i=0; i<ldm_srcToBase.size(); i++) {

      DMatch forward = ldm_srcToBase.get(i);
      DMatch backward = ldm_baseToSrc.get(forward.trainIdx);

      if(backward.trainIdx == forward.queryIdx){



  //Features No cross check

    System.out.println("Features Normal check execution");
    matcher.match(descriptersSrc, descriptersBase, matche);


  Mat matchResult = new Mat(src.rows()*2, src.cols()*2, src.type());
  Features2d.drawMatches(src, keyPointSrc, tempSrc, keyPointTemp, matche, matchResult);
  //↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓ Verification image output ↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓
  BufferedImage image = ImageConverter.convertMToBI(matchResult);
  File ouptut = new File("C:\\testlog\\sample_20171202.jpg ");
  ImageIO.write(image, "jpg", ouptut);

Load OpenCV library


Execution result

Although there are some misrecognitions, the features could be detected. 6-min

** What I thought about doing **

――It took me a long time to write the code because it was slow, but if you are quick to come up with or write the code, you can easily process the image if you understand the basics and explanations of various methods described in this article. I felt that I could do it. ――When planning your business, what you really want to use is "what to do before feature detection", so read this article and focus on the processing you really want to implement without extra man-hours. I would appreciate it if you could.

Tomorrow is @wc_moriyama. Thank you.

Recommended Posts

I tried using OpenCV with Java + Tomcat
I tried using Java REPL
I tried using JOOQ with Gradle
I tried to interact with Java
I tried UDP communication with Java
I tried using Java8 Stream API
I tried using JWT in Java
I tried using Java memo LocalDate
I tried using GoogleHttpClient of Java
[Java] I tried to connect using a connection pool with Servlet (tomcat) & MySQL & Java
I tried using the CameraX library with Android Java Fragment
I tried using Elasticsearch API in Java
I tried using Realm with Swift UI
I tried using Scalar DL with Docker
I tried using OnlineConverter with SpringBoot + JODConverter
I tried using Gson
I tried using TestNG
I tried using Galasa
I tried to make Basic authentication with Java
I tried to break a block with java (1)
I tried DI with Ruby
I tried to implement TCP / IP + BIO with JAVA
[Java 11] I tried to execute Java without compiling with javac
I tried using Google Cloud Vision API in Java
I tried to operate SQS using AWS Java SDK
I tried using azure cloud-init
I tried Drools (Java, InputStream)
I tried using Log4j2 on a Java EE server
I tried using Apache Wicket
I tried OCR processing a PDF file with Java
Using Mapper with Java (Spring)
I tried to implement Stalin sort with Java Collector
I tried using an extended for statement in Java
I tried UPSERT with PostgreSQL.
I tried BIND with Docker
I tried scraping a stock chart using Java (Jsoup)
I tried metaprogramming in Java
I tried to create a java8 development environment with Chocolatey
I tried to modernize a Java EE application with OpenShift.
I tried using Dapr in Java to facilitate microservice development
I tried OCR processing a PDF file with Java part2
I tried to get started with Swagger using Spring Boot
I tried using anakia + Jing now
Install Java and Tomcat with Ansible
I tried using Spring + Mybatis + DbUnit
I tried morphological analysis with MeCab
Try using Redis with Java (jar)
Enable OpenCV with java8. (For myself)
I tried the Java framework "Quarkus"
Using Java with AWS Lambda-Eclipse Preparation
What I learned with Java Gold
Html5 development with Java using TeaVM
I tried GraphQL with Spring Boot
I tried to summarize Java learning (1)
[Android] I tried using Coordinator Layout.
I tried Flyway with Spring Boot
I tried using Pari gp container
What I learned with Java Silver
I tried using WebAssembly Stadio (2018/4/17 version)
I tried to summarize Java 8 now
Using proxy service with Java crawling