Use OpenCV_Contrib (ArUco) in Java! (Part 2-Programming)


This article is a front and back part. If you haven't built the ArUco library, Use OpenCV_Contrib (ArUco) in Java! (Part 1-Build).

Also, the source code used this time is mostly in OpenCV_ArucoTest [GitHub], so I would appreciate it if you could refer to it.

This article describes the program when using the ArUco library in Java.

I was thinking of writing it sooner, but I was late. (I started writing the night before New Year's Eve ...) For the time being, I am registered for OpenCV Advent Calendar 2018. (What did you say after posting on New Year 2019)

Preparing to use ArUco in Eclipse

Follow the procedure for importing an existing OpenCV with the jar file included in Contrib.

Marker image generation

To make sure the ArUco library is properly built and set up Let's generate a Marker image.

The function to use is ・ Aruco.drawMarker (dictionary, markerID, sidePixels, markerImage) dictionary: Determines the type of Marker. The size, resolution, and number of types of squares are different. markerID: The ID of the Marker defined in the Dictionary, each with a different shape. sidePixels: Determines the resolution.

Source code below (Details: createMarker [GitHub])

	public static void createMarker() {
		Dictionary dictionary = Aruco.getPredefinedDictionary(Aruco.DICT_4X4_50);

		final int markerID = 0;
		final int sidePixels = 200;
		Mat markerImage = new Mat();
		Aruco.drawMarker(dictionary, markerID, sidePixels, markerImage);

		Imgcodecs.imwrite("F:\\users\\smk7758\\Desktop\\marker_2018-12-01.png ", markerImage);

marker_2018-12-01.png You should get something like this.

Marker recognition

Well, it's the production from here.

The following two functions are used ・ Aruco.detectMarkers (inputImage, dictionary, corners, markerIds, parameters); ・ Aruco.drawDetectedMarkers (inputImage, corners, markerIds); Well, as with the function, detectMarkers recognizes multiple markers. After that, using corners, drawDetectedMarkers does something like tracing the four corners of Marker. corners: Returns the on-screen coordinates of Marker. (The reason why List is because it recognizes multiple Markers and Mat is a matrix and can store coordinates.)

(Other arguments omitted)

Source code below (Details: detectMarker [GitHub])

	public static void detectMarker() {
		Dictionary dictionary = Aruco.getPredefinedDictionary(Aruco.DICT_4X4_50);

		Mat inputImage = Imgcodecs.imread("F:\\users\\smk7758\\Desktop\\marker_2018-12-01_test.png ");
		List<Mat> corners = new ArrayList<>();
		Mat markerIds = new Mat();
		DetectorParameters parameters = DetectorParameters.create();
		Aruco.detectMarkers(inputImage, dictionary, corners, markerIds, parameters);

		Aruco.drawDetectedMarkers(inputImage, corners, markerIds);

		Imgcodecs.imwrite("F:\\users\\smk7758\\Desktop\\marker_2018-12-01_detected.png ", inputImage);

marker_2018-12-01_test.png It was transformed with GIMP, but in the image above,

marker_2018-12-01_detected.png It should look like this.

Real-time Marker recognition using a camera

To be honest, when I came here, it was almost self-satisfaction, but next I combined JavaFX and VideoCapture (OpenCV) to perform real-time recognition.

The difference is that it runs on JavaFX, so using the ScheduledService, the value returned is of type Image. ScheduledService is a class used when performing multi-thread processing with JavaFX, and it seems that Task can be automatically restored. See the official JavaDoc for details. Also, the reason for returning with Image type is that the element that displays the image defined on the Controller class side is the argument. This is to take an image. For the one that converts Mat to Image, I used the one that came out after proper research.

Source code below (Details: [detectMarkerByCamera [GitHub]]( /

	protected Task<Image> createTask() {
		return new Task<Image>() {
			protected Image call() throws Exception {
				if (!vc.isOpened()) {
					System.err.println("VC is not opened.");
					return null;

				Mat inputImage = new Mat();

				if (! || inputImage == null) {
					System.err.println("Cannot load camera image.");
					return null;

				List<Mat> corners = new ArrayList<>();
				Mat markerIds = new Mat();
				// DetectorParameters parameters = DetectorParameters.create();
				Aruco.detectMarkers(inputImage, dictionary, corners, markerIds);

				Aruco.drawDetectedMarkers(inputImage, corners, markerIds);

				return convertMatToImage(inputImage);

	private Image convertMatToImage(Mat inputImage) {
		MatOfByte byte_mat = new MatOfByte();
		Imgcodecs.imencode(".bmp", inputImage, byte_mat);

		return new Image(new ByteArrayInputStream(byte_mat.toArray()));

SC_test_2018-12-2_21-37-9_No-00.png It should look like this.

Also, [detectMarker (Center) CoordinatesByCamera [GitHub]]( In), the function on the OpenCV side is used to find the coordinates of the center (center of gravity) from the points at the four corners of Marker and draw the points.

Real-time Marker attitude estimation using a camera

It is posture estimation that you want to do the most.

Well, I would like to do it immediately, but in fact there is something to do before estimating the posture. It's called camera calibration. The following function cannot be executed without the Mat obtained here, so refer to I did OpenCV camera calibration with Java. I want you to do it. (Actually, it took a couple of weeks to develop from the previous chapter. This is the cause of the delay)

Let's start with the function again. ・ Aruco.estimatePoseSingleMarkers (corners, 0.05f, cameraMatrix, distortionCoefficients, rotationMatrix, translationVectors); ・ Aruco.drawAxis (inputImage, cameraMatrix, distortionCoefficients, rotationMatrix, translationVectors, 0.1f);

As the name suggests, the above function estimates Marker's attitude and returns a rotation matrix and a translation vector. And the function below draws the Axis, or coordinate axis.

Source code below (Details: detectMarkerPoseByCamera [GitHub])

	List<Mat> corners = new ArrayList<>();
	Mat markerIds = new Mat();
	// DetectorParameters parameters = DetectorParameters.create();
	Aruco.detectMarkers(inputImage, dictionary, corners, markerIds);

	Aruco.drawDetectedMarkers(inputImage, corners, markerIds);

	Mat rotationMatrix = new Mat(), translationVectors = new Mat(); //receive
	Aruco.estimatePoseSingleMarkers(corners, 0.05f, cameraMatrix, distortionCoefficients,rotationMatrix, translationVectors);

	for (int i = 0; i < markerIds.size().height; i++) { // TODO
		Aruco.drawAxis(inputImage, cameraMatrix, distortionCoefficients, rotationMatrix, translationVectors, 0.1f);

SC_test_2018-12-31_0-36-22_No-00.png SC_test_2018-12-31_0-36-13_No-00.png SC_test_2018-12-31_0-35-37_No-00.png It should look like this.

in conclusion

It took me some time, but it started working for the time being. It was especially difficult to understand camera calibration ... I would like to continue to devote myself to the Java version of OpenCV.

(I enjoyed it at the end of 2018) [Although the article itself was written at the beginning of 2019]

Also, since this article is written by a beginner, if you have any mistakes or bad points, please let us know in the comments or on Twitter.

Reference link

The formula was the easiest to understand.

-ArUco marker detection (aruco module) --OpenCV officialDetection of ArUco Markers --OpenCV OfficialPose estimation with Opencv Aruco Part 1-Machine learning memorandumPose estimation with Opencv Aruco Part 2-Machine learning memorandum -OpenCV aruco marker-Personal notes related to the program ・ [Try using the aruco module --atinfinity / lab (GitHub)]( % E3% 83% BC% E3% 83% AB% E3% 82% 92% E4% BD% BF% E3% 81% A3% E3% 81% A6% E3% 81% BF% E3% 82% 8B) ・ Camera position and orientation estimation using markers (OpenCV + ArUco) -How to make a flying robotInvestigating ArUco, a lightweight AR libraryFor myself: OpenCV3.4.3 camera calibration

Recommended Posts

Use OpenCV_Contrib (ArUco) in Java! (Part 2-Programming)
Use OpenCV_Contrib (ArUco) in Java! (Part 1-Build) (OpenCV-3.4.4)
Constraint programming in Java
Use OpenCV in Java
Use PreparedStatement in Java
Use Redis Stream in Java
[Java] Basic terms in programming
Let's use Twilio in Java! (Introduction)
[Java] Do not use "+" in append!
Use composite keys in Java Map.
How to use classes in Java?
Creating lexical analysis in Java 8 (Part 2)
Do you use Stream in Java?
Creating lexical analysis in Java 8 (Part 1)
Multilingual Locale in Java How to use Locale
[Java] Use cryptography in the standard library
Use "Rhino" which runs JavaScript in Java
Creating a matrix class in Java Part 1
Let's think about what declarative programming is in Java and Elm (Part 1)
Partization in Java
The story of learning Java in the first programming
java programming basics
Changes in Java 11
[Java] Use of final in local variable declaration
Rock-paper-scissors in Java
java practice part 1
java Generic Programming
What I learned in Java (Part 2) What are variables?
[JAVA] [Spring] [MyBatis] Use IN () with SQL Builder
[Java] Use Collectors.collectingAndThen
Pi in Java
Why use setters/getters instead of public/private in Java
FizzBuzz in Java
Programming with direct sum types in Java (Neta)
[LeJOS] Let's program mindstorm-EV3 in Java [Environment construction part 2]
A quick review of Java learned in class part4
I want to use ES2015 in Java too! → (´ ・ ω ・ `)
What I learned in Java (Part 3) Instruction execution statement
A quick review of Java learned in class part3
A quick review of Java learned in class part2
[JAVA] [Spring] [MyBatis] Use GROUP BY in SQL Builder
Use of Abstract Class and Interface properly in Java
~ I tried to learn functional programming in Java now ~
Notes on how to use regular expressions in Java
Quick learning Java "Introduction?" Part 3 Talking away from programming
[java] sort in list
Read JSON in Java
Interpreter implementation in Java
Make Blackjack in Java
Rock-paper-scissors app in Java
Put java8 in centos7
NVL-ish guy in Java
Combine arrays in Java
"Hello World" in Java
Callable Interface in Java
Use java.time in Jackson
Comments in Java source
Azure functions in java
Format XML in Java
Simple htmlspecialchars in Java
Boyer-Moore implementation in Java