Introduction

I had a chance to find out about "detecting similar images" for a moment, and in the process I was thinking about videos.

Extract images from videos
Compare images

＼(＾o＾)／

I came up with the simple idea, and decided to make a little. ** Under development **

screen image

environment

Java (Java 9) is used on Windows, and Eclipse is used as the IDE. GUI creation is Scene Builder, or JavaFX.

Windows7 64bit
OpenCV 3.3.0 --Eclipse 4.7.1 (using pleiades ulitimate)
- e(fx)clipse
Scene Builder 9.0

Installing OpenCV will create a jar and a dll. Place them in a place where they can be read.

opencv/build/java/x64/opencv_java330.dll
opnecv/build/bin/opencv_ffmpeg330_64.dll

→ Copy to Windows \ System32 \

opencv/build/java/opencv-330.jar

→ Copy under classpath, add from Eclipse

Furthermore, in Java, it was necessary to describe the load process in a class that uses OpenCV as shown below.

	static {
		System.loadLibrary(Core.NATIVE_LIBRARY_NAME);
	}

Extract images from videos

Extract some frames from the video as images. Here, I decided to extract 9 images. There is no particular basis, but it is a judgment that it looks good when arranged in GUI with 3x3. Since the first and last frames are likely to be darkened, save 9 frames separated by 10. -> FPS difference detection ...?

Also, all the extracted images are scaled so that the width is 200px before saving. -> Detection of size difference ...?

↓ It looks like this.

	static public VideoMetadata captureFrame(Config config, String videoFile) {
		// ------------------------------------------------------------------
		//Open the video
		// ------------------------------------------------------------------
		System.out.println(videoFile);
		VideoCapture capture = new VideoCapture(videoFile);
		if (!capture.isOpened()) {
			System.out.println("file not opened.");
			return null;
		}

		String outputDir = config.getTempDir() + "/" + String.valueOf(videoFile.hashCode()).replace("-", "_");
		File dir = new File(outputDir);
		if (!dir.exists()) {
			dir.mkdir();
		}

		// ------------------------------------------------------------------
		//Acquisition of video information
		// ------------------------------------------------------------------
		double fps = capture.get(Videoio.CV_CAP_PROP_FPS);
		int resizeHeight = (int) (config.getImageWidth() / capture.get(Videoio.CV_CAP_PROP_FRAME_WIDTH)
				* capture.get(Videoio.CV_CAP_PROP_FRAME_HEIGHT));
		double allFrameCnt = capture.get(Videoio.CV_CAP_PROP_FRAME_COUNT);
		int captureInterval = (int) (allFrameCnt / (config.getCaptureCount() + 1));
		long time = (long) (allFrameCnt / fps);

		// ------------------------------------------------------------------
		//Get the number of frames specified in the settings
		// ------------------------------------------------------------------
		for (int i = 1; i < config.getCaptureCount() + 1; i++) {
			int frameIndex = captureInterval * i;
			Mat orgFrame = new Mat();
			Mat resizedFrame = new Mat();
			capture.set(Videoio.CV_CAP_PROP_POS_FRAMES, frameIndex - 1);
			if (!capture.read(orgFrame)) {
				continue;
			}
			Imgproc.resize(orgFrame, resizedFrame, new Size(config.getImageWidth(), resizeHeight));
			Imgcodecs.imwrite(outputDir + "/" + i + ".jpg ", resizedFrame);

		}

		VideoMetadata metadata = new VideoMetadata();
		metadata.setFilename(videoFile);
		metadata.setFrameDirname(outputDir);
		metadata.setPlayTime(time);
		return metadata;
	}

Compare images

In the previous section, we made 9 images per video. This time we will compare this.

For image comparison, I decided to compare histograms. I think there is much room for consideration in this regard.

The average of the comparison results of each of the nine images was used as the comparison result of the entire video.

The code is below.

	static public VideoComparison compareImages(Config config, VideoMetadata video1, VideoMetadata video2) {

		List<Double> histList = new ArrayList<Double>();
		// ------------------------------------------------------------------
		//Compare by image
		// ------------------------------------------------------------------
		for (int i = 1; i < config.getCaptureCount() + 1; i++) {
			String filename1 = video1.getFrameDirname() + "/" + i + ".jpg ";
			String filename2 = video2.getFrameDirname() + "/" + i + ".jpg ";
			File file1 = new File(filename1);
			File file2 = new File(filename2);
			if (!(file1.exists() && file2.exists())) {
				continue;
			}

			// ------------------------------------------------------------------
			//histogram
			// ------------------------------------------------------------------
			Mat img1 = Imgcodecs.imread(filename1);
			List<Mat> src1 = new ArrayList<Mat>();
			src1.add(img1);
			Mat hist1 = new Mat();
			Imgproc.calcHist(src1, new MatOfInt(0), new Mat(), hist1, new MatOfInt(256), new MatOfFloat(0, 256));

			Mat img2 = Imgcodecs.imread(filename2);
			List<Mat> src2 = new ArrayList<Mat>();
			src2.add(img2);
			Mat hist2 = new Mat();
			Imgproc.calcHist(src2, new MatOfInt(0), new Mat(), hist2, new MatOfInt(256), new MatOfFloat(0, 256));

			histList.add(Imgproc.compareHist(hist1, hist2, 0));

		}

		VideoComparison videoComparison = new VideoComparison();
		videoComparison.setVideo1(video1);
		videoComparison.setVideo2(video2);
		videoComparison.setHist(calcAvg(histList));
		return videoComparison;
	}

Try to make it GUI

I want to make it a GUI because it handles images. I was worried, but I decided to make it with JavaFX to commemorate the release of Java9. (JavaFX is 8 ...)

In JavaFX, the design layout can be described in XML (fxml), and Scene Builder can edit this graphically. The XML and code are linked by the fx: id attribute in XML and the @FMXL annotation in .java.

For example, if the control of the execute button is arranged as follows,

<Button fx:id="executeBtn" layoutX="214.0" layoutY="31.0" mnemonicParsing="false" onAction="#doExecuteAction" prefHeight="38.0" prefWidth="115.0" text="execute" />

On the Java code side, do the following:

	@FXML
	void doExecuteAction(ActionEvent event) {
        // ...
    }

Now you can attach the process that will be executed when you press the button.

The controls and actions used this time are as follows.

Button

--Execute

add to
Delete
ListView
TableView
Choice
ImgView

Finally

I wrote the code in Java for the first time in a long time, but there are many annoying aspects in personal use. Actually, I wrote it in Python at first, but I wanted to make it a GUI, what should I do with threads and so on ... → OpenCV can also be done in Java. So I chose Java.

I haven't had a GUI in Java since AWT, but JavaFX was better than I expected.

The rest is around the following.

--It takes time if there are many files

--Although it is processed by parallelStream, there seems to be room for tuning in terms of processing time.

--Visualization of processing status

--Implementation of progress bar

--Display processing status log

--Improved image comparison accuracy

--Comparison by features.

--Other ...?

Holiday programming was fun. end.