(Test automation) Make image recognition ambiguous

background

We are building and operating a test automation system for in-house system developers. (SikuliX + Testablish) Write down the technical issues and things you want to do on a daily basis.

Sikuli is fun!

Task

Sikuli is an OSS RPA tool. When it comes to this, I am very worried about recognition failure due to slight misalignment of images due to environmental differences that are common in RPA tools. In fact, even if the screen resolution and tool settings are the same in Win10 & Edge, there are many cases where it fails immediately due to the presence or absence of a monitor connection and small version differences. The main enemy of test automation is that it is difficult to secure idempotency. On the other hand, in the case of a manual test (person), even if there are some differences, the ambiguity can be interpreted in the head and the work can proceed. (So ​​it's sly. However, it may deteriorate the quality.) Sikuli also wants some forgiveness.

Correspondence

Sikuli's image recognition (OpenCV) has an option to set ambiguity. So why not start by searching for accuracy and then gradually lower it to increase ambiguity? Also, the accuracy seems to be 0.7 by default, but I wonder if that is also strictly speaking. (Sometimes you mistakenly recognize another similar element on the same screen.) I think it is logically good to start from around 0.99.

Thought logic Start with a recognition rate of 0.99, and if not found, decrease the recognition rate by 0.1 to make a judgment. (0.4 is Min)

Purpose

According to RaiMan, "Sikuli is God's eye" realizes image recognition with god-like forgiveness and tolerable image misalignment.

Contents

Verify with the following implementation.

----------------------- import sys dir = os.getcwd() acc = 0.99 saveFileName = "filename" saveFilePath = os.path.join(dir, saveFileName) while acc > 0.4:  if exists(Pattern(saveFilePath).similar(acc)):   find(Pattern(saveFilePath).similar(acc)).highlight()   wait(1)   highlightAllOff()   find(Pattern(saveFilePath).similar(acc)).click()   break  else:   acc = acc - 0.1 wait(1) -----------------------

Result or what you noticed

It is now possible to recognize even a slight deviation !! (It can be identified even if the target image is reduced to 80%) I'm not good at judging characters (because they are drawn with lines and the ratio is small in terms of area? Characters are better with the OCR function) Image recognition is much faster than OCR

Conclusion

May be essential for image recognition operations with RPA

Recommended Posts

(Test automation) Make image recognition ambiguous
(Test automation) Nesting images used for image recognition
Image recognition
CNN 1 Image Recognition Basics
Test automation for work
Deep learning image recognition 1 theory
Application of CNN2 image recognition
Image recognition with Keras + OpenCV