[JAVA] [Experiment] To what extent can static code analysis tools detect bugs and vulnerabilities? ～ FindBugs

** How well can static code analysis tools detect bugs and vulnerabilities? ** **

I tried to verify it by analyzing the Web application full of bugs that I made earlier with ** FindBugs **.

Verification content

A web application full of bugs (EasyBuggy 1.2.17) has the following 68 bugs and vulnerabilities currently implemented. The type.

** Failure **

--Memory leak (Java heap area) --Memory leak (Permanent area) --Memory leak (C heap area) --Deadlock (Java) --Deadlock (SQL) --Waiting for a process that does not complete

• infinite loop --Redirect loop --Forward loop --JVM crash --Network socket leak --Database connection leak --File descriptor leak --Thread leak --Mojibake --Integer overflow --Rounding error --Censoring error --Information loss

Vulnerability

--XSS (Cross-site scripting) --SQL injection --LDAP injection --Code injection --OS command injection --File upload with no size limit --File upload with no extension restrictions --Open redirectable login screen --Brute force attackable login screen --Too kind login error message --Dangerous file include --Unintentional file disclosure --CSRF (Cross Site Request Forgery)

** Performance issues **

--Delay due to regular expression analysis

error

• FactoryConfigurationError
• ExceptionInInitializerError
• NoClassDefFoundError
• OutOfMemoryError (Java heap space)
• OutOfMemoryError (Requested array size exceeds VM limit)
• OutOfMemoryError (unable to create new native thread)
• OutOfMemoryError (GC overhead limit exceeded)
• OutOfMemoryError (PermGen space)
• OutOfMemoryError (Direct buffer memory)
• StackOverflowError
• UnsatisfiedLinkError

** Unchecked exception **

• ArithmeticException
• ArrayIndexOutOfBoundsException
• ArrayStoreException
• BufferOverflowException
• BufferUnderflowException
• CannotRedoException
• CannotUndoException
• ClassCastException
• ConcurrentModificationException
• EmptyStackException
• IllegalArgumentException
• IllegalMonitorStateException
• IllegalPathStateException
• IllegalStateException
• IllegalThreadStateException
• ImagingOpException
• IndexOutOfBoundsException
• InputMismatchException
• MissingResourceException
• NegativeArraySizeException
• NoSuchElementException
• NullPointerException
• NumberFormatException
• UnsupportedOperationException

How many problems can FindBugs 3.0.1 detect? Try running FindBugs with two settings, default and maximum resolution.

result

The result is as follows. The "Result (D)" in the second column is the analysis result with the default settings, and the "Result (Large)" is the analysis result with the maximum analysis power setting.

As a result, for 68 types of bugs, only 2 types were detected by default, and only 5 types were detected even with the maximum analysis power setting. I had the impression that static analysis tools such as FindBugs gave quite a bit of detail, but they didn't seem to detect bugs more than I expected.

However, the total number of indications, including other indications, was 6 with the default setting and 144 with the maximum resolution setting (for about 3,300 lines of code).

Also, although we could not detect the open redirect vulnerability, we pointed out the HTTP response splitting vulnerability in the following line.

response.sendRedirect("/openredirect/login" + loginQueryString);

HTTP response splitting is a vulnerability that allows you to embed another response header with a line break if you specify a string that includes a line break in the argument of response.sendRedirect (), but most Servlet containers (Jetty and Tomcat) In newer versions of (including), line breaks in received header values are escaped to protect against this type of attack, so I think this point is often not the case.

• • Note: Even if the bug was detected in this verification, it may not be detected if the implementation method is different. On the contrary, even if the bug could not be detected by this verification, it may be detected if the implementation method is different. Even logic that causes a NullPointerException with a 100% probability may or may not detect a NullPointerException depending on how it is implemented. *

Summary

This time around, I got this result, but FindBugs pointed out a lot of code issues outside of the deliberately crafted bugs. FindBugs is certainly a very useful tool as it can point out simple coding mistakes and suggest better implementations.

However, it is not necessary to capture all the indications as in the case of HTTP response splitting mentioned above. On the other hand, it may add unnecessary code or reduce readability.

Also, from this result, it can be said that human code review is very important. It is useful to use static analysis tools for mechanical checks before code review, but I think that only human (excellent programmer) code reviews will ultimately prevent bugs from being created.

Summary,

--Many bugs may be latent even if FindBugs does not point out ――However, FindBugs is very useful because it can detect many problems. --You don't have to capture all the issues that FindBugs detects --Human code review is important