[JAVA] [Experiment] To what extent can static code analysis tools detect bugs and vulnerabilities? ~ SonarQube

** How well can static code analysis tools detect bugs and vulnerabilities? ** **

Last time, I tried to analyze Web application full of bugs with FindBugs However, this time I analyzed it with ** SonarQube **.

Verification content

A web application full of bugs (EasyBuggy 1.3.3) currently has the following 80 bugs and vulnerabilities. The type.

** Failure **

--Deadlock (Java) --Deadlock (SQL) --Waiting for a process that does not complete

infinite loop --Memory leak (Java heap area) --Memory leak (Permanent area) --Memory leak (C heap area) --Network socket leak --Database connection leak --File descriptor leak --Thread leak --Mojibake --Integer overflow --Rounding error --Censoring error --Information loss

Vulnerability

--XSS (Cross-site scripting) --SQL injection --LDAP injection --Code injection --OS command injection --Email header injection --Null byte injection --File upload with no size limit --File upload with no extension restrictions --Open redirectable login screen --Brute force attackable login screen --Login screen that allows session fixation attacks --Too kind authentication error message --Dangerous file include --Pastraversal --Unintentional file disclosure --CSRF (Cross Site Request Forgery) --Clickjacking --XEE (XML Entity Extension) --XXE (XML External Entity) --Delay due to regular expression analysis --Delayed string concatenation by plus operator --Delay due to unnecessary object creation

error

AssertionError
ExceptionInInitializerError
FactoryConfigurationError
GenericSignatureFormatError
NoClassDefFoundError
OutOfMemoryError (Java heap space)
OutOfMemoryError (Requested array size exceeds VM limit)
OutOfMemoryError (unable to create new native thread)
OutOfMemoryError (GC overhead limit exceeded)
OutOfMemoryError (PermGen space)
OutOfMemoryError (Direct buffer memory)
StackOverflowError
TransformerFactoryConfigurationError
UnsatisfiedLinkError

exception

ArithmeticException
ArrayIndexOutOfBoundsException
ArrayStoreException
BufferOverflowException
BufferUnderflowException
CannotRedoException
CannotUndoException
ClassCastException
ConcurrentModificationException
EmptyStackException
IllegalArgumentException
IllegalMonitorStateException
IllegalPathStateException
IllegalStateException
IllegalThreadStateException
ImagingOpException
IndexOutOfBoundsException
InputMismatchException
MalformedParameterizedTypeException
MissingResourceException
NegativeArraySizeException
NoSuchElementException
NullPointerException
NumberFormatException
SecurityException
UnsupportedCharsetException
UnsupportedOperationException

How many problems can SonarQube 6.3 detect?

result

The result is as follows (the same source code is verified with FindBugs as well as the verification result with SonarQube).

bug	SonarQube	FindBugs(Maximum analytical power)
Deadlock(Java)	×	×
Deadlock(SQL)	×	×
Waiting for a process that does not complete	×	×
infinite loop	○	×
Memory leak(Java heap area)	×	×
Memory leak(Permanent area)	×	×
Memory leak(C heap area)	×	×
Network socket leak	×	×
Database connection leak	○	○
File descriptor leak	○	○
Thread leak	×	×
Garbled characters	×	×
Integer overflow	×	×
Rounding error	×	×
Censoring error	×	×
Information loss	×	×
XSS (Cross-site scripting)	×	×
SQL injection	○	○
LDAP injection	×	×
Code injection	×	×
OS command injection	×	×
Email header injection	×	×
Null byte injection	×	×
File upload with no size limit	×	×
File upload with no extension restrictions	×	×
Open redirectable login screen	×	×
Brute force attackable login screen	×	×
Login screen that allows session fixation attacks	×	×
Too kind authentication error message	×	×
Dangerous file include	×	×
Past traversal	×	×
Unintended file disclosure	×	×
CSRF (Cross-site request forgery)	×	×
Clickjacking	×	×
XEE (XML entity extension)	×	×
XXE (XML external entity)	×	×
Delay due to regular expression parsing	×	×
Delayed string concatenation with plus operator	○	○
Delay due to unnecessary object creation	×	×
AssertionError	×	×
ExceptionInInitializerError / NoClassDefFoundError	×	×
FactoryConfigurationError	×	×
GenericSignatureFormatError	×	×
NoClassDefFoundError	×	×
OutOfMemoryError (Java heap space)	×	×
OutOfMemoryError (Requested array size exceeds VM limit)	×	×
OutOfMemoryError (unable to create new native thread)	×	×
OutOfMemoryError (GC overhead limit exceeded)	×	×
OutOfMemoryError (PermGen space)	×	×
OutOfMemoryError (Direct buffer memory)	×	×
StackOverflowError	×	×
TransformerFactoryConfigurationError	×	×
UnsatisfiedLinkError	×	×
ArithmeticException	○	×
ArrayIndexOutOfBoundsException	×	×
ArrayStoreException	×	×
BufferOverflowException	×	×
BufferUnderflowException	×	×
CannotRedoException	×	×
CannotUndoException	×	×
ClassCastException	×	×
ConcurrentModificationException	×	×
EmptyStackException	×	×
IllegalArgumentException	×	×
IllegalMonitorStateException	×	×
IllegalPathStateException	×	×
IllegalStateException	×	×
IllegalThreadStateException	×	×
ImagingOpException	×	×
IndexOutOfBoundsException	×	×
InputMismatchException	×	×
MalformedParameterizedTypeException	×	×
MissingResourceException	×	×
NegativeArraySizeException	×	×
NoSuchElementException	×	×
NullPointerException	×	○
NumberFormatException	×	×
SecurityException	×	×
UnsupportedCharsetException	×	×
UnsupportedOperationException	×	×

As a result, only 6 types were detected for 80 types of bugs. As you can see, there are some differences, but the detection ability is about the same as FindBugs. However, it is certainly a very useful tool because you can grasp simple mistakes and bad coding patterns by using a static analysis tool. By the way, the number of indications was 176, while the number of lines of code was about 4,700.

- Note: Even if the bug was detected in this verification, it may not be detected if the implementation method is different. On the contrary, even if the bug could not be detected by this verification, it may be detected if the implementation method is different. Even logic that causes a NullPointerException with a 100% probability may or may not detect a NullPointerException depending on how it is implemented. *

Other things I noticed

One thing I was a little worried about when using SonarQube was the large number of false positives.

For example, the following code points out that there is an unused method, but this method is actually used, and removing it will result in a compilation error.

The following code points out SQL injection, but it's okay because no value other than "ʻasc" or " desc` "is added to the SQL statement.

The following code points out the possibility of NPE, but since I'm checking null in ʻorg.apache.commons.lang.StringUtils.isBlank (String) , the variable fileName` becomes null and NPE It can't be ("might be thrown", so I wonder if it's okay ...).

Remarks

How to use SonarQube is easy, but [blog](https://t246osslab.wordpress.com/2016/11/20/sonnarqube%e3%81%a7%e3%82%bd%e3%83% bc% e3% 82% b9% e3% 82% b3% e3% 83% bc% e3% 83% 89% e3% 81% ae% e5% 93% 81% e8% b3% aa% e3% 82% 92% Please refer to e8% a7% a3% e6% 9e% 90% e3% 81% 99% e3% 82% 8b /).