A story about using the CoreImage framework to erase stains with Swift and implement a blur erase function

I was in charge of enhancing the image correction function to the Baitoru resume app, so I will summarize what I did.

The content of this enhancement is to perform a filter processing that is uniquely defined for the entire photo taken as an image correction function, and to partially perform image correction processing on the dark circles under the eyes and the spots around the cheeks. I implemented the functionality using only the Core Image framework.

It's not that difficult to filter the entire photo, so I'll just include the sample code.

import CoreImage
...

//Prepare the Sample image you want to filter
var image = UIImage(named: "sample.jpg ")

//Conversion from UIImage to CIImage
let ciImage: CIImage? = CIImage(image: image)

//Sepia filter preparation
var filter = CIFilter(name: "CISepiaTone")

//Pass the image to the filter
filter.setValue(ciImage, forKey: kCIInputImageKey)

//Get a filtered image
var filteredImage: CIImage? = filter.outputImage

Applying the sepia correction filter, the result looks like this. Actually, I made a unique filter by multiplying various image correction filters on the product and implemented it.

original Sepia correction
filter-sample-original.png filter-sample-sepia.png

Flow of partial image correction processing

Next, I will summarize the processing flow when partially correcting the image.

0aac97d9-f553-44cf-97ea-c622e28b864d-1920x3172r.png

Both the stain erasing function and the darkening function are the same in that the image is partially corrected, so both are the above processing flow.

Creating a mask image

To create a mask image, start by using Core Image's CI Detection to get the positional relationship of the facial parts. It is information that can be acquired in face detection, but it is possible to acquire the position of the mouth, the position of the eyes, whether the eyes are open, and the outline of the face, and based on that information, we will create a mask image.

Face detection is also supported by Core Image and can be obtained from the code below.

let options = [CIDetectorAccuracy: CIDetectorAccuracyHigh]
var faces: [CIFeature] = []
if let image = CIImage(image: faceImage), let faceDetector = CIDetector(ofType: CIDetectorTypeFace, context: nil, options: options) {
     faces = faceDetector.features(in: image)
}

To create a mask image, the position information of the detected facial parts is used, the position of the stain modeled numerically is calculated, and the CI Filter called CIRadialGradient is used. This Filter is a filter that can generate Gaussian by inputting position information, and for the mask for erasing stains, a total of 6 Gaussian waveforms are superimposed to create a mask image. A CI Filter called CISourceOverCompositing is used to synthesize Gaussian waveforms.

let maskCircles: [MaskCircle] = [
   ///Gaussian location information, etc....
]
for (index, circle) in maskCircles.enumerated() {
   guard let gaussian = CIFilter(name: "CIRadialGradient") else { return nil }
   gaussian.setValue(CIVector(cgPoint: circle.center), forKey: kCIInputCenterKey)
   gaussian.setValue(circle.innerRadius, forKey: "inputRadius0")
   gaussian.setValue(circle.outerRadius, forKey: "inputRadius1")
   gaussian.setValue(CIColor(red: 1.0, green: 1.0, blue: 1.0, alpha: 1.0), forKey: "inputColor0")
   gaussian.setValue(CIColor(red: 1.0, green: 1.0, blue: 1.0, alpha: 0.0), forKey: "inputColor1")

   if index == 0 {
      prev = gaussian
   } else {
      shimi = CIFilter(name: "CISourceOverCompositing")
      if let prevMask = prev?.outputImage, let gaussianMask = gaussian.outputImage {
         shimi?.setValue(gaussianMask, forKey: kCIInputImageKey)
         shimi?.setValue(prevMask, forKey: kCIInputBackgroundImageKey)
      }
      prev = shimi
   }
}

The image below shows the original image and the mask image superimposed on the original image. A mask image is created by superimposing Gaussian waveforms aiming at the spotted part of the face.

Original image Mask image for stain removal
Screenshot 2020-12-22 18.09.20.png Screenshot2020-12-2218.09.51.png

Image composition

Next, we will implement the stain removal function by synthesizing the image based on the created mask image for stain removal. Use the CIBlendWithMask filter for partial compositing with masked images.

In the CIBlendWithMask filter, the mask image, the original image, and the image with the image correction that you want to apply only to the mask part are prepared and the image is combined.

let blend = CIFilter(name: "CIBlendWithMask") else { return nil }

///Image settings with blur effect
blend.setValue(blurImage, forKey: kCIInputImageKey)
///Original image settings
blend.setValue(originalImage, forKey: kCIInputBackgroundImageKey)
///Mask image settings
blend.setValue(mask, forKey: kCIInputMaskImageKey)

///Acquisition of composited image
blend.outputImage
Original image Mask image for stain removal Blur image
Screenshot 2020-12-22 18.09.20.png Screenshot2020-12-2218.09.51.png Screenshot2020-12-2218.10.00.png

result

The result of partially correcting the image and implementing the stain erasing function is like this! You can see that the blur effect is partially applied only to the spots and the spots are less noticeable.

Original image Image after stain removal processing
Screenshot 2020-12-22 18.32.27.png Screenshot2020-12-2218.32.41.png

Recommended Posts

A story about using the CoreImage framework to erase stains with Swift and implement a blur erase function
[Swift] How to implement the Twitter login function using Firebase UI ①
[Swift] How to implement the Twitter login function using Firebase UI ②
[Swift] A note about function and closure
[Swift] How to implement the countdown function
[Swift] How to implement the LINE login function
[swift5] How to implement the Twitter share function
How to implement the breadcrumb function using gretel
[Swift] How to implement the fade-in / out function
A story about introducing Evolutions into the Play Framework
[Swift] About the inability to distinguish between full-width and half-width characters with NS Predicate
Let's implement a function to limit the number of access to the API with SpringBoot + Redis
Try to implement tagging function using rails and js
The story of pushing a Docker container to GitHub Package Registry and Docker Hub with GitHub Actions
[Swift] I tried to implement the function of the vending machine
Add a shadow to the Swift Button (and also the circle)
[Rails] I tried to implement "Like function" using rails and js
I tried to implement the image preview function with Rails / jQuery
A story about hitting the League Of Legends API with JAVA
A story about having a hard time aligning a testing framework with Java 6
How to operate IGV using socket communication, and the story of making a Ruby Gem using that method
I want to download a file on the Internet using Ruby and save it locally (with caution)
Rails API mode I tried to implement the keyword multiple search function using arrays and iterative processing.
The story of forgetting to close a file in Java and failing
The story of making a game launcher with automatic loading function [Java]
A story I was addicted to when testing the API using MockMVC
[Rails] Implement the product purchase function with a credit card registered with PAY.JP
A story about sending a pull request to MinGW to update the libgr version
Try to implement login function with Spring-Boot
[Swift] Easy to implement modal with PanModal
Authentication function with Play Framework [Registration and authentication]
How to implement TextInputLayout with validation function
[Swift5] How to implement animation using "lottie-ios"
I tried to implement a function equivalent to Felica Lite with HCE-F of Android
How to get the ID of a user authenticated with Firebase in Swift
A story about saving an image with carrierwave in a nested form using a form object.
Send a notification to slack with the free version of sentry (using lambda)
Be careful with requests and responses when using the Serverless Framework in Java