A story about using the CoreImage framework to erase stains with Swift and implement a blur erase function

I was in charge of enhancing the image correction function to the Baitoru resume app, so I will summarize what I did.

The content of this enhancement is to perform a filter processing that is uniquely defined for the entire photo taken as an image correction function, and to partially perform image correction processing on the dark circles under the eyes and the spots around the cheeks. I implemented the functionality using only the Core Image framework.

It's not that difficult to filter the entire photo, so I'll just include the sample code.

import CoreImage
...

//Prepare the Sample image you want to filter
var image = UIImage(named: "sample.jpg ")

//Conversion from UIImage to CIImage
let ciImage: CIImage? = CIImage(image: image)

//Sepia filter preparation
var filter = CIFilter(name: "CISepiaTone")

//Pass the image to the filter
filter.setValue(ciImage, forKey: kCIInputImageKey)

//Get a filtered image
var filteredImage: CIImage? = filter.outputImage

Applying the sepia correction filter, the result looks like this. Actually, I made a unique filter by multiplying various image correction filters on the product and implemented it.

original	Sepia correction

• Free materials are used for the photos: https://www.pakutaso.com/model.html

Flow of partial image correction processing

Next, I will summarize the processing flow when partially correcting the image.

Both the stain erasing function and the darkening function are the same in that the image is partially corrected, so both are the above processing flow.

Creating a mask image

To create a mask image, start by using Core Image's CI Detection to get the positional relationship of the facial parts. It is information that can be acquired in face detection, but it is possible to acquire the position of the mouth, the position of the eyes, whether the eyes are open, and the outline of the face, and based on that information, we will create a mask image.

Face detection is also supported by Core Image and can be obtained from the code below.

let options = [CIDetectorAccuracy: CIDetectorAccuracyHigh]
var faces: [CIFeature] = []
if let image = CIImage(image: faceImage), let faceDetector = CIDetector(ofType: CIDetectorTypeFace, context: nil, options: options) {
     faces = faceDetector.features(in: image)
}

To create a mask image, the position information of the detected facial parts is used, the position of the stain modeled numerically is calculated, and the CI Filter called CIRadialGradient is used. This Filter is a filter that can generate Gaussian by inputting position information, and for the mask for erasing stains, a total of 6 Gaussian waveforms are superimposed to create a mask image. A CI Filter called CISourceOverCompositing is used to synthesize Gaussian waveforms.

let maskCircles: [MaskCircle] = [
   ///Gaussian location information, etc....
]
for (index, circle) in maskCircles.enumerated() {
   guard let gaussian = CIFilter(name: "CIRadialGradient") else { return nil }
   gaussian.setValue(CIVector(cgPoint: circle.center), forKey: kCIInputCenterKey)
   gaussian.setValue(circle.innerRadius, forKey: "inputRadius0")
   gaussian.setValue(circle.outerRadius, forKey: "inputRadius1")
   gaussian.setValue(CIColor(red: 1.0, green: 1.0, blue: 1.0, alpha: 1.0), forKey: "inputColor0")
   gaussian.setValue(CIColor(red: 1.0, green: 1.0, blue: 1.0, alpha: 0.0), forKey: "inputColor1")

   if index == 0 {
      prev = gaussian
   } else {
      shimi = CIFilter(name: "CISourceOverCompositing")
      if let prevMask = prev?.outputImage, let gaussianMask = gaussian.outputImage {
         shimi?.setValue(gaussianMask, forKey: kCIInputImageKey)
         shimi?.setValue(prevMask, forKey: kCIInputBackgroundImageKey)
      }
      prev = shimi
   }
}

The image below shows the original image and the mask image superimposed on the original image. A mask image is created by superimposing Gaussian waveforms aiming at the spotted part of the face.

Original image	Mask image for stain removal

Image composition

Next, we will implement the stain removal function by synthesizing the image based on the created mask image for stain removal. Use the CIBlendWithMask filter for partial compositing with masked images.

In the CIBlendWithMask filter, the mask image, the original image, and the image with the image correction that you want to apply only to the mask part are prepared and the image is combined.

let blend = CIFilter(name: "CIBlendWithMask") else { return nil }

///Image settings with blur effect
blend.setValue(blurImage, forKey: kCIInputImageKey)
///Original image settings
blend.setValue(originalImage, forKey: kCIInputBackgroundImageKey)
///Mask image settings
blend.setValue(mask, forKey: kCIInputMaskImageKey)

///Acquisition of composited image
blend.outputImage

Original image	Mask image for stain removal	Blur image

result

The result of partially correcting the image and implementing the stain erasing function is like this! You can see that the blur effect is partially applied only to the spots and the spots are less noticeable.

Original image	Image after stain removal processing