I will write how to convert Cycle GAN to Core ML.
It is basically the same as the next article, but as of November 1, 2020, model conversion can be done with Core ML Tools instead of tf-coreml, so I will write the difference of that part.
Convert Cycle GAN to Mobile Model (CoreML) https://qiita.com/john-rocky/items/3fbdb0892b639187c234
In conclusion, you can convert each library version without any problem by doing the following.
tensorflow==2.2.0
keras==2.2.4
coremltools==4.0
I will write the procedure in Google Colaboratory.
** 1. Open the TensorFlow tutorial **
Open the tutorial from this page https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/tutorials/generative/cyclegan.ipynb
** 2. Match each library version to Core ML Tools 4.0 **
Uninstall tensorflow and keras that originally came with Colab, then reinstall the appropriate version. Also, install Core ML Tools 4.0. You may need to restart the Google Colab runtime.
Notebook
!pip uninstall -y tensorflow
!pip uninstall -y keras
!pip install tensorflow==2.2.0
!pip install keras==2.2.4
!pip install -U coremltools==4.0
** 3. Perform learning according to the tutorial **
You can learn Cycle GAN just by executing the tutorial from top to bottom, so continue as it is.
** 4. Save the model and convert it to Core ML **
Save the TensorFlow model.
Notebook
generator_g.save('./savedmodel')
Convert to Core ML. The arguments are much simpler than tf-coreml.
Notebook
import coremltools as ct
input_name = generator_g.inputs[0].name.split(':')[0]
print(input_name) #Check input_name.
keras_output_node_name = generator_g.outputs[0].name.split(':')[0]
graph_output_node_name = keras_output_node_name.split('/')[-1]
mlmodel = ct.converters.convert('./savedmodel/',
inputs=[ct.ImageType(bias=[-1,-1,-1], scale=2/255,shape=(1, 256, 256, 3,))],
output_names=[graph_output_node_name],
)
Save the converted Core ML model.
Notebook
mlmodel.save('./cyclegan.mlmodel')
** 5. Load and display Core ML model in Xcode project **
Create a project in Xcode.
Create an app that converts camera images with Cycle GAN.
Create a simple screen with only one UIImageView and UIButton on the Main.storyboard and connect it to the view controller. UIImage connects as a reference and UIButton connects as an action.
This time, let Vision Framework infer, but the inference result is returned as Multi Array type. I need to convert this to a UIImage, but it's easy with Core ML Helper. For that, copy the source code from CoreMLHelper.
In particular
https://github.com/hollance/CoreMLHelpers/tree/master/CoreMLHelpers
I have copied all the source code below to my project.
Here is a reference for how to use Core ML Helper.
Conversion from MultiArray to Image CoreMLHelper https://qiita.com/john-rocky/items/bfa631aba3e7aabc2a66
I will write the process in the view controller. I'm capturing a camera image, passing it to the Vision Framework, converting it to a UIImage, and pasting it into a UIImageView.
This time, I cut out the process of passing the camera image to the Vision Framework and extracting the inference result UIImage into a class called CycleGanConverter
.
ViewController.swift
import UIKit
import Vision
class ViewController: UIViewController, UIImagePickerControllerDelegate & UINavigationControllerDelegate {
@IBOutlet weak var imageView: UIImageView!
//Cycle GAN converter
let converter = CycleGanConverter()
override func viewDidLoad() {
super.viewDidLoad()
}
@IBAction func tappedCamera(_ sender: Any) {
//Start the camera when the camera button is pressed
if UIImagePickerController.isSourceTypeAvailable(UIImagePickerController.SourceType.camera) {
let picker = UIImagePickerController()
picker.modalPresentationStyle = UIModalPresentationStyle.fullScreen
picker.delegate = self
picker.sourceType = UIImagePickerController.SourceType.camera
self.present(picker, animated: true, completion: nil)
}
}
func imagePickerController(_ picker: UIImagePickerController, didFinishPickingMediaWithInfo info: [UIImagePickerController.InfoKey : Any]) {
defer {
picker.dismiss(animated: true, completion: nil)
}
guard let image = info[UIImagePickerController.InfoKey(rawValue: UIImagePickerController.InfoKey.originalImage.rawValue)] as? UIImage else {return}
//The image taken by the camera is entered in the variable image
//Converting to image size to be passed to Cycle GAN(Use CoreML Helper Extension)
let resized = image.resized(to: CGSize(width: 256, height: 256))
//Converted because it needs to be passed as cgImage type
guard let cgImage = resized.cgImage else {return}
//Pass to converter
converter.convert(image: cgImage) {
images in
//The converter returns the result in an images array if the conversion is successful
guard images.count > 0 else {return}
let image = images.first!
DispatchQueue.main.async {
//Paste into UIImageView
self.imageView.image = image
}
}
}
}
class CycleGanConverter {
func convert(image: CGImage, completion: @escaping (_ images: [UIImage]) -> Void) {
//Set up Vision Framework
let handler = VNImageRequestHandler(cgImage: image,options: [:])
let model = try! VNCoreMLModel(for: cyclegan().model)
let request = VNCoreMLRequest(model: model, completionHandler: {
[weak self](request: VNRequest, error: Error?) in
//Since the inference result is returned to the variable request, convert it with the perseImage method.
guard let images = self?.parseImage(request: request) else {return}
completion(images)
})
try? handler.perform([request])
}
func parseImage(request: VNRequest) -> [UIImage] {
guard let results = request.results as? [VNCoreMLFeatureValueObservation] else {return []}
//featureValue of each element in the results array.Since the converted image is included in multiArrayValue, convert this to UIImage
return results.compactMap {
$0.featureValue.multiArrayValue?
.image(min: -1, max: 1, channel: nil, axes: (3,2,1))? //A method of Core ML Helper that converts to UIImage
.rotated(by: 90) //Rotate for screen display
.withHorizontallyFlippedOrientation() //Invert for screen display
}
}
}
This time, due to time constraints, the number of trainings has been significantly reduced compared to the tutorial specification, so the conversion results are reasonable.
If you train according to the tutorial, you should get better conversion results.
Note regularly publishes about iOS development, so please follow us. https://note.com/tokyoyoshida
It is also posted on Twitter. https://twitter.com/jugemjugemjugem
Recommended Posts