[SWIFT] How to edit layers in Core ML Tools
When converting a model using Core ML Tools, you may want to add layers and insert calculations in between, or cut a part of the layer, but this article will explain how to do it.
Since Core ML Tools has few documents and little information on the net, I have to look it up while reading the source code, but I think that editing layers is a common use case for conversion to Core ML. I hope this article helps you.
Model case: Put reshape before output to change the shape of output
As a model case, let's add a reshape to make the model output shape (19) (1,1,19).
It doesn't make much sense because it's just adding dimensions, but I'm doing this because I want a simple case.
The model to be used is one that predicts (classifies) the result of adding two single-digit numbers. The classification result will be 19 arrays. This is because adding one-digit numbers gives a total of 19 types of classification problems from 0 to 18.
It looks like this in the figure.
This model is the same as the one used in the previous article. In the previous article, even the label name was given to the output, but this time we will simply output 19 probabilities as they are.
Run a simple model made with Keras on iOS using CoreML
procedure
Now let's add the Reshape layer. Work on Google Colaboratory.
The complete code for this work can be found here. https://gist.github.com/TokyoYoshida/2fada34313385d63b666253490b5f3f4
** 1. Load Keras model **
I will use a model made with Keras. The model creation part is omitted because it is in Previous article.
notebook
from keras.models import load_model
keras_model = load_model('my_model.h5')
** 2. Convert to CoreML **
Convert to CoreML. Since I will not put a label this time, I am simply converting it as it is.
notebook
from coremltools.converters import keras as converter
mlmodel = converter.convert(keras_model)
** 3. Load with builder **
Load it into Neural Network Builder in Core ML Tools.
notebook
import coremltools
spec = coremltools.utils.load_spec(coreml_model_path)
builder = coremltools.models.neural_network.NeuralNetworkBuilder(spec=spec)
Let's display the model information. From this you can see that the output of the model is named output1 and the shape is 19.
notebook
spec.description.output
#Output result
# [name: "output1"
# type {
# multiArrayType {
# shape: 19
# dataType: DOUBLE
# }
# }
# ]
Displays layer information.
notebook
builder.layers
#output
# ['dense_4',
# 'dense_4__activation__',
# 'dense_5',
# 'dense_5__activation__',
# 'dense_6',
# 'activation_18']
The target this time is activation_18, which is the output layer. Reshape the output of this layer.
** 4. Add Rehape layer **
NeuralNetworkBuilder has a method called add_reshape that allows you to add a reshape layer.
~~ It's important to note here that using the builder, I feel that I can intuitively add a layer to the builder that represents the original model, but this method does not work. ~~ (Added on October 27, 2020, I found that I can add a layer to the builder that represents the original model, so I will correct it)
For example, let's add the Reshape layer as it is to the model loaded in the builder earlier.
notebook
reshape = builder.add_reshape(name='Reshape', input_name='activation_18', output_name='output', target_shape=(1,1,19), mode=0)
When I read this with Xcode, I get this error.
Xcode error
There was a problem decoding this CoreML document
validator error: Layer 'Reshape' consumes an input named 'activation_18' which is not present in this network.
When I try to perform CoreML inference on Python, I get this error:
Python error
RuntimeError: Error compiling model: "Error reading protobuf spec. validator error: Layer 'Reshape' consumes an input named 'activation_18' which is not present in this network.".
I had a hard time finding it on the internet, but I'm sure there are people who have a hard time doing the same thing.
The reason for this error is that ʻinput_name ='activation_18'. Here you need to specify the name of the input or output displayed in builder.spec.description` in the original model.
~~ As a workaround, use NeuralNetworkBuilder to create a complete model with only one layer and add it to the original model. ~~ (Corrected on October 27, 2020) If the input_name argument of add_reshape matches the output_name of the original model, you can successfully connect the layers.
Add a reshape layer to your model.
input specifies the output of the original model, ʻoutput1. I felt like this was the input of the new model, ʻinput2, but it seems that the output of the original model and the input to the reshape layer do not work well.
output specifies ʻoutput2`, which is the output of the new model.
target_shape specifies (1,1,19) which is the shape you want to convert this time.
notebook
builder.add_reshape(name='Reshape', input_name='output1', output_name='output2', target_shape=(1,1,19), mode=0)
You now have a model with reshape added.
Check the layer information.
notebook
builder.layers
#output
# ['dense_4',
# 'dense_4__activation__',
# 'dense_5',
# 'dense_5__activation__',
# 'dense_6',
# 'activation_18',
# 'Reshape']
** 5. Change the output information of the model **
There was a reshape layer at the end of the model, but that only replaced the last layer of the model, and the output information of the model as a whole did not change.
If you check the output information as a trial, you can see that it is still (19) instead of the expected shape (1,1,19).
notebook
spec.description.output
# [name: "output1"
# type {
# multiArrayType {
# shape: 19
# dataType: DOUBLE
# }
# }
# ]
Changing the output information of the model is also very quirky and will result in an error if you try to assign it to builder.spec.description.output [0]. Therefore, specify it while popping or adding.
notebook
//Delete one output information. Since there was only one output information this time, the output information disappears
builder.spec.description.output.pop()
//Add one output information
builder.spec.description.output.add()
//Specify the attributes of the output information
output = builder.spec.description.output[0]
output.name = "output2"
output.type.multiArrayType.dataType = coremltools.proto.FeatureTypes_pb2.ArrayFeatureType.ArrayDataType.Value('DOUBLE')
//As shape information(1,1,19)To set
output.type.multiArrayType.shape.append(1)
output.type.multiArrayType.shape.append(1)
output.type.multiArrayType.shape.append(19)
(Supplement) In the above example, the information was added after making it clean by deleting it with pop () and adding it with add (), but it is possible to directly rewrite only the attribute value of the output that originally exists.
For example, rewrite the shape as follows.
Example of rewriting shape
builder.spec.description.output[0].type.multiArrayType.shape[0] = 100
(End of supplement)
If you check the output information, you can see that the settings are successful.
notebook
builder.spec.description.output
# [name: "output2"
# type {
# multiArrayType {
# shape: 1
# shape: 1
# shape: 19
# dataType: DOUBLE
# }
# }
# ]
** 6. Check the output result of the model with Jupyter Notebook **
Check the output result of the model with Jupyter Notebook running on Mac. Why Jupyter Notebook on Mac? However, since the back end of Google Colaboratory is Linux, CoreML cannot be inferred and it is not possible to confirm whether the model is working properly.
You can infer with Xcode, but Xcode may give you an error when you read a strange model, but it may also drop the following error at build time.
Xcode error
Command CoreMLModelCompile failed with a nonzero exit code
If you get such an error, it's a good idea to run it in Jupyter Notebook to see the details of the error.
So, start Jupyter Notebook.
Here is the complete code. https://gist.github.com/TokyoYoshida/632b4c8070aa6c937539e4ae261a2740
Give [2,3] as input to the model and try to perform the inference.
JupyterNoteBook
coreml_model_path= "my_model_with_builder.mlmodel"
import coremltools
spec = coremltools.utils.load_spec(coreml_model_path)
builder = coremltools.models.neural_network.NeuralNetworkBuilder(spec=spec)
mlmodel = coremltools.models.MLModel(spec)
mlmodel.predict({'input1': np.array([2.0,3.0])})
# {'output2': array([[[2.56040733e-21, 7.25779588e-15, 1.99342376e-10, 1.11184195e-09,
# 5.92091055e-05, 9.99939799e-01, 9.72097268e-07, 1.13292452e-14,
# 4.43997455e-23, 5.00404492e-33, 0.00000000e+00, 0.00000000e+00,
# 0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,
# 0.00000000e+00, 0.00000000e+00, 0.00000000e+00]]])}
Since this model is a model that adds two elements, the probability of the fifth element is almost 1, so you can see that it can be inferred correctly.
Also, since the output array is [[[19 arrays with probabilities]]], it is confirmed that the shape is (1,1,19) as intended. I can do it.
** 7. Check the output result of the model with Xcode & app **
Finally, put the output result of the model on Xcode and check it with the application.
The complete code is here. https://github.com/TokyoYoshida/CoreMLSimpleTest
ViewController.swift
let model = my_model_with_builder()
let inputArray = try! MLMultiArray([2,3])
let inputToModel = my_model_with_builderInput(input1: inputArray)
if let prediction = try? model.prediction(input: inputToModel) {
print(prediction.output2)
try! print(prediction.output2.reshaped(to: [19]))
}
// # Double 1 x 1 x 19 array
// # Double 19 vector
// # [2.422891379885771e-21,7.01752566646674e-15,1.959301054732521e-10,1.089580203839091e-09,5.933549255132675e-05,0.9999396800994873,9.530076567898504e-07,1.087061586308846e-14,4.16250629238845e-23,4.617410135639087e-33,1.401298464324817e-45,1.401298464324817e-45,1.401298464324817e-45,1.401298464324817e-45,1.401298464324817e-45,1.401298464324817e-45,0,0,0]
//
You can see that this is also as intended. The probability numbers may be slightly different, but don't worry, this is due to the retraining of the model.
Extra edition How to delete a layer
I have not confirmed the operation, but I will also write how to delete the layer.
Delete the layer.
notebook
layers = builder.spec.neuralNetwork.layers
#Get the second from the back of the layer(activation_18)
item = layers[-2]
#Delete that element
layers.remove(item)
If you look at the layer information, you can see that it has been deleted.
notebook
for layer in layers:
print(layer.name)
# dense_4
# dense_4__activation__
# dense_5
# dense_5__activation__
# dense_6
# Reshape
Finally
Note regularly publishes about iOS development, so please follow us. https://note.com/tokyoyoshida
It is also posted on Twitter. https://twitter.com/jugemjugemjugem
Recommended Posts