[SWIFT] How to edit layers in Core ML Tools

When converting a model using Core ML Tools, you may want to add layers and insert calculations in between, or cut a part of the layer, but this article will explain how to do it.

Since Core ML Tools has few documents and little information on the net, I have to look it up while reading the source code, but I think that editing layers is a common use case for conversion to Core ML. I hope this article helps you.

Model case: Put reshape before output to change the shape of output

As a model case, let's add a reshape to make the model output shape (19) (1,1,19).

It doesn't make much sense because it's just adding dimensions, but I'm doing this because I want a simple case.

The model to be used is one that predicts (classifies) the result of adding two single-digit numbers. The classification result will be 19 arrays. This is because adding one-digit numbers gives a total of 19 types of classification problems from 0 to 18.

It looks like this in the figure.

This model is the same as the one used in the previous article. In the previous article, even the label name was given to the output, but this time we will simply output 19 probabilities as they are.

Run a simple model made with Keras on iOS using CoreML

procedure

Now let's add the Reshape layer. Work on Google Colaboratory.

The complete code for this work can be found here. https://gist.github.com/TokyoYoshida/2fada34313385d63b666253490b5f3f4

** 1. Load Keras model **

I will use a model made with Keras. The model creation part is omitted because it is in Previous article.

notebook


from keras.models import load_model
keras_model = load_model('my_model.h5')

** 2. Convert to CoreML **

Convert to CoreML. Since I will not put a label this time, I am simply converting it as it is.

notebook


from coremltools.converters import keras as converter
mlmodel = converter.convert(keras_model)

** 3. Load with builder **

Load it into Neural Network Builder in Core ML Tools.

notebook


import coremltools
spec = coremltools.utils.load_spec(coreml_model_path)
builder = coremltools.models.neural_network.NeuralNetworkBuilder(spec=spec)

Let's display the model information. From this you can see that the output of the model is named output1 and the shape is 19.

notebook


spec.description.output

#Output result
# [name: "output1"
# type {
#   multiArrayType {
#     shape: 19
#     dataType: DOUBLE
#   }
# }
# ]

Displays layer information.

notebook


builder.layers
#output
# ['dense_4',
#  'dense_4__activation__',
#  'dense_5',
#  'dense_5__activation__',
#  'dense_6',
#  'activation_18']

The target this time is activation_18, which is the output layer. Reshape the output of this layer.

** 4. Add Rehape layer **

NeuralNetworkBuilder has a method called add_reshape that allows you to add a reshape layer.

~~ It's important to note here that using the builder, I feel that I can intuitively add a layer to the builder that represents the original model, but this method does not work. ~~ (Added on October 27, 2020, I found that I can add a layer to the builder that represents the original model, so I will correct it)

For example, let's add the Reshape layer as it is to the model loaded in the builder earlier.

notebook


reshape = builder.add_reshape(name='Reshape', input_name='activation_18', output_name='output', target_shape=(1,1,19), mode=0)

When I read this with Xcode, I get this error.

Xcode error


There was a problem decoding this CoreML document
validator error: Layer 'Reshape' consumes an input named 'activation_18' which is not present in this network.

When I try to perform CoreML inference on Python, I get this error:

Python error


RuntimeError: Error compiling model: "Error reading protobuf spec. validator error: Layer 'Reshape' consumes an input named 'activation_18' which is not present in this network.".

I had a hard time finding it on the internet, but I'm sure there are people who have a hard time doing the same thing. The reason for this error is that ʻinput_name ='activation_18'. Here you need to specify the name of the input or output displayed in builder.spec.description` in the original model.

~~ As a workaround, use NeuralNetworkBuilder to create a complete model with only one layer and add it to the original model. ~~ (Corrected on October 27, 2020) If the input_name argument of add_reshape matches the output_name of the original model, you can successfully connect the layers.

Add a reshape layer to your model. input specifies the output of the original model, ʻoutput1. I felt like this was the input of the new model, ʻinput2, but it seems that the output of the original model and the input to the reshape layer do not work well. output specifies ʻoutput2`, which is the output of the new model. target_shape specifies (1,1,19) which is the shape you want to convert this time.

notebook


builder.add_reshape(name='Reshape', input_name='output1', output_name='output2', target_shape=(1,1,19), mode=0)

You now have a model with reshape added.

Check the layer information.

notebook


builder.layers

#output
# ['dense_4',
#  'dense_4__activation__',
#  'dense_5',
#  'dense_5__activation__',
#  'dense_6',
#  'activation_18',
#  'Reshape']

** 5. Change the output information of the model **

There was a reshape layer at the end of the model, but that only replaced the last layer of the model, and the output information of the model as a whole did not change.

If you check the output information as a trial, you can see that it is still (19) instead of the expected shape (1,1,19).

notebook


spec.description.output
# [name: "output1"
# type {
#   multiArrayType {
#     shape: 19
#     dataType: DOUBLE
#   }
# }
# ]

Changing the output information of the model is also very quirky and will result in an error if you try to assign it to builder.spec.description.output [0]. Therefore, specify it while popping or adding.

notebook


//Delete one output information. Since there was only one output information this time, the output information disappears
builder.spec.description.output.pop()
//Add one output information
builder.spec.description.output.add()
//Specify the attributes of the output information
output = builder.spec.description.output[0]
output.name = "output2"
output.type.multiArrayType.dataType = coremltools.proto.FeatureTypes_pb2.ArrayFeatureType.ArrayDataType.Value('DOUBLE')
//As shape information(1,1,19)To set
output.type.multiArrayType.shape.append(1)
output.type.multiArrayType.shape.append(1)
output.type.multiArrayType.shape.append(19)

(Supplement) In the above example, the information was added after making it clean by deleting it with pop () and adding it with add (), but it is possible to directly rewrite only the attribute value of the output that originally exists.

For example, rewrite the shape as follows.

Example of rewriting shape


builder.spec.description.output[0].type.multiArrayType.shape[0] = 100

(End of supplement)

If you check the output information, you can see that the settings are successful.

notebook


builder.spec.description.output
# [name: "output2"
# type {
#   multiArrayType {
#     shape: 1
#     shape: 1
#     shape: 19
#     dataType: DOUBLE
#   }
# }
# ]

** 6. Check the output result of the model with Jupyter Notebook **

Check the output result of the model with Jupyter Notebook running on Mac. Why Jupyter Notebook on Mac? However, since the back end of Google Colaboratory is Linux, CoreML cannot be inferred and it is not possible to confirm whether the model is working properly.

You can infer with Xcode, but Xcode may give you an error when you read a strange model, but it may also drop the following error at build time.

Xcode error


Command CoreMLModelCompile failed with a nonzero exit code

If you get such an error, it's a good idea to run it in Jupyter Notebook to see the details of the error.

So, start Jupyter Notebook.

Here is the complete code. https://gist.github.com/TokyoYoshida/632b4c8070aa6c937539e4ae261a2740

Give [2,3] as input to the model and try to perform the inference.

JupyterNoteBook


coreml_model_path= "my_model_with_builder.mlmodel"

import coremltools
spec = coremltools.utils.load_spec(coreml_model_path)
builder = coremltools.models.neural_network.NeuralNetworkBuilder(spec=spec)

mlmodel = coremltools.models.MLModel(spec)

mlmodel.predict({'input1': np.array([2.0,3.0])})

# {'output2': array([[[2.56040733e-21, 7.25779588e-15, 1.99342376e-10, 1.11184195e-09,
#           5.92091055e-05, 9.99939799e-01, 9.72097268e-07, 1.13292452e-14,
#           4.43997455e-23, 5.00404492e-33, 0.00000000e+00, 0.00000000e+00,
#           0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,
#           0.00000000e+00, 0.00000000e+00, 0.00000000e+00]]])}

Since this model is a model that adds two elements, the probability of the fifth element is almost 1, so you can see that it can be inferred correctly.

Also, since the output array is [[[19 arrays with probabilities]]], it is confirmed that the shape is (1,1,19) as intended. I can do it.

** 7. Check the output result of the model with Xcode & app **

Finally, put the output result of the model on Xcode and check it with the application.

The complete code is here. https://github.com/TokyoYoshida/CoreMLSimpleTest

ViewController.swift


let model = my_model_with_builder()
let inputArray = try! MLMultiArray([2,3])
let inputToModel = my_model_with_builderInput(input1: inputArray)
if let prediction = try? model.prediction(input: inputToModel) {
print(prediction.output2)
try! print(prediction.output2.reshaped(to: [19]))
}

// # Double 1 x 1 x 19 array
// # Double 19 vector
// # [2.422891379885771e-21,7.01752566646674e-15,1.959301054732521e-10,1.089580203839091e-09,5.933549255132675e-05,0.9999396800994873,9.530076567898504e-07,1.087061586308846e-14,4.16250629238845e-23,4.617410135639087e-33,1.401298464324817e-45,1.401298464324817e-45,1.401298464324817e-45,1.401298464324817e-45,1.401298464324817e-45,1.401298464324817e-45,0,0,0]
// 

You can see that this is also as intended. The probability numbers may be slightly different, but don't worry, this is due to the retraining of the model.

Extra edition How to delete a layer

I have not confirmed the operation, but I will also write how to delete the layer.

Delete the layer.

notebook


layers = builder.spec.neuralNetwork.layers
#Get the second from the back of the layer(activation_18)
item = layers[-2]
#Delete that element
layers.remove(item)

If you look at the layer information, you can see that it has been deleted.

notebook


for layer in layers:
  print(layer.name)
# dense_4
# dense_4__activation__
# dense_5
# dense_5__activation__
# dense_6
# Reshape

Finally

Note regularly publishes about iOS development, so please follow us. https://note.com/tokyoyoshida

It is also posted on Twitter. https://twitter.com/jugemjugemjugem

Recommended Posts

How to edit layers in Core ML Tools
How to write a core mod in Minecraft Forge 1.15.2
How to use Lombok in Spring
How to find May'n in XPath
How to hide scrollbars in WebView
How to run JUnit in Eclipse
How to iterate infinitely in Ruby
[Rails] How to write in Japanese
How to run Ant in Gradle
How to master programming in 3 months
How to learn JAVA in 7 days
How to get parameters in Spark
How to install Bootstrap in Ruby
How to use InjectorHolder in OpenAM
How to introduce jQuery in Rails 6
How to use classes in Java?
How to name variables in Java
How to set Lombok in Eclipse
How to concatenate strings in java
How to install Swiper in Rails
[swift5] How to specify color in hexadecimal
How to implement search functionality in Rails
How to implement date calculation in Java
How to implement Kalman filter in Java
Multilingual Locale in Java How to use Locale
How to change app name in rails
How to get date data in Ruby
How to use custom helpers in rails
How to reflect seeds.rb in production environment
How to use named volume in docker-compose.yml
How to insert a video in Rails
How to standardize header footer in Thymeleaf
How to include Spring Tool in Eclipse 4.6.3?
Don't know how to build task'credentials: edit'
How to add jar file in ScalaIDE
How to do base conversion in Java
[Swift] How to fix Label in UIPickerView
How to have params in link_to method
How to use Docker in VSCode DevContainer
How to use MySQL in Rails tutorial
How to fix system date in JUnit
How to implement coding conventions in Java
How to embed Janus Graph in Java
[rails] How to configure routing in resources
How to map tsrange type in Hibernate
How to get the date in java
How to implement ranking functionality in Rails
How to use environment variables in RubyOnRails
How to implement asynchronous processing in Outsystems
How to publish a library in jCenter
How to specify id attribute in JSF
Understand in 5 minutes !! How to use Docker
How to overwrite Firebase data in Swift
How to use credentials.yml.enc introduced in Rails 5.2
How to assemble JSON directly in Jackson
How to execute multiple commands in docker-compose.yml
[For beginners] How to debug in Eclipse
How to use ExpandableListView in Android Studio
How to display error messages in Japanese
Summary of how to select elements in Selenium
How to get keycloak credentials in interceptor class