A mechanism to call a Ruby method from Python that can be done in 200 lines

I made a library that allows you to call Ruby methods from Python. I will introduce method chains and iterators because they can be used naturally to some extent.

https://github.com/yohm/rb_call

How it was made

We are developing a Rails app that manages jobs for scientific and technological calculations, and the behavior can be controlled by the Ruby API. However, since there are many Python users in the field of scientific computing, there were many requests for a Python API instead of Ruby.

What can you do?

For example, suppose you have the following Ruby code.

minimal_sample.rb


class MyClass
  def m1
    "m1"
  end

  def m2(a,b)
    "m2 #{a} #{b}"
  end

  def m3(a, b:)
    "m3 #{a} #{b}"
  end

  def m4(a)
    Proc.new { "m4 #{a}" }
  end

  def m5
    enum = Enumerator.new{|y|
      (1..10).each{|i|
        y << "#{i}" if i % 5 == 0
      }
    }
  end
end

if $0 == __FILE__
  obj = MyClass.new
  puts obj.m1, obj.m2(1,2), obj.m3(3,b:4)  #=> "m1", "m2 1 2", "m3 3 4"
  proc = obj.m4('arg of proc')
  puts proc.call                           #=> "m4 arg of proc"
  e = MyClass.m5
  e.each do |i|
    puts i                                 #=> "5", "10"
  end
end

The same thing can be written in Python as follows.

minimal_sample.py


from rb_call import RubySession

rb = RubySession()                          # Execute a Ruby process
rb.require('./minimal_sample')              # load a Ruby library 'sample_class.rb'

MyClass = rb.const('MyClass')               # get a Class defined in 'sample_class.rb'
obj = MyClass()                             # create an instance of MyClass
print( obj.m1(), obj.m2(1,2), obj.m3(3,b=4) )
                                            #=> "m1", "m2 1 2", "m3 3 4"
proc = obj.m4('arg of proc')
print( proc() )                             #=> "m4 arg of proc"

e = obj.m5()                                # Not only a simple Array but an Enumerator is supported
for i in e:                                 # You can iterate using `for` syntax over an Enumerable
    print(i)                                #=> "5", "10"

You can call Ruby libraries from Python almost as they are. You can do method chains, and you can iterate using for. Although not in the sample, list comprehensions can be used as expected. You can also access Ruby exceptions.

For example, if you combine it with Rails code, you can write it like this.

rails_sample.py


author = Author.find('...id...')
Book.where( {'author':author} ).gt( {'price':100} ).asc( 'year' )

If you use metaprogramming and external libraries well, you can realize it with compact code that fits in one file, about 130 lines for Python and about 80 lines for Ruby. Of course, there are restrictions as described later, but it works fine for most applications.

How did you implement it?

Problems with normal RPC

I'm using RPC to call Ruby methods from Python. This time, I used a library (specification?) Called MessagePack-RPC. Click here for the basic usage of MessagePack RPC. http://qiita.com/yohm13/items/70b626ca3ac6fbcdf939 Ruby is started as a sub-process of Python process, and interprocess communication is performed by socket between Ruby and Python. Roughly speaking, give the method name and arguments you want to call from Python to Ruby, and return the return value from Ruby to Python. At that time, the specification for serializing the transmitted / received data is defined in MessagePack-RPC.

If it is a process of "simply giving an argument and returning a value", there is no problem with this method. However, Ruby often makes you want to do method chains, and there are many libraries that assume that. For example, in Rails, you would frequently write the following code.

Book.where( author: author ).gt( price: 100 ).asc( :year )

Such a method chain cannot be realized by ordinary RPC.

The problem is essentially due to the inability to save state in the middle of the method chain. RPC between Ruby and Python can only exchange objects that can be serialized with MessagePack, so the object after Book.where will be serialized when it is returned from the Ruby process to the Python process. Even if you want to call the method of, you cannot call it.

In other words, it is necessary to hold some Ruby object in the Ruby process, and a mechanism to refer to it later as needed is required.

Solution

Therefore, this time, the Ruby object that cannot be serialized by the return value from Ruby is kept in the Ruby process, and only the ID and class of the object are returned to the Python side. Define a class called RubyObject on the Python side, keep the pair of (ID, class) coming from the Ruby side as a member, and delegate the method call to that RubyObject to the object in the Ruby process. To do.

The process when returning a value on the Ruby side is roughly as follows.

@@variables[ obj.object_id ] = obj          #Keep the object so that it can be referenced later by ID
MessagePack.pack( [self.class.to_s, self.object_id] )  #Return class and object ID to Python side

However, anything that can be serialized with MessagePack, such as String and Fixnum, is sent to Python as it is.

The processing when it is received on the Python side

class RubyObject():
    def __init__(self, rb_class, obj_id):  #RubyObject holds class name and ID
        self.rb_class = rb_class
        self.obj_id = obj_id

#Handling of values returned by RPC
rb_class, obj_id = msgpack.unpackb(obj.data, encoding='utf-8')
RubyObject( rb_class, obj_id )

When I write it in a picture, it looks like this, and the object on the Python side is an image that has only a pointer to a Ruby object.

RbCall.png

After that, it is OK if you transfer the method call made to RubyObject in Python to the actual Object on the Ruby side. Define a __getattr__ method (method_missing in Ruby) that is called when an attribute that does not exist is called for RubyObject.

class RubyObject():
    ...
    def __getattr__( self, attr ):
        def _method_missing(*args, **kwargs):
            return self.send( attr, *args, **kwargs )
        return _method_missing

    def send(self, method, *args, **kwargs):
        #Object ID in RPC,Send method name and arguments to Ruby
        obj = self.client.call('send_method', self.obj_id, method, args, kwargs )
        return self.cast(obj)       #Cast the return value to a RubyObject

Code called on the Ruby side

  def send_method( objid, method_name, args = [], kwargs = {})
    obj = find_object(objid)                       #Get the saved object from objid
    ret = obj.send(method_name, *args, **kwargs)   #Execute method
  end

Then, the method called for RubyObject in Python will be called as the method of Ruby. With this, Ruby objects also behave as if they were assigned to Python variables, and Ruby methods can be called naturally from Python.

Points to be careful about when implementing

Use MessagePack Extension Type

MessagePack has a specification that allows you to define a user-defined type called Extension type. https://github.com/msgpack/msgpack/blob/master/spec.md#types-extension-type

This time, I defined RubyObject (that is, String of class name and Fixnum of object ID) as Extension type and used it. On the Ruby side, I monkey patched Object and defined the to_msgpack_ext method. By the way, the latest msgpack gem supports Extension type, but msgpack-rpc-ruby seems to have stopped development and did not use the latest msgpack. Forked to rely on the latest gems. https://github.com/yohm/msgpack-rpc-ruby The code looks like this:

Object.class_eval
  def self.from_msgpack_ext( data )
    rb_cls, obj_id = MessagePack.unpack( data )
    RbCall.find_object( obj_id )
  end

  def to_msgpack_ext
    RbCall.store_object( self )   #Save the object in the variable
    MessagePack.pack( [self.class.to_s, self.object_id] )
  end
end
MessagePack::DefaultFactory.register_type(40, Object)

On the Python side as well, I wrote a process to convert the 40th Extension Type to the RubyObject type.

Counting the number of object references

At this rate, every time an object is returned from the Ruby side to the Python side, the variables saved in the Ruby process increase monotonically and a memory leak occurs. Variables that are no longer referenced on the Python side need to be dereferenced on the Ruby side as well.

That's why I overridden Python's RubyObject __del__. __del__ is a method called when the variable that refers to an object in Python is 0 and can be collected by GC. At this timing, the variables on the Ruby side are also deleted. http://docs.python.jp/3/reference/datamodel.html#object.del

    def __del__(self):
        self.session.call('del_object', self.obj_id)

The following code is called on the Ruby side.

  def del_object
    @@variables.delete(objid)
  end

However, if you simply use this method, it will not work properly if two Python variables refer to one Ruby Object. Therefore, the number of references returned from the Ruby side to the Python side is also counted, and the variables on the Ruby side are also released when it becomes zero.

class RbCall
  def self.store_object( obj )
    key = obj.object_id
    if @@variables.has_key?( key )
      @@variables[key][1] += 1
    else
      @@variables[key] = [obj, 1]
    end
  end

  def self.find_object( obj_id )
    @@variables[obj_id][0]
  end

  def del_object( args, kwargs = {} )
    objid = args[0]
    @@variables[objid][1] -= 1
    if @@variables[objid][1] == 0
      @@variables.delete(objid)
    end
    nil
  end
end

Objects deleted from @@ variables are properly released by Ruby's GC. There should be no problem managing the life of the object.

exception

Ruby exception information can also be obtained. In the published msgpack-rpc-ruby, when an exception occurs on the Ruby side, the exception is sent as to_s, but this method loses most of the exception information. Therefore, the exception object of Ruby is also sent as an instance of RubyObject of Python. Again, I've tweaked msgpack-rpc-ruby and made changes to serialize if Msgpack can serialize it, rather than always doing to_s.

The processing when an exception occurs is as follows. If an exception occurs on the Ruby side, msgpackrpc.error.RPCError will occur on the Python side. This is a specification of msgpack-rpc-python. Put an instance of RubyObject in the ʻargsattribute of the exception. If RubyObject is included, throwRubyExceptiondefined on Python side. At that time, the reference to the exception object generated on the Ruby side is stored in the attributerb_exception`. Now you can access the exceptions on the Ruby side.

The processing on the Python side is simplified and written as follows.

class RubyObject():
    def send(self, method, *args, **kwargs):
        try:
            obj = self.session.client.call('send_method', self.obj_id, method, args, kwargs )
            return self.cast(obj)
        except msgpackrpc.error.RPCError as ex:
            arg = RubyObject.cast( ex.args[0] )
            if isinstance( arg, RubyObject ):
                raise RubyException( arg.message(), arg ) from None
            else:
                raise

class RubyException( Exception ):
    def __init__(self,message,rb_exception):
        self.args = (message,)
        self.rb_exception = rb_exception

For example, when a Ruby ArgumentError is generated, the Python process is as follows.

try:
    obj.my_method("invalid", "number", "of", "arg")  #RubyObject my_Incorrect number of method arguments
except RubyException as ex:    #Exception called RubyException occurs
    ex.args.rb_exception       # ex.args.rb_Exception has a RubyObject that references a Ruby exception

Generator compatible

For example, consider performing the following Ruby processing.

articles = Article.all

ʻArticle.all` is not an Array but an Enumerable and is not actually expanded as an array in memory. The database is accessed for the first time when each is turned, and the information of each record can be acquired.

On the Python side as well, it is necessary to define a generator to run the loop in the form of for a in articles.

To do this, define the __iter__ method in the RubyObject class on the Python side. __iter__ is a method that returns an iterator, and this method is implicitly called in the for statement. This corresponds directly to Ruby's ʻeach, so call ʻeach in __iter__. http://anandology.com/python-practice-book/iterators.html https://docs.ruby-lang.org/ja/latest/class/Enumerator.html

In Python, when turning a loop, the __next__ method is called for the return value of __iter__. There is exactly the same correspondence in Ruby, and ʻEnumerator # next is the corresponding method. When the iteration reaches the end, the exception "StopIteration" is thrown on the Ruby side. Python has the same specifications, and when an exception occurs, an exception called StopIteration is thrown. (It happens to be an exception with the same name.)

class RubyObject():
    ...
    def __iter__(self):
        return self.send( "each" )
    def __next__(self):
        try:
            n = self.send( "next" )
            return n
        except RubyException as ex:
            if ex.rb_exception.rb_class == 'StopIteration': #When a Stop Iteration exception is thrown in Ruby
                raise StopIteration()  #Raise StopIteration exception in Python
            else:
                raise

Now you can use the loop from Python to Ruby's Enumerable.

others

Define methods in RubyObject so that Python's built-in functions work properly. The corresponding Ruby methods are:

-- __eq__ is == -- __dir __ is public_methods -- __str__ is to_s -- __len__ is size --__getitem__ is [] --__call__ is call

Limitations

--Cannot pass Python objects (those that cannot be serialized with MessagePack) to Ruby method arguments -Can't pass a Python function to a Ruby method that receives a block (essentially the same problem as above) --In principle, it should be possible to refer to Python objects from Ruby processes in the same way, but it seems to be complicated to implement, so I leave it as it is. --There is a method name that cannot be used on Python syntax. --For example, you can use method names such as .class, ʻis_a?in Ruby, but not in Python. --This problem can be avoided by calling.send ('class'), .send ('is_a?'). --Some Ruby libraries may have problems. --For example, in Mongoid, there is a class that undefs almost all public_methods for metaprogramming. - http://qiita.com/yohm13/items/40376eafc045492d5f4f --In such a case, the to_msgpack_extdefined in rb_call will also be undefined and will not work properly. --After requiring Mongoid, you can avoid it by redefiningto_msgpack_ext` in the corresponding class. --Unconfirmed, ActiveRecord may cause the same problem.

When I implemented it, I found that Python and Ruby have a very similar correspondence, and it can be implemented very neatly just by defining the corresponding methods well. It's actually about 200 lines of code, so if you are interested, please read the source.

Recommended Posts

A mechanism to call a Ruby method from Python that can be done in 200 lines
How to use the __call__ method in a Python class
How to set up a simple SMTP server that can be tested locally in Python
A record that GAMEBOY could not be done in Python. (PYBOY)
A story that heroku that can be done in 5 minutes actually took 3 days
A solution to the problem that the Python version in Conda cannot be changed
I tried to create a class that can easily serialize Json in Python
I made a Docker image that can call FBX SDK Python from Node.js
I made a familiar function that can be used in statistics with Python
How to install a Python library that can be used by pharmaceutical companies
How to install a Python library that can be used by pharmaceutical companies
A mechanism to call a Ruby method from Python that can be done in 200 lines
I tried to summarize the contents of each package saved by Python pip in one line
Investigation of DC power supplies that can be controlled by Python
[Python3] Code that can be used when you want to cut out an image in a specific size
From a book that programmers can learn ... (Python): Pointer
[Python] Created a method to convert radix in 1 second
[Python] How to call a c function from python (ctypes)
Convert images from FlyCapture SDK to a form that can be used with openCV
How to slice a block multiple array from a multiple array in Python
From a book that programmers can learn ... (Python): About sorting
Processing of python3 that seems to be usable in paiza
From a book that programmers can learn (Python): Decoding messages
Text analysis that can be done in 5 minutes [Word Cloud]
I tried "How to get a method decorated in Python"
Scripts that can be used when using bottle in Python
How to call Python or Julia from Ruby (experimental implementation)
[Python3] Code that can be used when you want to resize images in folder units
Fluid visualization BOS method that can be done at home to see what is invisible
A memorandum (masOS) that import tkinter could not be done with python installed from pyenv
[Python] A program to find the number of apples and oranges that can be harvested
Convert mesh data exported from SpriteUV2 to a format that can be imported by Spine
Call Matlab from Python to optimize
Call a Python function from p5.js.
[Python] Make a graph that can be moved around with Plotly
8 services that even beginners can learn Python (from beginners to advanced users)
Post a message from IBM Cloud Functions to Slack in Python
Call popcount from Ruby / Python / C #
About psd-tools, a library that can process psd files in Python
How to get a string from a command line argument in python
Make a Spinbox that can be displayed in Binary with Tkinter
From a book that programmers can learn (Python): Find the mode
From a book that programmers can learn ... (Python): Review of arrays
A timer (ticker) that can be used in the field (can be used anywhere)
Simple code to call a python program from Javascript on EC2
A function that measures the processing time of a method in python
Call dlm from python to run a time-varying coefficient regression model
Python standard input summary that can be used in competition pro
・ <Slack> Write a function to notify Slack so that it can be quoted at any time (Python)
Check if you can connect to a TCP port in Python
Make a Spinbox that can be displayed in HEX with Tkinter
From a book that programmers can learn (Python): Statistical processing-deviation value
I wrote a tri-tree that can be used for high-speed dictionary implementation in D language and Python.
There are no “private” instance variables in Python that can only be accessed from within an object.
[Python] A program that finds a pair that can be divided by a specified value
I made a web application in Python that converts Markdown to HTML
[Python] A program that calculates the number of socks to be paired
How to make a rock-paper-scissors bot that can be easily moved (commentary)
Serverless LINE Bot that can be done in 2 hours (source identifier acquisition)
Extract lines that match the conditions from a text file with python
[Can be done in 10 minutes] Create a local website quickly with Django
How to get a value from a parameter store in lambda (using python)
From a book that programmers can learn (Python): Conditional search (maximum value)
I tried to develop a Formatter that outputs Python logs in JSON