I made a tool to compile Hy natively

Introduction

In Previous post, I introduced a tool for executing and archiving source code. Introducing this time is a tool that natively compiles Hy to create executable files and shared libraries. You can also convert it to Python or C language. It is registered in PyPI under the name HyCC. However, there is still the possibility of ** unexpected bugs **, so please use it at your own risk **. By the way, ** all are implemented in Hy **.

--Hy official documentation -Hy's Github repository -Qiita: Squid Lisp written in Python: Hy -Qiita: I made a package to create an executable file from Hy source code

How to use

Please install from pip. ** However, the operation in the Windows environment has not been confirmed. ** **

$ pip install hycc

~~ In addition, if you use the latest version of PyPI, Hy (v0.12.1), some code may cause an error due to a bug in Hy, so please install the latest version from Github. ~~ ** 2017/07/01 postscript: This work is no longer necessary due to the update of Hy and HyCC. **

Now that the preparation is complete, let's take the following code as an example.

hello.hy


(defn hello []
  (print "hello!"))

(defmain [&rest args]
  (hello))

Create an executable file

$ hycc hello.hy

This will create an executable binary called hello in the current directory.

$ ./hello
hello!

Create a shared library

Build with the --shared option.

$ hycc hello.hy --shared

This will create a shared object called hello.so in the current directory. Python has the ability to import shared objects as modules, but Hy is of course the same. Therefore, this hello.so can be used as follows.

Hy to hello.import so


(import hello)
(hello.hello)
; >hello!

Hello from python.import so


import hello
hello.hello()
# >hello!

Natively compiled modules and executables are ** about 2-8 times faster **. So I think it's more useful than byte-compiling to a .pyc file using hyc. And of course ** faster than Python **. In addition, Cython is used internally, but the image of speed is as follows.

** C <Cython (with type specification) << Cython (without type specification) <HyCC <Python <Hy **

It is generally ** upward compatible with hyc **, but there is one caveat **. Since it is related to the internal mechanism, I will explain it while explaining the mechanism from the following.

How it works

Roughly speaking, Hy's code is converted to Python, converted to C via Cython, and compiled. However, as mentioned in Last post, Python code generated using hy2py that comes standard with Hy. ** Doesn't work as it is **. Therefore, you can't just hy2py and cythonize.

Convert Hy to Python

In the first place ** Why doesn't the code that hy2py spits work **? Because Python uses an invalid identifier. The invalid identifiers in Python are:

    • Identifier starting with a number 0-9 *
  1. Identifiers containing characters other than * _, a-z, A-Z, 0-9 *

Take the following code as an example.

sample.hy


(reduce + [1 2 3])
; > 6

Converting this with hy2py gives the following.

Code that hy2py spits


from hy.core.language import reduce
from hy.core.shadow import +
# +Is an invalid identifier
reduce(+, [1, 2, 3])

Here, + is the above 2. Because it corresponds to, it is useless. Simply replacing this with another valid name, such as ʻadd`, won't work.

Of the code that hy2py spits+Replace with add


from hy.core.language import reduce
from hy.core.shadow import add
# ImportError
reduce(add, [1, 2, 3])

Obviously, the hy.core.shadow module does not have the name ʻadd defined, so it will be ʻImportError. HyCC first generates the following Python code from Hy source to solve this problem.

HyCC way


import hy.core.language as _
reduce = _.getattr("reduce")
import hy.core.shadow as _
+ = _.getattr("+")
reduce(+, [1, 2, 3])

This code is equivalent to the code that hy2py spits, but you can replace+with any name. HyCC avoids errors by properly combining access at the AST level and access at the source code level. Similarly, the member access of the object is rewritten with getattr and setattr.

Precautions when using HyCC (** important **)

About the precautions when using HyCC mentioned earlier. As explained so far, HyCC replaces invalid identifiers with valid ones with some ingenuity. At this time, only one side effect or problem will occur.

HyCC bad guy


(def hoge/fuga 0)
(print (get (globals) "hoge/fuga"))
; > 0

The above code is converted by HyCC to the following Python code.

Code generated by HyCC


from __future__ import print_function
import hy
hogex2Ffuga = 0L
print(globals()[u'hoge/fuga'])

You can ignore the addition of some imports. In the original Hy code, hoge / fuga was an invalid name in Python. Therefore, it is replaced with hogex2Ffuga in the code generated by HyCC.

Executing this code will result in the following error.

  File "test.py", line 4, in <module>
    print(globals()[u'hoge/fuga'])
KeyError: u'hoge/fuga'

Did you understand? The negative effect of replacing invalid identifiers with valid ones is that the globals, locals, and ʻinspect` modules may not work properly in some cases.

We are considering some countermeasures for this problem, but ** Hy itself already has a similar problem ** in the first place. In Hy, hoge! Is replaced with hoge_bang andhoge?Is replaced with ʻis_hoge` at the stage of parsing. Therefore, the following code does not work properly.

Hy and bad guy


(def hoge! 0)
(print (get (globals) "hoge!"))
; > KeyError!

Therefore, the use of globals etc. can be said to be ** cautions when using HyCC rather than cautions when using HyCC **.

*** 2017/06/04 postscript *** ** Supported by update! ** Specifically, Key error is avoided by wrapping globals and locals in a mystery class like dict. As usual, the ʻinspect module does not work properly, but since ** Cython itself does not support ʻinspect **, there is nothing I can do about it. For details, please refer to this commit.

Bonus (convert to Python that works properly)

HyCC also has a function to convert from Hy to Python like hy2py.

$ hycc hello.hy --python

This will output hello.py to the current directory. If you pay attention to the above notes, it works well unlike the code output by ** hy2py **. Surprisingly, this feature may be in higher demand. Similarly, you can convert it to C language with the --clang option, but it is delicate whether it can be used.

Known bug (2017/06/13 postscript)

Shared library from import fails

As I wrote in github issues, is it a Python specification? When getting a submodule with gettatr for a module object It seems to be ʻAttribute Error` if the submodule contains a shared library. I'm thinking about how to deal with it.

in conclusion

Tool to compile Hy natively was introduced. All development is done on github, so feel free to [pull request](https://github.com/koji-kojiro/hylang-hycc/ You can throw pulls) or issues. Of course, comments here are also welcome. Hy is still a developing fucking minor language, but I hope that it will become more sophisticated as the number of users increases and discussions become more active. Thank you for reading this far.

Recommended Posts

I made a tool to compile Hy natively
I made a tool to get new articles
I made a tool to create a word cloud from wikipedia
[Titan Craft] I made a tool to summon a giant to Minecraft
I made a script to display emoji
I made a browser automatic stamping tool.
〇✕ I made a game
I made a useful tool for Digital Ocean
I made a tool to automatically browse multiple sites with Selenium (Python)
I made a CLI tool to convert images in each directory to PDF
I made a router config collection tool Config Collecor
I made a tool to convert Jupyter py to ipynb with VS Code
I made a tool to estimate the execution time of cron (+ PyPI debut)
I made a tool to easily display data as a graph by GUI operation.
I made a tool to generate Markdown from the exported Scrapbox JSON file
I made a tool to automatically back up the metadata of the Salesforce organization
I made a package to create an executable file from Hy source code
I made a library to separate Japanese sentences nicely
I made a cleaning tool for Google Container Registry
I made a script to put a snippet in README.md
I made a Python module to translate comment outs
I made a code to convert illustration2vec to keras model
I made a command to markdown the table clipboard
I made a python library to do rolling rank
I made a repeating text data generation tool "rpttxt"
I made a python text
I made a discord bot
I made a package to filter time series with python
I made a box to rest before Pepper gets tired
I made a command to generate a table comment in Django
I made a function to check the model of DCGAN
I tried to get started with Hy ・ Define a class
I made you to execute a command from a web browser
I made a script to say hello at my Koshien
I made a toolsver that spits out OS, Python, modules and tool versions to Markdown
I made a tool that makes it a little easier to create and install a public key.
I made a tool to get the answer links of OpenAI Gym all at once
I made a tool to automatically generate a simple ER diagram from the CREATE TABLE statement
I made a tool that makes it convenient to set parameters for machine learning models.
I made a C ++ learning site
I made a Line-bot using Python!
I made a wikipedia gacha bot
I made a fortune with Python.
I made a CUI-based translation script
A tool to convert Juniper config
I made a daemon with Python
[5th] I tried to make a certain authenticator-like tool with python
I made a program to solve (hint) Saizeriya's spot the difference
I made a library to easily read config files with Python
[2nd] I tried to make a certain authenticator-like tool with python
I made a web server with Raspberry Pi to watch anime
[3rd] I tried to make a certain authenticator-like tool with python
I made a client / server CLI tool for WebSocket (like Netcat for WebSocket)
[4th] I tried to make a certain authenticator-like tool with python
I made a scaffolding tool for the Python web framework Bottle
[1st] I tried to make a certain authenticator-like tool with python
I made a library that adds docstring to a Python stub file.
I made a command to display a colorful calendar in the terminal
I tried to create a linebot (implementation)
I made a Docker container to use JUMAN ++, KNP, python (for pyKNP).
I made a plugin to generate Markdown table from csv in Vim