I made a library that adds docstring to a Python stub file.

Recently, there have been cases where Python stub files are used for work. However, mypy's stub file generation command does not add docstring to the stub file, so I made a library to add it myself, so I will write about that.

What is a stub file in the first place?

It is a file with the extension .pyi, which describes Python type information and omits the logic part. Using a stub file has the following advantages.

--Due to the convenience of decorators and the convenience of third-party libraries, you can enable completion and checking for modules that do not work well. --Type information can be added by adding a separate stub file to a third-party library that does not have type annotations (checking is also enabled). ――For large modules that have a high load when many type inferences are run, using the minimum stub file that omits the mounting part reduces the load and reduces delays such as completion and checking.

On the contrary, as a disadvantage,

--Since the definition of type information is overwritten in the stub file for the original module, it is not preferable in the case where the project code is frequently updated because it deviates from the contents of the module (when using it in a project). It would be nice to have a mechanism to update the stub file in real time as much as possible). -* Basically, if it is a library code, etc., it will not be updated frequently, so it will be a good match for reducing the load on the editor.

And so on.

For details on how to create a stub file, see How to create a package containing type hints conforming to PEP 561.

Why did you start using stub files?

Some code in the project is no longer completed or type checked on VS Code (due to decorators etc.). Although it worked before, it is unknown whether it is due to the update or the library used on VS Code, but it is inconvenient if it is left as it is even after waiting for a while, so some Only the project module has stub files added in real time (I tried various other measures, but in the end, partial stub file support seemed to be quick and calm).

I don't want to create a stub file manually, so I use mypy's stub file generation command but the docstring disappears

It's okay to decide to use the stub file in your project, but if you need to manually update the stub file, it simply increases the man-hours and is not preferable. I would like you to update the stub file in real time while automating as much as possible.

Regarding the automatic generation of stub files, mypy provides a function of a command called stubgen, so I am using that.

Automatic stub generation (stubgen)

This avoids the manual generation of stub files, which greatly reduces the burden. However, the docstring disappears in the stub file generated using this stubgen.

For example, if you have the following module code,

from random import randint

sample_int: int = 100


def sample_func(a: int, b: str) -> bool:
    """
    Lorem ipsum dolor sit amet, consectetur adipiscing elit,
    ed do eiusmod tempor incididunt ut labore et dolore magna aliqua.

    Parameters
    ----------
    a : int
        Lorem ipsum dolor sit amet, consectetur adipiscing elit.
    b : str
        ed do eiusmod tempor incididunt ut labore et dolore magna aliqua.

    Returns
    -------
    c : bool
        Ut enim ad minim veniam, quis nostrud exercitation.
    """
    return True


class SampleClass:

    def __init__(self) -> None:
        """
        Lorem ipsum dolor sit amet, consectetur adipiscing elit.
        """

    @property
    def sample_property(self) -> int:
        """
        Lorem ipsum dolor sit amet, consectetur adipiscing elit.

        Returns
        -------
        d : int
            ed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
        """
        return randint(0, 100)

If you use the stubgen command to generate a stub file with the contents of this module, it will look like the following.

sample_int: int

def sample_func(a: int, b: str) -> bool: ...

class SampleClass:
    def __init__(self) -> None: ...
    @property
    def sample_property(self) -> int: ...

You can see that the type information remains, but the docstring has disappeared. In each extension used on VS Code (Pylance etc.), if the stub file is placed in the same directory as the module, the stub file seems to have priority, in this case the contents of docstring are displayed on the editor It will not be.

That's why I made it myself and registered it as PyPI (pip)

So I wrote a library that gives the docstring of the original module to the stub file and registered it in PyPI (pip) (MIT license).

It was okay to write it at work, but I felt that it might be used privately, so I wrote it as OSS privately (so that it can be used for other private projects etc.).

stubdoc - Github

I finished it in a short time (I have a lot of other things I want to do), so please forgive me for the rough cutting (because if I stick to it, it will not finish in a few days ...). I think I'll start using it at work soon, so if I notice something that doesn't work there, I'll update it from time to time.

Installation can be done with the pip command.

$ pip install stubdoc

The usage is the path of the original module file of the stub file in the -m argument (or --module_path) (since import is done internally by referring to this path as a module, .. in the upper hierarchy. Specify the path of the stub file to which you want to add the docstring to the -s argument (or --stub_path) in/or the root / path specification does not work.).

Command example:

$ stubdoc -m samples/sample.py -s out/samples/sample.pyi

Or you can handle it on Python.

from stubdoc import add_docstring_to_stubfile

add_docstring_to_stubfile(
    original_module_path='sample/path.py',
    stub_file_path='sample/path.pyi')

As a result, the contents of the stub file given as an example earlier will be replaced with the one with docstring as shown below.

sample_int: int

def sample_func(a: int, b: str) -> bool:
    """
    Lorem ipsum dolor sit amet, consectetur adipiscing elit,
    ed do eiusmod tempor incididunt ut labore et dolore magna aliqua.

    Parameters
    ----------
    a : int
        Lorem ipsum dolor sit amet, consectetur adipiscing elit.
    b : str
        ed do eiusmod tempor incididunt ut labore et dolore magna aliqua.

    Returns
    -------
    c : bool
        Ut enim ad minim veniam, quis nostrud exercitation.
    """

class SampleClass:
    def __init__(self) -> None:
        """
        Lorem ipsum dolor sit amet, consectetur adipiscing elit.
        """
    @property
    def sample_property(self) -> int:
        """
        Lorem ipsum dolor sit amet, consectetur adipiscing elit.

        Returns
        -------
        d : int
            ed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
        """

Limitations

It works only when the Ellipsis instance (...) and the pass keyword description are described after the colon of the function without line breaks (the same form as the output result of mypy's stubgen command). For example, it does not support stub files that have a line break before the Ellipsis instance (such as when a stub file is manually generated) as shown below.

def sample_func(a: int, b: str) -> bool:
    ...

class SampleClass:
    def __init__(self) -> None:
        ...
    @property
    def sample_property(self) -> int:
        pass

Also, it does not support nested functions. Only the methods of functions or classes defined at the top level are targeted. For example, the function that is nested as shown below is not checked.

def sample_func_1(a: int, b: str) -> bool:
    def sample_func_2(c: list) -> None: ...

References / Reference Site Summary

-How to make a package with PEP 561 compliant type hints

Recommended Posts

I made a library that adds docstring to a Python stub file.
I made a python library to do rolling rank
I made a configuration file with Python
I made a library to easily read config files with Python
[python] I made a class that can write a file tree quickly
I made a python dictionary file for Neocomplete
I want to write to a file with Python
[Python] I made a decorator that doesn't seem to have any use.
I made a web application in Python that converts Markdown to HTML
I made a library konoha that switches the tokenizer to a nice feeling
I made a library to separate Japanese sentences nicely
I made a Line Bot that uses Python to retrieve unread Gmail emails!
I made a library to operate AWS CloudFormation stack from CUI (Python Fabric)
I made a VM that runs OpenCV for Python
I made a Python module to translate comment outs
I want to randomly sample a file in Python
[Python] How to write a docstring that conforms to PEP8
I made a python text
A story that I was addicted to when I made SFTP communication with python
I made a system that automatically decides whether to run tomorrow with Python and adds it to Google Calendar.
A Python script that saves a clipboard (GTK) image to a file.
I made a package to filter time series with python
I made a Line-bot using Python!
I made a fortune with Python.
I made a daemon with Python
I made a toolsver that spits out OS, Python, modules and tool versions to Markdown
[Python] I made a Line bot that randomly asks English words.
[Python3] I made a decorator that declares undefined functions and methods.
[Python] I made my own library that can be imported dynamically
I made a package that can compare morphological analyzers with Python
I want to use a wildcard that I want to shell with Python remove
I tried to convert a Python file to EXE (Recursion error supported)
Python> I made a test code for my own external file
I created a Python library to call the LINE WORKS API
[Python] A memo that I tried to get started with asyncio
I made a shuffle that can be reset (reverted) with Python
I made a Python wrapper library for docomo image recognition API.
When writing to a csv file with python, a story that I made a mistake and did not meet the delivery date
I made Othello to teach Python3 to children (4)
I made a payroll program in Python!
I made a character counter with Python
I made Othello to teach Python3 to children (2)
I want to build a Python environment
I made a garbled generator that encodes favorite sentences from UTF-8 to Shift-JIS (cp932) in Python
I made Othello to teach Python3 to children (5)
I made a script to display emoji
I made a Hex map with Python
After studying Python3, I made a Slackbot
I made a roguelike game with Python
I made Othello to teach Python3 to children (3)
I made Othello to teach Python3 to children (1)
I made a simple blackjack with Python
I made a library for actuarial science
I made a neuron simulator with Python
I made a Docker container to use JUMAN ++, KNP, python (for pyKNP).
I made a password generator to teach Python3 to children (bonus) * Completely remade
I made a tool to automatically browse multiple sites with Selenium (Python)
I made a Discord bot in Python that translates when it reacts
I tried to develop a Formatter that outputs Python logs in JSON
I tried to discriminate a 6-digit number with a number discrimination application made with python
[Python] I made a utility that can access dict type like a path