Note that cibuildwheel builds python bwheel (including C ++ module) in bulk with CI and uploads it to PyPI

If you are a Han, you want to install a prebuilt wheel that includes C ++ modules on Windows, macOS, and Linux with Python pip!

Let's do it with cibuildwheel!

https://github.com/joerick/cibuildwheel

It's a script that builds in various environments and python versions. It's great. You can incorporate this into your own project CI (travis, GitHub Actions, etc).

It's a little difficult to understand,

$ python3 -m cibuildwheel --output-dir wheelhouse

By executing, python setup.py bdist_wheel equivalent will be built and executed for each python version environment. (For Linux and macOS, use Docker).

Run the script before build

In cibuildwheel, Linux and macOS are executed in the Docker container, so when performing CI build with Travis etc., something like git submodule update must be executed inside the docker container.

You can set some in the environment variables of cibuildwheel.

https://cibuildwheel.readthedocs.io/en/latest/options/

Upload to pypi

Notes on uploading packages in Linux binaries to PyPI https://qiita.com/syoyo/items/6185380b8d9950b25561

Please refer to

Azure pipeline ...?

With Azure pipeline, upload automation after build with twine etc. seems to be troublesome at present, but is the setting of vispy helpful?

https://github.com/vispy/vispy/blob/master/azure-pipelines.yml

Templates have been proposed for the time being, but it seems that there are some restrictions.

https://github.com/joerick/cibuildwheel/issues/229

I actually tried using it with tinyobjloader, but the online documentation was hard to understand and it felt more troublesome than Travis. (It would be nice if the yaml description could be syntax checked with an online editor)

If you can use Travis, we recommend using travis (or Github actions).

Build and publish flow

      - task: PublishBuildArtifacts@1
        inputs:
          path: $(Build.ArtifactStagingDirectory)
          artifactName: tinyobjDeployWindows

In deploy (upload to e.g. PyPI), First check that the dependent task is complete (dependsOn) and that it is a tag commit.

  - job: deployPyPI
    # Based on vispy: https://github.com/vispy/vispy/blob/master/azure-pipelines.yml
    pool: {vmImage: 'Ubuntu-16.04'}
    condition: and(succeeded(), startsWith(variables['Build.SourceBranch'], 'refs/tags/v'))
    dependsOn:
      - linux
      - macos
      - windows

Then use DownloadBuildArtifacts to download the job artifacts (actually folders).

      - task: DownloadBuildArtifacts@0
        inputs:
          artifactName: 'tinyobjDeployWindows'
          downloadPath: $(Pipeline.Workspace)

At this time, a folder is created with ʻartifactName ($ (Pipeline.Workspace) / tinyobjDepoloyWinwdows`), and the build is placed in it. For debugging, it seems good to output it to the log with ls etc. so that you can check it.

Then publish it on twine and you're done!

      # Publish to PyPI through twine
      - bash: |
          cd $(Pipeline.Workspace)
          find .
          python -m pip install --upgrade pip
          pip install twine
          echo tinyobjDeployLinux/python/dist/*
          echo tinyobjDeployLinux/python/wheelhouse/* tinyobjDeployMacOS/python/wheelhouse/* tinyobjDeployWindows/python/wheelhouse/*
          twine upload -u "__token__" --skip-existing tinyobjDeployLinux/python/dist/* tinyobjDeployLinux/python/wheelhouse/* tinyobjDeployMacOS/python/wheelhouse/* tinyobjDeployWindows/python/wheelhouse/*
        env:
          TWINE_PASSWORD: $(pypiToken2)

For the TWINE password (secret token), register the secret token with variable on the Azure Pipeline website with the name pypiToken2.

Impressions

I do git push every time, trigger the build, wait for about 10 minutes, fix it if an error occurs & attach the tag again and push is troublesome, so I want an environment where I can try it at hand.

Twine token

Azure Pipeline also has a secret variables function that can handle Twine (PyPi) secret tokens, so you can use that.

https://docs.microsoft.com/en-us/azure/devops/pipelines/process/variables?view=azure-devops&tabs=yaml%2Cbatch#secret-variables

The Secret variable is specified via an environment variable in the build script.

Use PublishBuildArtifacts

PublishPipelineArtifact only prints logs that aren't even the error message ʻOne or more errors occurred` (no error reason is displayed) and it doesn't work.

https://stackoverflow.com/questions/58841733/how-to-debug-azure-devops-task-publishpipelineartifact-when-one-or-more-errors-o

Use PublishBuildArtifacts.

Actual sample with C ++ code

cibuildwheel, pypi upload,

pytinyexr (PyEXR) uses Travis.

https://github.com/syoyo/PyEXR

tinyobjloader uses Azure Pipeline.

https://github.com/tinyobjloader/tinyobjloader/blob/master/azure-pipelines.yml

Py2.7(or pypy) + Windows + pybind11

At least pypy will cause a build error, so be sure to include pp27-win32 and pp36-win32 in CIBW_SKIP and do not build. (Is anyone using pypy?)

Also, python2.7 + C ++ 11 is basically deprecated.

https://cibuildwheel.readthedocs.io/en/stable/cpp_standards/

Unless you have a strong reason to use py27, you shouldn't support python 2.7.

Recommended Posts

Note that cibuildwheel builds python bwheel (including C ++ module) in bulk with CI and uploads it to PyPI
Object-oriented in C: Refactored "○ ✕ game" and ported it to Python
Note that I understand the least squares algorithm. And I wrote it in Python.
How to generate permutations in Python and C ++
Convert PDFs to images in bulk with Python
A Python script that crawls RSS in Azure Status and posts it to Hipchat
A module that makes it easier to write Perl-like filter programs in Python fileinput
[Note] How to write QR code and description in the same image with python
It is easy to execute SQL with Python and output the result in Excel
Try to make a Python module in C language
How to run tests in bulk with Python unittest
[C / C ++] Pass the value calculated in C / C ++ to a python function to execute the process, and use that value in C / C ++.
A note that deployed a Python application from Circle CI to Elastic Beanstalk and notified Slack
Recursively get the Excel list in a specific folder with python and write it to Excel.
Try to make it using GUI and PyQt in Python
[Road to intermediate Python] Install packages in bulk with pip
How to do Bulk Update with PyMySQL and notes [Python]
Install CaboCha in Ubuntu environment and call it with Python.
How to log in to AtCoder with Python and submit automatically
Module summary that automates and assists WebDriver installation with Python
A story that didn't work when I tried to log in with the Python requests module
Unittest and CI in Python
Note that it supports Python 3
processing to use notMNIST data in Python (and tried to classify it)
How to install OpenCV on Cloud9 and run it in Python
I want to solve APG4b with Python (only 4.01 and 4.04 in Chapter 4)
Regular expressions that are easy and solid to learn in Python
Read big endian binary in Python and convert it to ndarray
Throw something to Kinesis with python and make sure it's in
Note that Python decodes base64 format image and saves it locally
Note that I understand the algorithm of the machine learning naive Bayes classifier. And I wrote it in Python.
[AWS lambda] Deploy including various libraries with lambda (generate a zip with a password and upload it to s3) @ Python