Fast file transfer with fabric

Postscript: The method of http://qiita.com/NPoi/items/46364461f0ab76e986c3 is smarter!

Isn't fabric put slow? Let's use rsync. In the case of transferring the entire directory with many static files such as images, HTML, JS, etc., the execution time of fabric is dramatically different, so I will introduce it. (I am assuming people who use fabric to some extent)

Rsync as a non-root user

This is an official feature provided by fabric so it's easy to use.

from fabric.contrib.project import rsync_project

@task
def hoge():
	rsync_project(
		local_dir='./mydir',
		remote_dir='/usr/local/destination/mydir',
		exclude=['.DS_Tore', '*.tmp'],
		delete=True
	)

rsync as root user

After rsyncing as a non-root user, you can do mv or cp -r, but the nice thing about rsync is that you can only transfer the differences, so I created this function. The arguments are aligned with rsync_project.

The behavior is to rsync once to / tmp / (directory name hash value) / and then sudo rsync on the server.

It depends on the is_dir function of fabtools, so you need pip install fabtools

import hashlib
from fabric.api import local, run, sudo
from fabric.contrib.project import rsync_project
from fabtools.files import is_dir

def root_rsync(local_dir, remote_dir, exclude=[], delete=False):
	def _end_with_slash(dir_path):
    	if dir_path[-1] == '/':
    	    return dir_path
    	else:
    	    return dir_path + '/'
    local_dir = _end_with_slash(local_dir)
    remote_dir = _end_with_slash(remote_dir)
    m = hashlib.md5()
    m.update(remote_dir)
    me = local('whoami', capture=True)
    remote_tmp_dir = '/tmp/%s/%s/' % (me, m.hexdigest())
    run('mkdir -p %s' % remote_tmp_dir)
    if is_dir(remote_dir):
        run('rsync -a %s %s' % (remote_dir, remote_tmp_dir))  # already exists
    rsync_project(
        remote_dir=remote_tmp_dir,
        local_dir=local_dir,
        exclude=exclude,
        delete=delete
    )
    sudo('rsync -a %s %s' % (remote_tmp_dir, remote_dir))

Recommended Posts

Fast file transfer with fabric
Bidirectional file transfer with Pythonista 3
Introducing Paramiko + scp ~ SSH connection ~ File transfer with SCP
File operations with open — "../"
File upload with django
Draw netCDF file with python
Transfer files with teraterm [Note]
File upload with Flask + jQuery
Download csv file with python
Create xlsx file with XlsxWriter
Do multi-stage SSH with Fabric
Extract the xz file with python
Get video file information with ffmpeg-python
[Python] Write to csv file with Python
[Automation with python! ] Part 1: Setting file
Implemented file download with Python + Bottle
Build a deb file with Docker
Output to csv file with Python
Serve static files with X-Send File
Create an Excel file with Python3
Output cell to file with Colaboratory
Giving idempotence to fabric with cuisine
Download the file deployed with appcfg.py
Automation of remote operations with Fabric
Use camera calibration file with OpenCvSharp4
I couldn't daemonize gunicorn with Fabric
[Automation with python! ] Part 2: File operation
Open the file with the default app
Easy deployment environment with gaffer + fabric
Create a file uploader with Django