Dump the contents of redis db with lua

It's a continuation of this I threw lua to redis-server and scanned it in the server to return the key and value, so I made a note of it.

environment

$ python -V
Python 3.7.2

$ redis-cli info | grep redis_version
redis_version:5.0.5

code

When using hash or set, read as HSCAN or ZSCAN as appropriate.

dump.lua


local cr    = KEYS[1]
local num   = KEYS[2]
local match = KEYS[3]

local cr2, keys = unpack(redis.call("SCAN", cr, "MATCH", match, "COUNT", num))

local values = redis.pcall("MGET", unpack(keys))
return {cr2, keys, values}

export.py


#!/usr/bin/env python
import os
import redis
import click

DEFAULT_REDIS_URL = "redis://localhost:6379/3"
DEFAULT_LUA_FILE  = "dump.lua"

def scan_iter(r, lua_script, bulk=2000, match="*"):
    f = r.register_script(lua_script)

    cr = "0"
    while True:
        cr, keys, values = f(keys=[cr, bulk, match])
        if cr == b"0":
            break

        yield from zip(keys, values)


@click.command()
@click.option("--redis-url", envvar="REDIS_URL", default=DEFAULT_REDIS_URL)
@click.option("--lua-file", envvar="LUA_FILE", default=DEFAULT_LUA_FILE, type=click.File("r"))
@click.option("--output", default=os.sys.stdout, type=click.File("w"))
def cmd(redis_url, lua_file, output):
    r = redis.StrictRedis.from_url(redis_url)
    r.ping()

    for k, v in scan_iter(r, lua_file.read()):
        output.write(k.decode("utf-8"))
        output.write("\n")
        output.write("%d" % len(v))
        output.write("\n")


if __name__ == "__main__":
    cmd()

Run

$ pip install redis click

Sample data input

$ jq -n -r 'range(10000)|"set key\(.) value\(.)"' | redis-cli -n 8

Try to dump

$ ./export.py --redis-url redis://localhost:6379/8

end.

Recommended Posts

Dump the contents of redis db with lua
Try to get the contents of Word with Golang
Simulation of the contents of the wallet
Understand the contents of sklearn's pipeline
See the contents of Kumantic Segumantion
Process the contents of the file in order with a shell script
[python, ruby] fetch the contents of a web page with selenium-webdriver
The story of making a standard driver for db with python.
I checked the contents of docker volume
You can also check the communication of DB and cache with curl
Align the size of the colorbar with matplotlib
[Introduction to Python] How to sort the contents of a list efficiently with list sort
Setting to debug test by entering the contents of the library with pytest
The third night of the loop with for
Read all the contents of proc / [pid]
The second night of the loop with for
Count the number of characters with echo
Get the contents of git diff from python
Note: Prepare the environment of CmdStanPy with docker
2016 The University of Tokyo Mathematics Solved with Python
The contents of the Python tutorial (Chapter 2) are itemized.
[Note] Export the html of the site with python.
See the behavior of drunkenness with reinforcement learning
Increase the font size of the graph with matplotlib
The contents of the Python tutorial (Chapter 8) are itemized.
Calculate the total number of combinations with python
The contents of the Python tutorial (Chapter 1) are itemized.
Check the date of the flag duty with Python
The contents of the Python tutorial (Chapter 10) are itemized.
Eliminate the inconveniences of QDock Widget with PySide
Challenge the Tower of Hanoi with recursion + stack
Rewrite the name of the namespaced tag with lxml
Fill the browser with the width of Jupyter Notebook
Automating simple tasks with Python Table of contents
About the development contents of machine learning (Example)
Tucker decomposition of the hay process with HOOI
Find out the day of the week with datetime
The basis of graph theory with matplotlib animation
The contents of the Python tutorial (Chapter 6) are itemized.
Visualize the behavior of the sorting algorithm with matplotlib
Convert the character code of the file with Python3
The contents of the Python tutorial (Chapter 3) are itemized.
[Python] Determine the type of iris with SVM
Create a table of contents with IPython notebook
Contents of __name__
Extract the table of image files with OneDrive & Python
Add the attribute of the object of the class with the for statement
Coordinates of the right end of Label made with tkinter
The story of stopping the production service with the hostname command
Learn Nim with Python (from the beginning of the year).
Find the sum of unique values with pandas crosstab
The story of replacing Nvidia GTX 1650 with Linux Mint 20.1.
Add information to the bottom of the figure with Matplotlib
Get the sum of each of multiple columns with awk
The story of sharing the pyenv environment with multiple users
Destroy the intermediate expression of the sweep method with Python
Take a screenshot of the LCD with Python-LEGO Mindstorms
Visualize the range of interpolation and extrapolation with python
Calculate the regression coefficient of simple regression analysis with python
Predict the number of people infected with COVID-19 with Prophet
Log the value of SwitchBot thermo-hygrometer with Raspberry Pi