Redis Pipeline in Python

Intro

Redis offers a feature called pipeline that allows you to bulk send commands. This can drastically improved performance if you are running queries that can be batched together. The reason for this is that you only need to traverse the network to redis once, then Redis will run all the commands and return the result. In this article, we will learn how to use redis pipelining with Python.

Setting up Redis

For setting up Redis, I would recommend using a service for you in prod. Azure for example, has a great redis service that scales easily. However, you will want to learn redis and eventually how to scale it yourself. This will help with debugging cloud services or eventually, saving money and not using them.

We will start our intro to redis via using docker compose. Create a docker-compose.yml file and add the following.

version: "3.2"
services:
  redis:
    image: "redis:alpine"
    command: redis-server
    ports:
      - "6379:6379"
    volumes:
      - $PWD/redis-data:/var/lib/redis
      - $PWD/redis.conf:/usr/local/etc/redis/redis.conf
    environment:
      - REDIS_REPLICATION_MODE=master

Ensure you have docker installed and run

docker-compose up

Installing Redis Modules

In python, the main used redis module is called redis-py and can be installed using the follows.

pip install redis

Writing the Code

Let's open up a new file, index.py and go through many of the common commands you will used with lists in redis.

pipeline = r.pipeline()

Next, we can append our commands to this pipeline. For example, here are a series of counter operations. Each of these commands will be appeneded to our pipeline and not sent.

pipeline.set("count-user1", 1)
pipeline.del("count-user1-old-day")
pipeline.get("count-user1")
pipeline.incr("count-user1")
pipeline.get("count-user1")

Finally, to send all commands, we can use the exec function on the pipeline. Redis will execute all the commands and returns the results in an array.

pipeline.execute()
print(results)

// [
//   [ null, 'OK' ],
//   [ null, 0 ],
//   [ null, '1' ],
//   [ null, 2 ],
//   [ null, '2' ]
// ]

Below is the full code for context.

import redis

r = redis.Redis(host='localhost', port=6379, db=0)

pipeline = r.pipeline()

pipeline.set("count-user1", 1)
pipeline.del("count-user1-old-day")
pipeline.get("count-user1")
pipeline.incr("count-user1")
pipeline.get("count-user1")

results = pipeline.execute()
print(results)