Python Redis Pub Sub

Intro

Redis provides a Pub/Sub api that scales well and allows for quick real time connections. Real time apps are very popular, so redis can help you solve these business problems. If you are familiar with the popular socket.io, you will see in their docs an option to use Redis as an adapter to scale. In this article, we will learn how to use PubSub in Redis with Python.

Use Cases

• Sending DB updates to the client
• Displaying real time edits (like Google Docs)
• Real time chat

Just to give a more clear view, let's say you have the following setup with some microservices because you have scaled to many users

• One Server to write to the Database
• Two servers running web apis (like Express.js).
  • These severs both have socket.io to connect to a Front End (like React or Vue)
• A static website hosted on Netlfiy or AWS with just your React or Vue code
  • This website has a socket appliation to talk to Express

In this scenario, you can have a number of users connect to either API server app which allows you to handle more connections than a single server. Then, you can have redis pub/sub send quick communication between the three servers, the API servers and the DB. This allows you to quickly add more backend servers to scale while keeping connections to the client.

Setting up Redis

For setting up Redis, I would recommend using a service for you in prod. Azure for example, has a great redis service that scales easily. However, you will want to learn redis and eventually how to scale it yourself. This will help with debugging cloud services or eventually, saving money and not using them.

We will start our intro to redis via using docker compose. Create a docker-compose.yml file and add the following.

version: "3.2"
services:
  redis:
    image: "redis:alpine"
    command: redis-server
    ports:
      - "6379:6379"
    volumes:
      - $PWD/redis-data:/var/lib/redis
      - $PWD/redis.conf:/usr/local/etc/redis/redis.conf
    environment:
      - REDIS_REPLICATION_MODE=master

Ensure you have docker installed and run

docker-compose up

Installing Redis Modules

In python, the main used redis module is called redis-py and can be installed using the follows.

pip install redis

Creating the Subscriber

Let's start be creating a subscriber to list to messages. To start, we will need to import redis and connect to our server. We will also need the time library for an infinite loop later.

# subscriber.py
import redis
import time

r = redis.Redis(host='localhost', port=6379, db=0)

Now, let's create an instance of PubSub from the redis library and use the subscribe method to subscribe to two different channels.

p = r.pubsub()
p.subscribe("my-channel-1", "my-channel-2")

Finally, we will create an infinite loop and use the get_message command. If we receive a message, we will print it out.

while True:
  message = p.get_message()
  if message:
    print(message)
  time.sleep(0.01)

Open a terminal and run python subscriber.py to start the listener.

Here is the full file for context.

# subscriber.py
import redis
import time

r = redis.Redis(host='localhost', port=6379, db=0)

p = r.pubsub()

p.subscribe("my-channel-1", "my-channel-2")

while True:
  message = p.get_message()
  if message:
    print(message)
  time.sleep(0.01)

Creating the Publisher

The publisher is much simpler. We simply connect to our redis instance, then use the publish method to send a message.

# publisher.py
import redis

r = redis.Redis(host='localhost', port=6379, db=0)

r.publish('my-channel-1', 'my data')

Open a new terminal tap and run python publisher.py. This will send a message to our other python script.

Pattern Matching Publish and Subscribe

Redis also provides a way to pattern match when using the publish and subsribe feature. Let's say we had the following publisher sending events for multiple users.

import redis

r = redis.Redis(host='localhost', port=6379, db=0)

r.publish('user-1', 'my data')
r.publish('user-2', 'my data')

We can subscribed to all user events using the psubscribe method which stands for pattern subsrcibe. Here we subscribe to user-* which means all channels with the prefix user-

# subscriber.py
import redis
import time

r = redis.Redis(host='localhost', port=6379, db=0)

p = r.pubsub()

p.subscribe("user-*")

while True:
  message = p.get_message()
  if message:
    print(message)
  time.sleep(0.01)