Configuring a Redis Pipeline

Overview

DCS supports Redis pipelining. This technique sends multiple commands to the Redis server at once, reducing network latency and improving performance.

Without a pipeline, a client sends a command to Redis, waits for the server to return the result, then sends the next command, and so on. With a pipeline, a client sends commands to Redis without waiting for the results. After all commands are sent, the client closes the request, starts to receive responses, and matches them to the commands in sequence.

Figure 1 Comparing the network communication with and without a pipeline

Generally in pipelining, Redis clients send commands in batches, receive all results, and then return them to upper-layer services. This mechanism reduces the network round-trip time (RTT), system calls of read() and write(), and process switchovers, and improves program efficiency and performance.

Use pipelining to perform Redis operations in batches for better performance, if your services does not need to obtain the result of each operation immediately.

• Pipelines exclusively use the client-server connection. Other operations cannot be performed until the pipelines are closed. To perform other operations at the same time, set up a connection dedicated to pipelines.
• For more information, see Redis pipelining.

Notes and Constraints

• Pipelining cannot ensure atomicity.

  Pipelining sends client commands in batches. The server parses each command and executes them one by one in sequence. During this process, the server may execute commands from other clients. For atomicity, use transactions or Lua scripts.

• Pipelining does not support rollback if an error occurs.
• Pipelining does not feature transactions. Do not use it if commands depend on each other.

  Some clients, such as redis-py, wrap pipelines in transactional commands MULTI and EXEC. Note the differences between pipelines and transactions. For restrictions on transactions, see Redis transactions.

Comparing Performance

The following code compares the performance with and without a pipeline.

public static void main(String[] args) {
    // Set the Redis instance connection address and port.
    Jedis jedis = new Jedis("127.0.0.1", 6379);
 
    // Run commands consecutively.
    final int COUNT=5000;
    String key = "key";
    
    // 1 ---No pipelines are used.---
    jedis.del(key);   // Initialize the key.
    long t1 = System.currentTimeMillis();
    for (int i = 0; i < COUNT; i++) {
        // Send a request and receive a response.
        jedis.incr(key);
    }
    long t2 = System.currentTimeMillis();
    System.out.println("No Pipeline > value:"+jedis.get(key)+" > Duration:" + (t2 - t1) + "ms");

    // 2 ----A pipeline is used.---
    jedis.del(key);   // Initialize the key.
    Pipeline p1 = jedis.pipelined();
    long t3 = System.currentTimeMillis();
    for (int i = 0; i < COUNT; i++) {
        // Send a request.
        p1.incr(key);
    }
    // Receive a response.
    p1.sync();
    long t4 = System.currentTimeMillis();
 
    System.out.println("Pipeline used > value:"+jedis.get(key)+" > Duration:" + (t4 - t3)+ "ms");
    jedis.close();}

The result shows that the performance is better when a pipeline is used.

No pipeline > value:5000 > Duration:1204ms
Pipeline used > value:5000 > Duration:9ms

Processing the Response Data

The following code shows two methods of processing the response data when a pipeline is used in Jedis.

public static void main(String[] args) {
    // Set the Redis instance connection address and port.
    Jedis jedis = new Jedis("127.0.0.1", 6379);
    String key = "key";
    jedis.del(key);  // Initialize

    // -------- Method 1 --------
    Pipeline p1 = jedis.pipelined();
    System.out.println("-----Method 1-----");
    for (int i = 0; i < 5; i++) {
        p1.incr(key);
        System.out.println("Pipeline sends a request.");
    }
    // The request is sent. Start receiving the response.
    System.out.println("The request is sent. Start receiving the response.");
    List<Object> responses = p1.syncAndReturnAll();
    if (responses == null || responses.isEmpty()) {
        jedis.close();
        throw new RuntimeException("Pipeline error: no response.");
    }
    for (Object resp : responses) {
        System.out.println("Pipeline receives a response: " + resp.toString());
    }
    System.out.println();

    // -------- Method 2 --------
    System.out.println("-----Method 2-----");
    jedis.del(key);   // Initialize
    Pipeline p2 = jedis.pipelined();

    // Declare a response.
    Response<Long> r1 = p2.incr(key);
    System.out.println("Pipeline sends a request.");
    Response<Long> r2 = p2.incr(key);
    System.out.println("Pipeline sends a request.");
    Response<Long> r3 = p2.incr(key);
    System.out.println("Pipeline sends a request.");
    Response<Long> r4 = p2.incr(key);
    System.out.println("Pipeline sends a request.");
    Response<Long> r5 = p2.incr(key);
    System.out.println("Pipeline sends a request.");
    try {
        r1.get();   // An exception is thrown because the response is still pending.
    } catch (Exception e) {
        System.out.println(" <<< Pipeline error: Receiving the response has not started yet.  >>> ");
    }
    // The request is sent and responses start to be received.
    System.out.println("The request is sent. Start receiving the response.");
    p2.sync();
    System.out.println("Pipeline receives the response: " + r1.get());
    System.out.println("Pipeline receives the response: " + r2.get());
    System.out.println("Pipeline receives the response: " + r3.get());
    System.out.println("Pipeline receives the response: " + r4.get());
    System.out.println("Pipeline receives the response: " + r5.get());
    jedis.close();}

Result:

-----Method 1-----
Pipeline sends a request.
Pipeline sends a request.
Pipeline sends a request.
Pipeline sends a request.
Pipeline sends a request.
The request is sent. Start receiving the response.
Pipeline receives the response: 1
Pipeline receives the response: 2
Pipeline receives the response: 3
Pipeline receives the response: 4
Pipeline receives the response: 5
-----Method 2-----
Pipeline sends a request.
Pipeline sends a request.
Pipeline sends a request.
Pipeline sends a request.
Pipeline sends a request.
 <<< Pipeline error: Receiving the response has not started yet.  >>> 
The request is sent. Start receiving the response.
Pipeline receives the response: 1
Pipeline receives the response: 2
Pipeline receives the response: 3
Pipeline receives the response: 4
Pipeline receives the response: 5