phpredis Retry

Overview

phpredis is a common SDK for accessing Redis using a PHP script. Phpredis is only capable of basic connection and interaction. In scenarios where packets may be lost in a complex cloud network, or Redis master/standby switchover can be triggered when hardware is faulty, automatic reconnection and retry are unavailable. This document provides a practice for reliability reconstruction of a PHP script.

Example default phpredis connection

Function connect of phpredis is used to establish a persistent connection to Redis. A key value is increased every 2s in a loop. Neither this example nor function connect uses a reconnection or retry mechanism. When master/standby switchover occurs, the program enters the abnormal state and exits.

#!/usr/bin/env php
<?php
try {
    $redis = new Redis();
    // Redis: 127.0.0.1:6379, timeout 2s, interval 100 ms
    if (!$redis->connect("127.0.0.1", 6379, 2, null, 100)) {
        throw new Exception("Redis server cannot be connected.");
    }

    // Delete the existing keys (counting from 0).
    $redis->del("redisExt");

    echo "Start the increase counter. [press Ctrl + C to exit]\n";

    while (true) {
        // Execute the increase and obtain the new value.
        $newValue = $redis->incr("redisExt");

        // Print the current value and time.
        printf("[%s] redisExt = %d\n", 
               date('Y-m-d H:i:s'), 
               $newValue);

        // Wait for 2s.
        usleep(2000000);
    }
} catch (Exception $e) {
    // Error handling
    echo "Error occurred:\n";
    var_dump($e);
    exit(1);
}

Solution

By default, phpredis does not automatically reconnect and retry. But they can be configured.

Common phpredis reconnection implementation:

1. Use pconnect for persistent connection. However, this cannot avoid disconnection (such as Redis server restart and network fluctuation).
2. Capture connection exceptions and attempt to reconnect and retry.

phpredis provides the following parameters to implement reconnection and retry. For details, see Table 1.

$redis->connect(host, port, timeout, reserved, retry_interval, read_timeout, others);

Proper algorithms and retry intervals are required to prevent retry storms caused by specific batch disconnections, and support quick retry in some disconnections. phpredis implements a backoff algorithm through parameter OPT_BACKOFF_ALGORITHM. Set it to Redis::BACKOFF_ALGORITHM_DECORRELATED_JITTER in most scenarios to randomize the backoff delay.

The optimized example is as follows. When Redis exceptions such as network faults and master/standby switchovers occur, the program can still retry operations, and output clear error information and exit safely during a fault.

#!/usr/bin/env php
<?php
try {
    $redis = new Redis();

    // Configure the Redis connection.
    $redis->setOption(Redis::OPT_MAX_RETRIES, 3);       // Set the maximum retries.
    $redis->setOption(Redis::OPT_BACKOFF_ALGORITHM, Redis::BACKOFF_ALGORITHM_DECORRELATED_JITTER);          // Set the backoff algorithm.
    $redis->setOption(Redis::OPT_BACKOFF_BASE, 100);    // Basic delay 100 ms.
    $redis->setOption(Redis::OPT_BACKOFF_CAP, 2000);    // Maximum delay 2,000 ms.

    // Redis: 127.0.0.1:6379, timeout 2s, automatic retry interval 100 ms
    if (!$redis->connect("127.0.0.1", 6379, 2, '', 100)) {
        throw new Exception("Redis server cannot be connected.");
    }

    // Delete the existing keys (counting from 0).
    $redis->del("redisExt");

    echo "Start the increase counter. [press Ctrl + C to exit]\n";

    $lastSuccess = time();
    $consecutiveFails = 0;

    while (true) {
        try {
            // Execute the increase and obtain the new value.
            $newValue = $redis->incr("redisExt");

            // Reset the failure count.
            $consecutiveFails = 0;
            $lastSuccess = time();

            // Print the current value and time.
            printf("[%s] redisExt = %d\n", 
                   date('Y-m-d H:i:s'), 
                   $newValue);

            // Normally wait for 2s.
            usleep(2000000);

        } catch (RedisException $e) {
            // Process consecutively failed.
            $consecutiveFails++;

            // Print error information.
            printf("[%s] error: %s (consecutive failures %d )\n", 
                   date('Y-m-d H:i:s'), 
                   $e->getMessage(), 
                   $consecutiveFails);

            // Backoff wait: 100 ms, 200 ms, 400 ms, 800 ms ...
            $waitTime = min(100 * pow(2, $consecutiveFails), 5000); // Max. 5s
            usleep($waitTime * 1000);

            // Consecutive failures exceed the threshold. Attempt to completely reconnect.
            if ($consecutiveFails >= 5) {
                echo "Completely reconnecting...\n";
                try {
                    $redis->close();
                    $redis->connect("127.0.0.1", 6379, 2);
                    $consecutiveFails = 0; // Reset the failure count.
                    echo "Reconnected\n";
                } catch (Exception $reconnectEx) {
                    echo "Reconnect failed: " . $reconnectEx->getMessage() . "\n";
                }
            }

            // Forced exit after no successful operation for long.
            if ((time() - $lastSuccess) > 60) {
                throw new Exception("No successful operation for 60s. Script terminated.");
            }
        }
    }
} catch (Exception $e) {
    // Error handling
    echo "\n Error occurred:\n";
    echo "[" . date('Y-m-d H:i:s') . "] " . $e->getMessage() . "\n";

    // Try recording the final counter value.
    try {
        if (isset($redis) && $redis->isConnected()) {
            $finalValue = $redis->get("redisExt");
            echo "Final counter value: " . ($finalValue ?: 'N/A') . "\n";
        }
    } catch (Exception $finalEx) {
        echo "Obtain the final value failed: " . $finalEx->getMessage() . "\n";
    }

    exit(1);
}