Encryption Parsing Failure

Symptom

The out-of-path TLS traffic fails to be parsed.

Possible Causes

• The SSL private key uploaded to the database server is incorrect.
• The private key algorithm and format are incorrect.
• The protocol and suite settings are incorrect.
• The two-way handshake traffic captured during connection establishment is incomplete.

Solutions

First, check for errors other than decryption failures. For example, check for incorrect database configuration, incorrect agent configuration, network disconnection, incorrect security group configuration that blocks connection, and incorrect port configuration that blocks connection.

If these configurations are correct, check the following items one by one:

1. Upload the SSL private key of the database server, not the certificate file.
2. Private key algorithm and format: The private key must be in PEM format, password-free, and use the RSA algorithm. ECC/ECDSA certificates are not supported. Ensure that the private key strictly matches the current database certificate.

   You can use openssl x509 -noout -modulus -in server.crt | openssl md5 and openssl rsa -noout -modulus -in server.key | openssl md5 to compare the modulus hashes. Their values should be the same.

   Good example:

   -----BEGIN RSA PRIVATE KEY-----
   ...
   -----END RSA PRIVATE KEY-----

   Bad example: An encryption key is specified.

   -----BEGIN RSA PRIVATE KEY-----
   Proc-Type: 4,ENCRYPTED
   DEK-Info: AES-256-CBC,...
   ...
   -----END RSA PRIVATE KEY-----

   Other errors:

   -----BEGIN CERTIFICATE-----
   ...
   -----END CERTIFICATE-----

3. Protocol and suite configuration: TLS 1.3 and ECDHE/DHE suites (forward secrecy) must be disabled. Only TLS_RSA_* suites (for example, AES128-SHA) can be configured. Other suites will prevent the session key from being derived using a static private key.
4. If ssl = on, ssl_min_protocol_version = 'TLSv1', and ssl_max_protocol_version = 'TLSv1.2' (TLS 1.3 is disabled) are configured in the postgresql.conf file of PostgreSQL, ssl_ciphers must contain only static RSA key exchange suites. You can specify one algorithm (for example, ssl_ciphers = 'AES256-SHA') or multiple algorithms (for example, ssl_ciphers = 'AES256-SHA:AES128-SHA:AES128-SHA256:AES256-SHA256'). (ECDHE and DHE algorithms must not be included.)

   GCM and RC4 are not recommended. As a synchronous stream cipher, RC4 cannot recover from packet or byte loss, which may result in subsequent parsing failures. In out-of-path environments, GCM is susceptible to counter desynchronization caused by TCP retransmissions or empty packets, which may also result in parsing failures.

   An example of the configuration in the postgresql.conf file is as follows:

   # Enable SSL.
   ssl = on
   # 1. Specify a cipher suite.
   ssl_ciphers = 'AES256-SHA:AES128-SHA:AES128-SHA256:AES256-SHA256'
   # 2. Specify a protocol version range.
   ssl_min_protocol_version = 'TLSv1'
   ssl_max_protocol_version = 'TLSv1.2'
   # 3. Specify the certificate and private key. (Ensure the permission is 600.)
   ssl_cert_file = 'server.crt'
   ssl_key_file = 'server.key'
   ssl_ca_file = 'cacert.pem'

5. Handshake traffic capture: Decryption depends on the Client Key Exchange message in the handshake packet. The complete two-way handshake traffic during connection establishment must be captured. If the connection has been established before packet capture or key upload, or the handshake packet is lost, decryption cannot be performed.