503 Service Unavailable - SSL Handshake Failure

API Monitoring

Procedure #1: Using API Monitoring

To diagnose the error using API Monitoring:

1. Sign in to Apigee Edge UI as a user with an appropriate role.

2. Switch to the organization in which you want to investigate the issue.

   
3. Navigate to the Analyze > API Monitoring > Investigate page.
4. Select the specific timeframe in which you observed the errors.

5. Plot Fault Code against Time.

6. Select a cell which has the fault code messaging.adaptors.http.flow.SslHandshakeFailed as shown below:

   ( view larger image)

   

7. Information about the fault code messaging.adaptors.http.flow.SslHandshakeFailed is displayed as shown below:

   ( view larger image)

   

8. Click View logs and expand the row for the failed request.

   ( view larger image)

   
9. From the Logs window, note the following details:
   • Request Message ID
   • Status Code: 503
   • Fault Source: target
   • Fault Code: messaging.adaptors.http.flow.SslHandshakeFailed

Trace

Procedure #2: Using Trace tool

To diagnose the error using the Trace tool:

1. Enable the trace session and either
   • Wait for the 503 Service Unavailable error with error code messaging.adaptors.http.flow.SslHandshakeFailed to occur, or
   • If you can reproduce the issue, make the API call to reproduce the issue 503 Service Unavailable

2. Ensure Show all FlowInfos is enabled:

   
3. Select one of the failing requests and examine the trace.
4. Navigate through different phases of the trace and locate where the failure occurred.

5. You will find the error typically after the phase Target Request Flow Started as shown below:

   ( view larger image)

   
6. Note the values of the following from the trace:
   • error: SSL Handshake failed sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
   • error.cause: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
   • error.class: com.apigee.errors.http.server.ServiceUnavailableException
   • The value of the error SSL Handshake failed sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target indicates that the SSL Handshake failed, as Apigee Edge’s Message Processor was unable to validate the backend server’s certificate.
7. Navigate to the AX (Analytics Data Recorded) Phase in the trace and click it.

8. Scroll down to the Phase Details Error Headers section and determine the values of X-Apigee-fault-code and X-Apigee-fault-source, and X-Apigee-Message-ID as shown below:

   ( view larger image)

   
9. Note the values of X-Apigee-fault-code, X-Apigee-fault-source, and X-Apigee-Message-ID:

Error headers	Value
X-Apigee-fault-code	messaging.adaptors.http.flow.SslHandshakeFailed
X-Apigee-fault-source	target
X-Apigee-Message-ID	MESSAGE_ID

NGINX

Procedure #3: Using NGINX access logs

To diagnose the error using NGINX access logs:

1. If you are a Private Cloud user, then you can use NGINX access logs to determine the key information about HTTP 503 Service Unavailable.

2. Check the NGINX access logs:

   /opt/apigee/var/log/edge-router/nginx/ORG~ENV.PORT#_access_log

3. Search to see if there are any 503 errors with error code messaging.adaptors.http.flow.SslHandshakeFailed during a specific duration (if the problem happened in the past) or if there are any requests still failing with 503.

4. If you do find any 503 errors with the X-Apigee-fault-code matching the value of messaging.adaptors.http.flow.SslHandshakeFailed, then determine the value of the X-Apigee-fault-source.

   Sample 503 error from NGINX access log:

   ( view larger image)

   

   The above sample entry from NGINX access log has the following values for X-Apigee-fault-code and X-Apigee-fault-source:

   Headers	Value
   X-Apigee-fault-code	messaging.adaptors.http.flow.SslHandshakeFailed
   X-Apigee-fault-source	target

Message Processor logs

Procedure #4: Using Message Processor Logs

1. Determine the message ID of one of the failing requests using API Monitoring, Trace tool, or NGINX Access Logs as explained in Common diagnosis steps.

2. Search for the specific request message ID in the Message Processor log (/opt/apigee/var/log/edge-message-processor/logs/system.log). You may observe the following error:

   org:myorg env:test api:MyProxy rev:1
   messageid:myorg-28247-3541813-1
   NIOThread@1 ERROR HTTP.CLIENT - HTTPClient$Context.handshakeFailed() :
   SSLClientChannel[Connected: Remote:X.X.X.X:443
   Local:192.168.194.140:55102]@64596 useCount=1
   bytesRead=0 bytesWritten=0 age=233ms  lastIO=233ms
   isOpen=true handshake failed, message: General SSLEngine problem
   

   The above error indicates that the SSL handshake failed between the Message Processor and the backend server.

   This will be followed by an exception with detailed stack trace as shown below:

   org:myorg env:test api:MyProxy rev:1
   messageid:myorg-28247-3541813-1
   NIOThread@1 ERROR ADAPTORS.HTTP.FLOW - RequestWriteListener.onException() :
   RequestWriteListener.onException(HTTPRequest@1522922c)
   javax.net.ssl.SSLHandshakeException: General SSLEngine problem
   	at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1478)
   	at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535)
   	... <snipped>
   Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
   	at sun.security.ssl.Alerts.getSSLException(Alerts.java:203)
   	at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
   	... <snipped>
   Caused by: sun.security.validator.ValidatorException: PKIX path building failed:
   sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid
   certification path to requested target
   	at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397)
   	at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
   	... <snipped>
     

   Note the handshake failure is due to:

   Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

   This indicates that the SSL Handshake failed as Apigee Edge’s Message Processor was unable to validate the backend server’s certificate.

Phase 1

Phase 1: Determine the backend server’s certificate chain

Use one of the following methods to determine the backend server’s certificate chain:

openssl s_client -connect BACKEND_SERVER_HOST_NAME:PORT#

Certificate chain
 0 s:/CN=mocktarget.apigee.net
   i:/C=US/O=Google Trust Services LLC/CN=GTS CA 1D4
 1 s:/C=US/O=Google Trust Services LLC/CN=GTS CA 1D4
   i:/C=US/O=Google Trust Services LLC/CN=GTS Root R1
 2 s:/C=US/O=Google Trust Services LLC/CN=GTS Root R1
   i:/C=US/O=Google Trust Services LLC/CN=GTS Root R1

Phase 2

Phase 2: Compare the backend server’s certificate and certificates stored in the Message Processor’s truststore

1. Determine the backend server’s certificate chain.
2. Determine the certificate stored in the Message Processor’s truststore using the following steps:

   1. Get the truststore reference name from the TrustStore element in the SSLInfo section in the TargetEndpoint.

      Let’s look at a sample SSLInfo section in a TargetEndpoint configuration:

      <TargetEndpoint name="default">
      ...
         <HTTPTargetConnection>
            <Properties />
            <SSLInfo>
               <Enabled>true</Enabled>
               <ClientAuthEnabled>true</ClientAuthEnabled>
               <KeyStore>ref://myKeystoreRef</KeyStore>
               <KeyAlias>myKey</KeyAlias>
               <TrustStore>
                  ref://myCompanyTrustStoreRef
               </TrustStore>
            </SSLInfo>
         </HTTPTargetConnection>
         ...
      </TargetEndpoint>

   2. In the above example, the TrustStore reference name is myCompanyTruststoreRef.

   3. In the Edge UI, select Environments > References. Note the name in the Reference column for the specific truststore reference. This will be your truststore name.

      ( view larger image)

      

   4. In the above example the truststore name is:

      myCompanyTruststoreRef: myCompanyTruststore

3. Get the certificates stored in the truststore (determined in the previous step) using the following APIs:

   1. Get all certificates for a keystore or truststore. This API lists all the certificates in the specific truststore.

      Public Cloud user:

      curl -v -X GET https//api.enterprise.apigee.com/v1/organizations/ORGANIZATION_NAME/environments/ENVIRONMENT_NAME/keystores/KEYSTORE_NAME/certs -H "Authorization: Bearer $TOKEN"
      

      Private Cloud user:

      curl -v -X GET http://MANAGEMENT_HOST:PORT_#/v1/organizations/ORGANIZATION_NAME/environments/ENVIRONMENT_NAME/keystores/KEYSTORE_NAME/certs -H "Authorization: Bearer $TOKEN"
      

      Where:

      • ORGANIZATION_NAME is the name of the organization
      • ENVIRONMENT_NAME is the name of the environment
      • KEYSTORE_NAME is the name of the keystore
      • $TOKEN is set to your OAuth 2.0 access token as described in Obtain an OAuth 2.0 access token
      • curl options used in this example are described in Use curl

      Sample output:

      The certificates from the example truststore myCompanyTruststore are:

      [
        "serverCert"
      ]

   2. Get Cert Details for the specific certificate from a Keystore or Truststore. This API returns the information about a specific certificate in the specific truststore.

      Public Cloud user:

      curl -v -X GET https//api.enterprise.apigee.com/v1/organizations/ORGANIZATION_NAME/environments/ENVIRONMENT_NAME/keystores/KEYSTORE_NAME/certs/CERT_NAME -H "Authorization: Bearer $TOKEN"
      

      Private Cloud user

      curl -v -X GET http://MANAGEMENT_HOST:PORT_#>/v1/organizations/ORGANIZATION_NAME/environments/ENVIRONMENT_NAME/keystores/KEYSTORE_NAME/certs/CERT_NAME -H "Authorization: Bearer $TOKEN"
      

      Where:

      • ORGANIZATION_NAME is the name of the organization
      • ENVIRONMENT_NAME is the name of the environment
      • KEYSTORE_NAME is the name of the keystore
      • CERT_NAME is the name of the certificate
      • $TOKEN is set to your OAuth 2.0 access token as described in Obtain an OAuth 2.0 access token
      • curl options used in this example are described in Use curl

      Sample Output

      Details of serverCert shows the subject and issuer as follows:

      Leaf/Entity Certificate:

      "subject": "CN=mocktarget.apigee.net",
      "issuer": "CN=GTS CA 1D4, O=Google Trust Services LLC, C=US",

      Intermediate Certificate:

      "subject" : "CN=GTS CA 1D4, O=Google Trust Services LLC, C=US",
      "issuer" : "CN=GTS Root R1, O=Google Trust Services LLC, C=US",

4. Verify that the actual server certificate obtained in step 1 and the certificate stored in truststore obtained in step 3 match. If they do not match, then that’s the cause for the issue.

   From the example shown above, let’s look at one certificate at a time:

   1. Leaf certificate:

      From the backend server:

      s:/CN=mocktarget.apigee.net
      i:/C=US/O=Google Trust Services LLC/CN=GTS CA 1D4

      From Message Processor’s (client) truststore:

      "subject": "CN=mocktarget.apigee.net",
      "issuer": "CN=GTS CA 1D4, O=Google Trust Services LLC, C=US",

      The leaf certificate stored in the truststore matches that of the backend server.

   2. Intermediate certificate:

      From the backend server:

      s:/C=US/O=Google Trust Services LLC/CN=GTS CA 1D4
      i:/C=US/O=Google Trust Services LLC/CN=GTS Root R1

      From Message Processor’s (client) truststore:

      "subject" : "CN=GTS CA 1D4, O=Google Trust Services LLC, C=US",
      "issuer" : "CN=GTS Root R1, O=Google Trust Services LLC, C=US",

      The intermediate certificate stored in the truststore matches that of the backend server.

   3. Root certificate:

      From the backend server:

      s:/C=US/O=Google Trust Services LLC/CN=GTS Root R1
      i:/C=US/O=Google Trust Services LLC/CN=GTS Root R1

      The root certificate is completely missing in the Message Processor’s truststore.

   4. Since the root certificate is missing in the truststore, the Message Processor throws the following exception:

      sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

      and returns 503 Service Unavailable with the error code messaging.adaptors.http.flow.SslHandshakeFailed to the client applications.

Correct FQDN

Update backend server’s keystore with correct FQDN, valid and complete certificate chain:

1. If you don’t have a backend server’s certificate with the correct FQDN, then procure the proper certificate from an appropriate CA (Certificate Authority).

2. Validate that you have a valid and complete backend server’s certificate chain.

3. Once you have the valid and complete certificate chain with the correct FQDN of the backend server in the leaf or entity certificate that is identical to the host name specified in the target endpoint, update the backend’s keystore with the complete certificate chain.

Correct backend server

Update the target endpoint with the correct backend server’s host name:

1. If the host name was incorrectly specified in the target endpoint, then update the target endpoint to have the correct host name that matches the FQDN in the backend server’s certificate.

2. Save the changes to the API proxy.

   In the example discussed above, if the backend server host name was incorrectly specified, then you can fix it by using the FQDN from the backend server’s certificate, that is backend.apigee.net as follows:

   <TargetEndpoint name="default">
      …
      <HTTPTargetConnection>
         <Properties />
         <SSLInfo>
            <Enabled>true</Enabled>
            <TrustStore>ref://myTrustStoreRef</TrustStore>
         </SSLInfo>
         <URL>https://backend.apigee.net/resource</URL>
      </HTTPTargetConnection>
   </TargetEndpoint>