400 Bad Request - SSL Certificate Error

You're viewing Apigee Edge documentation.
Go to the Apigee X documentation. info

Symptom

The client application receives an HTTP 400 - Bad request response with the message "The SSL certificate error". This error is typically sent by the Edge Router in a two way TLS setup enabled for the incoming connection to Apigee Edge.

Error Message

The Client application gets the following response code:

HTTP/1.1 400 Bad Request

Followed by the below HTML error page:

<html>
  <head>
    <title>400 The SSL certificate error</title>
  </head>
  <body bgcolor="white">
    <center> <h1>400 Bad Request</h1>
    </center>
    <center>The SSL certificate error</center>
    <hr>
    <center>nginx</center>
  </body>
</html>

Possible Causes

The possible causes for this issue are as follows:

Cause	Description	Troubleshooting Instructions Applicable For
Expired client certificate	The certificate sent by the client is expired.	Edge Private and Public Cloud users
Incorrect Certificate sent by Client	This error is thrown if the certificate sent by the client application does not match with the certificate stored in the truststore of Edge's Router.	Edge Private and Public Cloud users
Missing Client Root Cert in Truststore	This error is thrown if the client’s CA signed root certificate is missing in the truststore of Edge's router.	Edge Private and Public Cloud users
Client Certificates not loaded in the Edge Router	This error is thrown if the client certificates uploaded to the truststore are not loaded on the Router.	Edge Private Cloud users

Cause: Expired Client Certificate

This issue typically happens for a 2-Way TLS, when the certificate sent by the client is expired. In a 2-way TLS, both client and server exchange their public certificates to accomplish the handshake. The client validates the server certificate and the server validates the client certificate.

In Edge, 2-way TLS is implemented at virtual host, where the server certificate is added to the Keystore and the client certificate is added to truststores.

During the TLS handshake if it is found that the client certificate is expired, then the server will send 400 - Bad request with the message "The SSL certificate error".

Diagnosis

Log in to the Edge UI and view the specific Virtual Host configuration (Admin > Virtual Hosts) for which the API request is being made, or use Get virtual host API management API to get the definition of the specific Virtual Host.

Typically a virtual host for two-way TLS communication looks as follows:

<VirtualHost name="myTLSVHost">
    <HostAliases>
        <HostAlias>api.myCompany.com</HostAlias>
    </HostAliases>
    <Port>443</Port>
    <SSLInfo>
        <Enabled>true</Enabled>
        <ClientAuthEnabled>true</ClientAuthEnabled>
        <KeyStore>ref://myKeystoreRef</KeyStore>
        <KeyAlias>myKeyAlias</KeyAlias>
        <TrustStore>ref://myTruststoreRef</TrustStore>
    </SSLInfo>
</VirtualHost>

Determine the Truststore reference used in the Virtual Host. In the above example, the Truststore reference name is myTruststoreRef.
Determine the Truststore pointed to by the Truststore reference.
1. In the Edge UI navigate to Admin > Environments > References and search for the Truststore reference name.
2. Note the name in the Reference column for the specific Truststore reference. This will be your Truststore name.
  
  Figure 1
  
  In the above example, notice that myTruststoreRef has the reference to myTruststore. Therefore, the Truststore name is myTruststore.
In Admin > Environments > TLS Keystores in the Edge UI, navigate to TLS Keystores and look for the Truststore found in step # 3.
Select the certificate under the specific Truststore (determined in step #3 above) as shown below:

Figure 2

The certificate with alias client-cert-markw in the above example, shows that it's expired.
Check if the certificate is expired for the certificate alias for your truststore.
If the certificate is not expired then move to Common Diagnosis Steps for the other Causes.

Resolution

Procure a new certificate and upload the certificate:

Create a new truststore, for example myNewTruststore.
Upload the new certificate to the newly created truststore.
Modify the trustore reference used in the specific Virtual Host to point to the new truststore using the steps given in Modifying a reference.

In the example described above, point the reference myTruststoreRef to myNewTruststore.

Common Diagnosis Steps for the Other Causes

To investigate this issue, you will need to capture TCP/IP packets using the tcpdump tool.
1. If you are a Private Cloud user, then you can capture the TCP/IP packets on the client application or Router.
2. If you are a Public Cloud user, then capture the TCP/IP packets on the client application.
3. Once you've decided where you would like to capture TCP/IP packets, use the following tcpdump command to capture TCP/IP packets:
```
tcpdump -i any -s 0 host <IP address> -w <File name>
```
  Note: If you are taking the TCP/IP packets on the Router, then use the public IP address of the client application in the tcpdump command.
  
  If you are taking the TCP/IP packets on the client application, then use the public IP address of the host name used in the Virtual Host in the tcpdump command.
  
  Refer to tcpdump for more information about this tool and for other variants of this command.
Analyze the TCP/IP packets collected using the Wireshark tool or similar tool with which you are familiar.

Here's the analysis of sample TCP/IP packets data using Wireshark tool:

The Packet #30 in the tcpdump (image below) shows that the Client Application (source) sent a "Client Hello" message to the Router (destination).
Packet #34 shows that the Router acknowledges the Client Hello message from the client application.
The Router sends the "Server Hello" in packet #35 and then sends its certificate and also requests the client application to send its certificate in packet #38.
In the packet #38, where Router sends "Certificate Request" packet, check the "Distinguished Names" section which provides details about the client certificate, its chain and certificate authorities that are accepted by the Router (server).

The client application sends its certificate in Packet # 41. Check the Certificate Verify section in packet # 41 and determine the certificate that is sent by the client application.

Figure 4
Verify if the subject and the issuer of the certificate and its chain sent by the client application (packet #41) matches with the accepted certificate and its chain from the Router (packet #38). If there's a mismatch then that's the cause for this error. Hence the Router (Server) sends the Encrypted Alert (packet #57) followed by FIN, ACK (packet 58) to the Client Application and eventually the connection is terminated.
The mismatch of the certificate and its chain can be caused by the scenarios described in the following sections.

Cause: Incorrect certificate sent by client

This typically happens if the subject/issuer of either the certificate and/or its chain sent by the client application does not match the certificate and/or its chain stored in the truststore of the Router (Server).

Diagnosis

Sign in to the Edge UI and view the specific Virtual Host configuration (Admin > Virtual Hosts) for which the API request is being made, or use the Get virtual host API management API to get the definition of the specific Virtual Host.

Typically a virtual host for two-way TLS communication looks as follows:

    <VirtualHost name="myTLSVHost">
        <HostAliases>
            <HostAlias>api.myCompany.com</HostAlias>
        </HostAliases>
        <Port>443</Port>
        <SSLInfo>
            <Enabled>true</Enabled>
            <ClientAuthEnabled>true</ClientAuthEnabled>
            <KeyStore>ref://myKeystoreRef</KeyStore>
            <KeyAlias>myKeyAlias</KeyAlias>
                <TrustStore>ref://myCompanyTruststoreRef</TrustStore>
        </SSLInfo>
    </VirtualHost>

Determine the Truststore reference used in the Virtual Host.
In the above example, the Truststore reference name is myCompanyTruststoreRef.
Determine the Truststore pointed by the Truststore reference.
1. In the Edge UI navigate to Admin > Environments References and search for the Truststore reference name.
2. Note the name in the Reference column for the specific Truststore reference. This will be your Truststore name.
  
  Figure 5
  
  In the above example, notice that myCompanyTruststoreRef has the reference to myCompanyTruststore. Therefore, the Truststore name is myCompanyTruststore.
Get the certificates stored in the Truststore (determined in the previous step) using the following APIs:
1. List certificates for a keystore or truststore API.
  
  This API lists all the certificates in the specific Truststore.
2. Get cert details from a keystore or truststore API.
  
  This API returns information about a specific certificate in the specific Truststore.
Check if the issuer and subject of each of certificate and its chain stored in myCompanyTruststore matches with that of the certificate and its chain as seen in the TCP/IP Packets (refer packet #38) above. If there's a mismatch then it indicates that the certificates uploaded to the truststore are not being loaded in the Edge Router. Move to Cause: Client Certificates not loaded in the Edge Router.
If there was no mismatch found in Step #5, then that indicates that the client application did not send the right Certificate and its chain.

Resolution

Ensure that the correct certificate and its chain are sent by the client application to Edge.

Cause: Missing Client Root Certificate in Truststore

This error is thrown if the client’s CA signed root certificate is missing in the truststore of Edge's router.

Diagnosis

Sign in to the Edge UI and view the specific virtual host configuration for which the API request is being made (Admin > Virtual Hosts > virtual_host), or use the Get virtual host API to get the definition of the specific virtual host.

Typically a virtual host for two-way TLS communication looks as follows:

    <VirtualHost name="myTLSVHost">
        <HostAliases>
            <HostAlias>api.myCompany.com</HostAlias>
        </HostAliases>
        <Port>443</Port>
        <SSLInfo>
            <Enabled>true</Enabled>
            <ClientAuthEnabled>true</ClientAuthEnabled>
            <KeyStore>ref://myKeystoreRef</KeyStore>
            <KeyAlias>myKeyAlias</KeyAlias>
            <TrustStore>ref://myCompanyTruststoreRef</TrustStore>
        </SSLInfo>
    </VirtualHost>

Determine the truststore reference used in the virtual host. In the previous example, the truststore reference name is myCompanyTruststoreRef.
Determine the actual truststore being used by the truststore reference.
In the Edge UI, navigate to Admin > Environments > References and search for the truststore reference name.
The truststore name for the specific truststore reference is in the Reference column.

Figure 6

In this example, notice that myCompanyTruststoreRef has myCompanyTruststore in the Reference column. Therefore, the truststore name is myCompanyTruststore.
Get the certificates stored in the truststore (determined in the previous step) using the following APIs:
1. List certificates for a keystore or truststore API. This API lists all the certificates in the truststore.
2. Get cert details from a keystore or truststore API. This API returns information about a specific certificate in the truststore.
Check to see if the certificate includes a complete chain, including the root certificate sent by the specific client as seen in the TCP/IP Packets (see Figure 4). The truststore must include the root certificate as well as the client’s leaf certificate or leaf and intermediate certificate. If the client’s valid root certificate is missing in the truststore, that’s the cause of the error.

However, if the client’s complete certificate chain, including the root certificate, exists in the truststore, then it indicates that the certificates uploaded to the truststore are possibly not loaded in the Edge Router. If that's the case, see Cause: Client Certificates not loaded in the Edge Router.

Resolution

Ensure that the correct client’s certificate, including the root certificate, is available in the truststore of the Apigee Edge router.

Cause: Client Certificates not loaded in the Edge Router

If you are a Public Cloud user, contact Apigee Edge Support.
If you are a Private Cloud user, follow the below instructions on each Router:
1. Check if the file /opt/nginx/conf.d/OrgName_envName_vhostName-client.pem exists for the specific virtual host. If the file does not exist, move to the Resolution section below.
2. If the file exists, use the below openssl command to get the details of the certificates that are available on the Edge Router:
```
openssl -in <OrgName_envName_vhostName-client.pem> -text -noout
```
3. Check the issuer, subject and expiry date of the certificate. If any of these do not match with what was observed in the Truststore in Edge UI or using the management APIs, then that's the cause of the error.
4. It's possible that the Router did not reload the uploaded certificates.

Resolution

Restart the Router to ensure the latest Certificates are loaded using the below step:

apigee-service edge-router restart

Re-run the APIs and check the results. If the problem persists, go to Gather Diagnostic Information.

Gather Diagnostic Information

If the problem persists even after following the above instructions, please gather the following diagnostic information. Contact and share the information you collect with Apigee Edge Support:

If you are a Public Cloud user, then provide the following information:
1. Organization Name
2. Environment Name
3. API Proxy Name
4. Virtual Host Name
5. Host Alias Name
6. Complete curl command to reproduce the error
7. TCP/IP packets captured on the Client Application
If you are a Private Cloud user, provide the following information:
1. Virtual Host Name and its definition using Get virtual host API
2. Host Alias Name
3. Complete Error Message observed
4. TCP/IP packets captured on the Client Application or Router.
5. Output of List the certificates from the keystore API API and also the details of each Certificate obtained using Get cert details API.
Details about what sections in this Playbook you have tried and any other insights that will help us to fastrack resolution of this issue.