503 Service Unavailable - Proxy tunnel creation failed with 403

You're viewing Apigee Edge documentation.
Go to the Apigee X documentation. info

Symptom

The client application gets an HTTP status code of 503 Service Unavailable with the error code protocol.http.ProxyTunnelCreationFailed as a response for API calls.

Error message

Client application gets the following response code:

HTTP/1.1 503 Service Unavailable

In addition, you may observe the following error message:

{
   "fault":{
      "faultstring":"Proxy refused to create tunnel with response status 403",
      "detail":{
         "errorcode":"protocol.http.ProxyTunnelCreationFailed"
      }
   }
}

Forward proxy and tunnelling

Apigee Edge allows your API proxies to communicate with your backend server through a proxy server as explained in Configure forward proxy. The proxy server opens a secure (HTTPS) or non-secure (HTTP) connection to the backend server depending on the proxy type (indicated by the property HTTPClient.proxy.type) used and transfers the data in both directions. This is known as tunneling.

By default, Apigee Edge uses tunneling for all traffic. To disable tunneling, the property HTTPClient.use.tunneling needs to be set to false.

Error Code: protocol.http.ProxyTunnelCreationFailed

Apigee Edge returns the error code protocol.http.ProxyTunnelCreationFailed if the proxy server is unable to create a tunnel between Apigee Edge and the backend server due to any issues such as firewall, ACL (Access Control List) restrictions, DNS issues, backend server unavailability, timeouts, etc.

The status code in the faultstring of the response from Apigee Edge typically indicates a possible high level cause that has led to this error.

Faultstring template:

Proxy refused to create tunnel with response status STATUS_CODE

Possible causes for some of the status code observed in faultstring:

The following table describes possible causes depending on the status code indicated in the faultstring:

Faultstring	Description
Proxy refused to create tunnel with response status 403	403 - Forbidden This can happen possibly due to firewall or ACL restrictions configured on the backend server that prevents creation of tunnel.
Proxy refused to create tunnel with response status 503	503 - Service Unavailable This can happen due to DNS issues, firewall restrictions, backend server’s unavailability that prevents creation of tunnel
Proxy refused to create tunnel with response status 504	504 - Gateway Timeout This can happen if there are timeouts during creation of tunnel

Depending on the status code observed in the faultstring, you need to use appropriate techniques to troubleshoot the issue. This playbook explains how to troubleshoot the issue if you observe the status code 403 in the faultstring for the error code protocol.http.ProxyTunnelCreationFailed.

Possible causes

This error (status code 403) occurs if there are any firewall or ACL (Access Control List) restrictions configured on the backend serverthat prevent the tunnel from being created between Apigee Edge and the backend server by the proxy server.

Cause	Description	Troubleshooting instructions applicable for
Proxy refused to create tunnel with response status 403	Proxy server refuses to create the tunnel as it receives the Proxy server host name instead of the Backend server host name in the Host header.	Edge Private Cloud users only

Common diagnosis steps

Use one of the following tools/techniques to diagnose this error:

Cause: Proxy refused to create tunnel with response status 403

Diagnosis

1. Determine the Fault Code and Fault Source for 503 Service Unavailable using Trace Tool or NGINX access logs as explained in Common diagnosis steps.
2. Review the Error Message and determine the status code indicated in the faultstring for failure in creation of the tunnel.
3. In this scenario, the status code is 403, which means Forbidden.
4. This means that there are insufficient rights or privileges to create the tunnel. This could typically happen if there are any firewall or ACL (Access Control List) restrictions that prevent creation of the tunnel.
5. Review any firewall and/or ACL restrictions that are configured on your backend server that can prevent creation of the tunnel.
6. Depending on the type of firewall and/or ACL restrictions, you need to fix the issue appropriately.

7. Let’s take an example firewall restriction to explain how to troubleshoot and resolve this issue:

   Scenario: Firewall restriction on backend server expects Host Header should always contain backend server host name

   You can use one of the following ways to determine the Host Header passed by Apigee Edge:

   Trace

   To determine the Host Header using Trace:

   1. Ascertain that the faultstring contains Proxy refused to create tunnel with response status 403 using trace as explained in Common diagnosis steps.
   2. Navigate to the phase Target Request Flow Started and review the Request Headers
   3. Verify the value of the host name specified in the Host header in the Request Headers section.
   4. If the Host header contains the proxy host name, then that’s the cause for this error.
   5. This is because the firewall is configured on the backend server to accept the requests only if the Host Header contains the backend server’s name.
   6. So when the proxy server tries to create the tunnel with the backend server, it fails with the error

      Proxy refused to create tunnel with response status 403.

      Sample trace showing Host Header having proxy host name

      ( view larger image)

      

      In the sample trace shown above, it shows that the Host Header contains the name of the proxy host www.proxyserver.com. Since there is a firewall restriction configured on the backend server which expects only the backend server host name to be contained in the Host Header, you get the error Proxy refused to create tunnel with response status 403.

   tcpdump

   To determine the Host Header using tcpdump

   1. Capture a tcpdump on the proxy server for the requests coming from Apigee Edge’s Message Processor component with the following command:

      tcpdump -i any -s 0 host MP_IP_ADDRESS -w FILE_NAME
      

      For more information on using the tcpdump command, see tcpdump.

   2. Analyze the tcpdump data using the Wireshark tool or a similar tool.

   3. Here's a sample analysis of the tcpdump using Wireshark:

      ( view larger image)

      
   4. Packet numbers 13, 14, and 15 show that the Message Processor is establishing connection to the proxy server through a three-way TCP handshake process.
   5. In packet 16, the Message Processor has connected to the proxy host httpbin.org (shown in the example above).

   6. Select packet 16 and examine the content of the packet in detail and specifically the Host Header being passed to the proxy server by the Message Processor.

      
   7. The sample above shows Host Header httpin.org, which is the hostname of the proxy server. Therefore, when the proxy server tries to create the tunnel with the backend server by passing the above Host Header httpin.org, it fails with the error Proxy refused to create tunnel with response status 403.

Resolution

Scenario: Firewall restriction on proxy server expects Host Header should always contain backend server hostname

If you have ascertained that this error is caused because the firewall on the backend server is configured such that it expects Host Header should always contain backend server host name, while the Message Processor is sending the proxy server hostname, then perform the following steps to resolve the issue:

1. Set the property use.proxy.host.header.with.target.uri to true in the TargetEndpoint as shown in the following example:

   Sample TargetEndpoint configuration:

   <TargetEndpoint name="default">
     <HTTPTargetConnection>
       <URL>https://mocktarget.apigee.net/json</URL>
       <Properties>
         <Property name="use.proxy.host.header.with.target.uri">true</Property>
       </Properties>
     </HTTPTargetConnection>
   </TargetEndpoint>
   

2. Ensure that the other properties related to forward proxy are configured on the Message Processor as follows:

   1. Review the file /opt/apigee/customer/application/message-processor.properties on each of the Message Processors.

   2. Ensure that the following properties are set as per your use case or requirements:

      Sample values for properties:

      conf_http_HTTPClient.use.proxy=true
      conf/http.properties+HTTPClient.proxy.type=HTTP
      conf/http.properties+HTTPClient.proxy.host=PROXY_SERVER_HOST_NAME
      conf/http.properties+HTTPClient.proxy.port=PORT_#
      conf/http.properties+HTTPClient.proxy.user=USERNAME
      conf/http.properties+HTTPClient.proxy.password=PASSWORD
      

Must gather diagnostic information

If the problem persists even after following the above instructions, gather the following diagnostic information, and then contact Apigee Edge Support:

If you are a Private Cloud user, provide the following information:

• Complete error message observed for the failing requests
• Environment name
• API proxy bundle
• Trace file for the API requests

• NGINX access logs

  /opt/apigee/var/log/edge-router/nginx/ORG~ENV.PORT#_access_log

  Where: ORG, ENV and PORT# are replaced with actual values.

• Message Processor system logs

  /opt/apigee/var/log/edge-message-processor/logs/system.log
  

References

• Configure forward proxy
• SSL Tunneling