Envoy proxy fails with HTTP 403 Forbidden error in Apigee Adapter for Envoy

You're viewing Apigee Edge documentation.
Go to the Apigee X documentation. info

Symptom

Envoy Proxy fails with HTTP 403 Forbidden error when invoked through Apigee Adapter for Envoy.

Error message

The following error message is displayed:

HTTP/1.1 403 Forbidden
content-length: 19
content-type: text/plain
date: Tue, 03 Nov 2020 00:20:10 GMT
server: istio-envoy

Possible causes

The Envoy proxy will raise an HTTP 403 error if one of the following conditions occur:

Cause	Description	Troubleshooting instructions applicable for
API product is not enabled	API product is not enabled for the specific environment.	Edge Public and Private Cloud users
Missing target service URI path in API Product	The URI path of the target service is missing or not added to the API product under API resources.	Edge Public and Private Cloud users
Missing hostname in the API Product	The hostname given in the client API request is missing in the API product under Apigee remote service targets.	Edge Public and Private Cloud users
Missing API key in the request header	The API key is not passed in the x-api-key HTTP header.	Edge Public and Private Cloud users
Invalid API key	The API key passed as part of the request is invalid.	Edge Public and Private Cloud users
Apigee Adapter for Envoy is unable to communicate with remote-service API proxy	Apigee Adapter for Envoy is unable to communicate with the remote-service API proxy.	Edge Public and Private Cloud users
Envoy proxy is unable to communicate with Apigee Adapter for Envoy	Envoy proxy is unable to communicate with Apigee Adapter for Envoy	Edge Public and Private Cloud users

Before you begin

1. Verify that you are getting the 403 Forbidden response message from the Envoy proxy. For example:

   curl -i -H "x-api-key: $API_KEY" http://httpbin:8080/echo
   
   HTTP/1.1 403 Forbidden
   content-length: 19
   content-type: text/plain
   date: Tue, 12 Jan 2021 08:18:08 GMT
   server: envoy
   RBAC: access denied
   

2. Enable debug logs:

   Ensure you have enabled debug logs in Apigee Adapter for Envoy to capture more details about the error. If not, stop Apigee Adapter for Envoy and start it again, enabling debug logs using the following command:

   apigee-remote-service-envoy -c config.yaml -l debug
   

Cause: API product is not enabled

This error will occur if the specific API Product used by the Envoy Proxy is not enabled in the specific environment in which the API calls are invoked.

Diagnosis

Perform the following steps to diagnose the issue:

1. Enable debug logs as explained in step 2 above.
2. Check the Apigee Adapter for Envoy logs and verify that the following message is displayed under the section Authorizing request:

   product: API_PRODUCT_NAME not found
   

   Sample Debug Log Output:

   2021-01-12T08:18:08.124Z        DEBUG   auth/auth.go:98 Authenticate: key: 7mQIG..., claims: map[string]interface {}(nil)
   2021-01-12T08:18:08.124Z        DEBUG   auth/verify_api_key.go:106      fetchToken fetching: 7mQIG...
   2021-01-12T08:18:08.589Z        DEBUG   auth/auth.go:125        using api key from request
   2021-01-12T08:18:08.589Z        DEBUG   auth/auth.go:157        Authenticate success: &auth.Context{Context:(*server.Handle
   r)(0xc0001a0600), ClientID:"7mQIG...", AccessToken:"", Application:"ENVOY-APP-1", APIProducts:[]string{"ENVOY-PRODUCT-1"},
   Expires:time.Time{wall:0x0, ext:63746037188, loc:(*time.Location)(0x14a3be0)}, DeveloperEmail:"[---masked---]", Scopes:[]
   string{""}, APIKey:"7mQIG..."}
   2021-01-12T08:18:08.589Z        DEBUG   product/manager.go:89
   Authorizing request:
     products: [ENVOY-PRODUCT-1]
     scopes: []
     operation: GET /echo
     target: httpbin:8080
     - product: ENVOY-PRODUCT-1
       not found
   

   The above example shows that the API product ENVOY-PRODUCT-1 was not found in Apigee Adapter for Envoy.

   For more information about Apigee Adapter for Envoy logging, see Logging.

3. If you see this message while authorizing the API request, then it most likely indicates that the specific API Product is not enabled for a specific environment in which you are making the API calls.
4. Perform the following steps to verify this:
   1. Log in to the Edge UI.
   2. On the Publish > API products page, click the specific API product that you used for configuring Apigee Adapter for Envoy.
   3. Verify that the specific environment in which you are making the API requests, is enabled in the API product.
   4. If the specific environment is not enabled in the API Product, then that’s the cause for this issue.
5. If the specific environment is already enabled, then go to Cause: Missing target service URI path in API Product.

Resolution

If the specific environment is not enabled in the API product, perform the following steps to resolve the issue:

1. Log in to the Edge UI.
2. On the Publish > API products page, click the specific API product that you used for configuring Apigee Adapter for Envoy.
3. On the API products > Product name page, click Edit.
4. Enable the specific environment in which you would like to make API requests, by selecting the relevant environment checkbox.
5. Click Save.

Cause: Missing target service URI path in API product

This error will occur if the target’s URI path is not specified in the specific API Product used by the Envoy Proxy.

Diagnosis

Perform the following steps to diagnose the issue:

1. Enable debug logs as explained in step 2 above.

2. Check the Apigee Adapter for Envoy logs and verify that the following message is displayed for the specific API Product associated with a specific target under the section Authorizing request:

   no path: REQUEST_URI_PATH
   

   Sample Debug Log Output:

   2021-01-12T08:09:02.604Z        DEBUG   auth/auth.go:98 Authenticate: key: 7mQIG..., claims: map[string]interface {}(nil)
   2021-01-12T08:09:02.605Z        DEBUG   auth/auth.go:125        using api key from request
   2021-01-12T08:09:02.605Z        DEBUG   auth/auth.go:157        Authenticate success: &auth.Context{Context:(*server.Handle
   r)(0xc0001a4180), ClientID:"7mQIG...", AccessToken:"", Application:"ENVOY-APP-1", APIProducts:[]string{"ENVOY-PRODUCT-1"},
   Expires:time.Time{wall:0x0, ext:63746036507, loc:(*time.Location)(0x14a3be0)}, DeveloperEmail:"[---masked---]", Scopes:[]
   string{""}, APIKey:"7mQIG..."}
   2021-01-12T08:09:02.605Z        DEBUG   product/manager.go:89
   Authorizing request:
     products: [ENVOY-PRODUCT-1]
     scopes: []
     operation: GET /echo1
     target: httpbin:8080
     - product: ENVOY-PRODUCT-1
       no path: /echo1
   2021-01-12T08:09:02.605Z        DEBUG   server/authorization.go:228     sending ok (actual: PERMISSION_DENIED)

   The sample output shows the message:

   no path: /echo1
   

   This indicates the path /echo1 was not found in the API product ENVOY-PRODUCT-1.

3. If you see the message no path: REQUEST_URI_PATH in the Apigee Adapter for Envoy debug logs, then that’s the cause of this issue. If not, go to Cause: Missing hostname in the API product.

Resolution

If the specific request URI is not added to the API Product for the specific target, then perform the following steps to resolve the issue:

1. Log in to the Edge UI.
2. On the Publish > API products page, click the specific API product that you used for configuring Apigee Adapter for Envoy.
3. On the API products > Product name page, click Edit.
4. In the API resources pane, add the API request URI to the API product.
5. Monitor the Apigee Adapter for Envoy logs and wait until the Apigee Adapter for Envoy fetches the updated API product. Afterwards, send another API request to verify the fix.

Cause: Missing hostname in the API product

This error will occur if the target hostname and port combination is not added to the specific API Product used by the Envoy Proxy.

Diagnosis

Perform the following steps to diagnose the issue:

1. Enable debug logs as explained in step 2 above.

2. Check the Apigee Adapter for Envoy logs and verify that the following message is displayed for the specific API Product associated with a specific target under the section Authorizing request:

   no targets: HOSTNAME:PORT
   

   Sample Debug Log Output:

   2021-01-12T08:12:06.019Z        DEBUG   auth/auth.go:98 Authenticate: key: 7mQIG..., claims: map[string]interface {}(nil)
   2021-01-12T08:12:06.019Z        DEBUG   auth/auth.go:125        using api key from request
   2021-01-12T08:12:06.019Z        DEBUG   auth/auth.go:157        Authenticate success: &auth.Context{Context:(*server.Handle
   r)(0xc0001a4180), ClientID:"7mQIG...", AccessToken:"", Application:"ENVOY-APP-1", APIProducts:[]string{"ENVOY-PRODUCT-1"},
   Expires:time.Time{wall:0x0, ext:63746036507, loc:(*time.Location)(0x14a3be0)}, DeveloperEmail:"[---masked---]", Scopes:[]
   string{""}, APIKey:"7mQIG..."}
   2021-01-12T08:12:06.019Z        DEBUG   product/manager.go:89
   Authorizing request:
     products: [ENVOY-PRODUCT-1]
     scopes: []
     operation: GET /echo
     target: httpbin1:8080
     - product: ENVOY-PRODUCT-1
       no targets: httpbin1:8080
   2021-01-12T08:12:06.020Z        DEBUG   server/authorization.go:228     sending ok (actual: PERMISSION_DENIED)

   The above example shows that the hostname and port combination httpbin1:8080 was not found in the API product ENVOY-PRODUCT-1.

3. If Apigee Adapter for Envoy logs contain an entry with the message no targets: HOSTNAME:PORT while authorizing the request, then this is the cause of the issue. If not, go to Cause: Missing API key in the request header.

Resolution

If the target hostname and port combination is not added to the API product, perform the following steps to resolve the issue:

1. Log in to the Edge UI.
2. On the Publish > API products page, click the specific API product that you used for configuring Apigee Adapter for Envoy.
3. On the API products > Product name page, click Edit.

4. In the Apigee remote service targets pane, add the target hostname and port and click Save.

   If you do not see the Apigee remote service targets section in the UI, add a custom attribute to the API product with the name apigee-remote-service-targets and add the HOSTNAME:PORT value using the Edge API. For example:

   curl https://api.enterprise.apigee.com/v1/organizations/$ORG/apiproducts/$ENVOY_PRODUCT \
       -X GET \
       -H "Authorization: Bearer $ACCESS_TOKEN" \
       -H "Content-Type:application/json" \
       -d \
   {
       "apiResources": [
           "/echo",
           "/verifyApiKey"
       ],
       "approvalType": "auto",
       "attributes": [
           {
               "name": "access",
               "value": "public"
           },
           {
               "name": "apigee-remote-service-targets",
               "value": "localhost:8080"
           }
       ],
       "createdAt": 1610435989556,
       "createdBy": "---masked---",
       "description": "",
       "displayName": "ENVOY-PRODUCT-1",
       "environments": [
           "test"
       ],
       "lastModifiedAt": 1612234134060,
       "lastModifiedBy": "---masked---",
       "name": "ENVOY-PRODUCT-1",
       "proxies": [
           "remote-service"
       ],
       "scopes": []
   }
   

5. Once the above task is done, monitor the Apigee Adapter for Envoy logs and wait until the Apigee Adapter for Envoy fetches the updated API product. Afterwards, send another API request to verify the fix.

Cause: Missing API key in the request header

This error will occur if API Key is not passed as part of the request headers.

Diagnosis

Perform the following steps to diagnose the issue:

1. Enable debug logs as explained in step 2 above.
2. Check the Apigee Adapter for Envoy logs and verify that you see the [missing authentication] message under the Authenticate error section.

   Sample Debug Log Output:

   2021-01-12T08:20:31.461Z        DEBUG   auth/auth.go:98 Authenticate: key: , claims: map[string]interface {}(nil)
   2021-01-12T08:20:31.461Z        DEBUG   auth/auth.go:159
   Authenticate error: &auth.Context{Context:(*server.Handler)
   (0xc0001a0600), ClientID:"", AccessToken:"", Application:"", APIProducts:[]string(nil), Expires:time.Time{wall:0x0, ext:0,
   loc:(*time.Location)(nil)}, DeveloperEmail:"", Scopes:[]string(nil), APIKey:""} [missing authentication]
   2021-01-12T08:20:31.461Z        DEBUG   server/authorization.go:205     sending denied: UNAUTHENTICATED
   2021-01-12T08:20:32.448Z        DEBUG   server/header_context.go:68     No context header x-apigee-api, using target header
   : :authority

   The sample output shown above has the message [missing authentication]. This message indicates that the API key is not passed as part of the request header.

3. If Apigee Adapter for Envoy logs contain a log entry with the message [missing authentication]under the Authenticate error section, then this is the cause of the issue. If not, go to Cause: Invalid API key.

Resolution

If the error [missing authentication] was displayed in the Apigee Adapter for Envoy logs, perform the following steps to resolve the issue:

1. Check if the client has sent the API key using the HTTP header x-api-key in the API request. If not, request the client send the API key in the HTTP header x-api-key.
2. Check the Apigee Adapter for Envoy configuration file and verify that the default API key header name x-api-key has been changed, for an example:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: apigee-remote-service-envoy
     namespace: apigee
   data:
     config.yaml: |
       global:
         tls:
           ...
       tenant:
         ...
       auth:
         target_header: api-key
   

   In the above example, the default API Key header name has been modified to api-key. In this case, you need to pass the API Key as part of the header api-key.

3. If the default API key header name has been changed, request the client use the updated API key header name and send another API request and verify if it solves the issue.

Cause: Invalid API key

This error will occur if invalid API Key is passed as part of the request header.

Diagnosis

Perform the following steps to diagnose the issue:

1. Enable debug logs as explained in step 2 above.
2. Check the Apigee Adapter for Envoy logs and verify that you see the message [permission denied] under the Authenticate error section. This is usually displayed after API Key is fetched by the Adapter which is indicated by the message fetchToken fetching: API_KEY.

   Sample Debug Log Output:

   2021-01-12T05:01:07.198Z        DEBUG   auth/auth.go:98 Authenticate: key: 123, claims: map[string]interface {}(nil)
   2021-01-12T05:01:07.198Z        DEBUG   auth/verify_api_key.go:106      fetchToken fetching: API_KEY
   2021-01-12T05:01:09.102Z        DEBUG   server/header_context.go:68     No context header x-apigee-api, using target header: :authority
   2021-01-12T05:01:09.831Z        DEBUG   auth/auth.go:159        Authenticate error: &auth.Context{Context:(*server.Handler)(0xc0001640c0), ClientID:"", AccessToken:"", Application:"", APIProducts:[]string(nil), Expires:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}, DeveloperEmail:"", Scopes:[]string(nil), APIKey:""} [permission denied]
   2021-01-12T05:01:09.832Z        DEBUG   server/authorization.go:228     sending ok (actual: PERMISSION_DENIED)
   

   In this example the API key sent in the API request was not valid.

3. If Apigee Adapter for Envoy logs contain a log entry with the [permission denied] under the Authenticate error section, then it indicates that the API Key passed as part of the request is invalid and is the cause of the problem. If not, go to Cause: Apigee Adapter for Envoy is unable to communicate with remote-service API proxy.

Resolution

If the message [permission denied] is observed under the Authenticate error section in the Apigee Adapter for Envoy logs, then perform the following steps to resolve the issue:

1. Check the API key sent in the API request against the API key value found in the application connected to the API product.
2. If the API key used by the client is not valid, request the client send the valid API Key.
3. If the API key used by the client is valid and if you are still seeing an HTTP 403 error, please contact Apigee Edge Support to investigate this further.

Cause: Apigee Adapter for Envoy is unable to communicate with remote-service API proxy

This error will occur if Apigee Adapter for Envoy is unable to communicate with the remote service API Proxy if the remote service host configured is invalid.

Diagnosis

Perform the following steps to diagnose the issue:

1. Enable debug logs as explained in step 2 above.

2. Check the Apigee Adapter for Envoy logs and verify that you see the following message:

   Error retrieving products: REQUEST_URI: no such host
   

   Sample Debug Log Output:

   2021-01-12T08:29:06.499Z        DEBUG   product/manager.go:188  retrieving products from: https://foo/remote-service/products
   2021-01-12T08:29:06.505Z        ERROR   product/manager.go:164  Error retrieving products: GET "https://foo/remote-service/pro
   ducts": dial tcp: lookup foo on 169.254.169.254:53: no such host
   github.com/apigee/apigee-remote-service-golib/product.(*manager).start.func1
           /go/pkg/mod/github.com/apigee/apigee-remote-service-golib@v1.4.0/product/manager.go:164
   github.com/apigee/apigee-remote-service-golib/util.(*Looper).Run
           /go/pkg/mod/github.com/apigee/apigee-remote-service-golib@v1.4.0/util/looper.go:87
   github.com/apigee/apigee-remote-service-golib/util.(*Looper).Start.func1
           /go/pkg/mod/github.com/apigee/apigee-remote-service-golib@v1.4.0/util/looper.go:59
   

   In this example, Apigee Adapter for Envoy was not able to communicate with the remote-service API proxy because the hostname provided in the remote server API Proxy URL is not valid as indicated by the error no such host .

3. If Apigee Adapter for Envoy logs contain a log entry with the message no such host, then this is the cause of the issue. If not, go to Cause: Envoy proxy is unable to communicate with Apigee Adapter for Envoy.

Resolution

If the above errors are displayed in the Apigee Adapter for Envoy logs, perform the following steps to resolve the issue:

1. Check the Apigee Adapter for Envoy configuration file and verify that the given remote-service API proxy URL is valid.

   If not, stop Apigee Adapter for Envoy, fix the remote-service API proxy URL in the configuration file, start Apigee Adapter for Envoy, and send another API request and verify the fix.

   Sample configuration:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: apigee-remote-service-envoy
     namespace: apigee
   data:
     config.yaml: |
       tenant:
         internal_api: https://istioservices.apigee.net/edgemicro
         remote_service_api: https://ORG-ENV.apigee.net/remote-service
         org_name: ORG
         env_name: ENV
         key: KEY
         secret: SECRET
         

2. Verify that the remote-service API proxy is deployed in the relevant Edge environment. If not, deploy the remote-service API proxy in the relevant Edge environment and try it again.
3. Verify network connectivity between the Apigee Adapter for Envoy and the remote-service API proxy endpoint. If there are any network connectivity issues found, contact your networking team and try to resolve the problem.

Cause: Envoy proxy is unable to communicate with Apigee Adapter for Envoy

Diagnosis

Perform the following steps to diagnose the issue:

1. Ensure you have enabled debug logs in Envoy. If not, stop Envoy and start it again, enabling debug logs. Then send another API request.

   Standalone deployments:

   envoy -c envoy-config.yaml -l debug
   

   Kubernetes/Istio based deployments:

   kubectl -n=istio-system get pods
   kubectl -n=istio-system exec -it INGRESS_GATEWAY_NAME bash -- curl -X POST localhost:15000/logging?connection=debug
   

2. Check the Apigee Adapter for Envoy logs and verify that there is a log entry with the message:

   connecting to APIGEE_ENVOY_ADAPTER_HOST:5000
   

   which is then followed by:

   upstream connect error or disconnect/reset before headers. reset reason: ACTUAL_REASON
   

   Sample Debug Log Output:

   [2021-03-23 05:44:41.867][1303661][debug][connection] [external/envoy/source/common/network/connection_impl.cc:769] [C4] connecting to 127.0.0.1:5000
   [2021-03-23 05:44:41.867][1303661][debug][connection] [external/envoy/source/common/network/connection_impl.cc:785] [C4] connection in progress
   [2021-03-23 05:44:41.868][1303661][debug][http2] [external/envoy/source/common/http/http2/codec_impl.cc:1173] [C4] updating connection-level initial window size to 268435456
   [2021-03-23 05:44:41.869][1303661][debug][connection] [external/envoy/source/common/network/connection_impl.cc:634] [C4] delayed connection error: 111
   [2021-03-23 05:44:41.869][1303661][debug][connection] [external/envoy/source/common/network/connection_impl.cc:203] [C4] closing socket: 0
   [2021-03-23 05:44:41.869][1303661][debug][client] [external/envoy/source/common/http/codec_client.cc:96] [C4] disconnect. resetting 0 pending requests
   [2021-03-23 05:44:41.869][1303661][debug][pool] [external/envoy/source/common/conn_pool/conn_pool_base.cc:314] [C4] client disconnected, failure reason:
   [2021-03-23 05:44:41.869][1303661][debug][router] [external/envoy/source/common/router/router.cc:1031] [C0][S6149963213555558594] upstream reset: reset reason: connection failure, transport failure reason:
   [2021-03-23 05:44:41.869][1303661][debug][http] [external/envoy/source/common/http/async_client_impl.cc:100] async http request response headers (end_stream=true):
   ':status', '200'
   'content-type', 'application/grpc'
   'grpc-status', '14'
   'grpc-message', 'upstream connect error or disconnect/reset before headers. reset reason: connection failure'

   The above example shows that Envoy was unable to communicate with Apigee Adapter for Envoy due to the reason connection failure.

3. The connection failure can be due to multiple reasons. Let’s look at each of the scenarios.

Scenario #1: Adapter process is not running

If the Apigee Adapter for Envoy process is not running, then this error can occur.

1. Verify that the Apigee Adapter for Envoy process is running by executing the following command. If the Apigee Adapter for Envoy process is running, the result of the following command should list it.

   ps -ef | grep apigee-remote-service-envoy
   

2. If it is not running, then that’s the cause of the issue.

Resolution

1. If the Apigee Adapter for Envoy process is not running, then start the Apigee Adapter for Envoy.
2. Make another API request and verify if the issue has been fixed.

Scenario #2: Adapter process is not listening on the specific port

If the Apigee Adapter for Envoy process is not listening on the specific port, then this error can occur.

If the Apigee Adapter for Envoy process is running, verify that there is a socket listening on port 5000: APIGEE_ENVOY_ADAPTER_HOST:5000. You can run the netstat command to verify this:

sudo netstat -lnp | grep 5000

Sample output:

sudo netstat -lnp | grep 5000

tcp6       0      0 :::5000                 :::*                    LISTEN      1596530/./apigee-re

If there is no socket listening on port 5000, that could be the reason for this issue.

Resolution

1. Stop the Apigee Adapter for Envoy and start it again.
2. Make another API request and verify if the issue has been fixed.

Scenario #3: Network Connectivity between Envoy and Apigee Adapter for Envoy

1. Verify network connectivity between Envoy and Apigee Adapter for Envoy:

   ssh $ENVOY_HOST
   telnet $APIGEE_ENVOY_ADAPTER_HOST 5000

   If telnet could make a TCP connection to Apigee Adapter for Envoy an output similar to following will be displayed:

   telnet $APIGEE_ENVOY_ADAPTER_HOST 5000
   
   Trying ::1...
   Connected to localhost.
   Escape character is '^]'.
   

2. If you observe the error Connection timed out with telnet, then that indicates there is a network connectivity issue between Envoy and Apigee Adapter for Envoy.

Resolution

If you see any network connectivity issues between Envoy and Apigee Adapter for Envoy, please engage your networking team and try to resolve the issue.

If the problem still persists, go to Must gather diagnostic information.

Must gather diagnostic information

If the problem persists after following the above instructions, gather the following diagnostic information and then contact Apigee Edge Support:

1. Apigee product used:

   Example: Apigee Edge Cloud, Apigee OPDK, Apigee hybrid, Apigee X

2. Apigee organization and environment

3. API product definition read using the Edge API:

   curl -i -u $USER:$PASSWORD $MANAGEMENT_SERVER_ENDPOINT/v1/organizations/$ORGANIZATION/apiproducts/$API_PRODUCT

   Reference: Apigee Edge APIs

4. Start a trace session in the remote-service API proxy using the Apigee Edge UI. Reproduce this problem and share the Trace session XML file.

   Reference: Using the Trace tool | Apigee Edge

5. Apigee Adapter for Envoy logs (complete logs related to the given time period)

   Standalone deployments:

   # by default Apigee Envoy write logs to stdout and stderr, check your deployment configuration and collect logs accordingly
   

   Kubernetes/Istio based deployments:

   kubectl -n=apigee get pods
   kubectl -n=apigee logs APIGEE_REMOTE_SERVICE_ENVOY_POD_NAME > apigee-remote-service-envoy.log

6. An API request sent to the Envoy proxy using a curl command (the complete output of the curl command):

   curl -v ENVOY_PROXY_ENDPOINT

7. An API request sent to the target service using a curl command (the complete output of the curl command):

   curl -v TARGET_SERVICE_ENDPOINT