You're viewing **Apigee Edge** documentation.  
[Go to the
**Apigee X** documentation](http://cloud.google.com/apigee/docs/release/notes/100-envoy-adapter-release-notes). info

## Symptom


Envoy Proxy fails with HTTP `403 Forbidden` error when invoked through
[Apigee Adapter for Envoy](https://docs.apigee.com/api-platform/envoy-adapter).

## Error message


The following error message is displayed:

```
HTTP/1.1 403 Forbidden
content-length: 19
content-type: text/plain
date: Tue, 03 Nov 2020 00:20:10 GMT
server: istio-envoy
```

## Possible causes


The Envoy proxy will raise an HTTP `403` error if one of the following conditions
occur:

| Cause | Description | Troubleshooting instructions applicable for |
|---|---|---|
| [API product is not enabled](https://docs.apigee.com/api-platform/troubleshoot/envoy-adapter/envoy-proxy-fails-403-forbidden-error.md.txt#api-product-not-enabled) | API product is not enabled for the specific environment. | Edge Public and Private Cloud users |
| [Missing target service URI path in API Product](https://docs.apigee.com/api-platform/troubleshoot/envoy-adapter/envoy-proxy-fails-403-forbidden-error.md.txt#missing-target-service-path) | The URI path of the target service is missing or not added to the API product under API resources. | Edge Public and Private Cloud users |
| [Missing hostname in the API Product](https://docs.apigee.com/api-platform/troubleshoot/envoy-adapter/envoy-proxy-fails-403-forbidden-error.md.txt#missing-hostname) | The hostname given in the client API request is missing in the API product under Apigee remote service targets. | Edge Public and Private Cloud users |
| [Missing API key in the request header](https://docs.apigee.com/api-platform/troubleshoot/envoy-adapter/envoy-proxy-fails-403-forbidden-error.md.txt#missing-api-key) | The API key is not passed in the `x-api-key` HTTP header. | Edge Public and Private Cloud users |
| [Invalid API key](https://docs.apigee.com/api-platform/troubleshoot/envoy-adapter/envoy-proxy-fails-403-forbidden-error.md.txt#invalid-api-key) | The API key passed as part of the request is invalid. | Edge Public and Private Cloud users |
| [Apigee Adapter for Envoy is unable to communicate with remote-service API proxy](https://docs.apigee.com/api-platform/troubleshoot/envoy-adapter/envoy-proxy-fails-403-forbidden-error.md.txt#unable-to-communicate-with-remote-service) | Apigee Adapter for Envoy is unable to communicate with the remote-service API proxy. | Edge Public and Private Cloud users |
| [Envoy proxy is unable to communicate with Apigee Adapter for Envoy](https://docs.apigee.com/api-platform/troubleshoot/envoy-adapter/envoy-proxy-fails-403-forbidden-error.md.txt#unable-to-communicate-with-envoy-adapter) | Envoy proxy is unable to communicate with Apigee Adapter for Envoy | Edge Public and Private Cloud users |

## Before you begin

1. Verify that you are getting the `403 Forbidden` response message from the Envoy proxy. For example:

   ```
   curl -i -H "x-api-key: $API_KEY" http://httpbin:8080/echo

   HTTP/1.1 403 Forbidden
   content-length: 19
   content-type: text/plain
   date: Tue, 12 Jan 2021 08:18:08 GMT
   server: envoy
   RBAC: access denied
   ```
2. Enable debug logs:


   Ensure you have enabled debug logs in Apigee Adapter for Envoy to capture more details about
   the error. If not, stop Apigee Adapter for Envoy and start it again, enabling debug logs
   using the following command:

   ```
   apigee-remote-service-envoy -c config.yaml -l debug
   ```

## Cause: API product is not enabled


This error will occur if the specific API Product used by the Envoy Proxy is not enabled in the
specific environment in which the API calls are invoked.

### Diagnosis


Perform the following steps to diagnose the issue:

1. Enable debug logs as explained in [step 2](https://docs.apigee.com/api-platform/troubleshoot/envoy-adapter/envoy-proxy-fails-403-forbidden-error.md.txt#enable-debug-logs) above.
2. Check the Apigee Adapter for Envoy logs and verify that the following message is displayed under the section `Authorizing request`:

   ```
   product: API_PRODUCT_NAME not found
   ```

   **Sample Debug Log Output:**

   ```
   2021-01-12T08:18:08.124Z        DEBUG   auth/auth.go:98 Authenticate: key: 7mQIG..., claims: map[string]interface {}(nil)
   2021-01-12T08:18:08.124Z        DEBUG   auth/verify_api_key.go:106      fetchToken fetching: 7mQIG...
   2021-01-12T08:18:08.589Z        DEBUG   auth/auth.go:125        using api key from request
   2021-01-12T08:18:08.589Z        DEBUG   auth/auth.go:157        Authenticate success: &auth.Context{Context:(*server.Handle
   r)(0xc0001a0600), ClientID:"7mQIG...", AccessToken:"", Application:"ENVOY-APP-1", APIProducts:[]string{"ENVOY-PRODUCT-1"},
   Expires:time.Time{wall:0x0, ext:63746037188, loc:(*time.Location)(0x14a3be0)}, DeveloperEmail:"[---masked---]", Scopes:[]
   string{""}, APIKey:"7mQIG..."}
   2021-01-12T08:18:08.589Z        DEBUG   product/manager.go:89
   Authorizing request:
     products: [ENVOY-PRODUCT-1]
     scopes: []
     operation: GET /echo
     target: httpbin:8080
     - product: ENVOY-PRODUCT-1
       not found
   ```


   The above example shows that the API product `ENVOY-PRODUCT-1` was not found in
   Apigee Adapter for Envoy.


   For more information about Apigee Adapter for Envoy logging, see
   [Logging](https://docs.apigee.com/api-platform/envoy-adapter/v2.0.x/operation.md.txt#logging).
3. If you see this message while authorizing the API request, then it most likely indicates that the specific API Product is not enabled for a specific environment in which you are making the API calls.
4. Perform the following steps to verify this:
   1. Log in to the [Edge UI](https://apigee.com/edge).
   2. On the **Publish \> API products** page, click the specific API product that you used for configuring Apigee Adapter for Envoy.
   3. Verify that the specific environment in which you are making the API requests, is enabled in the API product.
   4. If the specific environment is not enabled in the API Product, then that's the cause for this issue.
5. If the specific environment is already enabled, then go to [Cause: Missing target service URI path in API Product](https://docs.apigee.com/api-platform/troubleshoot/envoy-adapter/envoy-proxy-fails-403-forbidden-error.md.txt#missing-target-service-path).

### Resolution


If the specific environment is not enabled in the API product, perform the following steps to
resolve the issue:

1. Log in to the [Edge UI](https://apigee.com/edge).
2. On the **Publish \> API products** page, click the specific API product that you used for configuring Apigee Adapter for Envoy.
3. On the **API products \> Product name** page, click **Edit**.
4. Enable the specific environment in which you would like to make API requests, by selecting the relevant environment checkbox.
5. Click **Save**.

## Cause: Missing target service URI path in API product


This error will occur if the target's URI path is not specified in the specific API Product used
by the Envoy Proxy.

### Diagnosis


Perform the following steps to diagnose the issue:

1. Enable debug logs as explained in [step 2](https://docs.apigee.com/api-platform/troubleshoot/envoy-adapter/envoy-proxy-fails-403-forbidden-error.md.txt#enable-debug-logs) above.
2. Check the Apigee Adapter for Envoy logs and verify that the following message is
   displayed for the specific API Product associated with a specific target under the section
   `Authorizing request`:

   ```
   no path: REQUEST_URI_PATH
   ```

   **Sample Debug Log Output:**

   ```
   2021-01-12T08:09:02.604Z        DEBUG   auth/auth.go:98 Authenticate: key: 7mQIG..., claims: map[string]interface {}(nil)
   2021-01-12T08:09:02.605Z        DEBUG   auth/auth.go:125        using api key from request
   2021-01-12T08:09:02.605Z        DEBUG   auth/auth.go:157        Authenticate success: &auth.Context{Context:(*server.Handle
   r)(0xc0001a4180), ClientID:"7mQIG...", AccessToken:"", Application:"ENVOY-APP-1", APIProducts:[]string{"ENVOY-PRODUCT-1"},
   Expires:time.Time{wall:0x0, ext:63746036507, loc:(*time.Location)(0x14a3be0)}, DeveloperEmail:"[---masked---]", Scopes:[]
   string{""}, APIKey:"7mQIG..."}
   2021-01-12T08:09:02.605Z        DEBUG   product/manager.go:89
   Authorizing request:
     products: [ENVOY-PRODUCT-1]
     scopes: []
     operation: GET /echo1
     target: httpbin:8080
     - product: ENVOY-PRODUCT-1
       no path: /echo1
   2021-01-12T08:09:02.605Z        DEBUG   server/authorization.go:228     sending ok (actual: PERMISSION_DENIED)
   ```

   The sample output shows the message:

   ```
   no path: /echo1
   ```

   This indicates the path `/echo1` was not found in the API product
   `ENVOY-PRODUCT-1`.
3. If you see the message `no path: REQUEST_URI_PATH` in the Apigee Adapter for Envoy debug logs, then that's the cause of this issue. If not, go to [Cause: Missing hostname in the API product](https://docs.apigee.com/api-platform/troubleshoot/envoy-adapter/envoy-proxy-fails-403-forbidden-error.md.txt#missing-hostname).

### Resolution


If the specific request URI is not added to the API Product for the specific target, then
perform the following steps to resolve the issue:

1. Log in to the [Edge UI](https://apigee.com/edge).
2. On the **Publish \> API products** page, click the specific API product that you used for configuring Apigee Adapter for Envoy.
3. On the **API products \> Product name** page, click **Edit**.
4. In the **API resources** pane, add the API request URI to the API product.
5. Monitor the Apigee Adapter for Envoy logs and wait until the Apigee Adapter for Envoy fetches the updated API product. Afterwards, send another API request to verify the fix.

## Cause: Missing hostname in the API product


This error will occur if the target hostname and port combination is not added to the specific
API Product used by the Envoy Proxy.

### Diagnosis


Perform the following steps to diagnose the issue:

1. Enable debug logs as explained in [step 2](https://docs.apigee.com/api-platform/troubleshoot/envoy-adapter/envoy-proxy-fails-403-forbidden-error.md.txt#enable-debug-logs) above.
2. Check the Apigee Adapter for Envoy logs and verify that the following message is
   displayed for the specific API Product associated with a specific target under the section
   `Authorizing request`:

   ```
   no targets: HOSTNAME:PORT
   ```

   **Sample Debug Log Output:**

   ```
   2021-01-12T08:12:06.019Z        DEBUG   auth/auth.go:98 Authenticate: key: 7mQIG..., claims: map[string]interface {}(nil)
   2021-01-12T08:12:06.019Z        DEBUG   auth/auth.go:125        using api key from request
   2021-01-12T08:12:06.019Z        DEBUG   auth/auth.go:157        Authenticate success: &auth.Context{Context:(*server.Handle
   r)(0xc0001a4180), ClientID:"7mQIG...", AccessToken:"", Application:"ENVOY-APP-1", APIProducts:[]string{"ENVOY-PRODUCT-1"},
   Expires:time.Time{wall:0x0, ext:63746036507, loc:(*time.Location)(0x14a3be0)}, DeveloperEmail:"[---masked---]", Scopes:[]
   string{""}, APIKey:"7mQIG..."}
   2021-01-12T08:12:06.019Z        DEBUG   product/manager.go:89
   Authorizing request:
     products: [ENVOY-PRODUCT-1]
     scopes: []
     operation: GET /echo
     target: httpbin1:8080
     - product: ENVOY-PRODUCT-1
       no targets: httpbin1:8080
   2021-01-12T08:12:06.020Z        DEBUG   server/authorization.go:228     sending ok (actual: PERMISSION_DENIED)
   ```

   The above example shows that the hostname and port combination `httpbin1:8080`
   was not found in the API product `ENVOY-PRODUCT-1`.
3. If Apigee Adapter for Envoy logs contain an entry with the message `no targets:
   HOSTNAME:PORT` while authorizing the request, then this is the cause of the issue. If not, go to [Cause: Missing API key in the request header](https://docs.apigee.com/api-platform/troubleshoot/envoy-adapter/envoy-proxy-fails-403-forbidden-error.md.txt#missing-api-key).

### Resolution


If the target hostname and port combination is not added to the API product, perform the
following steps to resolve the issue:

1. Log in to the [Edge UI](https://apigee.com/edge).
2. On the **Publish \> API products** page, click the specific API product that you used for configuring Apigee Adapter for Envoy.
3. On the **API products \> Product name** page, click **Edit**.
4. In the **Apigee remote service targets** pane, add the target hostname and
   port and click **Save**.

   If you do not see the **Apigee remote service targets** section in the UI,
   add a custom attribute to the API product with the
   name `apigee-remote-service-targets` and add the
   <var translate="no">HOSTNAME</var>:<var translate="no">PORT</var> value using the Edge API. For example:

   ```
   curl https://api.enterprise.apigee.com/v1/organizations/$ORG/apiproducts/$ENVOY_PRODUCT \
       -X GET \
       -H "Authorization: Bearer $ACCESS_TOKEN" \
       -H "Content-Type:application/json" \
       -d \
   {
       "apiResources": [
           "/echo",
           "/verifyApiKey"
       ],
       "approvalType": "auto",
       "attributes": [
           {
               "name": "access",
               "value": "public"
           },
           {
               "name": "apigee-remote-service-targets",
               "value": "localhost:8080"
           }
       ],
       "createdAt": 1610435989556,
       "createdBy": "---masked---",
       "description": "",
       "displayName": "ENVOY-PRODUCT-1",
       "environments": [
           "test"
       ],
       "lastModifiedAt": 1612234134060,
       "lastModifiedBy": "---masked---",
       "name": "ENVOY-PRODUCT-1",
       "proxies": [
           "remote-service"
       ],
       "scopes": []
   }
   ```

   > [!NOTE]
   > **Note:** Where $ACCESS_TOKEN is set to your OAuth 2.0 access token, as described in [Using OAuth2 to access the Edge API](https://docs.apigee.com/api-platform/system-administration/using-oauth2.md.txt). For information about the curl options used in this example, see [Use curl](https://docs.apigee.com/api-monitoring/api.md.txt#curl).

5. Once the above task is done, monitor the Apigee Adapter for Envoy logs and wait until the Apigee Adapter for Envoy fetches the updated API product. Afterwards, send another API request to verify the fix.

## Cause: Missing API key in the request header


This error will occur if API Key is not passed as part of the request headers.

### Diagnosis


Perform the following steps to diagnose the issue:

1. Enable debug logs as explained in [step 2](https://docs.apigee.com/api-platform/troubleshoot/envoy-adapter/envoy-proxy-fails-403-forbidden-error.md.txt#enable-debug-logs) above.
2. Check the Apigee Adapter for Envoy logs and verify that you see the `[missing authentication]` message under the `Authenticate error` section.

   **Sample Debug Log Output:**

   ```
   2021-01-12T08:20:31.461Z        DEBUG   auth/auth.go:98 Authenticate: key: , claims: map[string]interface {}(nil)
   2021-01-12T08:20:31.461Z        DEBUG   auth/auth.go:159
   Authenticate error: &auth.Context{Context:(*server.Handler)
   (0xc0001a0600), ClientID:"", AccessToken:"", Application:"", APIProducts:[]string(nil), Expires:time.Time{wall:0x0, ext:0,
   loc:(*time.Location)(nil)}, DeveloperEmail:"", Scopes:[]string(nil), APIKey:""} [missing authentication]
   2021-01-12T08:20:31.461Z        DEBUG   server/authorization.go:205     sending denied: UNAUTHENTICATED
   2021-01-12T08:20:32.448Z        DEBUG   server/header_context.go:68     No context header x-apigee-api, using target header
   : :authority
   ```

   The sample output shown above has the message `[missing authentication]`.
   This message indicates that the API key is not passed as part of the
   request header.
3. If Apigee Adapter for Envoy logs contain a log entry with the message `[missing
   authentication]`under the `Authenticate error `section, then this is the cause of the issue. If not, go to [Cause: Invalid API key](https://docs.apigee.com/api-platform/troubleshoot/envoy-adapter/envoy-proxy-fails-403-forbidden-error.md.txt#invalid-api-key).

### Resolution


If the error `[missing authentication]` was displayed in the
Apigee Adapter for Envoy logs, perform the following steps to resolve the issue:

1. Check if the client has sent the API key using the HTTP header `x-api-key` in the API request. If not, request the client send the API key in the HTTP header `x-api-key`.
2. Check the Apigee Adapter for Envoy configuration file and verify that the default API key header name `x-api-key` has been changed, for an example:

   ```
   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: apigee-remote-service-envoy
     namespace: apigee
   data:
     config.yaml: |
       global:
         tls:
           ...
       tenant:
         ...
       auth:
         target_header: api-key
   ```

   In the above example, the default API Key header name has been modified to
   `api-key`. In this case, you need to pass the API Key as part of the header
   `api-key`.
3. If the default API key header name has been changed, request the client use the updated API key header name and send another API request and verify if it solves the issue.

## Cause: Invalid API key


This error will occur if invalid API Key is passed as part of the request header.

### Diagnosis


Perform the following steps to diagnose the issue:

1. Enable debug logs as explained in [step 2](https://docs.apigee.com/api-platform/troubleshoot/envoy-adapter/envoy-proxy-fails-403-forbidden-error.md.txt#enable-debug-logs) above.
2. Check the Apigee Adapter for Envoy logs and verify that you see the message `[permission denied] `under the `Authenticate error` section. This is usually displayed after API Key is fetched by the Adapter which is indicated by the message `fetchToken fetching: API_KEY`.

   **Sample Debug Log Output:**

   ```
   2021-01-12T05:01:07.198Z        DEBUG   auth/auth.go:98 Authenticate: key: 123, claims: map[string]interface {}(nil)
   2021-01-12T05:01:07.198Z        DEBUG   auth/verify_api_key.go:106      fetchToken fetching: API_KEY
   2021-01-12T05:01:09.102Z        DEBUG   server/header_context.go:68     No context header x-apigee-api, using target header: :authority
   2021-01-12T05:01:09.831Z        DEBUG   auth/auth.go:159        Authenticate error: &auth.Context{Context:(*server.Handler)(0xc0001640c0), ClientID:"", AccessToken:"", Application:"", APIProducts:[]string(nil), Expires:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}, DeveloperEmail:"", Scopes:[]string(nil), APIKey:""} [permission denied]
   2021-01-12T05:01:09.832Z        DEBUG   server/authorization.go:228     sending ok (actual: PERMISSION_DENIED)
   ```

   In this example the API key sent in the API request was not valid.
3. If Apigee Adapter for Envoy logs contain a log entry with the `[permission
   denied]` under the `Authenticate error `section, then it indicates that the API Key passed as part of the request is invalid and is the cause of the problem. If not, go to [Cause: Apigee Adapter for Envoy
   is unable to communicate with remote-service API proxy](https://docs.apigee.com/api-platform/troubleshoot/envoy-adapter/envoy-proxy-fails-403-forbidden-error#unable-to-communicate-with-remote-service).

### Resolution

If the message `[permission denied]` is observed under the **`Authenticate
error`**section in the Apigee Adapter for Envoy logs, then perform the following steps
to resolve the issue:

1. Check the API key sent in the API request against the API key value found in the application connected to the API product.
2. If the API key used by the client is not valid, request the client send the valid API Key.
3. If the API key used by the client is valid and if you are still seeing an HTTP `403` error, please contact [Apigee Edge Support](https://docs.apigee.com/api-platform/troubleshoot/support.md.txt) to investigate this further.

## Cause: Apigee Adapter for Envoy is unable to communicate with remote-service API proxy


This error will occur if Apigee Adapter for Envoy is unable to communicate with the remote
service API Proxy if the remote service host configured is invalid.

### Diagnosis


Perform the following steps to diagnose the issue:

1. Enable debug logs as explained in [step 2](https://docs.apigee.com/api-platform/troubleshoot/envoy-adapter/envoy-proxy-fails-403-forbidden-error.md.txt#enable-debug-logs) above.
2. Check the Apigee Adapter for Envoy logs and verify that you see the following message:

   ```
   Error retrieving products: REQUEST_URI: no such host
   ```

   **Sample Debug Log Output:**

   ```
   2021-01-12T08:29:06.499Z        DEBUG   product/manager.go:188  retrieving products from: https://foo/remote-service/products
   2021-01-12T08:29:06.505Z        ERROR   product/manager.go:164  Error retrieving products: GET "https://foo/remote-service/pro
   ducts": dial tcp: lookup foo on 169.254.169.254:53: no such host
   github.com/apigee/apigee-remote-service-golib/product.(*manager).start.func1
           /go/pkg/mod/github.com/apigee/apigee-remote-service-golib@v1.4.0/product/manager.go:164
   github.com/apigee/apigee-remote-service-golib/util.(*Looper).Run
           /go/pkg/mod/github.com/apigee/apigee-remote-service-golib@v1.4.0/util/looper.go:87
   github.com/apigee/apigee-remote-service-golib/util.(*Looper).Start.func1
           /go/pkg/mod/github.com/apigee/apigee-remote-service-golib@v1.4.0/util/looper.go:59
   ```

   In this example, Apigee Adapter for Envoy was not able to communicate with the
   remote-service API proxy because the hostname provided in the remote server API Proxy
   URL is not valid as indicated by the error `no such host` .
3. If Apigee Adapter for Envoy logs contain a log entry with the message `no such
   host`, then this is the cause of the issue. If not, go to [Cause: Envoy proxy is unable to communicate with Apigee Adapter for Envoy](https://docs.apigee.com/api-platform/troubleshoot/envoy-adapter/envoy-proxy-fails-403-forbidden-error.md.txt#unable-to-communicate-with-envoy-adapter).

### Resolution

If the above errors are displayed in the Apigee Adapter for Envoy logs, perform the following
steps to resolve the issue:

1. Check the Apigee Adapter for Envoy configuration file and verify that the given
   remote-service API proxy URL is valid.

   If not, stop Apigee Adapter for Envoy, fix the remote-service API proxy URL in the
   configuration file, start Apigee Adapter for Envoy, and send another API request and
   verify the fix.

   **Sample configuration:**

   ```
   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: apigee-remote-service-envoy
     namespace: apigee
   data:
     config.yaml: |
       tenant:
         internal_api: https://istioservices.apigee.net/edgemicro
         remote_service_api: https://ORG-ENV.apigee.net/remote-service
         org_name: ORG
         env_name: ENV
         key: KEY
         secret: SECRET
         
   ```
2. Verify that the `remote-service` API proxy is deployed in the relevant Edge environment. If not, deploy the `remote-service` API proxy in the relevant Edge environment and try it again.
3. Verify network connectivity between the Apigee Adapter for Envoy and the `remote-service` API proxy endpoint. If there are any network connectivity issues found, contact your networking team and try to resolve the problem.

## Cause: Envoy proxy is unable to communicate with Apigee Adapter for Envoy

### Diagnosis

Perform the following steps to diagnose the issue:

1. Ensure you have enabled debug logs in Envoy. If not, stop Envoy and start it again,
   enabling debug logs. Then send another API request.

   **Standalone deployments:**

   ```
   envoy -c envoy-config.yaml -l debug
   ```

   **Kubernetes/Istio based deployments:**

   ```
   kubectl -n=istio-system get pods
   kubectl -n=istio-system exec -it INGRESS_GATEWAY_NAME bash -- curl -X POST localhost:15000/logging?connection=debug
   ```
2. Check the Apigee Adapter for Envoy logs and verify that there is a log entry with the message:

   ```
   connecting to APIGEE_ENVOY_ADAPTER_HOST:5000
   ```

   which is then followed by:

   ```
   upstream connect error or disconnect/reset before headers. reset reason: ACTUAL_REASON
   ```

   **Sample Debug Log Output:**

   ```
   [2021-03-23 05:44:41.867][1303661][debug][connection] [external/envoy/source/common/network/connection_impl.cc:769] [C4] connecting to 127.0.0.1:5000
   [2021-03-23 05:44:41.867][1303661][debug][connection] [external/envoy/source/common/network/connection_impl.cc:785] [C4] connection in progress
   [2021-03-23 05:44:41.868][1303661][debug][http2] [external/envoy/source/common/http/http2/codec_impl.cc:1173] [C4] updating connection-level initial window size to 268435456
   [2021-03-23 05:44:41.869][1303661][debug][connection] [external/envoy/source/common/network/connection_impl.cc:634] [C4] delayed connection error: 111
   [2021-03-23 05:44:41.869][1303661][debug][connection] [external/envoy/source/common/network/connection_impl.cc:203] [C4] closing socket: 0
   [2021-03-23 05:44:41.869][1303661][debug][client] [external/envoy/source/common/http/codec_client.cc:96] [C4] disconnect. resetting 0 pending requests
   [2021-03-23 05:44:41.869][1303661][debug][pool] [external/envoy/source/common/conn_pool/conn_pool_base.cc:314] [C4] client disconnected, failure reason:
   [2021-03-23 05:44:41.869][1303661][debug][router] [external/envoy/source/common/router/router.cc:1031] [C0][S6149963213555558594] upstream reset: reset reason: connection failure, transport failure reason:
   [2021-03-23 05:44:41.869][1303661][debug][http] [external/envoy/source/common/http/async_client_impl.cc:100] async http request response headers (end_stream=true):
   ':status', '200'
   'content-type', 'application/grpc'
   'grpc-status', '14'
   'grpc-message', 'upstream connect error or disconnect/reset before headers. reset reason: connection failure'
   ```

   The above example shows that Envoy was unable to communicate with
   Apigee Adapter for Envoy due to the reason `connection failure`.
3. The `connection failure` can be due to multiple reasons. Let's look at each of the scenarios.

### Scenario #1: Adapter process is not running

If the Apigee Adapter for Envoy process is not running, then this error can occur.

1. Verify that the Apigee Adapter for Envoy process is running by executing the following command. If the Apigee Adapter for Envoy process is running, the result of the following command should list it.

   ```
   ps -ef | grep apigee-remote-service-envoy
   ```
2. If it is not running, then that's the cause of the issue.

### Resolution

1. If the Apigee Adapter for Envoy process is not running, then start the Apigee Adapter for Envoy.
2. Make another API request and verify if the issue has been fixed.

### Scenario #2: Adapter process is not listening on the specific port

If the Apigee Adapter for Envoy process is not listening on the specific port,
then this error can occur.

If the Apigee Adapter for Envoy process is running, verify that there is a socket listening on
port 5000: <var translate="no">APIGEE_ENVOY_ADAPTER_HOST</var>`:5000`. You can run the
`netstat` command to verify this:

```
sudo netstat -lnp | grep 5000
```

**Sample output:**

```
sudo netstat -lnp | grep 5000

tcp6       0      0 :::5000                 :::*                    LISTEN      1596530/./apigee-re
```

If there is no socket listening on port 5000, that could be the reason for this issue.

### Resolution

1. Stop the Apigee Adapter for Envoy and start it again.
2. Make another API request and verify if the issue has been fixed.

### Scenario #3: Network Connectivity between Envoy and Apigee Adapter for Envoy

1. Verify network connectivity between Envoy and Apigee Adapter for Envoy:

   ```
   ssh $ENVOY_HOST
   telnet $APIGEE_ENVOY_ADAPTER_HOST 5000
   ```

   If telnet could make a TCP connection to Apigee Adapter for Envoy
   an output similar to following will be displayed:

   ```
   telnet $APIGEE_ENVOY_ADAPTER_HOST 5000

   Trying ::1...
   Connected to localhost.
   Escape character is '^]'.
   ```
2. If you observe the error `Connection timed out` with telnet, then that indicates there is a network connectivity issue between Envoy and Apigee Adapter for Envoy.

### Resolution


If you see any network connectivity issues between Envoy and Apigee Adapter for Envoy, please
engage your networking team and try to resolve the issue.


If the problem still persists, go to
[Must gather diagnostic information](https://docs.apigee.com/api-platform/troubleshoot/envoy-adapter/envoy-proxy-fails-403-forbidden-error.md.txt#must-gather).

## Must gather diagnostic information


If the problem persists after following the above instructions, gather the following diagnostic
information and then contact [Apigee Edge Support](https://docs.apigee.com/api-platform/troubleshoot/support.md.txt):

1. Apigee product used:

   **Example:** Apigee Edge Cloud, Apigee OPDK, Apigee hybrid, Apigee X
2. Apigee organization and environment
3. API product definition read using the Edge API:

   ```
   curl -i -u $USER:$PASSWORD $MANAGEMENT_SERVER_ENDPOINT/v1/organizations/$ORGANIZATION/apiproducts/$API_PRODUCT
   ```

   **Reference:** [Apigee Edge APIs](https://apidocs.apigee.com/management/apis)
4. Start a trace session in the `remote-service` API proxy using the
   Apigee Edge UI. Reproduce this problem and share the Trace session XML file.

   **Reference:** [Using the Trace tool \| Apigee Edge](https://docs.apigee.com/api-platform/debug/using-trace-tool-0.md.txt)
5. Apigee Adapter for Envoy logs (complete logs related to the given time period)

   **Standalone deployments:**

   ```
   # by default Apigee Envoy write logs to stdout and stderr, check your deployment configuration and collect logs accordingly
   ```

   **Kubernetes/Istio based deployments:**

   ```
   kubectl -n=apigee get pods
   kubectl -n=apigee logs APIGEE_REMOTE_SERVICE_ENVOY_POD_NAME > apigee-remote-service-envoy.log
   ```
6. An API request sent to the Envoy proxy using a `curl` command (the complete output of the `curl` command):

   ```
   curl -v ENVOY_PROXY_ENDPOINT
   ```
7. An API request sent to the target service using a `curl` command (the complete output of the `curl` command):

   ```
   curl -v TARGET_SERVICE_ENDPOINT
   ```