You're viewing **Apigee Edge** documentation. [Go to the **Apigee X** documentation](http://cloud.google.com/apigee/docs). info > [!NOTE] > Was this troubleshooting playbook helpful? Please let us know by clicking [Send Feedback](). ## Symptom Analytics data is missing in the New Edge UI due to the Qpidd server not transferring analytics messages to PostgreSQL. In Edge, the `edge-qpid-server` component corresponds to the Qpidd server. Qpidd maintains two queues for each analytics group: - **ax-q-axgroup001-consumer-group-001** This queue holds analytics messages pushed from the Message Processors and Routers. Messages get pulled from here by the `edge-qpid-server` which parses messages and inserts them into PostgreSQL. Once messages are processed successfully they are removed from the queue. - **ax-q-axgroup001-consumer-group-001-dl** This queue is the dead letter queue. It acts as the destination for messages that `edge-qpid-server` failed to process and thus no longer wishes to receive. This typically gets populated when the maximum delivery count is exceeded, or if PostgreSQL rejected insertion of new data due to runtime errors. ## Error Message The root cause could be due to various runtime errors from the `edge-qpid-server` component. Typically if `edge-qpid-server` receives a runtime error from PostgreSQL, it creates the dead letter queue if it doesn't exist already, and then sends the following message there: ``` yyyy-MM-dd HH:mm:ss,SSS ax-q-axgroup001-consumer-group-001-persistpool-thread-6 WARN c.a.a.m.MessageConsumer - MessageConsumer.process() : Sending message batch to the DLQ. ``` ## Possible Causes | Cause | Description | Troubleshooting Instructions Applicable For | |---|---|---| | [Messages stuck in dead letter queue of qpidd](https://docs.apigee.com/api-platform/troubleshoot/analytics/analytics-data_stuck_qpidd#cause_messages_stuck_in_dead_letter_queue_of_qpidd) | `edge-qpid-server` could not understand messages that it read from Qpidd broker or was unable to persist messages to PostgreSQL. | Edge Private Cloud Users | ## Common Diagnosis Steps Run the following command on the Qpid node to view Qpidd queue stats: ``` /opt/apigee/apigee-qpidd/bin/qpid-stat -q ``` > [!NOTE] > **Note:** On Edge for Private Cloud 4.52.x and earlier, you can also run `qpid-stat -q` directly if `/usr/bin/qpid-stat` is in your `PATH`. The output returns the set of queues registered with the broker. If the queue with a name ending in `-dl` has messages populated (`Q-Depth(Messages)` in Edge for Private Cloud 4.53.00 and later, or `msg` in Edge for Private Cloud 4.52.x and earlier), then messages are stuck in the dead letter queue. **Example output in Edge for Private Cloud 4.53.00 and later (including 4.54.00, Apache Qpid Broker-J):** ``` +---+---+---+---+---+---+---+---+---+ | Queue Name | Messages Enqueued | Total Enqueued(B) | Messages Dequeued | Total Dequeued(B) | Consumers | Bindings | Q-Depth(Messages) | Q-Depth(B) | +---+---+---+---+---+---+---+---+---+ | ax-q-axgroup001-consumer-group-001 | 185 | 13800000 | 185 | 13800000 | 6 | 1 | 0 | 0 | +---+---+---+---+---+---+---+---+---+ | ax-q-axgroup001-consumer-group-001-dl | 70 | 3900000 | 0 | 0 | 0 | 1 | 70 | 3900000 | +---+---+---+---+---+---+---+---+---+ ``` **Example output in Edge for Private Cloud 4.52.x and earlier (legacy C++ `qpidd` broker):** ``` Queues queue dur autoDel excl msg msgIn msgOut bytes bytesIn bytesOut cons bind ======================================================================================================================== ax-q-axgroup-001-consumer-group-001 Y 0 185 185 0 13.8m 13.8m 6 2 ax-q-axgroup-001-consumer-group-001-dl Y 0 70 70 0 3.9m 3.9m 0 2 ``` ## Cause: Messages stuck in dead letter queue of qpidd ### Diagnosis This condition can happen under the following scenarios: 1. An upgrade had taken place in the past, during which time PostgreSQL was down. 2. A temporary outage of PostgreSQL due to network issues. 3. The `edge-qpid-server` attempted to send a message to PostgreSQL but PostgreSQL returned a runtime error. ### Resolution Follow the steps for your Edge for Private Cloud version to move stuck messages from the dead letter queue back to the main analytics queue. #### Edge for Private Cloud 4.53.00 and later (including 4.54.00) Edge for Private Cloud 4.53.00 and later (including 4.54.00) uses Apache Qpid Broker-J (`10.0.1` in 4.54.00 and `8.0.6` in 4.53.x). The legacy `qpid-tool` CLI is removed in Edge for Private Cloud 4.53.00 and later and cannot be used. Instead, use the Apache Qpid Broker-J HTTP Management REST API (`moveMessages`) or the Qpid Web Management Console to move messages from the dead letter queue back to the main analytics queue: 1. Note down the exact names of the main queue and dead letter queue from the [Common Diagnosis Steps](https://docs.apigee.com/api-platform/troubleshoot/analytics/analytics-data_stuck_qpidd#common_diagnosis_steps) output (for example, `ax-q-axgroup001-consumer-group-001` and `ax-q-axgroup001-consumer-group-001-dl`, or `ax-q-axgroup-001-consumer-group-001` and `ax-q-axgroup-001-consumer-group-001-dl`). 2. On the Qpid node, run the following `curl` command to invoke the `moveMessages` operation on the dead letter queue through the Qpid Broker-J HTTP Management API (port `8090` by default): ``` curl -u QPID_USER:QPID_PASSWORD -X POST \ -H "Content-Type: application/json" \ -d '{"destination": "ax-q-axgroup001-consumer-group-001"}' \ http://127.0.0.1:8090/api/latest/queue/default/default/ax-q-axgroup001-consumer-group-001-dl/moveMessages ``` Where: - `QPID_USER` and `QPID_PASSWORD` are the Qpid management credentials configured in `/opt/apigee/customer/application/qpidd.properties` or `/opt/apigee/token/application/qpidd.properties` (the default is `guest:guest`). - Replace `ax-q-axgroup001-consumer-group-001` and `ax-q-axgroup001-consumer-group-001-dl` if your analytics queue names in `qpid-stat -q` differ. - To move messages in batches, include `"limit"` in the JSON request body (for example, `"limit": 100000`). - If SSL/TLS is enabled on the Qpid Management API (`conf_qpidd_qpid_management.https.enabled=true`), use `https://` and pass `--cacert` (or `-k`) to `curl`. 3. Verify the command output. A successful call returns a JSON array of the moved internal message IDs: ``` [ 3, 4, 5 ] ``` If the returned array is empty (`[]`), there were no messages in the dead letter queue to move. > [!NOTE] > **Note:** Alternatively, if you have enabled the [Qpid Web > Management Console](https://docs.apigee.com/private-cloud/latest/configure-qpid-management-portal) (`http://QPID_NODE_IP:8090`), you can navigate to **Java-Broker \> virtualhosts \> default \> queues** , select `ax-q-axgroup001-consumer-group-001-dl`, select the stuck messages, click **Move Messages** , and choose `ax-q-axgroup001-consumer-group-001` as the destination queue. 4. Wait 5 minutes and then run `/opt/apigee/apigee-qpidd/bin/qpid-stat -q` from [Common Diagnosis Steps](https://docs.apigee.com/api-platform/troubleshoot/analytics/analytics-data_stuck_qpidd#common_diagnosis_steps). Verify that the messages on the main queue (`ax-q-axgroup001-consumer-group-001`) are getting processed and that `Q-Depth(Messages)` on the dead letter queue (`ax-q-axgroup001-consumer-group-001-dl`) remains at `0`. #### Edge for Private Cloud 4.52.x and earlier 1. Note down the name of the queues from the [Common Diagnosis Steps](https://docs.apigee.com/api-platform/troubleshoot/analytics/analytics-data_stuck_qpidd#common_diagnosis_steps). For example: - `ax-q-axgroup-001-consumer-group-001` - `ax-q-axgroup-001-consumer-group-001-dl` 2. Run the `qpid-tool` command to enter an interactive `qpid` prompt: ``` qpid-tool ``` This command returns the following: ``` Management Tool for QPID qpid: ``` 3. Run `list broker` to obtain a list of active brokers: ``` list broker ``` This command returns the following: ``` Object Summary: ID Created Destroyed Index ======================================= 125 21:00:00 - amqp-broker ``` Where the `ID` column specifies the ID of the broker. 4. Note the ID of the broker. In the example it is `125`. 5. Run the following command to move the messages from the dead letter queue back to the actual queue: ``` call 125 queueMoveMessages ax-q-axgroup-001-consumer-group-001-dl ax-q-axgroup-001-consumer-group-001 100000 {} ``` This command returns the following: ``` OK (0) - {} ``` If there is no output then there was nothing to do, meaning no messages to move. If you don't see `OK (0)` then you should contact [Apigee Edge Support](https://docs.apigee.com/api-platform/troubleshoot/support). 6. Quit the `qpid-tool` terminal: ``` quit ``` 7. Wait 5 minutes and then run the diagnosis steps again from [Common Diagnosis Steps](https://docs.apigee.com/api-platform/troubleshoot/analytics/analytics-data_stuck_qpidd#common_diagnosis_steps). Verify that the messages on the actual queue are getting processed and ensure the dead letter `msg` count remains at `0`. If the problem still persists, go to the next section. ## Must Gather Diagnostic Information If the problem persists after following the preceding instructions, gather the following diagnostic information and share it with [Apigee Edge Support](https://docs.apigee.com/api-platform/troubleshoot/support): - Qpidd logs: `/opt/apigee/var/log/apigee-qpidd/qpidd.log` (Edge for Private Cloud 4.53.00 and later) and `/opt/apigee/var/log/apigee-qpidd/apigee-qpidd.log` - PostgreSQL logs: `/opt/apigee/var/log/apigee-postgresql/apigee-postgresql.log` - Edge-qpid-server logs: `/opt/apigee/var/log/edge-qpid-server/logs/system.log` - Edge-postgres-server logs: `/opt/apigee/var/log/edge-postgres-server/logs/system.log` - Qpidd queue stats: ``` /opt/apigee/apigee-qpidd/bin/qpid-stat -q ``` - Analytics group returned by the following curl command: ``` curl -u sysadminEmail:password http://mgmt:8080/v1/analytics/groups/ax ```