Analytics data stuck in Qpidd dead letter queue

You're viewing Apigee Edge documentation.
Go to the Apigee X documentation. info

Symptom

Analytics data is missing in the New Edge UI due to the Qpidd server not transferring analytics messages to PostgreSQL. In Edge, the edge-qpid-server component corresponds to the Qpidd server.

Qpidd maintains two queues for each analytics group:

ax-q-axgroup001-consumer-group-001

This queue holds analytics messages pushed from the Message Processors and Routers. Messages get pulled from here by the edge-qpid-server which parses messages and inserts them into PostgreSQL. Once messages are processed successfully they are removed from the queue.
ax-q-axgroup001-consumer-group-001-dl

This queue is the dead letter queue. It acts as the destination for messages that edge-qpid-server failed to process and thus no longer wishes to receive. This typically gets populated when the maximum delivery count is exceeded, or if PostgreSQL rejected insertion of new data due to runtime errors.

Error Message

The root cause could be due to various runtime errors from the edge-qpid-server component. Typically if edge-qpid-server receives a runtime error from PostgreSQL, it creates the dead letter queue if it doesn't exist already, and then sends the following message there:

yyyy-MM-dd HH:mm:ss,SSS ax-q-axgroup001-consumer-group-001-persistpool-thread-6 WARN c.a.a.m.MessageConsumer - MessageConsumer.process() : Sending message batch to the DLQ.

Possible Causes

Cause	Description	Troubleshooting Instructions Applicable For
Messages stuck in dead letter queue of qpidd	`edge-qpid-server` could not understand messages that it read from Qpidd broker or was unable to persist messages to PostgreSQL.	Edge Private Cloud Users

Common Diagnosis Steps

Run the following command on the Qpid node to view Qpidd queue stats:

/opt/apigee/apigee-qpidd/bin/qpid-stat -q

Note: On Edge for Private Cloud 4.52.x and earlier, you can also run qpid-stat -q directly if /usr/bin/qpid-stat is in your PATH.

The output returns the set of queues registered with the broker. If the queue with a name ending in -dl has messages populated (Q-Depth(Messages) in Edge for Private Cloud 4.53.00 and later, or msg in Edge for Private Cloud 4.52.x and earlier), then messages are stuck in the dead letter queue.

Example output in Edge for Private Cloud 4.53.00 and later (including 4.54.00, Apache Qpid Broker-J):

+---------------------------------------+-------------------+-------------------+-------------------+-------------------+-----------+----------+-------------------+------------+
|               Queue Name              | Messages Enqueued | Total Enqueued(B) | Messages Dequeued | Total Dequeued(B) | Consumers | Bindings | Q-Depth(Messages) | Q-Depth(B) |
+---------------------------------------+-------------------+-------------------+-------------------+-------------------+-----------+----------+-------------------+------------+
|   ax-q-axgroup001-consumer-group-001  |        185        |      13800000     |        185        |      13800000     |     6     |    1     |         0         |     0      |
+---------------------------------------+-------------------+-------------------+-------------------+-------------------+-----------+----------+-------------------+------------+
| ax-q-axgroup001-consumer-group-001-dl |         70        |      3900000      |         0         |         0         |     0     |    1     |         70        |  3900000   |
+---------------------------------------+-------------------+-------------------+-------------------+-------------------+-----------+----------+-------------------+------------+

Example output in Edge for Private Cloud 4.52.x and earlier (legacy C++ qpidd broker):

Queues
  queue                                     dur  autoDel  excl  msg   msgIn  msgOut  bytes  bytesIn  bytesOut   cons  bind
  ========================================================================================================================
  ax-q-axgroup-001-consumer-group-001       Y                   0     185    185     0       13.8m   13.8m      6      2
  ax-q-axgroup-001-consumer-group-001-dl    Y                   0     70     70      0        3.9m    3.9m      0      2

Cause: Messages stuck in dead letter queue of qpidd

Diagnosis

This condition can happen under the following scenarios:

An upgrade had taken place in the past, during which time PostgreSQL was down.
A temporary outage of PostgreSQL due to network issues.
The edge-qpid-server attempted to send a message to PostgreSQL but PostgreSQL returned a runtime error.

Resolution

Follow the steps for your Edge for Private Cloud version to move stuck messages from the dead letter queue back to the main analytics queue.

Edge for Private Cloud 4.53.00 and later (including 4.54.00)

Edge for Private Cloud 4.53.00 and later (including 4.54.00) uses Apache Qpid Broker-J (10.0.1 in 4.54.00 and 8.0.6 in 4.53.x). The legacy qpid-tool CLI is removed in Edge for Private Cloud 4.53.00 and later and cannot be used. Instead, use the Apache Qpid Broker-J HTTP Management REST API (moveMessages) or the Qpid Web Management Console to move messages from the dead letter queue back to the main analytics queue:

Note down the exact names of the main queue and dead letter queue from the Common Diagnosis Steps output (for example, ax-q-axgroup001-consumer-group-001 and ax-q-axgroup001-consumer-group-001-dl, or ax-q-axgroup-001-consumer-group-001 and ax-q-axgroup-001-consumer-group-001-dl).
On the Qpid node, run the following curl command to invoke the moveMessages operation on the dead letter queue through the Qpid Broker-J HTTP Management API (port 8090 by default):
```
curl -u QPID_USER:QPID_PASSWORD -X POST \
  -H "Content-Type: application/json" \
  -d '{"destination": "ax-q-axgroup001-consumer-group-001"}' \
  http://127.0.0.1:8090/api/latest/queue/default/default/ax-q-axgroup001-consumer-group-001-dl/moveMessages
```
Where:
- QPID_USER and QPID_PASSWORD are the Qpid management credentials configured in /opt/apigee/customer/application/qpidd.properties or /opt/apigee/token/application/qpidd.properties (the default is guest:guest).
- Replace ax-q-axgroup001-consumer-group-001 and ax-q-axgroup001-consumer-group-001-dl if your analytics queue names in qpid-stat -q differ.
- To move messages in batches, include "limit" in the JSON request body (for example, "limit": 100000).
- If SSL/TLS is enabled on the Qpid Management API (conf_qpidd_qpid_management.https.enabled=true), use https:// and pass --cacert (or -k) to curl.
Verify the command output. A successful call returns a JSON array of the moved internal message IDs:
```
[ 3, 4, 5 ]
```
If the returned array is empty ([]), there were no messages in the dead letter queue to move.

Note: Alternatively, if you have enabled the Qpid Web Management Console (http://QPID_NODE_IP:8090), you can navigate to Java-Broker > virtualhosts > default > queues, select ax-q-axgroup001-consumer-group-001-dl, select the stuck messages, click Move Messages, and choose ax-q-axgroup001-consumer-group-001 as the destination queue.
Wait 5 minutes and then run /opt/apigee/apigee-qpidd/bin/qpid-stat -q from Common Diagnosis Steps. Verify that the messages on the main queue (ax-q-axgroup001-consumer-group-001) are getting processed and that Q-Depth(Messages) on the dead letter queue (ax-q-axgroup001-consumer-group-001-dl) remains at 0.

Edge for Private Cloud 4.52.x and earlier

Note down the name of the queues from the Common Diagnosis Steps. For example:
- ax-q-axgroup-001-consumer-group-001
- ax-q-axgroup-001-consumer-group-001-dl
Run the qpid-tool command to enter an interactive qpid prompt:
```
qpid-tool
```
This command returns the following:
```
Management Tool for QPID
qpid:
```

Run list broker to obtain a list of active brokers:

list broker

This command returns the following:

Object Summary:
ID   Created   Destroyed  Index
=======================================
125  21:00:00  -          amqp-broker

Where the ID column specifies the ID of the broker.

Note the ID of the broker. In the example it is 125.
Run the following command to move the messages from the dead letter queue back to the actual queue:
```
call 125 queueMoveMessages ax-q-axgroup-001-consumer-group-001-dl ax-q-axgroup-001-consumer-group-001 100000 {}
```
This command returns the following:
```
OK (0) - {}
```
If there is no output then there was nothing to do, meaning no messages to move. If you don't see OK (0) then you should contact Apigee Edge Support.
Quit the qpid-tool terminal:
```
quit
```
Wait 5 minutes and then run the diagnosis steps again from Common Diagnosis Steps. Verify that the messages on the actual queue are getting processed and ensure the dead letter msg count remains at 0.

If the problem still persists, go to the next section.

Must Gather Diagnostic Information

If the problem persists after following the preceding instructions, gather the following diagnostic information and share it with Apigee Edge Support:

Qpidd logs: /opt/apigee/var/log/apigee-qpidd/qpidd.log (Edge for Private Cloud 4.53.00 and later) and /opt/apigee/var/log/apigee-qpidd/apigee-qpidd.log
PostgreSQL logs: /opt/apigee/var/log/apigee-postgresql/apigee-postgresql.log
Edge-qpid-server logs: /opt/apigee/var/log/edge-qpid-server/logs/system.log
Edge-postgres-server logs: /opt/apigee/var/log/edge-postgres-server/logs/system.log

Qpidd queue stats:

/opt/apigee/apigee-qpidd/bin/qpid-stat -q

Analytics group returned by the following curl command:

curl -u sysadminEmail:password http://mgmt:8080/v1/analytics/groups/ax