Custom Dimensions not appearing when multiple axgroups have been configured

You're viewing Apigee Edge documentation.
Go to the Apigee X documentation. info

Symptom

The custom variable created using the Statistics Collector policy is not visible under Custom Dimensions in the Analytics Custom Reports in the Edge UI.

Error Message

The edge-postgres-server system log will have the following error after a restart for each organization/environment combination where the path is missing:

KeeperErrorCode = NoNode for
/organizations/<ORG>/environments/<ENV>/properties
2018-03-29 16:14:48,980 pool-10-thread-2 ERROR ZOOKEEPER -
ZooKeeperServiceImpl.getData() : Could not get data for path:
/organizations//environments//properties, reason:
KeeperErrorCode = NoNode for
/organizations//environments//properties

Possible Causes

Cause: ZooKeeper properties path is missing for organization and environment combination

Diagnosis

Analytics is enabled for an organization-environment combination through the Enable analytics API, which populates the /organizations//environments//properties in the ZooKeeper tree. When this API fails, the edge-postgres-server component cannot create custom dimensions, and the following error will appear in its system.log:

KeeperErrorCode = NoNode for
/organizations/example/environments/prod/properties
2018-03-29 16:14:48,980 pool-10-thread-2 ERROR ZOOKEEPER -
ZooKeeperServiceImpl.getData() : Could not get data for path:
/organizations/example/environments/prod/properties, reason:
KeeperErrorCode = NoNode for
/organizations/digi/environments/sandbox/properties

Typically this happens when an organization was on-boarded at a time when more than one analytics group existed that used the same postgres-server UUIDs. To check how many analytics groups exist, run the following API call:

curl -u username:password "http://management-server-host:8080/v1/analytics/groups/ax"

If this API returns multiple groups, then it is most likely the cause of the issue. In the following example, notice that two groups exist, differentiated by only a hyphen: axgroup-001 and axgroup001

[ {
  "name" : "axgroup-001",
  "properties" : {
  },
  "scopes" : [ "VALIDATE~test" ],
  "uuids" : {
    "qpid-server" : [ "94c96375-1ca7-412d-9eee-80fda94f6e07" ],
    "postgres-server" : [ "8ee86b70-5b33-44b6-b2f8-1b0ec0ec8d77" ]
  },
  "consumer-groups" : [ {
    "name" : "consumer-group-001",
    "consumers" : [ "94c96375-1ca7-412d-9eee-80fda94f6e07" ],
    "datastores" : [ "8ee86b70-5b33-44b6-b2f8-1b0ec0ec8d77" ],
    "properties" : {
    }
  } ],
  "data-processors" : {
  }
}, {
  "name" : "axgroup001",
  "properties" : {
  },
  "scopes" : [ "example~prod" ],
  "uuids" : {
    "qpid-server" : [ "94c96375-1ca7-412d-9eee-80fda94f6e07" ],
    "postgres-server" : [ "8ee86b70-5b33-44b6-b2f8-1b0ec0ec8d77" ]
  },
  "consumer-groups" : [ {
    "name" : "consumer-group-001",
    "consumers" : [ "94c96375-1ca7-412d-9eee-80fda94f6e07" ],
    "datastores" : [ "8ee86b70-5b33-44b6-b2f8-1b0ec0ec8d77" ],
    "properties" : {
    }
  } ],
  "data-processors" : {
  }

Specifically, the issue is caused if the postgres-server UUID that is configured for each axgroup is the same. A quick way to determine this is to run the following:

curl -u username:password "http://management-server-host:8080/v1/analytics/groups/ax"|grep -B7 "postgres-server"

The following sample output quickly compare the postgres-server UUIDs to determine they are the same:

{  "name" : "axgroup-001",  "properties" : {  },  "scopes" : [ "VALIDATE~test" ],
"uuids" : {    "qpid-server" : [ "94c96375-1ca7-412d-9eee-80fda94f6e07" ],
"postgres-server" : [ "8ee86b70-5b33-44b6-b2f8-1b0ec0ec8d77" ]--
"name" : "axgroup001",  "properties" : {  },  "scopes" : [ "myorg~prod" ],
"uuids" : {    "qpid-server" : [ "94c96375-1ca7-412d-9eee-80fda94f6e07" ],
"aries-datastore" : [ ],    "postgres-server" : [ "8ee86b70-5b33-44b6-b2f8-1b0ec0ec8d77" ],

Resolution

The same set of PostgreSQL servers cannot be assigned to two axgroups. The quickest way to resolution is to delete the postgres server information from one axgroup and assign all scopes (organization, environment combinations) to the other group. To do this, follow these steps:

1. Remove the postgres server components from one of the two axgroups using the instructions at Adding and deleting analytics components in analytics groups. If you are seeing problems only with non-production scopes, then choose the axgroup which does NOT have any production scope for this exercise. Otherwise, choose the axgroup with the least number of scopes.
2. Remove all scopes from the axgroup that no longer has any postgres servers, and those for which you are seeing the custom dimension issue from their axgroup. Assuming axgroup-001 from the example above is the group you want to use, you would have to use the following to remove the scope for the myorg organization, and the prod environment:

   curl -u username:password -X DELETE 'http://management-server-host:8080/v1/analytics/groups/ax/axgroup001/scopes?org=myorg&env=prod'
   

3. Enable analtyics using the Enable analytics API.
4. Restart the edge-postgres-server component on the postgres master node, and all edge-qpid-server components you may have installed, using the following commands:

    /opt/apigee/apigee-service/bin/apigee-service edge-postgres-server restart
   

    /opt/apigee/apigee-service/bin/apigee-service edge-qpid-server restart
   

5. Verify that the error provided in the Error Message section above no longer occurs in the Postgres server's log file /opt/apigee/var/log/edge-postgres-server/logs/system.log.
6. Confirm if the custom dimensions show up in the Edge UI.

If the problem persists, go to Must Gather Diagnostic Information.

Must Gather Diagnostic Information

If the problem persists even after following the previous instructions, please gather the following diagnostic information. Contact and share them to Apigee Edge Support:

1. The edge-postgres-server /opt/apigee/var/log/edge-postgres-server/logs/system.log from the time of the latest restart.
2. The output of the following Management API call:

   curl -u username:password "http://management-server-host:8080/v1/analytics/groups/ax"
   

3. The output of the ZooKeeper tree obtained using the following command:

   /opt/apigee/apigee-zookeeper/contrib/zk-tree.sh > zktree-output.txt