Postgres server running out of disk space

You're viewing Apigee Edge documentation.
Go to the Apigee X documentation. info

Symptom

The Postgres Server containing the Analytics data has run out of disk space.

In the following example, you can see that the disk /u01 has filled up 90% (176GB/207GB) of the disk space.

$df -g

Filesystem Size User Avail Use% Mounted on
/dev/mapper/sysvg-syslv09 207G 176G 176G 21G 90% /u01

Error messages

You may not observe any error message unless the disk space is completely filled on the Postgres Server.

Possible causes

The following table lists possible causes of this issue:

Inadequate disk space

Diagnosis

One of the typical causes for disk space errors on Postgres Servers is that you don't have adequate disk space to store the large volumes of analytics data. The steps provided below will help you to determine if you have enough disk space or not and take appropriate action to address the issue.

1. Determine the rate of incoming API traffic to Edge by referring to the Analytics Proxy Performance Dashboard.

   Sample Proxy Performance showing average TPS

   

2. Consider the following scenario:
   1. The incoming API traffic for your org is 22 TPS (transactions per second).
      1. This means that the API traffic is 1,900,800 transactions per day (22 * 60 * 60 * 24).
      2. Note each transaction/message in Analytics is 1.5K bytes in size.
      3. Therefore, each day generates 2.7GB of Analytics data (1,900,800 * 1.5 K).
   2. You have a requirement to retain 30 days worth of Analytics data on your Postgres Servers for reference.
      1. The total data generated for 30 days = 81GB (2.7GB * 30)
   3. Therefore, to store 30 days worth of Analytics data at traffic rate of 22 TPS, you need to have 150 GB of disk space.
      1. 81GB (Analytics data) + 50GB (other data such as logs, etc) + 20GB (additional buffer space) = 150GB.
3. If you have less disk space on the system i.e., less than 150 GB of space (as per the example scenario above), then you don't have adequate disk space to store the Analytics data.

Resolution

Add adequate disk space to the Postgres Server machine.

Lack of Analytics data pruning

Diagnosis

With the increase in API traffic to Edge, the amount of analytics data getting stored in the Postgres database will also increase. The amount of analytics data that can be stored in Postgres database is limited by the amount of disk space available on the system.

Therefore, you cannot continue to keep storing additional analytics data on the Postgres database without taking one of the following actions:

1. Add more disk space.

   This is not a scalable option as we can't keep adding more disk space as it limited and expensive.

2. Prune the data beyond the required retention interval.

   This is a preferred solution as you can ensure that the data that is no longer required is being removed at regular intervals of time.

If you don't prune the data at regular intervals manually or by using a cron job, then the amount of analytics data continually increases and can eventually lead to your running out of disk space on the system.

Resolution

To prune the data that is beyond your required retention interval:

1. Determine the retention interval, that is the duration for which you want to retain the Analytics data in the Postgres Database.
2. Run the following command to prune data for a specific organization and environment:

   /opt/apigee/apigee-service/bin/apigee-service apigee-postgresql pg-data-purge
     org env number_of_days_to_retain [Delete-from-parent-fact - N/Y] [Skip-confirmation-prompt - N/Y]
   

The script has the following options:

• Delete-from-parent-fact Default : No. Will also delete data older than retention days from parent fact table.
• Skip-confirmation-prompt. Default: No. If No, the script will prompt for confirmation before deleting data from parent fact. Set to Yes if the purge script is automated.

For more information, see Pruning Analytics data.

If the problem persists, contact Apigee Edge Support.