Troubleshooting OpenLDAP Problems

You're viewing Apigee Edge documentation.
Go to the Apigee X documentation. info

This section provides information and guidance on troubleshooting OpenLDAP problems.

SMTP is disabled and users need to reset password

Symptom

When SMTP is not set up on the Edge UI, new users added to Edge need a way to set a password.

Error Messages

Unknown username and password combination.

Possible Causes

New users are unable to get an email from the “Forgot your password?” link to set a password because SMTP is not set up.

Resolution

You can address this issue in one of the following ways:

Solution #1: Configure SMTP Server

Configure the SMTP Server to set a new password for the user using the instructions provided in the documentation.

Solution #2: Using LDAP

If you are unable to configure the SMTP server, then use the below LDAP commands to set the new password for a user:

An existing org admin needs to add the specific user via the Edge UI as shown below:

Use the ldapsearch command to find the user’s distinguished name (dn) and redirect the output to a file:

ldapsearch -w Secret123 -D "cn=manager,dc=apigee,dc=com" -b "dc=apigee,dc=com" -LLL -h localhost -p 10389 > ldap.txt

Here is an example of a dn entry for a user, along with the attributes for the user:

dn:uid=f7a4a4a5-7c43-4168-a47e-6e9a1417cc29,ou=users,ou=global,dc=apigee,dc=com
mail: apigee_validator@apigee.com
userPassword:: e1NTSEF9b0FrMFFXVmFjbWRxM1BVaFZzMnllWGZMdkNvNjMwNTJlUDZYN3c9PQ=
 =
uid: f7a4a4a5-7c43-4168-a47e-6e9a1417cc29
objectClass: inetOrgPerson
sn: Validator
cn: apigee

Open the ldap.txt file and find the dn of the new user that has been added based on the new user's email attribute.

Execute the ldappassword command to add a password for the new user using its dn. In this example, you are setting the user's password to Apigee123:

ldappasswd -h localhost -p 10389 -D "cn=manager,dc=apigee,dc=com" -W -s Apigee123
"uid=f7a4a4a5-7c43-4168-a47e-6e9a1417cc29,ou=users,ou=global,dc=apigee,dc=com"

Log in to the Edge UI as the new user with the password defined in the previous step. The user can set a new password once logged into the UI.

LDAP is not Replicating

Symptom

Many Edge installations have multiple data centers, for example DC-1 and DC-2. When logging into the Edge UI in DC-1 as an org admin you can view the list of users, but the same user list does not appear in the Edge UI in DC-2.

Error Messages

No errors appear, the Edge UI simply does not show the list of users that should have been replicated across all OpenLDAP servers.

Possible Causes

Typically the cause of this issue is a misconfigured OpenLDAP replication configuration, not the installation itself. Also, replication may break if the network between the OpenLDAP servers does not allow traffic on port 10389.

Diagnosis

Use the following steps to diagnose the problem:

Check if ldapsearch returns data from each OpenLDAP server:

ldapsearch -W -D "cn=manager,dc=apigee,dc=com" -b "dc=apigee,dc=com" -LLL -h <host-ip> -p 10389

Check if you can connect to each OpenLDAP node from the other OpenLDAP nodes on port 10389. If telnet is installed, use the following command:
```
telnet <OpenLDAP_Peer_IP> 10389
```
If telnet is not available, use netcat to check the connectivity as follows:
```
nc -vz <OpenLDAP_Peer_IP> 10389
```

Check the replication configuration in the following file:

/opt/apigee/data/apigee-openldap/slapd.d/cn=config/olcDatabase={2}bdb.ldif

The file should contain configuration like this:

olcSyncRepl: rid=001
  provider=ldap://__OTHER_LDAP_SERVER__/
  binddn="cn=manager,dc=apigee,dc=com"
  bindmethod=simple
  credentials=__LDAP_PASSWORD__
  searchbase="dc=apigee,dc=com"
  attrs="*,+"
  type=refreshAndPersist
  retry="60 1 300 12 7200 +"
  timeout=1

Also check the same file for the value of the olcMirrorMode attribute. It should be set to the value TRUE:
```
grep olcMirrorMode /opt/apigee/data/apigee-openldap/slapd.d/cn=config/olcDatabase={2}bdb.ldif
```
Check for iptables and tcp wrapper rules. Please remove any rules that do not allow the peer OpenLDAP servers to communicate with each other. Work with your network administrator to set the rules appropriately.
Make sure the OpenLDAP system password is the same on each OpenLDAP node.
Check for hidden characters in the ldif configuration files that are being used to configure N-Way OpenLDAP replication by running dos2unix against the ldif files that have been created to update the configuration. Typically a ldif file that has bad characters would cause the ldapmodify command to fail to run and so replication may not be set up. Remove any bad characters and save the configuration files.

If the problem persists, contact Apigee Support for assistance with setting up N-Way OpenLDAP replication.

Unable to start OpenLDAP

Symptom

OpenLDAP does not start.

Error Messages

SLAPD Dead But Pid File Exists

Possible Causes

This issue is typically caused by a lock file that is left behind on the file system and needs to be removed.

Diagnosis

Use the following steps to diagnose this issue:

Check for an OpenLDAP slapd process lock or pid file in the following location:

/opt/apigee/var/run/apigee-openldap/apigee-openldap.lock
/opt/apigee/var/run/apigee-openldap/apigee-openldap.pid

Delete the lock and pid file, if found, and try to restart openldap.

rm /opt/apigee/var/run/apigee-openldap/apigee-openldap.lock
Rm /opt/apigee/var/run/apigee-openldap/apigee-openldap.pid

If the OpenLDAP slapd process starts, then skip the below steps.
If the OpenLDAP slapd process does not start, try running slapd in debug mode and look for any errors:
```
slapd -h ldap://:10389/ -u apigee -F /opt/apigee/data/apigee-openldap/slapd.d -d 255
```
Errors may point to resource issues. Check memory and CPU utilization on the system.
Check the version of OpenLDAP and upgrade if it is old. Check for the supported versions of OpenLDAP in our Supported Software document.
```
slapd -V
```

Use strace to troubleshoot slapd process, and to provide strace output to Apigee Support:

strace -tt -T -f -F -i -v -e read=all -s 8192 -e write=all -o /tmp/strace.out -p <pid>

OpenLDAP Data Corruption

Symptom

Users are no longer able to run management calls or log in to the Edge UI. Using ldapsearch utility to query the users may indicate the user exists in the LDAP datastore, or may identify possible missing users or roles.

Error Messages

Unknown username and password combination.

Possible Causes

Typically, this issue may be observed due to corruption of OpenLDAP data. Usually, OpenLDAP data does not get corrupted. But in the rare case that it does, the corruption could be due to system disk failure or disk space issues.

Diagnosis

Check the disk space on the system having OpenLDAP installed using the below command:
```
du -m /opt
```
If you see the disk space used is very close to 100%, then that would indicate the cause for this issue is your system running out of disk space.

Resolution

If your system has run out of disk space or very near to running out of disk space, then add more disk space to ensure you have sufficient disk space.

Once you have sufficient disk space use one of the below solutions to address the LDAP data corruption issue:

Restore the OpenLDAP data from the backup.
Clean up the OpenLDAP database.

Solution #1 Restore the LDAP data from the backup

On a working OpenLDAP node make a backup. The backup should be performed regularly. See Apigee Private Cloud Operations Guide for best practices on backups:

slapcat -F /opt/apigee/data/apigee-openldap/slapd.d -l /tmp/ldap-backup.ldif

The following steps can be used to restore the OpenLDAP data from a good backup.

Stop the OpenLDAP node for which the data needs to be restored:

/opt/apigee/apigee-service/bin/apigee-service apigee-openldap stop

Change directory to the OpenLDAP data directory:
```
cd /opt/apigee/data/apigee-openldap
```
Back up the existing OpenLDAP data using the move command:
```
mv ldap ldap_orig
```
Switch to the apigee user:
```
su apigee
```
From /opt/apigee/data/apigee-openldap directory, create a new OpenLDAP data directory with the original name:
```
mkdir ldap
```
Take the backup of ldap_orig/DB_CONFIG subdirectory from step 3, and copy it to the openldap directory.
```
cp ldap_orig/DB_CONFIG ldap
```
To restore data from backup taken with slapcat, use slapadd to import the ldif which contains the good data:
```
slapadd -F /opt/apigee/data/apigee-openldap/slapd.d -l /tmp/ldap-backup.ldif
```

Start the OpenLDAP process:

/opt/apigee/apigee-service/bin/apigee-service apigee-openldap start

Solution #2 Clean up the LDAP database

The following steps wipe out the OpenLDAP database to provide a fresh start. This solution can be used if there is no data backup of the last state where the OpenLDAP data was working.

Stop the OpenLDAP service:

/opt/apigee/apigee-service/bin/apigee-service apigee-openldap stop

Change directory to the OpenLDAP data directory:
```
cd /opt/apigee/data/apigee-openldap
```
Back up the existing OpenLDAP data using the move command:
```
mv ldap ldap_orig
```
Switch to the apigee user:
```
su apigee
```
Create a new OpenLDAP data directory with the original name:
```
mkdir ldap
```
Take the backup ldap_orig/DB_CONFIG subdirectory from step 3, and copy it to the openldap directory:
```
cp ldap_orig/DB_CONFIG ldap
```

Restart the OpenLDAP process:

/opt/apigee/apigee-service/bin/apigee-service apigee-openldap start

Restart the Management Server to force a refresh of the connections to OpenLDAP:
```
/opt/apigee/apigee-service/bin/apigee-service edge-management-server restart
```

Solution #3 Reset OpenLDAP to a basic installation

If Solution #2 does not resolve the issue, you can reset OpenLDAP to a basic installation, as described in this section.

Prerequisites

To reset OpenLDAP, you need the following prerequisites:

The ability to run the setup with system admin and root LDAP credentials.
Access to ldapadd utility.
The saved original silent configuration file for the management/LDAP node.

Note: Before resetting OpenLDAP, be sure to back up the /opt/apigee/customer/application configuration files.

To reset OpenLDAP, do the following steps:

Re-install OpenLDAP and the Management Server.
1. Stop the openldap server:
```
apigee-service apigee-openldap stop
```
2. Delete the corrupt openldap data folder:
```
rm -rf /opt/apigee/data/apigee-openldap
```
3. Uninstall the openldap component:
```
apigee-service apigee-openldap uninstall
```
4. Reinstall the openldap component using the same configuration file that was used for the initial installation:
```
/opt/apigee/apigee-setup/bin/setup.sh -p ld -f configfile
```
  where configfile is the name of the configuration file.
5. Re-install Management Server using the original configuration file:
```
/opt/apigee/apigee-setup/bin/setup.sh -p ms -f configfile
```
After doing these steps, Management API calls should work again using the sysadmin account only. However, it will not be possible to log into the Edge UI, and Management API calls with any other users will not work.

Add missing openldap entries for orgs that existed prior to the corruption.

After the above step is completed, openldap will be missing entries for orgs that existed when the corruption occurred. The majority of the actual org data such as proxy bundles is stored in Cassandra or Zookeeper is not lost. However, running the setup-org script will not automatically add the openldap data for the orgs that already exist in Cassandra, and Zookeeper. This data has to be added manually for every org that existed prior to the corruption using the following steps:

Create an ldif file called missingLDAP.ldif with the following content:

# orgname, organizations, apigee.com
dn: o=orgname,ou=organizations,dc=apigee,dc=com
objectClass: organization
O: orgname

# userroles, orgname, organizations, apigee.com
dn: ou=userroles,o=orgname,ou=organizations,dc=apigee,dc=com
ou: userroles
objectClass: organizationalUnit

# orgadmin, userroles, orgname, organizations, apigee.com
dn: cn=orgadmin,ou=userroles,o=orgname,ou=organizations,dc=apigee,dc=com
objectClass: organizationalRole
cn: orgadmin
roleOccupant: uid=admin,ou=users,ou=global,dc=apigee,dc=com

# resources, orgadmin, userroles, orgname, organizations, apigee.com
dn: ou=resources,cn=orgadmin,ou=userroles,o=orgname,ou=organizations,dc=apigee,dc=com
ou: resources
objectClass: organizationalUnit

# @@@, resources, orgadmin, userroles, orgname, organizations, apigee.com
dn: cn=@@@,ou=resources,cn=orgadmin,ou=userroles,o=orgname,ou=organizations,dc=apigee,dc=com
roleOccupant: ou=delete,ou=permissions,dc=apigee,dc=com
roleOccupant: ou=get,ou=permissions,dc=apigee,dc=com
roleOccupant: ou=put,ou=permissions,dc=apigee,dc=com
labeledURI: /
objectClass: organizationalRole
objectClass: labeledURIObject
cn: @@@

where orgname is the organization that you are trying to recreate.

Add the missing LDAP entities using the following command:

ldapadd -x -w  -D "cn=manager,dc=apigee,dc=com" -H ldap://localhost:10389 -f missingLDAP.ldif

The above steps will create the missing entries for an existing org, and the orgadmin permissions for that org. Repeat the steps for each org that needs to be recreated. At this point, you can add orgadmin users to an org, but the other default roles do not exist yet, so you need to add them using the next step.

Add missing default roles and permissions to an existing org.

Using the same configuration file that was initially used to set up any of the existing orgs, run the following command:
```
/opt/apigee/apigee-service/bin/apigee-service apigee-provision create-roles -f configfile
```

After following this procedure, you need to manually:

Add any custom roles that existed prior to the corruption using the UI or management API,
Add any users to the relevant user roles.

(Optional) Reestablish Externalized Authentication Configuration on Management.

Restore the /opt/apigee/customer/application/management-server.properties if the configuration has not changed. Since the UI has not been reinstalled, the /opt/apigee/customer/application/ui.properties should remain the same given the credentials for the sysadmin has been restored with the new install of management.
(Optional) Reinstall Apigee mTLS on Management node.

If Apigee mTLS was previously installed, follow the Apigee mTLS Install Guide to reinstall it on the management server node.

If the problem persists, contact Apigee Support for further assistance.