Troubleshooting OpenLDAP Problems

This section provides information and guidance on troubleshooting OpenLDAP problems.

SMTP is disabled and users need to reset password

Symptom

When SMTP is not set up on the Edge UI, new users added to Edge need a way to set a password.

Error Messages

Unknown username and password combination.

Possible Causes

New users are unable to get an email from the “Forgot your password?” link to set a password because SMTP is not set up.

Resolution

You can address this issue in one of the following ways:

Solution #1: Configure SMTP Server

Configure the SMTP Server to set a new password for the user using the instructions provided in the documentation.

Solution #2: Using LDAP

If you are unable to configure the SMTP server, then use the below LDAP commands to set the new password for a user:

  1. An existing org admin needs to add the specific user via the Edge UI as shown below:

  1. Use the ldapsearch command to find the user’s distinguished name (dn) and redirect the output to a file:
    ldapsearch -w Secret123 -D "cn=manager,dc=apigee,dc=com" -b "dc=apigee,dc=com" -LLL -h localhost -p 10389 > ldap.txt
    

    Here is an example of a dn entry for a user, along with the attributes for the user:

    dn:uid=f7a4a4a5-7c43-4168-a47e-6e9a1417cc29,ou=users,ou=global,dc=apigee,dc=com
    mail: apigee_validator@apigee.com
    userPassword:: e1NTSEF9b0FrMFFXVmFjbWRxM1BVaFZzMnllWGZMdkNvNjMwNTJlUDZYN3c9PQ=
     =
    uid: f7a4a4a5-7c43-4168-a47e-6e9a1417cc29
    objectClass: inetOrgPerson
    sn: Validator
    cn: apigee
    
  1. Open the ldap.txt file and find the dn of the new user that has been added based on the new user's email attribute.
  1. Execute the ldappassword command to add a password for the new user using its dn. In this example, you are setting the user's password to Apigee123:
    ldappasswd -h localhost -p 10389 -D "cn=manager,dc=apigee,dc=com" -W -s Apigee123
    "uid=f7a4a4a5-7c43-4168-a47e-6e9a1417cc29,ou=users,ou=global,dc=apigee,dc=com"
    
  1. Log in to the Edge UI as the new user with the password defined in the previous step. The user can set a new password once logged into the UI.

LDAP is not Replicating

Symptom

Many Edge installations have multiple data centers, for example DC-1 and DC-2. When logging into the Edge UI in DC-1 as an org admin you can view the list of users, but the same user list does not appear in the Edge UI in DC-2.

Error Messages

No errors appear, the Edge UI simply does not show the list of users that should have been replicated across all OpenLDAP servers.

Possible Causes

Typically the cause of this issue is a misconfigured OpenLDAP replication configuration, not the installation itself. Also, replication may break if the network between the OpenLDAP servers does not allow traffic on port 10389.

Diagnosis

Use the following steps to diagnose the problem:

  1. Check if ldapsearch returns data from each OpenLDAP server:
    ldapsearch -W -D "cn=manager,dc=apigee,dc=com" -b "dc=apigee,dc=com" -LLL -h <host-ip> -p 10389
    
  1. Check if you can connect to each OpenLDAP node from the other OpenLDAP nodes on port 10389. If telnet is installed, use the following command:
    telnet <OpenLDAP_Peer_IP> 10389
    
  1. If telnet is not available, use netcat to check the connectivity as follows:

    nc -vz <OpenLDAP_Peer_IP> 10389
    
  2. Check the replication configuration in the following file:
    /opt/apigee/data/apigee-openldap/slapd.d/cn=config/olcDatabase={2}bdb.ldif
    

    The file should contain configuration like this:

    olcSyncRepl: rid=001
      provider=ldap://__OTHER_LDAP_SERVER__/
      binddn="cn=manager,dc=apigee,dc=com"
      bindmethod=simple
      credentials=__LDAP_PASSWORD__
      searchbase="dc=apigee,dc=com"
      attrs="*,+"
      type=refreshAndPersist
      retry="60 1 300 12 7200 +"
      timeout=1
    
  1. Also check the same file for the value of the olcMirrorMode attribute. It should be set to the value TRUE:
    grep olcMirrorMode /opt/apigee/data/apigee-openldap/slapd.d/cn=config/olcDatabase={2}bdb.ldif
    
  1. Check for iptables and tcp wrapper rules. Please remove any rules that do not allow the peer OpenLDAP servers to communicate with each other. Work with your network administrator to set the rules appropriately.
  1. Make sure the OpenLDAP system password is the same on each OpenLDAP node.
  1. Check for hidden characters in the ldif configuration files that are being used to configure N-Way OpenLDAP replication by running dos2unix against the ldif files that have been created to update the configuration. Typically a ldif file that has bad characters would cause the ldapmodify command to fail to run and so replication may not be set up. Remove any bad characters and save the configuration files.

If the problem persists, contact Apigee Support for assistance with setting up N-Way OpenLDAP replication.

Unable to start OpenLDAP

Symptom

OpenLDAP does not start.

Error Messages

SLAPD Dead But Pid File Exists

Possible Causes

This issue is typically caused by a lock file that is left behind on the file system and needs to be removed.

Diagnosis

Use the following steps to diagnose this issue:

  1. Check for an OpenLDAP slapd process lock or pid file in the following location:
    /opt/apigee/var/run/apigee-openldap/apigee-openldap.lock
    /opt/apigee/var/run/apigee-openldap/apigee-openldap.pid
    
  1. Delete the lock and pid file, if found, and try to restart openldap.
    rm /opt/apigee/var/run/apigee-openldap/apigee-openldap.lock
    Rm /opt/apigee/var/run/apigee-openldap/apigee-openldap.pid
    
  1. If the OpenLDAP slapd process starts, then skip the below steps.
  2. If the OpenLDAP slapd process does not start, try running slapd in debug mode and look for any errors:
    slapd -h ldap://:10389/ -u apigee -F /opt/apigee/data/apigee-openldap/slapd.d -d 255
    
  1. Errors may point to resource issues. Check memory and CPU utilization on the system.
  1. Check the version of OpenLDAP and upgrade if it is old. Check for the supported versions of OpenLDAP in our Supported Software document.
    slapd -V
    
  1. Use strace to troubleshoot slapd process, and to provide strace output to Apigee Support:
    strace -tt -T -f -F -i -v -e read=all -s 8192 -e write=all -o /tmp/strace.out -p <pid>
    

OpenLDAP Data Corruption

Symptom

Users are no longer able to run management calls or log in to the Edge UI. Using ldapsearch utility to query the users may indicate the user exists in the LDAP datastore, or may identify possible missing users or roles.

Error Messages

Unknown username and password combination.

Possible Causes

Typically, this issue may be observed due to corruption of OpenLDAP data. Usually, OpenLDAP data does not get corrupted. But in the rare case that it does, the corruption could be due to system disk failure or disk space issues.

Diagnosis

  1. Check the disk space on the system having OpenLDAP installed using the below command:
    du -m /opt
    
  1. If you see the disk space used is very close to 100%, then that would indicate the cause for this issue is your system running out of disk space.

Resolution

If your system has run out of disk space or very near to running out of disk space, then add more disk space to ensure you have sufficient disk space.

Once you have sufficient disk space use one of the below solutions to address the LDAP data corruption issue:

  1. Restore the OpenLDAP data from the backup.
  1. Clean up the OpenLDAP database.

Solution #1 Restore the LDAP data from the backup

On a working OpenLDAP node make a backup. The backup should be performed regularly. See Apigee Private Cloud Operations Guide for best practices on backups:

slapcat -F /opt/apigee/data/apigee-openldap/slapdd -l /tmp/ldap-backup.ldif

The following steps can be used to restore the OpenLDAP data from a good backup.

  1. Stop the OpenLDAP node for which the data needs to be restored:
    /opt/apigee/apigee-service/bin/apigee-service apigee-openldap stop
    
  1. Change directory to the OpenLDAP data directory:
    cd /opt/apigee/data/apigee-openldap
    
  1. Back up the existing OpenLDAP data using the move command:
    mv ldap ldap_orig
    
  1. Switch to the apigee user:
    su apigee
    
  1. From /opt/apigee/data/apigee-openldap directory, create a new OpenLDAP data directory with the original name:
    mkdir ldap
    
  1. Take the backup of ldap_orig/DB_CONFIG subdirectory from step 3, and copy it to the openldap directory.
    cp ldap_orig/DB_CONFIG ldap
    
  1. To restore data from backup taken with slapcat, use slapadd to import the ldif which contains the good data:
    slapadd -F /opt/apigee/data/apigee-openldap/slapd.d -l /tmp/ldap-backup.ldif
    
  1. Start the OpenLDAP process:
    /opt/apigee/apigee-service/bin/apigee-service apigee-openldap start
    

Solution #2 Cleanup the LDAP database

The following steps wipe out the OpenLDAP database to provide a fresh start. This solution can be used if there is no data backup of the last state where the OpenLDAP data was working.

  1. Stop the OpenLDAP service:
    /opt/apigee/apigee-service/bin/apigee-service apigee-openldap stop
    
  1. Change directory to the OpenLDAP data directory:
    cd /opt/apigee/data/apigee-openldap
    
  1. Back up the existing OpenLDAP data using the move command:
    mv ldap ldap_orig
    
  1. Switch to the apigee user:
    su apigee
    
  1. Create a new OpenLDAP data directory with the original name:
    mkdir ldap
    
  1. Take the backup ldap_orig/DB_CONFIG subdirectory from step 3, and copy it to the openldap directory:
    cp ldap_orig/DB_CONFIG ldap
    
  1. Restart the OpenLDAP process:
    /opt/apigee/apigee-service/bin/apigee-service apigee-openldap start
    
  1. Restart the Management Server to force a refresh of the connections to OpenLDAP:
    /opt/apigee/apigee-service/bin/apigee-service edge-management-server restart
    

If the problem persists, contact Apigee Support for further assistance.