HOWTO : OpenLDAP 2.4 Backup & Recovery on CentOS 6.2

This blog post will explain how to backup and restore our OpenLDAP 2.4 server. This is goal number nine.

1. Install OpenLDAP 2.4.
2. Configure Transport Layer Security (TLS).
3. Manage users and groups in OpenLDAP.
4. Configure pam_ldap to authenticate users via OpenLDAP.
5. Use OpenLDAP as sudo's configuration repository.
6. Use OpenLDAP as automount map repository for autofs.
7. Use OpenLDAP as NFS netgroup repository again for autofs.
8. Use OpenLDAP as the Kerberos principal repository.
9. Setup OpenLDAP backup and recovery.
10. Setup OpenLDAP replication.

The Maintenance chapter in the OpenLDAP Administrator's Guide on this topic is not very explicit. We hope this blog post will be more helpful.

Backup

We need to backup our OpenLDAP data, the OpenLDAP configuration and the OpenLDAP directory along with the system-wide LDAP configuration. Why take both the configuration and the directory? Because the configuration in LDIF format does not have the Kerberos keytab nor the /etc/sysconfig/ldap file.

Both the data and the configuration are dumped in LDIF files with the help of slapcat(8C). The directory and daemon configuration file are placed together in a compressed tar(1) file.

A quick shell script takes care of the backup procedure.

sudo mkdir -p /root/scripts
sudo vi /root/scripts/backup.slapd.sh


#!/bin/sh

# /root/scripts/backup.slapd.sh
#
# Backup the OpenLDAP data and configuration as compressed LDIF files.
# Also backup the entire OpenLDAP directory and daemon configuration.
#
# David Robillard, April 23rd, 2012.

umask 022

PATH="/bin:/usr/bin:/usr/local/bin:/sbin:/usr/sbin:/usr/local/sbin"
export PATH

DATE=`date +%Y%m%d`

BACKUP_DIR="/root/backup/slapd"

BACKUP_DATA_FILENAME="slapd.data.${DATE}.ldif"

BACKUP_CONFIG_FILENAME="slapd.config.${DATE}.ldif"

BACKUP_TAR_FILENAME="slapd.directory.${DATE}.tar.gz"

CA_TLS_CERT="/etc/pki/tls/certs/companyCA.crt"

DIT_CONFIG="cn=config"

DIT_SUFFIX="dc=company,dc=com"

SLAPD_CONFIG_FILENAME="/etc/sysconfig/ldap"

SLAPD_DIR="/etc/openldap"

SLAPD_LOG_ROTATION="/etc/logrotate.d/slapd"

SLAPD_TLS_CERT="/etc/pki/tls/certs/alice.company.com.pem"

SLAPD_TLS_KEY="/etc/pki/tls/certs/alice.company.com.key"

SLAPCAT_OPTIONS="-F /etc/openldap/slapd.d"

LOGFILE="/var/log/backup/slapd.log"

KEEP="30"

# Make sure we have a log file.
if [ ! -f ${LOGFILE} ]; then
touch ${LOGFILE}

if [ "$?" -ne "0" ]; then
echo "ERROR: could not create the log file."
exit 1
fi
fi

# Check if root is running this script.
if [ `id -u` -ne "0" ]; then
echo "ERROR: only root can run this script." | tee -a ${LOGFILE}
exit 1
fi

# Make sure we have a backup directory.
if [ ! -d ${BACKUP_DIR} ]; then
mkdir -p ${BACKUP_DIR}

if [ "$?" -ne "0" ]; then
echo "ERROR: could not create the backup directory." | tee -a ${LOGFILE}
exit 1
fi
fi

# Make sure we don't have too much backup files piling up in our backup directory.
FILES=`find ${BACKUP_DIR} -type f -name "slapd.*" -print | wc -l`

if [ "${FILES}" -gt "${KEEP}" ]; then

OVER=`echo ${FILES}-${KEEP} | bc`
RMFILES=`find ${BACKUP_DIR} -type f -name "slapd.*" -print | sort -r | tail -${OVER}`
echo "NOTE: removing ${RMFILES} from the backup directory." >> ${LOGFILE}
rm ${RMFILES}
fi

# Backup the DIT data.
slapcat ${SLAPCAT_OPTIONS} -b ${DIT_SUFFIX} -l ${BACKUP_DIR}/${BACKUP_DATA_FILENAME} >/dev/null 2>&1

if [ "$?" -eq "0" ]; then

gzip -f ${BACKUP_DIR}/${BACKUP_DATA_FILENAME} 2>&1 >> ${LOGFILE}

if [ "$?" -ne "0" ] ; then
echo "ERROR: dump file compression problem." | tee -a ${LOGFILE}
exit 1
fi
else
echo "ERROR: problem running slapcat(8C) for the DIT data backup." | tee -a ${LOGFILE}
rm ${BACKUP_DIR}/${BACKUP_DATA_FILENAME}
exit 1
fi

# Backup the DIT config as an LDIF file.
slapcat ${SLAPCAT_OPTIONS} -b ${DIT_CONFIG} -l ${BACKUP_DIR}/${BACKUP_CONFIG_FILENAME} >/dev/null 2>&1

if [ "$?" -eq "0" ]; then

gzip -f ${BACKUP_DIR}/${BACKUP_CONFIG_FILENAME} 2>&1 >> ${LOGFILE}

if [ "$?" -ne "0" ] ; then
echo "ERROR: dump file compression problem." | tee -a ${LOGFILE}
exit 1
fi
else
echo "ERROR: problem running slapcat(8C) for the DIT config backup." | tee -a ${LOGFILE}
rm ${BACKUP_DIR}/${BACKUP_CONFIG_FILENAME}
exit 1
fi

# Backup the entire configuration directory.
BACKUP_FILES_LIST="${CA_TLS_CERT} ${SLAPD_CONFIG_FILENAME} ${SLAPD_DIR} ${SLAPD_LOG_ROTATION} ${SLAPD_TLS_CERT} ${SLAPD_TLS_KEY}"

tar zcf ${BACKUP_DIR}/${BACKUP_TAR_FILENAME} ${BACKUP_FILES_LIST} >/dev/null 2>&1

if [ "$?" -ne "0" ]; then
echo "ERROR: problem running config directory tar." | tee -a ${LOGFILE}
rm ${BACKUP_DIR}/${BACKUP_TAR_FILENAME}
exit 1
fi

# EOF

Make sure this new script is executable.

sudo chmod a+x /root/scripts/backup.slapd.sh

Place the script in root's crontab.

sudo crontab -e

# root's crontab

# 

# The time and date fields are:

# 

#  field          allowed values

#  -----          --------------

#  minute         0-59

#  hour           0-23

#  day of month   1-31

#  month          1-12 (or names, see below)

#  day of week    0-7 (0 or 7 is Sun, or use names)

# Backup the OpenLDAP data, config, directory and daemon config.

00 22 * * * /root/scripts/backup.slapd.sh

# EOF

Come back tonight after 10 PM (or tomorrow :) to see if you have backup files.

IMPORTANT : make sure you copy the backup files off site!

Recovery

In the event of a catastrophic failure of the OpenLDAP server, we have everything to revive it. How we do it depends on the scenario.

1. Complete Server Lost
2. ACL Problems
3. Data Corruption (or Human Error...)
4. File System Full

Complete Server Lost

If the OpenLDAP server crashes beyond repair, the first thing one needs to do is simply find another computer (or have this one fixed), install CentOS and OpenLDAP. Then simply copy the backup files back to the new machine and run these commands :

sudo yum -y openldap-servers krb5-server-ldap nss-pam-ldapd

sudo mv /etc/openldap /etc/openldap.install

cd / && sudo tar zxvf /tmp/slapd.directory.20120515.tar.gz

sudo cp `rpm -ql openldap-servers | grep DB_CONFIG` /var/lib/ldap/DB_CONFIG

sudo chown ldap:ldap /var/lib/ldap/DB_CONFIG

sudo chkconfig slapd on

sudo /etc/init.d/slapd start

That should get you started. Of course, the rsyslog configuration will need to be reconfigured. See this blog post on how to proceed.

ACL Problems

It happens mostly when we build the server, but one can lock itself out of his own OpenLDAP server if the ACLs are modified. This one is quite easy to get back on our feet :

sudo /etc/init.d/nslcd stop

sudo /etc/init.d/slapd stop

sudo mv /etc/openldap /etc/openldap.broken

cd / && sudo tar zxvf /root/backup/slapd/slapd.directory.20120515.tar.gz

sudo /etc/init.d/slapd start

sudo /etc/init.d/nslcd start

Data Corruption (or Human Error...)

Well, sometimes even the administrators make mistakes. Normally one simply has to change it back to what it was via his prefered LDAP browser. But say one admin made changes on friday and left to go climbing. The other admin has no idea what happened and things are broken saterday morning at 2 AM. Well, in those circumstances, we admins tend to have less debug patience. So the easiest (and fastest!) way of fixing things is to bring back thursday's backup with slapadd(8C) while slapd(8C) is NOT running.

sudo /etc/init.d/nslcd stop

sudo /etc/init.d/slapd stop

sudo mv /var/lib/ldap /var/lib/ldap.broken

sudo mkdir /var/lib/ldap

sudo cp `rpm -ql openldap-servers | grep DB_CONFIG` /var/lib/ldap/DB_CONFIG

sudo zcat /root/backup/slapd/slapd.data.20120512.ldif.gz > /tmp/slapd.data.ldif

sudo slapadd -v < /tmp/slapd.data.ldif

sudo chown -R ldap:ldap /var/lib/ldap

sudo /etc/init.d/slapd start

sudo /etc/init.d/nslcd start

This will bring the data back to which ever backup data file we selected.

File System Full

Sometimes the /var filesystem gets filled up. If the LDAP data doesn't live in it's own /var/lib filesystem and the server generates a lot of logs, this can happen quite often. (HINT : place your LDAP data on a seperate filesystem! :) The /var fills up, one will see these errors in the log :

sudo: ldap_simple_bind_s: Can't contact LDAP server

Checking configuration files for slapd: bdb_db_open: DB_CONFIG for suffix dc=company,dc=com

has changed.

Performing database recovery to activate new settings.

bdb_db_open: Recovery skipped in read-only mode. Run manual recovery if errors are encountered.

bdb(dc=company,dc=com): file id2entry.bdb (meta pgno = 0) has LSN [1][1984522].

bdb(dc=company,dc=com): end of log is [1][28]

bdb(dc=company,dc=com): /var/lib/ldap/id2entry.bdb: unexpected file type or format

bdb_db_open: db_open(/var/lib/ldap/id2entry.bdb) failed: Invalid argument (22)

bdb(dc=company,dc=com): Unknown locker ID: 0

backend_startup_one: bi_db_open failed! (22)

slap_startup failed (test would succeed using the -u switch)

Fixing this error is quite easy. First, clean-up up /var!

Once /var has some space left, run the db_recover tool. This tool will make sure that all committed transactions are guaranteed to appear after db_recover has run, and all uncommitted transactions will be completely undone.

sudo /etc/init.d/nslcd stop
sudo /etc/init.d/slapd stop

sudo db_recover -vh /var/lib/ldap

sudo /etc/init.d/slapd start
sudo /etc/init.d/nslcd start

And that's it. If some data is missing, then use yesterday's backup.

We're now done with the backup and recovery of our OpenLDAP 2.4 server. This was goal number nine.

1. Install OpenLDAP 2.4.
2. Configure Transport Layer Security (TLS).
3. Manage users and groups in OpenLDAP.
4. Configure pam_ldap to authenticate users via OpenLDAP.
5. Use OpenLDAP as sudo's configuration repository.
6. Use OpenLDAP as automount map repository for autofs.
7. Use OpenLDAP as NFS netgroup repository again for autofs.
8. Use OpenLDAP as the Kerberos principal repository.
9. Setup OpenLDAP backup and recovery.
10. Setup OpenLDAP replication.

The next post will explain how to configure OpenLDAP replication.

HTH,

DA+