Jump to: navigation, search

Troubleshooting

This document should help FreeIPA users who are trying to troubleshoot why their setup is not working as expected. After following the steps and advises described in this article, users should be able to either fix the configuration themselves or provide the right information for developers/support to investigate and advise or to fix the issue.

Feedback is expected to be sent either to:

Before we dive into particular scenarios we offer you presentation about FreeIPA troubleshooting principles.

Contents

Reporting bugs

Providing the right information in a report, regardless whether it is filed by mail or in bug tracking system will help FreeIPA developers quickly identify the root cause of an issue and provide help.

The following information is expected to be passed in the report:

  • Version and distribution: we need to know the version (and distribution) of the FreeIPA you are running on. If other components (like NFS, sudo) are involved, their versions may be useful as well.
  • Problem description: What happened, in what part/component of FreeIPA it happened, how severe is the issue. The problem description should help us to either reproduce the problem or identify the faulty component.
  • What have you tried: transcript of what you already did to investigate the issue
  • Logs: Related (and sanitized) logs are essential source of information. At least the logs mentioned in the different scenarios below or the ones mentioned on Files to be attached to bug report page should be included.

If you want to read some theory about bug reporting we recommend you the famous article How to Report Bugs Effectively.

Troubleshooting scenarios

FreeIPA consists of many integrated technologies and components. Therefore, investigation of issues occurring in one part of FreeIPA will take different path and steps from investigation of issues in other part. The same applies for the list of information expected to be provided. The following sections tries to steer the users in the right directions when hitting issues in different components.

Installation

Server Installation

When installation crashes, check installation log in /var/log/ipaserver-install.log.

If the installation crashed on installing PKI server (Dogtag), check it's logs as well. The most useful logs are the following:

  • /var/log/pki/pki-ca-spawn.$TIME_OF_INSTALLATION.log
  • /var/log/pki/pki-tomcat/catalina.out
  • /var/log/pki/pki-tomcat/ca/system
  • /var/log/pki/pki-tomcat/ca/debug

pki-selinux policy not loaded properly

If you see in ipaserver-install.log line: /usr/bin/runcon: invalid context: unconfined_u:system_r:pki_ca_script_t:s0: Invalid argument" Then the culprit might be that pki-selinux failed to load its policy. The best thing to do is to force re-install pki-selinux (and check for any errors in the /var/log/messages file or journal).

Replica Installation

When installation crashes, check installation log in /var/log/ipareplica-install.log. When CA is being installed on a replica, check the aforementioned PKI logs as well.

Migrating from RHEL 6/CentOS 6
  • Installation of certificate server fails with:
    Clone does not have all the required certificates
    It indicates bug 1322059. Issue is that RHEL6, while creating replica file, uses certificates from a file which was created during server installation and potentially contains expired certificates instead of fetching the certs from database where they are valid. It is fixed on FreeIPA 3.2+. Recovery is to update the file with valid certs and then run ipa-replica-prepare again and try replica installation again:
    1. create a /root/dbpass file containing the 'internal' (not 'internaldb') password from /etc/pki-ca/password
    2. create a /root/dmpass file containing the DM password
    3. run PKCS12Export:
    4. #PKCS12Export -debug -d /var/lib/pki-ca/alias -p /root/dbpass -w /root/dmpass -o /root/cacert.p12
  • Installation of certificate server fails with:
    Error while updating security domain: java.io.IOException: 2
    It indicates bug 1256039. The issue is that certificate server users on master server has invalid, usually expired cert, in its database entry even though all certs tracked by certmonger are valid. Recovery is to update the database entry with correct certificate. Run following script or use manual method is described at freeipa users list mail.
    # /usr/share/pki/scripts/restore-subsystem-user.py -v

Client Installation

When installation does not work as expected, check installation log in /var/log/ipaclient-install.log. You can run installation in verbose mode if you run ipa-client-install with --debug option. (Log files always contain debug information, so you do not need to re-run installation with --debug option.)

Installation breaks on decoding/downloading CA certificate
  • `ipa-client-install` may crash with error like
    ipalib.errors.LDAPError: failed to decode certificate: (SEC_ERROR_INVALID_ARGS) security library: invalid arguments.
  • This may mean that PKI CA Certificate stored in LDAP was not properly imported during upgrade in some of the older versions
  • Verify that the CA certificate is stored correctly
    $ ldapsearch -h your.ipa.server.fqdn -x -b "cn=CAcert,cn=ipa,cn=etc,dc=example,dc=test"
  • The cACertificate;binary should contain the encoded certificate, typically starting with MII characters
  • If the certificate is missing, go to any FreeIPA master to let updater regenerate it:
    # kinit admin
    # ldapdelete -Y GSSAPI "cn=CAcert,cn=ipa,cn=etc,dc=example,dc=test"
    # ipa-ldap-updater --upgrade
Failed to update DNS records

When client cannot update the DNS record in FreeIPA managed DNS zone:

  • Make sure that the respective FreeIPA DNS zone has Dynamic Updates option enabled:
$ ipa dnszone-mod zone.name.example. --dynamic-update=TRUE
  • Make sure that the FreeIPA server with DNS service has port 53 opened for both UDP and TCP (related user case)

Directory Server issues

Replication issues

  • If changes done on one FreeIPA master are not replicated to another master, always verify errors log on both master and replica. It most often contains exact error what is wrong
  • Make sure that there are no DNS issues and both replicas can resolve each other's forward and reverse DNS records or that /etc/hosts does not contain address of the remote DS with it's short them being the primary (first) name.
  • Make sure that the system time difference on the FreeIPA masters is not greater than 5 minutes
  • In case of Kerberos issues in the log, verify that the DS keytab is correct and can be used to query other master:
    # kinit -kt /etc/dirsrv/ds.keytab ldap/`hostname`
    # klist
    # ldapsearch -Y GSSAPI -h `hostname` -b "" -s base
    # ldapsearch -Y GSSAPI -h the.other.master.fqdn -b "" -s base
  • Make sure that DS communication is not failing because of wrong default for SASL communication buffer (FreeIPA ticket with more information). This can be detected by seeing following error message in /var/log/messages or systemctl status dirsrv@YOUR-INSTANCE.service unit log in case of systemd:
    [01/Dec/2014:12:00:00 +0100] - sasl_io_recv failed to decode packet for connection xxxx

Obsolete RUV records

  • If the FreeIPA infrastructure keep getting obsolete RUV records (ipa-replica-manage list-ruv) which cannot be removed by ipa-replica-manage clean-ruv command and for example give status like RID XX Waiting to process all the updates from the deleted replica... but never finish, make sure that IPv6 is not disabled on the FreeIPA replicas (related freeipa-users thread)
  • Obsolete CA RUV records. FreeIPA < 4.4 doesn't have means to remove obsolete CA RUVs. They usually manifest in directory server log as
    attrlist_replace - attr_replace (nsslapd-referral, ldap://my.ipa.test:389/o%3Dipaca) failed.
    recovery is described at FreeIPA users list post

Performance tuning

See hardware recommendations and Directory server tuning.

Advanced Troubleshooting

See troubleshooting section of 389 Directory Server FAQ including the information how to retrieve a core dump and a stacktrace if the Directory Server is crashing.

Authentication/Kerberos

kinit does not work

  • On client, see the debug messages from the kinit process itself:
    KRB5_TRACE=/dev/stdout kinit admin
  • Make sure that there are no DNS Issues and that forward (A and/or AAAA) records of the client are OK.
  • Make sure that krb5kdc and dirsrv services on the FreeIPA server are running
  • Check for errors in /var/log/krb5kdc.log

Service does not start

  • See service log of the respective service for the exact error text. For example, the Directory Server stores the log in /var/log/dirsrv/slapd-REALM-NAME/errors
  • Make sure that the server the service is running on has a fully qualified domain name
  • Make sure that if /etc/hosts contains an entry for this server, the fully qualified domain name comes first, e.g.:
    192.168.1.1 ipa.example.com ipa
  • See what keys are in the keytab used for authentication of the service, e.g.:
    # klist -kt /etc/dirsrv/ds.keytab
  • Make sure that the stored principals match the system FQDN system name
  • Make sure that the version of the keys (KVNO) stored in the keytab and in the FreeIPA server match:
    $ kvno ldap/ipa.example.com@EXAMPLE.COM
  • Make sure that there are no DNS Issues and both forward and reverse DNS records of the are OK and match the system name and the stored principal keys
  • Make sure that the system time difference on the host and FreeIPA server is not greater than 5 minutes

Cannot authenticate on client

  • If FreeIPA was re-enrolled against different FreeIPA server, try removing SSSD caches (/var/lib/sss/db/*) and restarting the SSSD service (freeipa-users thread)

For further advise, see SSSD guide for troubleshooting problems on clients, including tips for gathering SSSD log files.

Failed auth increments failed login count by 2

  • This happens when migration mode is enabled. After normal auth attempt SSSD performs LDAP bind to generate Kerberos keys. This failure raises the counter for second time.
  • Resolution: disable migration mode when all users are migrated by
    ipa config-mod --enable-migration=False

Cannot authenticate user with OTP with Google Authenticator

  • This happens when hash function other that SHA-1 is used and OTP code is generated using Google Authenticator (encountered with 4.74). Google Authenticator ignores the hash function and uses SHA-1 anyway making the generated codes unusable. Use FreeOTP application or OTP tokens with SHA-1 hash function. related freeipa-users thread.

Smart Card authentication

See Troubleshooting SmartCard authentication for SmartCard authentication issues.

Trusts

Ubuntu distributions at this time don't support Trust feature of FreeIPA. See https://bugs.launchpad.net/ubuntu/+source/samba/+bug/1552249 for more details.

Cannot create trust with trust-add

See separate page with instructions how to debug trust creating issues.

DNS Issues

Getting logs

FreeIPA is using BIND as integrated DNS server. If you suspect that something is wrong with your DNS, inspect logs generated by BIND.

Depending on your distribution and FreeIPA version, the logs can be on accessed using three different techniques:

  • $ journalctl -u named-pkcs11
  • $ journalctl -u named
  • file /var/named/data/named.run

Kerberos does not work

From common experience, a great portion of issues with FreeIPA or the Kerberos authentication is caused by DNS misconfiguration. When investigating such issue make sure that:

  • Client forward record is OK both on FreeIPA server and the affected FreeIPA client:
    host $IPA_CLIENT_FQDN
  • Server forward and reverse record is OK both on FreeIPA server and the affected FreeIPA client:
    host $IPA_SERVER_FQDN
    host $IPA_SERVER_IP_ADDRESS
    • Returned hostname must be a fully qualified hostname with a trailing dot, e.g. server.example.com.
  • Check /etc/hosts on the client make sure there is not a wrong server entry or a server entry where the first name is not fully qualified.

named on server does not start

See article What to do when named with bind-dyndb-ldap cannot start

PTR synchronization does not work

Most common problems are caused by misconfiguration. Please see article How PTR record synchronization works.

Reporting bugs

Please follow instructions published by bind-dyndb-ldap project.

Forward zone does not work

At first please try following command:

dig @forwarder forwarding.zone.name. SOA

If command above returns NXDOMAIN or SERVFAIL, please check your forwarder.

DNSSEC validation
  • Do you use TLD domains you don't own (like .local or .corp)?
  • Do you have dnssec-validation yes; in /etc/named.conf?
  • Does journalctl -u named-pkcs11 show errors about record signatures?

Then DNSSEC validation prevents you from resolving records from the forward zone.

How to fix it:

  • at first please don't use domains you don't own (FreeIPA Deployment Recommendations)
  • if you really need those domains, you have to set dnssec-validation no; in /etc/named.conf on all FreeIPA DNS servers (and proceed restart)
missing zone delegation

Do you have a master zone that is the parent of your forward zone (both on FreeIPA server)? Example:

  • master zone: ipa.freeipa.org.
  • forward zone: others.ipa.freeipa.org.

Please check if master zone contains an NS delegation record and A glue records (HOWTO - Delegate a Sub-domain (a.k.a. subzone)).

Example:

$ORIGIN ipa.freeipa.org.

others NS ns.others.ipa.freeipa.org.
ns.others A 192.0.2.1
ns.others AAAA 2001:db8::1

Without zone delegation all queries are processed by master zone and NXDOMAIN is returned (Forward zones design page).

Forward policy set to none

Please check your forward policy:

ipa dnsforwardzone-show ipa.freeipa.org.

If forward policy is set to none, forwarding is disabled. Please set first or only as forward-policy to allow forwarding.

DNSSEC signing does not work

Related information how to use DNSSEC with FreeIPA can be found in DNSSEC howto.

DNSSEC signing is not enabled for the particular zone

ipa dnszone-show ipa.example

Allow in-line DNSSEC signing: TRUE

Use command ipa dnszone-mod ipa.example --dnssec=1 to enable DNSSEC signing for given zone.

DNSSEC master is not configured

Verify that one server is configured to be DNSSEC key master. Run following commands on one FreeIPA replica and check that exactly one LDAP entry is printed out:

  • kinit admin
  • ldapsearch -Y GSSAPI '(&(ipaConfigString=enabledService)(ipaConfigString=dnssecKeyMaster))'
dn: cn=DNSSEC,cn=vm-236.idm.lab.eng.brq.redhat.com,cn=masters,cn=ipa,cn=etc,dc=ipa,dc=example
objectClass: ipaConfigObject
objectClass: nsContainer
objectClass: top
ipaConfigString: dnssecKeyMaster
ipaConfigString: startOrder 100
ipaConfigString: enabledService
cn: DNSSEC

# numEntries: 1
  • If no entry was found, promote one FreeIPA replica to be the DNSSEC key master: ipa-dns-install --dnssec-master


DNSSEC key master services are not running

Run ipactl status on the DNSSEC key master and check that all services are running: All services should be in state RUNNING except ipa-ods-exporter service which is run only on-demand.

DNS keys are not generated by OpenDNSSEC

Here we begin with root account on the replica in DNSSEC key master role. First of all switch to user ods so you do not mangle filesystem permissions:

  • sudo -u ods -s /bin/bash
  • source /etc/sysconfig/ods
  • export SOFTHSM2_CONF

Now you can list zones managed by OpenDNSSEC:

  • ods-ksmutil zone list
Found Zone: ipa.example; on policy default

If the zone is not in the list, restart ipa-dnskeysyncd service which is responsible for LDAP->OpenDNSSEC synchronization and check its logs if the restart did not help.

If the zone is in the list, verify that DNSSEC keys were generated for the zone. You should see:

  • At least one key with type KSK in state publish or active.
  • At least one key with type ZSK in state active.
  • ods-ksmutil key list --verbose
Zone:         Keytype:   State:    Date of next transition (to):  Size:   Algorithm:  CKA_ID:     Repository:   Keytag:
ipa.example   ZSK        active    2015-06-03 12:52:07 (retire)   2048    8           623d723...  SoftHSM       60195
ipa.example   KSK        publish   2015-03-06 02:52:07 (ready)    2048    8           60b5ce3...  SoftHSM       6046

Missing keys indicate a problem with OpenDNSSEC or possibly lack of entropy. Check logs for ods-enforcerd service.

DNS keys are stored in local HSM on key master replica

Verify that keys shown by OpenDNSSEC key list command actually exist in local HSM on the DNSSEC key master replica:

  • sudo -u ods -s /bin/bash
  • python2 /usr/lib/python2.*/site-packages/ipapython/dnssec/localhsm.py

Every CKA_ID has to be listed in twice with boolean parameters shown below. Please ignore other values printed by localhsm.py command.

zone public keys
================
623d723...
{'ipk11label': u'623d723...', 'ipk11verify': True, ...}
...

zone private keys
=================
623d723...
{'ipk11label': u'623d723...', 'ipk11sign': True, 'ipk11extractable': True, ...}

PKI Issues

IPA won't start, expired certificates

Starting with IPA 3.0.0 all FreeIPA certificates are tracked by Certmonger and should be renewed automatically. In case of problems, see Certmonger#Manually_renew_a_certificate

If your Certificate Authority certificate is expired, see CA Certificate Renewal page .

For v2.0 see IPA_2x_Certificate_Renewal.

PKI-tomcatd fails to start

After an upgrade of IPA packages, pki-tomcatd fails to start. See Troubleshooting pki-tomcatd fails to start.

Authentication Errors

If you see something like 4301 (RPC failed at server. Certificate operation cannot be completed: FAILURE (Authentication Error)) or Invalid Credential the likely culprit is the RA agent certificate that IPA uses to authenticate against PKI.

Use getcert list -d /etc/httpd/alias -n ipaCert to show the current status of the certificate tracked by Certmonger. It should be MONITORING.

If it isn't in MONITORING, or it is and things still aren't working, compare the serial number of the certificate with that on other IPA masters:

# certutil -L -d /etc/httpd/alias -n ipaCert | grep Serial

The serial number should match the value of the 2nd integer at:

# ldapsearch -x -h localhost -p 389 -b uid=ipara,ou=People,o=ipaca description 

(use port 7389 for 2.x servers)

If they are different this suggests that one has been renewed. Only the most recent is allowed by dogtag. To repair this, go to the master with the most recent certificate:

# certutil -L -d /etc/httpd/alias -n ipaCert -a > /tmp/ra.crt 

This will export all the certificates. Edit this file and remove all but the first certificate, You can double-check the result with:

# openssl x509 -text -in /tmp/ra.crt

It should have only the cert with the latest serial #.

Now add it to your cert database:

# certutil -A -n ipaCert -d /etc/httpd/alias -t u,u,u -a -i /tmp/ra.crt
# service httpd restart

If the certificate is valid and the ou=People entry is ok then check the PKI logs /var/log/pki or /var/log/pki-ca.

If you see an error like "Failed to connect LDAP server" then try restarting the tomcat process, either pki-cad (for IPA 3.0) or pki-tomcatd@pki-tomcat.service.

CRL gets very old

If the main CRL file containing the list of invalidated certificates is old and not updated, make sure you check that:

  • There is at least one PKI master server generating the CRL - see CVE-2012-4546 for instructions.
  • CRL generation on that server is not blocked by wrong ownership of /var/lib/ipa/pki-ca/publish/ directory or there are no SELinux errors. Consult PKI system for details. (related user case)

External CA renewal with ipa-cacert-manage fails

The second step of external CA renewal may fail for a number of reasons:

  • Subject name mismatch
    The new CA certificate issued by the external CA uses a different subject name than the old CA certificate.
  • Subject name encoding mismatch
    The new CA certificate issued by the external CA uses the same subject name as the old CA certificate, but it is encoded differently.
    Some CAs like to re-encode the subject name from certificate signing requests in certificates they issue. This does not work well with NSS, which considers subject names to be equal only if they binary representation is exactly the same. This can be fixed by configuring the external CA to either respect the original encoding, or to use UTF8String encoding, which is IPA default.
    This usually happens with Microsoft Certificate Services, which use PrintableString as the default encoding in subject names. See this Microsoft knowledge base article for info on how to force UTF8String encoding.
  • Subject public key info mismatch
    The new CA certificate issued by the external CA uses a different public / private key pair than the old CA certificate.
  • Not a valid CA certificate
    The new CA certificate issued by the external CA is either missing the Basic Constraints extension, or the Basic Constraints extension indicates that the certificate is not a CA certificate.

Administration Framework

Privilege Separation

Starting with FreeIPA 4.5, management framework runs in separate processes and uses GSS-Proxy to obtain Kerberos credentials. Privilege Separation page describes this setup in detail, including how to debug privilege separation related issues.

ipa command returns Internal Server Error

  • See /var/log/httpd/error_log for traceback and potentially for more related information

ipa command crashes or returns no data

  • Try running the command with verbose output and see what exactly is being sent to the server:
    ipa -vv user-show admin
  • Try enabling debug level on server and see if there is useful information:
    • Add debug=True to [global] section of /etc/ipa/default.conf or /etc/ipa/server.conf and reload httpd service
    • Run the command again

Web UI

Cannot authenticate to Web UI

  • Make sure that the user can authenticate in CLI, e.g. with kinit $USER
  • Make sure that httpd, dirsrv and ipa_memcached services on the affected FreeIPA server are running.
  • Make sure there are no related SELinux AVCs
  • Make sure that cookies are enabled on the client browser
  • Make sure that the time on the FreeIPA server is up to date and there is no (significant) clock skew (freeipa-users thread)
  • Search for any related errors in /var/log/httpd/error_log

Browser shows ERR_CERT_COMMON_NAME_INVALID - missing Subject Alternative Name extension in certificate

For more details see Fraser's blog post Implications of Common Name deprecation for Dogtag and FreeIPA.

A certificate which is used for web needs to include Subject Alternative Name extension. If cert was issued without this extension then it needs to be renewed to include the extension in following way:

  • Use getcert list to find the REQUEST-ID to use; it will be the certificate in NSSDB /etc/httpd/alias with nickname Server-Cert.
  • Use getcert resubmit -i REQUEST-ID -D DNS-NAME to request a new HTTP certificate with the appropriate DNS-NAME Subject Alt Name value(s).

Integration with other software

It is difficult to predict issues with interaction with other software, the problem may be in bad configuration of the software, unexpected environment configuration or, of course, a flaw in any of the FreeIPA component. A general advise is to look for logs of the software and try to narrow down the area where the problem is, check it's documentation or know bugs or consult on freeipa-users list.

sudo does not work for hostgroups

SUDO requires netgroups to be working properly to use the hostgroups in the FreeIPA sudo rules. If the SUDO rules only work with individual hosts being set in SUDO rule, check the affected machine:

  • Make sure that NIS domain name matches your domain:
    nisdomainname
  • Make sure that netgroups can be read:
    getent netgroup hgroup1 
    if this command does not work, make sure that sss is listed for netgroup entry in /etc/nsswitch.conf

If the advice above does not help, edit /etc/sudo-ldap.conf (or /etc/ldap.conf, depending on what version of platform you're running) and add the following line to the top:

sudoers_debug 2

Then try another command, for example

sudo -l