Troubleshooting

• Troubleshooting page in the official Shibboleth documentation

Here is how you can get the metadata of your IdP or SP as they are currently published to the federation:

• Log in to the metadata administration tool.
• Select the Entity you are interested in and click the download icon in the actions column on the right side.

• Watch the log file idp-process.log (default path: /opt/shibboleth-idp/logs) and the Tomcat log.
• Increase the log level to DEBUG. While you can see warnings and errors without doing so, the content of SAML assertions stays hidden without the DEBUG log level. In most cases it is sufficient to set DEBUG for idp, messages and encryption.

  /opt/shibboleth-idp/conf/idp.properties

  idp.loglevel.idp = DEBUG
  idp.loglevel.messages = DEBUG
  idp.loglevel.encryption = DEBUG

• Trigger a configuration reload manually (touch /opt/shibboleth-idp/war/idp.war) or wait for the automatic reload of the logging configuration. The reload intervals can be adapted in /opt/shibboleth-idp/conf/services.properties.
• If you see any warnings or errors in the idp-process.log, read the log file from top to bottom beginning at the last restart of the IdP. This is important because you can mostly find the reason (like syntax errors in configuration files incl. the line number) in the first messages, whereas further below you only get consequential errors. They tend to tell little about where to dig further like in this example:

  'shibboleth.metrics.RegisterMetricSets$child#0' defined in file [C:\Program Files (x86)\shibboleth\idp\system\conf\..\..\conf\admin\metrics.xml]: Cannot resolve reference to bean 'shibboleth.metrics.AttributeResolverGaugeSet' while setting bean property 'arguments' with key [7]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'shibboleth.metrics.AttributeResolverGaugeSet' defined in file [C:\Program Files (x86)\shibboleth\idp\system\conf\general-admin-system.xml]: Invocation of init method failed; nested exception is net.shibboleth.utilities.java.support.component.ComponentInitializationException: Injected service was null or not an AttributeResolver

  Scroll up and search for errors that occur during the parsing of conf/attribute-resolver.xml.

• When editing the values in .properties files, please avoid trailing spaces at the line ends. A blank space at the end of a password renders the password unusable. If you get the LDAP error Invalid credentials, make sure that your credentials are correct and check for spaces at the line ends of the credentials. The password must not be written in between single or double quotes.
• Be precise when editing configuration files. An example: Entity IDs with or without a trailing slash are two different Entity IDs!

If your IdP does not log anything at all to the files in /opt/shibboleth-idp/logs (or any custom directory you configured), please check the Tomcat log files. On a Debian server they are located in /var/log/tomcat9. Watch all of them, e.g. with

root@idp:~# tail -f /var/log/tomcat9/*.log

and restart Tomcat in a different terminal window.

To see the full SAML assertion, set the log level to DEBUG for idp, messages and encryption as shown above. The SAML assertion will then be written into idp-process.log. Search for the string “Assertion before encryption”. In this example of a SAML assertion you can see that the IdP with the Entity ID https://idp.local/idp/shibboleth transmitted a persistentID (value: “NMTR6SVZOCUWF5MNGDDIYQQBY4QCB76N”) as well as the attributes eduPersonScopedAffiliation (with three values) and eduPersonEntitlement (with one value) to an SP with the Entity Id https://sp1.local/shibboleth.

/opt/shibboleth-idp/logs/idp-process.log

2020-10-27 14:54:07,079 - 127.0.0.2 - DEBUG [org.opensaml.saml.saml2.encryption.Encrypter:339] - Assertion before encryption:
<?xml version="1.0" encoding="UTF-8"?><saml2:Assertion xmlns:saml2="urn:oasis:names:tc:SAML:2.0:assertion" ID="_3a89275f3cd179685b22a3cf616c518a" IssueInstant="2020-10-27T13:54:06.179Z" Version="2.0">
   <saml2:Issuer>https://idp.local/idp/shibboleth</saml2:Issuer>
   <saml2:Subject>
      <saml2:NameID Format="urn:oasis:names:tc:SAML:2.0:nameid-format:persistent" NameQualifier="https://idp.local/idp/shibboleth" SPNameQualifier="https://sp1.local/shibboleth" xmlns:saml2="urn:oasis:names:tc:SAML:2.0:assertion">NMTR6SVZOCUWF5MNGDDIYQQBY4QCB76N</saml2:NameID>
      <saml2:SubjectConfirmation Method="urn:oasis:names:tc:SAML:2.0:cm:bearer">
         <saml2:SubjectConfirmationData Address="127.0.0.2" InResponseTo="_72e4da8e1fa8dfcea0f85a340ded8b7c" NotOnOrAfter="2020-10-27T13:59:06.581Z" Recipient="https://sp1.local/Shibboleth.sso/SAML2/POST"/>
      </saml2:SubjectConfirmation>
   </saml2:Subject>
   <saml2:Conditions NotBefore="2020-10-27T13:54:06.179Z" NotOnOrAfter="2020-10-27T13:59:06.179Z">
      <saml2:AudienceRestriction>
         <saml2:Audience>https://sp1.local/shibboleth</saml2:Audience>
      </saml2:AudienceRestriction>
   </saml2:Conditions>
   <saml2:AuthnStatement AuthnInstant="2020-10-27T13:53:52.456Z" SessionIndex="_d8a534b987abc20fe007a9cc80af7bdb">
      <saml2:SubjectLocality Address="127.0.0.2"/>
      <saml2:AuthnContext>
         <saml2:AuthnContextClassRef>urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport</saml2:AuthnContextClassRef>
      </saml2:AuthnContext>
   </saml2:AuthnStatement>
   <saml2:AttributeStatement>
      <saml2:Attribute FriendlyName="eduPersonScopedAffiliation" Name="urn:oid:1.3.6.1.4.1.5923.1.1.1.9" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:uri">
         <saml2:AttributeValue>staff@local</saml2:AttributeValue>
         <saml2:AttributeValue>member@local</saml2:AttributeValue>
         <saml2:AttributeValue>employee@local</saml2:AttributeValue>
      </saml2:Attribute>
      <saml2:Attribute FriendlyName="eduPersonEntitlement" Name="urn:oid:1.3.6.1.4.1.5923.1.1.1.7" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:uri">
         <saml2:AttributeValue>urn:mace:dir:entitlement:common-lib-terms</saml2:AttributeValue>
      </saml2:Attribute>
   </saml2:AttributeStatement>
</saml2:Assertion>

Shibboleth IdPs 4.x that were upgraded from a version 3.x do not automatically use the attribute registry. The configuration file attribute-Resolver.xml then still contains the transcoding rules for each defined attribute. When adding a new attribute definition you also have to specify a transcoding rule, or else the IdP will not be able to encode the attribute in any SP-readable form. Here is an example:

/opt/shibboleth-idp/conf/attribute-resolver.xml

    <AttributeDefinition xsi:type="Scoped" id="samlPairwiseID" scope="%{idp.scope}">
        <InputDataConnector ref="myStoredId" attributeNames="persistentId"/>
        <AttributeEncoder xsi:type="SAML2ScopedString" name="urn:oasis:names:tc:SAML:attribute:pairwise-id" friendlyName="pairwise-id" encodeType="false" />
    </AttributeDefinition>

Alternatively, you can reconfigure the IdP to use the attribute registry (documentation in German).

Note that the AttributeEncoder line must not be included in the attribute definition of a newly installed IdP 4.x. It will be fetched from the attribute registry underneath conf/attributes instead.

“opensaml::SecurityPolicyException Message was signed, but signature could not be verified.”

You see this error message whenever the IdP certificate published in the federation metadata does not match the one configured on the actual IdP. During installation the Shibboleth installer generates a self-signed certificate and preconfigures it in conf/idp.properties. Adapt that file to point to the certificate you want the IdP to use and make sure the same one is published.

By the way: The file metadata/idp-metadata.xml is autogenerated, too. It contains the initial post-installation IdP metadata. It is parsed when you first add the IdP to the metadata administration tool, but in the actual federation this file is ignored. The valid IdP metadata that you maintain are those in the administration tool.

“opensaml::FatalProfileException at (https://testsp2.aai.dfn.de/Shibboleth.sso/SAML2/POST)”

You get this error message when the Service Provider cannot find any metadata for the Identity Provider.

• Check if you have added the IdP to the metadata administration tool and if it was added to the respective federation correctly (DFN-AAI-Test, DFN-AAI-Basic, or DFN-AAI).
• Compare the Entity ID in conf/idp.properties with the one in the metadata entry. They have to be identical.
• After a change to the federation metadata, keep in mind that you have to wait for 60-90 minutes for the metadata to be aggregated and redistributed to all SPs.

When you see the following message in your idp-process.log, the Identity Provider cannot find any metadata for the Service Provider:

INFO [org.opensaml.saml.common.binding.impl.SAMLMetadataLookupHandler:171] (...) Message Handler:  No metadata returned for 
 (...) in role {urn:oasis:names:tc:SAML:2.0:metadata}SPSSODescriptor

• Please check where the IdP should know the SP from:
  • from federation metadata?
  • from your organisation's local metadata? (Did you add the SP as a local SP in the metadata administration tool?)
  • from an xml metadata file that you added manually to conf/metadata-providers.xml? Check the content of both files.
• Remember that - in the first two cases - you have to wait for 60 to 90 minutes for the changes to propagate.

“Web Login Service - Unsupported Request The application you have accessed is not registered for use with this service”, or in German: “Web Anmeldedienst - Nicht unterstützte Anfrage Die Applikation, auf die Sie zugreifen möchten, ist für die Benutzung dieses Dienstes nicht registriert.”

This error message is displayed when the IdP cannot find the SP in metadata.

• Check whether all required metadata providers have been added to conf/metadata-providers.xml (Documentation).
• Check the folder /opt/shibboleth-idp/metadata to see if up-to-date federation metadata have been downloaded. You can also access this information on the IdP status page (default: https://YOUR-HOST/idp/status). In the following example DFN_AAI metadata have expired:

  service: shibboleth.MetadataResolverService
  last successful reload attempt: 2020-12-22T07:58:12Z
  last reload attempt: 2020-12-22T07:58:12Z
  
  	metadata source: DFN_AAI
  	last refresh attempt: 2020-12-24T05:26:48Z
  	last update: 2020-12-24T05:26:48Z
  
  	metadata source: DFN_AAI_eduGAIN
  	last refresh attempt: 2021-01-05T08:57:55Z
  	last update: 2021-01-05T08:57:55Z
  
  	metadata source: DFN_AAI_TEST
  	last refresh attempt: 2021-01-05T09:36:13Z
  	last update: 2021-01-05T09:36:13Z

• Check the IdP's DEBUG-Log. Compare the saml:Issuer from the AuthnRequest with the EntityID you are trying to contact. If there is a different issuer string in the Authentication Request the IdP cannot find the issuer in the federation metadata. Contact the SP operator in this case.

If you see the error message “A non-proceed event occurred while processing the request: DecryptNameIDFailed” this might be after a certificate rollover and maybe it only occurs upon Single Logout. The Service Provider has encrypted data with a certificate for which the IdP does not have a private key (any more). Remove the old certificate from the metadata and wait for 60 to 90 minutes. By then, the information to only use the new certificate has propagated.

Your IdP keeps copies of all original files in the folder dist/conf. Here you can fetch files, if you want to start over or if you want to see what the original file looks like in the latest installed IdP version.

If you notice that your IdP 4.x transmits duplicate attributes, you probably have copied the file conf/attribute-resolver.xml from an IdP 3.x without adapting it. The IdP has duplicate transcoding rules: once in conf/attribute-resolver.xml and once in the Attribute Registry. Remove the Attribute Encoder lines from the resolver configuration so that it looks like in this example.

If you get the error message below, you probably have a duplicate attribute in your Attribute Registry. Maybe you imported attributes from a file like our dfnMisc.xml (German documentation) and you have defined one of the attributes in an individual .properties file underneath conf/attributes/custom/? Make sure that each attribute exists in the Attribute Registry exactly once.

java.lang.IllegalArgumentException: {urn:oasis:names:tc:SAML:2.0:assertion}NameID is
already the child of another XMLObject and may not be inserted into this list

Our downstream eduGAIN metadata (the eduGAIN metadata we distribute to DFN-AAI) have never contained entities from DFN-AAI. We filter them out because your systems already know them from DFN-AAI metadata and we do not want to distribute duplicates. To check whether an entity is part of the eduGAIN metadata, please search for it in the eduGAIN Entities Database.

You have added your Identity Provider to the federation but it doesn't show in discovery services? This can happen for several reasons:

• The Service Provider hasn't fetched the latest metadata yet. Please wait for 60-90 minutes before testing in DFN-AAI or our test federation). Wait for up to 24 hours before testing in eduGAIN. You can check if your IdP is already included in the eduGAIN metadata: https://technical.edugain.org/entities.
• You have ticked the checkbox “hide from discovery” in the IdP's settings in the metadata administration tool. Remove the tick and wait for 60-90 minutes.

A Service Provider can announce in its metadata that it

• signs Authentication Requests it sends to IdPs, and/or
• wants to get signed SAML assertions back.

Our metadata administration tool only displays this information if it is included in the xml files upon initial upload to the metadata administration. Please extend your SP metadata like this:

<md:SPSSODescriptor AuthnRequestsSigned="true" WantAssertionsSigned="true" protocolSupportEnumeration="urn:oasis:names:tc:SAML:2.0:protocol">