spid-sp-test 0.5.4

SAML2 SPID Service Provider validation tool that can be run from the command line

spid-sp-test

spid-sp-test is a SAML2 SPID Service Provider validation tool that can be executed from the command line. This tool was born by separating the test library already present in spid-saml-check.

Features

spid-sp-test is:

• able to test a SAML2 SPID Metadata file or http url
• able to test a SAML2 SPID AuthnRequest file or or http url
• able to test ACS behaviour, how a SP replies to a SAML2 Response
• able to dump the responses sent to an ACS and the HTML of the SP's response
• able to handle Attributes to send in Responses or test configurations of the Responses via json configuration files
• able to configure response template with Jinja2
• able to get new test-suite via multiple json files
• fully integrable in CI
• able to export a detailed report in json format, in stdout or in a file

Generally it's:

• extremely faster in execution time than spid-saml-check
• extremely easy to setup

Setup

apt install libxml2-dev libxmlsec1-dev libxmlsec1-openssl
pip install spid-sp-test --upgrade --no-cache

Overview

spid-sp-test can test a SP metadata file, you just have to give the Metadata URL, if http/http or file, eg: file://path/to/metadata.xml. At the same way it can test an Authentication Request.

In a different manner spid-sp-test can send a huge numer of fake SAML Response, for each of them it needs to trigger a real Authentication Request to the target SP.

If you want to test also the Response, you must give the spid-sp-test fake idp metadata xml file to the target SP. Get fake IdP metadata (--idp-metadata) and copy it to your SP metadatastore folder.

spid_sp_test --idp-metadata > /path/to/spid-django/example/spid_config/metadata/spid-sp-test.xml

To get spid-sp-test in a CI you have to:

• configure an example project to your application
• use the spid-sp-test fake idp metadata, configure it in your application and execute the example project, with its development server in background
• launch the spid-sp-test commands

An example of CI is here

Examples

Run spid_sp_test -h for inline documentation.

usage: spid_sp_test [-h] [--metadata-url METADATA_URL] [--idp-metadata] [-l [LIST [LIST ...]]] [--extra] [--authn-url AUTHN_URL] [-tr] [-nsr] [-tp TEMPLATE_PATH] [-tn [TEST_NAMES [TEST_NAMES ...]]]
                    [-tj [TEST_JSONS [TEST_JSONS ...]]] [-aj ATTR_JSON] [-report] [-o O] [-d {CRITICAL,ERROR,WARNING,INFO,DEBUG}] [-xp XMLSEC_PATH] [--production] [--html-path HTML_PATH] [--exit-zero]

src/spid_sp_test/spid_sp_test -h for help

optional arguments:
  -h, --help            show this help message and exit
  --metadata-url METADATA_URL
                        URL where SAML2 Metadata resides: it can be file://path or https://fqdn
  --idp-metadata        get example IdP metadata
  -l [LIST [LIST ...]], --list [LIST [LIST ...]]
                        esecute only selected checks
  --extra               execute extra checks
  --authn-url AUTHN_URL
                        URL where the SP initializes the Authentication Request to this IDP,it can also be a file:///
  -tr, --test-response  execute SAML2 responses
  -nsr, --no-send-response
                        print SAML2 Response without sending back to SP. It only works with '-tr'
  -tp TEMPLATE_PATH, --template-path TEMPLATE_PATH
                        templates containing SAML2 xml templates for response tests
  -tn [TEST_NAMES [TEST_NAMES ...]], --test-names [TEST_NAMES [TEST_NAMES ...]]
                        response test to be executed, eg: 01 02 03
  -tj [TEST_JSONS [TEST_JSONS ...]], --test-jsons [TEST_JSONS [TEST_JSONS ...]]
                        custom test via json file, eg: tests/example.test-suite.json
  -aj ATTR_JSON, --attr-json ATTR_JSON
                        loads user attributes via json, eg: tests/example.attributes.json
  -report               json report in stdout
  -o O                  json report to file, -report is required
  -d {CRITICAL,ERROR,WARNING,INFO,DEBUG}, --debug {CRITICAL,ERROR,WARNING,INFO,DEBUG}
                        Debug level, see python logging
  -xp XMLSEC_PATH, --xmlsec-path XMLSEC_PATH
                        xmlsec1 executable path, eg: /usr/bin/xmlsec1
  --production, -p      execute tests for system in production, eg: https and TLS quality
  --html-path HTML_PATH, -hp HTML_PATH
                        Only works with Response tests activated. Path where the html response pages will be dumped after by the SP
  --exit-zero, -ez      exit with 0 even if tests fails

examples:
        src/spid_sp_test/spid_sp_test --metadata-url file://metadata.xml
        src/spid_sp_test/spid_sp_test --metadata-url http://localhost:8000/spid/metadata --extra
        src/spid_sp_test/spid_sp_test --metadata-url http://localhost:8000/spid/metadata -l test_Organization test_Signature

        # export idp metadata
        src/spid_sp_test/spid_sp_test --idp-metadata

        # test an authentication request made by a SP
        src/spid_sp_test/spid_sp_test --metadata-url http://localhost:8000/spid/metadata --authn-url http://localhost:8000/spid/login/?idp=spid-idp-test

        # select which tests to execute
        src/spid_sp_test/spid_sp_test --metadata-url http://localhost:8000/spid/metadata --authn-url http://localhost:8000/spid/login/?idp=http://localhost:8080 --extra --debug ERROR -json -l xsd_check

        # execute Response tests
        src/spid_sp_test/spid_sp_test --metadata-url http://localhost:8000/spid/metadata --authn-url http://localhost:8000/spid/login/?idp=http://localhost:8080 --extra --debug ERROR -tr

        # select which response test to execute
        src/spid_sp_test/spid_sp_test --metadata-url http://localhost:8000/spid/metadata --authn-url http://localhost:8000/spid/login/?idp=http://localhost:8080 --extra --debug INFO -tr -tn 1 8 9 24 63

        # run a test suite configured in a json file
        src/spid_sp_test/spid_sp_test --metadata-url http://localhost:8000/spid/metadata --authn-url http://localhost:8000/spid/login/?idp=http://localhost:8080 --extra --debug INFO -tr -tj tests/example.test-suite.json

        # select which user attribute to return in response via json file
        src/spid_sp_test/spid_sp_test --metadata-url http://localhost:8000/spid/metadata --authn-url http://localhost:8000/spid/login/?idp=http://localhost:8080 --extra --debug DEBUG -aj tests/example.attributes.json

        # dump SP response as html page
        src/spid_sp_test/spid_sp_test --metadata-url http://localhost:8000/spid/metadata --authn-url http://localhost:8000/spid/login/?idp=http://localhost:8080 --extra --debug ERROR -tr --html-path dumps

Test metadata passing a file

spid_sp_test --metadata-url file://metadata.xml

Test metadata from a URL

spid_sp_test --metadata-url http://localhost:8000/spid/metadata

A quite standard test

spid_sp_test --metadata-url http://localhost:8000/spid/metadata --authn-url http://localhost:8000/spid/login/?idp=http://localhost:8088 --extra

Print only ERRORs

spid_sp_test --metadata-url http://localhost:8000/spid/metadata --authn-url http://localhost:8000/spid/login/?idp=http://localhost:8080 --extra --debug ERROR

JSON report (add -o filename.json to write to a file)

spid_sp_test --metadata-url http://localhost:8000/spid/metadata --authn-url http://localhost:8000/spid/login/?idp=http://localhost:8080 --extra --debug CRITICAL -json

Given a metadata file and a authn file (see tests/metadata and tests/authn for example) export all the test response without sending them to SP:

spid_sp_test --metadata-url file://tests/metadata/spid-django-other.xml --authn-url file://tests/authn/spid_django_post.html --extra --debug ERROR -tr -nsr

Get the response (test 1) that would have to be sent to a SP with a custom set of attributes, without sending it for real. It will just print it to stdout

spid_sp_test --metadata-url file://tests/metadata/spid-django-other.xml --authn-url file://tests/authn/spid_django_post.html --extra --debug ERROR -tr -nsr -tn 1 -aj tests/example.attributes.json

Test Responses and html dumps

By enabling the response dump with the --html-path HTML_PATH option, you will get N html files (page of your SP) as follows:

• test description, commented
• SAML Response sent, commented
• SP html page, with absolute src and href (god bless lxml)

Here an example of 1_True.html, where 1 is the test name and True is the status.

<!-- Response corretta. Risultato atteso: Ok -->

<!-- <?xml version="1.0"?>
<samlp:Response xmlns:ds="http://www.w3.org/2000/09/xmldsig#" xmlns:saml="urn:oasis:names:tc:SAML:2.0:assertion" xmlns:samlp="urn:oasis:names:tc:SAML:2.0:protocol" Destination="http://localhost:8000/spid/acs/" ID="_mwmzlidj-fppt-jryt-kvfx-ulaxcbnzhhtk" InResponseTo="id-LeCCYOOGkYFrysiKZ" IssueInstant="2021-03-21T18:12:43Z" Version="2.0">
    <saml:Issuer Format="urn:oasis:names:tc:SAML:2.0:nameid-format:entity">http://localhost:8080</saml:Issuer>


    <ds:Signature>
        <ds:SignedInfo>
            <ds:CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/>
            <ds:SignatureMethod Algorithm="http://www.w3.org/2001/04/xmldsig-more#rsa-sha256"/>
            <ds:Reference URI="#_mwmzlidj-fppt-jryt-kvfx-ulaxcbnzhhtk">
                <ds:Transforms>
                    <ds:Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature"/>
                    <ds:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/>
                </ds:Transforms>
                <ds:DigestMethod Algorithm="http://www.w3.org/2001/04/xmlenc#sha256"/>
                    <ds:DigestValue>7+hvbXYS5rczc1fDOD4YTnP7QzEBfaSq2LGrkQSg0yI=</ds:DigestValue>
            </ds:Reference>
        </ds:SignedInfo>
        <ds:SignatureValue>DJSOgjXJ0OrU1pgIhiv9EiI/zMDaZExcRtUSSz+2swpM9d/lQssgOxQ1wSRRD7jn
2CS4k8x7x9m92qQRIAUwXz4CScU3LVdH9/CJKPf1E0SDE7ENPlApQ6csi7USJpRL
h7lERyocGiPTsC1HbGs9AqZP+zVSenbMgmbLLidb2c9rdlzYLjm5leOG+qtec4jT
TeMEM0WGdC2iCZKJvp6Bahmpl5QrhADRad8g2ulwW2cJpB5CjRkpjtnLr+LAE/OV
CbfWzhUY3k9NiO1OE5nIqLNCW2iwe5+m8IrO5dstcFAhcNlF+/pkTvX+xnX2w7A4
HVBC/yGuGdm7iksopV0WoA==</ds:SignatureValue>
    </ds:Signature>

    <samlp:Status><samlp:StatusCode Value="urn:oasis:names:tc:SAML:2.0:status:Success"/></samlp:Status>


    <saml:Assertion xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" ID="_zpyvhqej-suvh-oekc-zvtt-iocdveoergib" IssueInstant="2021-03-21T18:12:43Z" Version="2.0">



    <ds:Signature>
        <ds:SignedInfo>
            <ds:CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/>
            <ds:SignatureMethod Algorithm="http://www.w3.org/2001/04/xmldsig-more#rsa-sha256"/>
            <ds:Reference URI="#_zpyvhqej-suvh-oekc-zvtt-iocdveoergib">
                <ds:Transforms>
                    <ds:Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature"/>
                    <ds:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/>
                </ds:Transforms>
                <ds:DigestMethod Algorithm="http://www.w3.org/2001/04/xmlenc#sha256"/>
                    <ds:DigestValue>dmd5Axp3vXNJmgOgBgfuhflWGb8wx6o2VoFeb8as35Q=</ds:DigestValue>
            </ds:Reference>
        </ds:SignedInfo>
        <ds:SignatureValue>HYfbyTCQzc/EIGEahYM9HenARSRpob3xiqUWFtQ6c9XVmznHAQgmAc/VvW5TQUWQ
b6JSPQsS6zp2/LClKXFAdka8sFV7qnz3C0Jbjc0AkeBoiMJqOcImGC1tLRNQ2j/w
JEMGnPoVDxTfsgUwYM1PWNKtnJ21Z+1G5ZFK6MsWUVpnB/NT6nDajmT5JLrqPMhb
p3Qstbfjmm4ZENfTGXdWshgZHR8qx0VRLJx1TgoSMJG0g6AyYFV6k/Xm6MCo7SOA
SWyL+3IsBJSz1rpOKZ8n2Lbo3L6z9zwexIsMklsVFq1VcNbEbtelwMAiVLRELbj8
/dM70O4D51WtO1fTr5T5CA==</ds:SignatureValue>
    </ds:Signature>

        <saml:Issuer Format="urn:oasis:names:tc:SAML:2.0:nameid-format:entity">http://localhost:8080</saml:Issuer>

        <saml:Subject>

            <saml:NameID Format="urn:oasis:names:tc:SAML:2.0:nameid-format:transient" NameQualifier="http://localhost:8080">
                    that-transient-opaque-value
            </saml:NameID>


            <saml:SubjectConfirmation Method="urn:oasis:names:tc:SAML:2.0:cm:bearer">
                <saml:SubjectConfirmationData InResponseTo="id-LeCCYOOGkYFrysiKZ" NotOnOrAfter="2021-03-21T18:17:43Z" Recipient="http://localhost:8000/spid/acs/"/>
            </saml:SubjectConfirmation>

        </saml:Subject>


        <saml:Conditions NotBefore="2021-03-21T18:12:43Z" NotOnOrAfter="2021-03-21T18:17:43Z">

            <saml:AudienceRestriction>
                <saml:Audience>http://localhost:8000/spid/metadata</saml:Audience>
            </saml:AudienceRestriction>

        </saml:Conditions>


        <saml:AuthnStatement AuthnInstant="2021-03-21T18:12:43Z" SessionIndex="_kyfjbqds-zbgy-nfhj-ewse-iqaddsdczdjd">

            <saml:AuthnContext>
                <saml:AuthnContextClassRef>https://www.spid.gov.it/SpidL1</saml:AuthnContextClassRef>
            </saml:AuthnContext>

        </saml:AuthnStatement>


        <saml:AttributeStatement>

    <saml:Attribute Name="spidCode">
    <saml:AttributeValue xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">AGID-001</saml:AttributeValue>
    </saml:Attribute>

    <saml:Attribute Name="name">
    <saml:AttributeValue xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">SpidValidator</saml:AttributeValue>
    </saml:Attribute>

    <saml:Attribute Name="familyName">
    <saml:AttributeValue xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">AgID</saml:AttributeValue>
    </saml:Attribute>

    <saml:Attribute Name="placeOfBirth">
    <saml:AttributeValue xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">Roma</saml:AttributeValue>
    </saml:Attribute>

    <saml:Attribute Name="countyOfBirth">
    <saml:AttributeValue xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">RM</saml:AttributeValue>
    </saml:Attribute>

    <saml:Attribute Name="dateOfBirth">
    <saml:AttributeValue xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:date">2000-01-01</saml:AttributeValue>
    </saml:Attribute>

    <saml:Attribute Name="gender">
    <saml:AttributeValue xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">M</saml:AttributeValue>
    </saml:Attribute>

    <saml:Attribute Name="companyName">
    <saml:AttributeValue xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">Agenzia per l'Italia Digitale</saml:AttributeValue>
    </saml:Attribute>

    <saml:Attribute Name="registeredOffice">
    <saml:AttributeValue xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">Via Listz 21 00144 Roma</saml:AttributeValue>
    </saml:Attribute>

    <saml:Attribute Name="fiscalNumber">
    <saml:AttributeValue xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">TINIT-GDASDV00A01H501J</saml:AttributeValue>
    </saml:Attribute>

        </saml:AttributeStatement>

    </saml:Assertion>

</samlp:Response>
 -->

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
  <head>

  </head>
  <body>
    <h1>SAML attributes</h1>
    <dl>

      <dt>spidCode:</dt>
      <dd>AGID-001</dd>

      <dt>name:</dt>
      <dd>SpidValidator</dd>

      <dt>familyName:</dt>
      <dd>AgID</dd>

      <dt>placeOfBirth:</dt>
      <dd>Roma</dd>

      <dt>countyOfBirth:</dt>
      <dd>RM</dd>

      <dt>dateOfBirth:</dt>
      <dd>2000-01-01</dd>

      <dt>gender:</dt>
      <dd>M</dd>

      <dt>companyName:</dt>
      <dd>Agenzia per l'Italia Digitale</dd>

      <dt>registeredOffice:</dt>
      <dd>Via Listz 21 00144 Roma</dd>

      <dt>fiscalNumber:</dt>
      <dd>TINIT-GDASDV00A01H501J</dd>

    </dl>

    <p><a href="http://localhost:8000/spid/logout/">Log out</a></p>
  </body>
</html>

Extending tests

spid-sp-test offers the possibility to extend and configure new response tests to be performed. The user can:

• customize the test suite to run by configuring a json file similar to tests/example.test-suite.json and passing this as an argument with --test-jsons option. More than one json file can be entered by separating it by a space

• customize the attributes to be returned by configuring these in a json file similar to example/example.attributes.json and passing this with the --attr-json option

• customize xml templates to be used in tests, indicating them in each test entry in the configuration file configured via --test-jsons and also the templates directory with the option --template-path. The templates are Jinja2 powered, so it's possible to extend src/spid_sp_test/responses/templates/base.xml with our preferred values

Looking at src/spid_sp_test/responses/settings.py or tests/example.test-suite.json we found that every test have a response attribute. Each element configured in would overload the value that will be rendered in the template. Each template can load these variable from its template context or use which ones was statically defined in it.

Finally you have batteries included and some options as well, at your taste.

Unit tests

That's for developers.

pip install requirements-dev.txt
pytest --cov=src/spid_sp_test tests/test_*

Authors

• Giuseppe De Marco
• Paolo Smiraglia
• Michele D'Amico

References

TLS/SSL tests

• https://github.com/nabla-c0d3/sslyze

  pip install --upgrade sslyze
  sslyze www.that-sp.org --json_out ssl.log
  

• https://testssl.sh/