Cutting Gordian knots: Funding improved Kerberos testing in Samba

by Andrew Bartlett

Enjoy this blog by Andrew Barlett as he reflects on his experience with improving Kerberos testing within Samba thanks to LMAX Group who are as passionate about building solid future-proof foundations as we are.

The old test suite the Samba team are replacing is similar to sand in the way that each time the team touched it or what was already built on - it just crumbled because it was sensitive - often a good thing in a test suite - but too sensitive to be practical in this case. 

We hope that others can build on this, leading to more secure Kerberos protocol features and finally to Windows 2012 Functional Level support.

Now let’s get technical. Samba as an Active Directory DC is intimately entangled with the Kerberos protocol and in particular the two major Kerberos implementations for Unix systems, Heimdal Kerberos and (later) MIT Kerberos.

Samba has embedded (known also as vendoring) a patched version of Heimdal Kerberos, which has sadly not been kept up to date. While known security issues are addressed via backports, this is a significant technical debt that needs to be addressed.

Samba is also a project that places a very high value on protocol-level testing, and we have therefore developed tests for our Kerberos KDC that rely deeply on the Heimdal Kerberos library we include.

While these tests were an incredible step up in terms of coverage, and they have done much to avoid regressions in our KDC, they have also become a Gordian knot - the tests became incredibly sensitive not only to the behaviour of the KDC but also to the library on which they were based.

For example; the tests would intercept the packet stream issued by the library, and in some cases even modify those packets. Improvements to Kerberos security broke the modifications and changes to packet orders broke the tests badly.

In attempting to import a new version of Heimdal we have had to find a way to cut this knot. The approach that the Samba team has chosen to take is to reimplement our Kerberos test suite in Python, directly building the ASN.1 packets and calling Python, not the Kerberos cryptographic functions.

This allows Samba to test protocol elements not exposed by Unix Kerberos libraries, as well as ensuring that behaviour is identical (or at least known not to be) regardless of which implementation or version Samba uses.

This approach was demonstrated initially by fellow Samba Team member Stefan Metzmacher of SerNet and is now being extended thanks to funding from Catalyst client LMAX Group, leading independent operator of institutional execution venues for FX and cryptocurrency trading.

This too was a Gordian knot: while many have seen the risks and challenges in Samba's reliance on an old version of Heimdal Kerberos, finding a way forward has been intractable until this new approach was taken.

With the work to start replacing the problematic krb5.kdc.the test suite has now started, we are on the path to a full upgrade to Heimdal. We are also, if we so chose, also closer to replacement with MIT Kerberos. As we implement more in the test suite we are becoming more confident that our testing can become a constant benchmark against which to measure our progress, not one changing concurrently with the upgrade.

Funding Samba improvements is also often a Gordian knot, while all users benefit from upgrades to the fundamental underlying infrastructure, big projects like our Kerberos upgrade need foresight from those willing to fund the early work.

Catalyst's goal is to bring Windows 2012 FL support to Samba, as this includes important Kerberos features like FAST (Kerberos armouring) and 'Claims', linking authentication to the user's device as well as their own account. We continue to seek partners to continue this important work, as with each project and partner we get closer to this goal.

 

Contact the Samba team if your organisation is interested in further improving Samba in this area.

Many thanks to LMAX Group for funding Kerberos FAST testing and parts of the Python test re-implementation.