by Andrew Bartlett
In November 2021, Samba and Microsoft did a coordinated release of a critical security issue that I like to call the “Dollar ticket” attack on Active Directory Kerberos.
In this blog I wanted to share some of the lessons from that year that led up to that final release in late 2021.
Staying curious and trusting your gut
My first worries about this issue came about when patches to add support for a curious additional alias were included into Samba’s KDC – the central Kerberos component in our Active Directory. Computers in Active Directory have accounts in the domain, ending with a $. My colleague on the Samba Team Stefan Metzmacher added support for a Kerberos login with the name computer to match an account with name computer$.
This work was done to match Microsoft’s windows and to fix a bug in Samba.
And so it sat, with this feeling of "this is not quite right" but nothing more. Thankfully on the Active Directory side, at the KDC, an exact match always wins.
The idle mind and turning over rocks
At the end of 2020, a work stream at Catalyst came to an end, and I was helping with the specifications for some testing work that Catalyst was planning, a job we were very pleased to have sponsored by our long-time client LMAX Group. The trailing-$ question came up again, and I started thinking again, this time turning my attention to traditional ‘MIT Style’ Kerberos – a system that has always been well specified in terms of string-based names, not the unique and numeric “SIDs” used in Active Directory.
I realised that nothing in Active Directory would prevent an unprivileged user from creating an account called called root$ and to then present a ticket with that name to a traditional MIT style Kerberos-accepting services and have them read as root (the main administrative user on Linux).
(The unprivileged bit comes from an Active Directory mis-feature known as MachineAccountQuota. Samba doesn’t implement this.)
Talk to and listen to experts
I reached out to Microsoft via our good friends at the Protocols Interoperability team, and got a call with Microsoft’s Kerberos Lead after the summer, the agenda of which was to tell me that this was exactly by design, and that nothing would be done.
Samba has since warned, about this behaviour, as it remains the case that unprivileged users in Active Directory can create such accounts via the MachineAccountQuota feature.
However the call did not go as planned – as we continued the call our thought-experiments came to other ways that this issue could cause trouble, and by the end I thought a security process had started.
This wasn’t the only time just talking it over mattered, later in the year, as we coordinated with Microsoft, some off-hand remarks alerted me to other critical issues we had missed in Samba, and we are very grateful to have been able to fix them as the same time.
Nothing starts until a case is lodged, with a full scripted exploit
An important lesson I learnt is how to interact professionally with MSRC, the Microsoft Security Response Centre.
Busy and stressed as I was, and on the back of a successful call with Microsoft’s engineers, I thought that a simple text-based walk-though and some hand-waving would then get the process started. Instead, it just lead to confused triage engineers and stalled cases.
What turned things around was the incredibly fine work of Joseph Sutton, who joined Catalyst in 2022 and quickly became quite skilled at building Kerberos testsuites.
Joseph built the infrastructure, and then full working exploits for the issues we knew about, and then was able to explore some new, even more dangerous ideas. At the time we only knew of an exploit via the RODC codepath, but with the tooling we were able to exploit any domain, to the full admin user, via another Kerberos feature S4U2Self, or Kerberos impersonation.
Flipping the attack above around, we instead got a ticket for a normal, unprivileged user DC, but were able to get the KDC to treat it as if it was DC$, the deeply trusted account of the domain controller.
This, as you might expect, got Microsoft’s attention.
Who to share with?
Holding in our now literally shaking hands was an exploit that could blow open any default-configuration AD domain currently in deployment, with only a user account.
I had tried to keep the issues quiet, even on the Samba Team, due to the concern, but developers who have not been told about the issue, confidentially, can’t fix the issue either. In the end I shared exploits with fellow Samba Team members who needed it, and the details with the broader team in line with our normal security handling.
Thankfully a group of developers from a number of Samba supporting companies, Red Hat, SuSE, SerNet and Symas joined Catalyst in the work upstream. I do want to make an extra thanks to Symas, who provided long-time team member Nadezhda Ivanova to the effort, despite not currently having a Samba product under support.
Later we would similarly take care in the reveal of the issues first to vendors, and then to the broader public, even then never publishing the full exploit.
Calling on a good friends, and making more.
At this stage we had a big issue, but no funding to fix it. Thankfully a Catalyst client was willing to fund a significant chunk of the work for the AD DC, but their use case didn’t extend to the fileserver, nor to the RODC, where we initially found the issue.
For the fileserver issues the broader Samba Team stepped up, but for the RODC issue I needed outside funding assistance. It was at this point when I called on a good friend, Arvid Requate of Univention. Univention was really awesome in supporting the required fixes for Samba’s AD DC at an early stage when we were deliberately vague in describing the full impact, and I greatly appreciate the trust they put in me.
Likewise, as mentioned above it looked for a time like we would need to revise the Access Control layer in Samba’s AD DC. This code is complex and while well tested, it hasn’t had a lot of work over the past few years, so few developers understand it really well. I was incredibly appriciative when Nadezhda Ivanova, a good friend and the original author of this feature was supported by her employer to return to Samba development to assist in this area.
Friends matter, and in security friends really matter and I thank all my good friends, both old and the new friends I made at Microsoft during the time.
The challenge of timing, and not saving the world
As weeks turned to months and deadlines were (thankfully) extended, we all were pretty exhausted. I had esimated the task, but with all the extra issues found along the way, I was incredibly grateful when the October deadline was moved to November. Even so, I’m reminded why my PM always asks me to triple my initial estimates, as we really pushed hard to get the patches all integrated and released.
In the end we made it, with less notice to our vendors than we would have liked, but everybody got patches, including back-ported patches for those who had commissioned those, before Microsoft released and the issue was disclosed.
But we couldn’t do everything. I mentioned above about a root$ -> root mapping that is still possible particularly in Microsoft Windows Active Directory, and I still wish that there was a practical way we might have been able to do coordinated disclosure and avoid that issue. In the end this has been left for each deployment to address, with few overarching mitigations possible.
Sadly as I was told early in 2021, this is pretty much “by design” in the protocols.
In the end I had to choose that while the number of deployments mixing Active Directory, MIT-style Kerberos and Linux (but not using Samba) is important, that I held in my hand an exploit all the way to Domain Administrator. We needed to fix that, and to fix that we had to disclose.
As someone who likes to ‘solve for everything’ and takes an outlook on life of choosing what is best for others, for the world, this was perhaps the hardest thing.
I also knew that after the 2021 we all went though that I couldn’t still be doing this work at this pace in 2022.
Final thoughts
Everything I ‘discovered’ and disclosed in this area is actually in public documentation. There is a great article on MachineAccountQuota at NetSPI and the problematic lookup algorithm is public in MS-ADTS. The default non-canonicalisation feature is standard Kerberos per RFC4120 and presumably all of this has been this was since Windows 2000 first introduced Active Directory. I wonder how many others found this before me, how many networks were already 0wned.
Thankfully it look a full month before security Twitter joined the dots between the coordinated releases. Hopefully some domains were patched by then. Exploits have now been written.
Find out how Catalyst can help with your Samba support and development needs.