DEF CON 29 - Laura Abbott, Rick Altherr -Breaking TrustZone M: Privilege Escalation on LPC55S69

00:17 - We are engineers at Oxide Computer working on building a new server.

00:21 - Computers haven’t changed much in many years, and we are looking to fix that.

00:25 - No, I promise this isn’t a sales pitch. I’m giving background to explain what we were building when we stumbled across this issue mostly by accident.

00:34 - One of the reasons we are looking to build a new server is to have some level of security assurance at the hardware level.

00:40 - As part of the server, we are building a hardware Root of Trust using a security-focused microcontroller.

00:46 - All of our software is written in Rust, and the goal is to have everything open-source by the time the product ships as we are big believers in transparency in software.

00:54 - This will be a theme throughout the presentation.

00:57 - We had some pretty specific requirements for what we were looking for in hardware.

01:01 - We need a strong assertion regarding the integrity and authenticity of Root of Trust firmware.

01:06 - We want to be able to use the Root of Trust to extend trust to other parts of the system.

01:10 - And if we’re not confident in what’s running on the Root of Trust, we can’t do that.

01:15 - Part of building up that trust involves having a tamper- and impersonation-resistant unique ID.

01:20 - If the device does get compromised, we need a way to reestablish that trust.

01:25 - Other features you might think of, such as cryptographic accelerators or key storage, were considered pluses, but not strict requirements.

01:34 - We did a pretty wide survey of hardware, including programmable logic devices.

01:39 - One requirement we didn’t explicitly state, but turned out to be pretty key, is that the hardware must be available for us to purchase.

01:46 - This ruled out more than one device. We were also biased towards devices that can be programmed with open-source tools, because again, transparency.

01:56 - There was no one standout winner among all the candidates.

01:59 - And based on our trade-offs we made, we decided the hardware that best met our needs was the LPC55S69 from NXP.

02:08 - The LPC55S69 is a dual-core ARMv8-M Cortex-M33.

02:16 - Core zero has TrustZone-M and Memory Protection Unit support for isolation.

02:22 - Core zero does not have TrustZone, but it can be restricted at the bus level, similar to other DMA engines.

02:27 - Rick will talk more about this in a bit. There’s cryptographic accelerators for AES, SHA, and doing field math with ECC.

02:35 - The chip supports secure boot using RSA signatures, as well as Authenticated Debug There’s a PUF, physically unclonable function, for use with keys.

02:45 - Part of the flash can be locked down after programming to prevent modification.

02:49 - Other variants of the chip support CAN, which is common in automotive use cases.

02:53 - There are a lot of appealing things about this chip, and it seems like it can be used in a lot of places.

02:59 - But, of course, we have to ask ourselves, why should we believe this is secure? NXP presents this as a great chip to use in your IOT and other edge cases.

03:08 - And a lot of the written material sounds great, but it’s marketing material.

03:13 - How does this actually work in practice? Trust, but verifying all that.

03:17 - What we found when we started working with the chip was that the documentation was not great.

03:22 - It was frequently pretty confusing. And this was especially true when it came to secure boot.

03:30 - This is the diagram for the user manual for the LPC55 showing the boot flow from power on.

03:35 - There’s a number of steps here and also ways things can go wrong.

03:39 - When trying to make secure boot work, we ran into some issues.

03:44 - So naturally when the manual doesn’t provide information, you turn to the application notes.

03:50 - And the application notes give a slightly different boot flow chart with a few more steps.

03:55 - This updated flow chart mentions the NMPA, which I’ll talk about more in a bit.

04:00 - It also mentions the Debug Mailbox, as well as the recovery boot path.

04:04 - These are two entirely new code paths that the manual apparently just didn’t feel the need to mention.

04:09 - This pattern of making a passing reference to something significant or omitting something completely was a recurring experience with the NXP documentation.

04:19 - We continued to run into surprises while reading the documentation.

04:22 - As part of our due diligence, we were reviewing all the application notes.

04:28 - I found one labeled camera interface in LPC55, which was very surprising, given the manual does not mention a camera hardware block anywhere.

04:38 - The application note contained the line, “There is a hidden coprocessor in LPC55S69 which can handle the signals of camera. ” When I mentioned this to my colleagues, they thought I was joking because a hidden coprocessor or even a camera sounds like a bad idea in a chip that’s supposed to be secure.

04:57 - We had to ask NXP for details about this, what this thing even was, because they usually don’t give out this information unless you’re a high-volume customer.

05:05 - We did eventually find out that this core is named EZH.

05:09 - It’s a custom core with a custom instruction set architecture really designed to offload wire protocol conversions and that don’t need to run on the main ARM processor.

05:22 - The documentation also contained a passing reference to DICE CDI.

05:26 - DICE stands for Device Identifier Composition Engine and Compound Device Identifier.

05:32 - This was a feature that comes out of the Trusted Computing Group, and is designed to give it cryptographically strong identity to the chip based off of the first mutable code.

05:40 - The idea is that if you change the first code that’s running, you get a new identity.

05:45 - This is a really nice feature for what we’re building in Root of Trust and being able to extend trust, but unfortunately, NXP decided to defeaturize this by removing detailed documentation.

05:55 - The feature is still actually there in hardware, you just can’t know anything about it.

06:00 - And then there’s the ROM patcher. But before we go into this part of the story, it helps to have an understanding of TrustZone-M.

06:07 - And I’ll pass this off to Rick to talk more about TrustZone-M.

06:13 - - Thank you, Laura. So what exactly is TrustZone-M? According to ARM, the ARMv8-M architecture extends TrustZone-M technology to…

06:21 - I can’t do this to you. I got bored the first time I read this, and I’ve practically fallen asleep every time I’ve tried to read it since.

06:30 - TrustZone-M is very similar to TrustZone-A, which you probably know as just TrustZone.

06:36 - This is the system we’ve had for a long time on A-profile devices used in cell phones and other embedded systems.

06:42 - In TrustZone-A, the hardware is divided into a secure world and a non-secure world, and then you have multiple execution modes within each of those worlds.

06:50 - Now, in an M-profile device, there’s a couple of key differences.

06:53 - The first is that you only have two execution modes instead of the four that you would normally have on A-profile device.

07:00 - On an M-profile device, you’re gonna have handler mode, which is similar to what you might have as an OS kernel, and thread mode, which is roughly what a user space thread would be.

07:10 - The second key thing is that M-profile devices have no MMUs.

07:15 - And there’s no concept of virtual addressing at all.

07:17 - In fact, everything is physically addressed.

07:20 - And while you do have the option to include an MPU in a given chip, it depends on the chip, and so you can’t rely on memory protection in any form in the chip.

07:31 - So given all of this, how does TrustZone-M work? TrustZone-A relies pretty heavily on the MMU, so how does M-profile actually deal with distinguishing between S world and NS world? Well, if all you have is physical memory addresses, you use physical memory addresses.

07:49 - You divide up the memory space into secure, non-secure, and non-secure callable ranges.

07:55 - If you’re executing from within a secure range, you can do data accesses to any address in the system, but your next instruction fetch must be from another secure range.

08:08 - Now, if you wanna leave the secure mode and go into non-secure, you have to use the special instructions BXNS or BLXNS in order to indicate I am switching to non-secure mode and therefore the next instruction fetch will come from a non-secure range.

08:22 - Once I’m in the non-secure range, data accesses can only go to non-secure ranges.

08:28 - Also, my next instruction either needs to come from a non-secure range, or it needs to come from a non-secure callable range, and specifically be a SG, or secure gateway, instruction within an NSC range.

08:43 - And why does this exist? Well, conceptually it’s okay for a secure world to arbitrarily call into any part of the non-secure world.

08:52 - We trust the secure world more, but a non-secure application shouldn’t be able to arbitrarily jump into any code path in the secure world.

09:01 - So instead, you create this special NSC range that you place these SG instructions in, and that creates an explicit entry point for the non-secure world to call into the secure world.

09:15 - And in practice, an NSC range typically gets used like a big jump table.

09:18 - You have an SG instruction immediately followed by a branch that goes back into the secure range.

09:25 - Now, when you have an NSC region, it more or less acts like a secure region.

09:29 - The only special detail is that it can actually have these SG instructions, and it’s where NS code can actually call into S code.

09:41 - Okay, so how do we actually set up these S, NS, or NSC ranges? Well, it depends on which part of the chip we’re talking about.

09:49 - If we’re talking about the CPU, you start with the SAU, or Security Attribution Unit.

09:57 - This is similar in concept to an MPU where you’re actually defining a range with a base and limit address, and then specifying what security attribute applies to that range.

10:08 - Now, this is actually defined architecturally as part of the ARMv8-M security extension that defines TrustZone-M.

10:15 - So if you have a part with TrustZone-M, you’re gonna have a SAU; however, the number of regions available to you is implementation-defined.

10:23 - In practice, you would define four, maybe eight ranges, which is kind of similar to how MPUs often provide.

10:32 - Now, one key piece about this is that the default state is that any region not covered by a SAU region, any address not covered by a SAU region is implicitly considered to be secure.

10:45 - So in effect, the SAU allows you to mark ranges as lower security state than what they start out as.

10:56 - Now, there’s another piece where the chip vendor might have reasons why they want certain areas of the chip to be considered secure or non-secure.

11:03 - So there’s the Implementation-defined Attribution Unit, or IDAU.

11:07 - And this is a piece of fixed logic attached to the CPU core that does an address to security attribute computation, it’s just that this is fixed by the chip vendor.

11:18 - Now, in one of ARM’s documents, they provide an example implementation where you just take bit 28 of the physical address and use that to indicate secure or not secure.

11:28 - Sadly, a lot of implementations actually just picked up on this and ran with it, including the LPC55S69.

11:36 - When you do that, you end up with a memory address space that looks like this.

11:40 - You see that things like SRAM and code are a single device, but they have double the address space, and it becomes a secure and a non-secure region.

11:52 - So this means that you actually have two different ways of accessing the same physical device, and we call those the non-secure and secure aliases.

12:05 - Why is this important? Well, when you do an access from the CPU core, that address is fed to both the SAU and the IDAU.

12:14 - Those independently compute what the security attribute is, and then the most secure of the two is what’s chosen to actually do the restrictions.

12:22 - So if you use the secure addresses… If you use a secure range from the IDAU’s perspective, it can never be lower privilege.

12:33 - The IDAU will always win as having it be a secure address.

12:36 - And this means that you can write secure code, a secure world code using secure aliases and never have a chance of it being anything else.

12:44 - On the other hand, if you use a non-secure region from the IDAU’s perspective, you have to go look at the SAU to decide what its security attributes actually gonna be.

12:53 - It could be secure, non-secure, or non-secure callable.

12:56 - And this all depends on what the chip vendor actually did as far as what the IDAU layout is, and then the application as far as what the SAU configuration is.

13:07 - Okay, so what happens once you have that request and you’ve computed what the security attribute is? Where does the policy get enforced? Well, this is mostly implementation defined, but they do have some common names for things.

13:21 - So first, you’re typically gonna have the CPU core sending out a request over the AHB interface.

13:27 - And in a TrustZone-M world, you actually extend the typical AHB matrix to include the security attributes.

13:34 - And that’s called “Secure AHB” or “S-AHB. ” Some implementations actually take this a step further and include the ability to do policy-based restrictions on passing through the bus matrix based upon the security policy.

13:50 - It’s not common, but it exists. When you reach the peripheral side of the matrix, then you encounter the MPCs and PPCs, which are really the policy going into peripherals.

14:03 - Now, why do we have two different ones? Well, the PPC is more of a general, here’s a single security policy for a peripheral.

14:10 - This is the, I wanna have this entire GPIO interface be available to the secure world or the non-secure world.

14:17 - But in the case of something like an SRAM, you might wanna do more fine grain policy of having it be something like every 2K, I wanna alternate through secure and non-secure.

14:28 - And so that gets a separate MPC that allows that fine granularity enforcing of security attributes.

14:37 - Okay, so what about other AHB initiators? We have more than just the CPU core.

14:42 - Well, again, left mostly to the implementers to find out, but common terminology, you might have a Secure AHB-aware initiator.

14:50 - This is really what’s happening with the SAU and IDAU in the Cortex-M33s, is you essentially have all the built-in logic to make the decision about what the security attribute should be, and so you’re just asserting that as part of the request.

15:05 - If you have a legacy AHB device and you wanna just insert it onto a Secure AHB, you do need a way to actually set what the security attribute should be.

15:15 - So this is typically called an “MSW” or “Master Security Wrapper. ” And the common implementation is to have it be a single security attribute gets applied to all requests from that initiator.

15:26 - So when you put this all together, you get something like this, which happens to be the AHB bus matrix diagram from the LPC55S69 user manual.

15:34 - And so you can see that it starts to look like a network flow diagram, roughly.

15:38 - And you can think of it in kind of firewalling rules, where the IDAU and SAU and the MSWs are sort of setting your ingress policy and labeling, and then you might have some enforcement within the matrix itself, but then when you hit the egress side, you can enforce more strict policies about, because it came from this source or it has this security attribute, it is allowed or not allowed to go into this peripheral.

16:03 - So the implication about all of this is that most peripherals are gonna show up in both the secure and non-secure ranges.

16:11 - This is because from the IDAU’s perspective, you need to have non-secure ranges in order for you to have non-secure access to a space at all.

16:20 - If I only had a secure alias for a given device, I could only ever access it from the secure world.

16:26 - And that could be useful. The rest of the system is designed to be implemented by the application software.

16:34 - You configure the SAU, the MSW, the MPC, and the PPC to set up whatever policies you want.

16:40 - This is where you say, “I want GPIO block A to be in the secure world. ” Now, given this level of configurability, you run into a lot of potential for mistakes, and those mistakes can have devastating consequences.

16:57 - For example, you could accidentally leave the non-secure alias active for a code section that is supposed to be secure only.

17:09 - So with all of that in mind, we’ll go back to Laura to talk about how we actually found the ROM patcher.

17:21 - - Thanks, Rick. So back to the ROM patcher.

17:24 - Actually, really back to the secure boot flow diagrams.

17:27 - It turns out that just having a box labeled “Validate image” and “Validate CMPA and NMPA” isn’t really descriptive for all the ways things can go wrong.

17:39 - I managed to brick a stack of boards by screwing up various settings.

17:43 - Physically, the chips are fine, but there’s no way to change any of the persistent settings or get new code on.

17:48 - As part of NXP’s security for the boot ROM, standard SWD debugging isn’t enabled at power on until right before the jump to the user image.

17:57 - This means that if you screw up a setting, there’s no good way to figure out where it’s looping at the ROM because you can’t attach a debugger.

18:04 - The lockout for these settings also happens before ISP or the Debug Mailbox Handler, which are designed to be able to access parts of the code, so there’s really no way to be able to undo this.

18:15 - In some respects this is a positive sign because it means that lockout will happen if something does get compromised, but it’s a pretty big problem for transparency in that we don’t know all the details of what’s going on.

18:27 - All I really wanted was a complete flow chart showing all the possible ways I could brick my board so I knew not to do that.

18:35 - NXP, unfortunately, did not provide that flow chart, but what they did provide was a ROM that was not read out protected, so it was possible to save the ROM, load it up in Ghidra, and start reverse engineering.

18:46 - I had never really done serious reverse engineering before, so this was a really fun project for me to figure out what exactly the chip was doing.

18:54 - I spent a lot of time comparing the addresses against the manual to see what was being accessed.

19:00 - NXP divides the flash space of the chip into two distinct parts: the regular user flash for applications and the protected flash region.

19:08 - The protective part of the protected flash region refers to the fact that pages in this region can only be changed under some circumstances, or maybe not at all.

19:17 - The protected flash region holds a bunch of different settings.

19:20 - The region can be divided up into a couple of different areas.

19:24 - The CFPA and the CMPA are designed to be programmed by the user of the chip in the field and at manufacturing time, respectively.

19:32 - That’s what the F and M stand for. They contain settings for secure boot, and screwing them up can result in a bricked board.

19:39 - The NMPA is controlled by NXP and can’t be modified.

19:45 - The details of what each field and protected flash reading does was not fully described in the manual.

19:51 - NXP’s region in particular was really not documented.

19:55 - But apparently everything was more fully described in an attached Excel spreadsheet.

20:00 - And by attached, they mean attached to the PDF.

20:04 - I was not aware this was something you could do in terms of attaching things to PDFs.

20:11 - But you’d download the attached Excel spreadsheet and some parts are missing.

20:17 - But we can show the rows, and we finally get to see the full address space of the protected flash area, including the NXP area which showed this really interesting region labeled “ROM_PATCH, and doubly interesting because nowhere in the documentation was the ability to change the ROM mentioned at all.

20:36 - Going back to why we were interested in this chip in the first place, part of how we build up our Root of Trust is assuming that the ROM always executes the same code.

20:44 - The ability to change the ROM is potentially concerning and breaks a lot of our assumptions.

20:49 - How concerning, though? The only way to figure to know is to figure out exactly how this block works.

20:56 - Fortunately, the part for the disassembly that references flash region give a full program sequence for the ROM patcher.

21:02 - Pretty sweet. NXP’s ROM patch controller can support up to 16 patches.

21:09 - A single patch slot can patch one 32-bit ROM address.

21:14 - The block cannot be used to patch addresses outside the ROM region.

21:18 - Up to eight slots can replace the specified address with any 32-bit value.

21:22 - Other slots can be replaced with an SVC instruction corresponding to the slot number.

21:27 - SVC is the system call instruction on ARMv8-M.

21:31 - What happens here is the ROM sets up the system call table in RAM to allow for patching longer instruction sequences.

21:37 - The ROM patch settings do not persist across reset, which does lessen the impact here, but changes made to a secure address are also aliased in the non-secure region, as Rick talked about earlier.

21:51 - This is a diagram showing the registered layout of the hardware block.

21:54 - To set patch, the entire path fulfiller must be turned off.

21:58 - The target address is set in one of the address registers.

22:01 - If we’re just doing the value replacement, you write the value to one of the corresponding registers.

22:07 - The value registers are reverse-indexed which is a bit of an interesting quirk.

22:11 - The control bit determines if you’re doing a value replacement or an SVC replacement.

22:16 - Individual patches can also be turned off and on.

22:19 - Importantly, these settings are not sticky, so you can change settings as much as you like.

22:24 - This means it’s possible to patch one set of addresses, turn off the controller, then patch another set of addresses.

22:32 - It’s worth noting that this is probably a custom NXP design.

22:36 - Previous generations of ARM had a standardized flash patch and breakpoint unit to a lot of remapping of arbitrary flash regions.

22:44 - This was associated with at least one persisting exploit that was presented at a previous DEFCON.

22:51 - ARMv8-M removed the patching feature of the unit probably because the designers realized that remapping could potentially break isolation with TrustZone.

23:00 - In fairness to NXP, there are legitimate reasons to want to have a hardware patcher.

23:05 - Running software is hard, and you do need a way to fix bugs in your ROM.

23:09 - There’s a reasonable motivation to want to bring this back, but they probably should have had a bit to be able to prevent modification from curious firmware engineers.

23:20 - The ROM is the first code that’s executed at startup.

23:23 - And if that were the only use of the ROM, this block would be significantly less interesting.

23:27 - The ROM also exports a bunch of functions which are designed to be called at runtime by user applications.

23:33 - These are all features that are either hard to sequence or NXP wants to keep proprietary and would also be served by providing source code to see what exactly these are all doing.

23:43 - All of these APIs expect to be called from secure mode as all the addresses are for secure aliases.

23:48 - The Image Authentication APIs also required privileged mode.

23:52 - The addresses for calling the functions are accessible via tables inside the ROM itself.

23:58 - Okay, this is DEFCON here, so clearly it’s time to do something interesting with this hardware.

24:03 - We can put this all together to do an exploit.

24:06 - For step one, we find ROM API used by secure code.

24:11 - Step two, we used the ROM patcher from non-secure mode to change the code of the API since that code exists in ROM space.

24:18 - Step three, have a secure code invoke that API, and now you’ve won.

24:25 - I’ll pass it back to Rick to talk about writing an exploit in TrustedFirmware-M.

24:31 - - So now that we have a plan of attack for how to use the ROM patcher to create a privilege escalation, we had to go actually build a full proof of concept.

24:41 - And initially we wanted to start very simple, but it turned out that it was much easier to illustrate the severity of the problem by taking a unmodified secure world and demonstrating it with a non-secure app only.

24:54 - In this case, we chose TrustedFirmware-M. So what is TrustedFirmware-M? This is a reference implementation of ARM’s Platform Security Architecture which is really a set of APIs for common security services offered from a secure world to a non-secure app.

25:12 - This is designed to allow you to have different vendors for your secure world and non-secure world code.

25:19 - Think of it as I buy a secure TEE type environment from one vendor, maybe it comes with the chip, and then I write my own code and I only have access to the non-secure world.

25:31 - So this provides a lot of core services like cryptography, attestation, and Internal Trusted Storage, and even some lifecycle management for your device, which makes a lot of sense, and it’s actually quite useful.

25:44 - Thankfully, TrustedFirmware-M has upstream support for the LPC55S69.

25:49 - So I could just grab this and build it. Now, the piece that we’re really interested in is the Internal Trusted Storage.

25:57 - And why are we interested in that? Well, as you might expect from something named “Trusted Storage,” it deals with persistent data.

26:07 - And specifically, it’s something that is required to be implemented in the secure world in most implementations of TrustedFirmware-M.

26:16 - So this gives us an API that’s callable from the non-secure service space, it’s required to be implemented in the secure world, and it’s going to interact with persistent data storage, which means it will touch the flash.

26:31 - Does this have a good chance of using a ROM API? Well, yes it does. It uses the ROM Flash API.

26:39 - So then we can get into building our actual PoC.

26:42 - Now remember, we’re just building the non-secure section of a full PoC.

26:49 - We’re using an unmodified TrustedFirmware-M.

26:53 - So we take that TrustedFirmware-M, I used version 1. 2, which was the most recent at the time, and I build that as an unmodified image.

27:02 - And then I build a completely independent non-secure app that does the following: we need to write a little payload that fits within the capabilities of the ROM patcher.

27:11 - Remember that we only have a limited number of words that we can actually write into or patch into the ROM address space.

27:17 - And so I needed a payload that would fit in that space.

27:22 - Then, the ROM Patch Controller will be used from non-secure mode via its non-secure alias to copy that payload into empty space in the ROM.

27:32 - So we’re essentially moving code from the non-secure world into the secure world.

27:37 - Once we’ve done that, we can patch the Flash_Write ROM API to invoke that payload.

27:44 - Once we do so, non-secure world can call into the ITS API to do any sort of write, and our payload should run.

27:51 - We just have to verify it. So we’re gonna talk a lot about addressing and layout, and so it’s important to understand, in these devices, you have a single flash region.

28:02 - And TrustedFirmware-M, in a configuration with no bootloader, simply puts the secure and non-secure images right next to each other in address space.

28:11 - And then you have, you know, your other persistent data storage.

28:14 - Now, this is important because it goes into how the SAU and the MPC get configured.

28:20 - The secure world actually sets this up. And when we flip on debug mode for the TF-M secure mode or secure world, it gives us the spew about that configuration.

28:30 - And I verified that it doesn’t change when you turn off debug mode, but it was just handy to be able to have it spit it out to me over a serial port.

28:36 - But if we look carefully, you’ll notice that the SAU was being configured to mark out that section of non-secure code.

28:43 - Now, if we were to go back and look at our IDAU implementation for the LPC55, we would find that the zero range up to 1000000 is actually marked as a non-secure range, and so this is really using the SAU to also set it as non-secure so that both sides agree.

29:06 - This is a non-secure range. The remainder of that flash region will automatically become secure because there’s no SAU region covering it.

29:16 - And then there will be a additional secure alias for the entire flash as well.

29:24 - Now, we also have the NSC base, which, if you note, actually starts with a one in the address, and that’s using the secure alias.

29:31 - Remember that NSC can actually be in a secure range, so it’s perfectly fine and legit to use secure region for that, the secure alias for that.

29:44 - But because of this being configured, when we’re executing in non-secure, we should not be able to access the secure mode flash.

29:54 - It’s there, but there should be no non-secure window into that region from zero to 40,000 hex.

30:07 - Okay, so the first thing we do is look at the Flash ROM API and look at specifically to find a method that we can instrument and provide a call-out to our payload.

30:17 - Now, it turned out that the Flash_Write method is actually pretty simple and provides a really nice entry point where I can replace two move instructions with a single branch and link instruction.

30:29 - And I have a lot of registers to play with it at that moment, and it’s really easy to replay those two instructions inside my payload.

30:36 - And so constructing the payload is primarily just doing a bunch of copies of constants from a non-secure section of memory and copying it over into secure.

30:47 - Now, this is really kind of a neat trick because remember, secure mode can do data accesses from non-secure regions.

30:55 - So I don’t actually have to put the full constant pool in the secure world; I just have to have the code that reads it be in the secure world so that it can write to the secure aliases of the other devices that I wanna interact with.

31:10 - So in this case, the particular example, we’re using constants to load in and overwrite the control registers for both the the SAU and the Secure AHB to basically extend the ranges and open up both the secure mode code region as we saw in the flash layout, as well as the secure memory region, and extend those in the SAU regions that thus turn them into a non-secure space.

31:42 - Now, if we were trying to access them via the secure aliases, this would fail, but a non-secure execution via the non-secure alias with the SAU in this configuration, it should be marked as non-secure, even though it’s accessing the secure code that’s at the zero range of the flash.

32:02 - So how do we verify this? Well, it’s pretty simple.

32:05 - You just need to dereference zero. Remember, everything’s physically addressed, so zero is where the secure code actually lives.

32:11 - And we just do that from a non-secure space.

32:14 - Assuming that everything worked, we’ll just get a dump out of the actual contents.

32:21 - So here you have an NXP LPCXpresso55S69. This is just an off-the-shelf dev board for that part.

32:28 - I have two USB connections at the top. The orange one is directly connected to the USB interface on the LPC55.

32:36 - The black one goes to an onboard SWD debugger called the “LPC-Link. ” And the main reason I have that is that I can, it has a serial console or is connected on main LPC55, and it exposes it as a USB ACM device.

32:53 - So just a convenient way to get the serial.

32:56 - We’re actually gonna use the ROM bootloader on the LPC55 for most of what we need to do.

33:04 - So the first thing is we need a copy of both the TrustedFirmware-M, I’m using the tag for the 1. 2 version, and then we’re also gonna get a copy of our PoC.

33:19 - Now, I’ve said that we use an unmodified version of TrustedFirmware-M.

33:24 - There’s actually a bug in their build system, so I grabbed an upstream patch that fixes that bug for this particular version.

33:32 - Then the build system for TrustedFirmware-M uses pip, so use a virtual env for that, and then go ahead and have it fetch all of its requirements.

33:47 - Then I’m gonna go ahead and install cmake that way.

33:52 - And then, they use cmake as part of their build system, so we’re going to go ahead and say, “Build for this development board, using the GNUARM too chain with the TrustedFirmware-M bootloader disabled. ” We’re not using that in this case.

34:10 - And then also build our PoC as the non-secure application.

34:24 - Okay. So now we’re gonna go ahead and build that.

34:26 - Now, I did grab our PoC during the recording of this from a private GitHub repository.

34:31 - By the time this is available, we will have made that same repository public, and set the default branch to be the appropriate code so you can play along at home.

34:45 - Okay. So with that built, we can look and here’s the non-secure world and the secure world binaries.

34:54 - Now, what we wanna do is erase the flash and load these in.

34:59 - So the first thing we’re gonna do is use NXP’s tool for this for interacting with the bootloader, which is called “blhost. ” And it’s not the most reliable tool, so what we have to do is hold down the ISP button and press the reset button, and it’s supposed to jump into the ISP mode.

35:21 - And then we can talk to it over USB. So in this case, we got it on the first try there.

35:26 - Now, I’m going to actually reset it again and do the same thing between each step, just because I get slightly better success that way.

35:41 - Oops. ‘Kay, so what we’re gonna do is write just the secure application, the secure world, into the flash.

35:58 - Great. Now, if I reset and let this run, you’ll see that we get the debug spew that I showed earlier, showing us the SAU configuration.

36:05 - And then it actually launches the secure application, and then hangs because there is no non-secure world to execute right now.

36:14 - So now we’re gonna go ahead and put it back into ISP mode.

36:19 - And we’re gonna write in the non-secure world.

36:23 - And then if I reset, you’ll see that it actually does that same dump of the SAU configuration.

36:30 - Now, this time, it actually starts to boot our non-secure world.

36:34 - There’s a bug that I couldn’t figure out where when I first tried to use the ITS it crashes, but if I just hit reset again, it’s gonna scroll by pretty quickly.

36:45 - And I’ll go back here to show what actually is happening.

36:53 - So we started up and we got into this normal thing, and the LPC ROM PoC started as the non-secure application.

37:01 - And within that, we start a couple of threads.

37:04 - One is this TOUCH FLASH API. And this thread is literally just incrementing a counter using the ITS.

37:10 - It reads from ITS, increments by one, and then pushes it back.

37:14 - Now, we actually wait for five seconds and before we apply our ROM patch.

37:20 - So this is just the operation normally, and then the patch begins.

37:25 - And this is just a separate thread from non-secure space executing the payload that we looked at earlier.

37:33 - It’s loading the payload, patching the Flash_Write API and doing all that work.

37:38 - And in the meantime, the counter thread then, you know, keeps on running, except now it’s actually running our PoC every time this counter is invoked.

37:47 - And so then a little bit later after we’ve let that just sleep for a while, the PoC thread then is also running from non-secure code again, actually goes in and dumps out the beginning addresses of the secure space.

38:02 - So that that happens, and then the counter just continues on, and we, you know, print out a tick every once in a while to show that everything is still alive.

38:10 - So now that we’ve gone through our proof of concept, let’s talk a little bit about how we went about disclosing this to NXP and doing a public disclosure.

38:20 - Like most people and most companies, we started with the responsible disclosure process with a 90-day response window.

38:27 - And we started off with coming up with our own template because, frankly, we’re a computer manufacturer, not a security company.

38:34 - We didn’t to have to do a disclosure and didn’t have anything ready for this.

38:40 - So I put something together that wrote up a lot of the details and tried to provide impact and CVSS scores.

38:46 - And then the first question was, well, how do I send this to the vendor? Where do I even find the contact info for their page? So this is what you have at the header of NXP’s website.

39:01 - And I wasn’t really clear to me where to go.

39:06 - Maybe the Support section? Nothing particularly obvious there.

39:12 - How about the Company section? Hmm, no.

39:17 - No, nothing there either. Okay, well, maybe that’s at the footer.

39:21 - That that could be. No. No, not the footer.

39:25 - Okay, well, back to the Company. There was a contact us.

39:27 - Maybe I can go there. Oh, no. No. But wait, there’s a section that says Contact Support.

39:35 - Oh, there it is hidden at the bottom of the Contact Support page.

39:39 - Now, when I did go to this page finally after finding it, it does tell me how to email NXP’s PSIRT and gives me their GPG key along with some details about their process.

39:51 - Okay. Sent that off. We did that on December 16th, 2020, and off it went.

39:57 - And 30 minutes later, we got a response. Now, this is not an automated response, but it does actually mostly give the equivalent of an automated response.

40:08 - Like, yes, we got it, but we’ll get in touch at some point.

40:12 - So I don’t really count this as part of the 90-day like timeline for a response and coordination.

40:22 - And in fact, I waited a very long time for them to follow up.

40:26 - If you noticed, this is December 16th, 2020.

40:29 - Here on January 11th, 2021, I had to ask. Like, it’s been almost a month. Have you confirmed anything? Is it look like a real issue? And they came back a day later and said, “Oh yes.

40:42 - In fact we did confirm it and it’s an issue, and thank you for reporting this.

40:46 - Yada yada. We’ll be following up. “ And then the coordination kinda went a little wonky.

40:55 - You’ll notice from the dates and times, like there was a lot of time between these interactions.

41:00 - And it started with, “Hey, we really looked at your PoC a little bit more closely and, you know, we think we were doing some silly things. ” Now, this was our first PoC that did not use TrustedFirmware-M.

41:12 - And so it’s a little bit understandable, but this is where we came back and said, “Look, here’s one with a very clean slate.

41:17 - This does not use any modifications to your example code for how to write a secure world application, and we can definitely do privilege escalation. “ What was really annoying about this was an insistence on trying to demote the severity of the issue in this case.

41:36 - A long time after that, there was a, “Hey. Yeah, this is a real problem.

41:41 - And can we have 45 more days?” Now, note this was almost 60 days after we originally reported the issue before they actually acknowledged that it was a real issue of the severity that we told them it was.

41:55 - So we had some internal deliberation and decided, “Hey, you know what, tell us about what else is in this chip because we are trying to use it.

42:05 - We wanna be a customer. And if you can tell us that, we’ll gladly give you 45 days. “ Well, they just took the 45 days and didn’t really hold up their end of the bargain.

42:16 - We did get some material under NDA answering a few of our questions, but not a whole lot.

42:24 - One of the more interesting ones is we asked to be able to access or audit their ROM API source code because we already reverse engineered most of it.

42:32 - We just wanted to see what else we were missing.

42:35 - And this is the statement we got back. And oddly enough, they gave us permission to share this publicly.

42:41 - This is not a statement that I would want to see from a company making a security chip.

42:50 - After we had had some further discussions and were working out a timeline, I asked about, “Hey, when are you gonna actually have a CVE? Like, what are you gonna do about that? We wanna have a public disclosure. This is a big issue. ” And honest to God, the response was, “What’s a CVE?” And we said, “Okay, you know what? We’ll just file ourselves.

43:12 - I’ve done this before. Not a big deal. “ But then it comes out a month later, they come back and say, “Oh.

43:21 - Hey, we’re gonna file a CVE on this. Who should we put it as the reporter?” And I’m like, “No, we already have the CVE number assigned.

43:28 - Here’s all the details. “ Like we kind of took the wind out of their sails, but it was very delayed.

43:37 - So in the end, we spent almost 4. 5 months working on this disclosure from start to finish.

43:45 - And a lot of it was waiting for them to even acknowledge it.

43:50 - In the end, from the Oxide side, we did push out the CVE, we did a blog post going into some of the technical details, we did some tweets.

43:58 - And actually, given how poorly the response went with NXP, that was enough to fuel a spite CFP for DEFCON.

44:07 - So here we are. From NXP’s side, their disclosure, I call this public, but in practice, they wrote a security bulletin that we received under NDA and we’re told that, in fact, they’re only going to email that out to select customers.

44:24 - I have searched and not been able to find this anywhere publicly.

44:29 - I also did not receive it outside of our discussions with the PSIRT channel.

44:33 - I have no idea how you get these security bulletins.

44:36 - Somebody gets them, but good luck. As far as off-the-shelf customers, they did update the user manuals.

44:44 - If you know what you’re looking for, the two most recent revisions to the LPC55S69 user manual do talk about the security regions, but they are very light on what the underlying cause was and are more prescriptive in terms of you must write these registers to these values without explaining why.

45:03 - So Laura, what should we take away from all of this? - Thanks, Rick.

45:09 - Wrapping this up, this is the full scope of devices with this ROM patcher.

45:13 - The title of this talk mentions the LPC55S69, but it affects multiple shifts in the LPC5500 family.

45:21 - All those numbers in there actually mean something, but I’m not a chip marketing expert.

45:27 - Anything in the LPC55S6x is affected. The S in the part name stands for “secure,” which means it has TrustZone.

45:36 - There is a variant, LPC552x, which does not have TrustZone.

45:40 - This means it isn’t vulnerable to a TrustZone breakout because it isn’t there.

45:45 - This vulnerability can still be used for privileged escalation however.

45:48 - Slightly worrying, the LPC552X does not have the extra level-setting bits without TrustZone, so the only prevention against it is the NPU.

45:58 - One of the questions we asked NXP when we were discussing was if there was an ability to lock the patch configuration to prevent changes.

46:05 - The answer there was no. And it sounded like this was a deliberate product choice with the idea that maybe they would need to give a patch to customers later.

46:13 - This seemed like a bad choice, but I don’t work for NXP product strategy.

46:17 - I mean, maybe they could’ve put the lock bit in there and let these risk guide, but again, out of my hands.

46:23 - NXP did mention that the upcoming LPC55S3x would include the ability to lock the ROM patch, which was really good to hear.

46:32 - We’re not sure if this is all effective devices.

46:36 - NXP was willing to acknowledge the TrustZone vulnerability, but wasn’t willing to admit this was a privilege escalation.

46:42 - There may be other chips which have the same controller we don’t know about, which is really concerning from a security perspective.

46:50 - Finally, some important takeaways. Code is hard.

46:54 - Configuring hardware is hard. Configuring hardware security is really hard.

46:58 - TrustZone-M is hard to configure correctly.

47:01 - There’s a lot of knobs. Requiring code to drop permissions is a great way to accidentally end up with something still exposed.

47:08 - We’ve seen this in Linux plenty. Reference implementations are frequently incomplete.

47:13 - The trusted firmware PoC we gave could’ve been prevented by setting a few lock bits to prevent changing the SAU, but they just weren’t set.

47:21 - Just taking a reference implementation is no guarantee you’ll be secure or you won’t hit odd edge cases.

47:28 - You need to be doing audits more view of all the reference code.

47:32 - Transparency is so important in your hardware.

47:35 - In order to drop privileges, you need to know that you need to do so.

47:39 - The ROM patcher block was labeled as reserved in various parts of the memory map because NXP did not want to acknowledge it existed.

47:46 - The old manual before their update would tell you not to write anything for the reserved area for various security protections.

47:53 - This means that even if you were following the manual to the letter, you would not be protected.

47:58 - Oxide and NXP really seemed to have a fundamental disagreement about how big of an issue this is.

48:03 - Our stance is that unless you have full knowledge about what your hardware is doing, it is not possible to write a truly secure implementation.

48:11 - Turning that reference code into something correct is incredibly difficult without full documentation.

48:15 - Hardware makers should really want to make it easy to write secure code and to treat undocumented hardware with the antithesis of security.

48:22 - And really, please just give us source code for the ROM.

48:26 - The conclusion here should not be that NXP should have re-protected the ROM so that we couldn’t find the bug.

48:32 - That is not driven to be an effective solution.

48:35 - Rick and I are not professional reverse engineers.

48:37 - We’re developers who think this kind of problem is really neat.

48:41 - I’d like to say there were no more bugs lurking in that ROM, but I really don’t know.

48:46 - People who are much more experienced in this area could probably find more bugs.

48:50 - There’s certainly been a history of bugs in the ISP mode, some of which have been presented at DEFCON.

48:55 - NXP could really increase the confidence here by letting us all review their code.

48:59 - Just saying that the third party reviewed the code doesn’t really tell us anything if we don’t know what was found or fixed, especially if the audit didn’t actually manage to find this hardware block.

49:10 - Hiding things doesn’t actually make it more secure; it only delays the time till bugs are found.

49:14 - I really look forward to the future when hardware manufacturers have complete correct hardware documentation available so we can all have more secure and open software.

49:22 - Thank you. .