DEF CON Safe Mode - Patrick Kiley - Reverse Engineering the Tesla Battery Management System Q&A

• There we go. - All right, welcome back everybody. We are here with another of our great Def Con speakers. We have Patrick Kiley here and you can check out his presentation on YouTube where he talked about reverse engineering the Tesla battery management system for more power. Welcome, Patrick. How are you doin’ today? - I’m great, how are you guys? - Doin’ all right. - Pretty good. - So can you just give us a little bit of an overview of who you are, let people hear a little bit about your presentation just in case they haven’t seen it or give a little bit of an overview recap? - Sure. So my name is Patrick Kiley, I’m a security consultant for Rapid 7, part of the penetration testing team and I got interested in this project back when I just started looking at how the Tesla worked inside, the Model S specifically.

01:08 - I’d seen how some people had hacked them and once I got a chance to peek under the hood, I just wanted to learn more and more. I was already into car hacking and I’m learning more about this and once I realized that there was an upgrade this car had that nobody had really researched and actually pushed information about, it kind of gave me a path forward. So I wanted to see like okay, how did Tesla actually make this car capable of ludicrous? And it sent me down a path that was probably the most complicated project I’ve ever worked on. - Dakka, that’s awesome. - And I believe this is your first time being a main track speaker for Def Con, is that right? - That’s correct, yes. - All right. Well, for anyone that is familiar with Def Con we have a tradition that you should be very familiar with by now, it’s called Shot the Noob.

01:56 - New speakers are welcomed into the Def Con speaker collective with a shot of a drink of their choice. That’s to bring them into the community, celebrate their joining the people that are giving back to the community knowledge. So thank you, here’s to you. Bottoms up. - Cheers, congrats! - Thank you. Woo, that’s medicine. (laughing) Okay. - All right, good. - Loosen you up before the questions start getting serious. - Yep. - Okay, so first off, I was blown away by the amount of different techniques and skill sets that you cruised through in your talk. You had hardware reversing, hardware debugging, assembly reversing, de compiling, compiled Python, diagnosing obscure third party issues, binary extraction, just a crazy amount of things.

02:58 - How did you go about piecing all these things together? How did you find your next steps to go through everything? - (laughs) Yeah, so piece by piece. I already had the mechanical skills. So in the past I’ve had a Mustang that I’ve either replaced the heads on or put a supercharger on or things along those lines. Replaced exhaust systems on, so I had the mechanical knowledge already. I’ve been doin’ that for longer than I care to admit. I think even when I was a teenager I was tinkering around with automobiles and taking carburetors apart, so complex, small items like that.

03:37 - And then just trying to take things apart and figure out how they work has been something I’ve been doing since I was a child. My parents told me about the time I locked them out of the bathroom and then proceeded to disassemble the toilet. (laughing) And I don’t remember that but they remind me of it on pretty much every occasion when I talk about something I’ve been working on, so there’s that. So really once I started getting into it, I already had the reverse engineering skills from just doing some other stuff and had the Canbus skills. If you look at some of the previous presentations I’ve done, I’ve been pokin’ around Canbus for some time.

04:17 - So having the DBC files, it was really trivial to just diagnose the stuff going on within the Canbus. From there it was just okay, how do I do these other things? I know that Can has this protocol that sits on top of the thing itself for running diagnostics called UDS and I found those ODX files and worked with Carfucker who helped show me how to use ‘em. He’s one of the main guys for the car hacking village. He showed me how to actually import those into Vehicle Spy and play around with ‘em, helped me figure out some of the security access stuff. I’d already seen some of that from Craig Smith’s work where he figured out how to pop air bags on cars, so I knew about security access I just didn’t understand the algorithm but Tesla’s algorithm, fortunately for this vehicle, was incredibly simple.

05:16 - It’s a static scene key, there’s no transform required. It’s like you request the scene and you reply with a fixed reply and boom, you’re in. So that made that part easy and then the rest of it was just piece by piece, digging through the Python code and then building a test bench, which for any reverse engineering project when you’re dealing with a very expensive piece of hardware, try and replicate it on a bench because that was invaluable. I never would have attempted half the stuff I did if I hadn’t proofed it out using a much cheaper variant of it that I just wired together and figured out how to work. - There was one jump in particular that it was right after you had your Tesla towed home, not figuring out what was going on with it.

06:05 - Do you remember the final thing that got you past that hump? (laughing) - Oh I remember it really well. - I imagine that there was some stress involved there. - Yeah, so I was into day two of trying to- - Oh. We’re having a little video loss, might be some Wifi interference or something. - Basically said no, I’m not here for the weekend, I’m over in Irvine, you’re here in Rancho Cucamonga, I’m not coming here, you can come Monday and I’m like no (laughs) I’m gonna bail out and figure the rest of this out at home.

06:40 - So I only needed your garage to drop the battery and do that stuff. If I have to come back for that, I’m really in an SOL type of situation. So stressing out that for a day and a half, I just started capturing a bunch of logs. Just from past experience, I was like maybe the logs are telling me something. The logs weren’t great but they did mention this one file and it was a firmware.rc.

07:07 - It mentioned that a couple of times, it just had that around the point where it said error and I could see that it failed. So basically I timestamped okay when the car said fuck you, I’m not actually gonna upgrade or let you redeploy the software, I noted where that time was and looked right there in the logs and saw an error about that and then I just did some searching. Turns out that the 10 Cent guys, when they did their Tesla gateway reversing, they mention that file. They mention how that file was loaded into memory. I’m like I’ve never seen that file before, I’ve seen it mentioned, I searched through all my stuff, found a couple references to it and then I just like well, let’s just go to the gateway and see if they’ll give it to me.

07:47 - And just to give you a little background, this is not while I have the car. I had already spent the two days, I was already flustered, I had to buy a last minute ticket home, flew home and I was messing around on my bench. So I went to my bench version that had the gateway and said gimme firmware.rc, it gave it to me, I’m like cool, okay, you’ve got this BMS error, I’ve already replicated that. Looked at the values in it and I actually have a copy of a map here, but there’s no way to share it.

08:17 - (laughing) But there’s basically a tab separated value file and within that file it tells you the firmware that that particular battery pack ID needs and it has a little CRC code. That CRC code needs to be in that firmware.rc file and once that was updated with the correct CRC code and the BMS I knew was flashed (drowned out) version of firmware, I could also see that CRC code being broadcast by the BMS on the can, I did that and I was like okay, well, there’s this final CRC file, how do I figure that out? I asked some people that I knew, it’s like hey, I don’t know a ton about CRC32s but here’s what I have and here’s the file itself with the CRC value and then someone said oh, it’s a jam CRC. Here’s this website you can go to to recreate it. So I recreated the last CRC line, uploaded it back to the gateway, rebooted the gateway and the error clears.

09:15 - I’m like good, we’ve got a path forward! (laughing) (drowned out) the car to actually try all this and the car was basically being an asshole as well, because the battery wasn’t engaging, it was just dealing with 12 volts, so a lot of the components would just shut down after a little while. So I basically had to make sure it had a good battery maintain on it so it sat there for a few minutes to make sure the 12 volt had enough juice and then I tried to get the gateway to go through it’s process and as soon as that car woke up and I heard the clicks of the contactors in the rear, it was like hallelujah! (laughing) I figured it out! - There was one other thing that you said you’d missed that you would only say over drinks. We did just have a drink, could you tell us? - No. - No? Okay. (laughing) - It was basically messing around with the car and putting on a feature that it didn’t really need to have that was causing the error. At different times I was screwing around with something I shouldn’t have been, so it was my own fault is really the crux of what I want to say there.

10:19 - - Fair, so it’s a catch you in person over drinks. - Yes. - I tried to slip that in, didn’t expect it to really work. - That was pretty sly. (laughing) - Nice try, (indistinct) - Yep. - Go ahead, go ahead. - Got one question comin’ in for you, that you would have had to pay Tesla about $5,000 to make this transition. What was the approximate cost for you to do it yourself? - (laughs) Counting the towing or not? - (laughs) Let’s say not ‘cause maybe somebody will learn from your lesson.

10:53 - - So the fuse itself I found on the used market for about $350. I found the contactors for about $200. The rest of that was just basically my time and effort and labor, so just basically meaning my labor. The rest of the stuff didn’t really cost me anything other than just time and then, of course, there were all the components to my bench. You can buy a Tesla MCU for about $700-$1,000 on Ebay or some of the other markets and the BMS was running between $150 and $300. I wanted to make sure I got a BMS with a shunt, that was a little bit harder, I just happened to get lucky and saw one but the bench parts I guess you can consider were major portions of that all.

11:46 - And then there was a second kit that I bought just to be safe that was actually the exact part number that Tesla used to do the upgrade. So basically people would pay for this kit and it’d be part of their whole upgrade package. I bought one of those as well to make sure there wasn’t something I was missing and that was about another $1,000. - Can people just go to their local junkyard and pull pieces themselves to create a test bench? If so, do you think the junkyards really understand the value of this equipment that you’re using? - Well so many people are doing weird things with Teslas so I’m sure they can but I’m not sure how many junkyards you’re going to be able to find that actually have Teslas out there. I found the secondary market to be really all but if you can get one and if you can get to a junkyard and you have one that has an MCU great, getting the BMS would be a pain in the… I don’t even want to think about it.

12:40 - (laughing) You basically have to drop the battery pack out of the vehicle to get at that BMS, there’s no way to get at it from any other means. So doing that at a junkyard would be problematic. But if you can get that center display and it’s not already been snagged, that’s the part that you would start with with any test bench because that’s where you learn how to root the car, that’s where you learn to do all the changes, that’s where the gateway is stored, it really is the central hub of the vehicle. - Yeah, go ahead (indistinct) - I was gonna say I keep hearing you mention the gateway, what exactly is the gateway in the car? Can you talk about that? Is it like actually a network router or is this something else? - So it’s called the security gateway. It’s a function you’ll see on newer Canbus automobiles, it basically is a device that exists sort of like a firewall that exists between multiple Canbuses and in this place the infotainment unit, the part that actually connects to the internet.

13:42 - So it acts as a ethernet to Canbus gateway. So the ethernet side is what connects to the central display, connects to the instrument cluster, and then from the central display there’s the cellular connection, the Wifi connection, a Bluetooth connection, the USB ports, and then has all the logic for how it communicates out on the internet and the rest of it communicates with the rest of the Tesla mothership but then the gateway also bridges the various Cambuses. So it can take a message from say the powertrain Cambus and copy it onto either the ethernet Cambus or the chassis Cambus, consequently it can take messages from those and other Cambuses and push ‘em onto the BMS as well. Because sometimes devices that are not on the same Bus need to communicate with one another. - So when you’re requesting the firmware, is it coming directly from the gateway or is the gateway asking these sub devices hey, send your firmware? - Just to clarify your question, are you talking about the firmware.

rc file or are you talking about 14:48 - the firmware itself? - Well, kind of both. I think you did both throughout your talk. There was one particular place where I noticed that you were issuing a command line to retrieve a file and then you made some changes, I think you even used VIM which represent. (laughing) And then you pushed it back. - Yeah, so that was all from the CID. So the CID on the vehicle that we’re talking about is Nvidia Tegra running Ubuntu.

15:18 - That and it shows basically it’s an arm version of Ubuntu. All the firmware for all the modules of the vehicle sit within that firmware image and then when people talk about how their Tesla gets updates, it has all new firmware and it pushes out firmware updates to all the various modules that need it. Some module way over here may not need an update so it may not get updated but that’s part of what the gateway and the main system do when they do an update. The gateway itself does store a few files but it doesn’t store a copy of the entire vehicle’s firmware. The entire vehicle’s firmware is stored on the central display, on that Tegra based Ubuntu system on that little EMMC chip.

16:00 - That’s the same EMMC that people complain about wearing out because of all the logging that’s going on. - Ah yes, I have heard that. - The (drowned out) image itself has that entire copy of the firmware. The gateway only has things like the firmware.rc file, the internal.dat file and a few others. There’s like hwidf.acq which is all the hardware IDs of the vehicle and it gets that from the Cambus. So the gateway does queries during an upgrade but there are also some crash files that are stored on the gateway because it’s easier for it to store it on the gateway, I don’t know why it does that, but some of the crash files are stored there as well because I saw on my bench system some images right before the vehicles crashed and oh.

16:47 - - Okay, so when you say crash, it’s not like software crashing, it’s like black box from the actual car crash. - Yeah, there’s an actual and I think there’s another actual black box device but the gateway stores quite a bit of that and I’m not gonna pretend to understand how the whole emergency data reporter function of the vehicle works but it’s there and some of the files are on the gateway. - Yeah, that might be an interesting future talk for someone. Just reversing the black box of Tesla. So you also do a lot of stuff with the car hacking village is there anything interesting going on that you want to announce while you’re here, get people to come and show up in the car hacking village? Anything like that? - Your question broke up right at the point where you were asking the critical part, so do you mind repeating it? - Oh yeah, sorry, sorry. So you’re very active in the car hacking village, is there anything interesting going on maybe Tesla related or related to your talk that you’re gonna continue working on? Something that you wanna pimp out? - Yes.

17:54 - So tomorrow at 10:00, I’ve got a deep dive into many of the techniques that I covered in the main talk and that’s basically because due to the whole change of the online format, some of the stuff got cut off to the talks. So I have some of the stuff that was cut from my talk itself but then I also expanded on some additional topics as well. So I go into some of the binary analysis, I load up the firmware insight IDA, I don’t get very far but I show that I was able to get in IDA. I do some of the live UDS techniques, the security access as well as the shunt calibration and then it’s just another brief overview. So it’s like three different things that I’m talking about and then there’s some Can DBC stuff as well.

18:45 - - Other than the setting up a bench itself, is there any particular software or tools that people require to do this kind of investigation and playground work with car hacking? - So I was using a not free tool called Vehicle Spy only because it was very powerful and made it a lot easier. - (indistinct) - The can utils and some of the other stuff out there pretty much make it possible to do whatever you want to out there. There are a ton of Canbus interfaces out there, one of my favorites is the Panda. So for this I’m gonna give a shout out to the comma.ai folks for actually making such an awesome Canbus interface. Comma.

ai is a open source self-driving tool 19:30 - that I’ll just let people go out and check it out on their own but they make a tool called the Panda, works really well with the Tesla, it can connect to three Canbuses at once. A lot of people that have done hacking on the Tesla use that and I’m very impressed with it. It’s a very inexpensive tool that’ll give you physical interface and from that you can either connect over Wifi or USB and from there directly start messing around with Linux tools that are all free and those are all in Can Utils. - That’s cool. - What’s been Tesla’s reaction to the kind of work that you’ve been doing to get you ludicrous speed? - Yeah, so when this talk was accepted I actually reached out several times and Tesla was very supportive. They actually just asked to review the slides to make sure there weren’t any surprises, I kind of told them hey, I’m gonna talk about this, this and this and they said yeah, just do us a favor, send us a copy of your slides, whitepaper, et cetera so we’ll make sure there aren’t any gotchas in there and they said yep, everything looks good, you’re good to go.

20:33 - So very supportive, they actually have a program called the security vehicle research program in addition to their (indistinct) So a vehicle that you’re interested in doing security research on can be registered with them and you won’t violate any warranties. And supposedly, I haven’t tried this, they’ll even help you with some bricking situations. Didn’t need their help on this one fortunately but they do some of the other stuff. - Sorry, excuse me, I’ve got a really annoying thing going on. There we go. (laughing) Sorry for anyone that was on the stream. I was just constantly getting calendar alerts for the next things that I’m responsible for. - Are there any other things that you’re aware of in a Tesla that you could possibly try to unlock or bypass or anything else that either you want to look into or you think that other people might want to look into to build upon your research? - I’d have to think about that one for a bit. Yeah, there are some things. I would like to see where the actual limits are in the BMS, I would like to understand those but I don’t really have a desire to push past them because you’re starting to get into dangerous territory at that point. I just want to find where they are. So it’s like all right, here’s the variable that actually controls max power, here’s the power curve that’s basically define state of charge, battery temperature and how much power’s available for the various driving rotors and the battery itself. I’d like to find those, hence the additional IDA stuff.

22:05 - And then just more or less actually get more underneath the hood of how the battery management itself and the driving rotors work. The driving rotors are not something that I actually have hardware on bench for because they’re very expensive, the whole drive inverter is inside the drive unit and the drive unit’s a multi-thousand dollar piece of equipment. - Excellent. - That’s kind of where I would like to go from there. I want to see where the steps are for the P90D and the P100D because those cars are faster. - So we did get a really good question that was for those not familiar with the whole gateway Canbus stuff, would the car be able to phone home after you’ve made the ludicrous speed modification? - Yes.

22:53 - So the vehicle, within it’s firmware itself, has a function where actually it uploads the config to Tesla on a regular basis. I think it’s vitals.json and puts it in json format. So basically they take the vehicle’s config and whenever, I think it’s when the car goes to sleep, it actually uploads that information to Tesla. So anytime someone makes a change that wasn’t done by someone else, or I’m sure even when the service center makes a change, that is gonna be seen on Tesla’s end unless of course you figure out a way around that. (laughing) - I imagine you’re probably no longer under warranty as soon as you root the front panel or something like that. - Magnus and Mossack protects a lot of stuff but I’m not a lawyer so I’m not even going to go there and try and figure it out.

23:48 - But it’s like if you root the center display and you have a problem with your brakes, how are the two related? - Yeah, fair. - But if you root your center display and you break your center display, it’s a totally different situation. That’s all the gray legal area that I’m not even going to dare delve into. (laughing) - Okay. - To follow up on that question, if Tesla does get made aware of this bypass that you put into place, could they then reverse it and remove it from you? - I’m sure they have. There are stories out there about how people had a ludicrous speed vehicle and then after they purchased it it was removed because Tesla said they audited it and found out that it shouldn’t have had it and removed it.

24:33 - And then of course, the person who bought the vehicle was pretty pissed off. I don’t know what the resolution on that was but there’s a huge amount of controversy around the whole supercharging thing. So if you buy a salvage Tesla it has supercharging disabled. Tesla does that, it’s their prerogative, it’s their supercharger network, but it makes it very difficult to go trips and people figured out, using the same techniques that I’m talking about here, on how to re-enable that again. And then there was a thing that came out recently that basically says you’re opening yourselves up to getting sued by us because you’re basically getting something that you shouldn’t be.

25:13 - My only suggestion for Tesla is why not verify the safety of it and then only enable it to where they have to pay for it because now you’ve got a source of revenue. - That was gonna be my next question because I know that after a certain point they stopped allowing superchargers to be free for Teslas but it sounds like it’s completely disabled on these. - On the salvage vehicles it’s disabled, yeah. The way I understand it is if the vehicle has free supercharging, it has free supercharging but as soon as the vehicle’s sold back to Tesla it’s lost. This particular vehicle has free supercharging ‘cause it was purchased with free supercharging so it should always have that.

25:57 - As I understand, they haven’t taken it away from anywhere they said this vehicle will have free supercharging for life unless it has been in an accident and been totaled out by an insurance company. - All right, so it wasn’t just Tesla purchasing it back and then reselling it with it disabled or something like that? - It’s like people trade in their cars. People trade in their Teslas. I think when they sell them out again they don’t have free supercharging, I don’t know. - That makes sense. - I don’t keep track of their market. - Fair, I didn’t know. It seems like a really sketchy area, so (indistinct) that I can pull out of you. - Yeah. - Their prerogative. - Yeah. - So it seems as though where electric cars are goin’ now that this is going to be a really great area of research.

26:43 - What kind of advice would you give to people that want to get started with car hacking research? Like if somebody wants to start from the ground up, what sorts of things could they do? Where should they start looking? - Start learning Canbus, that seems to be where a lot of it gets from there. Learn UDS and from there learn about binary reversing and just reverse engineering in general. Go to a junkyard, find a car that you find interesting, verify it has Canbus and start ripping modules out. We did something with our company called a junkyard hack-a-thon, where a bunch of us went out and just ripped apart various vehicles, Craig Smith helped us. He’s the writer of the car hackers handbook and we got several vehicles worth of modules.

27:25 - Even if you just get one module, take it apart, figure out what CPU it has, learn how to hardware hack it and from there have fun. - Yes, I’ve seen in some of the videos with people hacking electric cars, maybe I’m remembering this wrong, but is there any risk to a car sitting in a junkyard still holding any kind of electrical charge that people should be aware of? - Yeah, ‘cause high voltage is high voltage. It’ll hurt you either way. I have personally never been to a junkyard where I’ve seen electric cars. I’ve only seen the salvage yards, the warranty auction places, and you see those online but the actual junkyards themselves I don’t think I’ve ever seen an electric vehicle in a junkyard. Even a hybrid would be dangerous though because the hybrid still has high voltage ‘cause it has an electric motor. - All right, we got one question. Well it’s kind of more like…

28:25 - Hawkeye’s wondering if Teslas have a problem with shutting down if their error messages reach a critical amount of storage space. I’m guessing that has to do with the EMMC logging thing? - I’ve never seen one run out of storage space. It’s just the volume of logging and I believe at a certain point it overwrites itself but it’s the volume of constantly writing to the EMMC that wears it down, not for the device storage actually filling up. And it’s unfortunate because it’s a $20 part. The EMMC module itself on that Jaguar is a very inexpensive part but it’s a $2,000 repair if you take it to a service center. - That’s fine.

29:06 - So as we start wrapping this up, what sorts of takeaways do you want people to have from your presentation and what would be your call to action or what would you like to see come about based on your research and your presentation that you’ve put out there? - Well, I’d love for Tesla owners to be able to work on their own vehicles. For there to be a consumer version of the tools that are used to work on and diagnose the vehicles ‘cause right now you just have to take it to the garage and pay the service center to do that. I can understand how the high voltage components why you’d want to do that, but as these cars age and continue to be out there in the fleet there needs to be another method. In addition to that, this is kind of like the next version of hot rodding. People are gonna figure this out now that electric cars are becoming more and more mainstream, there are gonna be people that want to buy them and make them faster.

30:04 - And you can make them faster but either lightening them or you can make them faster by tweaking what’s under the hood just like you would on a standard internal combustion engine. So this is kind of like the next phase of where that’s gonna go. - Awesome. Thank you so much for doing this, thanks so much for your presentation that people can go see on the YouTube with the Def Con channel. So thanks for doin’ this, Patrick. Really enjoyed you discussing this with us. - All right, thank you. .