DEF CON 29 - Guillaume Fournier, Sylvain Afchain, Sylvain Baubeau - eBPF, I thought we were friends!

00:08 - And today Sylvain and I are going to present the rootkits that we implemented using eBPF.

00:14 - If you don’t know what eBPF is, don’t worry, we are going to present this technology and tell you everything you need to know in order to understand the talk.

00:22 - So let’s start with a few words about us. We are the cloud workload security team.

00:27 - We usually use eBPF for good. And our goal is to detect threats at runtime.

00:32 - Everything we do is added to the Datadog engine, which is an open source project.

00:36 - So feel free to check it out if you are interested.

00:39 - That being said, for DEFCON, we decided to use everything we knew about eBPF to build the ultimate rootkits.

00:46 - So, as I said before, we are going to start the talk with a brief introduction to eBPF.

00:50 - Then Sylvain will take it over to talk about how we implemented obfuscation and persistent access in our rootkits.

00:57 - After that I will come back to present the command-and-control feature along with some data exfiltration examples, and then I will talk about the network discovery, and RASP bypass features of the rootkits.

01:08 - And finally Sylvain will present a few detection and mitigation strategies that you can follow to detect rootkits such as ours.

01:17 - All right, so let’s start with eBPF. EBPF stands for extended Berkeley Packet Filter.

01:24 - It is a set of technologies that can run sandbox programs in the Linux kernel without changing the kernel source code or having to load kernel modules.

01:32 - It was initially designed for nitro packet processing, but many new use cases were progressively added.

01:38 - So for example, we can now use eBPF to do kernel performance tracing along with network security and runtime security in general.

01:46 - So how does it work? So eBPF is simply a two-step process.

01:51 - First, you have to load your eBPF programs in the Linux kernel and then you need to tell the kernel how to trigger your programs.

01:58 - So let’s have a look at the first step. EBPF programs are written in C.

02:03 - So it’s not exactly C, it’s more like a subset of C because of many restrictions that eBPF has to follow.

02:09 - But I’m gonna talk about this later. So once you have your C program, you can use LLVM to generate eBPF bytecode, which you can then load into the kernel using the BPF syscall.

02:22 - EBPF programs are really made of two different things, eBPF maps and the actual program.

02:28 - So there are a lot of different types of eBPF maps, but all you need to know is that they are the only way to persist data generated by your eBPF programs.

02:39 - Similarly there are a lot of different program types and each program type has its own use case.

02:45 - However, regardless of the program type, each program has to go through the same following two phases.

02:51 - So the first one is the verifier step. So I will talk about this later, but for now, just know that this ensures that your program is valid.

03:00 - And second, your eBPF bytecode will be converted into machine code by a just-in-time compiler.

03:08 - And when those two phases succeed, your program is ready to be executed.

03:15 - Step two is attaching eBPF programs. So in other words, this is when you tell the kernel how to trigger your program.

03:24 - So there are many different program types and I can’t present them all, but I’m just going to talk about four of them.

03:30 - So for example, you can use kprobe to trigger any BPF program whenever a specific symbol in the kernel is called.

03:38 - Tracepoints are similar to kprobes, but the hook points on which they can be attached have to be declared manually by the kernel developers.

03:46 - Those two programs require an older syscall in order to be attached.

03:52 - And this is called as the perf_ event_ open syscall.

03:55 - So the other two program types I wanted to talk about are TC classifiers, so it’s sched_cls and XDP programs.

04:02 - So those program types can be used to do packet processing.

04:07 - So whenever some network traffic is detected at the host level or at a specific network interface level, those two require the netlink to be attached.

04:17 - And the only thing to remember here is that each program type has its own set up and does not require a different level of access.

04:25 - Another very important fact about eBPF is that the BPF maps can be shared between different programs regardless of their program types.

04:37 - All right, so the eBPF verifier. So the verifier is used to ensure that eBPF programs will finish and won’t crash.

04:46 - To do so it’s really just a list of rules that the verifier checks and your program has to comply with those rules.

04:55 - So for example, your program has to finish, it cannot be like an infinite loop.

04:59 - So your program has to be a directed acyclic graph.

05:04 - You can’t have unreachable code, you can’t have unchecked dereferences.

05:07 - Your stack size is limited and your overall program size is also limited.

05:12 - And finally, one of the most infamous features of the verifier is it’s very cryptic outputs.

05:18 - So basically if your program doesn’t pass the verifier step, you will have a huge log of everything that the verifier looked into and eventually some kind of error telling you what happens.

05:31 - But yeah, basically you are in for a very painful beginning session.

05:37 - Last but not least, eBPF comes with a list of helpers that will help you access data or executive operations that you wouldn’t be able to write natively.

05:47 - For example, you have context helpers, you have map helpers, a lot of things that you wouldn’t be able to write in C and that you would need external instrumentation to do.

05:57 - In short, you have about 160 helpers. And most of the heavy lifting of your eBPF programs will be based on those helpers.

06:09 - So that concludes this introduction to eBPF.

06:11 - And I will hand it over to you, Sylvain, so that you can kick off the presentation of the rootkits.

06:17 - - Thank you, Guillaume. Before we get into the details, let’s see why eBPF is an interesting technology to write a rootkit.

06:26 - First, the safety guarantee. Broad eBPF mean that a bird in our rootkit cannot crush the host.

06:32 - (indistinct) will not cause any low message to be emitted.

06:36 - The user therefore has no way to know that something actually went wrong and notice the presence of the rootkit.

06:44 - As we saw earlier, the eBPF bytecode is converted to native code.

06:47 - And the number of instruction is limited, which limits by extension, the performance impact that our rootkit can have on the machine that could otherwise be detected by the user.

06:59 - On the commercial side, the eBPF is used by an increasing number of vendors in various use cases, network monitoring security, for instance.

07:07 - With eBPF becoming widespread, returns of one product being abused, too many issues programs also increases.

07:15 - The safety guarantee we just told you about should not give the security and (indistinct) before filling of security.

07:24 - There’s a lot of activity around eBPF and each new version of the Linux kernel comes with a new set of eBPF helpers, bringing new capabilities.

07:33 - As we wanted our rootkit to run on a widely used distribution on Linux such as RHEL/CentOS Linux, or the latest Ubuntu LTS, we used a limited number of helpers using RHEL/CentOS or a feature like KRSI will have probably made the development of the rootkit easier.

07:55 - One of the primary tasks of a rootkit is to ask itself, what does it mean in our case? EBPF programs are bound to a running process.

08:04 - If this process gets killed, all the attached eBPF programs will be unloaded.

08:09 - For that reason, it is essential that we boast either our program and protect it from being killed.

08:16 - The eBPF programs and maps used by the rootkit should also be hidden and we should forbid other programs to gain access to them, so there are five descriptors.

08:27 - So let’s see a rootkit in action. So let’s start with kit, would give us its PID.

08:36 - Then we can try your PS kernel in order to see if we can detect it from the output.

08:42 - We can try using its preface entry and nothing.

08:47 - We can even try using your such file or even a relative pass.

08:55 - And we still have the same issue, no such file or directory.

09:00 - And finally, we can try to send a signal to see what happened, and we get no such process error.

09:12 - Your prescription capabilities of our rootkit mainly rely on the use of two BPF helpers.

09:17 - So BPF provides user helper arose our BPF program to ride into machinery of the process that issues the syscall.

09:25 - This can be used for instance to alter the other data that is returned by your syscall.

09:30 - It’s also possible to alter the syscall arguments.

09:33 - So it’s one caveat with this helper, the memory to be modified has to be mapped into the kernel space.

09:40 - Otherwise a minor or major pitfall will be triggered causing the BPF provider user call to fail.

09:48 - User BPF helper is to ease the BPF override return.

09:52 - This one allows you to change the return that you have a syscall and as an interesting property.

09:57 - If this helper is used at the syscall exit, it will simply change the return that you will do at the syscall.

10:03 - But if we use it at the entry of the syscall, the execution of the syscall will be completely skipped.

10:09 - It is important to note that this helper can only be used at the entry of the syscall or at the exit.

10:18 - So let’s see how obfuscation of a file actually works.

10:21 - At startup, the rootkit will populate the map with the pass of it’s PID folder.

10:27 - Now the user space issues a file with syscall such as stat.

10:32 - The syscall usually come in two forms, one that accept the pass to the file as a string, another one that accepts the VFS for the file, but the user space program must have previous (indistinct) using an open syscall.

10:47 - So let’s consider the former. To properly identify the targeted file, the rootkit needs to do an accurate resolution of the pass as the pass specified could be a relative pass.

10:57 - At the entry of the syscall, that is not in context to the resolution.

11:01 - So we need to go deeper in the kernel in our case, in DBFS code.

11:05 - So we are at the region, but at this point we cannot block the syscall as we are outside of the override of points for the eBPF override return helper.

11:14 - So the only thing that we can do is to change the return value so that the user space believes that the syscall failed.

11:21 - We also need to scrub the content of the switcher that could have been filled by the kernel.

11:30 - Now let’s consider the latter, the ratio that accept the file as return.

11:34 - We do the same passage version as before, but instead of just pretending it was the descriptor file that the syscall failed, we still have the file descriptor that we can edit located into an eBPF map.

11:43 - If the same processes issue the syscall with the same file descriptor, we can add the syscall entry return an error and block the syscall.

11:51 - In this situation, the user has no way to know that the file descriptor exist.

11:55 - And as we control the read syscall, we can also hide all the references to the file descriptor in proxy phase.

12:06 - Blocking the syscall that accept the PID as an argument is trivial if you’re using BPF override return, same for in kernel modules.

12:17 - Now let’s see your demo. Let’s demo the obfuscation of our BPF programs and maps.

12:25 - So we still have the rootkit started and we’ve released the maps and the programs, thanks to the eBPF tool kernel online.

12:32 - We can’t see anything related to the rootkit.

12:35 - Now if we (indistinct) kprobes and maps, and we list, again, the programs, we can see the program related to the binary, but still nothing related to the rootkit.

12:46 - Then we can even try checking the kprobes and still nothing related to the rootkit.

12:55 - The ID is to hook into the eBPF syscall itself, two things are required here.

13:01 - We want to avoid the real space to utilize the two IDs of our own duties using the BPF_prog_get_next_id command.

13:08 - And we need to prevent the user space from getting a file descriptor to the program or map from the ID using BPF_prog_get_fd_by_id basically to prevent ID guessing.

13:20 - When these operations are handled by the rootkit, the user space is not able to view or modify our programs and match.

13:29 - BPF provider user is a crucial part of our rootkit but it comes with a pitfall.

13:35 - While an eBPF program that makes use of it is loaded, the journal printer pretty scary, but legitimate message in the kernel ring buffer.

13:45 - So let’s see how the rootkit handle this. First, let’s check what is the current statues of the kernel ring buffer using the kernel dnesg.

13:58 - So nothing special here. And we can even start a cache on the device used by the ring buffer.

14:07 - Then we start the rootkit. So we can see that the cache terminated with a pretty legit looking message.

14:17 - We can recheck the kernel ringer buffer, thanks to the message kernel online, and we can only see legit-looking messages, but I need more.

14:31 - And then we can stop the rootkit and we can run again, the message kernel online in order to see what was overridden by the rootkit.

14:42 - So let’s see how it works. In this diagram, Ronald is waiting for a new message and is blocked on a red syscall.

14:51 - Two BPF programs are loaded secondly by the rootkit.

14:54 - The first one, which doesn’t choose eBPF provide user work on our red syscall and we make it written zero and we’ll write legit-looking message.

15:03 - This will guarantee that the warning message is related to the BPF ProLite user won’t be read.

15:09 - Then the second program using BPF ProLite user is loaded.

15:13 - At this point, the red syscall can be unblocked and we can override the content of the warning messages with legit-looking messages.

15:23 - Another important task for the rootkit is to set up a persistent access across reboots, for instance.

15:29 - The rootkit can copy itself to a dedicated place in either it’s binary file with the same mechanism that we already saw.

15:37 - For the persistent access to the system, we can use a generic method close to what we described in the obfuscation path.

15:42 - We can replace the content of some critical files read by the root demands, such as crotab or even sshd.

15:51 - Let’s see an example targeting sshd and using the reader override approach.

15:57 - So the approach here is to open an sshd to the authorized key files.

16:03 - Only sshd should be impacted, meaning the file will remain the same for the user point of view.

16:09 - And we want to have it available to the command-and-control.

16:13 - So let’s see this in action. So let’s take the authorized keys content first.

16:24 - So we can see that only one key is present.

16:27 - So let’s start the connection and it’s in that password is required.

16:33 - So now we’re going to start the rootkit and we are going to specify that we want to inject an SSH key to the authorized keys but only for SSH.

16:49 - So we can try your connection again, and it seems to be successful.

16:58 - And now we can check what is the content of the authorized key from the user point of view and nothing changed apparently.

17:07 - Persistent access to an application database can also be set up using another type of eBPF program.

17:13 - Uprobes eBPF programs attached to user space function.

17:17 - In addition to being safer and easier to use than ptrace, they offer a valuable advantage.

17:24 - The kernel will automatically set up for us the hooks on every instance of the program.

17:29 - Let’s see a uprobe demonstration using (indistinct) So first, let’s try to connect to (indistinct) using the word bonsoir as password.

17:45 - This one seems to be the good one, then trying hello, and this one is rejected.

17:51 - Now we start the rootkit and we’ll get the opposite treasured.

17:57 - Now the value password is hello. So the idea here is to hook on the md5_cript_verify function of a progressive creator that check whether the user provided the right MD5 for its role, passwords, and the challenge sent by the server.

18:15 - Overwriting the expected hash contained in shadow_pass with a known value makes the comparison succeeded and give persistent access to the database to the attacher.

18:26 - Now I will hand over to Guillaume that will show you the command-and-control capabilities of the rootkit.

18:33 - - Thank you, Sylvain. Let’s talk about the command-and-control feature of the rootkits.

18:37 - So what exactly do we want to do? We want to be able to send commands to the rootkits to exfiltrate data and to get remote access to the infected hosts.

18:45 - Unfortunately, there are a few eBPF-related challenges that we need to face in order to implement those features.

18:52 - First, you can’t initiate a connection with eBPF.

18:55 - Second, you can’t open a port. However, eBPF can hijack an existing connection.

19:02 - So in order to show up this feature, we have set up a very simple infrastructure on AWS.

19:07 - A web app was installed on any C2 instance and we used a Classic Load Balancer to redirect HTTPS traffic to our instance over HTTP.

19:16 - In other words, the TLS termination is done at the Load Balancer level and its eBPF requests are sent to our instance and encrypted.

19:26 - So our goal is to implement CNC by hijacking the network traffic to our web app.

19:32 - First, we need to figure out which eBPF program types we’re going to use in order to implement this feature.

19:38 - Although eBPF provides a lot of options to choose from, we decided to go with two eBPF program types, XDP programs and TC classifier programs.

19:49 - So both of those programs are usually used to do deep packet inspection use cases.

19:55 - And while XDP only works for ingress, TC works on both ingress and egress traffic.

20:02 - Another difference between the two program types is that XDP programs can be offered to the network interface controller, which essentially means that your program will be run before the packet enters any subsystem into the network stack.

20:18 - On the other hand, TC programs have to be attached to a network interface, but much later in the network stack, which means that they are triggered later in the kernel.

20:32 - With both programs, you can drop, allow, and modify your packets.

20:35 - And with an XDP programming, you can also retransmit a packet.

20:39 - This option is actually super interesting for us because it means that you can essentially receive and answer to a packet even before it reaches the network stack, which in other words means that you can do this even before it reaches any kind of network firewall or monitoring on the hosts.

21:00 - Skipping the network side also explains why XDP programs are mainly used for DDoS mitigation and TC programs are usually used to monitor and secure network access at the pod or container level.

21:15 - So what you need to remember about this slide is that first, XDP programs can be used to hide natural traffic from the kernel entirely and TC programs can be used to exfiltrate data on its way out.

21:31 - First, let’s see how we used XDP programs to receive commands with the rootkits.

21:37 - So we implemented a client for the rootkit and this client communicates with the rootkit by sending simple https requests with custom routes and custom user agent.

21:49 - So after going through the Load Balancer, the request eventually reaches the host and triggers our XDP programs.

21:57 - Then our program pass the requests, the http routes and understand that this request is not meant for the web app, but it’s meant for us.

22:06 - So after sending the other, sorry, after reading the user agents, the rootkit executes the requests that come in and moves on to the final step.

22:14 - So this final step is probably the most important one.

22:18 - It overrides the entire request with a simple health check request, and we do this for two different reasons.

22:26 - First, we don’t want the manager’s request to reach the web app or any kind of user space monitoring tool that might be bringing and that might detect the unusual traffic.

22:37 - And second, we want the client to receive an answer in order to know if the request was successful.

22:44 - So, as I said before, we could also have drop the packet entirely, but since we’re using TCP, the Load Balancer would have retransmitted the packet over and over again until the request times out.

22:55 - And this would have generated noise and increase our chances of getting discovered.

23:00 - That said, if you were working with a UDP server, this would be a totally valid strategy.

23:09 - So let’s have a look at how we can send Postgres kernels remotely.

23:15 - All right, so on the left of the screen, you can see two different shells.

23:18 - Those shells are connected to the remote infected hosts on AWS and on the right, this is my local shell and this is the attacker machine.

23:28 - Okay, so let’s start with trying to log into the Postgres database using the normal password.

23:34 - And again, the rootkit is not running yet. So as you can see, the bonsoir passwords works fine.

23:39 - And then they start the rootkit and restart to log in again and as expected and as you’ve seen before during Sylvain demo, it doesn’t work.

23:49 - So you have to change into, hello, and this time it will work there you go.

23:52 - Okay, so we’re gonna try to do the same thing, but instead of hard coding the new password with the rootkit, we’re gonna define remotely through C&C what the new password should be.

24:03 - So, as you can see, we have a custom client that will make a request to its CPS, it needs to be a CPS request to defcon. demo. dog and then we will provide both the role and the secret to override the normal secret with.

24:19 - So the request that will go through is a very simple one with the custom routes and the user agent will contain the new password that will be used at runtime.

24:30 - So as expected, we get the 200 okay from the health check, which essentially means that we know that the new password now is defcon and not hello anymore.

24:41 - So as you can see, hello that doesn’t work, but if I’m change it to defcon, here you go.

24:48 - This time it does work. Okay, so this is how we send the command to the rootkits.

24:55 - Now let’s see how we can exfiltrate data. So to exfiltrate data, the client has to send an initial request to specify what kind of data and what kind of resource we want to exfiltrate.

25:08 - So the XDP part of this process is basically the same as before, but this time the XDP programs stores the network flow that made the request along with the requested resource, sorry, in an eBPF map.

25:23 - And the reason why we do so is because when the web app answers the health check, we want to be able to detect the packets that are meant to be sent back to the clients.

25:34 - So when the XDP answer reaches the TC egress classifier, our eBPF program looks at the network flow and overrides the answer with the requested data.

25:46 - Now, the question is, what kind of data can you exfiltrate with the rootkit? And the answer is, well, pretty much anything that is accessible to eBPF.

25:58 - And the reason for that is, as I said before, multiple program types can share data through eBPF maps, regardless of what those programs are supposed to do.

26:08 - So basically you can exfiltrate things like file content, environment variables, database dumps, in-memory data if you start looking at the stacks of the programs.

26:18 - Anyway, you can pretty much exfiltrate whatever you want.

26:22 - So let’s have a look at a simple demo that we can exfiltrate Postgres credentials along with the file content of etcpasswrd demo.

26:35 - All right, so again, the two shells on the left are the ones connected to the infected hosts on AWS and on the right, this is my local shell.

26:45 - So the first request that I make here is to do Progres list, which basically means please list all the credentials that you have detected so far since the rootkit has started.

26:59 - And as you can see, the answer was, sort of the health check answer was overridden with the content of a map that we used to store the passwords that we’ve collected at runtime.

27:10 - And again, remember that with Postgres, you don’t need the clear password to login.

27:14 - You just need the hash password that is stored in the database.

27:19 - Here you go. So now we’re gonna try to do the same thing to dump the contents of etc password.

27:25 - So to do so, this is a two-step process. The first thing you want to do is tell the rootkits to start looking for this specific file.

27:34 - And as soon as a user space process tries to open the file and read the content of the file, our rootkit will actually copy the data as it is sent to the user space application and say that into an eBPF map so that it can be retrieved later.

27:52 - So this first request will tell the rootkits to start looking for etc password.

27:58 - And now let’s go back to the host and trigger some kind of pseudo operation so that our user space process tries to open the file.

28:09 - Here you go. And then this time, instead of saying, add, we’re gonna say, get, and this will dump the content of the etc password file.

28:21 - Here you go. All right, so the cool thing about this technique is that it applies to any un-encrypted network protocol.

28:30 - So for example, we also implemented it for DNS, which means that you can actually use it to do DNS spoofing.

28:36 - So the only difference between the normal way of doing this and the DNS spoofing is that instead of using a TC program to override the answer of the request, you will actually switch to see an XDP program because DNS requests are made from the host instead of received by the host.

28:57 - All right, so let’s move on to our network discovery feature.

29:02 - So I know everybody knows what it is, but I have to say it anyway, network discovery is the ability to discover machines and services on the network so that you know where you want to go next in the infrastructure.

29:14 - And also discovering services is a super important step when you are trying to pivot between hosts, because it will tell you what kind of attacks you might want to try.

29:24 - So the rootkit has two different networks discovery features.

29:28 - One of them is passive, the other one is active.

29:30 - And you can control both of them through command-and-control.

29:35 - So I’m gonna get into more details later, but basically the only difference between the two is the kind of networks scanning you’re looking for and also the level of traffic that you are willing to generate on the network.

29:48 - So first, let’s have a look at the passive option.

29:52 - So the passive option is simply a basic network monitoring tool.

29:56 - So it will do pretty much the same thing as any other eBPF-based network monitoring tool, which means that it will listen for any ingress or egress traffic and then generate a graph from all the collected network flows.

30:09 - It will also show you the amount of data that was sent per network flow.

30:15 - And to implement this feature, we used our TC and XDP programs.

30:20 - So the TC programs were used to monitor the egress traffic and the XDP programs were used to monitor the ingress traffic.

30:29 - So for this version of the rootkits, we are limited to IPv4 and TCP UDP packets, that said support for IPv6 and all the protocols could have been headed easily.

30:41 - So the reason why the passive option is pretty cool is that it will not generate any traffic on the network.

30:48 - In other words, it is basically impossible to detect that someone is tapping into your network and more specifically at the network that creatures this specific infected host.

31:00 - However, this doesn’t work for services that do not communicate with the infected host.

31:06 - And so in other words, the graph will definitely not be complete.

31:10 - And that’s also why we implemented the active method.

31:17 - So the active method is a simple ARP scanner along with a SYN scanner.

31:22 - So we implement it using only our XDP programs, which means that the entire process is done without involving the kernel stack.

31:34 - And although this will be a slower process, you can use this method to discover hosts and services that are reachable by the infected host, but that are not communicating usually with the infected host.

31:48 - And again, the rootkit client will generate a nice network graph for you once the scan is complete.

31:56 - So on a technical level, this feature of the rootkit is actually quite interesting because, as I said before, eBPF cannot create a connection from scratch.

32:04 - So in other words, we had to figure out a way to generate hundreds of SYN requests while dealing with this limitation of eBPF.

32:13 - So let’s see how we solve this problem. So in order to send a SYN request, you first need to know the MAC address of the IP that you want to scan.

32:26 - To do so, we use the same trick that we’ve been using so far, which is to override the requests from the rootkit clients.

32:33 - So when our XDP program receives a scan request for a specific IP and a specific port range, it will override the entire request with an ARP request for the target IP.

32:47 - And then instead of returning XDP pass, which is what we’ve done so far, and also which would send the packets to the network stack, our eBPF program returns XDPTX.

32:59 - So what XDPTX does is send the packet out to the network interface controller it can inform.

33:06 - In other words, our HTTP packet was transformed into an ARP request and broadcasted back to the entire local network.

33:20 - So eventually the target IP will answer the ARP request and will be able to store the MAC address of this specific IP in an eBPF map.

33:32 - However, during this entire process, the TCP packet that was used to send the HTTP request was never acknowledged by the kernel.

33:41 - And that is simply because it never made it’s way to the kernel in the first place, which means that the Load Balancer or the client itself will eventually try to retransmit the packet.

33:53 - And when this packet is retransmitted and when it eventually reaches our XDP program, we will do the exact same thing.

34:03 - But this time, instead of, because we know the MAC address, instead of overriding the request with an ARP request, we’re gonna override the request with a SYN request.

34:13 - And more specifically a SYN request with the first port of the provided port range, the target IP and the MAC address of the target IP.

34:23 - And assuming that the remote IP or the remote host doesn’t have any kind of prediction against SYN request, sorry, SYN scanning, it will answer either resets or SYN plus acc to this first request.

34:38 - So reset would mean that the port is open and SYN PLUS acc would indicate that there might be a set of sprinting running on the host.

34:47 - And this is where basically the network loop happens and the reason why we were able to generate hundreds of packets while dealing with the limitation of eBPF, that is the inability to create packets.

35:05 - So whenever we get an answer from a SYN request, we override the received packet with another SYN request on the next port.

35:13 - And we also switch the IPs, switch to the MAC addresses, and send it back again to the target IP.

35:18 - And we do so in a loop until we go through the entire port range.

35:24 - So eventually the clients will try one last time to retransmit the initial HTTP requests.

35:31 - Because once again, during the network loop, we never answered the second retransmit.

35:38 - So eventually when this third retransmit reaches our XDP program, we will override the request with the usual health check requests so that the 200 okay answer will make its way back to the client after the request was handled by web app and user space.

36:00 - All right, so let’s see it in action. So on the right of the screen, this is a shell to the infected host on AWS and at the bottom here, it’s another one.

36:14 - And at the top, this is my local shell on my machine.

36:20 - So the first thing you want to do is to start the rootkit.

36:24 - Then second is to start dumping the logs of the rootkit.

36:28 - So an eBPF can actually generate logs using a trace pipe.

36:32 - Obviously you would not want to do this in a real use case for a rootkit, but this is a great way of visualizing the scan as it goes through.

36:43 - Yeah, so that’s why I’m doing this and that’s why you will see what the rootkit does at runtime.

36:48 - And then let’s make the scan request. So what I’m saying here is please scan the IP 10. 0. 2. 3 from port 7990 and for the next 20 ports after this one.

37:04 - So the first thing you can see is the request is immediately changed into an ARP request and we already got the answer for this ARP request.

37:13 - So next step, when we get a retransmit, we will change this into a SYN request.

37:19 - Here you go. So the SYN request went through and then you can see the loop that happens.

37:24 - And we increased port one by one until we reached the final port requested by the port branch.

37:33 - And then now we are waiting for the third retransmit and this retransmit will be the one that we override with the health check request, which means that we will eventually get here, go the 200 okay, in other words, the answer from the user space web app.

37:50 - All right, so now what you want to do is retrieve the output of the scan and exfiltrate all the network flows that were detected at runtime, and here you go.

38:00 - So you would say network_discovery_get and eventually…

38:04 - So it actually requires a lot of different requests because there’s a lot of data to exfiltrate, but eventually you will get the entire list of network flows that were captured by the rootkit.

38:16 - Here we go. So you have all the different individual flows and then more importantly, you will have a graph generated for you.

38:27 - So this one is the passive, sorry, active graph.

38:31 - So as you can see, range there, you can see the ARP requests and replies between those different hosts.

38:39 - And then in gray, those are the SYN requests and the reset answers.

38:44 - And in red is the only SYN plus acc, and so from the remote host.

38:50 - All right, and then you have also the passive graph, which is the one that we saw before.

39:03 - Okay, so now let’s move on to our RASP bypass.

39:06 - So RASP stands for runtime application self-protection.

39:11 - So in a few words, a RASP is a new generation of security tools that uses runtime instrumentation to detect and block application level attacks.

39:20 - And more importantly, it leverages its insight into the application in order to make more intelligent decisions.

39:28 - So simply put, it is some kind of advanced input monitoring tool that can detect malicious parameters and can understand if a malicious input will successfully exploit a weakness from one of your apps.

39:43 - So the textbook example of a RASP is usually a SQL injection.

39:47 - So the RASP route implement multiple functions, instruments, sorry, multiple functions in the libraries that you use such as for example, the HTTP server library or the SQL library.

40:00 - And it will check at runtime that the user control parameters in your queries are properly sanitized.

40:07 - If not, the RASP will stop the query before it reaches the database and redirect the client to an error page or some kind of error message.

40:16 - In other words, a RASP relies on the assumption that the application runtime has not been compromised, which is exactly what we can do with eBPF.

40:26 - So just a little disclaimer before I move forward.

40:31 - I want to stress the fact that we are playing outside of the boundaries of what a RASP can protect you from.

40:38 - And more importantly, this bypass does not apply to one specific RASP, but to all of them because this is one of the core principles of how a RASP works.

40:50 - So let’s have a look at how RASP products a good web app from a SQL injection.

40:58 - So let’s say that you have a web app with a simple products page and get parameter to specify the category of the products that you want to see.

41:09 - Chances are your web app uses the default go database SQL interface.

41:15 - So this is a generic interface that you can use to query your database without having to worry about the underlying driver and the database type that you’re using.

41:24 - More importantly for us, since it is such a generic interface, this is usually where the RASP tools instrument your code because simply it’s much easier to hook at this layer rather than having to hook onto all the underlying drivers.

41:42 - So when your request is handled by the web server, the query will be formatted with the provided category parameter and eventually the web app will call the query context function of this SQL interface.

41:57 - This is when the RASP checks the query and makes sure that everything is normal.

42:03 - And if it is, the execution will resume its normal flow and the underlying driver will be called.

42:10 - So in our example, we use SQLite, so the SQLite driver is called.

42:16 - Eventually the query makes its way to the database and the answer is sent back to the client.

42:21 - However, if the RASP detects that something is wrong or detects some kind of SQL injection, it will block the query and redirect the client to an error page.

42:33 - All right, so now let’s see what we did to bypass this protection.

42:37 - Well, the answer is actually pretty simple.

42:39 - We added a uprobe on both the database SQL interface and the SQLite driver interface.

42:47 - What this allows us to do is call one of our eBPF programs right before the RASP checks the SQL query and trigger another one right before the SQL query is executed by the database itself.

43:04 - And thanks to the BPF ProLite user helper, we can override the input parameters of the hooked functions so that the RASP sees a benign query and the database executes our SQL injection.

43:20 - And the cool thing about this is that we can even do it conditionally, which means that we can bypass the RASP only if one specific secret password was added to the beginning of the query.

43:36 - Perfect, so let’s move on to the demo. So as you can see, we have a very simple web app.

43:42 - So it’s a shoes retailer. So you have a lot of different products and you can filter by category.

43:49 - So let’s try to do a SQL injection using the get parameter.

43:55 - So the injection was simply be select star from user and because the RASP is not running right now, the SQL injection should work.

44:04 - Here you go. And if you scroll down this time, you will see the users along with the passwords, perfect.

44:12 - So now let’s restart the web app with the RASP, which is what I’ve just done and try this again.

44:20 - So let’s go to the shop and then override the category parameter, perfect.

44:30 - And this time the RASP blocked the request because it detected that someone tried to do a SQL injection and the SQL injection would actually have succeeded.

44:42 - Great, so now let’s start the rootkits by providing the path to the web app and then try to refresh the page.

44:51 - So again, this should also be blocked by the RASP because we haven’t provided the secret password for the bypass to work.

44:59 - And the secret password is of course, defcon.

45:02 - And when we say defcon, the entire process that I described before will be triggered.

45:08 - And as you can see, the RASP did not detect it.

45:11 - So that’s all for our RASP bypass. I hope you had fun.

45:15 - Just before I hand it over to Sylvain for the detection and mitigation strategies, I wanted to say that, unfortunately, we won’t have time to talk about the container breakouts that are (indistinct) into the rootkit.

45:26 - However, they have been presented during our Black Hat call this year.

45:30 - So if you are interested, feel free to check it out.

45:33 - That being said, Sylvain, take it away. - So let’s talk about detection and mitigation.

45:41 - How could we detect and protect ourselves from this type of rootkit? We could do this at different level.

45:47 - First, if a vendor provided you eBPF programs, you should go through an audit and an assessment phase of their programs.

45:54 - Some changes that the code has to be GPL. It probably uses some internal candid symbols, so you can ask for it.

46:01 - What should we be looking for? The program types that are used, but also the eBPF helpers used.

46:07 - So communication. So maps between programs may indicate the control risk in the case of that the vendor program is compromised.

46:15 - We developed a tool to assist in this auditing phase by inspecting the health files containing eBPF programs, it is able to list the use entities, programs, and maps, and compare to graph of the interactions between them.

46:29 - The tool was run on our rootkit with the following result.

46:32 - We can identify on the graph that the XDP program has stored information into maps that are also used by some kprobes, which correspond to the command-and-control capabilities of the rootkits.

46:47 - It is also possible to mitigate at runtime the loading of such programs by monitoring calls to the eBPF syscall and logging (indistinct) It would even be possible to protect the eBPF syscall itself by either (indistinct) syscall to it to need some trusted processes.

47:05 - As the programs inspected before then and rejected if they contain suspicious patterns or make use of some dangerous helpers.

47:13 - We could also compute and validate the signature of the programs before (indistinct) exist to the kernel itself.

47:24 - Using TLS everywhere for network traffic also helps mitigating the risk of a rogue eBPF program that intercept network data.

47:33 - Now if we are not able to block the loading of such a rootkit, how difficult would it be to detect its presence? Even if it’s possible, though very challenging to write anonymous perfect rootkit, we should concentrate on the action the rootkit will have to block and lie about the result of such actions.

47:51 - For instance, our rootkit (indistinct) we are loading kernel modules because such a module would have the ability to lead the eBPF programs and the active kprobes.

48:01 - Now let’s imagine that we insert a module that executes a specific action only known to us.

48:07 - The blocking of the module by the rootkit will then be easy to detect.

48:12 - Monitoring the network traffic at the infrastructure level could that detecting hijacked connection or strange back into our transmission.

48:20 - Our rootkit being far from complete and far from perfect, it should be relatively easy to detect it.

48:27 - That being said, we hope it will bring to light the potential and the risk of such an eBFP base rootkit while presenting some interesting technique.

48:39 - The code of both the rootkit and the monitor is available at the servers, please have a look.

48:45 - Thanks for your attention and have a great conference. .