DEF CON 29 - Shir Tamari, Ami Luttwak - New class of DNS Vulns Affecting DNS-as-Service Platforms

00:08 - My name is Shir, and I lead the research team at Wiz, the cloud security company.

00:13 - With me in the room is Ami Luttwak- - Hi. - The CTO of Wiz.

00:20 - - Thank you, Shir, and it’s really great to be here.

00:22 - And so, let’s start with a bit of background about us.

00:25 - So we are the Wiz research team. Wiz is a cloud security company.

00:30 - The team is composed of experienced security researchers, many of them with background from Microsoft Cloud Security group, and our goal is to do groundbreaking cloud research to find vulnerabilities, misconfigurations, risks in cloud environments that customers are not aware of, and would impact anyone using the cloud.

00:50 - So we started looking into DNS-as-a-service.

00:53 - Why DNS-as-a-service? So first of all, DNS, as we all know, is the lifeblood of the internet.

00:59 - It’s probably one of the most important services that we have.

01:02 - Now, DNS-as-a-service has huge impact. If you think about it, it holds your domain, it holds your internet routing.

01:09 - Now, what’s cool about DNS-as-a-service, unlike usual cloud services, is that also it controls not only cloud activities, right, when you use DNS-as-a-service, it also impacts your on-premise activities.

01:22 - So potentially, DNS-as-a-service has huge impact on an organization.

01:27 - Now, on top of that, what is really great for researchers is the DNS protocol is one of the oldest protocols that we use.

01:35 - It’s more than 20 years old. It’s incredibly complex, and there’s so many different implementations of it, both from the DNS providers, but also think about millions of DNS clients that we have out there, each of them implemented in different way, creating a very, very interesting attack vector for researchers to explore.

01:54 - So we started from looking into Route 53, which is the DNS service provided by AWS, and it’s highly popular across AWS customers.

02:05 - - So Route 53 is built on thousands of DNS servers that host DNS zones for all AWS customers.

02:14 - We mapped around 2,000 servers in the Route 53 platform.

02:18 - Whenever the customer is hosting a domain name on the platform, they get four shared name servers to manage the desired domain.

02:26 - On that right side of the slide, you can see a simulation of one out of the four DNS servers Amazon provides each domain.

02:35 - The server stores the DNS zones for wiz. io, for example, and several other AWS customers.

02:43 - If the server is queried for one of the domains under management, it will return the appropriate records for that domain.

02:52 - While studying Route 53, we discovered that anyone can register any domain they want on the platform.

03:02 - There is no restriction on whether the domain is already hosted on the platform, and there is no ownership verification.

03:09 - The only limitation is that if the domain already exists in one of the name servers, it will not be possible to register it again on the same server.

03:18 - And it makes a lot of sense, and it’s indeed not possible.

03:24 - Trust us, we tried. Basically, anyone can register any domain on any of the name servers, as long as it does not already exist there.

03:34 - But is there anything dangerous here? For me, as a security researcher, it felt like we had too much control here.

03:43 - But almost any DNS-as-a-service provider works this way.

03:47 - So is it really a security problem? All right, this is an example that if you try to register Wiz again on the server, it would be impossible, because it’s already there.

04:00 - - So we started with a very, very simple and interesting research question.

04:05 - If we can register any domain, right, what domain can we possibly register that will give us interesting access to data, right? So we wanted to register a domain that is not already present or the name server, and that for some reason DNS clients will actually query for that.

04:21 - So we started into a quest to think what domains can we register that no one thought about, and we could actually somehow break the DNS? - So we thought about it, and came up with an idea.

04:36 - What would happen if we register one of the official AWS name servers on the platform? What would happen if we registered one of the Route 53 name servers under the same server? So we chose an arbitrary name server.

04:51 - In this case, it’s the ns-1611, as you see on the slide, and we tried to register it on the platform enough times that at least once it would fall under the management of the same name server.

05:04 - Let’s do an illustration of that. So as you can see in the slide, we have an illustration of the ns-1611 name server, which is an official Route 53 name server, and you can see it already contains DNS zones for several of a lot of these three customers.

05:21 - What do you think would happen if you managed to register the name server domain name ns-16-11 under its own management? I totally don’t know what would happen, but we must check it out. - Definitely.

05:35 - (Ami laughs) - So we tried it, and it worked As you can see on the slide, our new domain is now managed within a name server with the same domain name.

05:49 - We were really excited. We didn’t know if it will have any impact, but we had a really good feeling about this one.

05:57 - So to test the effect of what we did, we decided to specify an IP record, IP record pointing to our server.

06:08 - So now, if a DNS client will connect to the ns-1611 name server and ask it about itself it is our IP address that will be returned.

06:20 - At this point, we were really curious to know if anyone is asking Amazon’s server about itself, and if anyone is trying to connect with us, because of the manipulation we did.

06:31 - So we ran tcpdump on our server, and hoped to see something interesting.

06:38 - Surprisingly, we started receiving thousands of requests from thousands of different IPs.

06:44 - We realized we were onto something. The next step in the research was to analyze all these DNS queries, and figure out why these queries are being sent to us.

06:57 - - So wait, why are we getting any traffic? And no one was supposed to ask the name server for their own domain.

07:04 - The name server actual domain is registered on other name servers, right? So why are we getting any traffic? And what’s more weird was that it wasn’t actual regular traffic.

07:15 - It wasn’t even DNS traffic. It was dynamic DNS traffic, which is a very specific protocol that you wouldn’t even expect to see in this type of internet traffic.

07:26 - Now, we were getting a lot of data. I’m talking about IP addresses, computer names, domain names.

07:32 - So we started investigating. We started to look into the data.

07:37 - So basically, we saw, we were getting millions of extra requests from endpoints, and when we started looking into it, we saw it’s a lot of data.

07:47 - We were seeing internal IPs, external IPs, names of computers.

07:52 - We very quickly understood these are names of endpoints within many, many different organizations across the globe.

08:00 - And the scale was totally unbelievable. I’d say within a few hours of sniffing the traffic, we saw more than a million unique endpoints, and they would belong to, based on the initial analysis we did, we saw more than 50,000 organization, 15,000 organizations, all of them using AWS as a DNS service.

08:22 - Now, we said, “Okay, let’s try to figure out “what organizations are we seeing here,” right? So it was quickly we understood that we are stepping into an unbelievable tap of worldwide organization and traffic.

08:37 - We saw Fortune 500 companies. We saw more than 130 government agencies right in the traffic.

08:45 - So we knew we were probably onto something big, but the problem is that we had no clue what we were seeing and why.

08:54 - So what do we know so far? We registered a name server domain on the name server, right? We have no idea why, but for some reason, millions of endpoints started sending tons of dynamic queries to us.

09:07 - But again, why? Why are we seeing dynamic DNS? Before we are able to answer that simple, mysterious question, we have no way to actually understand what we are seeing.

09:19 - So we decided we are going to step into the world of dynamic DNS.

09:25 - - So what is exactly dynamic DNS? Dynamic DNS is an extension to the DNS protocol, which is specified in RFC 2136.

09:36 - It allows clients to dynamically update DNS records of a target DNS server, and it’s commonly used to help devices find each other in internal Windows networks.

09:48 - Let’s see how it works. So when my Windows computer joined the company’s network, and receive a new IP address, as you see on the slide, it updates the local DNS server, which is called master server on its new IP address.

10:05 - Now, when Ami is trying to connect to my computer, he can query the local DNS server about SHIR-PC, and the master server will answer it with my current IP address.

10:18 - So far, sounds like a great feature. At the moment, it is still not clear to us why endpoints sends dynamic DNS updates to our server.

10:27 - These are requests that should never leave the internal network.

10:31 - Could it be that the endpoints think of our server as their master server? How does it even know to find the master server? So it turns out Microsoft has its own algorithm for finding the master server, and it does not work exactly as specified in the RFC.

10:54 - Just before we go into the logic of the algorithm, let’s do a brief refresh on DNS records for those who have not touched DNS recently.

11:04 - So in order to fully understand the algorithm, it is important to remember only three types of records, and we’ll make it very quick.

11:10 - The first will be the A record. A records specify the IP address of which the domain points to.

11:18 - The second would be the NS record, which specify the name server of the domain name.

11:24 - The third is the SOA record, which is short for Start of Authority.

11:29 - The SOA records contain administrative information about the domain, and its first parameter is the master server.

11:37 - This is the server in which clients would attempt to update doing dynamic DNS updates.

11:42 - Usually, this server will be on one of the domain’s name servers, and in Route 53, the default value of this field will be the first Amazon’s name server from the name servers list.

11:56 - And now that we have all refreshed our memory, let’s get into the algorithm.

12:01 - Now, this is the primary name server. So when a Windows endpoint want to update the internal master server for its new IP address, it first needs to find it.

12:12 - The endpoint will send the SOA query for the internal DNS resolver on its own fully qualified domain name.

12:19 - The internal DNS resolver knows that the wiz. io DNS zone is managed internally, so when queried it will return the internal master server within the SOA response.

12:35 - Now, when the endpoints find the master server, it will update it as we saw in the previous illustration, and the update succeed.

12:47 - But what happens if the corporate DNS resolver is not set correctly, and does not contain a DNS zone for the local domain? Or what would happen when the computer leaves the organization, and starts working with external DNS resolvers provided by home routers? In that cases, when the computer query for the corporate domain, the DNS resolver will treat it as an external domain and will return the master server of the external domain, just as it will do with any other domain.

13:20 - This is where the problem starts. So imagine an employee of Wiz decided to work from home, like most of us lately, and connected to their home Wi-Fi.

13:31 - The computer got an internal IP address from the home router, and now trying to find the local master server to update it.

13:40 - Because the external DNS resolver are not familiar with wiz. io, it is going to resolve it just like any other domain.

13:50 - Because the domain is managed on Route 53, it will return the wiz. io master server as specified in Route 53.

13:59 - The endpoint will then try to update the master server, which is an Amazon shared name server that manages thousands of customers.

14:11 - Obviously, Amazon servers does not support dynamic DNS, so the update request would fail.

14:18 - The thing is that Microsoft’s algorithm does not give up here.

14:23 - The algorithm believes it still has a chance to find the master server.

14:28 - So the next step would be to check if the wiz. io’s name servers have records for the master server.

14:37 - So the DNS client connects directly to the IP address of the name server, and ask them what is the IP address of the master server.

14:46 - In our case, it is the same domain name as the name server.

14:51 - And here happens the interesting part. In fact, Windows DNS clients queries the name server for itself.

14:58 - And if you remember, we have registered this DNS zone, and here we can return any record we want.

15:05 - So we returned our IP address to the DNS client, and now, the computer will update our server with DNS updates, which is the malicious actor server, and this is very crazy.

15:22 - - So now we reached the point that we started understanding what’s happening, right? Because what we know, we know that Windows endpoints, they use a custom algorithm to find the master DNS.

15:34 - The algorithm queries the name server for its own address.

15:38 - When you are in external and using Route 53, what happens is that this means that you’re actually querying the name server for its own domain, and that explains what happened.

15:47 - That’s why we are able to register our malicious DNS server, and we were receiving this dynamic DNS traffic from millions of endpoints, because all of the organizations using Route 53, when these devices are actually outside of the domain and outside of the company, they will actually use this algorithm, and we would be able to get traffic from those devices.

16:09 - So we understood what’s going on. The next step was to figure out how much data are we actually getting, and what can we find from it? So when we started looking into the data, we quickly understood that we are actually building here what we called a nation state intelligence capability, because think about it.

16:29 - We saw IPs for millions of endpoints from more than 15,000 organizations, but it’s not just DNS requests.

16:38 - We are seeing external IPs, internal IPs, computer names.

16:43 - We are starting to map all of those organizations.

16:46 - So let’s see what we can do with this scale of intelligence capabilities.

16:51 - So we started from what we call IP-based intelligence.

16:54 - Imagine that you can map companies and a portion of companies around the world and map their global sites, map where their employees are at, right? So we looked at one company, for example, and view, and just see how amazing this is.

17:08 - So this is one of the largest services companies in the world, and we got around 40K endpoints actively reporting from this company.

17:16 - Now, we are seeing a mapping of all of their global sites, all of their actual offices, the employee locations, right? So from just the external IPs, we can map, create a map of offices and home locations of employees across all of those companies.

17:35 - It’s pretty amazing. And we can go even deeper, right? This is an exhibit of a specific office, where we detected 600 endpoints of that company, right? So imagine an intelligence capability that maps for you in one single tap into the DNS without any trace actual structures of organizations across the world and all of their different offices and locations.

18:02 - So, but it doesn’t stop there. Then we started thinking, “Okay, what interesting data can we pull from this “if we’re an intelligence agency now?” So we started looking at companies that are in violation of the Office of Foreign Assets Control.

18:17 - And we saw interesting things. For example, this mining company, it’s an international mining company, and interestingly, we found six employees working from the Ivory Coast, which is definitely a place that is not allowed in that regulation at all, right? And we saw so many interesting examples like that.

18:37 - So we found a subsidiary of a large credit union with a branch in Iran.

18:44 - Again, 13 endpoints working from Iran based on our newly revised intelligence capability, right? So it’s not only mapping all organizations.

18:56 - We can now start finding violations. We can ask questions across so many different agencies.

19:02 - You remember, we are actually seeing also government agencies.

19:05 - I wonder where are all of their offices, right? Now, it doesn’t stop there, because if you remember, external IPs is just a small portion of the data, right? We also have internal IPs.

19:17 - So what can you do with the internal IPs? If I have internal IPs from different endpoints from the company, I can start building the map of the network, the internal logical network of the company.

19:29 - So for example, these are the segments. This is the employee segment.

19:33 - This is the ICT. Here’s the operational network.

19:35 - Remember that we also see names of devices, so we can start really understanding all of the segments, so building an intelligence map of the organization externally and internally across thousands of organizations.

19:46 - Now, we also had computer names and the computer names actually hold information about the endpoint, right? And many times, you get employee names.

19:55 - You understand the actual role of the machine.

19:58 - You can see the specific, the build the machines is using, right? So we are actually getting quite a lot of information about those companies based on the IPs, the computer names.

20:12 - Here, we see this is finance, and we see all of those machines are part of a specific segment.

20:17 - Okay, perfect, I am starting to build my internal map of this company.

20:21 - That’s perfect. Now, just so we understand the scope here, we looked at a specific DNS provider.

20:35 - Then we asked, “Wait, is this only this DNS provider?” So we started looking at others, and we soon found there is many others also susceptible to the same vulnerabilities, right? This is not just a Route 53 vulnerability.

20:48 - This seems to be something that is shared across most of the DNS service providers, and if you think about it, we don’t have to stop at the cloud providers.

20:56 - You have shared hosting. You have domain registrars.

20:59 - There is so many different service providers using, again, I think the shared a concept here is that they provide DNS services for many, many different companies, and there is a chance that many, many of them are vulnerable to this attack of name server hijacking.

21:18 - So we started from AWS. We reported the vulnerability, and it was fixed really, really quickly by the AWS Route 53 team.

21:31 - Again, I think that within a week, or two it was fixed, and no one can now utilize this vulnerability in Route 53, because you are not allowed anymore to register those special domains in the name server, and we are in disclosure process with several additional cloud providers, and we believe there is many, many more to come, and it’s part also of what we call the industry to start looking into, and actually check across all of the DNS providers.

22:00 - - So as Ami just said, AWS fixed the vulnerability very easily.

22:06 - They added all the names of their name servers to an ignore list, which is simple, and very effective.

22:14 - Users trying to register one of the official AWS name servers on the Route 53 platform are now receiving the following error message, which says that the domain is invalid, and you can see it on the slide.

22:29 - When we reported our discovery to Microsoft, they explained to us that this is a known misconfiguration that occur when the organization works with the external DNS resolvers, and it’s not considered as a vulnerability.

22:45 - So we would like to offer a solution for both platforms and organizations who would like to protect themselves from these kind of attacks.

22:55 - So first, we will start with the platforms.

22:58 - DNS providers who want to ensure they are not vulnerable should make sure it is impossible to register their own domains on the platform.

23:07 - DNS providers who want to have even a better security can also do ownership verification to ensure users only register their own domains.

23:18 - And in addition, it is very important to make sure that the platform user cannot register a reserved name, as specified in the DNS RFC.

23:28 - The RFC is full of reserved domains that should not be allowed to register, and their registration may lead to unexpected behavior.

23:38 - For organizations that want to make sure they’re not vulnerable, we recommend making sure that the primary name server in their SOA record does not point to a different domain owned by the DNS provider.

23:50 - As you can see in the slide, and now you can see it, the default primary name server that a domain owner receives when they add their domain to the Route 53 is the first of the four name servers, which manage the DNS zone.

24:07 - Changing the primary name server to an invalid sub-domain of the organization, or the even the real primary master server will fix the issue, and attackers, and will not allow potential attackers to register domain on the platform, as you can see in the slide.

24:29 - And yeah, we are very close to the end. - So just a few summaries and take-aways.

24:37 - First of all, what’s really cool here is that we were able to get to nation state intelligence capabilities from a simple domain registration.

24:46 - Just a simple for the registration got us so much power, and we believe what we saw here is a new class of DNS vulnerabilities.

24:53 - This is just one idea of a domain. Think about how many different interesting domains you can try to register, and remember today, you can basically, right, register any domain that you want that would trigger unexpected results.

25:05 - No one thought, and we honestly didn’t understand initially what would be the impact of registering the name server on itself.

25:12 - I’m sure there’s many other magic domains that you can register, and it opens up so many interesting questions, like all of this traffic was dynamic DNS.

25:20 - Why is it actually, dynamic DNS was built as a protocol like 20 years ago for on-premise networks.

25:28 - Why are we still seeing it outside in the internet? What are the implications of this protocol still being active on the internet, and potentially endangering both the endpoints and the DNS servers, right? So there’s so much here that we see as potential research areas for us and also for the community, and we believe the scope here is huge, ‘cause it’s not a single service.

25:53 - We found it across multiple DNS providers, and we are pretty sure that this vulnerability and many others will probably impact many, many, many of those DNS providers.

26:07 - - All right, thank you very much, guys. - Thank you. - It was very fun.

26:11 - - Yeah?.