In this section

DNS Forensics Foundations: Resolver Logs, NXDOMAIN Bursts, DNS Tunnels and Encrypted DNS

Module 0

Introduction

Almost every connection on a network starts with a name lookup, and the lookup is recorded before the connection exists.

That makes DNS the earliest and often the most complete record of what a host intended to reach: which names, how often, and with what result. It is also one of the most abused protocols, because attackers use it to find their infrastructure, to generate names faster than defenders can block them, and to carry data where other protocols are watched.

A cold lookup, through the chainws45stub resolverResolver10.20.0.53Authoritative203.0.113.53A www.example.net?A www.example.net? (from 198.51.100.10)answer, with a TTLanswer; cached at the resolverlater, ws46 asks the same nameanswered from cache; nothing sent outside

ws45 asks the company resolver; the resolver, translated to the firewall's address, asks the authoritative server and caches the answer. The next client to ask is answered from the cache, and nothing leaves.

This lesson reads four minutes of the lab's DNS through a real resolution chain: a company resolver, root and top-level servers and an authoritative server, with sensors inside and outside the firewall and the resolver's own reply log. It covers how a lookup moves through the chain, what each position records, failed lookups, a host the sensor never saw, a tunnel and encrypted DNS.

Module 5 teaches each part in depth.

1

A Lookup Is a Record of Intent

Before the connection, the question

At the start of the window, ws45 asked for www.example.net. The resolver logged the question and its reply:

$ grep 'www.example.net. A IN' resolver.log | cut -c17- | head -4
unbound[5753:0] info: 10.20.30.45 www.example.net. A IN
unbound[5753:0] info: 10.20.30.45 www.example.net. A IN NOERROR 0.000208 0 49
unbound[5753:0] info: 10.20.30.46 www.example.net. A IN
unbound[5753:0] info: 10.20.30.46 www.example.net. A IN NOERROR 0.000000 1 49

Each reply line carries the fields unbound's documentation lists for its reply log: the client's address, the name, the type and class, the return code, the time to resolve, whether the answer came from cache, and the response size.

The first pair is ws45's lookup: NOERROR, resolved in a fraction of a second, from cache 0, so the resolver had to go and ask. The second pair is ws46 asking for the same name a few seconds later: from cache 1, answered without any outside query.

A lookup records intent rather than action. It shows that software on ws45 wanted the address of www.example.net at that moment; whether a connection followed is a separate record, from a firewall, a sensor or a flow export, joined on the answer and the time.

That separation is useful in both directions. A lookup with no connection after it can mean the connection was blocked or never attempted; a connection with no lookup before it means the address came from somewhere else, a cache, a hard-coded value or a configuration file.

The sensor on the LAN recorded the same exchange from the client's side, in Zeek's DNS log:

$ zeek-cut ts id.orig_h query qtype_name rcode_name answers TTLs \
  < zeek-lan/dns.log | grep -m2 'www.example.net'
1794298562.219119	10.20.30.45	www.example.net	A	NOERROR	203.0.113.80	300.000000
1794298568.230548	10.20.30.46	www.example.net	A	NOERROR	203.0.113.80	294.000000

Zeek adds the answers and their TTLs, the time the answer may be cached, which the reply log does not carry. Two records from two positions, agreeing on the client, the name and the time.

They also differ in one useful way. The resolver's reply log has a line for every answer it gave, including answers from its cache, but nothing about what the answer contained. Zeek's log has the answer and its TTL, but only for clients on the link it watches.

An investigation that needs to know which address a name pointed to at a given minute reads the sensor or an outside capture; one that needs every host that asked reads the resolver. Note also the second TTL: 294 seconds rather than 300, because ws46's answer came from a cache that had already held it for six seconds.

2

Through the Chain

What each position can see

A lookup that the resolver cannot answer from cache goes out through the firewall to the servers responsible for the name. The outside sensor shows the beginning of that, for the first names of the window:

$ tshark -r outside.pcap -Y 'dns && frame.number>=5 && frame.number<=18' \
  -T fields -e ip.src -e ip.dst -e dns.flags.response -e dns.qry.name
198.51.100.10	203.0.113.4	False	intranet.corp.example.net
203.0.113.4	198.51.100.10	True	intranet.corp.example.net
198.51.100.10	203.0.113.5	False	intranet.corp.example.net
203.0.113.5	198.51.100.10	True	intranet.corp.example.net
198.51.100.10	203.0.113.53	False	intranet.corp.example.net
203.0.113.53	198.51.100.10	True	intranet.corp.example.net
198.51.100.10	203.0.113.4	False	intranet
203.0.113.4	198.51.100.10	True	intranet
198.51.100.10	203.0.113.53	False	intranet.corp.example.net
203.0.113.53	198.51.100.10	True	intranet.corp.example.net
198.51.100.10	203.0.113.4	False	intranet
203.0.113.4	198.51.100.10	True	intranet
198.51.100.10	203.0.113.53	False	www.example.net
203.0.113.53	198.51.100.10	True	www.example.net

Every query outside carries the source 198.51.100.10, the firewall's address. The resolver asked; the firewall translated; the client that started it is invisible from here. Outside, ws45's lookup appears once, as the resolver asking the authoritative server 203.0.113.53 for www.example.net. ws46's identical lookup a few seconds later does not appear at all, because the resolver answered it from cache.

That is the rule for DNS evidence by position:

Where a lookup can be recorded

On the client's link

The question and the answer, with the client's address

In the resolver's log

Every client it serves, cached answers included, in whatever detail it is set to log

Outside the network

Only cache misses, and only with the resolver's or firewall's address

The client's link and the resolver's log name the client. Outside the network, only cache misses appear, under the resolver's address, so a name looked up by fifty workstations may show up outside once. An organization that monitors only its internet edge sees a thin and anonymized fraction of its own DNS.

The same capture shows the chain at work. For the first name of the window, intranet.corp.example.net, the resolver asked the root server at 203.0.113.4, then 203.0.113.5, then the authoritative server 203.0.113.53: each referral points one step closer, which is how a recursive resolver finds the server for a name it has not seen.

By the time ws45 asked for www.example.net, the resolver already knew where example.net lived and went straight to 203.0.113.53.

Read the addresses too. Every query leaves from 198.51.100.10 and every response returns to it, so an outside capture can pair them, measure how long each server took and see what each referral said, but it cannot attribute any of them to a workstation.

Joining an outside query back to a client takes the resolver's log, matched on the name and the second, or the firewall's translation records if the resolver itself is the client being investigated. The outside capture earns its place for a different question: what the internet was told.

Internal names reaching the root servers, and the order in which the resolver walked the chain, are visible only there. Module 5's first lesson follows one lookup through every referral, and through the caches that make the second lookup cheap.

3

The Resolver Names Hosts the Sensor Never Saw

An office without a sensor

The resolver serves every office. The LAN sensor watches one segment. The two lists of clients are not the same:

Replies by client and result, from the resolver's logws45 (LAN, sensor)94ws46 (LAN, sensor)76219ws47 (second office, no sensor)48NOERRORNXDOMAINSERVFAIL

Three clients, three different pictures. ws47 sits in an office with no network sensor, so this log is the only record that it looked up anything at all.

ws45 and ws46 appear in both the reply log and the sensor's DNS log. ws47 appears only in the reply log. It sits in a second office, 10.20.40.0/24, with no sensor, and its lookups reach the resolver over the company's internal routing. Every one of them failed:

$ grep '10.20.40.47' resolver.log | grep NXDOMAIN | cut -c33- | head -4
grep -c '10.20.40.47' resolver.log
info: 10.20.40.47 bkqcaiqopbreeq.net. AAAA IN NXDOMAIN 0.000257 0 101
info: 10.20.40.47 bkqcaiqopbreeq.net. A IN NXDOMAIN 0.000046 0 101
info: 10.20.40.47 xepptuhrbprtnh.org. AAAA IN NXDOMAIN 0.000696 0 104
info: 10.20.40.47 xepptuhrbprtnh.org. A IN NXDOMAIN 0.000111 0 104
96

Twenty-four names that look random, each asked for as an A and an AAAA record, every one NXDOMAIN, the response for a name that does not exist. A host trying many generated names, none of which has been registered, is the pattern domain generation algorithms produce: malware computes a long list of candidate names, and its operator registers one or two of them when needed.

Until then, every attempt fails. The names also carry no history: none of them appears in the resolver's baseline log, recorded before the window, and no other client asked for any of them. A typo, a misconfigured application or a retired service produces failures for names that recur; a generator produces a fresh set every time it runs.

The reasonable mistake

Reading a quiet sensor as a quiet network

An analyst queries the sensor's DNS log for the 24 random names a threat feed lists, finds none, and reports that no internal host contacted them. ws47 looked up all 24; it sits in an office whose traffic the sensor never sees, and its lookups reached the resolver over the WAN.

Instead: Search the resolver's own log first for any question about which host looked up a name: it serves every office. The sensor's log answers only for the links the sensor watches.

The point is general. A sensor answers questions about the links it watches and is silent about everything else, and its silence is easy to mistake for a negative result.

The resolver is often the single place where every internal host's lookups converge, which is why its logs are among the most valuable a network keeps, and why Module 5 treats them as a primary source rather than a supplement to packet capture.

The same reasoning applies to remote workers on a VPN, to servers in a cloud subnet and to any segment added after the sensors were placed: wherever DNS still goes through the company resolver, the resolver sees it.

The reply log has to be switched on to be useful. unbound's documentation says log-replies defaults to no and that printing the lines makes the server significantly slower; many resolvers in production log nothing about individual queries, and the convergence point goes unrecorded.

Before relying on a resolver during an investigation, check which logging it has on and how long it keeps the lines, because the answer decides whether a question about last week can be answered at all.

4

Failed Lookups Are Evidence Too

Not every failure is the same

Failures are a large share of the window, and they come from different causes:

What failed lookups usually are

A burst of random names

Software generating domains until one resolves

The same internal name, repeatedly

A search suffix leaking a name meant for the inside

SERVFAIL for one zone

A broken delegation, often a decommissioned service

ws47's burst is the first kind. ws46's failures are mostly the second and third. Its configuration appends a search suffix, corp.example.net, to short names, and it repeatedly looks up a name called intranet: the workstation asks for intranet.corp.example.net and then for intranet on its own, the resolver forwards both, and both fail.

The outside capture above shows the consequence: the resolver sent those internal names to the root and top-level servers on the internet. A name meant for the inside leaked out, which is a privacy and sometimes a security problem, and it shows up only in an outside capture or the resolver's log.

ws46's SERVFAIL replies all come from one name, backup.legacy.example.com. The resolver's error lines say why: all servers for the zone legacy.example.com failed and their addresses were not usable, a lame delegation, where the parent zone points at name servers that cannot answer.

That pattern is usually a decommissioned service someone forgot to remove from a configuration, and the failing lookups themselves are harmless; but a lame delegation in a domain the organization owns can be taken over by whoever registers the abandoned name, which makes it worth reporting.

The resolver records the reason in its own error lines: all servers for legacy.example.com failed, and no address for them was usable. That text, rather than the client's bare SERVFAIL, is what separates a broken delegation from a resolver that was simply overloaded or cut off from the internet.

Reading failures by client and pattern, not as one count, is the habit. A dashboard showing NXDOMAIN volume for the whole network would show all three causes as one number; split by client and name, they are three different findings, one of which is an infected host.

The split takes one command against the reply log, grouping by client, name and return code, and it is the first thing worth running on any resolver log handed over in an incident. Module 5's lesson on failed lookups reads each cause on its own records.

5

Names at the Length Limit

When DNS carries data

One zone in the window behaves like no other. ws45 sent queries for names under stats.example.org, and none of them looks like a name anyone would type:

$ grep 'stats.example.org. NULL IN$' resolver.log | awk '{print length($7)}' \
  | sort -n | uniq -c | tail -3
grep 'stats.example.org. NULL IN$' resolver.log | head -1 | cut -c33-112
      1 168
      1 180
     26 253
info: 10.20.30.45 yrbu51.stats.example.org. NULL IN

91 queries for 91 distinct names, 26 of them 253 characters long as the log writes them, all of them type NULL, a type no other query in the window asked for. Ordinary names in the same window are short; the longest outside this zone is 62 characters.

These names are not lookups. They are data, encoded into the labels of the name, sent through the company's own resolver to an authoritative server the tunnel's operator controls, which reads the data out of the query and sends its own back in the answer.

That is a DNS tunnel, and the lab's is a real one, iodine 0.7.0 carrying a file. It works because the company resolver will forward a query for any name to whichever server is authoritative for it, so a host that cannot reach the internet directly can still exchange data with it through DNS.

The firewall allowed nothing unusual; the resolver did its job. Nothing in the tunnel's packets is malformed. Each query is a valid question for a valid name, answered by a server that really is authoritative for the zone; the abuse is in the volume and the content, not the protocol.

That is why stopping workstations from sending DNS to the internet does not stop a tunnel through the company's own resolver. Only measuring the traffic, or refusing to resolve the zone, does.

What gives a tunnel away is its shape rather than any single query: long names, many distinct names under one zone, an unusual record type, a steady rhythm, and far more bytes per lookup than ordinary resolution needs. Module 5 measures each of those against a baseline and follows the tunnel through its setup, transfer and keep-alive phases.

6

What Encrypted DNS Leaves Visible

The query is hidden on the wire, not at the resolver

Partway through the window, ws46 sent lookups to the resolver over DNS over TLS on port 853 and DNS over HTTPS on port 443. The LAN sensor's DNS log has nothing for them; its TLS log has this:

$ zeek-cut id.orig_h id.resp_h id.resp_p server_name < zeek-lan/ssl.log
10.20.30.46	10.20.0.53	853	resolver.corp.example.net
10.20.30.46	10.20.0.53	443	resolver.corp.example.net

Two TLS sessions from ws46 to the company resolver, with the resolver's own name in the server name field. The port tells the two transports apart: 853 is DNS over TLS, and 443 is DNS over HTTPS, which on the wire looks like any other web session to that address.

The sensor can say that ws46 used encrypted DNS and to which resolver; it cannot say which names it asked for. The resolver, which decrypts every query in order to answer it, logged them exactly as before.

Encryption here protects one hop, from the client to the resolver. It is a real protection against anyone watching the client's link, and it is no protection at all against the resolver's own log.

That changes the investigative picture only for clients that bypass the company resolver: a host using an outside encrypted resolver on port 443 hides its lookups from the company completely, and from the sensor looks like ordinary HTTPS to a well-known address.

The resolver's log for the two seconds in which those sessions started shows what the sensor missed:

$ grep -E '08:17:(17|20) .*10.20.30.46' resolver.log | grep -v -e ' IN$' -e hash.rep \
  | cut -c33-
grep 'cdn-sync' zeek-lan/dns.log | wc -l
info: 10.20.30.46 cdn-sync.example.org. A IN NOERROR 0.001073 0 468
info: 10.20.30.46 www.example.net. A IN NOERROR 0.000000 1 60
0

cdn-sync.example.org appears in the resolver's log and nowhere in the sensor's DNS log: the lookup went inside the TLS session. The www.example.net answer came from cache, as before, and its logged response size is 60 bytes where the earlier answers were 49, a difference Module 5 takes up when it reads encrypted DNS sizes.

The window also holds ws45's lookup for use-application-dns.net, answered NXDOMAIN: the canary domain Firefox checks before switching on DNS over HTTPS by default, which a network answers that way to say no. Module 5's last lesson reads all of it: what the LAN, the resolver log, the outside capture and the TLS metadata each keep when DNS is encrypted, and what padding does to the sizes.

7

What Module 5 Builds On It

DNS investigations in depth

Module 5 starts with resolution chains and the DNS message read field by field, then turns to the investigative questions: unusual domains measured against a baseline, failed lookups read by client and pattern, query patterns over time, tunneling indicators on the iodine tunnel this lesson met, and encrypted DNS visibility.

Its exercise is Case 2: a suspected DNS tunnel mixed with legitimate automated queries, where each busy zone has an explanation to test before it is called malicious.

Module 5, in order

015.1 to 5.2Resolution chains, and the DNS message field by field
025.3 to 5.4Unusual domains against a baseline; failed lookups by pattern
035.5 to 5.6Query patterns over time; tunneling indicators
045.7 to 5.8Encrypted DNS visibility; Case 2

Every lesson in the module uses this lesson's lab and its records. Nothing has to be installed or captured again before starting it.

The resolver's reply log, the LAN sensor's Zeek and Suricata logs and the outside capture are the same files, read more deeply: the full referral chain behind each name, the per-host baselines that make an unusual domain stand out, and the timing that separates a tunnel's keep-alive from an update agent's polling.

The order matters. Reading one lookup completely, from the stub through the resolver to the authoritative server and back, is what makes the later patterns readable.

A tunnel looks like nonsense until the reader knows what a normal query and answer contain, and what sizes an ordinary lookup produces in the reply log; a burst of failures looks like noise until the reader knows which failures are normal for a given client.

The habit to carry into it is to treat the resolver as the first source for any question about names, and to read every DNS record for its position: a client-side record names the host, a resolver record names every host it served, and an outside record names only the resolver and only for what it did not have cached.

Write the position down next to every finding, because a later reader needs to know which of the three the evidence came from before they can judge what it leaves out.

Do

Practice

What does the resolver know?
the lab resolver

The prepared evidence below is the window's DNS in three views: replies by client and result from the resolver's log, queries grouped by client and zone, and where each of ws46's transports can be read.

Find the host the sensor never saw, the tunnel, and where encrypted lookups survive. Each answer comes with the reasoning behind it, so check the reasoning against the records rather than only the choice. The Clients tab joins the reply log to the sensor's log by address; the Zones tab is the same grouping Module 5 builds into a baseline.

If you run your own resolver, or a home router that can log queries, turn on query logging for an hour and read which devices looked up what. The habit of asking which device, which name and which result transfers directly to the module.

A lookup records what a host intended to reach, before any connection, and each position in the resolution chain records a different share of it: the client's link and the resolver's log name the host, the outside sees only the resolver and only cache misses.

The resolver's log is often the only record of hosts beyond the sensor's reach; failures are read by client and pattern; names at the length limit are data; and encrypted DNS hides queries on the wire but not at the resolver.

The next lesson reads the web traffic those lookups usually lead to, encrypted and not.

Versions this lesson relies on: Wireshark and TShark 4.6.9; Zeek 9.0.1 (LTS); Suricata 8.0.8; unbound 1.19.2; iodine 0.7.0.