dns Featured

DM on GitLab Slack: I need your DNS skills!

Michael Friedrich

• 5 min read
DM on GitLab Slack: I need your DNS skills!

How to start a good story?

Uh oh. I used to actively maintain DNS infrastructures from 2009-2012 at the University of Vienna, only ops troubleshooting since then.

Ok ... let's see. I am located near Nuremberg, Brendan joins from Maryland. Dummy check in the browser - works for me. Probably not the most helpful reply, I know -  this gives the other side confirmation where to look next, async :)

First steps to debug

The CLI tool dig is my Swiss army knife. Every DNS question is followed by opening a terminal and typing in ...

dig gumroad.com
dig gumroad.com +short
dig gumroad.com +trace

The reason for the trace command is to see the full nameserver chain and how to reach the domain. This can help later to troubleshoot network routes and is just there to have it in console memory.

We already knew that the primary nameserver is Cloudflare. The traces from Brendan suggested that something was really fishy going on with the resolver or the local network.

Also, the works-in-VPN test makes this theory more appealing.

Ask the nameservers

Another way to check whether the recursive resolvers provide the same results is to ask them directly with @<nameserver IP/FQDN. This overrules the local resolver setting for this query.

Looks good. Now it is time to compare my environment to Brendan's environment.

In theory, someone plays proxy

In this quick example, two engineers collaborate and pick each others brain.

At this stage, I already knew that either the pihole or the router would intercept the DNS traffic. If you cannot do it that quickly, common questions involve:

  • Do you have any sorts of device or firewall between your router and the Internet?
  • Any child protection settings enabled on the router which may influence this?
  • Which router hardware and versions are involved, get a screenshot of the admin interface (then google your way through the handbook settings of privacy or DNS protection and ask the remote side to show what's there).

Is it the router that filters DNS queries?

Since I learned that Ubiquiti is involved, I could refine my parallel Google search.

  • ubiquiti router to get an idea about model names and specifically ask for these
  • ubiquiti router dns problem for a generic search case, maybe firmware related or people with generic search cases and wrong config settings.
  • ubiquiti router dns filter to learn more about proxies and filtering (my suspicion)

I learned a new term EdgeRouter during my research, and that it may allow to forward DNS queries. Pretty advanced stuff, so this piece of hardware certainly has more controls to keep in mind (note to self).

The search for filters led to this and this entry, both not solving the problem but giving an indication about children protection.

Maybe still the pihole?

Inspecting the DNS resolver settings of the router now.

Then I asked Brendan to query the local resolvers to see an anti-pattern for these errors maybe.

Bingo!

Look at the SOA from the first screenshot. This overrides the original SOA of the domain, and has cleanbrowsing.org inside.

DNS firewall and MITM

Hello again Google! There is a DNS firewall shared as feature on cleanbrowsing.org . That smells fishy, as it also says "protective DNS" ... others would say "Man in the Middle attack" with re-routing your DNS requests to a different destination.

Then I found this Reddit post:

It hijacks your DNS and sends it to cleanbrowsing.org. Ubiquiti could have subscribed to their DNS feeds, filtered locally, and continued forwarding to your chosen DNS server so that it wouldn't break your local DNS... but that would cost them money so fuck you.

Not sure how this works exactly since the cleanbrowsing pricing page says that API access is granted only for pro plans.

Making things worse, all of that is hidden behind this setting on a Uniquiti router!

It worked again after disabling the knob 🧐

DNS filters for families?

Guess why I do not like Cloudflare for families - because you cannot debug the above in an easy manner (you can open a support ticket and wait). The more alarming thought: If they ever decide to censor gumroad.com they just can do it and claim it to be an error after public discussion. Only because you have set their resolvers and thrown away the power of the distributed approach of DNS. You've given up control. Don't do that, protect your family.

Agreed, child protection and safe surfing is a needed thing. Truly not at the cost of giving away self-control and doing everything automagically. If Ubiquiti would use local filter lists and work more transparent, this whole thing would be easy to "self heal".

It always is DNS

In this case, it needed two engineers and 23 minutes time each. I would not want to imagine your frustration digging into this ... if you need to, this blog post hopefully can help.

  • Learn about dig
  • Ask/know key questions about the environment (router, proxies, etc.)
  • Run debug DNS queries
  • Rule out segments and devices, as if it is a game
  • Don't trust the other side. Let them show you screenshots of settings and results, and verify them if you are both sides.
  • Know your Google foo. Adopt searches in new tabs as you gather new information.

Additional tools if you're debugging from the couch

These are handy if you don't want to switch on the work Macbook, and just continue on your iPad to quickly help. I did not do the above, but wanted to add this just in case you are :)