A dedicated server can run its own nameservers, which is what lets you offer ns1.yourbrand.com rather than pointing clients at someone else's. It is also one of the easier things to get subtly wrong, because a misconfiguration takes domains offline rather than producing an error you can see.
Two halves, in two places
This trips up nearly everyone who tries it.
At your registrar, register the nameserver hostnames as glue records, ns1.yourbrand.com and ns2.yourbrand.com, each pointing at an IP address on your server. This is what makes those names exist on the internet at all.
On the server, run a DNS service that is authoritative for the zones and answers for them.
Doing only the server half is the common failure. Everything looks configured, and client domains fail because the nameserver names they were told to use resolve to nothing.
Use two addresses
Two nameservers is the minimum, and pointing both at the same IP address gives you no redundancy at all; the machine going down takes both.
If the server has two addresses, use them. Better still, put a secondary nameserver on different hardware entirely, since the point of the second one is surviving the first being unreachable.
Verify before telling anyone
dig ns1.yourbrand.com dig NS clientdomain.com dig clientdomain.com @ns1.yourbrand.com
The first confirms the glue record resolves. The second confirms the domain delegates to you. The third confirms your server actually answers for that zone.
All three, before a single client points a domain at you. Debugging this with a live client site is considerably less pleasant than debugging it with your own test domain.
Do not run an open resolver
An authoritative nameserver answers for zones you host. A recursive resolver answers any question for anyone.
Leaving recursion enabled and reachable from the internet means your server is used in amplification attacks, and the traffic, the complaints and the blocklisting all land on you.
Disable recursion, or restrict it to your own network. This is the single most important setting on a public nameserver and it is off by default in some software and on in others, so check rather than assume.
The record that stops your mail
Every zone needs its own MX records, SPF and DKIM. A zone created from a bare template resolves the website perfectly and silently has no mail configuration at all.
That failure is quiet: the site works, so the migration looks successful, and mail stops without anything reporting it. Check MX explicitly for every zone you create. For what has to be there, see understanding MX records.
TTLs, before you change anything
Lower the TTL on a record a day before you intend to change it. The change then propagates in minutes rather than hours, and a mistake corrects equally fast.
Lowering it at the same moment as the change does nothing, because the resolvers you are waiting for cached the old record with the old TTL. See understanding DNS propagation.
Whether to run DNS at all
Worth asking. A dedicated nameserver is a service to secure, patch and monitor, and DNS failures take everything down at once rather than degrading.
If the reason is branding, private nameservers on a reseller plan achieve the same appearance without the operational burden. Private nameservers in WHM explains that route.
Run your own when you need control the platform does not offer, and accept that DNS is now something you monitor.
Publishing an AAAA record is a promise that the server can be reached that way, and a half-finished setup breaks only for some visitors. Understanding IPv6 and When You Need It explains the order that avoids it.
Confirm both servers answer identically
Running your own nameservers means two systems must agree, and a difference between them produces a domain that resolves inconsistently.
for ns in ns1.example.com ns2.example.com; do printf '%-22s %s\n' "$ns" "$(dig "@$ns" example.com SOA +short | head -1)" done dig example.com NS +short
The serial number in the start of authority record is the version. Different serials mean one server has not received the latest zone, and roughly half of visitors are being answered from the older copy.
Check after every change rather than assuming the transfer happened. A synchronisation that silently stopped produces a domain that appeared to work for months and then behaves unpredictably after an edit.
Make sure the delegation matches what you run
The names published at the registrar and the servers you actually operate have to correspond exactly.
dig example.com NS @a.gtld-servers.net +short dig ns1.example.com A +short dig ns2.example.com A +short
Each published name must resolve to an address that answers for the domain. A name pointing at a machine that no longer runs the service produces intermittent failures, since resolvers try the servers in no fixed order.
Where the nameservers are under the domain they serve, the parent zone also needs address records for them, since resolving the nameserver would otherwise require asking the nameserver.
Decide whether this is worth running yourself
Operating nameservers means accepting responsibility for a service whose failure takes everything down at once.
Two servers on the same machine, in the same rack, or on the same network are not redundancy. Genuine separation means different networks and ideally different providers, which is more infrastructure than most estates justify.
dig example.com NS +short | while read n; do printf '%-24s %s\n' "$n" "$(dig +short "$n" | head -1)"; done
Read the addresses. If they are adjacent, they share everything that matters, and the arrangement provides the appearance of redundancy without its substance. Secondary DNS covers what real separation involves.
Confirm both servers survive a restart
Nameservers are configured once and rebooted rarely, so the arrangement that works today has frequently never been tested against a restart. Reboot each one deliberately, at a time you choose, and confirm the service starts and answers afterwards. Do them separately rather than together, so the domain continues resolving throughout. A machine that has run for a year without restarting has not demonstrated that its configuration survives one, and discovering otherwise during an unplanned reboot takes the whole domain down rather than half of it.