Discussion:
DNS problems yesterday?
(too old to reply)
Patrick Mchale
2007-01-18 20:26:16 UTC
Permalink
Hey There,

I was wondering if anyone noticed an odd external DNS problem happening yesterday, the National Library's
main website was affected for several hours. We could resolve internally but nothing from outside would get
resolved. Nothing was changed on our DNS servers. Vodafone seemed to be affected also.

Default Server: planet.natlib.govt.nz
Address: 192.122.171.130
server 210.55.131.76
Default Server: jupiter.natlib.govt.nz
Address: 210.55.131.76
www
Server: jupiter.natlib.govt.nz
Address: 210.55.131.76

Name: slbweb.natlib.govt.nz
Address: 210.55.131.96
Aliases: www.natlib.govt.nz
server defiant.netgate.net.nz
Default Server: defiant.netgate.net.nz
Address: 202.37.245.17
www.natlib.govt.nz
Server: defiant.netgate.net.nz
Address: 202.37.245.17

Name: slbweb.natlib.govt.nz
Address: 210.55.131.96
Aliases: www.natlib.govt.nz
server reliant.net.nz
*** Can't find address for server reliant.net.nz: Non-existent host/domain
server reliant.netgate.net.nz
Default Server: reliant.netgate.net.nz
Address: 202.37.245.20
www.natlib.govt.nz
Server: reliant.netgate.net.nz
Address: 202.37.245.20

Name: slbweb.natlib.govt.nz
Address: 210.55.131.96
Aliases: www.natlib.govt.nz
server 203.96.152.4
Default Server: rachel.paradise.net.nz
Address: 203.96.152.4
www.natlib.govt.nz
Server: rachel.paradise.net.nz
Address: 203.96.152.4

*** rachel.paradise.net.nz can't find www.natlib.govt.nz: Non-existent host/domain
server 202.73.198.15
Default Server: isdn1ldv.vodafone.co.nz
Address: 202.73.198.15
www.natlib.govt.nz
Server: isdn1ldv.vodafone.co.nz
Address: 202.73.198.15

*** isdn1ldv.vodafone.co.nz can't find www.natlib.govt.nz: Non-existent host/domain


Cheers

Patrick McHale
National Library of New Zealand.
Nathan Ward
2007-01-18 20:32:28 UTC
Permalink
natlib.govt.nz SOA dns1.natlib.govt.nz. networks.natlib.govt.nz.
2007011817 3600 1200 1728000 86400

The serial suggests (of course, it doesn't prove, as the serial is a
freeform number), that changes were done lots yesterday.
It is my assumption that someone was fiddling.
Post by Patrick Mchale
Hey There,
I was wondering if anyone noticed an odd external DNS problem
happening yesterday, the National Library's
main website was affected for several hours. We could resolve
internally but nothing from outside would get
resolved. Nothing was changed on our DNS servers. Vodafone seemed to be affected also.
Default Server: planet.natlib.govt.nz
Address: 192.122.171.130
server 210.55.131.76
Default Server: jupiter.natlib.govt.nz
Address: 210.55.131.76
www
Server: jupiter.natlib.govt.nz
Address: 210.55.131.76
Name: slbweb.natlib.govt.nz
Address: 210.55.131.96
Aliases: www.natlib.govt.nz
server defiant.netgate.net.nz
Default Server: defiant.netgate.net.nz
Address: 202.37.245.17
www.natlib.govt.nz
Server: defiant.netgate.net.nz
Address: 202.37.245.17
Name: slbweb.natlib.govt.nz
Address: 210.55.131.96
Aliases: www.natlib.govt.nz
server reliant.net.nz
*** Can't find address for server reliant.net.nz: Non-existent host/
domain
server reliant.netgate.net.nz
Default Server: reliant.netgate.net.nz
Address: 202.37.245.20
www.natlib.govt.nz
Server: reliant.netgate.net.nz
Address: 202.37.245.20
Name: slbweb.natlib.govt.nz
Address: 210.55.131.96
Aliases: www.natlib.govt.nz
server 203.96.152.4
Default Server: rachel.paradise.net.nz
Address: 203.96.152.4
www.natlib.govt.nz
Server: rachel.paradise.net.nz
Address: 203.96.152.4
*** rachel.paradise.net.nz can't find www.natlib.govt.nz: Non-
existent host/domain
server 202.73.198.15
Default Server: isdn1ldv.vodafone.co.nz
Address: 202.73.198.15
www.natlib.govt.nz
Server: isdn1ldv.vodafone.co.nz
Address: 202.73.198.15
*** isdn1ldv.vodafone.co.nz can't find www.natlib.govt.nz: Non-
existent host/domain
Cheers
Patrick McHale
National Library of New Zealand.
_______________________________________________
NZNOG mailing list
http://list.waikato.ac.nz/mailman/listinfo/nznog
!DSPAM:22,45afd7fe128422051574865!
Patrick Mchale
2007-01-18 22:08:51 UTC
Permalink
Thanks for the responses, We kinda thought there was a bit of fiddling going on but at to
who and finding a definitive answer, this problem may be a total mystery forever.

We are though inquiring into one particular DNS provider to check if anything went
astray yesterday. The report was very interesting and a very useful website for
finding out about a particular sites DNS issues and security. I need to read up more
on the info it presented and sort out some changes.

Cheers

Patrick McHale
Hey There,

I was wondering if anyone noticed an odd external DNS problem happening yesterday, the National Library's
main website was affected for several hours. We could resolve internally but nothing from outside would get
resolved. Nothing was changed on our DNS servers. Vodafone seemed to be affected also.

Default Server: planet.natlib.govt.nz
Address: 192.122.171.130
server 210.55.131.76
Default Server: jupiter.natlib.govt.nz
Address: 210.55.131.76
www
Server: jupiter.natlib.govt.nz
Address: 210.55.131.76

Name: slbweb.natlib.govt.nz
Address: 210.55.131.96
Aliases: www.natlib.govt.nz
server defiant.netgate.net.nz
Default Server: defiant.netgate.net.nz
Address: 202.37.245.17
www.natlib.govt.nz
Server: defiant.netgate.net.nz
Address: 202.37.245.17

Name: slbweb.natlib.govt.nz
Address: 210.55.131.96
Aliases: www.natlib.govt.nz
server reliant.net.nz
*** Can't find address for server reliant.net.nz: Non-existent host/domain
server reliant.netgate.net.nz
Default Server: reliant.netgate.net.nz
Address: 202.37.245.20
www.natlib.govt.nz
Server: reliant.netgate.net.nz
Address: 202.37.245.20

Name: slbweb.natlib.govt.nz
Address: 210.55.131.96
Aliases: www.natlib.govt.nz
server 203.96.152.4
Default Server: rachel.paradise.net.nz
Address: 203.96.152.4
www.natlib.govt.nz
Server: rachel.paradise.net.nz
Address: 203.96.152.4

*** rachel.paradise.net.nz can't find www.natlib.govt.nz: Non-existent host/domain
server 202.73.198.15
Default Server: isdn1ldv.vodafone.co.nz
Address: 202.73.198.15
www.natlib.govt.nz
Server: isdn1ldv.vodafone.co.nz
Address: 202.73.198.15

*** isdn1ldv.vodafone.co.nz can't find www.natlib.govt.nz: Non-existent host/domain


Cheers

Patrick McHale
National Library of New Zealand.
Jean-Francois Pirus
2007-01-18 23:21:53 UTC
Permalink
Post by Patrick Mchale
I was wondering if anyone noticed an odd external DNS problem happening
yesterday

Yes we had some issues as well.

We kept getting the following messages:
named[4498]: client 127.0.0.1#42127: no more recursive clients: quota reached
(We have a limit of 1000 and an average of 25)

And we could not get to some of the root servers, ie: a.ROOT-SERVERS.NET

So I am assuming there was a DOS attack somewhere which affected DNS traffic.

Assumption: The name server could not get to some root servers, the queries
kept pilling up and we hit the quota.

It looks like it was an issue on the
- 14th ( 422333 quota reached)
- 17th ( 170472 quota reached)
- 18th ( 547309 quota reached)

To put it into perspective here are the numbers for the previous 5 weeks:
4228
28
16
856
801
--
------------------------------------------------------------------------
Jean-Francois Pirus <***@clearfield.com> Senior Software Engineer
Phone (+64-9) 358 2081 Clearfield Software Ltd
Fax (+64-9) 358 2083 4th Floor 8-10 Whitaker Place
Mob (+64-21) 640 779 P O Box 3901 Auckland, New Zealand
------------------------------------------------------------------------
Joe Abley
2007-01-18 23:37:27 UTC
Permalink
Post by Patrick Mchale
Post by Patrick Mchale
I was wondering if anyone noticed an odd external DNS problem
happening
yesterday
Yes we had some issues as well.
named[4498]: client 127.0.0.1#42127: no more recursive clients: quota reached
(We have a limit of 1000 and an average of 25)
Do you run open resolvers, or do you restrict use of your recursive
servers (by source address) to your customers only?

Almost every case I've seen where bind9 suffers query spikes like
you're describing (and are not just being hammered by an enormous
throng of customers) it has been because the server was being used by
someone far away as a packet amplifier. Throw on an ACL to restrict
recursive lookups (and to deny queries, if the servers aren't also
authority servers) and the problem frequently goes away.
Post by Patrick Mchale
And we could not get to some of the root servers, ie: a.ROOT-
SERVERS.NET
In case it's useful to know for future testing, F and I are the
servers that you have the greatest chance of reaching locally.


Joe
Juha Saarinen
2007-01-18 23:48:16 UTC
Permalink
Post by Joe Abley
Throw on an ACL to restrict
recursive lookups (and to deny queries, if the servers aren't also
authority servers) and the problem frequently goes away.
I'd be interested to see a working BIND 9 ACL to restrict recursion to
certain clients only.
--
Juha Saarinen
www.geekzone.co.nz/juha | Skype: juha_saarinen
blogs.pcworld.co.nz/pcworld/techsploder
www.computerworld.co.nz | MSN: ***@msn.com
Voice: +64 9 950 3023 Subtle recursive jokes in .sigs are not funny.
joshua sahala
2007-01-19 00:02:17 UTC
Permalink
Post by Juha Saarinen
Post by Joe Abley
Throw on an ACL to restrict
recursive lookups (and to deny queries, if the servers aren't also
authority servers) and the problem frequently goes away.
I'd be interested to see a working BIND 9 ACL to restrict recursion to
certain clients only.
in named.conf:

acl "localonly" {
192.168.1.0/24;
...
192.168.250.0/24;
};

options {

....

allow-recursion {
"localonly";
};

....

};

see the BIND admin reference manual for more info (or one of the many
howtos available on teh intarwebs)

/joshua
--
A common mistake that people make when trying to design something
completely foolproof is to underestimate the ingenuity of complete
fools.
- Douglas Adams -
Michael Fincham
2007-01-19 00:16:06 UTC
Permalink
Post by joshua sahala
Post by Juha Saarinen
Post by Joe Abley
Throw on an ACL to restrict
recursive lookups (and to deny queries, if the servers aren't also
authority servers) and the problem frequently goes away.
I'd be interested to see a working BIND 9 ACL to restrict recursion to
certain clients only.
acl "localonly" {
192.168.1.0/24;
...
192.168.250.0/24;
};
options {
....
allow-recursion {
"localonly";
};
....
};
see the BIND admin reference manual for more info (or one of the many
howtos available on teh intarwebs)
/joshua
Is there a significant difference between doing this and setting up two
different BIND "views"? I'm currently using two view, one for our
internal networks, and one for external networks, with an ACL to decide
which view applies and recursion disabled for the external view.

I've noticed that with a "views" configuration, the external view is
very slow to update (the servers are run as slaves) when the master is
updated. The internal view updates almost immediately, but it can be up
to an hour or so before queries hitting the external view get the
up-to-date records.

Would I be losing anything important if I switched to just using the
allow-recursion ACL? I suspect views might have been designed for a
different configuration scenario...
--
--Michael Fincham
Unleash Technology Solutions
Simon Allard
2007-01-19 00:58:23 UTC
Permalink
Post by Juha Saarinen
I'd be interested to see a working BIND 9 ACL to restrict recursion to
certain clients only.
http://www.cymru.com/Documents/secure-bind-template.html

Very good base point for securing your bind setup.
Joe Abley
2007-01-19 01:09:03 UTC
Permalink
Post by Juha Saarinen
Throw on an ACL to restrict recursive lookups (and to deny
queries, if the servers aren't also authority servers) and the
problem frequently goes away.
I'd be interested to see a working BIND 9 ACL to restrict recursion
to certain clients only.
Ask Team Cymru and ye shall receive. All that, and more!

http://www.cymru.com/Documents/secure-bind-template.html


Joe

Jean-Francois Pirus
2007-01-19 00:02:59 UTC
Permalink
Post by Joe Abley
Do you run open resolvers,
Noooo, we've had ACL's on bind for a long time.
Only our subnets can do recursive queries.

This server does a lot of recursive queries as it's a mail server and has to
check all those #%$#$% spam connections.
Post by Joe Abley
In case it's useful to know for future testing, F and I are the servers
that you have the greatest chance of reaching locally.
Thanks, I'll keep that in mind for next time.
--
------------------------------------------------------------------------
Jean-Francois Pirus <***@clearfield.com> Senior Software Engineer
Phone (+64-9) 358 2081 Clearfield Software Ltd
Fax (+64-9) 358 2083 4th Floor 8-10 Whitaker Place
Mob (+64-21) 640 779 P O Box 3901 Auckland, New Zealand
------------------------------------------------------------------------
joshua sahala
2007-01-18 23:55:34 UTC
Permalink
Post by Jean-Francois Pirus
named[4498]: client 127.0.0.1#42127: no more recursive clients: quota reached
(We have a limit of 1000 and an average of 25)
from the BIND 9.2 ARM:

recursive-clients

The maximum number of simultaneous recursive lookups the server
will perform on behalf of clients. The default is 1000. Because
each recursing client uses a fair bit of memory, on the order
of 20 kilobytes, the value of the recursive-clients option may
have to be decreased on hosts with limited memory.
Post by Jean-Francois Pirus
And we could not get to some of the root servers, ie: a.ROOT-SERVERS.NET
how were you testing this? did you try any of the other root servers? If
any of the root server were inaccessible it would have caused a whole lot
more noise on teh intarwebs.
Post by Jean-Francois Pirus
So I am assuming there was a DOS attack somewhere which affected DNS traffic.
where
somewhere == your server
Post by Jean-Francois Pirus
Assumption: The name server could not get to some root servers, the
queries kept pilling up and we hit the quota.
I think you need to look closer to home, so to speak...

Is this DNS server of your publicly accessible? If so, does it allow
recursion from anywhere/anyone? If so, then you become a free DNS server
for others, like spammers.

Try restricting recursion to your network only. If it is already
restricted, then check all your network hosts for signs of
malware/virii/backdoors/etc as it seems possible that an internal host or
two was spewing spam and making a lot of bogus recursive queries


hth
/joshua
--
A common mistake that people make when trying to design something
completely foolproof is to underestimate the ingenuity of complete fools.
- Douglas Adams -
Jean-Francois Pirus
2007-01-19 00:09:19 UTC
Permalink
Yes, I have read the documentation.
Post by joshua sahala
how were you testing this? did you try any of the other root servers? If
any of the root server were inaccessible it would have caused a whole lot
more noise on teh intarwebs.
It was an intermittent problem. ie: there was some packet loss probably
restricted to DNS traffic.
Post by joshua sahala
Is this DNS server of your publicly accessible? If so, does it allow
recursion from anywhere/anyone?
Nope.
--
------------------------------------------------------------------------
Jean-Francois Pirus <***@clearfield.com> Senior Software Engineer
Phone (+64-9) 358 2081 Clearfield Software Ltd
Fax (+64-9) 358 2083 4th Floor 8-10 Whitaker Place
Mob (+64-21) 640 779 P O Box 3901 Auckland, New Zealand
------------------------------------------------------------------------
Continue reading on narkive:
Loading...