Directory Servers

[L.C.]

Hey L.C.,

 

I've tryed the same directoryservers to query for SSCV Dragonball Z using my own directory client;

 

IP		PORT	POP	SCORING	VERSION	TIMESTAMP	NAME			DESCRIPTION
============================================================================================================================================
sscentral.com
74.86.4.98	21000	4	257	134	1267777114146	SSCV Dragonball Z	ss://ds1.hlrse.net Fast pace action with intense battles in a multitude of carefully constructed bases. Fight with your team to capture the seven dragonballs and defend your base to the death with your friends.

ds.hlrse.net
74.86.4.98	21000	4	257	134	1267777234557	SSCV Dragonball Z	ss://ds1.hlrse.net Fast pace action with intense battles in a multitude of carefully constructed bases. Fight with your team to capture the seven dragonballs and defend your base to the death with your friends.

nanavati.net
74.86.4.98	21000	1	257	134	1267777328487	SSCV Dragonball Z	ss://ds1.hlrse.net Fast pace action with intense battles in a multitude of carefully constructed bases. Fight with your team to capture the seven dragonballs and defend your base to the death with your friends.

ds1.krslynx.com
74.86.4.98	21000	4	1	134	1267777278111	SSCV Dragonball Z	ss://ds1.hlrse.net Fast pace action with intense battles in a multitude of carefully constructed bases. Fight with your team to capture the seven dragonballs and defend your base to the death with your friends.

ssdir.playsubspace.com
74.86.4.98	21000	1	1	134	1267777418076	SSCV Dragonball Z	ss://ds1.hlrse.net Fast pace action with intense battles in a multitude of carefully constructed bases. Fight with your team to capture the seven dragonballs and defend your base to the death with your friends.

ds1.hlrse.net
74.86.4.98	21000	1	1	134	1267777461621	SSCV Dragonball Z	ss://ds1.hlrse.net Fast pace action with intense battles in a multitude of carefully constructed bases. Fight with your team to capture the seven dragonballs and defend your base to the death with your friends.

 

I see no problem with the zone's directory entries. (My directory client seems to have problem interpreting the scoring boolean though.)

 

It's possible that the Continuum client doesn't receive the zone because the description contains "ss://ds1.hlrse.net" . The client will try to resolve the IP of the zone by resolving the DNS name after "ss://" . More information here, here and here

I am not sure what I am missing here. O_o This problem only started to happen starting a few days ago, earlier this week. DBZ's description was not changed and has been the same for quite a long time. The description also begins with ss://ds1.hlrse.net/ ... so........

[»Lynx]

I was just curious as to whether anybody else has noticed this:

 

http://dl.krslynx.com/junk_pics/drpepper.jpg

 

It's using port 4990, and is sending requesting UDP packets. I've not checked to see exactly what packets it's requesting (but I am assuming the directory server stats, as I /think/ that uses UDP). If I block it, it will cease requesting packets until midnight UTC, and then will slowly request more and more (up until around 10MBPS).

 

Anyway, in total it's consumed around 150GB of bandwidth, so unless anybody has any answers I'll investigate more and if nothing appears, that IP is getting banned.

[PoLiX]

dreamhost... it is probably maverick.

[rootbear75]

150GB over how long?

[»Lynx]

24-48 hours. The bandwidth consumption doesn't bother me too much, I've still got plenty I'm just curious as to where it's coming from.

 

Edit: Speeds are lower above, as I killed it for a while to investigate/prepare the server to monitor future traffic.

Edited March 9, 2010 by Lynx

[Samapico]

Something tells me a directory server shouldn't use 150GB of bandwidth over 2 days

 

 

That's just me...

[»Lynx]

Likewise, either way it's using port 4990 (unless dirservers use 4991, I forgot) but if it belongs to nobody in the community I'll just block the IP etc.

[»Maverick]

I'm querying all the directory servers every minute for the population statistics; (drpepper.dreamhost.com is the server it's running on)

I have a basic directory client that doesn't do anything more then just send the appropiate packets (encryption request, zone list request), receives the zone directory list and saves it to a file.

 

I seriously doubt it would use 150 GB although it may add up over time. If it causes problems, I will stop the querying process.

 

EDIT: Quick and dirty calculations;

Each time a zone list is requested and returned, it takes around 8000 bytes (that's the size of the file that's stored with the results)

Every day, that would take [24*60*8000] 11520000 bytes = 10,9 MB . Every month that would be (31 days) 340 MB. That's still a substantial amount and load on the directory server (which would only be queried seldomly by normal clients) but shouldn't be a real problem (I hope).

 

 

L.C., PoLiX, can you check if your directory server is also generating such bandwith consumption ASAP?

I will stop all directory server querying if it does. We don't want the few directory servers that we have to go down because of my population statistics

Edited March 9, 2010 by Maverick

[»Maverick]

Lynx, do you know since when the bandwith consumption has been like this?

[»Lynx]

Well, I got some E-Mails from my provider when it started to hit 5MBPS upstream, and then I just found what was causing the surge, made sure it wasn't (too) important and killed it. The first spike began on 24th Feb to 1MBPS upstream in UDP packets, then returned to the normal (800 or so kbps) then continued to swell over the next few days up to a peak on 27th Feb where data from upstream UDP packets reached around 10MBPS. Here are some bandwidth recordings for when I realized it started again (at 3MBPS this time).

 

http://bw.krslynx.com/

 

Seeing as things have gone down to a mere few kbps, it should be fine. Next time it requests a lot of packets, I'll send you a message so you can check on your end too.

[»Maverick]

I'm really puzzled by this.

The only thing I can think of is my directory client not closing the connection properly somehow (although it does send disconnection packets and closes the UDP socket when it's done) which causes the directory server to keep sending packets (the zone list most likely). If this takes longer then a minute, this will result in a repeating effect as my directory client would again open up a connection to retrieve the zone list.

[Samapico]

http://bw.krslynx.com/

Nice peak there...

[L.C.]

4990 is the port that the directory server broadcasts, and the port directory clients use to retrieve listings from directory servers.

 

4991 is the port that ASSS and Subgame2 use to broadcast themselves to the directory server, and is the port on which the directory server accepts broadcasts from zones.

 

You also have to factor in these things:

* With UpstreamServers enabled, every 60 seconds (assuming this is what you have ZoneRefreshSeconds set to) your directory server will download a list from each directory server listed in UpstreamServers; all but ds1.hlrse.net does this -- I think doc flabby's Central does this as well

* People other than Maverick's Perl instance is checking the directory server too

* Doc flabby is also running the Perl script

 

Maverick, how often does yours check the directory server?

Edited March 9, 2010 by L.C.

[»Lynx]

I would be inclined to say that there are multiple sockets opening up on the script at once (refer to the iftop image above, you'll notice four different silmultanious connections from drpepper.dreamhost.com at once), I can only assume that the directory server is sending an unexpected packet or something unexpected is happening causing connections to stack up and cause a swarm, that's a wild guess though. Is your directory client code open source/available to inspect by any chance?

 

Either way, I'm monitoring all packets I'll inspect further as soon/if it happens again.

[Guest doc flabby - away form hom]

My directory server (Central) does not have this problem. Alothough it does end up stacking up connections which eventually crash it (still not worked out why this happens)

 

Are you using the latest version of snrrubs dir server?

[»Maverick]

Maverick, how often does yours check the directory server?

Every minute.

 

Is your directory client code open source/available to inspect by any chance?

Not really. I can send you the source code for inspection but first I want to try to debug it myself

 

Several things may be causing this behaviour. My directory client does receive an unknown packet (0x90) upon connecting to the directory server but everything works fine.

The directory client also identifies itself as using the Continuum protocol while it communicates using the known Subspace protocol (this was needed to communicate with Doc Flabby's directory server). This identification mismatch shouldn't really be a problem as no encryption is used anyway but I intend to restore this back to Subspace identification to see if it helps.

 

I hope to find out what I changed on the 24th february. If it was this continuum / subspace identification then this might be the culprit.

[»Lynx]

Yeah, I'm using version 1.1 of Snrrrubs directory server. Good luck with finding out the error, Mav.

[»Maverick]

I can't find the problem. Unfortunately Eclipse removed the Local History so I couldn't look back to what was done on the 24th.

 

I've checked the packets being sent and received and there is nothing out of the ordinary.

The directory server returns a correct Disconnection Request (0x07) packet after it has done sending the directory list and my directory client closes the connection.

Changing my directory client to identify itself as Continuum or Subspace doesn't seem to matter anything to the packets being received.

 

Communicating with ds1.krslynx.com results in 155 bytes send, 10049 bytes received, totalling in 10204 bytes transferred (about 10 kB).

(Note: tested it on my own computer at home, not on the server.)

 

I've tried running Snrrrub's directory server at my own computer to see if the directory server would give any strange error messages but nothing was reported in the (debug) logging.

 

 

EDIT: Might it be possible that this URL is requested alot and results in bandwith usage? http://forums.trenchwars.org/showthread.php?38966-Directory-Servers

It uses Doc Flabby's script for querying if a directory server is online and also runs on the DreamHost server. I doubt this script would be giving any problems as it's caching the page for 30 minutes (changed it to 60 minutes now) though.

Edited March 9, 2010 by Maverick

[»Maverick]

sscentral.sscuservers.net is back online

 

I've made a few small changes to my directory client to garantuee that a directory server is only queried at a maximum of once a minute.

Edited March 9, 2010 by Maverick

[»doc flabby]

You also have to factor in these things:

* With UpstreamServers enabled, every 60 seconds (assuming this is what you have ZoneRefreshSeconds set to) your directory server will download a list from each directory server listed in UpstreamServers; all but ds1.hlrse.net does this -- I think doc flabby's Central does this as well

 

Thats a very aggressive settings. I would recommend Upstream Servers is set to 1 hour or more.

 

Central only downloads every 4 hours.

[»Lynx]

Hrmm, I increased mine to 1 hour, thanks for the tip. Edited March 9, 2010 by Lynx

[PoLiX]

No major bandwidth spikes for me.

[»Maverick]

Alright, PoLiX.

 

Lynx, do you still have the dreamhost server IP banned on your end?

[L.C.]

k, everybody running Sharvil's directory server set your ZoneRefreshSeconds to 240 (4 hours).

[»Lynx]

Hey Mav, my firewall script kicked in and I never saved after banning your IP, it is unbanned again now.

 

LC, 4 hours = 14,400, 240 = 4 minutes.