»doc flabby Posted July 27, 2009 Report Posted July 27, 2009 (edited) Firstly the current use of 00 as a prepend for protocol packets is very wasteful: If we adjust it so only the first byte is used to indicate the packet type for all packets, we can save space and simplify the protocol: Secondly the current protocol is inconsitant with regards to endianness. ALL Subspace 2 packets are BIG-ENDIAN (network byte order). Packet xx [data] 00-0F Protocol Packets (used for protocol) 10-FF User Packets (user can use) 0x00 Connection =============== UDP is a connectionless protocol. Therefore we must develop a protocol similar to TCP to handle "connections" only in a more optimised way than TCP. This is what the original subspace protocol was designed to do. Using a 4way handshake and cookies, we avoid the risk of connection spoofing. Client->Server SYN Packet 00 01 Server->Client ACK Packet 00 02 [cookie] Client->Server COOKIE Packet 00 11 [cookie] Server->Client Ready Packet 00 12 After the server sends ready packet the client can begin sending data. Any differenciation between client and server ends after this connection sequence. Closing connection (same as continuum): Host A->Host B FIN Packet (disconnect) 00 FF Host B->Host A FIN Packet (disconnect) 00 FF 0x01 Clock Syncronisation ========================= In order for accurate timing (Pretty much identical to the Subspace Sync Packet) A->B SYN 01 20 [A Current DateTime] B<-A ACK 01 21 [A Current DateTime][b Current DateTime] DateTime format --------------- Uses the format from here: http://doc.trolltech.com/4.5/datastreamformat.html struct DateTime { uint32 JulianDay // http://en.wikipedia.org/wiki/Julian_day uint32 Milliseconds since midnight uint8 [0|1] //0=localtime 1=UTC } These packets can then be used to syconise the clocks using this formula. From RFC 2030 Timestamp Name ID When Generated ------------------------------------------------------------ Originate Timestamp T1 time request sent by client Receive Timestamp T2 time request received by server Transmit Timestamp T3 time reply sent by server Destination Timestamp T4 time reply received by client The roundtrip delay d (ping) and local clock offset t are defined as d = (T4 - T1) - (T2 - T3) t = ((T2 - T1) + (T3 - T4)) / 2. It also allows us to caluclate the latency of the connection. This feature is also used as a keep-alive mechanism. Failure to receive syncornisation will result in the connection being dropped. Some mainipulation afterwards of the data collected can be used to compensate for lag more accurately, http://www.mine-control.com/zack/timesync/timesync.html Other Packet Types: =================== There are only 3 types of data needed Low Latency (packet positions etc): Datagram Reliable Datagram High Latency (download of level, chat messages): Ordered Reliable Stream Datagram ======== Sent plain using UDP. Starts with 10-FF Max Size 510 bytes; xx [data] Reliable Datagram ================= Sent plain using UDP. Max Size 510 bytes; Data: 03 xx [Datagram] ACK: 04 xx Resends Data if ACK not received xx incremented with each packet sent. Max 128 packets in transport. Stream ====== TBA Gonna finish this later Let me know what you think or any other network improvements you would make. Edited August 6, 2009 by doc flabby
Dr Brain Posted July 31, 2009 Report Posted July 31, 2009 Why not leverage the accumulated knowledge of thousands of engineers and use TCP/IP for your reliable packet stream? Reinventing the wheel is fun and all, but it's still reinventing the wheel (and the existing wheel is pretty damn good).
Kilo Posted July 31, 2009 Report Posted July 31, 2009 Given that bandwidth is an issue (mostly for hosts, but also for some users), isn't the overhead TCP has bad? Reliability and in-order is nice, but both are not always needed, and the other stuff that comes with TCP is less needed, which is why (as far as I know) most real-time games use UDP and figure it out on their own. If reliable packets were very occasional, then why go to the trouble of making a separate packet stream for them? If they are not occasional, why deal with overhead when it's not necessary to create the desired effect?
Dr Brain Posted July 31, 2009 Report Posted July 31, 2009 With reasonable connections, the TCP overhead is minimal. On connections with a lot of loss, the TCP overhead becomes greater, but UDP won't be doing much better in those situations anyway. The reason to use TCP is to reduce programmer burden. It's less trouble to "make a separate packet stream for them" than to make (and debug) a reliable UDP system.
»doc flabby Posted July 31, 2009 Author Report Posted July 31, 2009 (edited) With reasonable connections, the TCP overhead is minimal. On connections with a lot of loss, the TCP overhead becomes greater, but UDP won't be doing much better in those situations anyway. The reason to use TCP is to reduce programmer burden. It's less trouble to "make a separate packet stream for them" than to make (and debug) a reliable UDP system. The problem with TCP is latency rather than overhead. TCP can be influenced by certain QoS algos that ISPs use to manage connections which can add alot of latency. Also using TCP at the same time as UDP can lead to weird interactions between the two. Also you can handle a much larger number of connections on the server side using UDP. With alot of connections, each of which is related to one and other in a real time way, TCP becomes problematic. IE if one packet is dropped on one connection, it will hold up ALL the other clients whilst they wait for that packet to arrive.Also from a programming point of view. With udp you can simply have a loop waiting for packets to arrive. With TCP you need to check each individual socket. On a webserver TCP is a perfect choice because each webpage download has no effect on any of the other downloads. In the case of a game. You are directly influencing the information the other clients receive the packets you send. The majority of real-time multiplayer games use UDP as a communication protocol. There is a strong argument however for using TCP (most likely by running a HTTP server) to handle large file downloads (ie level/map/program updates), that occur when the user is not playing as its perfectly suited to this. Also QoS is useful in this situation. For many games TCP is fine, but for a Real-Time game like Subspace, its unfortunately not suitable. If you look at any other real-time game it uses UDP as its core-protocol) Unfortunately this means we have to fake some aspects of TCP we like (ie streams - for chat commands for example) (cluster packets which are incidentally a poor mans nagle's algorithm) Theres a pretty good essay about this here: (read the comments too)http://gafferongames.com/networking-for-game-programmers/udp-vs-tcp/ Problems with TCP and UDP interacting (in a bad way) Basically TCP connections causes UDP packetloss, the more connections, the more lag (this is more of a problem on the server side, as the client will only have one TCP connection to the server) Also its why people who download using P2P (which created ALOT of TCP connections) and play subspace end up lagging.http://www.isoc.org/INET97/proceedings/F3/F3_1.HTM But this has made me have an interesting thought, the only packet (apart from the big ones like level downloads) I can see needing more than 512bytes (and thus require the stream protocol) is the chat packet (ok we could do what continuum does and not allow you to send packets that long, but if I want to support Unicode, in some languages that would only give you 128 character to express yourself, that said, we don't want people spamming the whole screen Edited July 31, 2009 by doc flabby
Samapico Posted July 31, 2009 Report Posted July 31, 2009 Interesting... so the only reason to limit the message length in continuum is the UDP packet size limit? I thought it was just a pseudo-anti-spam measure
»jabjabjab Posted August 1, 2009 Report Posted August 1, 2009 Makes sense because every other game does the same thing by limiting to 200 or 250 characters (255?? I dunno)
Bak Posted August 1, 2009 Report Posted August 1, 2009 There is no set UDP packet-size limit; the internet is capable of breaking up packets into smaller pieces and reassembling them (although this will slow things down, see http://en.wikipedia.org/wiki/IP_fragmentation ). You are usually okay if you stay below 1400 bytes per packet. The subspace limit (~400 bytes per packet) is imposed by subspace itself, possibly to reflect the state of the internet (typical MTU) when subspace was designed. I don't understand why one packet dropped will hold up connections on all the clients. I think brain was proposing using TCP for reliable streams, such as chat, and UDP for real-time droppable packets such as position updates.
»doc flabby Posted August 1, 2009 Author Report Posted August 1, 2009 (edited) There is no set UDP packet-size limit; the internet is capable of breaking up packets into smaller pieces and reassembling them (although this will slow things down, see http://en.wikipedia.org/wiki/IP_fragmentation ). You are usually okay if you stay below 1400 bytes per packet. The subspace limit (~400 bytes per packet) is imposed by subspace itself, possibly to reflect the state of the internet (typical MTU) when subspace was designed. I don't understand why one packet dropped will hold up connections on all the clients. I think brain was proposing using TCP for reliable streams, such as chat, and UDP for real-time droppable packets such as position updates. Its based on the MTU of a dial up modem which is 576. which is the smalled MTU allowed on the internet. Once you take into account IP/UDP headers this gives you a max of 508 bytes to play with. I think subgame limits itself to 476 or something similar (maybe trying to be on the safe side?) http://stackoverflow.com/questions/1098897/what-is-the-largest-safe-udp-packet-size-on-the-internet Packet fragmentation is really best avoided with UDP because it can cause alot of packet loss, and it can't be measured. Ill explain why. My House (ethernet) ADSL Router My ISP Random Router Random Router Friend's ISP My Friend (poor guy is on dialup) MTU: 1500 1454 1500 1300 1500 1500 576 Packet: 1500 1454 1454 1300 1300 1300 576 46 46 154 154 154 576 46 46 46 148 154 46 So you can see a single datagram can end up as 5 packets, If just one packet is lost, the entire packet is dropped. All these fragmentations will slow down the packet as well adding latency, some routers will refuse to fragment packets. TCP uses MTU path discovery to find the optimal MTU for your connection, to avoid fragmentation. Fragmentation is a bad thing. The problem with using just TCP (and no udp) is that if you write a chat message, that then loses a packet, all your position packets will be held until that chat message is received by the server, which means ALL the clients will receive the wrong position for you whilst the server is waiting for a position update, because the chat packet is taking up the stream. Of course you could use two tcp connections, but this all begins to add to overhead... Why not leverage the accumulated knowledge of thousands of engineers and use TCP/IP for your reliable packet stream? Reinventing the wheel is fun and all, but it's still reinventing the wheel (and the existing wheel is pretty damn good).I think i need to read more carefully, I thought he was suggesting using TCP for ALL packets....man im so dumb sometimes -_- And as i've discussed in my previous post I'm pretty much agreeing with brain that TCP is the answer for streams Edited August 1, 2009 by doc flabby
Dr Brain Posted August 1, 2009 Report Posted August 1, 2009 Yeah, I was only suggesting TCP for chats, bricks, maps, arena changes, etc. Definitely not position packets.
»doc flabby Posted August 6, 2009 Author Report Posted August 6, 2009 Yeah, I was only suggesting TCP for chats, bricks, maps, arena changes, etc. Definitely not position packets.Updated with clock syncronisation
Gannon8 Posted August 7, 2009 Report Posted August 7, 2009 Also, could we have all files compressed instead of the map and news files? It takes forever to download all the graphics on the HS zone in the current system (no offense dr brain). It's probably caused by sending packets with max size of 510 bytes, but compression would be nice even over a TCP stream (of course make it optional.) Supposedly we could do a UDP packet type "Request TCP Connection" and just use TCP for file transfers and other relatively uncommon stuff.Force users to stay in spec when they transfer a file so that they won't lag out Aparently the attach file button is not the Submit posed button
Samapico Posted August 7, 2009 Report Posted August 7, 2009 Graphics (lvz) are already compressed (zlib) lvl files aren't though. Continuum's file transfer system sucks balls.
Dr Brain Posted August 7, 2009 Report Posted August 7, 2009 LVL files are compressed using zlib before being transferred. Effectively, everything is already compressed. It's just the transfer system doesn't utilize the maximum bandwidth of the connection.
»doc flabby Posted August 7, 2009 Author Report Posted August 7, 2009 I was going to have level/settings downloads done via HTTP. That way you could host level files on a different computer to the game server, thus avoiding level downloads causing lag on the game server because of people downloading a level using up all the bandwidth (which is why they are so slow in continuum, its by design). Its also pretty simple to implement a basic http server, to have for a server start pack, and port 80 generally is available to all users (www access is not normally blocked). Also it avoids the problem of trying to tie up the UDP connection with a TCP one for each client, which would get complex.
Recommended Posts