what would you like to see in the vegastrike multiplayer?

A forum for online playing, administration, bugs and feature requests
Post Reply
rogue
Explorer
Explorer
Posts: 14
Joined: Sun Aug 07, 2005 4:52 pm
Contact:

Post by rogue »

Sorry to burst your bubbles, but the string based argument is better.
You can still negotiate compressed streams when you need to, and if that's just not enough, you can negotiate bit-based transfers VERY easily. Read on.


{Edited: Added missing numbers}
0000
0001
0010
0011
0100
0101
0110
0111
1000
1001
1010
1011
1100
1101
1110
1111
(Now I think that's all)

Code: Select all

- your counting is incorrect, which perhaps indicates that you have very limited familiarity with working on binary data.)
You're right, I forgot a few combinations in there. but not so many that it changes the point.


Sure you can put them back to back to form more complex patterns, but if you did that, you'd be making a char, which is what strings are made out of. A char is:
0000 0000 through 1111 1111

Strings do not incur much overhead. If the overhead of 8-20 bytes is to much for job X, impliment compression to reduce that further, easy enough, or use the other suggestion below.

The advantages out weigh any miniscule speed advantage users on 28.8k modem MIGHT see if they're downloading pr0n at the same time.
Being: If it were done an existing and VERY PROVEN methods (HTTP, TELNET, SMTP, POP3), could all be implimented in the same server over the same port AT the same time, without ANY noticeable speed hit to anybody.
(HTTP being the most important, as it can provide any data about the server anybody wanted to provide. I know of a lot of people who like setting up a website about their game server, and even know some who have gone as far as working their assess off to make callbacks to their game to extract data from it to have it be displayed live on a website (Which can be done in a number of ways, but this way is the easiest, as the code to do it ALREADY EXISTS)


And then if there are things you absolutely MUST argue the speed of a bit based protocol is absolutely necessary or it would all keel over and die, you can use the string based methods to negotiate a bit based stream for what needs it, then switch back - The same way one would negotiate a compressed stream, or even encrypted streams if that's needed - All without ruining compatibility with existing protocols.

My argument is so I can add Vegastrike support to my completely self written multi-protocol HTTP / telnet mmorpg server. (No, I didn't download smaug and hack it up and call it my own. I started from scratch, so I do know what I'm talking about. don't believe me? Look at http://ant.infice.com:5555/index.html )
and try telnet://ant.infice.com:5555 (wow, both work, http and telnet. Not even Apache does that.)

I have been working in the Vegastrike engine, and added a few basic necessitiies to start implimenting this stuff (the command processor).
if using XML, an externally verified parsing engine
Not XML, XML-LIKE tags. We would take something like::
<FOO>arguements</FOO>
and translate it to something the command processor recognizes, Exactly like:
FOO arguments

And with this method, it would be VERY easy to processess new tags, without having to add something new to the input routine to decipher a message, as all the tags can be deciphered exactly the same. All that would be needed is for the command "FOO" to be added to the command processor after the function itself to do FOO is created.
If the advantage you seek from a readable format is intelligible code for processing such, this is what enums are for.
Enums are intigers and have absolutely nothing to do network communications and being able to safely parse arbitrary commands between two applications.

I don't care how the internals of vegastrike work, this grids, graphs, edges are nonsense pertaining to this particular post. These posts I've made are about how to actually transfer data from a client, to a server, and from a server, to a client. Once the protocol is defined, it can be implimented in anyway, such as a distributed network of server applications running on multiple machines to create a larger universe that's faster and supports more clients, or a single server application with a universe and player connections limited to the resources on that machine.
jackS
Minister of Information
Minister of Information
Posts: 1895
Joined: Fri Jan 31, 2003 9:40 pm
Location: The land of tenure (and diaper changes)

Post by jackS »

rogue wrote: I don't care how the internals of vegastrike work, this grids, graphs, edges are nonsense pertaining to this particular post. These posts I've made are about how to actually transfer data from a client, to a server, and from a server, to a client. Once the protocol is defined, it can be implimented in anyway, such as a distributed network of server applications running on multiple machines to create a larger universe that's faster and supports more clients, or a single server application with a universe and player connections limited to the resources on that machine.
That's why the section was prefaced with "On a different note". It pertains to organizational imperatives in developing the server, and inter-server communications. Do note that this is a many-paged thread, and not all comments are directed at the immediately preceding post.
rogue wrote:
If the advantage you seek from a readable format is intelligible code for processing such, this is what enums are for.
Enums are intigers and have absolutely nothing to do network communications and being able to safely parse arbitrary commands between two applications.
You are correct that they have nothing to do with networking. This is also an irrelevant observation. The above was directed at the claim that it is somehow harder to see whether or not a value has been previously assigned to an existing command than a string. I fail to see how that does not remain a vacuous argument.
rogue wrote: You're right, I forgot a few combinations in there. but not so many that it changes the point.

Sure you can put them back to back to form more complex patterns, but if you did that, you'd be making a char, which is what strings are made out of. A char is:
0000 0000 through 1111 1111
Forgive me, but I still fail to see what point you were trying to make (some negative connotation as to the number of combinations available?).

rogue wrote: The advantages out weigh any miniscule speed advantage users on 28.8k modem MIGHT see if they're downloading pr0n at the same time.
Being: If it were done an existing and VERY PROVEN methods (HTTP, TELNET, SMTP, POP3), could all be implimented in the same server over the same port AT the same time, without ANY noticeable speed hit to anybody.
(HTTP being the most important, as it can provide any data about the server anybody wanted to provide. I know of a lot of people who like setting up a website about their game server, and even know some who have gone as far as working their assess off to make callbacks to their game to extract data from it to have it be displayed live on a website (Which can be done in a number of ways, but this way is the easiest, as the code to do it ALREADY EXISTS)


And then if there are things you absolutely MUST argue the speed of a bit based protocol is absolutely necessary or it would all keel over and die, you can use the string based methods to negotiate a bit based stream for what needs it, then switch back - The same way one would negotiate a compressed stream, or even encrypted streams if that's needed - All without ruining compatibility with existing protocols.

My argument is so I can add Vegastrike support to my completely self written multi-protocol HTTP / telnet mmorpg server. (No, I didn't download smaug and hack it up and call it my own. I started from scratch, so I do know what I'm talking about. don't believe me? Look at http://ant.infice.com:5555/index.html )
and try telnet://ant.infice.com:5555 (wow, both work, http and telnet. Not even Apache does that.)

I have been working in the Vegastrike engine, and added a few basic necessitiies to start implimenting this stuff (the command processor).

if using XML, an externally verified parsing engine
Not XML, XML-LIKE tags. We would take something like::
<FOO>arguements</FOO>
and translate it to something the command processor recognizes, Exactly like:
FOO arguments

And with this method, it would be VERY easy to processess new tags, without having to add something new to the input routine to decipher a message, as all the tags can be deciphered exactly the same. All that would be needed is for the command "FOO" to be added to the command processor after the function itself to do FOO is created.
Given that you aren't proposing a data side packet description format (which would be novel, although code changes would still be needed for additional functionality), I don't see this as particularly easier (yes, there's a single parsing and a switch dispatch rather than a switch parse and dispatch, but for our purposes, I don't see that being much more difficult to implement) except insofar as it allows you reuse of your own existing code. This would appear to be an argument for "more convenient" (assuredly, not an insignificant issue) rather than "better".

I am, in all honesty, much more concerned with the lack of clarity and soundness of reasoning in the presentation of your arguments than what appears to be the content of them. Aside from some overhead (which may or may not be a significant factor - this will depend on the size of the enclosed data and the required rates of data transferrance and processing - but the overhead is often not a determining performance factor) there's nothing intrinsically horrid about a string based encapsulation approach. However, I've yet to see a convincing argument as to why one homegrown low-level parser is preferable to a new, yet to be spec'd low-level parser excepting that there exist solutions of the former category.
Last edited by jackS on Sun Oct 16, 2005 11:23 pm, edited 1 time in total.
chuck_starchaser
Elite
Elite
Posts: 8014
Joined: Fri Sep 05, 2003 4:03 am
Location: Montreal
Contact:

Post by chuck_starchaser »

Well, the positions and velocities of ships and missiles and bolts have to be sent as a binary block; I don't think there's a choice about that, given the amount of data involved, IMO, and compressing a text representation would probably not get nearly the same efficiency. But if other things could benefit of XML-like things, then they could be added in text format to the binary block, I suppose?

There's more to all this than transmission speed. The speed of the executable as a whole is probably very adversly affected, currently, by so much parsing, IMHO, (python, xml, etc.), though I'd have to do some homework to prove it. Modern CPU's can typically have 100 microinstructions in flight at most times, due to parallelism and pipelining, which means, a dozen or two assembler instructions per clock-cycle, which means several dozen assembler instructions per NANO-second. Unfortunately they spend most of their time twiddling thumbs waiting for memory (hudndreds of nanoseconds latencies, therefore thousands of wasted cycles), on cache misses; or flushing the pipelines on branch prediction misses.

Compressing data as much as possible reduces memory footprint and increases cache coherence.

Text parsers are VERY branchy things, which takes up precious space in the branch prediction cache.

A hand-optimized binary format will, instead:
a) often achieve much higher compression than any generic compression algo
b) obviate the need for a middle step, --i.e.: de-compressing then parsing
c) often use switch statements that the compiler will optimize to offset tables rather than multiple, nested, conditional branches.
jackS
Minister of Information
Minister of Information
Posts: 1895
Joined: Fri Jan 31, 2003 9:40 pm
Location: The land of tenure (and diaper changes)

Post by jackS »

chuck_starchaser wrote: Modern CPU's can typically have 100 microinstructions in flight at most times, due to parallelism and pipelining, which means, a dozen or two assembler instructions per clock-cycle, which means several dozen assembler instructions per NANO-second.
<Pedantic> Actually, the width (in terms of execution units) is usually only 3-4 instructions on a general purpose processor. VLIW machines may be 6 wide or so. The bypass network and register file scale very poorly in both area and power, and clustering is not a panacea.</Pedantic>
chuck_starchaser
Elite
Elite
Posts: 8014
Joined: Fri Sep 05, 2003 4:03 am
Location: Montreal
Contact:

Post by chuck_starchaser »

Well, I was speaking of my (shorter but wider) Athlon 64... :D ... but it's true a lot of people are still buying Intel's effluent crap... (Athlons have 9 pipes, IIRC, though some share resources, like the FPU's).
jackS
Minister of Information
Minister of Information
Posts: 1895
Joined: Fri Jan 31, 2003 9:40 pm
Location: The land of tenure (and diaper changes)

Post by jackS »

10 execution units. 3 Integer, 3 Address Generators, 3 FP, 1 load-store. I misspoke; replace "execution units" in my previous post with "issue width" or "feeding execution units".

however, we're wandering off topic ;-)
klauss
Elite
Elite
Posts: 7243
Joined: Mon Apr 18, 2005 2:40 pm
Location: LS87, Buenos Aires, República Argentina

Post by klauss »

Haha...

Ok...

Rogue, don't take the following the wrong way. I love your command processor. It rocks. It gives much needed UI power. But...

...about networking, you're talking nonsense.
XML? HTTP (not HTML)? Are you serious?
In any data-intensive protocol, with is also a time-critical one, HTTP is out of the question. Any text-based protocol is out of the question. Basically, if you add a compression layer to a text parsing layer, CPU usage goes way above the acceptable threshold (for the server, I mean). That's what makes encryption also such a debatable need: yes, you need security. But encryption is comparable to compression in CPU usage. Actually, encryption usually requires a compresion layer before encryption to increase entropy. Anyway... no. No way it's justified. What for? As JackS said... so that humans could read the protocol? Why? They wouldn't be able to read that much traffic in a reasonable time, anyways, nor interact usefully through telnet, not even for testing.

You want machine independence? You want a standard protocol?
Try ASN.1 PER. It's a little hard to parse correctly, but it can be done efficiently since it basically amounts to a standardized binary protocol. I myself have written a (difficult to use) library for that. But... then again... even with simpler structures the machine dependency is easily overcame by accounting for endian formats (which, if you have 8-bit connections granted as it is the case with TCP/IP, is the only remaining issue).

Do you know what's involved in generating an ASCII string out of a float? At least as many divisions as digits, for start. Usually, those are implemented as multiplications by inverse. Still, 6 cycles per digit MINIMUM, plus the loop overhead. If you consider the potentially single cycle (if scheduling is right) that a floating point register load takes on native-endian CPUs, that's 60 times slower for a 10-digit float (which is the norm). Same thing for reading an ASCII string.

Also...
To write such a protocol is no small task (though it's not as enourmous as writing a 3D engine such as vegastrike)
Ehm... did you ever code an HTTP parser? Yes... splitting header/body is not that hard. But, now try retrieving the contents of a header field. That implies, at least, an access into a map. That map may be a hashmap, or a red-black tree, or a simple list. Whatever. But, in any case, pretty expensive. Not hard, I know, but expensive.

Now... you have a message that conveys a position update. If the message uses up 400 bytes (which is the norm in HTTP requests of the simple kind), you have an absolute maximum throughput of 327 messages per second on a 128kbps connection (normal upstream wideband), or 1308 on a 512kbps one (normal downstream wideband). Now... that implies, for a sustained 10hz update rate, a maximum of 130 units being updated. That's for a single client. What would be the upstream bandwidth required at the server to serve, say, 10 clients? 5 megabits. I'm not sure, but that seems quite a lot for a measley 10 clients. Now... replace the 400-byte update with a 40-byte one (binary protocol). Now, you go from 5 megabits to 512kbps at the server, or from 130 units to 1300 units at the client (without degrading the 5Mbps at the server).
Do you still think that a text-based (or even HTTP-based) protocol is justified?
Did I miscalculated any of the figures in such a way that would break my point? (I tend to mess up numbers, but I doubt any kind of messing up would break my point)

Really... do some research before proposing such a protocol. If you wanted to stick to text-based protocols, you would require massive compression. And, still, you wouldn't get anywhere close to the binary protocol's efficiency.
Oíd mortales, el grito sagrado...
Call me "Menes, lord of Cats"
Wing Commander Universe
ace123
Lead Network Developer
Lead Network Developer
Posts: 2560
Joined: Sun Jan 12, 2003 9:13 am
Location: Palo Alto CA
Contact:

Post by ace123 »

Well here's the way it is in the VS code so far is a simple 16-byte header (I don't know the specifics, because I don't have the code in front of me):
Each packet contains a unique unit serial # (binary - 4 bytes), a packet ID (binary - 4 bytes), and a timestamp of when it was sent (8 bytes, double). This results in a 16-byte header.

Since position updates are so important, I'm also planning on adding a position in the header (doubles - 24 bytes), if not also a velocity/rotation in every packet (7 floats - 28 bytes).

This would make the header at least 40 bytes, or 28 if we used floats for posision change from last full update rather than doubles.

On the other hand, if this would be a text format, aside from rewriting a lot of code, this would, assuming a two-byte serial and a 1-byte packet id, result in 8 bytes, with another 16-byte timestamp, assuming one makes the header in hex with no surrounding tags. We are already at 25 bytes, not to mention a position, which may bring it up 18*3 more bytes to a total of around 80 bytes, twice what we would have had with the ascii header.

I wouldn't worry about making it easier to parse. In fact, the easiest packet to parse is one already in binary. I wouldn't want a packet that has to be read by an XML parser. Just think of the speed difference in loading XMESH vs. BFXM. Since VS is open source, there shouldn't be a need for someone to look at it. Debugging is also best done in the game.

The only reason that I have used a text-based format before is so that it can interface with telnet (such as the soundserver, for example).

I wouldn't want to be the one fighting a VS space battle MUD-style from telnet, or Begin-style except real-time.
> fire all weapons
You have been hit 15 times since your last command. Your hull is down to 20%.
> turn left 20 degrees
> fire all weapons
You have died. Game over.
The best way to help would be to look at the code and try it out for yourself and fix bugs. There's a lot still to be done with interpolation and syncronizing time, not to mention everything else in VS has not yet been implemented because this is not done yet.

I'll be happy to answer any questions or talk with you about the code itself, but I just don't think that re-implementing a packet format is a good use of time right now. It's already implemented, and since everything is abstracted into a class, that would just as easily be done later, when everything actually works, than now.
Guest

Post by Guest »

Ryder P. Moses wrote:Less than one actual human per system? Geh. I'd think the magic number would be around at least five per. After all, the whole point of MP is human interaction. If that means fewer overall systems, well, so be it.

Around 20-50 online at a time is probably a more realistic estimate, though, yeah. This is not a mass-market game, and it will never have the tens of thousands of users that mass-market stuff gets.

But I guess that's the advantage of fully procedural environments, is they're scalable. If there's only an average of twenty people online at a time, well, the universe will be sized just right for that. If there's twenty hundred, hey, the universe is perfect sized for that, now!
How about including an "exploratory service" into the game. When the amount of players reaches a certain threshold, new systems could be explored by the exploratory service.
rogue
Explorer
Explorer
Posts: 14
Joined: Sun Aug 07, 2005 4:52 pm
Contact:

Post by rogue »

for intelligent conversation, talk to me on freenode :)
Ryder P. Moses
Daredevil Venturer
Daredevil Venturer
Posts: 593
Joined: Mon Sep 26, 2005 3:59 am

Post by Ryder P. Moses »

Guest: Arr, that'd work, although I still think that'd totally defeat the idea of MP.

Possibly a comp-stomp setup, sorta similar to dungeons in other MMOs, where there's a smallish default 'friendly' universe accessible at any time, and then some larger system chains at least partly populated by Aera or somesuch that you can form raiding parties and go out into.
CoffeeBot
Intrepid Venturer
Intrepid Venturer
Posts: 676
Joined: Wed Jul 06, 2005 5:25 am
Location: On the counter by the toaster
Contact:

Post by CoffeeBot »

Ryder P. Moses wrote:Guest: Arr, that'd work, although I still think that'd totally defeat the idea of MP.

Possibly a comp-stomp setup, sorta similar to dungeons in other MMOs, where there's a smallish default 'friendly' universe accessible at any time, and then some larger system chains at least partly populated by Aera or somesuch that you can form raiding parties and go out into.
How...does that defeat the purpose of MP?

My understanding was that the whole galaxy is going to be "hostile" or possibly unsafe, but if you stick in areas where it's dominated by friendly factions, you should theoretically be protected. Venturing into unpopulated areas, or areas controlled by hostile factions is where the danger comes in. There's always the possibility of being attacked wherever you go...it's just the chances of survival are greater in the more friendly-populated areas.

It should be noted that "friendly-populated" areas are completely relative -- if you're friends with the Aera, the Sol system ain't gonna be as safe for you as it would be for a player who's Confed.

WRT uncharted areas: It would be nice to find a way for characters who like to explore to get a chance to make some money doing what they love. With several thousand systems available to us, there's no reason we couldn't "close off" a few hundred and leave them temporarily hidden until such time as some cartography team looks for fearless adventurers looking to fly sluggish and poorly defended scanning vessels.
Duality
Daredevil Venturer
Daredevil Venturer
Posts: 583
Joined: Sun Feb 16, 2003 12:58 am
Location: West Coast of USA
Contact:

Post by Duality »

I don't know if this is a valid working idea but like IRC networks, a couple of servers located all over the world can connect all to gether as one universe.

One server contains a couple of systems. Like around 10 or something like that. You would still need to have a powerful server to hold large numbers of players.

As long as the universe, systems, and the contents are original it would be good enough.
CoffeeBot
Intrepid Venturer
Intrepid Venturer
Posts: 676
Joined: Wed Jul 06, 2005 5:25 am
Location: On the counter by the toaster
Contact:

Post by CoffeeBot »

@Duality: That's how most of the MMOs work, currently. Each "server" that you log into is in reality a rack or two, with each zone residing on a server, maybe a few on one server if it's small/unpopular. At least, that's how City of Heroes did it.

Well, on second though, they don't work like IRC does, but they do work like how you described. IRC's a little different in that each server acts as a form of relay, containing connections to every channel -- hence the "fork" errors that make half the people in a room disappear.
Ryder P. Moses
Daredevil Venturer
Daredevil Venturer
Posts: 593
Joined: Mon Sep 26, 2005 3:59 am

Post by Ryder P. Moses »

Coffeebot: How does structuring the game so that people are totally out on their own in a computer-populated universe defeat the purpose of multiplayer? Hmm, let me think on that for a second... maybe because it's not multiplayer if there's no other people there.

It doesn't matter whether you've got an open universe of 100 systems per person or you've got an option for people to leave the smaller 'populated' universe and go explore 100 computer-populated systems, either way the player is not interacting with other humans, and what you have is in fact a single-player game with multiplayer lag and connection instabilities.

If you've got 100 open systems per person, people will disperse and there won't be a 'populated' area. There needs to be a mechanism to keep people more or less in a relatively small area. If having a hundred Aera cruisers show up to whale on the player as soon as he steps out of 'populated' space is the best you can do for a containment system, something needs rethinking. That's just the space trader equivalent of an invisible wall.
chuck_starchaser
Elite
Elite
Posts: 8014
Joined: Fri Sep 05, 2003 4:03 am
Location: Montreal
Contact:

Post by chuck_starchaser »

I think the number of "active" systems could grow as more people come online, and shrink as they go off line. Servers could serve one or more systems depending on their CPU muscle and connection bandwidth, and they could negotiate which hosts which system. When you go online, latencies are tested and a server gets assigned to you, which may be hosting one or more systems to choose from. Unless location is saved, of course. Anyways, perhaps there could be numbers next to jump points... As you approach a jump point, you see the latency to the server hosting the system on the other side of it, so you may choose to go to some other system with lower latency.
klauss
Elite
Elite
Posts: 7243
Joined: Mon Apr 18, 2005 2:40 pm
Location: LS87, Buenos Aires, República Argentina

Post by klauss »

chuck_starchaser wrote:As you approach a jump point, you see the latency to the server hosting the system on the other side of it, so you may choose to go to some other system with lower latency.
Interesting idea.
Oíd mortales, el grito sagrado...
Call me "Menes, lord of Cats"
Wing Commander Universe
CoffeeBot
Intrepid Venturer
Intrepid Venturer
Posts: 676
Joined: Wed Jul 06, 2005 5:25 am
Location: On the counter by the toaster
Contact:

Post by CoffeeBot »

klauss wrote:
chuck_starchaser wrote:As you approach a jump point, you see the latency to the server hosting the system on the other side of it, so you may choose to go to some other system with lower latency.
Interesting idea.
Definitely. But, it also gives an indirect clue to someone running from other players, possibly even NPCs -- if he sees a latency that implies even mild activity over one of the gates, he won't use it, and choose another with a lower number. It sadly doesn't work in the other direction, because the hunters don't know where he ran, and for all they know, he went into a populated area.

But, I'm also a "worst-case scenario" guy. I always plan on the worst possible outcome. So, take my comments with a grain of salt :)
chuck_starchaser
Elite
Elite
Posts: 8014
Joined: Fri Sep 05, 2003 4:03 am
Location: Montreal
Contact:

Post by chuck_starchaser »

Well, that's kind of natural; if I wander off into a shady area and I get chased by some thieves, of course I will run towards an area I consider safer, better lit, with more police; and the chasers might give up the chase, judging the newer conditions disadventageous. So, if I'm in some system and some players who logged in China are after me, I will run for a low latency system. This is not a "problem", game-play-wise, as once I'm in the low latency system, there may be other threats there who also happen to have low latency like me. The net result would be to keep people in a geographical region hanging together most of the time, which is desirable, except for multiculturalism :D
rogue
Explorer
Explorer
Posts: 14
Joined: Sun Aug 07, 2005 4:52 pm
Contact:

rebuttal

Post by rogue »

klauss wrote:Haha...

Ok...

Rogue, don't take the following the wrong way. I love your command processor. It rocks. It gives much needed UI power. But...

...about networking, you're talking nonsense.
XML? HTTP (not HTML)? Are you serious?
In any data-intensive protocol, with is also a time-critical one, HTTP is out of the question. Any text-based protocol is out of the question. Basically, if you add a compression layer to a text parsing layer, CPU usage goes way above the acceptable threshold (for the server, I mean). That's what makes encryption also such a debatable need: yes, you need security. But encryption is comparable to compression in CPU usage. Actually, encryption usually requires a compresion layer before encryption to increase entropy. Anyway... no. No way it's justified. What for? As JackS said... so that humans could read the protocol? Why? They wouldn't be able to read that much traffic in a reasonable time, anyways, nor interact usefully through telnet, not even for testing.

You want machine independence? You want a standard protocol?
Try ASN.1 PER. It's a little hard to parse correctly, but it can be done efficiently since it basically amounts to a standardized binary protocol. I myself have written a (difficult to use) library for that. But... then again... even with simpler structures the machine dependency is easily overcame by accounting for endian formats (which, if you have 8-bit connections granted as it is the case with TCP/IP, is the only remaining issue).

Do you know what's involved in generating an ASCII string out of a float? At least as many divisions as digits, for start. Usually, those are implemented as multiplications by inverse. Still, 6 cycles per digit MINIMUM, plus the loop overhead. If you consider the potentially single cycle (if scheduling is right) that a floating point register load takes on native-endian CPUs, that's 60 times slower for a 10-digit float (which is the norm). Same thing for reading an ASCII string.

Also...
To write such a protocol is no small task (though it's not as enourmous as writing a 3D engine such as vegastrike)
Ehm... did you ever code an HTTP parser? Yes... splitting header/body is not that hard. But, now try retrieving the contents of a header field. That implies, at least, an access into a map. That map may be a hashmap, or a red-black tree, or a simple list. Whatever. But, in any case, pretty expensive. Not hard, I know, but expensive.

Now... you have a message that conveys a position update. If the message uses up 400 bytes (which is the norm in HTTP requests of the simple kind), you have an absolute maximum throughput of 327 messages per second on a 128kbps connection (normal upstream wideband), or 1308 on a 512kbps one (normal downstream wideband). Now... that implies, for a sustained 10hz update rate, a maximum of 130 units being updated. That's for a single client. What would be the upstream bandwidth required at the server to serve, say, 10 clients? 5 megabits. I'm not sure, but that seems quite a lot for a measley 10 clients. Now... replace the 400-byte update with a 40-byte one (binary protocol). Now, you go from 5 megabits to 512kbps at the server, or from 130 units to 1300 units at the client (without degrading the 5Mbps at the server).
Do you still think that a text-based (or even HTTP-based) protocol is justified?
Did I miscalculated any of the figures in such a way that would break my point? (I tend to mess up numbers, but I doubt any kind of messing up would break my point)

Really... do some research before proposing such a protocol. If you wanted to stick to text-based protocols, you would require massive compression. And, still, you wouldn't get anywhere close to the binary protocol's efficiency.
To bad you think it's nonsense, because it makes perfect sense if you take the time to read it all. Don't take it personally ... "(Famous words to say just before someone say's something offending to be an ass when their view differs)"

1) I never said XML , I said XML-TAGS, which is not XML, it's <FOO>. (See how that resembles XML or HTML? wow.) XML-LIKE. No fucking calls to XML. This is the third time I've said this.

2) Think outside the box, why would position info have to be sent as a string? What I posted was an example usage of the protocol, and how it would work. Not a definitate: Positions have to be strings . If that's what you think I said, you're being anal.

3) A barebones mmorpg server exists that already supports 2000 concurrent users.

4) These are computers, they arn't limited to "Do it one way and only this way or eat shit and die". No, humans are the ones that decide that shit.

5) There is no combat system in the ANT engine. Do you know what MUD stands for? MULTI-USER-DUNGEON. EVERY Mmorpg today uses a MUD TYPE SERVER as it's base. Something that does portal occullusion, something that passess messages between clients, something that accepts commands, and sends responses. ANT is a BARE-BONES MutliProtocol MMORPG SERVER. There is almost nothing game specific to muddle things up or confuse things. It was also designed with 3d mmorpg specifically in mind (as an eventuality, and the time has come)

6) NEGOTIATIONS . Negotiate sending of a position class between the client and server, this can be done in 1-3-5 bytes if that's how it gets set up. (Even if it's string based. WOOO. 1-5 bytes. Shit my pants huge)

7) I've already made the vegastrike module, it works, it connects, and is ready to start developing a multiplayer game on. It's easy to use, and doesn't require anything special to start it (just open the console, type: connectto host port , or connectto default). It parses commands from the server, and can send commands typed by the user to the server. It's 227 lines in a .cpp, and maybe 10-15 lines in a header.

it doesn't mean everything you've written has to be re-written. What you've written can be used, but how it can be used will have to be determined.

You all need to do research before going to tell me to do research. My research is my WORKING code, and my years of school and Cisco training, and my years of experience developing servers and clients.
Show me your ijmplimentation, show me it support 2000+ users on 256mb of ram simultaneously(Before any optimizations for size), and be damned fast, show me it support 3 completely different types of clients, one of which you wrote yourself, then you can tell me I'm speaking nonsense.
And read what I say. I never said string _ONLY_. That's narrow.
klauss
Elite
Elite
Posts: 7243
Joined: Mon Apr 18, 2005 2:40 pm
Location: LS87, Buenos Aires, República Argentina

Post by klauss »

@rogue:

Ok... not string only.

So... what are the tags for, then? I really don't understand what you were talking about now. I guess I never did. I'll just read your code... that will be much better than discussing it on the forum.

But... just so that you don't keep that (obviously bad) idea of myself... if you had said XML. If you had proposed using HTTP. Wouldn't what I said make sense?

I did network programming. Not cisco, but I think I know what I'm talking about. The problem, obviously, is that I don't know what you are talking about.

And... about the 2000+ users. I've done tests. If the kernel has no socket limit (I know, it does, but let's stick to theory... we can always cluster, as I was talking about network bandwidth and not CPU resources). Still, one CPU handling 2000 connections would require not much more than 1Gb of memory. The server, I mean. Assuming, of course, that they're not all in different systems. If they were... well... I didn't test that. But as we're talking about making a server serve only a bunch of systems at once, that would work, wouldn't it?
Oíd mortales, el grito sagrado...
Call me "Menes, lord of Cats"
Wing Commander Universe
rogue
Explorer
Explorer
Posts: 14
Joined: Sun Aug 07, 2005 4:52 pm
Contact:

Post by rogue »

Sorry if I seem a little pissy, I'm out of Mary jane to keep me calm :-P.

Ok, the tags (which would be 3 bytes overhead total), would only seperate commands that must be parsed and executed on the client from any other random string based data that may be sent.

(Nobody take this out of context, this is an example only)
Say you want to make a menu, the steps to doing this would be like:
create menu with "FOO" as the display text and menu trigger text
Create a menuitem with FOO as it's display text and BAR as what it executes when selected
Call the menu.

Now with even a small 16 byte structure as a header, this means you must send 16 byte header with the "cmd" variable set to "menuadd" or something.
Then you must transfer any string related data about the menu.

Then you must send another 16 byte header with it's cmd set to menuitem
Then transfer the display value for the menuitem, and the action the menuitem will take when selected (function to call or whatever).

Then after that you finally have to send a 16 byte header with the cmd set to Display menu, then (possibly) send yet another string with the menus name to actually call it.


Whereas with a string based protocol like the one I suggested and wrote (mmorpgclient.cpp & .h)
It *could* be simply:
<M="FOO"> //menu named foo
<MI="FOO" "BAR"> //menuitem named foo that does bar
<C> // call the last menu created
or: <C="FOO"> // call menu FOO

The overhead of the "tags" (<=>) is 3 bytes.
The overhead of each charactor is 1 byte.

the size of the strings "FOO" or "FOO""BAR" is constant between the two protocols.

The way I see it, 3 bytes overhead for the tags, 1 byte for the newline charactor which will trigger anything that's buffered. 4 bytes overhead for this protocol.
Then the name of the command can be contained in 1 byte as a regular charactor, once the 1 byte range runs out, new commands can be added at 2 bytes.

The overhead of creating a menu with a binary based protocol with a fixed sized 16 byte header is:
16 bytes for each header (which may be sent three times in this case), plus the size of the string data that must already be sent. (What good is a menu if you can't read it.)
The overhead of creating it with my proposed method is 4 bytes(<=>\n), plus 1 byte per charactor in the command. (which can be as small as 1 char)

after that the data the menuitem holds will be the same size over both protocols.




So let's count, if we send a fixed sized 16 byte header:

1 header to tell it the string it will be recieving is a menu
1 header to tell it the string it will be recieving is a menuitem
1 header to tell it to call the menu
3x16 = 48 bytes overhead for 3 commands.

With a ascii based protocol like I'm suggesting the size is completely variable, but can be as low as 5 bytes per command (and commands can be grouped, so it could be done: <M="FOO"><MI="FOO""BAR"><C>\n ,

that's 8 bytes for the tags AND the newline of overhead for 3 commands.
Plus 5 bytes for the commands themselfs.
8+5 is still less than 48.

("FOO" and "FOO""BAR" are not counted since they will remain the same over both protocols)


I hope this, and mmorpgclient.cpp &.h will make this clearer :).





---------------------------------------------------------------
the 2000 users is what my mmorpg server supports with 256 mb of ram (it's actually slightly more). before any optimizations for size.


----------------------------------------------------------------
I would touch on some other stuff here, but I'm damned tired and don't feel like it right now ;)
klauss
Elite
Elite
Posts: 7243
Joined: Mon Apr 18, 2005 2:40 pm
Location: LS87, Buenos Aires, República Argentina

Post by klauss »

@rogue...

If you don't like the fixed header, you can use ASN.1.

Take a look, at
http://asn1.elibel.tm.fr/en/standards/index.htm

It will seem a bit overwhelming and complicated. It is. But it can be coded efficiently, and the PER encoding method is comparable (in littleness of overhead) to most binary protocols. And... with ASN.1, since it's binary, you can send binary data much more efficiently (about 30% more efficient if you used base64 for the ascii encodings).

When most of the traffic are position updates, a binary protocol is much better. In your server, that is not the case. Or... if it is... it's not as efficient as it could be.
Oíd mortales, el grito sagrado...
Call me "Menes, lord of Cats"
Wing Commander Universe
Wisq
ISO Party Member
ISO Party Member
Posts: 453
Joined: Sat Jul 30, 2005 10:21 am

Post by Wisq »

As has been mentioned, there's two kinds of overhead at work here. There's network overhead (bytes versus amount of useful info), and that is indeed a real concern, and I'm glad it's being so thoroughly discussed.

The other kind, however, is processing overhead. Independent of the network speed between client and server, each side still has to understand what the other is saying.

Even the most rudimentary means of parsing a string-based protocol to extract the meaningful info is going to take more processing than a well-designed binary one. Essentially, you have to handle every single bit of the incoming packet, convert that to an internal structure, and process from there. A binary protocol (again, a well-designed one) can essentially skip that step; data can be directly extracted from the packet on an as-needed basis.

Like the network overhead, this is a tiny cost initially, but grows with traffic volume.

There are ways to make binary protocols more network- and processor-terse. Look at IP -- it has all the basic fields for an IP packet, then a field with the payload type, and then a payload. A specialised parser takes over, such that all the space needed for TCP or IPSEC is not wasted on UDP or ICMP. This saves both bytes sent and time spent processing.

AFAIK, the only way to make an ASCII protocol easier to parse is to always expect certain data tidbits at certain indexes... and then you've completely defeated the purpose of it.

If the network overhead on the binary packets is large enough to be considered a problem by some -- as I seem to understand from reading the last few pages of this thread -- then the protocol obviously needs fixing. But even with bit-packing, I would sincerely doubt that an ASCII protocol (particularly an XMLish one, even if no XML parser is used) could ever be as efficient (network and processing both) as a binary one.
jackS
Minister of Information
Minister of Information
Posts: 1895
Joined: Fri Jan 31, 2003 9:40 pm
Location: The land of tenure (and diaper changes)

Post by jackS »

looking over mmorpg.cpp, there appears to be currently no capacity to handle embedded binary data due to the use of newline as the delimiter.
Post Reply