On the Road Again (to Beta 7)
<i>This post was originally made by <b>skyjake</b> on the dengDevs blog. It was posted under the categories: Beta 7, Engine, Games.</i>
Here's an update on how the work towards Beta 7 is progressing, and some thoughts on the next steps.
The <a href="http://www.dengine.net/blog/?p=280">previous post on the subject</a> listed a number of bullet points with things to do. Below, I comment on how these have turned out.
<blockquote>Discard the current libdeng <tt>cl_*.c</tt> and <tt>sv_*.c</tt>. Not much there to write home about. Particular offenders are <tt>cl_player.c</tt> and <tt>cl_mobj.c</tt>, which never really worked that great and duplicate gameside logic (insufficiently).</blockquote>
This has been done. The contents of the old client and server implementation in libdeng has been #ifdef'd out.
<blockquote>Also discard the current low-level network communications code that sends and receives packets over TCP/UDP.</blockquote>
This has also been done.
<blockquote>Start from the basics. Using new libdeng2 classes, set up the data link between the server and client. I'm planning on using TCP streams only, initiatiated upon the client opening a TCP connection via the port the server is listening on.</blockquote>
Also done. libdeng2 now has a fully functional network subsystem. The server is able to listen and accept incoming TCP connections, and the client is able to connect to the server. dengcl starts a local instance of the server as a background process.
The concept of "sessions" was introduced: a Session is an on-going game session on the server, containing the game world and one or more maps. With single-player games, the idea is that when the player starts a new game, the client will tell the local dengsv to start a new session with the desired parameters. The player then joins the session as the only user.
The implementation is able to synchronize the initial state of all users and the world between all users of the session. However, the libdeng2 user/world state currently does not yet have anything to do with the libdeng map data and mobjs, as none of the libdeng data structures have yet been connected or migrated to libdeng2.
<blockquote>Redo the communications protocol and the way network packets are created and (de)serialized.</blockquote>
Also done. Interestingly, the communication protocol uses the same basic objects as the scripting subsystem. These are called records, variables, and values. The protocol simply serializes these data structures and sends them over the network stream. The benefit here is that scripts will be able to process network messages with ease. Other benefits include flexibility and robustness when faced with malformed data.
Records are simple data structures that contain a set of variables and a set of subrecords. Records can be nested inside other records up to any arbitrary depth. Variables are libdeng2's replacement for cvars. A variable can have any kind of value. Currently we have quite a nice set of value types: Number, Text, Array, Dictionary, Block, Record (plus None and Ref for scripting).
The dot notation familiar from just about every programming language is used when accessing variables in subrecords.
It is worth mentioning that the scripting subsystem is already present in libdeng2 with most major functionality implemented. I see it as a crucial piece in the puzzle -- it will be in a central role in the interactive command shell (i.e, the "console"), game scripting, UI logic, briefing/finale animations, definitions, and configuration. The implementation has been written from scratch by me and originates mostly from the work I've been doing on Hawthorn over the past five years. The language closely resembles Python, although the syntax has been relaxed a bit, plus there's a hint of Ruby. It is not an object-oriented language, though — there are no classes or derivation. If you're interested in what it can do and what it looks like at the moment, check out the <a href="http://deng.git.sourceforge.net/git/gitweb.cgi?p=deng;a=blob;f=doomsday/tests/script/kitchen_sink.de;h=28d9f945ed48b65edbe14dae5aface2e6d7b076b;hb=new-order-phase4"><tt>kitchen_sink.de</tt> test script</a>.
<blockquote>The principle of how things work is that the game world exists on the server, and the clients mirror copies of it. The clients are using local game logic for moving the players with zero latency. Impulses such as attacking and opening doors are signaled to the server, where they get performed in the real game world. This causes changes to occur in the map data, all of which can be recorded as a set of DMU operations. These operations are prioritized based on distance to clients and sent over the data links to them. Mobjs need to be tracked separately (unless we do a DMU-like interface for them as well -- which might not be a bad idea). On clientside, the DMU operations are applied to the client's copy of the world. On clientside, mobjs are there only for appearances and basic collision testing. On clientside, the only mobj controlled by the game logic is the player. On serverside, mobjs are controlled by the game logic as usual. Other events such as sounds, chat messages, state changes (e.g., inventory updates) are handled with gameside messages.</blockquote>
Most of this is not done yet. The current hurdle is how to get the map data and objects under libdeng2's control in a sensible way: one that doesn't cause too much breakage (and introduce new bugs), but in a way that doesn't waste time and effort.
I'm planning on concentrating on the handling of objects next. Currently they are omitted from DMU, so it is fertile ground for some new solutions. The ultimate goal is that the objects are under the ownership and control of libdeng2's de::Map, but the existing game logic can still access them like it does now. This probably requires switching from C to C++ compilation for the game plugins, but that is unavoidable in the long run anyway. I am hoping to do this with the minimum amount of changes in game plugin code.
<blockquote>Savegames are done on the server, as it has all the "official" information about the game. I'm considering that the savegame data could be copied to clientside as well, so that the game could be resumed if any of the involved players start hosting a game. (Since the savegame system is identical for local singleplayer games and networked games, this should do nicely.) At least the client's 3D view at the time of the save will need to be included in the clientside copy of the savegame. As to the format, it will be a ZIP directory containing a .jpg for the screen capture, and a separate file for each binary array of data, e.g., a file called "sectors" for the sector data, etc.</blockquote>
While the details of saving games haven't yet been clarified, the mechanism for saving and restoring world and user state already exists: it is the same one that sessions use. The open questions here are how the server and client will cooperate when saving the game, where the files go, and what is the exact layout of the archive containing the data. (libdeng2 already has the ability to create and modify ZIP archives.) None of these seem like big challenges, though.
<blockquote>Finally, demo recording will work as before by simulating the incoming data stream from a server, accompanied with a record of the local clientside camera movements. The demo file will be a ZIP directory with some metadata files included in addition to the serialized packet stream. For instance, a .jpg could be included in the demo file to act as an icon in the GUI.</blockquote>
The plan regarding demos is unchanged. No work has yet been done regarding demos.
So, my next step is to start looking very closely at how the objects should be managed in libdeng2, and how the data will be shared in sessions so that the clients will be up to date on where the objects are and what they're doing.
My vacation is now over and going forward I have significantly less time for the project. This inevitably means that progress will be somewhat slower. Although, I have to say that now that I have a clear(er) vision of where I want to go and how things should be, finding the motivation is much easier. (Here's hoping that work doesn't suck all energy out of me during the fall.)
Here's an update on how the work towards Beta 7 is progressing, and some thoughts on the next steps.
The <a href="http://www.dengine.net/blog/?p=280">previous post on the subject</a> listed a number of bullet points with things to do. Below, I comment on how these have turned out.
<blockquote>Discard the current libdeng <tt>cl_*.c</tt> and <tt>sv_*.c</tt>. Not much there to write home about. Particular offenders are <tt>cl_player.c</tt> and <tt>cl_mobj.c</tt>, which never really worked that great and duplicate gameside logic (insufficiently).</blockquote>
This has been done. The contents of the old client and server implementation in libdeng has been #ifdef'd out.
<blockquote>Also discard the current low-level network communications code that sends and receives packets over TCP/UDP.</blockquote>
This has also been done.
<blockquote>Start from the basics. Using new libdeng2 classes, set up the data link between the server and client. I'm planning on using TCP streams only, initiatiated upon the client opening a TCP connection via the port the server is listening on.</blockquote>
Also done. libdeng2 now has a fully functional network subsystem. The server is able to listen and accept incoming TCP connections, and the client is able to connect to the server. dengcl starts a local instance of the server as a background process.
The concept of "sessions" was introduced: a Session is an on-going game session on the server, containing the game world and one or more maps. With single-player games, the idea is that when the player starts a new game, the client will tell the local dengsv to start a new session with the desired parameters. The player then joins the session as the only user.
The implementation is able to synchronize the initial state of all users and the world between all users of the session. However, the libdeng2 user/world state currently does not yet have anything to do with the libdeng map data and mobjs, as none of the libdeng data structures have yet been connected or migrated to libdeng2.
<blockquote>Redo the communications protocol and the way network packets are created and (de)serialized.</blockquote>
Also done. Interestingly, the communication protocol uses the same basic objects as the scripting subsystem. These are called records, variables, and values. The protocol simply serializes these data structures and sends them over the network stream. The benefit here is that scripts will be able to process network messages with ease. Other benefits include flexibility and robustness when faced with malformed data.
Records are simple data structures that contain a set of variables and a set of subrecords. Records can be nested inside other records up to any arbitrary depth. Variables are libdeng2's replacement for cvars. A variable can have any kind of value. Currently we have quite a nice set of value types: Number, Text, Array, Dictionary, Block, Record (plus None and Ref for scripting).
The dot notation familiar from just about every programming language is used when accessing variables in subrecords.
It is worth mentioning that the scripting subsystem is already present in libdeng2 with most major functionality implemented. I see it as a crucial piece in the puzzle -- it will be in a central role in the interactive command shell (i.e, the "console"), game scripting, UI logic, briefing/finale animations, definitions, and configuration. The implementation has been written from scratch by me and originates mostly from the work I've been doing on Hawthorn over the past five years. The language closely resembles Python, although the syntax has been relaxed a bit, plus there's a hint of Ruby. It is not an object-oriented language, though — there are no classes or derivation. If you're interested in what it can do and what it looks like at the moment, check out the <a href="http://deng.git.sourceforge.net/git/gitweb.cgi?p=deng;a=blob;f=doomsday/tests/script/kitchen_sink.de;h=28d9f945ed48b65edbe14dae5aface2e6d7b076b;hb=new-order-phase4"><tt>kitchen_sink.de</tt> test script</a>.
<blockquote>The principle of how things work is that the game world exists on the server, and the clients mirror copies of it. The clients are using local game logic for moving the players with zero latency. Impulses such as attacking and opening doors are signaled to the server, where they get performed in the real game world. This causes changes to occur in the map data, all of which can be recorded as a set of DMU operations. These operations are prioritized based on distance to clients and sent over the data links to them. Mobjs need to be tracked separately (unless we do a DMU-like interface for them as well -- which might not be a bad idea). On clientside, the DMU operations are applied to the client's copy of the world. On clientside, mobjs are there only for appearances and basic collision testing. On clientside, the only mobj controlled by the game logic is the player. On serverside, mobjs are controlled by the game logic as usual. Other events such as sounds, chat messages, state changes (e.g., inventory updates) are handled with gameside messages.</blockquote>
Most of this is not done yet. The current hurdle is how to get the map data and objects under libdeng2's control in a sensible way: one that doesn't cause too much breakage (and introduce new bugs), but in a way that doesn't waste time and effort.
I'm planning on concentrating on the handling of objects next. Currently they are omitted from DMU, so it is fertile ground for some new solutions. The ultimate goal is that the objects are under the ownership and control of libdeng2's de::Map, but the existing game logic can still access them like it does now. This probably requires switching from C to C++ compilation for the game plugins, but that is unavoidable in the long run anyway. I am hoping to do this with the minimum amount of changes in game plugin code.
<blockquote>Savegames are done on the server, as it has all the "official" information about the game. I'm considering that the savegame data could be copied to clientside as well, so that the game could be resumed if any of the involved players start hosting a game. (Since the savegame system is identical for local singleplayer games and networked games, this should do nicely.) At least the client's 3D view at the time of the save will need to be included in the clientside copy of the savegame. As to the format, it will be a ZIP directory containing a .jpg for the screen capture, and a separate file for each binary array of data, e.g., a file called "sectors" for the sector data, etc.</blockquote>
While the details of saving games haven't yet been clarified, the mechanism for saving and restoring world and user state already exists: it is the same one that sessions use. The open questions here are how the server and client will cooperate when saving the game, where the files go, and what is the exact layout of the archive containing the data. (libdeng2 already has the ability to create and modify ZIP archives.) None of these seem like big challenges, though.
<blockquote>Finally, demo recording will work as before by simulating the incoming data stream from a server, accompanied with a record of the local clientside camera movements. The demo file will be a ZIP directory with some metadata files included in addition to the serialized packet stream. For instance, a .jpg could be included in the demo file to act as an icon in the GUI.</blockquote>
The plan regarding demos is unchanged. No work has yet been done regarding demos.
So, my next step is to start looking very closely at how the objects should be managed in libdeng2, and how the data will be shared in sessions so that the clients will be up to date on where the objects are and what they're doing.
My vacation is now over and going forward I have significantly less time for the project. This inevitably means that progress will be somewhat slower. Although, I have to say that now that I have a clear(er) vision of where I want to go and how things should be, finding the motivation is much easier. (Here's hoping that work doesn't suck all energy out of me during the fall.)
Comments
In order to carry on working on the map data representation I would like to shift my work over to the new-order-phase4 branch so that I can begin by re-implementing the BSP builder algorithm. Naturally, this work will necessarily involve breaking map loading.
Also there has been a sizeable amount of work done to quash bugs in the beta6 branch since the new-order branch split from the trunk.
So, how do we progress forward from here? Should we take the opportunity to reconcile the new-order-phase4 branch with my private merge of the beta6 and beta6-with-mapcache branches? Once merged I could then branch the result for the work on the map data.
Maybe this is what should be done:
<ol>
<li>In new-order-phase4, I will finish the thinker/object refactoring and merge the changes to the trunk. This ends phase4. All games (apart from D64?) will be fully functional in the trunk, using the new thinker/objects. Savegame functionality will be disabled (until replaced with the new serialization mechanism).
<li>The beta6-with-mapcache + beta6-head changes (i.e., your new map code and all changes since the trunk swap) are merged to the trunk.
<li>Phase5 begins with the new map data code and refactored thinker/objects.
</ol>
As for D64; its fine by me if you want to leave updating this to the new thinker/object handling for now as currently it is not possible to even load a map.