A New Core
<i>This post was originally made by <b>skyjake</b> on the dengDevs blog. It was posted under the category: Engine.</i>
I've been thinking about the best way to approach <a href="http://dengine.net/dew/index.php?title=Unified_networking">unified networking</a>. Central to the concept is clearer separation of the server and client sides. To make this more feasible, I believe we should convert most of the current engine into a shared library (libdeng) that can be linked against the executables and all the plugins.
I believe this is a good opportunity to implement a new core for Doomsday. It would handle the things that dd_(u/p)init.c and dd_loop.c currently do, plus the dynamic loading and unloading of plugins and other libraries. The unloading is important as well: I would like to make it possible to switch game plugins at runtime. Also, IPC between the server and client processes running on the same computer in single player games would be aided by the core.
To allow us to take things to the Next Level (so to speak), I would like to implement the new core in C++. The libdeng itself would remain a C library. Using C++ also makes it possible to borrow some of the relevant code from Hawthorn.
With the new core in place it would be much more elegant to launch a process with just the server side (dedicated server), or a process with the client UI (network player), or both processes simultaneously (single player).
I have to still figure out the details, but one possibility would be that the current Doomsday executable is split into three parts:
<ul>
<li>Server executable. Runs the new core, linked against libdeng. The server-specific code from the current Doomsday executable is moved here. Has no UI (basically a daemon process).
<li>Client executable. Runs the new core, linked against libdeng. Client-specific code from Doomsday is moved here. Graphical OpenGL UI. IPC with the local server executable, or over the network with a remote one.
<li>libdeng. Shared routines needed by all. C API.
</ul>
The core could be a shared library of its own (with a C++ interface).
I've been thinking about the best way to approach <a href="http://dengine.net/dew/index.php?title=Unified_networking">unified networking</a>. Central to the concept is clearer separation of the server and client sides. To make this more feasible, I believe we should convert most of the current engine into a shared library (libdeng) that can be linked against the executables and all the plugins.
I believe this is a good opportunity to implement a new core for Doomsday. It would handle the things that dd_(u/p)init.c and dd_loop.c currently do, plus the dynamic loading and unloading of plugins and other libraries. The unloading is important as well: I would like to make it possible to switch game plugins at runtime. Also, IPC between the server and client processes running on the same computer in single player games would be aided by the core.
To allow us to take things to the Next Level (so to speak), I would like to implement the new core in C++. The libdeng itself would remain a C library. Using C++ also makes it possible to borrow some of the relevant code from Hawthorn.
With the new core in place it would be much more elegant to launch a process with just the server side (dedicated server), or a process with the client UI (network player), or both processes simultaneously (single player).
I have to still figure out the details, but one possibility would be that the current Doomsday executable is split into three parts:
<ul>
<li>Server executable. Runs the new core, linked against libdeng. The server-specific code from the current Doomsday executable is moved here. Has no UI (basically a daemon process).
<li>Client executable. Runs the new core, linked against libdeng. Client-specific code from Doomsday is moved here. Graphical OpenGL UI. IPC with the local server executable, or over the network with a remote one.
<li>libdeng. Shared routines needed by all. C API.
</ul>
The core could be a shared library of its own (with a C++ interface).
Comments
<strong>EDIT</strong>: What are your thoughts on the high level architecture? Do we (ultimately) intend to have backend console functionality in the core?
What about low-level network communication? Should that be done via the core? e.g., server wishes to update a client; server process constructs the packet(s) but uses a core-level service API to send. Upon receiving a packet the core determines if it is for the client process, if so it is queued awaiting the client to retrieve.
My rational for doing network communication in the core is to (eventually) eliminate the need for a local server process entirely, when running a pure client connected to a remote server.
My overall thoughts about the core at this stage is that it would primarily provide services to both the server and client that would be better placed here than within libdeng.
When that is functional, and the two processes can talk to each other (through sockets), I will start segregating the server-side and client-side functionality from the common libdeng stuff, moving the code to the component where it belongs.
Since the core is a shared library, too, I see it as a natural place to start replacing the various subsystems of the engine with their next-generation (C++) counterparts. IMO we should discontinue the use of C for implementing anything completely new; C++ has many benefits that can aid in the development. Plus personally I prefer writing C++ to C. However, since libdeng should remain a C library (since the plugins remain implemented in C) I don't think we should use C++ in it, though (unless there are very good arguments for doing that).
<blockquote>Do we (ultimately) intend to have backend console functionality in the core?</blockquote> You mean like the current dedicated server text console? I think the server process should be just a UI-less background process. We can have a separate text mode console app that communicates with the background process if we want interactive control over it (probably we do).
<blockquote>What about low-level network communication? Should that be done via the core?</blockquote> Absolutely. My plan includes new network communication services in the core, providing a secure and robust way to define a packet structure, and serialize/deserialize them in network byte order.
This will most likely mean that libdeng will be using functionality provided by the core library (libcore) until the affected subsystems are replaced by their next-gen implementation (and moved to the libcore).
So, as the overall plan, I envision the legacy code in libdeng being replaced/moved part by part to the new libcore.
<blockquote>My rational for doing network communication in the core is to (eventually) eliminate the need for a local server process entirely, when running a pure client connected to a remote server.</blockquote> Yes, ultimately the client process will be free of anything related to server functionality. That's the point of separating the server and the client processes (at the source level, too).
Let's look at it this way. In the new architecture (of which I should draw a diagram), libdeng and libcore are pretty much on the same level. libdeng contains the current implementation (also of the console backend), while libcore contains redesigned subsystems. I see the console backend as an extremely important subsystem, but the current implementation is not up to par. There will definitely be a console backend in the libcore, but it will be quite radically revised and reimplemented in C++.
There's plenty of things in the current implementation that should be combined to achieve a better design. For example, consider the console backend, InFine, and the DEDs. To me the logical way forward is to combine these into one, more powerful scripting engine that can handle interactive commands, UI scripting/functionality, and definitions. This is what the new "console backend" is about in the libcore. (Also, it will elegantly allow us to implement the <a href="http://dengine.net/dew/index.php?title=Definition_parser_with_extensions" rel="nofollow">2nd gen DED system</a>.)
Other subsystems I've been considering are the virtual file directory, the model file hash, and the resource locator. These could be combined into one "file tree" in libcore that maintains both real and virtual files and knows how to efficiently locate specific type of resources.
There's probably plenty of other functionality that could be similarly unified into better-designed, larger and more capable subsystems.
However, this is quite a long-term plan. The first step is to just use the subsystems that we currently have and keep them in libdeng. Over time subsystems will be revised/unified/updated and moved to the libcore. Gradually, libdeng will be transformed into a legacy role where it will basically act as a C wrapper API for the libcore subsystems, available for plugins to use. In the distant future, plugins would be also able to link with the libcore and use the C++ functionality directly.
PS. Now that I think about it, libcore is probably not a very good name for the new library, considering what it's about. I'll need to come up with a new name...
I agree that the name libcore is not ideal as it was partly the name that was throwing my perception of its role.