Quick Status Update
<i>This post was originally made by <b>skyjake</b> on the dengDevs blog. It was posted under the categories: Engine, Games, Releases, Version 1.9.</i>
My personal goal was to get something released around Christmas, as has been the tradition in past years. However, due to time constraints this was not possible. Also, the goals of the 1.9 release were redefined in October/November, adding to the workload. Now, one might ask why was this done? The major reason is that the codebase as a whole needs to be refined so that it is more future-proof and a better foundation for 3rd party development. As redundancies between the games are reduced and more functionality is moved into the engine, it will become easier to fix bugs and maintain the code. Once the planned changes such as engine-side map data have been implemented, it should be much easier to improve netgames and fix the bugs currently remaining.
And of course, since we are talking about an increment to the minor version number, significant changes are called for.
At the moment I am working with Dani to update the game-side code to conform to the new requirement of having to access map data such as sectors and lines through a set of engine functions specifically reserved for this purpose. This is called the Doomsday Map Update API (DMU). It has several benefits. The engine will now learn about each change to map data as the change occurs. Hitherto games have been able to make changes at any time, and the engine has had to poll through all the data and check if any of it has changed compared to the previous known state. This is quite inefficient and complicated. The Doomsday netgame server currently works using this kind of polling, and benchmarks show it to be quite a performance hog. Even the renderer needs to rely on polling in certain situations. Another benefit of DMU is that it reduces the games' dependency on the engine. Specifically, DMU narrows the dependencies more closely to the actual Doomsday Public API. Since the beginning of the project, games have had direct access to certain data arrays maintained by the engine. If for some reason the arrays changed the games would be automatically broken. Not a great thing for maintaining compatibility between releases. Thinking forward, we want to be in a situation where we can replace the entire engine while providing the same API to the games. That way the game logic can be transitioned to a next-generation engine with a minimum amount of work and high level of accuracy.
Now that DMU has been added, all map data is internal to the engine and not directly visible to the games. This means quite a bit of code needs to be updated, since as you would expect, map data is accessed in numerous places in the games. The amount of work needed for refactoring the games is the largest drawback.
The current status is that DMU API is mostly complete, although not all map data is yet accessible. I have been concentrating on updating jHexen to use DMU. This work is about halfway complete. Dani is working on updating jDoom and jHeretic.
I don't dare make any predictions on release dates. One thing is for certain, though. We are in for an extended beta testing period, because of the magnitude of changes done for DMU, and other various changes (which Dani will probably tell about at some point). It is entirely possible that we have been introducing a host of new bugs as we have been refactoring the code.
My personal goal was to get something released around Christmas, as has been the tradition in past years. However, due to time constraints this was not possible. Also, the goals of the 1.9 release were redefined in October/November, adding to the workload. Now, one might ask why was this done? The major reason is that the codebase as a whole needs to be refined so that it is more future-proof and a better foundation for 3rd party development. As redundancies between the games are reduced and more functionality is moved into the engine, it will become easier to fix bugs and maintain the code. Once the planned changes such as engine-side map data have been implemented, it should be much easier to improve netgames and fix the bugs currently remaining.
And of course, since we are talking about an increment to the minor version number, significant changes are called for.
At the moment I am working with Dani to update the game-side code to conform to the new requirement of having to access map data such as sectors and lines through a set of engine functions specifically reserved for this purpose. This is called the Doomsday Map Update API (DMU). It has several benefits. The engine will now learn about each change to map data as the change occurs. Hitherto games have been able to make changes at any time, and the engine has had to poll through all the data and check if any of it has changed compared to the previous known state. This is quite inefficient and complicated. The Doomsday netgame server currently works using this kind of polling, and benchmarks show it to be quite a performance hog. Even the renderer needs to rely on polling in certain situations. Another benefit of DMU is that it reduces the games' dependency on the engine. Specifically, DMU narrows the dependencies more closely to the actual Doomsday Public API. Since the beginning of the project, games have had direct access to certain data arrays maintained by the engine. If for some reason the arrays changed the games would be automatically broken. Not a great thing for maintaining compatibility between releases. Thinking forward, we want to be in a situation where we can replace the entire engine while providing the same API to the games. That way the game logic can be transitioned to a next-generation engine with a minimum amount of work and high level of accuracy.
Now that DMU has been added, all map data is internal to the engine and not directly visible to the games. This means quite a bit of code needs to be updated, since as you would expect, map data is accessed in numerous places in the games. The amount of work needed for refactoring the games is the largest drawback.
The current status is that DMU API is mostly complete, although not all map data is yet accessible. I have been concentrating on updating jHexen to use DMU. This work is about halfway complete. Dani is working on updating jDoom and jHeretic.
I don't dare make any predictions on release dates. One thing is for certain, though. We are in for an extended beta testing period, because of the magnitude of changes done for DMU, and other various changes (which Dani will probably tell about at some point). It is entirely possible that we have been introducing a host of new bugs as we have been refactoring the code.
Comments
Over the course of time this API has changed quite drastically with different types of routines added at different times. There are probably still some remnants from the old 1.0 architecture and there are certainly some exported routines that would never/shouldn't be used now that we have DMU.
<ul>
<li>No exported pointers such as thinkercap, validcount, topslope, etc. such as currently in game_import. The addresses to these variables (if direct access even is necessary) must be queried using a function like DD_GetAddress().</li>
<li>The game exports struct has to go. Instead, there is a single function exported by the game that the engine can use to get the address of a function.</li>
<li>Since the engine can't be sure how many parameters each exported game function has (it may change over time), we should use a system where the engine first puts the arguments in a name-tagged array, which the game can then access using functions in the Doomsday Public API (named e.g. ExportArg(int name)). This way, we can freeze the binary API completely and even retain runtime compatibility. If the function interface has changed, for example a new parameter is provided by the engine, the old implementation in the game can still function since it accesses the arguments by their name tags.</li>
</ul>
Naturally this means some performance overhead, but it should be quite negligible.
Points 2/3: Definetly. That will greatly improve forward compatibility.
Yeah I doubt performance will be an issue considering how few exported game functions we actually require.
Though we are again adding to the 1.9.0 workload...
I guess before we attempt any of this other stuff we should try and get beta-4 released?
Re: Dummy objects - skyjake wrote:
True, but what I was suggesting is basically just a helper mechanism for allocating a block of extra memory for each dummy so that the game wouldn't have to do it manually. (The game isn't obligated to use the engine's dummy memory helper since it can allocate the extra memory itself and not call P_XDummy.)
---
If we go with the idea of Doomsday allocating the space for the xdummy then I think we should suggest that the games shouldn't mix and match between local and game side management.
What happens in the case of game-side objects such as things (the engine has no local objects of this type)?
Ultimately I would like to see Doomsday handling all custom map data object properties and objects themselves.
I worry that applying different logic to different types of data object will lead to confusion.
I think I'll make a new post for the dummy issue.
For example the engine-side player profile management, would be a rather large change to introduce midway through the 1.9.x series as similar to the DMU - would require a significant refactoring of the game-side code. Player profile management is an integral part of supporting multiple local players as it has very close ties to input events (inc bindings), network and has a significant impact on the game logic.
After that, I think we should indeed try to tidy up the engine/game split as much as possible before 1.9.0. These things need to be done at some point anyway, and now is a good time since changes are already in motion.
Let's aim to get the profiles, cfg management, and revised savegames working for Beta 5, after Beta 4 has been released.
Shouldn't we hide the engine side data structs so that object ptrs returned to the game (although opaque void*) don't actually link to the engines' objects? We don't want the games to be able to attempt to free() a sector_t for example.
When you think about it, from the game's point of view, opaque pointers are actually just an alternative indexing system for the objects. The reason the *p versions of the DMU functions exist is because we don't want to go and rewrite all the sector_t*, line_t* and other references in the games to work with indices instead of pointers.
What I think is necessary is to make the sector_t* and other opaque pointers more type safe. Currently you can use a sector_t pointer in a DMU_LINE call and it will go undetected in the engine. One way to fix this is to add a type member to the beginning of each internal data structure, so that the type can be checked without knowing the type of the object.
I was thinking the pointers we return to the games could be handles (ie sector_t**). That way we can be sure that they can't?
<blockquote>When you think about it, from the game's point of view, opaque pointers are actually just an alternative indexing system for the objects. The reason the *p versions of the DMU functions exist is because we don't want to go and rewrite all the sector_t*, line_t* and other references in the games to work with indices instead of pointers.</blockquote>
Sure, I understand why it's been done that way. Certainly changing all the exisiting logic to work with indices is impractical and unecessary.
<blockquote>What I think is necessary is to make the sector_t* and other opaque pointers more type safe. Currently you can use a sector_t pointer in a DMU_LINE call and it will go undetected in the engine. One way to fix this is to add a type member to the beginning of each internal data structure, so that the type can be checked without knowing the type of the object.</blockquote>
Yeah that sounds like a good solution. Do we have a need to have a method of versioning each structure also?
But sector_t** would be the address of a pointer to a sector_t. That doesn't change anything from the game's point of view, since it can still dereference it?
<blockquote>Do we have a need to have a method of versioning each structure also?</blockquote>
I don't think so, because these are runtime data structures internal to the engine. If the need ever arises, we can just define new type constants.
Even as (sector_t** opaqued to void*)? I suppose so but at least they couldn't free the engine's handle on the allocated memory inadvertantly, so the engine could still perform cleanup?
<pre>
void** handle = 0;
free(*handle);
</pre>
Do you have a situation in mind where the game could inadvertantly free the engine's data? I don't think it is something to worry about.
Probably just hyperbole