Bringing Build Engine games to Doomsday
Hello Doomsday team,
Rather than spamming the other thread with off-topic discussion I decided to create a separate thread. Basically, I would like to see the Build Engine games, most notably Duke Nukem 3D and Shadow Warrior, supported in the Doomsday engine. Primarily Shadow Warrior, because that one has no proper port at all, but Duke Nukem 3D would be a first stepping stone because it lacks a few features found in Shadow Warrior (transparent water and voxels for instance).
As I understood there is no interest from the developer team to go for this issue in the foreseeable future, so I thought I could try my hand at it and see how far I can get. From what I gathered the Build engine is very much like a technical superset of the Doom engine, both work with 2D sectors for level design and sprites for world objects. What Build adds is sloped surfaces (different heights within a single sector), room-over-room (by "teleporting" the player to another sector), under water parts (similar to room over room), destructible terrain (achieved by altering sector data on the fly during runtime) and being able to look and walk into sectors that aren't adjacent on the map (this creates the seamless room-over-room experience).
The first step would be to allow Doomsday to load GRP files (like Doom WADs); that should be simple enough, GRP files consist just of a list of files and then the files themselves glued together in sequence, there is no compression. The next step would be loading the assets and expanding Doomsday's capabilities the handle the new features (slopes and so on). Unless of course Doomsday can already do that, I don't know, otherwise this would be the hard part. Once we can walk around in the levels it's time to code the actual game mechanics (would this be libduke then?).
What level of programming expertise would I need for these tasks and does it even make sense to start now or should I wait until Doomsday gets closer to version 2.0? If this were successful we could even think about supporting Blood, which would be awesome.
Rather than spamming the other thread with off-topic discussion I decided to create a separate thread. Basically, I would like to see the Build Engine games, most notably Duke Nukem 3D and Shadow Warrior, supported in the Doomsday engine. Primarily Shadow Warrior, because that one has no proper port at all, but Duke Nukem 3D would be a first stepping stone because it lacks a few features found in Shadow Warrior (transparent water and voxels for instance).
As I understood there is no interest from the developer team to go for this issue in the foreseeable future, so I thought I could try my hand at it and see how far I can get. From what I gathered the Build engine is very much like a technical superset of the Doom engine, both work with 2D sectors for level design and sprites for world objects. What Build adds is sloped surfaces (different heights within a single sector), room-over-room (by "teleporting" the player to another sector), under water parts (similar to room over room), destructible terrain (achieved by altering sector data on the fly during runtime) and being able to look and walk into sectors that aren't adjacent on the map (this creates the seamless room-over-room experience).
The first step would be to allow Doomsday to load GRP files (like Doom WADs); that should be simple enough, GRP files consist just of a list of files and then the files themselves glued together in sequence, there is no compression. The next step would be loading the assets and expanding Doomsday's capabilities the handle the new features (slopes and so on). Unless of course Doomsday can already do that, I don't know, otherwise this would be the hard part. Once we can walk around in the levels it's time to code the actual game mechanics (would this be libduke then?).
What level of programming expertise would I need for these tasks and does it even make sense to start now or should I wait until Doomsday gets closer to version 2.0? If this were successful we could even think about supporting Blood, which would be awesome.
Comments
However, it will be quite challenging due to the issues you've mentioned. Doomsday indeed lacks support for slopes and portal-based rendering. However, this would provide an excellent opportunity to enhance Doomsday's renderer.
- As you guessed, a "libduke" would be needed for all the game logic, physics, etc. The existing Duke sources would have to be heavily modified, though, to work with Doomsday's runtime APIs and renderer instead of its own.
- Doomsday handles map data in its own internal format, and we import the Doom/Hexen maps into it using the wadmapconverter plugin. This is DaniJ's area of expertise. Support for importing Duke format maps would have to go either into wadmapconverter or another map conversion plugin.
- The internal map data would have to be extended to handle the more advanced concepts of the Build engine, such as slopes. However, extending the internal map format long been the plan anyway — we've isolated it from the game plugins, meaning that the internal map data structures can be radically altered without breaking any of the games, as long as the map APIs stay the same toward the plugins.
- There is one specific exception to internal map data that might provide problematic for now: mobjs are not fully internal to the engine, but are instead treated as POD data structures with a predefined set of common fields in the beginning. I don't know how Duke handles objects, but it might clash with this approach. We've been long planning to handle objects internally as C++ objects and provide some sort of API to manipulate them, so that the "raw data" is not exposed and game plugins could be freed to handle objects in any way they please, as long as they inform the engine about their common properties (like position, speed, etc.)
- And of course, the world renderer itself would need improving (for slopes, etc.). We are planning to rewrite most of it, however it doesn't do much good to wait until the entire renderer has been updated, as it might take years. The wisest course of action might be to start modifying the current renderer, and incorporate the added features in such as a way that they can be integrated into the new renderer (i.e., well decoupled and written using C++/Doomsday 2 APIs).
This would be a huge undertaking, but at least I could provide assistence as needed.In practice, I would like to see this developed at Github (or some other public git repository) as a separate branch that periodically merges to the main Doomsday master branch with new stuff as it gets implemented.
I am working on OS X 10.9 (Mavericks), so what do I need to get started. According to the Getting Started page I need XCode, which I already have. The Compiling and Running page list a whole bunch of other dependancies, do I need all of those as well? Can I use XCode as my IDE? That's the one I'm already used to.
Yes, I expected that. Honestly I don't really want to actually port the source itself, just make the game compatible with Doomsday and make it as faithful as it makes sense. If people want a 1:1 faithful port they can use DOSBox or Chocolate Duke Nukem 3D. In fact, I think using that port as a reference instead of the original code would be a better idea because it's more readable. Another plugin (grpmapconverter), the Duke 3D level format has (to my knowledge) nothing to do with the Doom level format, so it would just be bloating the code.
I have some familiarity with the Duke 3D map format and there are indeed some fundamental design differences that will require some extensive changes to Doomsday's internal map data format. However, I plan to continue work on expanding the map data format for Doomsday 1.15
You are right to begin this task by supporting the GRP container format. We can introduce data format interpreters as and when they become viable. Accurately interpreting the map data is however impossible at present. For now I would suggest you focus on the myriad graphic and sound formats.
However, I still would recommend using Qt Creator as the IDE — I've tried setting up an Xcode project like this a few times but the builds never worked quite as fluently as with Qt Creator. I also recall Xcode having some issues indexing all the symbols in the project, as some of the sources are currently being compiled several times but with different preprocessor definitions.
About the other dependencies: Qt is of course mandatory (4.8.5 is what we're using currently), SDL is used only for joystick input (but is required for compilation). SDL_mixer is optional (if using "CONFIG += deng_nosdlmixer"). I recommend getting the FMOD Ex SDK for audio.
The FluidSynth audio plugin (for playing midi music using SF2 soundfonts) needs Homebrew, because you need to install GLib 2.0. Homebrew can also provide ccache to speed up your compilation, Doomsday is quite a large project and can take a while to compile from scratch.
How would I go about testing whether the graphics and sounds are properly read?
However the first step is of course supporting the GRP container itself. In practice this will mean a parallel component to the other ZIP/WAD containers. I should point out that we are currently transitioning to a new "FS2" file system implemented in libdeng2. Presently this means that in order for such resources to be available to the majority of the engine it is necessary to implement the GRP container in the old "FS1" file system.
Without wishing to discourage you, I wonder if extending support to such a significantly different game is perhaps a little too soon, considering the current state of the architecture. While some parts are new and clean, with some decent documentation. The older parts have some fairly quirky behavior and the design of which isn't up to the same standard.
We can see about the rest as we go along, if certain aspects have to be put on hold then so be it. But first I'll need to get everything set up on my side, then we'll see. As I said, I am in no hurry to get this done soon.
The project has been configured for the SDL 1 frameworks (see dep_sdl.pri).
I would advise against relying on FS1 at all. As DaniJ says we're migrating to FS2 and other libdeng2 based classes, so any new code that handles GRP containers should be built around the latter. The FS1 based implementation heavily relies on conventions and quirks unique to Doom WADs; mixing any of it with GRP container logic does not seem like a good idea.
We will soon need a way to bypass or otherwise get rid of the FS1 classes so that development of the resource subsystem can proceed unhindered and — quite crucially — outside of the client app, in a library of its own.
I think the wisest way forward is to start collecting smaller-scope classes, like a GRP container for FS2, and image/sound importers for the Duke formats, and keep them as decoupled as possible from any existing Doomsday functionality except the SDK libraries (libdeng2, libgui, libappfw, libshell). Initially these could be developed in test apps that just verify that the components are working, e.g., open a GRP file and save the images as .png for instance. Later on, when the client's internal architecture has been cleaned up, these can then be incorporated in the revised resource management system.
I've encountered this in the past a few times. I got around it by creating a hard link to /usr/bin/qmake and then using that as the qmake binary.
However, now when I want to click on Build->Build Project "doomsday" I get an error "Unspecified SDK configuration" and general message "Project ERROR: Unspecified SDK configuration.". What does that mean? Also, do you guys have any code formatting rules I should stick to? (new lines, spaces, indentation, tab VS space,...)
Yes, we do follow our own code formatting rules. The source files of libdeng2 are a pretty good style reference. Indentation is 4 spaces, never tabs.
One rather non-standard convention is that we put const after the typename, for instance "String const &name" instead of "const String& name". & and * therefore are always attached to the identifier:
We of course would prefer that you try to match our style, although changing it with a reformatting utility is also pretty straightforward.
[EDIT: maybe it would be best if you copy-pasted you config_user.pri or PMed it to me]
I could never understand the reasoning behind this. I always use tabs for indentation and spaces for alignment, but I will stick to your rules.
It's mostly for simplicity and consistency. This basically sums it up: http://programmers.stackexchange.com/a/657
A good IDE like Qt Creator of course makes this a non-issue, though.
You shouldn't need pkg-config on Mac unless you are building the FluidSynth plugin.
Note that the general messages tab is somewhat useless as it also prints messages that occur during the background re-evaluation of the .pro/.pri files for the purposes of the IDE knowing which files are part of the project, and during that processing it doesn't seem to respect the CONFIG variable. This means all kinds of errors might get printed.
(BTW, I updated my config_user.pri, see above, to look more like yours; I hadn't realised the code field was scrollable)
You could also try rerunning qmake (via Build -> Run qmake).
Everything that is accessed from Homebrew or another external package system is configured via pkg-config. So as long as pkg-config is in the path you should be fine.
If not, try rebuilding libdeng2 and see if it failed for some reason.
You may have to specify "unsupported/macx-clang" as the Qt mkspec. This can also be given to qmake as a command line option "-spec unsupported/macx-clang".
So I commented away the part about fluidsynth:
Commenting away ccache: http://pastebin.com/QageG9Dz
I always trashed the build-doomsday-Desktop-Debug folder after each failed attempt.
You should look for and install pkg-config, GLib 2.0 development files, and ccache from Macports (or Homebrew), if you wish to compile FluidSynth and use caching.
http://pastebin.com/zRmnAE8t
BTW, do we have a proper documentation of the GRP format somewhere? I've started digging through the source, but if there was already something properly written that would be better.
Note that the "-appdir . -vdmap .. }data -bd Doomsday.app/Contents/Resources" options are mandatory, as they instruct the engine where the resources are located in the developer directory layout.
The "ccache" package has to be installed from Macports/Homebrew, if you wish to use it.
I've checked our post-build script and it turns out that for FluidSynth, Homebrew actually is required as it assumes the libraries can be found under /usr/local/Cellar and copies them from there to the Doomsday.app bundle. So you might wish to not build the FluidSynth plugin for now.
What's the issue with FMOD?
Also, there is no Snowberry, it takes me straight to the homes screen (ring zero?), is that intended?
I do have ccache installed via MacPorts, but it won't recognise it, might that be the same problem as with FluidSynth and glib2? The problem with FMOD is the following: I downloaded the FMOD Programmer's API from the website and there is no "lib" folder inside "api" it, just "lowlevel" and "studio". It builds fine without FMOD, so I'm not really concerned about it, I just ant the compiler to get working. Having ccache would be nice though, compilation really takes a long time.
Perhaps you downloaded the wrong one? Note that you need the FMOD Ex Programmer's API, under FMOD Ex, not the Studio one.
Having this as part of the Doxygen documentation would be fine, although we also have the wiki for information like this. Markdown is also ok, as we're using Doxygen 1.8.6.
If I'm not posting often here it doesn't mean that I have lost interest already, I'm currently juggling my time between being part-time at the hospital and renovating, so I can't devot much time to this at the moment. I'm definitely interested in continuing though.