Source Directory Structure Proposal
<i>This post was originally made by <b>skyjake</b> on the dengDevs blog. It was posted under the categories: Engine, Game Resource Packs, Games.</i>
Now that we are using Subversion, it's feasible to think about such things as bringing some consistency into the Doomsday source directory structure. I think the following new principles should form the basis of a new directory structure.
<ul>
<li>All file and directory names should be in lower case. Upper case letters should only be used in special cases (for example, Unix-style README, INSTALL files).</li>
<li>Instead of using file type as the root level structure (Defs, Src, Include) we should use the logical module (jhrp, common, engine, plugins).</li>
</ul>
The new structure would look something like this:
<pre>
build/ (all build/project-related files, incl. built binaries)
win32/ (windows projects and scripts)
mac/ (Xcode project)
unix/ (Unix/Linux build scripts)
engine/
scripts/ (engine scripts: makedmt.py)
src/
include/ (engine's internal files)
api/ (doomsday.h, dd_share.h, doomsday.def, etc.)
defs/ (engine's definition files)
plugins/
jdoom/ (the games should be just plugins like everything else)
src/
include/
defs/
jheretic/
jhexen/
common/
src/
include/
defs/
dehread/
mapload/
d3d/
opengl/
packs/
scripts/ (resource pack scripts: "compile PK3s")
jdrp/
jhrp/
jxrp/
examples/ (examples like shiny surfaces for jHexen map01)
tests/ (assorted resource tests)
</pre>
What this would mean in practice:
<ul>
<li>All project files, build scripts, and makefiles will need to be revised. (Tedious, but doable.)</li>
<li>The modules are nicely separated: no more confusion with jHRP's definition files being under Defs/jHeretic.</li>
<li>The Doomsday Public API can be separated cleanly into its own directory.</li>
<li>More resource packs can be included in the official repository. (including data files, also thanks to SVN).</li>
<li>Games are considered appropriately to be plugins.</li>
<li>Game definition files are stored under the game's own source directory.</li>
<li>The runtime directory structure either has to mirror this new structure, or perhaps it would be a good idea to figure out a separate runtime structure optimal for runtime operations under a runtime/ root-level directory. It might look something like this:
<pre>
runtime/
common/ (common.cfg for settings shared by all; executed last?)
jdoom/ (jDoom's runtime directory)
bspcache/
demo/
savegame/
jheretic/
jhexen/
</pre>
When a binary installation is made, only the runtime directory is needed. (Also a bin/ directory for executables on win32. On the Mac, the executables are in the application bundle. In Unix they are in /usr/local/bin and /usr/local/lib or somesuch place.) Something different has to be done with the definitions, though. Maybe even "runtime/jdoom/defs" (virtually)?</li>
<li>Something needs to be done to existing resource packs. Maybe just make sure that the old directory locations are supported also.
</ul>
Comments? If we decide to do this, it would probably need to happen post-beta4, even post-1.9.0.
Now that we are using Subversion, it's feasible to think about such things as bringing some consistency into the Doomsday source directory structure. I think the following new principles should form the basis of a new directory structure.
<ul>
<li>All file and directory names should be in lower case. Upper case letters should only be used in special cases (for example, Unix-style README, INSTALL files).</li>
<li>Instead of using file type as the root level structure (Defs, Src, Include) we should use the logical module (jhrp, common, engine, plugins).</li>
</ul>
The new structure would look something like this:
<pre>
build/ (all build/project-related files, incl. built binaries)
win32/ (windows projects and scripts)
mac/ (Xcode project)
unix/ (Unix/Linux build scripts)
engine/
scripts/ (engine scripts: makedmt.py)
src/
include/ (engine's internal files)
api/ (doomsday.h, dd_share.h, doomsday.def, etc.)
defs/ (engine's definition files)
plugins/
jdoom/ (the games should be just plugins like everything else)
src/
include/
defs/
jheretic/
jhexen/
common/
src/
include/
defs/
dehread/
mapload/
d3d/
opengl/
packs/
scripts/ (resource pack scripts: "compile PK3s")
jdrp/
jhrp/
jxrp/
examples/ (examples like shiny surfaces for jHexen map01)
tests/ (assorted resource tests)
</pre>
What this would mean in practice:
<ul>
<li>All project files, build scripts, and makefiles will need to be revised. (Tedious, but doable.)</li>
<li>The modules are nicely separated: no more confusion with jHRP's definition files being under Defs/jHeretic.</li>
<li>The Doomsday Public API can be separated cleanly into its own directory.</li>
<li>More resource packs can be included in the official repository. (including data files, also thanks to SVN).</li>
<li>Games are considered appropriately to be plugins.</li>
<li>Game definition files are stored under the game's own source directory.</li>
<li>The runtime directory structure either has to mirror this new structure, or perhaps it would be a good idea to figure out a separate runtime structure optimal for runtime operations under a runtime/ root-level directory. It might look something like this:
<pre>
runtime/
common/ (common.cfg for settings shared by all; executed last?)
jdoom/ (jDoom's runtime directory)
bspcache/
demo/
savegame/
jheretic/
jhexen/
</pre>
When a binary installation is made, only the runtime directory is needed. (Also a bin/ directory for executables on win32. On the Mac, the executables are in the application bundle. In Unix they are in /usr/local/bin and /usr/local/lib or somesuch place.) Something different has to be done with the definitions, though. Maybe even "runtime/jdoom/defs" (virtually)?</li>
<li>Something needs to be done to existing resource packs. Maybe just make sure that the old directory locations are supported also.
</ul>
Comments? If we decide to do this, it would probably need to happen post-beta4, even post-1.9.0.
Comments
The installation structure may need some more thinking on, considering that most existing mods rely heavily on the current structure (at least virtually) (even down to "hardcoded in DED", Doomsday-base/root level paths) due to the limitations in Doomsday's virtual filesystem/zip reader of the time.
I think we'd better leave this stuff until after beta4. Though I WOULD prefer that if we are to make (huge) changes to the directory structure that we do so for 1.9.0
I know of at least three third party developers waiting in the wings to begin working on new game plugins once 1.9.0 is released. Two of these projects have already begun development using the 1.8.x code base and will have a huge job on their hands in updating their work to 1.9.0 anyway. Therefore I think it would be "less than helpful" for us to make these changes in 1.9.1 thus causing them another batch of substantial updating. I think it would be better to change the structure "now".
<blockquote>The runtime directory structure either has to mirror this new structure, or perhaps it would be a good idea to figure out a separate runtime structure optimal for runtime operations under a runtime/ root-level directory.</blockquote>
Personally I think we should devise a slightly different structure for installations rather than mirror that of the source (though it would be convenient for developers if the structure were the same).
"runtime/GAME/defs" folders - since they only contain definition files which, once read, are not needed during gameplay - we should be OK to deal with them purely via virtual paths.
The defs for each game could go into eg (DOOM.PK3) which would then live in runtime/GAME/data. If we allowed DED files to be read from runtime/GAME/data/Auto we could then get rid of the /defs folders completely in the installation directory (and thus remove the distinction between /defs and /data in the eyes of users, which they deem to be an arbitary limitation ("a definition IS data, so why not GAME/data/defs!?" - is a common complaint)).
The only problem is existing resource packs that are physically installed to the Doomsday root folder (those installed and used as self-contained archives are not a problem (since we can juggle the file paths virtually where necessary).
Personally - I'm of the opinion that we DONT support addons which are phyisically unpacked into the Doomsday base directory, as multiple data files (eg WAD(s) + n* textures, + n* MP3s etc) in the 1.9.x series (however they COULD be made to work using a -vdmap (perhaps Snowberry could help by unpacking the archive into a an .addon first?)). This isn't as drastic as it sounds, all of the "offical" j-packs have all since been updated to work as self-contained PK3 archives (other than jDRP which has jDoom.DED external to the archive for reasons I've previously explained) so we are only talking about a handful (to my knowledge) of VERY old mods (which probably don't work 100% in modern Doomsday as it stands anyway), since ZIP support was introduced in 1.7.10 - 90% of mods are now distributed in "run-from-zip" form other than DED-only mods.
I think that it would be a good idea for us to enforce usage of the supported container formats (ZIP, WAD and Lump Directory + -vdmap) onto mod authors.
Currently I think Doomsday allows them a bit too much freedom to "mess about" in the Doomsday base folder. The by-product of this freedom is that I have seen several extremely badly organised mods that:
<ul>
<li>conflict with others.</li>
<li>don't work under nix due to filename/case sensitivity issues.</li>
<li>"MUST" be physically installed due to the way they are distributed/organised.</li>
<li>change original DED files (such as jDoom's Objects.DED)</li>
<li>are a complete pain for users to uninstall ("which file is part of Doomsday and which is part of mod X?").</li>
</ul>
Not to mention the number of problems reported on the forums, which stem from the poor organisation of some of this variety of mod.
IMO now that we have -vdmap to allow creation of virtual paths AND the various container formats - there is no longer a need to support collections of seperate data files in the Doomsday base folder and instead, do so only virtually.
Can you think of a situation where this wouldn't work? (obviously there is nothing to stop people from installing stuff into the base directory and using -vdmap(s) though).
<blockquote>common.cfg for settings shared by all; executed last?</blockquote>
I would have thought that common settings should be executed first(?) That way it could act like the "Defaults" profile in Snowberry (bare in mind we'll have mutliple cfgs per game once profiles support is implemented (needed to manage multiple local player settings cleanly)).
Currently I install addons into the static data directory, and do not support any addon that relies on hardcoded paths. After trying to support them I've decided it's not worth it, so I agree with DaniJ in not supporting addons that don't use the .zip packages.
My concern is with the proposed runtime directory, I require the ablity to split off the static and dynamic data parts of this. Is this runtime directory proposed to hold the "static" data such as addons, and dynamic such as savegames ?
<blockquote>My concern is with the proposed runtime directory, I require the ablity to split off the static and dynamic data parts of this. Is this runtime directory proposed to hold the "static" data such as addons, and dynamic such as savegames ?</blockquote> The thing is, we have a lot of freedom thanks to the virtual path mappings. We need to think of this from several points of view:
<ul>
<li>The user sees PK3 files that are stored somewhere, e.g., Snowberry's addons folder or some other folder that is loaded automatically by Doomsday.
<li>The engine does not see the PK3/ZIP files, only the virtual files that are "unpacked" from the archives. The directory structure of these files is the one that the engine is actually using. <i>Only this structure is really relevant.</i> It may be partially or even fully virtual, but this is the directry structure that the engine relies on.
</ul>
The runtime directory structure would really only need to hold the dynamic data such as savegames and cvar config files (the files that Doomsday itself writes). This is user-specific stuff so it would belong in ~/.deng or "~/Library/Application Support/Doomsday Engine" on the Mac. A well-behaved Windows app would use "%HOME%\Application Data\Doomsday Engine" or something like that.
In this scenario, the launcher would worry about where the addons and resource packs are placed and where they are loaded from. The auto-loading addon folder could be under the runtime directory so the user can easily put files in there (without Admin rights).
<blockquote>I think we'd better leave this stuff until after beta4. Though I WOULD prefer that if we are to make (huge) changes to the directory structure that we do so for 1.9.0</blockquote> Definitely after beta4, or otherwise we'll see another 3-5 months delay.
So I propose that once beta4 has been released (before summer, I hope...), we'll get our hands dirty and carry out this directory structure revision. It should make things a lot easier down the road, especially if we design good (strict!) guidelines for addons.
<blockquote>Personally I think we should devise a slightly different structure for installations rather than mirror that of the source (though it would be convenient for developers if the structure were the same).</blockquote> Yes, I think that one of the major problems with the current setup is that it is intended both for source development and runtime usage, at the same time. It works, but as we've seen it can get a bit tricky.
In fact, when the comes to a directory called "runtime", there shouldn't even be one in an installation directory, since the installation directory itself is the "runtime" directory. For developers we can have a "runtime" directory as part of the source hierarchy, which is a copy of the actual installation layout. It would be used for running the game without doing a real install.
<blockquote>and thus remove the distinction between /defs and /data in the eyes of users, which they deem to be an arbitary limitation</blockquote> That's a good point.
<pre>
(physical user dir)/
doom2/
jdoom.cfg (config file)
savegame/
data/ (PK3s, defs, etc. are loaded automatically from here)
coolstuff.pk3
jdoom/
colors.cfg (Doomsday UI color scheme)
jheretic/
jhexen/
</pre>
The question is, what should this look from the engine's point of view? Where are resource files such as 3D models, texture bitmaps, and definition files actually located? One possibility would be:
<pre>
(runtime dir)/
jdoom/ (stuff common to all jDoom games)
models/ (virtual by default)
sergeant/
(data + defs for sergeant)
data/
doom2/ (game called "doom2")
textures/
flats/
jheretic/
jhexen/
</pre>
This would mean the PK3's internal structure would use paths with a prefix "jdoom/", if the pack can be used with all jDoom games (jdoom.pk3), or a prefix like "doom2/" if it is only for Doom 2. When Doomsday would load a resource, it would first check the game-specific location, e.g., "doom2/textures/", and if that fails, the generic "jdoom/textures/" directory.
3D models would be loaded from under "doom2/models/". The organization under that directory depends on the definition files, but could be like "(class)/(name)", e.g., "actors/sergeant/", or "item/bullets". Or maybe we should just flatten it out to "doom2/models/(name)", e.g., "doom2/models/sergeant/".
<blockquote>The only problem is existing resource packs that are physically installed to the Doomsday root folder</blockquote> I think the only way these can be supported is via Snowberry and some sort of automatic vdmap emulation. We can create Snowberry manifests that describe how old packs should be used (manifests are "standalone" addon metadata files).
<blockquote>I think that it would be a good idea for us to enforce usage of the supported container formats</blockquote> Definitely. PK3 or ZIP should be the official recommendation for addons. Another alternative is Snowberry bundle, but that ties the addon to Snowberry (unless we add Snowberry bundle support into Doomsday... could be tricky).
<blockquote>I would have thought that common settings should be executed first(?) That way it could act like the "Defaults" profile in Snowberry.</blockquote> You're right, a common.cfg needs to be loaded first. The problem is, though, that the game-specific .cfg includes all cvars, including those set in common.cfg. After one session the game-specific .cfg would override all the common settings. To make a common.cfg work we would need to only save the "changed" cvars of a game, a task that may be a bit difficult to pull of properly. Maybe we'll just need to forget about common.cfg...
We need to bare in mind the proposed player profiles in Doomsday 1.9.0 Should the player profiles used by Doomsday mirror system users (ie each system user gets a profile generated for them by default ) as well as allow for other non-system user profiles?
Ideally the cfg stuff needs to be seperated into three types of settings (to handle them cleanly, they should be in seperate files) and each of these groups needs to support profiles:
<ul>
<li>Player settings: Containing ONLY player specific stuff. SHOULD be non game-specifc but probably can't (eg player class) eg: Player name, color, control bindings etc.
Profiles: danij.cfg, skyjake.cfg, yagisan.cfg...</li>
<li>Game-specific settings eg: Multiplayer server options, compatibility options, gameplay options, ui colors etc.
Profiles: jdoom.cfg, jheretic.cfg, jhexen.cfg...</li>
<li>Engine settings: Affect all games eg: rendering, network (ip, master-server paths...) options etc.
Profiles: lowquality.cfg, hiquality.cfg, pixelly.cfg...</li>
</ul>
<blockquote>A well-behaved Windows app would use "%HOME%\Application Data\Doomsday Engine" or something like that.</blockquote>
Need to bare in mind that Windows users tend to dislike putting data where the OS wants it. So if this location can be changed during installation (at least on Windows) it would be nice eg System_AppData|Doomsday base folder|custom.
<blockquote>So I propose that once beta4 has been released (before summer, I hope...), we'll get our hands dirty and carry out this directory structure revision. It should make things a lot easier down the road, especially if we design good (strict!) guidelines for addons.</blockquote>
Sounds OK to me.
<blockquote>In fact, when the comes to a directory called "runtime", there shouldn't even be one in an installation directory, since the installation directory itself is the "runtime" directory. For developers we can have a "runtime" directory as part of the source hierarchy, which is a copy of the actual installation layout. It would be used for running the game without doing a real install.</blockquote>
I agree it would be a good idea to loose the runtime folder for standard user installations.
<blockquote>The question is, what should this look from the engine's point of view? Where are resource files such as 3D models, texture bitmaps, and definition files actually located?</blockquote>
I'm inclined to think that for continuity purposes (and because virtual paths can be tweaked easily) that it might be wiser if we kept the same virtual file structure. Meaning that only the physical structure will change but internally, Doomsday still sorts files into the existing structure.
Changes to the virtual file structure don't affect the game plugins themselves very much at all, so we can tweak that after 1.9.0 if we want.
<blockquote>3D models would be loaded from under "doom2/models/". The organization under that directory depends on the definition files, but could be like "(class)/(name)", e.g., "actors/sergeant/", or "item/bullets". Or maybe we should just flatten it out to "doom2/models/(name)", e.g., "doom2/models/sergeant/".</blockquote>
In jDRP1.1 I decided to organise the models at class level first eg:
jDRP/Items/AmmoBox/*.DMD *.PNG (etc)
Considering that ultimately each "module" will contain more than just models it seemed a good idea to remove "models" from the base level so that it can be re-introduced at the top level at a latter point (if necessary) eg:
jDRP/Actors/Cacodemon/Models/*.DMD *.PNG (etc)
jDRP/Actors/Cacodemon/Scripts/*.DSF -- eventually anyway
jDRP/Actors/Cacodemon/Sounds/*.WAV
<blockquote>Definitely. PK3 or ZIP should be the official recommendation for addons. Another alternative is Snowberry bundle, but that ties the addon to Snowberry (unless we add Snowberry bundle support into Doomsday... could be tricky).</blockquote>
I don't think thats necessary. Snowberry bundles currently offer the best way to configure aggregate mods that contain options to configure which modules are used. In jDRP 1.1 I have implemented this so that the customisation works by setting command-line flags that branch the include statements in DED files. This means jDRP has the same level of customisation available from the command-line as that offered through use of the Snowberry GUI (though it is FAR easier to customise jDRP via Snowberry).
<blockquote>You're right, a common.cfg needs to be loaded first. The problem is, though, that the game-specific .cfg includes all cvars, including those set in common.cfg. After one session the game-specific .cfg would override all the common settings. To make a common.cfg work we would need to only save the "changed" cvars of a game, a task that may be a bit difficult to pull of properly. Maybe we'll just need to forget about common.cfg...</blockquote>
Considering we are going to be implementing the profiles stuff it would be nice to try and figure common.cfg into the equation though personally think it is lower priority.
<pre>
build/
engine/
plugins/
packs/
external/ (external libs and other source here, if needed)
libpng/
zlib/
</pre>
In Linux I would assume the external components can be dealt with as dependencies to other packages (e.g., libpng).