Week 41/2012: Build woes, FS evolution
Last week we got off to a rocky start with the 1.9.10 unstable builds — not a single build had a complete set of packages for all platforms. This was because of the ongoing refactoring in the engine's file system: due to differences in how Microsoft Visual C++ and GCC/Clang interpret C++, often code that works on one platform fails to build on the others. The situation is further compounded by the various versions of the compilers we are using: counting all the platforms, we are using MSVC 2008, MSVC 2010, GCC 4.0, 4.2, 4.4, and Apple clang 4.1. Going forward, we'll try to be more careful before merging code into the master so that it remains always operational.
The situation with file systems could use a little bit of explaining. Back when we were in the 1.9 betas, I was working on the Hawthorn branch, writing a new file system (de::FS) from scratch. The end result was a clean but rather bare-bones implementation of the Doomsday 2.0 file system. It could access regular files and ZIP archives, but not any of the Doom format files (WADs). Meanwhile, DaniJ was working on the original file system in the main code base. From an architectural point of view, this original file system was unclean and inelegant: it grew out of the original Doom/Hexen WAD lump index, where I bolted on support for virtual files and ZIP/PK3 archives. It is basically a large collection of kludges. This prompted DaniJ to start improving it gradually toward the architecture of the Doomsday 2.0 file system.
Now we are at the point where de::FS, the libdeng2/Hawthorn file system, can finally be introduced in the main code base. However, the first iteration of the file system (FS1) is still there and is used for locating and loading resources. Over time these two systems are destined to evolve into a single system that follows the ideas of the 2.0 file system, but in the short term we will have two parallel systems in use — this way we can avoid disrupting existing functionality and add-on support.
Initially, when the libdeng2 de::FS enters the master branch, it will only be used for locating shared libraries and for the needs of the libdeng2 scripting engine. When it comes to libraries/plugins, these have thus far always been located using platform-specific special purpose code, so this will unify and clean up that portion of the engine nicely.
Another important goal of the file system refactoring is to fix the aforementioned multithreaded caching bug. We were planning to release it as a patch for 1.9.9, but the amount of changes has grown so large that including this work in the possible patch is looking less and less likely. In all likelihood it will take weeks before the revised code can be deemed stable release quality.
We will release a patch for 1.9.9 if suitable trivial fixes accumulate in time; I will evaluate the suitability of the new file system code when the time arrives. The long term objective is to avoid patch releases because the initial stable releases should already be of high enough quality.
The situation with file systems could use a little bit of explaining. Back when we were in the 1.9 betas, I was working on the Hawthorn branch, writing a new file system (de::FS) from scratch. The end result was a clean but rather bare-bones implementation of the Doomsday 2.0 file system. It could access regular files and ZIP archives, but not any of the Doom format files (WADs). Meanwhile, DaniJ was working on the original file system in the main code base. From an architectural point of view, this original file system was unclean and inelegant: it grew out of the original Doom/Hexen WAD lump index, where I bolted on support for virtual files and ZIP/PK3 archives. It is basically a large collection of kludges. This prompted DaniJ to start improving it gradually toward the architecture of the Doomsday 2.0 file system.
Now we are at the point where de::FS, the libdeng2/Hawthorn file system, can finally be introduced in the main code base. However, the first iteration of the file system (FS1) is still there and is used for locating and loading resources. Over time these two systems are destined to evolve into a single system that follows the ideas of the 2.0 file system, but in the short term we will have two parallel systems in use — this way we can avoid disrupting existing functionality and add-on support.
Initially, when the libdeng2 de::FS enters the master branch, it will only be used for locating shared libraries and for the needs of the libdeng2 scripting engine. When it comes to libraries/plugins, these have thus far always been located using platform-specific special purpose code, so this will unify and clean up that portion of the engine nicely.
Another important goal of the file system refactoring is to fix the aforementioned multithreaded caching bug. We were planning to release it as a patch for 1.9.9, but the amount of changes has grown so large that including this work in the possible patch is looking less and less likely. In all likelihood it will take weeks before the revised code can be deemed stable release quality.
We will release a patch for 1.9.9 if suitable trivial fixes accumulate in time; I will evaluate the suitability of the new file system code when the time arrives. The long term objective is to avoid patch releases because the initial stable releases should already be of high enough quality.