End of Beta and Start of Automated Builds
<i>This post was originally made by <b>skyjake</b> on the dengDevs blog. It was posted under the categories: Blog, Engine, Games, Platforms, Practices, Releases, Version 1.9.</i>
Last week I had some time off from work due to illness and decided to spend the time productively: we now have a system that is building a binary release automatically on the three major platforms: Windows, Mac OS X, and Ubuntu.
<strong>Gravity</strong>
Let us consider for a moment the significance of this development. As Doomsday is a hobby project, it is only natural that team members stray out from the source trunk into developing interesting and fancy new things. However, this does not help at all with producing actual releases that people at large might want to use. In essence, the situation is missing <em>gravity: </em>the far-flung development branches need a constant pull to become part of the official release stream. Hand-made releases lack gravity because it is cumbersome to make them and no one wants to do that on a regular basis. Hence, no releases for months on end. Also, important bug fixes are left stranded on some development branch with a bunch of half-finished changes with no way to get them released quickly. But with an automated build system churning out releases effortlessly (and on a regular schedule), we have the necessary gravity to motivate team members to finalize their work and integrate it back to the trunk (in reasonably-sized chunks) to get it released.
I have decided to make some other important changes as well. The never-ending "Beta" phase is now over and done with. The team's goal is not to make "Beta" quality releases — each release should be stable and good enough for normal use. The trunk (i.e., master branch) is henceforth intended to be as bug-free and stable as possible. New code will be merged in only after it has been finished with in a development branch. Consequently, the version number of the Doomsday Engine is upgraded to 1.9.7 (from beta7).
<strong>Builds and Numbers</strong>
As the build system will be generating new releases on a regular basis, there is a need to identify each build. The builds created by the system are tagged with a build number. Since these are (at most) daily builds, the build number equals the number of days since January 1, 2011. For instance, today's build number is 59. When we increase the Doomsday version number, the build number will not reset.
As to the build schedule, for now I have set it up for biweekly (Monday/Friday) builds. Every week, at exactly 12:00 (EET) on Monday and Friday, the system tags and builds a new release. The releases will become available for download within a couple of hours from the build time.
<strong>Official Stable Releases</strong>
As the automated builds are not intended to be the "final" releases, but rather work-in-progress versions, we will still need to manually mark certain builds as official stable releases. These will be linked to on the project web site. However, in practice, this means that the latest release files are simply uploaded to SourceForge and the build tag is removed from the file names.
<strong>Nifty Things</strong>
The automated build system has a couple of useful features. First of all, the builds can be tracked using RSS. One can subscribe to the <a href="http://code.iki.fi/builds/events.rss">build event feed</a> and be immediately notified of each build via the RSS app of choice (or Google Reader). This is extremely helpful for both the team members and the users. The feed itself provides links to the build files, compiler logs, and the Git changelog of all revisions since the previous build.
Another nice feature is that the build system maintains an apt repository for Ubuntu users. Currently I am able to only build 32-bit binaries. On the plus side, the .deb packages also include Snowberry, so the experience should be on par with the other platforms. As a bonus, the .deb package contains an apt source list file that automatically retrieves updates from the Doomsday Builds repository. I.e., once the .deb is installed, new builds can be installed simply by running apt-get.
The 64-bit build can be done manually from source.
<strong>Source Releases?</strong>
For the time being I have decided not to make any source releases. There are two options available if you wish to have access to the source:
<ul>
<li>For just viewing the source, the <a href="http://deng.git.sourceforge.net/git/gitweb.cgi?p=deng/deng">SF.net repository source browser</a> is quite helpful.</li>
<li>For full access to all revisions, clone the <a href="https://sourceforge.net/scm/?type=git&group_id=74815">Git repository</a> on your local computer.</li>
</ul>
<strong>Development Practices</strong>
For team members, it is worthwhile to appreciate the importance of the master branch from here on out. It is crucial that any changes made in the master will not break the automated releases. The RSS feed will guarantee that everyone knows the current state of the master, and also who is responsible for any breakage (as the committed revisions are logged). In case a breakage occurs, my first action is to remove the offending revisions from the master, returning it to a known working state.
This also means that no day-to-day development can occur in the master branch, as it must remain releasable at any given moment. Branches must be used for working on new features until they are ready for merging into the master. Trivial bug fixes are an exception: they should be done in the master branch so that they are released immediately. (Team members should take extra care of fixing bugs on the correct branch!)
An important thing to notice is that currently the master is based on the Beta6.9 release. DaniJ has been working on lots of improvements and fixes in the old beta6 branch in the past months, and these are currently not part of the official release stream. I expect that in the coming weeks the most important changes from the beta6 branch will be merged to the master to get them out in the open.
My focus has recently been on the hawthorn branch (i.e., the 2.0 stuff), but due to the new release system I am reconsidering my approach to the 2.0 development. Stay tuned for a post of my thoughts on that a bit later on.
<strong>In Closing...</strong>
I hope that the changes outlined in this post will bring enhanced vitality to the project. At a minimum there will be a steady stream of fresh binaries for people to play with.
Once more, here's the build events RSS feed: <a href="http://code.iki.fi/builds/events.rss">http://code.iki.fi/builds/events.rss</a>
Happy Doomsdaying,
Jaakko
Last week I had some time off from work due to illness and decided to spend the time productively: we now have a system that is building a binary release automatically on the three major platforms: Windows, Mac OS X, and Ubuntu.
<strong>Gravity</strong>
Let us consider for a moment the significance of this development. As Doomsday is a hobby project, it is only natural that team members stray out from the source trunk into developing interesting and fancy new things. However, this does not help at all with producing actual releases that people at large might want to use. In essence, the situation is missing <em>gravity: </em>the far-flung development branches need a constant pull to become part of the official release stream. Hand-made releases lack gravity because it is cumbersome to make them and no one wants to do that on a regular basis. Hence, no releases for months on end. Also, important bug fixes are left stranded on some development branch with a bunch of half-finished changes with no way to get them released quickly. But with an automated build system churning out releases effortlessly (and on a regular schedule), we have the necessary gravity to motivate team members to finalize their work and integrate it back to the trunk (in reasonably-sized chunks) to get it released.
I have decided to make some other important changes as well. The never-ending "Beta" phase is now over and done with. The team's goal is not to make "Beta" quality releases — each release should be stable and good enough for normal use. The trunk (i.e., master branch) is henceforth intended to be as bug-free and stable as possible. New code will be merged in only after it has been finished with in a development branch. Consequently, the version number of the Doomsday Engine is upgraded to 1.9.7 (from beta7).
<strong>Builds and Numbers</strong>
As the build system will be generating new releases on a regular basis, there is a need to identify each build. The builds created by the system are tagged with a build number. Since these are (at most) daily builds, the build number equals the number of days since January 1, 2011. For instance, today's build number is 59. When we increase the Doomsday version number, the build number will not reset.
As to the build schedule, for now I have set it up for biweekly (Monday/Friday) builds. Every week, at exactly 12:00 (EET) on Monday and Friday, the system tags and builds a new release. The releases will become available for download within a couple of hours from the build time.
<strong>Official Stable Releases</strong>
As the automated builds are not intended to be the "final" releases, but rather work-in-progress versions, we will still need to manually mark certain builds as official stable releases. These will be linked to on the project web site. However, in practice, this means that the latest release files are simply uploaded to SourceForge and the build tag is removed from the file names.
<strong>Nifty Things</strong>
The automated build system has a couple of useful features. First of all, the builds can be tracked using RSS. One can subscribe to the <a href="http://code.iki.fi/builds/events.rss">build event feed</a> and be immediately notified of each build via the RSS app of choice (or Google Reader). This is extremely helpful for both the team members and the users. The feed itself provides links to the build files, compiler logs, and the Git changelog of all revisions since the previous build.
Another nice feature is that the build system maintains an apt repository for Ubuntu users. Currently I am able to only build 32-bit binaries. On the plus side, the .deb packages also include Snowberry, so the experience should be on par with the other platforms. As a bonus, the .deb package contains an apt source list file that automatically retrieves updates from the Doomsday Builds repository. I.e., once the .deb is installed, new builds can be installed simply by running apt-get.
The 64-bit build can be done manually from source.
<strong>Source Releases?</strong>
For the time being I have decided not to make any source releases. There are two options available if you wish to have access to the source:
<ul>
<li>For just viewing the source, the <a href="http://deng.git.sourceforge.net/git/gitweb.cgi?p=deng/deng">SF.net repository source browser</a> is quite helpful.</li>
<li>For full access to all revisions, clone the <a href="https://sourceforge.net/scm/?type=git&group_id=74815">Git repository</a> on your local computer.</li>
</ul>
<strong>Development Practices</strong>
For team members, it is worthwhile to appreciate the importance of the master branch from here on out. It is crucial that any changes made in the master will not break the automated releases. The RSS feed will guarantee that everyone knows the current state of the master, and also who is responsible for any breakage (as the committed revisions are logged). In case a breakage occurs, my first action is to remove the offending revisions from the master, returning it to a known working state.
This also means that no day-to-day development can occur in the master branch, as it must remain releasable at any given moment. Branches must be used for working on new features until they are ready for merging into the master. Trivial bug fixes are an exception: they should be done in the master branch so that they are released immediately. (Team members should take extra care of fixing bugs on the correct branch!)
An important thing to notice is that currently the master is based on the Beta6.9 release. DaniJ has been working on lots of improvements and fixes in the old beta6 branch in the past months, and these are currently not part of the official release stream. I expect that in the coming weeks the most important changes from the beta6 branch will be merged to the master to get them out in the open.
My focus has recently been on the hawthorn branch (i.e., the 2.0 stuff), but due to the new release system I am reconsidering my approach to the 2.0 development. Stay tuned for a post of my thoughts on that a bit later on.
<strong>In Closing...</strong>
I hope that the changes outlined in this post will bring enhanced vitality to the project. At a minimum there will be a steady stream of fresh binaries for people to play with.
Once more, here's the build events RSS feed: <a href="http://code.iki.fi/builds/events.rss">http://code.iki.fi/builds/events.rss</a>
Happy Doomsdaying,
Jaakko
Comments
<code>deb http://code.iki.fi/builds/apt unstable main</code>
I can not access that repo, I get a "not found" message.
Also may I suggest adding a .desktop file into the package aswell, so then it is an easy way to start up snowberry. I am working on getting it working. I'm not sure which Icon to use though, the snowberry Icon, or the deng Icon?
A .desktop icon would be very good. It needs to be inserted into the application menu, too. To be consistent, I think the icon should be the Snowberry "D E three orbs" icon, as on other platforms.
Note that the "launch-doomsday" script is the correct way to start Snowberry now.
but I can not even browse the repo through firefox (http://code.iki.fi/builds/apt/). I get a "Not found" from MobileMe.
I can access this page though (http://code.iki.fi/builds/build59/)
I'll consider my options for including 64-bit builds in the repository. However, at the moment it looks like they will require some manual effort, as I don't have a 64-bit Ubuntu running continuously (maybe some kind of a cross-compilation scheme, hmm?).
I have set up a signing key for the repository. It can be installed with apt-key. The key is here: http://code.iki.fi/builds/apt/doomsday-builds-key.asc
(The same key is also here: http://dl.dropbox.com/u/11948701/doomsday-builds-key.asc )
W: GPG error: http://code.iki.fi unstable Release: The following signatures were invalid: BADSIG BACCFCFB98DB2EDC Jaakko Keränen (skyjake)
This is what I get:
<pre>
pub 2048R/98DB2EDC 2011-03-01
uid Jaakko Keränen (skyjake) <jaakko.keranen@iki.fi>
sub 2048R/3A1695CD 2011-03-01
</pre>
=
pub 2048R/98DB2EDC 2011-03-01
uid Jaakko Keränen (skyjake)
sub 2048R/3A1695CD 2011-03-01
apt-get update
=
W: GPG error: http://code.iki.fi unstable Release: The following signatures were invalid: BADSIG BACCFCFB98DB2EDC Jaakko Keränen (skyjake)
gpg: Signature made Fri 04 Mar 2011 03:52:39 PM EET using RSA key ID 98DB2EDC
gpg: Good signature from "Jaakko Keränen (skyjake) <jaakko.keranen@iki.fi>"</pre>
AFAICT the files seem ok, maybe it's an issue with file caching at some step along the way? Let's wait a while and see if the issue persists.
Does anyone have any idea what could be causing it?
I remember seeing something like this when installing wolf-et on ubuntu some time back.
I think I was trying to get the game update patch.