On Week 19/2012
Last week I was running the 1.9.8 candidates through a series of tests on my various Macs to see if any regressions had cropped up. I did manage to find and/or fix a few, for instance resizing the game window in client mode, and things/monsters not being always placed on the floor in MP. This is the price of refactoring -- while it keeps the codebase fresh and nimble, it will also cause regressions as unexpected dependencies break or there are programmer oversights.
In the last couple of years I've been spending increasing amounts of time thinking about release engineering and quality control. Quality control is funny because the most important question isn't how high the quality is. The most important thing is reliable measurements so that the resultant products have a predictable level of quality -- the level of quality can only be increased as a secondary effect after we can reliably determine what the current level actually is.
Measuring the quality of the builds is something we are currently not doing very well. The instruments we have in place are mostly superficial, and the rest is reliant on manual work by developers and users. Issues are bound to slip through the cracks in this kind of ad hoc setup. However, I'm afraid there is no silver bullet: automated testing could catch some issues, but as far as game/engine behavior is concerned, only human operators are able to determine whether things are working as intended or not. Therefore, even stable releases cannot be considered 100% stable and unchanging; we will be making patch releases every now and then (such as 1.9.7-1).
As a matter of good engineering, I would prefer to do high-quality, bug-free releases. However, given the history of the project's codebase, there is a such a large number of known and unknown bugs that it is virtually impossible to make a bug-free release. Every stable release we make has a large number of issues, some of which are recorded in the Bug Tracker and some of which are yet to be discovered. That being said, I don't think the present situation is too dire. We only have a couple of critical bugs in the Tracker, and most of the other issues fall under the category of annoyances.
We were planning to release the stable 1.9.8 on Sunday, but instead were stuck investigating a couple of issues with screen refresh on the Windows platform. It turned out to be related to the version of Qt we were using on the Windows autobuilder -- after switching it back to an earlier one everything seems to be working as expected.
Now the plan is to start work on 1.9.9 with a review of the scheduled roadmap items and bugs in the tracker.
In the last couple of years I've been spending increasing amounts of time thinking about release engineering and quality control. Quality control is funny because the most important question isn't how high the quality is. The most important thing is reliable measurements so that the resultant products have a predictable level of quality -- the level of quality can only be increased as a secondary effect after we can reliably determine what the current level actually is.
Measuring the quality of the builds is something we are currently not doing very well. The instruments we have in place are mostly superficial, and the rest is reliant on manual work by developers and users. Issues are bound to slip through the cracks in this kind of ad hoc setup. However, I'm afraid there is no silver bullet: automated testing could catch some issues, but as far as game/engine behavior is concerned, only human operators are able to determine whether things are working as intended or not. Therefore, even stable releases cannot be considered 100% stable and unchanging; we will be making patch releases every now and then (such as 1.9.7-1).
As a matter of good engineering, I would prefer to do high-quality, bug-free releases. However, given the history of the project's codebase, there is a such a large number of known and unknown bugs that it is virtually impossible to make a bug-free release. Every stable release we make has a large number of issues, some of which are recorded in the Bug Tracker and some of which are yet to be discovered. That being said, I don't think the present situation is too dire. We only have a couple of critical bugs in the Tracker, and most of the other issues fall under the category of annoyances.
We were planning to release the stable 1.9.8 on Sunday, but instead were stuck investigating a couple of issues with screen refresh on the Windows platform. It turned out to be related to the version of Qt we were using on the Windows autobuilder -- after switching it back to an earlier one everything seems to be working as expected.
Now the plan is to start work on 1.9.9 with a review of the scheduled roadmap items and bugs in the tracker.