Bringing Wolfenstein 3D to Doomsday
(thread split from viewtopic.php?f=23&t=1756)
I've had a change of plans. I still want to do this, but I won't have the time to devote myself to such a large undertaking for quite a while. Instead I'd like to go for a simpler goal first to get familiar with the Doomsday structure.
I'd like to port Wolfenstein 3D instead: it's smaller and the source is very well laid out and properly commented (Build's source is an utter mess in comparison - no, scratch that, it's an utter mess, period). It appears the Wolfenstein 3D isn't using a single WAD (or whatever other format), but a bunch of loose WL6 files. Those should probably be put in a ZIP archive by the user and then treated similarly to a WAD, right?
Is this OK with you guys or are there any reasons why the Wolfenstein 3D engine would be unsuitable?
I've had a change of plans. I still want to do this, but I won't have the time to devote myself to such a large undertaking for quite a while. Instead I'd like to go for a simpler goal first to get familiar with the Doomsday structure.
I'd like to port Wolfenstein 3D instead: it's smaller and the source is very well laid out and properly commented (Build's source is an utter mess in comparison - no, scratch that, it's an utter mess, period). It appears the Wolfenstein 3D isn't using a single WAD (or whatever other format), but a bunch of loose WL6 files. Those should probably be put in a ZIP archive by the user and then treated similarly to a WAD, right?
Is this OK with you guys or are there any reasons why the Wolfenstein 3D engine would be unsuitable?
Comments
That's the idea. If I were to write my own code I would decide everything myself, but when working within an existing codebase one has to adhere to the style and conventions of the codebase. otherwise the project will turn into a mess.
Progress so far:
https://github.com/HiPhish/Game-Source- ... D.markdown
Comments and suggestions are welcome.
EDIT: finished how to decompress maps
http://tinypic.com/r/9vl8bm/8
In order to help me test my findings about the game formats I wrote a small extractor tool over the weekend. It's a command-line program written in C that loads the data from the files and prints out the raw binary contents.
https://github.com/HiPhish/Wolf3DExtract
One still needs to make sense of what the data actually means, so if someone wants to help out while i work on the other data formats that would be great. Now that we can extract the maps it's not even hard, just print it to a file, open it up in a hex editor and and cross-reference with a graphical map. Or find it in the source code.
http://tinypic.com/r/2jdetxc/8
About finding out the meaning of the bytes, you can scratch that part, I found a list. The DOS level editor Nmap comes with a file called "MAPDATA.DEF" that's a text file that lists the hex numbers and the corresponding objects. That will shave off several hours of work.
EDIT: And by the way, Vim is awesome!
Yes, I am now in control of the image format. There is still some cleanup to do before committing though, but it works!
I also think VIM is pretty cool
Might I suggest the topic title being re-named or at least amended though
https://github.com/HiPhish/Game-Source-Documentation
https://github.com/HiPhish/Wolf3DExtract
So far we are in control of maps and bitmap pictures, but there is still a long way to go. As far as graphics are concerned we still need sprites and textures. Then there is the audio side where we need both sounds and music.
I am kind of nuts when it comes to documentation; if you look at the code of my extractor you'll see that everything has doxygen comments. I don't consider "the source is available" to be a good substitute for a proper documentation, self-documenting code is just an ideal. I want the documentation to be sufficient enough that one could write a port without ever looking at the original code. It's enough if one person has to look at this unportable mess.
The extractor tool is there to help me put the documentation to the test, I write it without referencing the original code. It's slower than just copy-pasting the parts I need, but the result is cleaner and more readable that way. I'm particularly proud of how modular the code is, one could literally take a module, compile it into a library and drop it into another project.
To make proper visible images from the bitmap pics I have written a little converter tool that converts the picture into a PPM image. PPM is a very obscure image format, but it is the simplest one I could find that was standardised. All I need to do then to get a PNG is pipe three programs into each other in the Terminal: The last program is ImageMagick. Once the picture has been converted to PPM I am no longer responsible for it and it's up to the user what they want to do with it.
Next up are sprites or textures then, they are both stored differently from pictures.
It seems like I accidentally figured out textures rather than sprites, but the sprites can't be far away.
I have now fully understood the format for both sprites and textures and I am able to extract them. The repositories need a bit of cleanup before I update them though.
Textures, sprites and sound effects are all stored in the same file; first the textures, then the sprites and finally the sound effects. I had forgotten to add the sprite offset when i was trying to extract them, that's why I was getting textures instead. However, the format of textures and sprites is not the same, so it was pure luck that some of the textures turned out right. But knowing that helped me figure out textures in a couple of seconds: unlike sprites textures are not compressed and can be read just as they are stored.
With this all graphics formats are concluded. Now we have just sound and music left. I'm afraid I have no ida how sound is stored in general, so i have no idea how long that will take.
As for music and SFX check out this link:
http://www.shikadi.net/moddingwiki/AudioT_Format
According to that source the SFX are stored in the WSWAP in raw PCM which is the most basic sound format. So all you have to do is figure out the offsets. The music is stored in IMF format in the AUDIOT file (Offsets in the AUDIOHED). The IMF format was used in a several old DOS games and should be pretty well documented if you google around.
http://devinsmith.net/backups/bruce/wolf3d.html
The problem with both has been that they have always been missing some details, or in the case of the Modding Wiki being too broad because an article covers multiple games. These sources are good for pointing me in the right direction rather than digging aimlessly through the source code. Speaking of code, a better alternative to the original code is Chocolate Wolfenstein 3D, which is mostly the same, but replaces all the non-portable parts (assembly code, language extensions, undefined features) with portable ones:
https://github.com/fabiensanglard/Choco ... enstein-3D
Without those sources there is no way I could have made progress this fast.
http://tinypic.com/r/33ely0l/8
I can't make sense of them though, I used another application to convert them to MIDI, to make sure I was getting proper results. If Doomsday doesn't already support the IMF format we'll have to either use a library like AdPlug or roll our own. I can also extract PC-speaker and AdLib sound effects, but I can't play them, so i have no idea if the result works.
For the purposes of your extraction tool I would simply output the "raw" IMF format data. (Doomsday does not presently support this format but a MIDI conversion is probably fairly straightforward).
For images I needed to know in what order the pixels were, so I wrote a converter that outputs PPM imaged, that can be converted to PNG for example. That's how I learned that textures were not rotated 45°, but transposed.
That said though, the music tracks and the sound effects are extracted the same way using the same code, so I think it's safe to assume they were extracted correctly as well. That only leaves digitised sound effects. I know they are stored in the VSWAP file, the question is how and where.
You have already documented how to locate the sound-chunks in the VSWAP. For example the first sound-chunk will be offset at 0x116a00 and having the length of 0x1000. The data is in raw PCM format, 1-channel, 8-bit, ~7000 Hz samplerate. All one have to do to make a WAV file is to prepend a WAV-header. The info on the header can be found here:
https://ccrma.stanford.edu/courses/422/projects/WaveFormat/
To verify the method I have extracted the first 3 sound-chunks manually:
https://www.dropbox.com/s/02ghct3z7zxhbio/sound1.wav?dl=0
https://www.dropbox.com/s/cv8ib72al01dkpq/sound2.wav?dl=0
https://www.dropbox.com/s/xs4q8o0k8kj3ffp/sound3.wav?dl=0
Have you got a grip on the adlib sound FX yet? The format looks rather complex though according to the moddingwiki it should be possible to convert them to IMF (with just one channel).