Bringing Wolfenstein 3D to Doomsday

edited 2015 Jan 13 in Developers
(thread split from viewtopic.php?f=23&t=1756)

I've had a change of plans. I still want to do this, but I won't have the time to devote myself to such a large undertaking for quite a while. Instead I'd like to go for a simpler goal first to get familiar with the Doomsday structure.

I'd like to port Wolfenstein 3D instead: it's smaller and the source is very well laid out and properly commented (Build's source is an utter mess in comparison - no, scratch that, it's an utter mess, period). It appears the Wolfenstein 3D isn't using a single WAD (or whatever other format), but a bunch of loose WL6 files. Those should probably be put in a ZIP archive by the user and then treated similarly to a WAD, right?

Is this OK with you guys or are there any reasons why the Wolfenstein 3D engine would be unsuitable?
«1

Comments

  • do it and test it your self to get the best option if you ask others they will always tell you what is the best and i agree with your zip idea.
  • I don't think you should force the users to zip up their Wolf3D install.
  • It depends on how Doomsday handles things. If it's possible to plug in a folder full of files instead of one file that's fine by me, but that's not my decision. Besides, zipping up a folder takes like two seconds, so I think people will be able to overcome that obstacle.
    Gordon wrote:
    if you ask others they will always tell you what is the best
    That's the idea. If I were to write my own code I would decide everything myself, but when working within an existing codebase one has to adhere to the style and conventions of the codebase. otherwise the project will turn into a mess.
  • The Doomsday file systems can load game resource files as native files and folders from the local file system. There is no need to package them into a ZIP. Wolfenstein uses a precursor to what became the WAD file format. It may be that support can be folded into the existing WAD file interpreter if the differences between the two can be reliably detected.
  • I have put my documentation so far on GitHub. I have a good overview of how maps are stored and I have documented the Carmack compression algorithm. Next up is figuring out the RLEW compression.

    Progress so far:
    https://github.com/HiPhish/Game-Source- ... D.markdown

    Comments and suggestions are welcome.
  • RLEW compression explanation is up.

    EDIT: finished how to decompress maps
  • Have you ever wondered what a Wolfenstein 3D map looks like decompressed in all its binary glory? then today's your lucky day:

    http://tinypic.com/r/9vl8bm/8

    In order to help me test my findings about the game formats I wrote a small extractor tool over the weekend. It's a command-line program written in C that loads the data from the files and prints out the raw binary contents.
    https://github.com/HiPhish/Wolf3DExtract

    One still needs to make sense of what the data actually means, so if someone wants to help out while i work on the other data formats that would be great. Now that we can extract the maps it's not even hard, just print it to a file, open it up in a hex editor and and cross-reference with a graphical map. Or find it in the source code.
  • ha, that is cool. you are doing well
  • I fixed my code, I had mixed up two variables in the Carmack decompression algorithm. Now the ASCII rendering looks like the actual map. I had also found a weird quirk in the Carmack-decompressed data where it needs to be shifted by one word before RLEW-decompressing. Why? I have no idea, might be a leftover from Id's older games, but that's what was hardcoded into the original code. That's the sort of thing why I wrote the extractor utility. Here is the "pics or didn't happen":
    http://tinypic.com/r/2jdetxc/8

    About finding out the meaning of the bytes, you can scratch that part, I found a list. The DOS level editor Nmap comes with a file called "MAPDATA.DEF" that's a text file that lists the hex numbers and the corresponding objects. That will shave off several hours of work.

    EDIT: And by the way, Vim is awesome!
  • It's getting really late, so I'll just drop this by: 2r7bhg7.png

    Yes, I am now in control of the image format. There is still some cleanup to do before committing though, but it works!
  • At this rate it won't be long before we can consider implementing data conversion from the Wolfenstein 3D formats in Doomsday. I particularly like the fact you are documenting your findings as you go. Keep up the great work HiPhish.
  • nice, good to see the progress.

    I also think VIM is pretty cool
  • The progress is indeed cool.

    Might I suggest the topic title being re-named or at least amended though :)
  • OK, both repositories are up to date now. I'm not sure if I had shared the links, so here they are again:
    https://github.com/HiPhish/Game-Source-Documentation
    https://github.com/HiPhish/Wolf3DExtract

    So far we are in control of maps and bitmap pictures, but there is still a long way to go. As far as graphics are concerned we still need sprites and textures. Then there is the audio side where we need both sounds and music.

    I am kind of nuts when it comes to documentation; if you look at the code of my extractor you'll see that everything has doxygen comments. I don't consider "the source is available" to be a good substitute for a proper documentation, self-documenting code is just an ideal. I want the documentation to be sufficient enough that one could write a port without ever looking at the original code. It's enough if one person has to look at this unportable mess.

    The extractor tool is there to help me put the documentation to the test, I write it without referencing the original code. It's slower than just copy-pasting the parts I need, but the result is cleaner and more readable that way. I'm particularly proud of how modular the code is, one could literally take a module, compile it into a library and drop it into another project.

    To make proper visible images from the bitmap pics I have written a little converter tool that converts the picture into a PPM image. PPM is a very obscure image format, but it is the simplest one I could find that was standardised. All I need to do then to get a PNG is pipe three programs into each other in the Terminal:
    ./wolf3dextract -pic 3 | ./vga2ppm | convert /dev/stdin pic_3.png
    
    The last program is ImageMagick. Once the picture has been converted to PPM I am no longer responsible for it and it's up to the user what they want to do with it.

    Next up are sprites or textures then, they are both stored differently from pictures.
  • Well, this isn't really what I was looking for: opmjv8.png

    It seems like I accidentally figured out textures rather than sprites, but the sprites can't be far away.
  • you will find the sprites in time as you were just looking at images and wanted sprites but found textures. maby the textures come before the sprites in the code or in the files so either way your bound to find them first.
  • 2wegh7d.pngs5hhf7.png Halt, zis Sread is verboten for allied Schweinehunds!

    I have now fully understood the format for both sprites and textures and I am able to extract them. The repositories need a bit of cleanup before I update them though.

    Textures, sprites and sound effects are all stored in the same file; first the textures, then the sprites and finally the sound effects. I had forgotten to add the sprite offset when i was trying to extract them, that's why I was getting textures instead. However, the format of textures and sprites is not the same, so it was pure luck that some of the textures turned out right. But knowing that helped me figure out textures in a couple of seconds: unlike sprites textures are not compressed and can be read just as they are stored.

    With this all graphics formats are concluded. Now we have just sound and music left. I'm afraid I have no ida how sound is stored in general, so i have no idea how long that will take.
  • You are making very fast progress and it's cool to see your updates. Pretty exciting I think.
  • Keep up the good work HiPhish. It will be awesome to one day play Wolf3d in Doomsday.

    As for music and SFX check out this link:
    http://www.shikadi.net/moddingwiki/AudioT_Format

    According to that source the SFX are stored in the WSWAP in raw PCM which is the most basic sound format. So all you have to do is figure out the offsets. The music is stored in IMF format in the AUDIOT file (Offsets in the AUDIOHED). The IMF format was used in a several old DOS games and should be pretty well documented if you google around.
  • Thanks, I am already aware of the Modding Wiki, that's one of my two main sources, along with this:
    http://devinsmith.net/backups/bruce/wolf3d.html

    The problem with both has been that they have always been missing some details, or in the case of the Modding Wiki being too broad because an article covers multiple games. These sources are good for pointing me in the right direction rather than digging aimlessly through the source code. Speaking of code, a better alternative to the original code is Chocolate Wolfenstein 3D, which is mostly the same, but replaces all the non-portable parts (assembly code, language extensions, undefined features) with portable ones:
    https://github.com/fabiensanglard/Choco ... enstein-3D

    Without those sources there is no way I could have made progress this fast.
  • to the IMF format that wolf uses Wikipedia has a huge article on it tho i did not read it all but i do know that music format was made by ID software and that several games used it. tho inferior to midi thats basically what it is. http://en.wikipedia.org/wiki/IMF_%28file_format%29
  • I can now extract music tracks:
    http://tinypic.com/r/33ely0l/8

    I can't make sense of them though, I used another application to convert them to MIDI, to make sure I was getting proper results. If Doomsday doesn't already support the IMF format we'll have to either use a library like AdPlug or roll our own. I can also extract PC-speaker and AdLib sound effects, but I can't play them, so i have no idea if the result works.
  • I wouldn't worry too much about translating the resource data into more common formats. So long as we can locate and understand the format of the source data we can solve the format conversion at a later time.

    For the purposes of your extraction tool I would simply output the "raw" IMF format data. (Doomsday does not presently support this format but a MIDI conversion is probably fairly straightforward).
  • I would like to be able to verify the output, so I know I extracted the right thing. For example, when I thought I had the map format figured out I noticed that the bytes of the map were all wrong after a certain point and so I knew my analysis was wrong. It helped me track down the error in the extractor and I was able to update the documentation accordingly.

    For images I needed to know in what order the pixels were, so I wrote a converter that outputs PPM imaged, that can be converted to PNG for example. That's how I learned that textures were not rotated 45°, but transposed.

    That said though, the music tracks and the sound effects are extracted the same way using the same code, so I think it's safe to assume they were extracted correctly as well. That only leaves digitised sound effects. I know they are stored in the VSWAP file, the question is how and where.
  • If you haven't already figured the digitzed sound part out (though I suspect you might have looking at the rate you are progressing right now) - here's how:

    You have already documented how to locate the sound-chunks in the VSWAP. For example the first sound-chunk will be offset at 0x116a00 and having the length of 0x1000. The data is in raw PCM format, 1-channel, 8-bit, ~7000 Hz samplerate. All one have to do to make a WAV file is to prepend a WAV-header. The info on the header can be found here:
    https://ccrma.stanford.edu/courses/422/projects/WaveFormat/

    To verify the method I have extracted the first 3 sound-chunks manually:
    https://www.dropbox.com/s/02ghct3z7zxhbio/sound1.wav?dl=0
    https://www.dropbox.com/s/cv8ib72al01dkpq/sound2.wav?dl=0
    https://www.dropbox.com/s/xs4q8o0k8kj3ffp/sound3.wav?dl=0
  • That's fantastic, I haven't looked into digitised sound yet, I was still working on the PC speaker sound today. It's working now, I can extract the sound effects and then run them through my converter to make them into wave files. I had to learn how wave files work and I came across the same link you posted :)
  • Great news about the PC-speaker sounds :)

    Have you got a grip on the adlib sound FX yet? The format looks rather complex though according to the moddingwiki it should be possible to convert them to IMF (with just one channel).
  • I haven't looked at it yet. If you want to go ahead feel free, I'm still documenting PC speaker, then I'll have a look at digitised sound.
  • OK. I will try and look into it though I suspect I'm in over my head here.. I have some experience with MIDI files but this is a little bit more excotic :)
  • Then you know more than I do, I don't even know how to read notes.
Sign In or Register to comment.