Future Doomsday Add-on Distribution

DaniJ

Prior to our host's web server upgrade just before Christmas we were fortunate to have been granted unlimited bandwidth for this site and our Add-on repository (files.dengine.net). Back when we first set up the Add-on repository the amount of bandwidth required to provide that service was relatively small. However just before the web server upgrade we were confronted with the fact that our bandwidth usage had risen dramatically. Although good in the sense that Doomsday is clearly becoming a lot more popular, the flip side is that our host is no longer willing to absorb that amount of punishment :-/

As you may recall, in our first month on the new web server we consumed our entire bandwidth allowance (500gb) in less than a week. Subsequently the site itself went offline while I negotiated a temporary increase to this limit.

The reality is that we simply cannot continue to provide this service in its current form and that is why the Add-on repository is currently offline.

So, Deng team is calling upon the community to help us decide the future of the Doomsday Add-on repository.

Yagisan

My recommendation would be to seed the content via bittorent in logical packs. Don't bother "zipping" up the contents to make them smaller, leave them uncompressed so people can choose which parts they want to download. You'll need a machine to do the initial seeding, and to be the only seed to stop the torrents from dieing - I think your current host can do that fine - just bandwidth limit the seeding to say 192kb/s maximum combined total for all seeds, and you should stay under your 500GB every 28 days. I'd use the openbittorent ( http://openbittorrent.com/ ) and public bittorent (http://publicbt.com/ ) and possibly a _free_ Google app engine install of atrack ( http://repo.cat-v.org/atrack/ ) as a third backup tracker. For an efficient swarm set the block size to 64KB regardless of actual total resource pack size, and make sure your initial seed is in super-seeding mode.

reg

What is a "logical pack"? I, for instance, don't like audio packs. Some people play Doom only and don't need Heretic/Hexen stuff. So
I agree that packs should contain unpacked files as they appeared on original addons page. This will allow me & others who have them to seed, too.

Yagisan

reg wrote:

What is a "logical pack"?

Way back in the dawn of time, when I used to make packages for Debian (5 Mar 2005 - 1.8.6 packaged) , I would separate the packages in a way that made logical sense to me. I had the sycraft doom audio packs in one logical pack, the jdtp in another, the jdrp in another, xccp in another and so on. KuriKai and DaniJ would no doubt be familiar with how I split up doomsdays packs pre-snowberry, and that's roughly the sort of structure I'd have in mind.

KuriKai

you mean having packages more separate? cause I think things should stay in pk3's cause it make it easier for people to move into the addons folder, and we don't know how deng 2.0 will support addons, with it's in game addon browser

DaniJ

There is no reason to change the way the addons themselves are packaged. We just need to determine how best to distribute them.

KuriKai

Best way is defiantly via p2p

JonusC

Whatever makes most sense to a non-technical player/user I would assume is the most 'logical', much like it has been already in chunks-by-type. Doom Textures PK3, Doom Models PK3, Doom SFX PK3... I guess having a seperate .torrent file for each PK3 is best, since not everyone uses the (for example) per-file selection that uTorrent has.

Torrent is a cool thing, as far as I know if the Torrent/Seed files are updated then all it takes is a "force hash check" and most Torrent clients will see the difference between new file and currently download file (although I think everything is wasted due to invalidation after the first chunk mismatch; torrent doesn't use a smart binary compare per chunk does it?)

P.S. This forum is REALLY fast.

reg

JonusC wrote:

Whatever makes most sense to a non-technical player/user I would assume is the most 'logical', much like it has been already in chunks-by-type. Doom Textures PK3, Doom Models PK3, Doom SFX PK3... I guess having a seperate .torrent file for each PK3 is best, since not everyone uses the (for example) per-file selection that uTorrent has.

I agree. However, there're some addons which are < 100 Kb. Making a separate .torrent file is just not worth it. It's possible to either combine them in 1 torrent or offre them for direct download(with speed cap of course).

reg

Mr. Rocket posted a direct download link to a pack of resource packs. Maybe DDLs should be provided for people who don't want to bother with torrents or can't use the?
By the way, have any decision about distribution been made? Putting back the page on files.dengine.net looks like trivial, because torrent links were alreeady there.

DaniJ

Restoring the page would indeed be trivial but we can't restore it as it was.

We were hoping that this thread would have gained rather more replies/viewpoints than it has. No decision can even be considered yet as we haven't nearly enough opinions on the subject.

reg

DaniJ wrote:

Restoring the page would indeed be trivial but we can't restore it as it was.

We were hoping that this thread would have gained rather more replies/viewpoints than it has. No decision can even be considered yet as we haven't nearly enough opinions on the subject.

I see.However, IMHO, if you leave everything as is, you won't get much more replies. Who has replied to this thread? Yagisan (former developer), KuriKai (deng team member) and 2 users (me and JonusC). I think this indicates that current forum members just don't care - they got the addons already.
As for newcomers, they just come to files.dengine.net, see the message and leave. Only maybe 1% find this thread.
But they probably won't post their opinion - they want to get addons, not to discuss how they would be distributed. It's much more effective for them to just use Google or something similar. BTW, searching Doomsday in Google gives link to its former site at 2th place, and Dengine.net only the 10th.
So, to get more opinions, IMHO, you should:
1. Put a link to this thread to page at files.dengine.net
2. Restore the page with direct links, links to Rpaidshare etc., torrents, or anything else.
3. Expect complaints about how bad the chosen distribution method is. :-(

Vermil

Indeed, there are a few people asking why they can't download the add-on's over at NewDoom.

DaniJ

reg wrote:

Restore the page with direct links, links to Rapidshare etc., torrents, or anything else & Expect complaints about how bad the chosen distribution method is. :-(

We can certainly restore it but only the torrents could remain.

BTW, searching Doomsday in Google gives link to its former site at 2th place, and Dengine.net only the 10th.

I'm not sure what you searched for but when I google "Doomsday Engine" this site is #2 while our old domain is #1 - which is understandable given the amount of amassed history at the old site.

reg

DaniJ wrote:

We can certainly restore it but only the torrents could remain.

OK, I just wanted to ask - what do you think about links to file-hosting services? I personally don't like them (I prefer torrents) but some users might not want to get a BitTorrent client.
Your decision to wait just seems odd - 3 weeks should've been enough for any forum member to express his/her opinion on the subject, and almost all newcomers don't even know that this forum exists (as I said). Restoring page with just torrent links is simple, can be easily reversed (because it probably won't use up bandwidth quota). And it's better than nothing.

DaniJ wrote:

I'm not sure what you searched for but when I google "Doomsday Engine" this site is #2 while our old domain is #1 - which is understandable given the amount of amassed history at the old site.

I googled "doomsday" if you're interested.

DaniJ

reg wrote:

I just wanted to ask - what do you think about links to file-hosting services? I personally don't like them (I prefer torrents) but some users might not want to get a BitTorrent client.

I don't like them either and my understanding is that you need to continually resubmit files to such services.

Your decision to wait just seems odd - 3 weeks should've been enough for any forum member to express his/her opinion on the subject...

You would think so but it seems you are correct; most simply don't care.

...and almost all newcomers don't even know that this forum exists (as I said).

Not much more we do about that, there is a link to the forum clearly visible on the homepage.

I googled "doomsday" if you're interested.

That term is too generic for this site to rate highly with google at present. I surmise the only reason the old domain is rated as highly as it does for that term is because of its age.

reg

DaniJ wrote:

...and almost all newcomers don't even know that this forum exists (as I said).

Not much more we do about that, there is a link to the forum clearly visible on the homepage.

Many come directly to files.dengine.net. There's no link back to the main site.

DaniJ

Exactly. They are quite happy to use the service but really don't care about the project so long as it gives them what they want.

What we could do for Doomsday 2.0 is to build in a method of locating addons (we could torrent them, using a plugin), thus circumventing the whole issue.

Lightning Hunter

I'll tell you why my opinion hasn't really been voiced on this subject. I am personally still waiting for a stable release of Doomsday (a modern release; not 1.8.6). At the moment, there is no point for me to discuss add-ons when Doomsday itself isn't stable enough to play them. I'll be honest; I haven't sat down to play doom in years - since the beta versions of Doomsday started. All activity has revolved around the beta versions of Doomsday for several years now, so the amount of feedback by users has decreased significantly as a result. Back when there was a stable version of Doomsday and Dani J was working on the resource pack, the feedback and amount of activity was abounding. To the average player, the progression of Doomsday via beta releases just isn't that exciting - especially when things have to seemingly go backwards first before going forward.

I don't think many responses can be expected until Doomsday is stable once again, and resource packs (or something exciting in the eyes of the players) is actually being worked on. At the moment, people are still lingering around NewDoom where they remember the activity as being prevalent at one point. This forum will eventually get there too, but it will take a stable version of Doomsday and something "exciting" to revive the community. Until then, it's just us "hardcore" Doomsday fans that hang around.

Also, I'm not sure what kind of responses are expected here. Either the resource repository can be kept up, or it can't. It seems to me that it's all about funding. If it can't be afforded, then it can't be kept up. I'm not exactly sure what kind of solution is hoped to be gained with this thread. Even after reading the initial post a few times, I just don't know what kind of answer to give!

Mancubus

My opinion is: rather than having the "archive is not available" page it's better to have torrents at least. Maybe with some pre-words, that 'due to bandwidth problem we are unable to provide direct downloads for the moment but we are working on this, please come here and help us'.

You don't like torrent's? Either come and propose your way to solve it, or wait until other will do it for you.
If you are able to solve all the add-ons thing in Snowberry, torrent is nothing complex to you.

Psychikon

Maybe if you have torrents for now and/or dedicated Rapidshare or what have you sites specifically for certain chunks of the data (so the bandwidth isn't exceeded), then you might also leave a message that will both put people at ease and might even spark their interest in the development aspect. Something like this:

"Doomsday is expected to adapt a filesearch blah blah in it for your convenience, but until that time, we have these links here:
-blah
-blickity
-blah

To track news on the project, or to see what the developers are doing, go here, or visit our forums."

My assumption is that what you want is a way in which to not only have it over and done with, but a stable framework so that it can not only be easily managed but easily accessed as well - is this correct, Dani?
If so, it might not be so easy at this time, taking its toll on your mind about the project, and might be better suited to put the completion of it on hiatus (using an easy, temporary method) until such formerly-proposed (in-engine) methods are implemented; leaving a message of hope for others, and a way for them to ease their mind on the project by providing the proper links.

DaniJ

How about we submit Doomsday addons to /idgames and then direct users there instead? After all, it is already hosting pretty much every DOOM related file known to man.

Lightning Hunter

That sounds like a good idea to me.

reg

DaniJ wrote:

How about we submit Doomsday addons to /idgames and then direct users there instead?

Will they be accepted? There's many files >50 mb.
EDIT: Some of them are accesible on their sites; for instance, DHTP or Sycraft's music pack.

SkullDrago

Why don´t you make a svn repo on sourceforge.net where you upload the newest addons and everyone can download them via tortoisesvn. torrent is a p2p which many people don´t want to use because of many viruses. Many have fear because the police is viewing in that p2p networks.

DaniJ

reg wrote:

DaniJ wrote:

How about we submit Doomsday addons to /idgames and then direct users there instead?

Will they be accepted? There's many files >50 mb.
EDIT: Some of them are accesible on their sites; for instance, DHTP or Sycraft's music pack.

Nearly all of those that are larger than 50mb are available for download on the original authors' sites. We could link to those and the rest could go to /idgames.

SkullDrago wrote:

Why don´t you make a svn repo on sourceforge.net where you upload the newest addons and everyone can download them via tortoisesvn.

Although convenient for those maintaining the addons, that solution would be unworkable for the users. Very few users even have an idea what an svn is, let alone configure (and install depending on the OS) a client just to get Doomsday addons.

Alwique

I favor idgames for the download section.

reg

If addons will be accepted for /idgames, then it's a good idea.

haramanai

Hi.
Torrents and file hosting services is the way to go.
What I will say it may sound bad but sounds right to me.
I just ordered Doom collectors edition and it's second hand. You cannot find a new hard copy of this game anywhere. Also the price to download the game from the id store is 20 dollars.
Why I am saying all this?
Because this is the reason why people who are going to search for the Add-ons are familiar with torrents.

Psychikon

haha, I've actually used torrents when I've lost my CDs too actually

Alwique

Idgames or file hosting. I prefer idgames do to the fact that it is stable and secure.