Har

I have an idea.

To preface:

-Manual addon management is not user friendly for people who aren't at least a little bit of a power user.
-Developers donate their time and generally do an awesome job keeping their addons up to date for relatively little compensation.
-Hosting addons is a bandwidth heavy operation that requires an expensive connection to the internet.

My idea is an application that allows users to download, install, and update their addons. The application would be open source, and the downloads would be peer to peer (using bittorrent or something like it). This would remove the necessity of having a central cost point for the bandwidth. Products are already available that do this (curse updater, wowinterface updater, etc), however they all represent ways to monetize the donated work of addon developers and have a company behind them to cover the bandwidth expenses. On top of this, while they all work, none of them have any automatic versioning control or any other advanced features that could reduce the overall amount of resources consumed.

Features I would like:

-Autodiscovery of installed addons
-Automated addon bug reporting capabilities (specifically identifying when addons don't work well together)
-Automated detection and download of new versions of installed addons
-A comments page that would allow new comments, as well as import comments from wowinterface and curse (if we're allowed)
-Developers could have a tip jar (through one of those micropayment tip jar web pages), and choose if they want to display adwords or something on the description/download page. All revenue goes straight to the developer
-Downloads of an update could compare (on a byte level) the previous version and only download the changed portions of the file
-Developers could "seed" and maintain their addon themselves, or users would be able to upload addon files they got through traditional means (assuming they have the right to share the addon)

I'm not a developer, but hopefully someone here is and thinks this is a pretty cool way to get more money into the hands of the volunteer devs and make a better addon experience overall.

 

Rayyeter

I like this idea, but there are a couple things that stand out to me as somewhat flawed/problematic:

-If downloads are peer to peer, there still needs to be a site to track those downloads (a bittorrent-esque implementation could be done tracker-less, I suppose, but then how would people get the addon?). This could potentially take a lot of bandwidth.
-Comparing on a byte level would require downloading the whole file, would it not? Perhaps a hash compare (providing devs make a hash for each file) would work better. BT-esque protocol would cover this.


I'm wondering if I might be able to implement something in C# to do this. This would definitely be faster than the downloaders are (from my memory) on patch days, though who knows if the time it takes to do something like this would be worth it when there's already a couple solid (albeit centralized) options in place.

You must obey the Dance Commander!

 

Har

The byte level comparison would be done by the author when he seeds the new file. He would have an option of seeding a file delta, which would simply be the differences between the last release and the one being uploaded.

There would have to be a tracker (assuming we use the bittorrent protocol), but the amount of bandwidth consumed by a tracker is orders of magnitude less than serving the file again and again in a series of one way centralized transfers.

The existing services are good, but they are not as good as they could be. As such, I believe that there is enough room in the "market" to justify this application. If it does everything that (for example) Curse.com does, but does it for free with a few features not currently available with the Curse addon manager, people would use it. All it would take for the project to gain steam is a few thousand users who would probably have the majority of common addons installed between them.

I got a sourceforge project up and running here: SourceForge.net: WOW Addon Manager - Project Web Hosting - Open Source Software. Anyone interested in developing should go there and sign up.

 

Kjallstrom

I think a number of these issues are largely solved by distributed version control systems like Git and Mercurial. The interesting task then is to make hosting, and really any aspect of dealing with the repository beyond "there's a new tagged version available", transparent to the end user while leaving them accessible to the developer.

 

GSH

The major issue here is how to handle new versions. If you allow anyone on the network to simply introduce new versions, you might end up with problems between conflicting versions.

Right now, the download sites essentially act as an authority to say that "Yes, this developer is permitted to add new versions." You're going to need a point of central authority to determine correct versioning, and that point will end up using a lot of bandwidth, which defeats the point of a distributed system.

 

Har

Git and Mercurial are ways to coordinate multiple developers on open source projects. Neither of them are "WOW aware", and they both charge for hosting. I actually imagine that the devs would continue to use Git and Mercurial exactly how they do now, but also seed their versions on this application and set up a tip jar.

 

Har

Not anyone would be able to seed new versions- only the owner of the project. The only time someone other than the addon developer would do this is if they had permission to do it. Also, only the addon developer would be able to ask for tips.

Also, "a point of central authority to determine correct versioning" does not cost bandwidth, it costs time. Bandwidth is used only for file transfers

 

Har

...

 

Tifordin

If the project is open source then the protocol used to upload new versions is also open source. You could use key pairs or some sort of encryption to attempt to ensure that only the developer uploads new versions, but it does seem somewhat prone to error and security issues.

A better approach would be to have some sort of central server, but only store a minimal amount of information on it (versions, name, description, other metadata), with the actual content being distributed by the peers. Starting to sound familiar? Well it should, because this is exactly what torrents do :P

Of course you could further modify and streamline it to have multiple torrents under the same project as versions. Then again, torrents don't lend themselves well to partial downloads (diffs) due to the way they represent data in pieces of a fixed size, and perform hashes on those pieces.

In fact, the only way I could see it working efficiently is if you somehow combined the concepts of a versioning system (CVS, SVN) with a distributed system (Bittorrent). Ie. a centralised server would inform peers of a list of projects and revisions, in addition to other peers who also maintain a copy of that project and/or version. Then your client would contact other peers to download all the change logs (basically a list of changes between the current version and the previous version) between your version and the most updated version. Perhaps these logs could be downloaded using the hashed piece model of bittorrent. On your computer would be stored both the change logs (similar to an SVN repository), and the actual version which appears in the Addons folder (similar to a working copy, kept up to date as new change logs arrive). The change logs present on your client would then in turn be shared with other peers.

The drawback to this approach is that if an overzealous developer releases 5 new revisions every day then the overhead associated with grabbing a new change log for each revision would get a little over-the-top.

All in all, a fairly complex system to model and implement. Definitely doable though I'd say... The hard part is making it worthwhile for addon developers to migrate to your distribution system.

 

sarf

I would recommend using a least-effort system - use Pirate Bay as the centralized platform, with each torrent being one addon. Have some sort of naming convention that the client uses (prefix), coupled with using an extra tag/part of the .torrent file to contain signatures of the torrent file contents if necessary. Make an "uploading client" for authors to facilitate "releasing" an addon version to the torrent site.

TPB has a comment system and would probably be amenable to the bandwidth usage even if they would not be able to get their ads viewed by the client simply for ideological reasons. Simply web scraping a few hundred pages of search results (using the prefix or naming convention) would get the available list of addons (far from efficient, but the client could save the list and just scrape the newest until it finds a match with previously downloaded information when checking for new addons).

Be like a child - greedy, self-centered and immune from prosecution.

 

Har

A few clarifications: the management software would be open source, but it would not simply allow anyone to modify an addon's page- only users with admin rights to that addon would be able to. Secondly, I do suggest that we use bittorrent to drive the traffic behind the scenes. Thirdly, if there's no easy way to take a new version of an addon and automatically generate a small file containing only the changes from the last version that could be downloaded and then somehow integrated into the installed addon, then we'll just stick with full addon downloads. When a user identifies an updated version they want to have, our software will archive (move and rename, presumably) the old version and install the new one.

Basically, this project is going to tie together functions that are done in a few different places. Development will still be done with git (or whatever), but once a developer has a version of the addon they want people to use, they can seed it to our users. Our users have our application installed and use it to see what addons (and what versions) they have installed, and automatically download updates. Behind the scenes, every user with an addon is seeding the files to other users who want the addon.

 

Har

Public trackers were an option I considered, but their drawbacks are significant. They're not WOW aware, they can't notify a user when a new version of their favorite addon comes out, they don't install these addons by default, and (most importantly) there's absolutely no reason for a developer to seed his addon files. He makes no money from this, and many WOW users are not the type of people who know how to use bittorrent.

Why would the user periodically search TPB for new versions of their addon list when they could just check it out on curse? It's as manual a process as we have in place right now.

This application would take care of tracking the torrents, notifying users about new versions, installing the addons they download, uninstalling (archiving) the older versions, and most importantly, accepting donations for the developers.

 

Jelloshots

You cite bandwidth costs as a major reason why you'd want a decentralized repository, but it's hard to have some sort of oversight and a lot of your requested features otherwise. Meanwhile, there are plenty of free centralized platforms for freeware and donationware. Using one of those seems like a better option than using some low bandwidth site that taps into public bittorrent trackers for the heavy lifting. The concept is similar, and there's certainly the risk of e.g. Google shutting down Google Code, but it seems worth the risk. The real problem is getting developers to sign on to yet *another* addon portal.

 

koaschten

You could work with a user profile controlled rss/xml newsfeed that the torrent client reads. It would show new versions of the addons he subcribed to, the torrent client would read the feed and download the new version and at the same time seed it once it finishes the download of a new version.

The problem i see with this is, the P2P nature of distribution is it only works if the client is:
- always on
- there is a reasonable amount of users interested in the download, so it doesnt die once the initial downloaders are done
- people willing to have it run more or less 24/7 on their machine, this requires a reasonable amount of trust into the developing community

 

Har

Jello, the main reason I think this project should exist it that it would merge two completely different products: end user addon management, and addon distribution. The free centralized platforms you speak of only do one of these- not both. Getting the devs to sign on to yet another addon portal is a problem, I admit, however since we'd be a revenue source for them and getting their addon made available through our application would be easy, I imagine we'd get most of them.

Koaschten, we would be using the Bittorrent protocol, but not using a standard implementation that shows the "behind the scenes" info like who's downloading what. Users would not have the ability to stop seeding files they download unless they indicate in their setting that they want to be a leecher with no outgoing bandwidth. Even the archived addons would be kept available for uploading as long as the user did not manually delete their archived addons.

Additionally, they way I imagine ensuring that there are always enough peers to make downloads fast would be to set the application up so that when a user turns it on to check their addons and download any new releases, a daemon (or something sort of background process) would remain active for a definable amount of time slowly contributing uploads to the swarm.

 

slant

I always wondered why nobody thought to use APT for WoW addons. It may be analagous to picking a sledgehammer to drive a tenpenny nail, but APT is custom-made for package management, supports versioning, hash checking, dependency management for non-embedded libraries, decentralized repositories, a local database of installed packages, multiple package sources, and so on. It wouldn't alleviate the need for bandwidth of course, but since packages can be spread out amongst multiple mirrors, that would be a much more manageable problem.

I don't see peer to peer as the right solution to this problem. It's dependent on peers uploading, and people tend to update their addons before playing an online videogame. They don't want to leave that stuff running, because it'll hurt their latency in-game. I know I wouldn't leave it running. Also people are selfish and won't upload until forced via a ratio system, and WoW addons are extremely small, and thus not really suited to P2P anyway.

I believe the root of the problem is twofold. If both of these issues are addressed, bandwidth would be a much smaller issue.

1) Wowaceupdater didn't support actual releases. Every time an addon was updated in any way, no matter how small, every single player re-downloaded it, wasting a ton of bandwidth. Curse's implementation fixed that problem, APT would fix it too.

2) All of these package managers are terribly rudimentary and don't support any sort of dependencies. So addons used embedded libraries, which slowed game loading significantly and players were forced to have a couple dozen copes of libfubarplugin, librock, libace, libsharedmedia, etc, on their computers. The biggest problem is that embedded libraries inflated the size of every single addon. APT would fix this too.

All of this pretty much applies to RPM/yum as well, I'm just more familiar with APT. RPM may be a superior solution because the packaging is much easier.

 

sarf

My idea was to have the client scrape TPB, using some nifty prefixes and a particular category for the addons. The client would then present the updated list to the user. This is wasteful of the users bandwidth (and TPBs) as to get the metadata, the client would have to grab the torrent file, but it would make it possible for the user to have ease of use. In order to take care of the initial seeding, it'd be best of the "release addon" functionality was integrated into the application itself, and then have the application running in the background unless told to stop (sneaky, perhaps, but would probably net enough seeders to make it worthwhile).

Your solution seems to imply that you are going for a centralized repository of *some* information, and then using bittorrent mostly to alleviate the patch day problems (please do correct me if I am wrong). *Someone* will have to seed the addon; I don't see why the addon author shouldn't be the initial seeder (and perhaps seeder of last resort).

It seems you want something like freenet coupled with the good parts of centralized file hosting (stable host, ease of lookup and update). This is not a trivial task.

I'd recommend that effort is instead expended upon having wowinterface/curse into providing addon trackers (and automatically seeded torrents of their addons) to handle patch day load as well as adding the BT functionality to their clients.
Perhaps they should nudge people into using the BT way by having http-addons download at capped speeds for non-premium users - it would still handle the patch day load lots better than the current system while still maintaining the centralized portion.

This is mostly unsolicited advice - sorry for being negative to your idea. It is a neat idea, but your scope is a bit too big to be feasible.

Be like a child - greedy, self-centered and immune from prosecution.

 

Har

Slant, I'm willing to concede that p2p might not be the best for this, but I was unable to think of any other way to avoid having to rely on companies that require ad exposures to subsidize bandwidth. I'm not familiar with APT, but if there's some other back end distribution system more effective than p2p, I'll advocate the usage of it.

Sarf, this is all solicited advice

I didn't mean to imply that I was looking for something like freenet- all I was looking for was (as an end user) a free (preferably open source) tool for managing and downloading my addons. I hope this discussion leads to one, but worst case scenario, I'll continue to manage them manually.

 

slant

There are a number of places that will host packages for free. Many of them include an APT or yum source as well as SVN/CVS/git. Obviously none of these services were intended to host gaming addons, and I would certainly talk to them before hosting my addon there, but as long as the project adheres to an open-source license and source code is freely available I don't see why they'd have any major objections. These repositories and services are supported primarily by donations and angel investors, so their bandwidth isn't by any means free or limitless, but they have a great deal more resources than your average fly by night gaming site. And keep in mind that by addressing points 1 and 2 above, several orders of magnitude less bandwidth would be required.

 

Zakariya

You might consider looking at GitTorrent.

From their homepage:

It seems to be just the thing you were looking for. Of course, it's highly alpha... but holds some promise.