The OP's system is involved, but I wouldn't call it elegant. It works on the model of Strat/BRS having player caps reduced, which admittedly is one way of handling content accessibility for larger playerbases. This motion in fact sounds quite similar to cries for 5-man raids years ago. Blizzard gave us Heroics in response. And that worked quite well, until they nerfed heroics so badly. Perhaps part of the solution to "challenging 5-mans" is to revert their difficulty. Possibly add another difficulty tier, similar to the Arena style solution hinted at. Add "Legendary" difficulty above Heroic, make everything hit harder, give the bosses some new tricks, raise the ilvl and quantity of epic drops including gems, and drop 2x badges/boss. Continue the trend every so often as raid/PvP gear outpaces the 5-man selection.
As for accessing lore, wait till the expansion, get to 80 and 5-10 man the old lvl 70 25-mans. Just pray Blizz doesn't break something like the Emps and not care enough to go back and fix them... Casual players without the time or desire to be "bleeding edge" will not access elite content when it comes out, when it's still challenging. That's fine. Otherwise, why would it be considered elite, what would the bleeding edge strive for? The player-cap reduction system the OP proposed would actually be a big hit on more casual raiding guilds like mine who are "months behind" world-first guilds. On our relatively relaxed schedule, "Age n" would have come and gone before we ever got around to accessing the content in true raid format. The existence of 5-man versions (even if taking it on 25-man was an option) would belittle our desire to complete it. We in turn would be punished simply because our timetables and compositional resources limited the rate of progress compared to world-first guilds. Even the OP acknowledges any system of making content accessible to a wider mass should be done over time. Waiting for the expansion to raise the level cap should function just as well for the most part, and requires virtually no backwards resource allocation on Blizzard's part. You want to see Illidan the first week he's out? Raid everyday on the PTR when he's being tested, then spend 24/7 downing him the first week he's Live. You want to see Illidan eventually and are willing to put in the time and effort to 25-man him? Keep at it, and you'll see him in a few months. You want to see Illidan, but can't or won't put in the time to 25-man him as he's intended? Wait till the expansion for it to become legacy.
As for 15 hours of raiding, not all of our raiders can stay the full 5 hrs/night. Some can't make it all 3 nights. Some even take entire weeks/months off. Our leadership works around that, and yes it does get frustrating, but results are achieved. 1-2 hours worth of dailies, farming/grinding are enough to cover raid costs. We even offer a standby system which allows excess raiders to go and farm during raid times while still earning credit. While we certainly ask that our raiders reserve some time for us, we expect them to do it efficiently. That's simply not something you can expect from rag-tag pugs, and I see no convincing reason to mold elite content to their needs. You don't have time for 25 mans, you don't have time for it. 5-mans exist for you. You want a challenging 5-man? Ask for harder heroics. Ask for new 5-mans everyone can enjoy.
Expansion content now includes:
# 4x 25-man raids (4 - 9 bosses)
# 2x 25-man encounters (1 - 2 bosses)
# 2x 10-man raids (6 - 16 bosses)
# 15x 5-mans (2 - 5 bosses)
# 15x heroics (2 - 5 bosses)
There already exists quite a lot of content accessible to a wide playerbase compared to limited content. I for one don't buy the argument that Blizzard ignores the casual masses in favor of raiders. Also make note that all 15 5-mans + heroics were available for enjoyment on release, while 2 of the 25-mans and 1 of the 10-mans had to be patched in over long spans of time, bugs and all. Perhaps it would feel like 5-mans were better catered for if Blizz had only released 2 per hub and turned on the remaining over time? Nonetheless, I wholeheartedly encourage asking Blizz for more new 5-mans. Emphasis on new, not backward capped. There are still several zones waiting for fresh instance hubs to unlock lore and excitement
I know I may come off as an elitist jerk with too much time... but I've run the gauntlet from having nothing but 80+ hrs/wk of playtime, to barely having 1-2 hrs every other week, to taking months off. I didn't expect to raid 25-mans when I didn't have time, despite my guild asking my under-geared toon to tag along. I quested, leveled alts, did dailies, occasionally 5-manned, chatted, and sometimes simply enjoyed the view. And I enjoyed it quite well. When I wanted to see Vashj, Kael, Archi and Illidan, I put time into the guild and game. Or rather, when I had time to put in, I decided I'd like to work towards seeing them...