You seem to be shocked that games from niche genres don't have monsterous budgets. As it hasn't been this way in pretty much every single entertainment medium.
Modern niche games often have bigger budgets that what you considered "large" in late 90s. However around the time Half-life debuted the acceptance for niche dropped to hell. People were stiill making them, but in majority of cases those titles turned into swan songs. One niche game after another was launching and then flopping epically, dragging it's devs down. Nowadays niche games are doing fine, while back then they often led to death of the developer who was making them. The moment FPS and RTS populairity reached their peak was the moment everything else started to slowly die out.
...
They did tempt devs, but what's more important is that they tempted gamers. Every market has limited ammount of money in it. As long as there are only small fishes in that pond it's ok, but once you put couple giants in there the food is no longer enough to sustain the whole ecosystem, the smaller predators die out. Eventualy the big fished grew into sizes that were too big even the whole pond, so they needed to jump to another to keep growing.
However once they left, there suddenly was once again plenty of food there, thus the population of small fishes started to slowly getting rebuilt.
If you look at sales of big PC hits you will notice that before late 90s it was rare for a game to sell many millions of copies. When million or even multimillion sellers started to become more common, they simply have eaten a large portion of how much money there was left to be spent on less pretty, more niche and less advertised products. PC market was constantly growing during 90s, but not fast enough.
...
DD helps with two biggest hurdles for any niche product: avaibility on shelves and profit margins. This along with less competition from giants has created much more friendly enviorement for independent devs (I don't mean just indie, but broadly those devs that aren't owned by everybody else) and it shows. Look at it this way, on consoles DD is much weaker and there market is filled with big budgeted games, have you noticed that in few shorts years independent console devs became almost extinct? While on PC they still form the majority of active studios.
...
You see, this is where I completely disagree with you. Modern niche games have bigger budgets and far bigger production values than they used to have ten years ago. To me that's a good thing, while to you it's a huge problem that niche games don't have 20+ mln budgets. I don't understand where did you get the idea that niche games should have non-niche budgets. It doesn't make any sense and it's completely out of touch with reality. Did you complain that Donnie Darko, Dr Terrible or Memento didn't have 100+ mln budgets either? Or that a small theatre doing difficult niche plays can't afford to have Broadway-quality of costumes and big-name actors? In every entertainement form it's accepted as obvious thing that if you make something niche you can't get the budget mainstream provides.
Even those niche genres are more popular now than they were in their "golden eras" or ten years ago. More people play those games than ever before. The only difference is that other genres' popularity grew faster, that's all.
In late 90s and early 00s the only reason why you saw plenty of games from many genres is because devs were too blind to see that by making them they were killing their company. There never was a period where FPS/RTS and niche genres were both hugely popular. Once the former became popular the market turned to slaughterhouse to everything else. To me it's not "PCgaming at it's best" when niche releases mostly meant death to it's devs.
...
We already are way past the point where tech was the primary apsect that determined production values of games. I can directly compare production values and budgets and see both are higher now even for niche. I can't see how can you assume that in market that's fully commercionalized and valued in dozens of billions of dollars there would be no budget differences between products that are aimed at smaller audiences vs ones that are aimed for mass public. THose less mainstream genres are doing now much betteer than 10 years ago, they sell better, they are more popular than ever, they have bigger budgets and production values than ever before. They grew a lot, you're just complaining it's not enough and they should have grew as much as mainstream genres, despite the fact that you won't find a single market on Earth that would develop in such way.
...
I'm not comparing them to 1994, I'm comparing them to 2000 ones and I don't see why I should expect budget of niche games to rise as much as those for more mainstram types. Especially since those types of games never were about pushing the boundaries of what can be done with graphics anyway, so there need for budget growth is much more limited than in action games. Similiarly to how it happened in movies. Obviously budget for small dramas did grew, but not anywhere near as much as they did for blockbusters filled with insane special effects.
I just don't understand why do you assume every segment of the market has to grow exactly at same rate in both popularity and resources being spent on projects to not be considered as "having a problem".
You need to stop imagining those titles ever were "large scale productions". They never were, they always were niche and always were small productions by any rational means. They just stayed that way and continued to grow inside their own niche, while we also have gotten pc titles that managed to grow out of their own niche and became mainstream. But just because other games didn't achieve that doesn't mean they are in sorry state They are doing better than they were 10 years ago and that's a simple fact. The only difference between us is that I'm satisfied with just that, while you think they didn't grow anywhere enough.