Though you'll probably disagree, I think that it was largely due to the reassertion of the mainstream record industry's control (or the guys that ran it - the 'whiteification' of Motown management following the move to LA being an good example) over the minors and RnB/race-based sub-labels that led to a large extent to the watering down of the 'proper stuff' towards lowest common denominator vinyl mush aimed at maximum general chart success - as it did in the disco era (with the changeover from gay and hardcore to mainstream and fucking Abba, Bonnie Tyler and the BeeGees) and as it has done in recent years through Syco, One Dimension and the like... All those 'controversial'/emotionally demanding soul recordings (by fabulously talented artists, writers and producers that only sold in limited numbers into the ghettos and to relatively tiny numbers of enlightened white fans) dumbed down into musical magnolia, with the result that the old hardcore fans lost faith, whilst the pop buying public had limited interest because the more mainstream releases STILL weren't bland enough.
Dx