This arguement has been going on, and will go on for ever. In my view it's demand first that influences price, since if nobody wants a record, no matter how rare it is, then it has no value. Demand is obviously created through exposure of a record to an audience, could be that demand is based on quality, or rarity, or both, if it's both then the value rises accordingly, and that means to the price that people are willing to pay for it. We're a mad, competitive bunch, and more often than not it's the need to have a record ahead of anyone else drives the early pricing up.
It may well be that more copies of a rare record turn up over time, but then the demand will also increase over time, but then the demand usually will peak, a record will get tired, fall off playlists, and drop in value, and it's true worth will bottom out, that's when you find that you "rare" Ace Spectrum that you paid 50 quid for is only worth a fiver. On the other hand if the record is actually both good and rare - Parliaments - Cabel, then the vinyl will hold or improve it's value, it's the risk you take
As Chalky says, there's some great tunes out there valued at less than 100 quid, but that's only because they're not widely known, and demand is low. You have to get to 'em early, there's no point in wingeing about the price of Ruby Andrews or any other mass appeal tune if you've missed the boat.