Mainstream, commercial publishers will publish just about anything they think they can make money on. So when a book is widely rejected (assuming it’s been submitted to the right people and places), it’s because the people who know and understand the market best consider that it’s not commercial enough to sell well. That it’s not going to appeal enough to readers to sell in any quantity.
If a writer then goes ahead and self-publishes the book, chances are it won’t sell many copies. Not only do they have a product which has already been judged insufficiently commercial, they won’t have the same support that commercially-published books receive—no editorial advice, no marketing clout, no sales team, and no promotional material or expertise.
Because they have no editorial support, their book is unlikely to be edited as well as a commercially-published title. Sure, they could get the book worked on by an editorial agency: but this edit is unlikely to be carried out in much detail, and might well focus more on typo-hunting than real line-editing.
Because the first-time self-published writer has no reputation for producing good books they are unlikely to be able to attract the attention of any of the major reviewers, and so that particular promotional route is closed to them.
So long as they have an ISBN, their books will be listed with Gardners and other wholesalers: but they’ll stand very little chance of getting any real distribution for their books; and even if they manage that, they’ll have no sales activity other than that which they carry out themselves—which means they’re restricted to sales in local bookshops only, with little hope of national stock placement. It’s not surprising, then, that few self-published books sell more than fifty or a hundred copies.
The suggestion is often made that self-published books might achieve better sales figures if given the same sort of support that is routine for a commercially-published title.
While some might, I doubt that many would. I've seen a lot of self-published books over the last few years and it's clear to me that most were rejected by the commercial presses because they simply weren’t good enough to command a decent level of sales. I don't think they'd sell many more copies if given more support: I'm just surprised that so many have managed to sell more than five or ten copies each, given their general lack of expertise involved in bringing them to publication.