That's a nice (but not really) way of calling something self-published.
It's a euphemism for what you really want to say:
"Oh, that's cute, no 'real' publishing house wanted your shit so you self-published it, huh?"
As in, the only people with the nerve to self-publish are those not skilled enough to write something commercially worthwhile yet too vain to realize that.
This might have been true years ago, but no longer.
In fact, self-publishing is anything but vanity nowadays. Self-published authors are typically some of the most interesting, hard-working, boot-strapping artists out there. From having to find editors, to designing a cover and interior, to acquiring copyrights and ISBNS, to getting the book up for sale online and in stores, to figuring out how to market the work and perhaps even sell a few copies. Investing their own money and working without the net known as an advance. And I haven't even discussed the actual writing of the damn thing.
Self-publishing isn't vanity, it's a labor of love. It's done by people that truly have something to say and would die inside if they weren't allowed to say it! Even if they don't expect to make a buck or garner any fame doing it. Is that vain? I sure wouldn't say so.
You want to discuss "vanity"? How about Gallery publishing a book penned by Snooki. St. Martin's publishing books by the Kardashian sisters. Or pretty much every "important" publisher under the sun releasing ghostwritten books for Donald Trump over the years. These aren't worthwhile books, they're nothing more than commercial packages "written by" vain people to further their fame, published by publishing houses to make an easy buck.
(If the person that "wrote" the book is pictured on the front of the book, it's usually a true vanity project. Think of the book as a mirror, with Snooki or Khloe Kardasian or Donald Trump holding it up to their face, admiring the cover of something they'll never even read.)
Snooki, the Kardashians, Donald Trump, none of them would ever have the balls or chutzpah to self-publish. They wouldn't even know where to start. If we can even assume they could come up with a unique idea then hole themselves up for a few years to write the damn thing, it would still be hard to believe they'd figure out how to accomplish everything else to get the book to market.
But for them to get a book out there from a major publishing house, all that involves is them signing their name to a contract (or writing an X in Snooki's case), smiling wide for the cameras at a few signing events, and letting a huge team of people at the publishing house do all the rest of the work.
Now how vain is that?
My "vanity" work: