Very occasionally, a publisher will ask a writer it has under contract to pay back the advance that was paid.
This might happen if, for example, the deadline passed months ago but the writer still shows no sign of submitting a finished manuscript; or if the writer submits a vampire novel about World War I instead of the non-fiction book about the history of cockle fishing off the Welsh coast that he was contracted to write.
If a book is published but fails to sell in any significant numbers, though, the advance is the author's to keep. Even if the book doesn't earn out. Despite what many vanity presses insist.