When a writer is signed up to a commercial publishing house, the contract that both parties sign will set out in some detail the terms of payment involved.
The first thing to remember is Yog’s Law: money flows to the writer. So the contract will specify how much money the publisher will pay to the writer, and when: if any mention is made of the writer paying the publisher, this is probably a contract that the writer should not sign.
The money will almost always be paid as a royalty on books sold. So the writer will receive a royalty from of every single copy of the book that the publisher sells, the amount of which is determined by the terms in the contract.
It is usually best for the writer if the royalties are calculated as a percentage of the cover price of the book, rather than on profits, as profit can be difficult to define: for example, if a particular book is selling well but its publisher occupies over-large offices, then those book sales might well end up subsidising the over-large offices instead of earning their author royalties.