It varies widely. Usually these deals are structured so you pay the licensor a royalty, so they get a slice of each sale you make. Most commonly, you then pay an advance against royalties. How big the royalty and advance are depend on the power of the property and the scope of the license. So when WotC made that deal for Star Wars, they were getting RPG, miniature, and card game rights. That was a broad license of a big property so their advance was many millions of dollars. Advances that big are not common for RPGs. They are usually in the thousands and mostly less than $25,000.
Green Ronin Publishing