One alternative to patents and prizes not often discussed is just to keep your innovation secret. But is it worth it? Secrets are costly to keep and can be found out.
A forthcoming paper, "Keeping Secrets: The Economics of Access Deterrence" by Emeric Henry and Francisco Ruiz-Aliseda, in the American Economic Journal: Microeconomics looks a this question.
The abstract reads:
Keeping valuable secrets requires costly protection efforts. Breaking them requires costly search efforts. In a dynamic model in which the value of the secret decreases with the number of those holding it, we examine the secret holders' protection decisions and the secret breakers' timing of entry, showing that the original secret holder's payoff can be very high, even when protection appears weak, with implications for innovators' profits from unpatented innovations. We show that the path of entry will be characterized by two waves, the first of protected entry followed by a waiting period, and a second wave of unprotected entry. (Emphasis added)Payoffs to keeping a secret can be high even in a world where protection looks weak, so sometimes shutting the hell up is the best policy.
Of course this can't be the perfect solution to rewarding innovation since firms can always try to keep their developments secret, but they don't. In many cases they do takeout patents.