What is a setup? What is a payoff? Actually it’s easier to define the payoff first. A payoff is a story event for which the main character (and the reader) need to prepare. If your fantasy novel has at its conclusion a sword-fight between your hero and the Dark Lord, you need to set that up. Your hero needs a sword, and needs to know how to use it. Perhaps he needs a magic sword in order to win a fight with that particular Dark Lord.
Payoffs don’t always happen at the conclusion of a novel. Perhaps in your historical novel your heroine needs to be put into a position where marrying a (handsome) stranger seems like her only choice. This marriage is a payoff, though it will probably happen closer to the beginning of your novel. To set up for that payoff, you may perhaps need to turn your comfortable middle-class historical heroine into an orphan, and have a loathsome step-brother offer her a position as an unpaid nanny in his home. Perhaps the step-brother tells her that is her only choice to avoid ending up in a workhouse or worse. That kind of setup would make a marriage to a stranger seem like a plausible choice.
Setup scenes make the reader expect a payoff. If we are shown a firearm in an early scene, we expect that firearm to be fired in a later scene— not necessarily a self-defense or murder type firing, it could be hunting or target practice. But if something is mentioned in a significant way, we expect that it is a setup that will be paid off later in the story.
In the same way, big payoffs in a story have to be set up. Is your character going to undergo a major change— perhaps learning skills or becoming more independent? You can’t just state that fact in your climatic scene, you have to set it up in earlier scenes, or the change seems unrealistic and unmotivated.
Chris Fox says that to avoid ‘plot holes’ you have to pay off your setoffs and set up your payoffs. Setups make your future payoff scenes seem less random and ‘out of the blue.’ Actually having the payoff happen means that those setup scenes had meaning for the story as a whole.
One way to start thinking about setups and payoffs is to think about your climactic scenes. What payoffs will happen at that time? What might you need to do to set up those payoffs? And if you have a scene, often near the beginning, that feels like a setup, think how that scene can lead to a payoff. (If your scene sets up something that will never lead to any kind of payoff, it may not be an important scene for your novel.)
Assignment: Read a favorite novel and note scenes that seem to be setups or payoffs.