Does Osric think that Hamlet hasn’t understood the proposition? Is he genuinely trying to clarify the wager? Or is he just flustered by Hamlet’s refusal to play the game? Of course he doesn’t have an answer for what would happen if Hamlet answered “no” – finding a response to that question would require a much higher pay grade.
It’s like asking the most rule-following cultural-norm-fulfilling, rote society-participator what would happen if you broke the rules; he doesn’t know and cannot even begin to imagine a world where people don’t follow their expected roles.