This is one of those lines that Hamlet can mean one way and Rosencrantz and Guildenstern can interpret another. It makes sense that Hamlet would be talking about reason as it relates to the little intellectual game they’ve been playing around ambition, etc – that he cannot reason anymore along those lines. And it would make sense that Rosencrantz and Guildenstern, primed to look for Hamlet’s madness, would see this as an admission of his mental difficulties. It might inspire a knowing look between them, which in turn might explain why Hamlet doesn’t take them straight to the court as he just suggested he would but starts to investigate their motives. And thus this transition in the scene, which has baffled me before suddenly makes sense. This may be why that trip to the court stalls for a moment.