Thinking about the architecture of a prison makes this line make a good deal more sense to me. The world is a goodly prison, a big one, like Alcatraz or The Tower of London. It’s not just one cell, it’s a whole complex with many places to lock a person up. You got your solitary confinements, your dark dark dungeons, your cell blocks, your minimum security, your maximum security. An old school prison probably has a great deal more lock-up variety than the modern prison or even the pre-fab office buildings which so often remind one of a prison.
I bet there are even some prisons where parts of them don’t even feel like prison. A garden perhaps, or a library. So one could live in a prison of a world and never really feel confined.