Grounds is a very funny word. In this case it is most clearly linked to the legal sense of grounds, as in grounds for arrest, grounds for divorce, etc. But where does that come from? A sort of metaphorical sense? That you must be able to stand firmly on the thing – that you need ground enough to stand on?
The college in my hometown calls its campus The Grounds, which gives it a sort of stately mansion feel, as one does tend to wander the grounds of a nobleman’s estate.
And then, there are coffee grounds, as in something that has been ground up, I guess. That is probably not a place where you stand.