Exactly what right would I be violating by demanding information from you?

The only right that I can think of is the right to own property. However, that is a very tenuous link