Just tell them to mind their own business. It isn’t their business to know anything you don’t want them to know. I would just shrug if off and not respond to them.
People will think whatever they want – rumors and all. At the end of the day, it will pass and something new will come up as the next rumor.