How can I stop others from interfering with my phone?

When I’m out and listening to music I’ll experience a lot of annoying things that make me think somehow other people are controlling my iPhone - not intentionally, but it seems like it’s receiving commands from elsewhere. For example, the volume will go up or down; the songs will skip; and most weird - voice control will open up and do weird stuff. Had this happened to anyone else? Do you know how to stop it? No one seems to be accessing my personal data or anything, it just feels like crossed wiring or something, but its really really annoying. Only happens in crowded places like shopping places and the bus.