It is not as simple as that. If one has physical access, one could in principle access the content of the drive.
One could, for example, start the Mac in Target Disk Mode, connected to another Mac, and have basically access to everything in it.
I prefer not to use whole disk encryption, because I value more my own access to my data, compared to the possibility of someone else getting access. But I understand that different people have different security needs, and sensitive information needs to be protected. Disk encryption makes it nearly impossible for a third party to get to the data without the encryption key, even if they have physical access and all the time in the world. As they say: priceless (in some cases, at least).