Remove Memoji from my Apple ID

Hello,

Is there a way to remove Memoji from my Apple ID completely? I have changed my Apple ID Profile Picture in Settings > Apple ID to a picture of my real face, but whenever I iMessage someone new and it gives me the "Share Contact?" prompt, it offers to show the other person my old, unwanted Memoji I made in 2018-2019 instead of my new profile picture.

I am currently using an iPhone SE (2020 Model) on the latest iOS (currently 16.2). I have not used Animoji on my Apple ID, if it matters.

Thanks in advance.