Mac Mail "Priority"

I would like to remove the automatically appearing "priority" drop down when composing email. It never used to be there, and is now persistent, regardless whether I remove it from the "header fields" dropdown. How do I set Mail to now show the priority unless I specifically select it?