How to clear saved colors?

I know I have been able to clear or reset saved colors in the palette window (see attached image for reference) before by simply selecting the little square and striking the delete key but not having luck this time.

Would anyone know how to do that?

macOS Sequoia 15.0

Pages 13.1

Thank you.