I tried this and couldn't see a problem, except for the subtle way the color picker works. I'll try to explain, and you can see if that covers your situation.
To set a color for an object, use Inspector -> Format -> Style -> Fill

I assume you're clicking the color swatch at the bottom of the highlighted area to bring up the color picker?

What's not entirely clear here is that this picker has two states. I don't know their technical term, but I think of them as 'locked' and 'unlocked':
Locked:

Unlocked:

The 'lock' state is toggled by clicking on the color wheel itself.
Clicking the lock state button opens the color picker (if it's not already open), which makes most people think that is what the icon is for, but that's only half the story.
The reality is that picking a color in the color picker ONLY applies that color to the current object IF the color wheel is in LOCKED state. If it's unlocked you're free to choose any color from the picker, but it won't be applied to the current selection.
For example:

Here, the picker is locked, so selecting a color on the color wheel (or using the eyedropper) will apply that color to the selected item (the square).
If I UNLOCK the color picker, I can now choose another color from the color wheel (or eyedropper), but it is NOT applied to the selected item:

Does that match what you're seeing?