Safari Shared for you links won't remove

So I have an issue where I can't remove the shared links under safari when I hold down on the link and click remove, it doesn't delete it. I know I can disable shared links completely for safari, but I still want that feature, but just want to get rid of those that I don't need.