If I leave Family Sharing, would the organizer be alerted?

I don't know about an active notification as such but an observant person would notice your name had disappeared from the list:

This document has an image of a family list on an iPhone --> Set up Family Sharing on iPhone --> Set up Family Sharing on iPhone - Apple Support

yes an alert will pop up on the organizer's phone