Are you running a Managed or a Referenced Library?
A Managed Library, is the default setting, and Photos copies files into the Photos Library when Importing. The files are then stored within the Library package
A Referenced Library is when Photos is NOT copying the files into the Photos Library when importing because you made a change at Photo -> Preferences -> General. (You unchecked the option to copy files into the Library on import) The files are then stored where ever you put them and not in the Library package. In this scenario you are responsible for the File Management.
It sounds very like you're running a Referenced Library. Are you?