Quick Time Player problems

I assume you did launch QuickTime Player from the Dock, Launchpad, or from the Applications folder.

When you cancel the Open dialog box, then QuickTime should still be active, but without any windows (possibly showing other open windows ‘behind’ it), thus only the Menu bar as visible clue. Clicking anywhere else may bring one of the other apps to the front and send QuickTime to a lower layer.

If QuickTime Player is no longer the front-most application: Command-Tab (⌘⇥) should act as an application switcher, allowing the user to switch to any other currently open application; repeat to go from app to app, thus returning to QuickTime Player if needed. Alternatively use the Dock, or the initial way of opening the application.

To start a recording, go to the File menu, and select New Screen Recording.

The selected sidebar shortcut looks for QuickTime documents in your iCloud space. You would get an empty window if there are no such documents there. If you don’t want to open an existing document, then just Cancel the Open dialog box and do what you wanted to do (e.g. any function from the menu bar). When you do want to open an existing local document, then select a local option in the sidebar first.

Apparently your Mac is currently set up to use iCloud for your Desktop and Documents folders.

If you do not want to use iCloud for your QuickTime documents, then you may disable that in System Preferences (10.12)﹥iCloud﹥iCloud Drive﹥Options﹥QuickTime Player, or System Preferences (10.15)﹥Apple ID﹥iCloud﹥iCloud Drive﹥Options﹥QuickTime Player.