Unable to change the default downloads location on Safari - macOS Sequoia 15.5

I am trying to change the download location on Safari from 'Downloads' to 'Ask for each download'. However, as soon as I click the later option, it immediately changes back to 'Downloads' without showing any error or a hint. I do not want 'Downloads' as my default location for productivity purposes. Help would be appreciated. Let me know what you need.