This is something that’s a minor annoying thing; as the title says. It’s just generally annoying to have files there that don’t belong to anything. Could it, in the future, be added as at least an option (which can be set to stick/always remember).
Early 2021 clean-up: duplicate