It seems this was a bug in iOS 16 beta 1 which Apple fixed with beta 2 – at least that’s what 9to5mac.com writes on their homepage. According to that, it only effected TestFlight apps.
Update (June 22, 2022):Apple has fixed this bug with iOS 16 beta 2. Users can now run TestFlight apps without having to enable Developer Mode. You can read the original article below.