Is this server publicly accessible? If it’s only connected to your private network, I personally would not firewall it off like this. Also, you can just turn off SSH if you are not using it. Another option would be to only allow key based authentication.
Regarding your UFW setup: Could you post your rules/the output of ufw status verbose?
The server is used by my family so it does need to be accessible remotely.
I didn’t think of just turning off SSH, thanks.
After doing the verbose status, I realised that although I specified to Allow all outogoing connections, I didn’t realized that Incoming was set to deny by default. After changing that to Allow everything worked.