As someone who ran Napp-IT on OmniOS (and other OS’s) for many years on multiple servers, I can appreciate the benefits of ZFS. I really do for the most part like ZFS, but to say that ZFS is completely rock solid isn’t the entire truth. Google “ZFS data corruption” and start reading. I personally have had a number of data integrity issues over the years. I came very close to losing significant data a handful of times. Most of the time I was able to Google-Fu my way through it but a few times I had to engage paid professional services. The last and final issue I had ate up many days worth of time from myself and multiple online experts and even OmniTI could not get my data back.
That last issue was the final straw on my back. Yes overall the number of problems I had with ZFS were few and far between. Yes, I could have uptime in the years. But when a problem DID occur it had a very high chance of being rocket science and I felt like I was walking a thin edge which if I failed I would lose data.
Before you go down that path ALL of my servers are enterprise grade (mostly Supermicro, I love SM, but I have HP, Dells and a few IBM/Fujitsu bits) all with ECC ram and enterprise hard drives/SSDs. I do run a few non-server grade boxes doing a few things here and there (mostly Dell Micro 7020/40s) but those are not running ZFS.
So in the end, I moved mostly from ZFS. I’ve tried a number of solutions, some better than others but over the last few years I sleep better AND I haven’t lost a single file. I do value my data, some of it more than others and things like my wife’s dissertation, research, and family pics most of all. I’ve learned that depending on ANY solution (even enterprise ones like ZFS) isn’t enough. All my PC’s run on mirrored drives, these are incrementally backed up onto designated NAS boxes with RAID6 (or equivalent) configs. Anything that is important is then backed up onto Google Drive and Backblaze B2.
So yes I can sit here on a high horse and tell you that it really just comes down to how valuable your data is to you. If you don’t value your data VERY much then run in on ZFS. But if you REALLY value your data then use the formula of 3+ copies = 1+ copy. But then I would sound kind of like a bit of a jerk.
Maybe you have just had some bad luck or I had good luck because I have been using ZFS and Freenas for over 6 years, since the 8.0 days and had very few issues when it came to data integrity. Don’t get me wrong there have been issues with Freenas throughout the years, but when it came to my data that is one area that I had not experienced any problems. I have gone through multiple setups, arrays, and replace multiple failed drives over the years without any issues. I only use RaidZ2, which really should be the minimum, RaidZ is really not redundant enough.
Recently I even upgraded an 8 drive array to larger drives replacing each one until all have been re-built and then the array expanded automatically. This went with out any issues. So why yes there can be issues with any filesystem, from my own experience ZFS is pretty darn stable, but nothing is completely rock solid period.
I also did not use Freenas 10, I went right to 11 from 9. That is the one area that iXsystems really failed on badly, and the whole situation was poorly handled. Thankfully Freenas 11 seems to be prioritizing reliability/stability over features, for the most part.