Big files (larger than 2GiB and than 4GiB) works fine; that has been true for about a decade now.
As geek said: Any process can have at most 4GiB address space; in practice it's a bit less, because some is reserved for libraries, mmap, and such. For most people, this makes no difference, since very few people (outside HPC and crazy scientific computing) run programs that use more memory than that.
The real hardware limit is, as said above, a difficult question. Without PAE, it is actually not even 4GiB; on my Atom system, I only get 3GiB recognized of the 4 installed in hardware. And PAE used to be gambling, with restrictions and frequent crashes. I think it is more stable now, but I haven't started using it (my system works so well with 3GiB, why put any effort into messing with it).
ZFS works fine on these systems; I've been using it for years, and have not encountered any problems. Follow the instructions in the handbook for low-memory configuration of ZFS. There are rumours around that beginning with 11.1, ZFS has become even more memory hungry; I've not seen any issue from that. Performance of ZFS is appropriate for a home server (depending on workload, it ranges from 50 MB/sec to being disk limited).
My home server is a 1.8 GHz 32-bit Atom (an Intel D525, on a JetWay tiny motherboard), and it does home service just fine: Firewall, PF, routing, network services (DNS/DHCP/SMPD/...), a handful of file systems (5 disks total), NFS, and similar things. Neither lack of memory nor lack of CPU speed are a problem for it at all. This is not a giant cluster for HPC, but a system the size of a shoebox.
EDITed to add an anecdote: Two weeks ago, there was a large wildland fire near our house. So near that the fire department ordered us evacuated, since they had given up on defending our neighborhood. One of the first things that went into the trunk of the car was the external backup disk (that went at the same time as passports, credit cards, and other such documents). Since we had another 5 minutes to spare, I ended up putting the whole home server in: if our house burns down, that would save me many hours of having to buy a new server, reinstall, and restore the backup. Unfortunately, the next thing I threw into the car was a box of cookies and a case of soda (in case we get stuck in a hotel parking lot overnight, we'd have something to eat and drink). That's unfortunate because servers have corners and sharp edges, which caused one of the cans of lemonade to explode, and spray sticky lemonade all over the trunk, including the server. BIG OOPS! DON'T DO THAT!
So after we evacuation was lifted (fortunately, none of the houses in our neighborhood were damaged, except for heavy smoke smell for a week), I took the home server apart, and carefully wiped it clean. Fortunately, none of the sticky sugary lemonade got on the motherboard, fans or disks; the external connectors I cleaned with Q-tips, water, and isopropyl alcohol. Works fine again: uptime is now over 14 days, so it has not gone down since we got back home.