I've used ports on one machine, packages on another one, and I've also tried mixing them.
Packages are not very up to date. This means that you don't have the latest version of the program, which in it self is not that important, but what is important is the security. Many of the programs I install using packages have already security holes in them. If the programs don't have any holes when I install them, they will sooner or later end up with one or two, and waiting for someone to update that package can take ages. I always have to solve this problem by replacing the package with a port, since they are updated quite fast.
If I don't update the server to the latest version all the time, pkg_add will start having problems finding some of the packages (this can probably be fixed by setting the PACKAGESITE variable). It also seems like many of the mirror serves out there are not really mirrors. They don't have the same packages as some of the other servers.
Ports are always up to date and are updated as soon as a security hole has been found. If I only use ports I rarely see any trouble with my applications or updates I do. If I start mixing, or use packages only I do believe I get more errors and problems than if I had stayed with ports only.
Compiling is the biggest problem, it takes time. Sometimes it can take half a day if you have a slow computer, whatever a slow computer is these days
Another thing with ports is that you can add or remove support for certain things, while this is not possible with packages since they are precompiled.
People also talk about optimization when using ports, but I've never really noticed, nor tried to notice any difference.
Personally I try to only use packages if I'm in a hurry and just need to test something. I wish I could use packages only since I find it rather.... time consuming and a bit 80's to have to compile for hours before you can use a program.