Solved Where should application data folders go?

lloyd

lloyd

Where should the data folders be located for programs like Syncthing, Apache/Nginx, Lychee (self-hosted photo galleries), etc.?

These programs generate large amount of non-user-specific data. I'm trying to learn and adhere to hier(7) by creating a folder in the right place, but I can't quite find this specific answer.

From what I've seen, programs sometimes use /var/ and other times /usr/local/:

MySQL/MariaDB/var/db/mysql/
Nginx/usr/local/www/nginx-dist/
Apache/usr/local/www/apache22/data/

So for something like Syncthing, should the share data folders go in:
  • /usr/local/syncthing/shares/
  • /usr/local/syncthing/
  • /var/syncthing/
  • ...Somewhere else?
I like /var/syncthing/ because it's clean and simple, but I would appreciate to hear what others with more experience think.


P.S. Why am I asking? I want to contribute to documentation, so knowing where folders like this should go helps me write better guides.

P.P.S. This is my first post and I'm trying my best to learn and follow the forum rules!
 
A lot of programs put "shared stuff" in /usr/local/shared/nameofprogram if it's data to be shared between all users of a program. Examples: /usr/local/share/gimp, gtk-doc, etc. Often documentation for or icons will wind up here.
/var/db/mysql is I think the default location of the actual databases, but that can be controlled by a config variable (I think).

To my understanding, /var/db is for things more "system base" related. NTP, Ports, Pkg, etc. If Syncthing is a user application (Port or Pkg) it's shared data probably should not go on /var, I'd guess either /usr/local/share/syncthing or /usr/local/var/syncthing.

"Syncthing" what is in the "shared data folders?" On your system go to /usr/local/share and look at a few of the directories there to get a feel for what kinds of things are there. That may help you decide.

Above is my opinion, I defer to others that have a say in where things go.
 
Things that are mostly static and don't change tend to go in /usr/local.

However things that do change, such as the MySQL database files, then generally go in var (along with domain sockets, etc).

So a /var/syncthing is probably a good bet.
 
I like to use /opt for softwares installed outside the ports/packages system.
I feel it is necessary to keep an eye on applications that are not updated by pkg.
Is it evil to use a Linux naming convention for optional softwares?
 
I always stick to the hier(7). The correct one to be exact, i.e. FreeBSD hier(7). If I'm working on an OpenBSD machine, then it'll be OpenBSD hier(7).
There's one exception. Suppose, I have a workstation (or server) which it's all about doing one special thing, let's say "audio codec transcoding".

I have a large amount of files with complex hierarchy (many branches and sub-directories), contained in one parent directory.
* Uncompressed Original files. (AIFF)
* Intermediate files
* Separate meta data files, and related transcode/batch scripts
* Separate uncompressed target (WAVE) and compressed target (MP3)

+ Sub-directories for Artist, Album, etc.

I need them to stay together in one master directory.
The size is massive -- maybe, it's better to keep it on a dedicated HDD.

This unit needs special
* permissions (rwx, setuid)
* mount-ing flags, e.g. MNT_NOEXEC (what about scripts?!)
* Some tunefs(8) tweaking.

Sharing
* Constraint on how to share it over the network
* Move it to another OS, temporarily or permanently.

Backup/Update
* OS upgrade? keep them out of way.
* Dedicated backup-strategy.
* Keep an optimal size (partition): to reduce backup/dump/fsck time

To keep it simple in such situations, I deploy a strategy similar to the Phishfry 's:
Using a dedicated partition, and mounting it on its own directory/mounting-point. Then, set whatever permission/flags/etc, which is necessary,
 
vigole said:
I think that (/opt) comes from SVR4.
Click to expand...
That is probably correct. I recall old SunOS and then Solaris machines applications (outside of base) would install into /opt by default.

The "correct hier(7)" is important, not everything is consistent.

Your example of the audio stuff is interesting, to me it sounds like something that would be on a "shared network drive" and in my mind is simply data (output of an application) that needs to be shared by different users (think a NAS with audio/video files on it that different people can watch), not "data for an application that multiple people use", which gets to the heart of my first post, "what is he talking about sharing". Sharing data used by the application like icons, fontsets or sharing data produced by an application?

Anyway, my opinions, hopefully we've given enough opinions so the OP can figure something out.
 
  • Thanks
Reactions: a6h
Alain De Vos said:
Another question is where should configuration info go..
Click to expand...
Per user configuration or overall application config?
Per user tends to go into $HOME/.config/appname
Overall config seems to wind up in /usr/local/etc/appname, if it's a service, /usr/local/etc/rc.conf with rc scripts in /usr/local/etc/rc.d (mirrors what is in /etc for base).
 
Thanks everyone for the info so far, this has been really helpful.

To clarify, Syncthing is like Dropbox. The directory I'm looking to place properly is the one that stores the content of shared files synced between computers. (The application's config files go in /usr/local/etc/syncthing/)

The closest comparison would be how MySQL/MariaDB stores the actual DB files themselves in /var/db/mysql/ while the application's config files are put in /usr/local/etc/mysql/.

- I agree with kpedersen's assessment that these files are subject to frequent change, so /var/ makes sense.

- mer brings up a good point that /var/ might be best for system-base related content, but: (a) there's no /usr/var/ equivalent and (b) MySQL/MariaDB puts their DB files in /var/ and that's not a system base application.

So based on all this convo so far and hier(7), /var/syncthing/ feels like the right spot.
 
Oh, one thing, if it is like dropbox. Does it get run by individual users or does it run as a system service?

If system service, then yeah /var/syncthing would be my recommendation. However if it is expected individual users run it (i.e with their own credentials), then perhaps $HOME/.syncthing would be more appropriate (and allow multiple user instances without worry about collision).
 
lloyd said:
Thanks everyone for the info so far, this has been really helpful.

To clarify, Syncthing is like Dropbox. The directory I'm looking to place properly is the one that stores the content of shared files synced between computers. (The application's config files go in /usr/local/etc/syncthing/)0
Click to expand...

Ultimately, unless you're producing a port or package, it doesn't matter where you store things, so long as you're consistent.

For example, I never store dynamic stuff in /usr/local. As kpedersen says correctly, this is for mainly static stuff.

/var is historically for changing data. It also (IMHO) should be mounted as a separate partition, NOT part of the root schema. Then should it fill up, ok your program probably dies but not your system.


lloyd said:
The closest comparison would be how MySQL/MariaDB stores the actual DB files themselves in /var/db/mysql/ while the application's config files are put in /usr/local/etc/mysql/.
Click to expand...

And that's basically the standard. Configuration files that are global go into /usr/local/etc. Private configurations should always go in the user's directory.


lloyd said:
- I agree with kpedersen's assessment that these files are subject to frequent change, so /var/ makes sense.

- mer brings up a good point that /var/ might be best for system-base related content, but: (a) there's no /usr/var/ equivalent and (b) MySQL/MariaDB puts their DB files in /var/ and that's not a system base application.
Click to expand...

But /var has always historically stored changeable files (printer spool files, mail boxes, usenet spools and of course logs). However, to confuse the issue, /var/cron is mainly static content. Go figure! (Personally I think it should be moved to /usr/local/cron).

Again, I think it gets back to being tradition. It used to always be a given that you would create a separate partition for /var to protect growing data from crashing your system. Things like /usr never changed until you started adding packages/ports and then, often, you would add a new disk/partition and link off /usr/local. As Phishfry said, on Solaris, you would use /opt to throw all the packages into and this was also often a separate partition (often so you could export it from another server).



lloyd said:
So based on all this convo so far and hier(7), /var/syncthing/ feels like the right spot.
Click to expand...

If it is global repository for something, then yes.
 
/opt is generally place to put softwares you want for Unix/Linux. Another option on Unix(I dont think that one exist for linux too) is /usr/local/libexec.
As long as I remember GhostBSD distro was using that path to load gbi(ghostbsd installer).
 
kpedersen said:
Oh, one thing, if it is like dropbox. Does it get run by individual users or does it run as a system service? If system service, then yeah /var/syncthing would be my recommendation. However if it is expected individual users run it (i.e with their own credentials), then perhaps $HOME/.syncthing would be more appropriate (and allow multiple user instances without worry about collision).
Click to expand...

Correct, Syncthing runs as a system service.

kpedersen said:
The Apache2 port puts the default static data location to be served in /usr/local. A number of other ports / operating systems do put it in /var.
Click to expand...

On Debian, Apache's content is served from /var/www/, and I think that's a better place for that content than /usr/local/. I can imagine /usr/local/ being ok for small, static site content, but for a dynamic website with lots of growing files it makes more sense for put it in /var/, especially with what was said above about mounting /var/ on a separate partition/disk.
 
Alain De Vos said:
What about webservice. The http pages you serve don't change often. Except poudriere logs.
Do they go in var or not ?
Click to expand...
Well I say yes, but it seems the port maintainers believe otherwise, throwing them into /usr/local/www. I do not, and never have, used /usr/local/www, preferring /var/www.

The pages may change, but the static files (php, perl, python etc) will be mostly static apart from programmatic changes.

The argument for building ports probably follows your reasoning for a webservice being placed where it is, too?
 
Since FreeBSD 13.0, using kern.elf32.allow_wx=0 and kern.elf64.allow_wx=0 kernel supports W^X. There's also elfctl(1) for adding exceptions. But AFAIK you can't use filesystem level exception, i.e. there's no MNT_WXALLOWED flag in the mount(2). Systems with MNT_WXALLOWED support in their mount(2) are able to exclude an entire filesystem from W^X. /usr/local is good candidate for such scenarios:

Enforce the W^X, and put W^X non-friendly entities in the /usr/local, e.g. 'JIT compilation', Chromium, etc. Finally mount(8) the /usr/local with wxallowed option.

FreeBSD still doesn't support that -- or I'm not aware of, if it does. But it may happen in the future. Therefore I think it's a good idea to keep /usr/local on its own dedicated filesystem.

References:
* FreeBSD 13.0-RELEASE Release Notes
* Implement enforcing write XOR execute mapping policy. | cgit.freebsd.org
* Implement enforcing write XOR execute mapping policy. | reviews.freebsd.org
 
mark_j said:
Well I say yes, but it seems the port maintainers believe otherwise, throwing them into /usr/local/www.
Click to expand...
Putting them there is what the upstream server software vendors are doing and the port maintainers are just following that quasi-standard. All three server software I used (apache, nginx and h2o along with a few others I tried) put them in the same location on Linux and FreeBSD.

I have heard of people moving them elsewhere but that's not often the case.
 
You must log in or register to reply here.
Back
Top