Samba, macOS and Spotlight: Possible on FreeBSD?

herrbischoff

herrbischoff

I have researched online resources for quite some time now, did my own testing and am now turning to this board to possibly help me along.

The scenario is as follows: Apple has deprecated the AFP protocol some time ago. Starting with macOS 10.13, AFP will not be able to serve files from a APFS partition, the new mandatory file system (see developer.apple.com). AFP client support is kept for the time being, although not officially supported any longer. That disqualifies it as a viable medium- to long-term solution, which includes Netatalk as a server component.

As quick server-side search for file names (not necessarily file contents or metadata, although macOS users have become accustomed to this) is a hard requirement for file servers, I am looking how to implement this with FreeBSD. Having switched from Linux as my main server OS a couple of years ago, I have implemented every single customer server on a FreeBSD base and it has been remarkably stable the whole time. Hence I'd like to use FreeBSD as a Samba server as well.

The Samba basics are simple. Problems appear when you try to make indexed searching available, in the style of Netatalk's spotlight support. There is a single page of not very comprehensive information based on (as it appears on the surface) instructions for Fedora Linux, offering zero information for other platforms: https://wiki.samba.org/index.php/Spotlight

Having tried to replicate the steps on FreeBSD, I couldn't get it to work, even when manually compiling in vfs_fruit and Spotlight support, which are both unavailable as configuration options in the ports tree version of Samba.

The most recent information I was able to find is from the Ports mailing list: https://www.mail-archive.com/freebsd-ports@freebsd.org/msg67346.html This lead me nowhere as well.

So I'm looking for input, pointers, ideas and ideally solutions for this situation. It astounds me most that in 2017 there still appears to be no modern (as in capable of quick indexed search), reliable, cross-platform sharing protocol for file servers, despite Samba being supported by all relevant platforms.
 
Interesting to see that there are zero replies as of yet. Is using FreeBSD as a macOS (or any kind of for that matter) file server not something users are interested in?
 
herrbischoff said:
Interesting to see that there are zero replies as of yet. Is using FreeBSD as a macOS (or any kind of for that matter) file server not something users are interested in?
Click to expand...

Perhaps many people like I simply don´t share your hypotheses "AFP client support is kept for the time being, although not officially supported any longer. That disqualifies it as a viable medium- to long-term solution, which includes Netatalk as a server component."

My take on this is that Netatalk is a perfectly viable and working solution for the time being. When time comes we'll see.
 
In this case, I'd appreciate any hints you're willing to share about how to set up Spotlight search with Netatalk. Possibly this can be adapted to work with Samba.
 
herrbischoff said:
Interesting to see that there are zero replies as of yet. Is using FreeBSD as a macOS (or any kind of for that matter) file server not something users are interested in?
Click to expand...
You are looking for an answer to a very specific question which applies for the narrow user case scenario (or maybe not so narrow considering how many FreeBSD developers use OS X as their desktop:D). I can assure you that many of serious users frequently this forum use FreeBSD as a file server, myself included. However personally (an OpenBSD guy supporting medium Red Hat desktops/compute nodes shop) I could care less abut OS X clients (didn't Apple fix their NFS client after 10 years of bug reports?) Your concerns do not involve core fileserver functionality (serving files via NFS or Samba) but rather premium proprietary Spotlight search of Samba server in the style of EOL Netatalk AFS fileserver. That is all very Apple specific.

My recommendation would be to try mailing lists (I just checked and I don't see freebsd-samba of freebsd-cif so I am not sure what would be the appropriate mailing list maybe questions or even better hackers). You are also likely to get more attention on FreeNAS forum which is frequented by seasoned storage guys. Don't be discouraged by initial noise on FreeNAS forums. There are few idiots "answering" all questions. Wait for the signal :)

Presumably your clients are paying for OS X. Doesn't that include some kind support from Apple? I am sure you are not the first person which wants to reproduce that Spotlight gizmo with Samba.
 
herrbischoff said:
Interesting to see that there are zero replies as of yet. Is using FreeBSD as a macOS (or any kind of for that matter) file server not something users are interested in?
Click to expand...

I do use my FreeBSD server as a fileserver for my macOS desktop and I have for years. But it's just me, not an entire company. I've never even bothered to see if there was some sort of Spotlight functionality. My file server stored structured data (media files, archives of completed projects). Search isn't a priority. The documents that need the sort of search you get from Spotlight are on the local disk.

I initially used NFS and it worked okay for year. Then I got a windows laptop and switched it over to Samba so I could more easily share the files with both computers.

To make the long answer short, although I am interested, I've never tried.
 
Oko said:
You are looking for an answer to a very specific question which applies for the narrow user case scenario (or maybe not so narrow considering how many FreeBSD developers use OS X as their desktop:D).
Click to expand...

Yeah, I know that this Apple stuff is rather non-standard. But dealing with clients using macOS runs you into those kinds of situations. I was under the impression that even Gnome nowadays leverages Tracker to index local files and remote Samba volumes, given the server itself supports it. Personally I don't exactly need Spotlight support but many of my clients rely on it. I was hoping to be running a file server on a reliable and reasonable machine instead of being limited to some absurd Apple hardware which would do little more than serving files. Especially considering the non-availability of server hardware for macOS. I'm not going to suggest a Mac mini as a "server" machine or a woefully out-of-date Mac Pro, just to run the macOS Server app.

Since all Linux machines I manage have Netatalk Spotlight support just one manual compile away, I figured that at least this should be replicable on FreeBSD. Once Tracker runs smoothly, adapting it to Samba might be possible.

Regarding support from Apple: there is zero. Even on an enterprise level, Apple support is close to useless, as they are always referring back to their own solutions — and if they don't provide one, you don't get one. Basically, you are on your own, despite paying up to tens of thousands of dollars a year. They appear to be almost exclusively focussed on iOS, MDM and their snazzy little devices to even bother to consider the needs of professional business users. Most of that support is, ironically, handled by IBM anyway.
 
So, from what I gather this is a dead end. I tried Netatalk and Samba mailing lists, FreeBSD forums and Apple support.

Conversation with Apple:

Apple: "Use macOS Server".
Me: "The is no Apple server hardware."
Apple: "Buy a Mac mini or Mac Pro with macOS Server."
Me: "How would that work with a large ZFS data pool?"
Apple: "It wouldn't. Use an external RAID attached via Thunderbolt."

Conversation with the community:

Community: "I don't need this".
Community: "Ask Apple."
Community: "Linux appears to work for you, use that."
Community: "Use Windows and Acronis Files Connect/ExtremeZ-IP."
Me: "But... Shouldn't there be a cross-platform solution that works with existing file servers?"
Community: *tumbleweeds tumbling*

All I can say at this point is that it's disheartening, frustrating and frankly a bit depressing. I wonder why Windows is still the dominant platform among enterprises and will likely never be replaced by any flavor of Unix desktop or macOS or whatever...
 
herrbischoff said:
There is a single page of not very comprehensive information based on (as it appears on the surface) instructions for Fedora Linux, offering zero information for other platforms: https://wiki.samba.org/index.php/Spotlight

Having tried to replicate the steps on FreeBSD, I couldn't get it to work
Click to expand...

The page you reference on samba.org now has a FreeBSD section, underneath the Linux section. I haven’t tried it yet but I’m excited to. It does have one broken link in it however (to some configuration files), which I just reported (hopefully correctly) to the webmaster.
 
Right. How about some constructive ideas instead of blunt fanboy trolling? I thought this was usually reserved for happening within the Linux community due to misguided feelings of superiority.

I will gladly do what you blurted out as soon as FreeBSD becomes a mainstream platform, when all essential Adobe and Microsoft products are developed as native FreeBSD applications for use on all my client's machines, FreeBSD gains 100% support for all Apple hardware without regressions and an open source display server environment turns up that can safely and satisfyingly be used by non-technical people, typographers, graphic designers, print processing, photographers and project managers. Oh yeah, I forgot that not even Electron is currently available on FreeBSD so you couldn't even run Microsoft Teams because there's so little developer interest to do so.
 
  • Thanks
Reactions: Oko
Dear herrbischoff,
I wonder why you demand support for some niche proprietary app, which apparently only Fedora (as RH's testing hotbed) was interested enough to temporarily take a look at.

If you like to keep that applet alive, why don't you yourself delve into the code and its integration into the whole system and find out the (probably little) mistakes that break it?

Edit:
Btw... Apple sucks. Nobody can force you to support their products. This would be against the Human Rights Declaration.
 
Datapanic said:
Gee, sorry. We don't have that problem with Macs and large storage systems where I work.
Click to expand...
Good for you. However most of us have no luxury of cheery picking job assignments and alienating paid customers. Even people who work in academia like myself have often times to do things the way our PI investigators want or feel the most comfortable with rather than what we think is the most optimal approach.
 
Oko said:
Good for you. However most of us have no luxury of cheery picking job assignments and alienating paid customers. Even people who work in academia like myself have often times to do things the way our PI investigators want or feel the most comfortable with rather than what we think is the most optimal approach.
Click to expand...

herrbischoff said:
Exactly.
Click to expand...

I take back my apology. Good luck finding a solution to your problem.
 
Oko said:
Good for you. However most of us have no luxury of cheery picking job assignments and alienating paid customers. Even people who work in academia like myself have often times to do things the way our PI investigators want or feel the most comfortable with rather than what we think is the most optimal approach.
Click to expand...

If you've got nothing constructive to add, please do not bother commenting. Thank you.
 
Come on guys. I've opened this thread to check if other people might had some success in making this work but what I find are useless bickering which I would expect to find in sites like 4chan/g/.

Anyway I've managed to install the new net/samba47 port with spotlight enabled in a jail. I've also followed the instructions on the samba wiki website referenced here. Everything seems to be connecting fine and there are logs in smbd.log indicating that samba is actually trying to do something. Setting the server as external actually launches the tracker-* daemons. From the client perspective however nothing seems to happen. It could help if I could see the dbus log messages but I have no idea how to do that. I will do some research later and try when I return to the site.
 
ulzeraj
Regarding dbus logging, maybe this information helps?
The freedesktop.org dbus monitor is not the only one. There are some others as graphical utilities, part of the Gnome and Qt infrastructure.

I wish you all good luck finding what is wrong
 
Thank you! Just a heads up that dubs-monitor will complain about the X11 Display so if you are running it on command line only system you should execute the following commands before running it:


# dbus-launch --sh-syntax > .dbus_settings
# . .dbus_settings


First some info dump. My physical machine is FreeBSD 11.1-RELEASE-p6 amd64 running with a custom kernel in which I've added VIMAGE support. The file server jail is a simple one without vimage. On this jail I've enabled mount.devfs (ruleset 4) and mount.procfs.

The net/samba47 options (omitting unset options):
Code: 
OPTIONS_FILE_SET+=ADS
OPTIONS_FILE_SET+=AESNI
OPTIONS_FILE_SET+=CUPS
OPTIONS_FILE_SET+=DEBUG
OPTIONS_FILE_SET+=DOCS
OPTIONS_FILE_SET+=FAM
OPTIONS_FILE_SET+=LDAP
OPTIONS_FILE_SET+=PROFILE
OPTIONS_FILE_SET+=SPOTLIGHT
OPTIONS_FILE_SET+=SYSLOG
OPTIONS_FILE_SET+=UTMP
OPTIONS_FILE_SET+=NSUPDATE
OPTIONS_FILE_SET+=AVAHI

Bellow is a /usr/local/etc/smb4.conf. Jail is a member of a samba4 AD domain.
Code: 
[global]
    security = ADS
    workgroup = LAN
    realm = LAN.PSANTOS.ETI.BR
    interfaces = 10.211.0.20
    bind interfaces only = yes
    server services = -nbt
    dedicated keytab file = /etc/krb5.keytab
    kerberos method = secrets and keytab
    aio read size = 16384
    aio write size = 16384
    aio write behind = yes
    use sendfile = yes
    delete readonly = yes
    log level = 3
    vfs objects = fruit zfsacl
    store dos attributes = yes
    ea support = yes
    fruit:model = Xserve
    fruit:aapl = yes
    rpc_server:mdssvc = embedded

[homes]
   read only = no
   vfs objects = fruit zfsacl
   inherit acls = yes
   inherit owner = yes
   nfs4:mode = special
   nfs4:acedup = merge
   guest ok = no
   delete readonly = yes
   writable = yes
   spotlight = yes

[pkg]
    path = /tank/pkg
    vfs objects = fruit zfsacl
    inherit acls = yes
    inherit owner = yes
    nfs4:mode = special
    nfs4:acedup = merge
    nfs4:chown = yes
    guest ok = no
    delete readonly = yes
    writable = yes
    spotlight = yes

[srv]
    path = /tank/srv
    vfs objects = fruit zfsacl
    inherit acls = yes
    inherit owner = yes
    nfs4:mode = special
    nfs4:acedup = merge
    nfs4:chown = yes
    guest ok = no
    delete readonly = yes
    writable = yes
    spotlight = yes

I've followed the https://wiki.samba.org/index.php/Spotlight FreeBSD instructions verbatim. The testing part contains an error because it suggests the tracker-store binary is in /opt/tracker/libexec when in your FreeBSD machine it will be most likely in /usr/local/libexec along with the other tracker-* programs.

I'v also added the following lines to the /usr/local/etc/rc.d/samba_server init script bellow the line that defines extra_commands which is the last before the functions are defined:
Code: 
export TRACKER_BUS_TYPE=system

Before launching samba_server I've run the dbus-monitor in a separate terminal window:
Code: 
# dbus-launch --sh-syntax > .dbus_settings
# . .dbus_settings
# dbus-monitor
signal time=1513991615.089886 sender=org.freedesktop.DBus -> destination=:1.0 serial=2 path=/org/freedesktop/DBus; interface=org.freedesktop.DBus; member=NameAcquired
   string ":1.0"
signal time=1513991615.090266 sender=org.freedesktop.DBus -> destination=:1.0 serial=4 path=/org/freedesktop/DBus; interface=org.freedesktop.DBus; member=NameLost
   string ":1.0"

Launching the programs will not generate any specific information on syslog or dbus-monitor. However as soon as I've mounted the homes share from finder the following message appears on /var/log/messages:

Code: 
Dec 22 23:28:18 file dbus[7769]: [system] Activating service name='org.freedesktop.Tracker1' (using servicehelper)

The log.smbd outputs a lot of stuff with loglevel 3 enabled. The full logs are in a pastern bellow. There is this relevant message:
Code: 
(process:98726): Tracker-WARNING **: Locale '0' is not set, defaulting to C locale

Nice. The tracker processes also show up. They are owned by root.
Code: 
 # ps x | grep tracker
74418  -  IJ   0:00.00 /bin/sh /usr/local/libexec/tracker-wrapper tracker-store
75699  -  IJ   0:00.05 /usr/local/libexec/tracker-store
36481  1  S+J  0:00.00 grep tracker

Now for the real test I try to search something on the mounted shares from finder (you have to switch to the second option as it defaults to "My Mac") but nothing happens. Every time I try to search something the following message is produced on log.smbd:

Code: 
[2017/12/22 23:39:22.084660,  3] ../source3/rpc_server/srv_pipe.c:1528(api_rpcTNP)
  api_rpcTNP: rpc command: MDSSVC_CMD
Full smbd output of here: https://pastebin.com/AV6VCjTS

The dbus-monitor output shows nothing. I wonder if its really working.

Then I had the idea to start everything by hand to watch the tracker-store output. Issuing the following commands, mounting the directories and using finder to search for "FreeBSD" in my /tank/pkg/iso folder (smb://pkg/iso uri) returns no results but tracker returns the following output:

Code: 
(tracker-store:77551): Tracker-DEBUG: Resetting collator in db interface 0x8049ec0d0
(tracker-store:77551): Tracker-DEBUG: [ICU collation] Initializing collator for locale 'C'
(tracker-store:77551): Tracker-DEBUG: Preparing query: 'PRAGMA journal_mode = WAL;'
Tracker-Message:   Setting page size to 8192
Tracker-Message:   Setting cache size to 250
(tracker-store:77551): Tracker-DEBUG: Preparing query: 'SELECT "2_u" COLLATE TRACKER AS "2_u" FROM (SELECT "fts1"."docid" AS "1_u", "fts1"."docid" AS "ID", tracker_rank(matchinfo("fts1"."fts", 'cl'),fts_column_weights()) AS "obj_u_rank", "nie:DataObject2"."nie:url" AS "2_u" FROM "fts" AS "fts1", "nie:DataObject" AS "nie:DataObject2" WHERE "fts1"."docid" = "nie:DataObject2"."ID" AND "2_u" IS NOT NULL AND "fts1"."fts" MATCH 'FreBSD*' AND SparqlUriIsDescendant(? COLLATE TRACKER, "2_u" COLLATE TRACKER))'
(tracker-store:77551): Tracker-DEBUG: ---> [1|smbd|42535] Success, no error given


So apparently samba is indeed launching the daemons and registering those dbus services but those sql-like queries (the results are kept on a sqlite database I believe) won't return anything so I think something is missing like some step to index stuff if it doesn't do it automatically as you access those files.

Running the tracker status tool increases my suspicion:
Code: 
root@file:~ # setenv TRACKER_BUS_TYPE system
root@file:~ # tracker status
(tracker status:35246): Tracker-WARNING **: Locale '0' is not set, defaulting to C locale

Currently indexed: 0 files, 0 folders
Remaining space on database partition: 1.8 TB (70.97%)
All data miners are idle, indexing complete

So maybe if we find out how the index processing works we might make this work. I don't know much about dbus and freedesktop. Enabling the spotlight option on samba will pull a lot of tracker related programs. They all seem to be X11 dependent software. They will run if you setenv TRACKER_BUS_TYPE system but their parameters suggest they are graphic utilities.
 
ulzeraj, that's an interesting to read report.

I do not know whether it's relevant, but when I look at that snippet you quoted...
Code: 
MATCH 'FreBSD*'
...I wonder, will it ever match with only one 'e'?!?
 
Yeah thats wrong but I also tried other keywords like windows and searching for specificword in a folder filled with specificword_*.jpg files.

My last edit happened while you were replying so apparently the tracker system is working but nothing is being indexed. If we find out how to index our files then the samba module will start showing results.

Code: 
root@file:~ # setenv TRACKER_BUS_TYPE system
root@file:~ # tracker status
(tracker status:35246): Tracker-WARNING **: Locale '0' is not set, defaulting to C locale

Currently indexed: 0 files, 0 folders
Remaining space on database partition: 1.8 TB (70.97%)
All data miners are idle, indexing complete
 
I do know nothing about tracker, but you say it uses a sqlite backend.
Do you find evidence for tracker's request to the backend db being processed in the sqlite or whatever log?
 
If you launch tracker-store manually you can see evidence that its an sqlite database.

Code: 
root@file:~ # /usr/local/libexec/tracker-store -v 3
GLib-GIO-Message: Using the 'memory' GSettings backend.  Your settings will not be saved or shared with other applications.
Tracker-Message: Starting tracker-store 1.6.1
Tracker-Message: General options:
Tracker-Message:   Verbosity  ............................  3
Tracker-Message: Store options:
Tracker-Message:   Readonly mode  ........................  no
Tracker-Message:   GraphUpdated Delay ....................  1000
Tracker-Message: Log verbosity is set to 3, enabling D-Bus client lookup
Tracker-Message: Registering D-Bus object...
Tracker-Message:   Path:'/org/freedesktop/Tracker1/Status'
Tracker-Message:   Type:'TrackerStatus'
Tracker-Message: Registering D-Bus object...
Tracker-Message:   Path:'/org/freedesktop/Tracker1/Statistics'
Tracker-Message:   Type:'TrackerStatistics'
Tracker-Message: Registering D-Bus object...
Tracker-Message:   Path:'/org/freedesktop/Tracker1/Resources'
Tracker-Message:   Type:'TrackerResources'
Tracker-Message: Registering D-Bus object...
Tracker-Message:   Path:'/org/freedesktop/Tracker1/Steroids'
Tracker-Message:   Type:'TrackerSteroids'
Tracker-Message: Registering D-Bus object...
Tracker-Message:   Path:'/org/freedesktop/Tracker1/Backup'
Tracker-Message:   Type:'TrackerBackup'
Tracker-Message: Registering D-Bus service...
  Name:'org.freedesktop.Tracker1'
(tracker-store:81558): Tracker-WARNING **: Locale '0' is not set, defaulting to C locale
(tracker-store:81558): Tracker-DEBUG: Locale 'TRACKER_LOCALE_LANGUAGE' was set to 'C'
(tracker-store:81558): Tracker-DEBUG: Locale 'TRACKER_LOCALE_TIME' was set to 'C'
(tracker-store:81558): Tracker-DEBUG: Locale 'TRACKER_LOCALE_COLLATE' was set to 'C'
(tracker-store:81558): Tracker-DEBUG: Locale 'TRACKER_LOCALE_NUMERIC' was set to 'C'
(tracker-store:81558): Tracker-DEBUG: Locale 'TRACKER_LOCALE_MONETARY' was set to 'C'
Tracker-Message: Setting database locations
Tracker-Message: Checking database directories exist
Tracker-Message: Checking database version
Tracker-Message: Checking database files exist
Tracker-Message: Loading databases files...
Tracker-Message: Opened sqlite3 database:'/root/.cache/tracker/meta.db'

I can't help but feel the frequent "GLib-GIO-[B]Message[/B]: Using the 'memory' GSettings backend. Your settings will not be saved or shared with other applications." message that appear in every dbus related command may be a sign of a missing configuration file or running daemon.

The program that run the indexes is probably /usr/local/libexec/tracker-miner-fs and the settings for indexing are defined by gsettings parameters. The thing is that setting them as the wiki suggests will not change those settings.

Issuing the following commands will not change the value of org.freedesktop.Tracker.Miner.Files index-recursive-directories:
Code: 
root@file:~ # dbus-launch --csh-syntax > .dbus_settings
root@file:~ # source .dbus_settings
root@file:~ # setenv TRACKER_BUS_TYPE system
root@file:~ # gsettings set org.freedesktop.Tracker.Miner.Files index-recursive-directories "['/tank/pkg']"
GLib-GIO-Message: Using the 'memory' GSettings backend.  Your settings will not be saved or shared with other applications.



root@file:~ # gsettings list-recursively | grep index-recursive-directories
GLib-GIO-Message: Using the 'memory' GSettings backend.  Your settings will not be saved or shared with other applications.
org.freedesktop.Tracker.Miner.Files index-recursive-directories ['&DESKTOP', '&DOCUMENTS', '&DOWNLOAD', '&MUSIC', '&PICTURES', '&VIDEOS']

Update: installing devel/dconf fixes the issue of not saving the gsettings parameters.
 
You must log in or register to reply here.
Back
Top