searx installation guide

I know this is very late but this thread appeaers in top google results,

I wrote a guide on how to install it here https://blog.westofer.ga/posts/installing-searx-on-freebsd/

It is my first writing and I am new to freeBSD so keep your expectations low. I am really sorry about the quality and I will try to re write it once I am done

For time travelers this site will probably be down in the future so I will copy and paste it in the next message
 
A port of searx already exists for freeBSD, so why bother? this port uses a deprecated version of python (py27)
Installing packages
pkg install py37-virtualenv python py37-pip \
py37-lxml /py37-babelfish/ git libxml2\
hs-ShellCheck \
uwsgi #optional


Fetching ports
portsnap fetch this might take a LONG time
Installing packages from ports
sudo pkg install py37-babel

creating user
Now we need to create a searx user
sudo adduser
with the following settings:
Username: searx
Full name: Searx Search Engine
Uid (Leave empty for default): *EMPTY*
Login group [searx]: *EMPTY*
Login group is searx. Invite searx into other groups? []: *EMPTY*
Login class [default]: *EMPTY*
Shell (sh csh tcsh bash rbash fish git-shell nologin) [sh]: *EMPTY*
Home directory [/home/searx]: /usr/home/searx
Home directory permissions (Leave empty for default): *EMPTY*
Use password-based authentication? [yes]: no
Lock out the account after creation? [no]: no

Install Searx and dependencies
Start an interactive shell from new created user and clone searx:
sudo -H -u searx -i
cd /usr/home/searx
git clone [URL]https://github.com/asciimoo/searx.git[/URL] ./searx-src

In the same shell create a virtualenv
python3 -m venv /usr/home/searx/searx-pyenv
echo . /usr/home/searx/searx-pyenv/bin/activate >> /usr/home/searx/.profile

To install searx's dependencies, exit the bash session open a new one (CTRL+D).
sudo -H -u searx -i
# update pip's boilerplate ..
pip install -U pip
pip install -U setuptools
pip install -U whee
l
Tip
open a second terminal for configuring the tasks and leave '(searx)$' terminal open

Configuration
In another terminal type:
sudo mkdir /usr/local/etc/searx
Then copy the settings config file to /usr/local/etc/searx
sudo -H cp /usr/home/searx/searx-src/searx/settings.yml /usr/local/etc/searx/settings.yml
*I have no idea why do we need this but it is included in the docs*

Now replace the secret key and instance name
sudo -H sed -i -e s/{ultrasecretkey}/\367ed0f4f4f300bd1457e5c87d036ab2/g /usr/local/etc/searx/settings.yml
sudo -H sed -i -e s/{instance_name}/searx@\ryzen/g /etc/searx/settings.yml

Sed didn't work for me for some reason, so change it manually! (line #3 and #15)

Check
To check searx , enable optional debugging and start the webapp. (line #2)
Code: 
debug: False
#change it to
debug True

Then we can start the web app
sudo -H -u searx -i
cd /usr/local/searx/searx-src
export SEARX_SETTINGS_PATH=/usr/local/etc/searx/settings.yml
python searx/webapp.py


Testing!
if freeBSD is installed in your local machine then simply open localhost:8888 in a browser, if you are in a ssh session, then you need to bind address to your local machine.
#run in your local machine
ssh -L 8888:localhost:8888 username@server [cmd]

now you should be able to open searx in localhost:8888 in your browser!
At this point searx is not demonized, this why we should use [B]uwsgi[/B] or [B]pm2[/B]. You should also change Debug to False in setting.yml [file] /usr/local/etc/searx/settings.yml [/file]
[cmd]
debug: False
#change it to
debug True


Demonizing Searx with pm2

Official docs use uwsgi, however I believe that pm2 is way easier
Install node if you dont have it already
sudo pkg install node www/npm
Install pm2 with
sudo npm i -g pm2

start pm2 on startup with the following
pm2 startup
you will then be given a command to copy/paste (note that it doesn't work in fish shell, use sh)
Now start searx with pm2

sudo -H -u searx -i
cd /usr/home/searx/searx-src
sudo pm2 start searx/webapp.py
[cmd]

Also save pm2 settings for next startup

[cmd]sudo pm2 save

Installing nginx
This guide is getting quit long and I think dividing it would be better :)
 
Setting up Nginx For Searx in freeBSD

This blog post was written by a freeBSD noob. I just wrote how I did it, which might not be the most efficient way possible :)
Click to expand...

Installing packages
Install nginx
Install nginx with
sudo pkg install nginx
Install certbot
sudo pkg install py37-cerbot

Configuring
configure nginx
Add nginx to /etc/rc.conf

Code: 
 #/etc/rc.conf
nginx_enable="YES"
and enable it
sudo service nginx start
Create a sites-enabled directory
sudo mkdir /usr/local/etc/nginx/sites-enabled
Create a conf file for searx
sudo touch /usr/local/etc/nginx/sites-enabled/searx.conf
Then we need to edit nginx.conf this way we are going to have the search engine in a subdomoin (for example searx.westofer.ga)
This file is going to be the homepage (in my example https://westofer.ga)

Code: 
#/usr/local/etc/nginx/nginx.conf
worker_processes 1;
events {
    worker_connections 1024;
}

http {
    include mime.types;
    default_type application/octet-stream;
    include /usr/local/etc/nginx/sites-enabled/*;
    sendfile on;
    keepalive_timeout 65;

    server {
        listen 80;
        server_name YOUR-DOMAIN.COM;
        #charset koi8-r;
        #access_log  logs/host.access.log  main;

    location / {
        root /usr/local/www/nginx;
        index index.html index.htm;
    }
}

Now for the /usr/local/etc/nginx/sites-enabled/searx.conf
Code: 
server {
    listen 80;
    listen [::]:80;
    access_log /dev/null;
    error_log /dev/null;

    root /var/www/subdomain.example.com;
    index index.html index.htm index.nginx-debian.html;
    server_name searx.domain.com;

    location / {
        proxy_pass http://localhost:8888; #port for searx
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
    }
}
Restarting nginx server
sudo service nginx restart

SSL certificate (https)
now you should see your website running in http://searx.domain.com, which is insecure, so we should upgrade to https with a SSL certificate from certbot.
 
Nginx Certbot In Freebsd

Installation
First we need to install certbot
sudo pkg install py37-certbot
Stop nginx to install the certificates
sudo service nginx stop
now we can safley run certbot
sudo certbot certonly --standalone
type your email , domain and sub domains when prompted
you should recieve a success message saying
  • Congratulations! Your certificate and chain have been saved at: /usr/local/etc/letsencrypt/live/DOMAIN.COM/fullchain.pem Your key file has been saved at: /usr/local/etc/letsencrypt/live/DOMAIN.COM/privkey.pem Your cert will expire on 2020-10-24. To obtain a new or tweaked version of this certificate in the future, simply run certbot again. To non-interactively renew all of your certificates, run " certbot renew"
  • If you like Certbot, please consider supporting our work by:........
run the following to get auto-renewal (all in one line)
echo "0 0,12 * * * root python -c 'import random; import time; time.sleep(random.random() * 3600)' && certbot renew -q" | sudo tee -a /etc/crontab > /dev/null
Modifing nginx conf
Now go to /usr/local/etc/nginx/sites-avialable/searx.conf and change it to
Code: 
server {

    listen 80;
    listen [::]:80;
    access_log /dev/null;
    error_log /dev/null;

    server_name searx.DOMAIN.COM;
    return 301 https://$server_name$request_uri;
}

server {

    listen 443 ssl;
    server_name www.searx.DOMAIN.COM;
    ssl_certificate /usr/local/etc/letsencrypt/live/DOMAIN.com/fullchain.pem;
    ssl_certificate_key /usr/local/etc/letsencrypt/live/DOMAIN.com/privkey.pem;


    location / {
        proxy_pass http://localhost:8888;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
}
}
 
Wow this guide is great! Thanks a lot! Somehow I didn't get a notification that I got an answer. Sorry for the extremely late response.
I'll probably try to install it this weekend :)
 
Recently, I also stumbled across this. First we need to decide, for what purpose we want to run searx, because this determines the installation procedure. The official guides describe the installation procedure of public searx instances on Linux with all the bells & whistles of bot blocking, proxying of search requests to the upstream engines, in order to avoid captcha problems, and searx multiprocess operation for load balancing.
  • If you want to setup a public instance, then you want to follow the guide of Westofer and revise your setup regarding bot blocking using filtron. For load balancing with multiprocess searx, you would actually want to utilize uwsgi instead of pm2.

  • In case you want to setup a local instance either for use in your LAN only, or even a private instance running on the loopback interface of your computer, then all these guides are overly complicated and bring in stuff that you do not need and actually do not want.
For installation of local instances you may want to consider the following.

The port www/searx is not that bad. It is only outdated — searx 0.16, while the current release is v0.18. The port comes with a working rc startup script, which utilizes the generic daemon(8) utility of FreeBSD for daemonizing searx, and it pre-installs most of the dependencies. Furthermore, searx is a well-designed python software package, having its own setup.py script. Therefore almost everything is already in place to get a local/private searx instance up and running in a few minutes.

First install the outdated port:
# pkg install -y searx

If not already done, complete the python installation, so we get /usr/local/bin/python and /usr/local/bin/python3 properly linked to the actual python binary.
# pkg install -y python python3

Now manually update the searx installation from upstream — this step can be repeated occasionally.
# fetch -o searx-master.zip https://github.com/searx/searx/archive/master.zip
# unzip searx-master.zip; cd searx-master
# python setup.py install
# rm -r `python3 -c 'import sysconfig; print(sysconfig.get_paths()["purelib"])'`/searx

At at the time writing, python3.7 is the current version, if you got another version, then in the commands below, replace 3.7 with 3.x of your version. In addition note the current searx version, and use it in the following commands (assuming tcsh is your shell).
# setenv SEARXV `sed -n '/^Version: /{s///;p;}' searx.egg-info/PKG-INFO`; echo $SEARXV
# ln -s searx-$SEARXV-py3.7.egg/searx `python3 -c 'import sysconfig; print(sysconfig.get_paths()["purelib"])'`/searx

# cp searx/settings.yml /usr/local/etc/searx-settings.yml
# cp searx/settings.yml /usr/local/etc/searx-settings.yml.sample
# sed -e "s|ultrasecretkey|`/usr/bin/openssl rand -base64 32`|" -i "" /usr/local/etc/searx-settings.yml
# ln -sf /usr/local/etc/searx-settings.yml `python3 -c 'import sysconfig; print(sysconfig.get_paths()["purelib"])'`/searx/settings.yml

In case you want to run searx as a private instance on localhost of your computer, then you are ready to go:
# sysrc searx_enable="YES"
# service searx start

In case you want to run searx as a local instance in your LAN, you need to change the setting bind_address from 127.0.0.1:8888 to the correct lanip-address, for example 192.168.0.11:8888, and restart searx, in case it was started already.

Now point your browser to either of it:

http://localhost:8888/

or

http://lanip-address:8888/

I need to adapt the timeout setting from 3.0 to 5.0, otherwise I see a lot of timeout errors.
 
I think you forgot to include the command to install searx into the virtual environment ;)
pip install -U wheel
pip install -e .

But all in all your guide worked great! I installed it in a jail and ended up using uwsgi and the unix socket of uwsgi.
I case someone is confused. I chose /var/spool/searx to be the home directory of searx.

/usr/local/etc/uwsgi/uwsgi.ini

[uwsgi]

# uWSGI core
# ----------
#
# https://uwsgi-docs.readthedocs.io/en/latest/Options.html#uwsgi-core

# Who will run the code
uid = searx
gid = searx

# set (python) default encoding UTF-8
env = LANG=C.UTF-8
env = LANGUAGE=C.UTF-8
env = LC_ALL=C.UTF-8

# chdir to specified directory before apps loading
chdir = /var/spool/searx/searx-src/searx

# searx configuration (settings.yml)
env = SEARX_SETTINGS_PATH=/usr/local/etc/searx/settings.yml

# disable logging for privacy
disable-logging = true

# The right granted on the created socket
chmod-socket = 660

# Plugin to use and interpretor config
single-interpreter = true

# enable master process
master = true

# load apps in each worker instead of the master
lazy-apps = true

# load uWSGI plugins
#plugin = python3,http

# By default the Python plugin does not initialize the GIL. This means your
# app-generated threads will not run. If you need threads, remember to enable
# them with enable-threads. Running uWSGI in multithreading mode (with the
# threads options) will automatically enable threading support. This *strange*
# default behaviour is for performance reasons.
enable-threads = true


# plugin: python
# --------------
#
# https://uwsgi-docs.readthedocs.io/en/latest/Options.html#plugin-python

# load a WSGI module
module = searx.webapp

# set PYTHONHOME/virtualenv
virtualenv = /var/spool/searx/searx-pyenv

# add directory (or glob) to pythonpath
pythonpath = /var/spool/searx/searx-src


# speak to upstream
# -----------------
#
# Activate the 'http' configuration for filtron or activate the 'socket'
# configuration if you setup your HTTP server to use uWSGI protocol via sockets.

# using IP:
#
# https://uwsgi-docs.readthedocs.io/en/latest/Options.html#plugin-http
# Native HTTP support: https://uwsgi-docs.readthedocs.io/en/latest/HTTP.html

#http = 192.168.69.69:8888

# using unix-sockets:
#
# On some distributions you need to create the app folder for the sockets::
#
# mkdir -p /run/uwsgi/app/searx
# chmod -R searx:searx /run/uwsgi/app/searx
#
socket = /var/run/uwsgi/uwsgi.socket


The relevant part of nginx.conf
/usr/local/etc/nginx/nginx.conf

server {
listen 80;
server_name searx.domain.com;
return 301 https://$server_name$request_uri;
}

server {
listen 443 ssl http2;
server_name searx.domain.com;
ssl_certificate /usr/local/etc/letsencrypt/live/domain.com/fullchain.pem;
ssl_certificate_key /usr/local/etc/letsencrypt/live/domain.com/privkey.pem;
include snippets/ssl.conf;
include snippets/headers.conf;
location / {
include uwsgi_params;
uwsgi_pass unix:/zroot/iocage/jails/searx/root/var/run/uwsgi/uwsgi.socket;
}
}


/etc/rc.conf

uwsgi_enable="YES"
uwsgi_socket="/var/run/uwsgi/uwsgi.socket"


PS:
I don't think installing sudo only to switch the user and home directory is necessary.
su should also do the job.

su -l searx
 
Last edited:
I found out, that Searx’s webapp.py can be run on a unix domain socket (UDS) instead of an IP socket (IPS). This is beneficial for a number of reasons:
  1. Communication over unix domain sockets is much faster.

  2. We won’t need the uWSGI daemon in the middle, which adds to the complexity of the setup and of course comes with a performance penalty. Instead, Apache or Nginx would serve directly as a proxy to the UDS.

  3. At least on FreeBSD (I am not sure for Linux), many independently launched processes may listen on the same UDS, while more then one process may listen on the same IPS only if these are child processes forked from a mother process which actually opened the IPS and shared it to it’s children.

  4. When it comes to multi-threading, Python got a huge barrier called Global Interpreter Lock, which is imposed by the memory management system. You can setup multithreading with Python, but because of the GIL, the threads are effectively not running in parallel but sequentially. This is a considerable issue for web services programmed in Python, which even on multiprocessor systems cannot easily process several requests in parallel. The official suggestion is not to use multithreading but multiprocessing. This means for Searx, we would need to launch as many webapp.py which would serve our needs.

  5. Now with UDS on FreeBSD you may launch 10 and more, which all would listen and respond in parallel on the same unix domain socket. If you stay with IPS, then things turn out way more complicated (uWSGI and friends).

For enabling UDS, I replaced in searx-settings.yml the definition bind_address : "127.0.0.1" by bind_address : "unix:///tmp/searx.sock"

Then for Apache I added the following virtual host settings for my LAN:
Code: 
<VirtualHost *:80>
   ServerName            searx.example.com:80

   <Proxy *>
      Require            ip 192.168.0.0/24
   </Proxy>

   ProxyPass             / unix:/tmp/searx.sock|http://searx.example.com/
</VirtualHost>

For launching as many Searx processes as needed by the rc script which had been installed in the course of the instructions of my other message above, I created a shell script mulitsearx.sh like follows, which would launch 4 Searx instances:
Bash: 
#!/bin/sh

"$1" "$2" &
pid1="$!"

"$1" "$2" &
pid2="$!"

"$1" "$2" &
pid3="$!"

"$1" "$2" &
pid4="$!"

trap "kill -TERM $pid1 $pid2 $pid3 $pid4" exit INT TERM
wait

Then I would replaced in /usr/local/etc/rc.d/searx the definition of command_args with the following:
command_args=" -c -f -P ${pidfile} /path/to/multisearx.sh ${python} ${script_py}"

Note, /path/to/multisearx.sh must be executable by user www.

For testing:
# service searx start
# ps -axj | grep python
Code: 
www      52524 52523 52523 52523    0 S     -     0:00.01 /bin/sh /root/bin/multisearx.sh /usr/local/bin/python3.7 /usr/local/lib/python3.7/site-packages/searx/webapp.py
www      52525 52524 52523 52523    0 R     -     0:05.22 /usr/local/bin/python3.7 /usr/local/lib/python3.7/site-packages/searx/webapp.py
www      52526 52524 52523 52523    0 R     -     0:05.20 /usr/local/bin/python3.7 /usr/local/lib/python3.7/site-packages/searx/webapp.py
www      52527 52524 52523 52523    0 R     -     0:05.22 /usr/local/bin/python3.7 /usr/local/lib/python3.7/site-packages/searx/webapp.py
www      52528 52524 52523 52523    0 R     -     0:05.25 /usr/local/bin/python3.7 /usr/local/lib/python3.7/site-packages/searx/webapp.py
root     52531 45043 52530 45043    2 S+    1     0:00.01 grep python
# service searx stop
# ps -axj | grep python
Code: 
root     52707 45043 52706 45043    2 S+    1     0:00.01 grep python
 
I would like to run my searx instance using searx port. That is what ports are for, right? :)

Port installs fine, but trying to start it does nothing. I guess it should at least throw something to console or to some logfile but it just silently does nothing.
 
pacija said:
That is what ports are for, right?
Click to expand...
The searx port got the advantage of easy-installation, while my installation method got the advantage that it comes up with a working searx :p

That said, searx does not come with provisions for logging.

For finding out what is wrong, you want to start searx onetime manually on the command line in the foreground.

I assume you got installed the default Python v3.7, in case not, replace the version number in below command:

python3.7 \
`python3.7 -c 'import sysconfig; print(sysconfig.get_paths()["purelib"])'`/searx/webapp.py

Then you would see all the diagnostic output as it happens directly in the terminal. Starting searx takes some time (on slow machines upto 15 seconds). So do not press ctrl-C too early.
 
I am getting:

Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/searx/webapp.py", line 50, in <module>
from werkzeug.contrib.fixers import ProxyFix
ModuleNotFoundError: No module named 'werkzeug.contrib'

Although www/py-werkzeug is installed:

% pkg info -x werkzeug
py37-werkzeug-1.0.1

searx is installed from package built in my own poudriere.

I'm more interested in getting searx port to work out of the box than getting searx to work in other way (virtualnenv etc.)

Thank you in advance,
 
You must log in or register to reply here.
Back
Top