PyTorch

Hello!

I've seen that the port has been tried by a FreeBSD community member:

Is there anyone who knows if there exits a port of PyTorch or could make it work otherwise on FreeBSD?

Many thanks!
 

SirDice

Administrator
Staff member
Administrator
Moderator
Is there anyone who knows if there exits a port of PyTorch
Did you know that Freshports has a search option?

There are also various other search options you could use on the command line, but the Freshports website is usually a lot easier to use. For new and seasoned users alike.

Code:
root@molly:/usr/ports # make search name=pytorch
Port:   py38-pytorchvideo-0.1.3
Path:   /usr/ports/math/py-pytorchvideo
Info:   Video understanding deep learning library
Maint:  nc@FreeBSD.org
B-deps: gettext-runtime-0.21 indexinfo-0.3.1 libffi-3.3_1 mpdecimal-2.5.1 py37-setuptools-57.0.0 python37-3.7.13_1 python38-3.8.13_1 readline-8.1.2
R-deps: Imath-3.1.5 aom-3.3.0_1 argp-standalone-1.3_4 arm-gnueabi-binutils-2.37_2,1 brotli-1.0.9,1 ca_root_nss-3.77 dav1d-1.0.0 expat-2.4.8 ffmpeg-4.4.2_1,1 fontconfig-2.13.94_2,1 freetype2-2.12.0 fribidi-1.0.12 gcc10-10.3.0 gdk-pixbuf2-2.40.0_4 gettext-runtime-0.21 giflib-5.2.1 glib-2.70.4_5,2 gmp-6.2.1 gnutls-3.7.4 graphite2-1.3.14 harfbuzz-4.2.1 highway-0.16.0_2 indexinfo-0.3.1 jbigkit-2.1_1 jpeg-turbo-2.1.3 lame-3.100_4 lcms2-2.12 libX11-1.7.2,1 libXScrnSaver-1.2.3_2 libXau-1.0.9 libXdmcp-1.1.3 libXext-1.3.4,1 libXfixes-6.0.0 libXft-2.3.4 libXrender-0.9.10_2 libass-0.15.2 libdrm-2.4.110,1 libepoll-shim-0.0.20210418 libffi-3.3_1 libiconv-1.16 libidn2-2.3.2 libimagequant-2.17.0 libjxl-0.6.1_5 libogg-1.3.5,4 libpciaccess-0.16 libpthread-stubs-0.4 libraqm-0.9.0 libtasn1-4.18.0 libtheora-1.1.1_7 libudev-devd-0.5.0 libunistring-1.0 libv4l-1.23.0 libva-2.14.0 libvdpau-1.5 libvorbis-1.3.7_2,3 libvpx-1.11.0 libx264-0.163.3060 libxcb-1.14_1 libxml2-2.9.13_2 libyaml-0.2.5 metis-5.1.0_9 mpc-1.2.1 mpdecimal-2.5.1 mpfr-4.1.0_1 nettle-3.7.3 openblas-0.3.20,1 openexr-3.1.5 openjpeg-2.4.0 opus-1.3.1 p11-kit-0.24.1 pciids-20220322 pcre-8.45_1 png-1.6.37_1 py37-setuptools-57.0.0 py38-av-9.0.1 py38-fvcore-0.1.5.post20210924 py38-iopath-0.1.9 py38-olefile-0.46 py38-parameterized-0.8.1 py38-pillow-9.1.0 py38-portalocker-2.3.2 py38-tabulate-0.8.9 py38-termcolor-1.1.0_1 py38-tqdm-4.64.0 py38-yacs-0.1.8 py38-yaml-5.4.1 py39-numpy-1.20.3,1 py39-tkinter-3.9.12_6 python37-3.7.13_1 python38-3.8.13_1 python39-3.9.12_1 readline-8.1.2 shared-mime-info-2.2_1 suitesparse-amd-2.4.6 suitesparse-camd-2.4.6 suitesparse-ccolamd-2.9.6 suitesparse-cholmod-3.0.14 suitesparse-colamd-2.9.6 suitesparse-config-5.12.0 suitesparse-umfpack-5.7.9 svt-av1-1.0.0 tcl86-8.6.12 tiff-4.3.0 tk86-8.6.12 vmaf-2.3.1 wayland-1.20.0_2 webp-1.2.2 x265-3.4_2 xorgproto-2021.5 xvid-1.3.7,1
WWW:    https://github.com/facebookresearch/pytorchvideo
Or
Code:
root@molly:/usr/ports # pkg install psearch
Updating dicelan-server repository catalogue...
dicelan-server repository is up to date.
All repositories are up to date.
Checking integrity... done (0 conflicting)
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        psearch: 2.1.0

Number of packages to be installed: 1

Proceed with this action? [y/N]: y
[1/1] Installing psearch-2.1.0...
[1/1] Extracting psearch-2.1.0: 100%
root@molly:/usr/ports # psearch pytorch
math/py-pytorchvideo      Video understanding deep learning library
 
Hello & Welcome to this FreeBSD community!

Is there anyone who knows if there exits a port of PyTorch or could make it work otherwise on FreeBSD?
I actually had to look into this a few weeks ago. As per my knowledge, there is no port available. Therefore, I started compiling from source. I got libtorch to compile successfully after a few very minor patches and careful selection of configuration options. I can share that if you like.
While I was also able to build a program using libtorch it is still encountering some runtime issue that I have to figure out.

What I mean to say is this: Compiling the source on FreeBSD does appear to be an option without too much hassle. Therefore, creating a port would be too but I don't know much about the python side.
 
Did you know that Freshports has a search option?

There are also various other search options you could use on the command line, but the Freshports website is usually a lot easier to use. For new and seasoned users alike.

Code:
root@molly:/usr/ports # make search name=pytorch
Port:   py38-pytorchvideo-0.1.3
Path:   /usr/ports/math/py-pytorchvideo
Info:   Video understanding deep learning library
Maint:  nc@FreeBSD.org
B-deps: gettext-runtime-0.21 indexinfo-0.3.1 libffi-3.3_1 mpdecimal-2.5.1 py37-setuptools-57.0.0 python37-3.7.13_1 python38-3.8.13_1 readline-8.1.2
R-deps: Imath-3.1.5 aom-3.3.0_1 argp-standalone-1.3_4 arm-gnueabi-binutils-2.37_2,1 brotli-1.0.9,1 ca_root_nss-3.77 dav1d-1.0.0 expat-2.4.8 ffmpeg-4.4.2_1,1 fontconfig-2.13.94_2,1 freetype2-2.12.0 fribidi-1.0.12 gcc10-10.3.0 gdk-pixbuf2-2.40.0_4 gettext-runtime-0.21 giflib-5.2.1 glib-2.70.4_5,2 gmp-6.2.1 gnutls-3.7.4 graphite2-1.3.14 harfbuzz-4.2.1 highway-0.16.0_2 indexinfo-0.3.1 jbigkit-2.1_1 jpeg-turbo-2.1.3 lame-3.100_4 lcms2-2.12 libX11-1.7.2,1 libXScrnSaver-1.2.3_2 libXau-1.0.9 libXdmcp-1.1.3 libXext-1.3.4,1 libXfixes-6.0.0 libXft-2.3.4 libXrender-0.9.10_2 libass-0.15.2 libdrm-2.4.110,1 libepoll-shim-0.0.20210418 libffi-3.3_1 libiconv-1.16 libidn2-2.3.2 libimagequant-2.17.0 libjxl-0.6.1_5 libogg-1.3.5,4 libpciaccess-0.16 libpthread-stubs-0.4 libraqm-0.9.0 libtasn1-4.18.0 libtheora-1.1.1_7 libudev-devd-0.5.0 libunistring-1.0 libv4l-1.23.0 libva-2.14.0 libvdpau-1.5 libvorbis-1.3.7_2,3 libvpx-1.11.0 libx264-0.163.3060 libxcb-1.14_1 libxml2-2.9.13_2 libyaml-0.2.5 metis-5.1.0_9 mpc-1.2.1 mpdecimal-2.5.1 mpfr-4.1.0_1 nettle-3.7.3 openblas-0.3.20,1 openexr-3.1.5 openjpeg-2.4.0 opus-1.3.1 p11-kit-0.24.1 pciids-20220322 pcre-8.45_1 png-1.6.37_1 py37-setuptools-57.0.0 py38-av-9.0.1 py38-fvcore-0.1.5.post20210924 py38-iopath-0.1.9 py38-olefile-0.46 py38-parameterized-0.8.1 py38-pillow-9.1.0 py38-portalocker-2.3.2 py38-tabulate-0.8.9 py38-termcolor-1.1.0_1 py38-tqdm-4.64.0 py38-yacs-0.1.8 py38-yaml-5.4.1 py39-numpy-1.20.3,1 py39-tkinter-3.9.12_6 python37-3.7.13_1 python38-3.8.13_1 python39-3.9.12_1 readline-8.1.2 shared-mime-info-2.2_1 suitesparse-amd-2.4.6 suitesparse-camd-2.4.6 suitesparse-ccolamd-2.9.6 suitesparse-cholmod-3.0.14 suitesparse-colamd-2.9.6 suitesparse-config-5.12.0 suitesparse-umfpack-5.7.9 svt-av1-1.0.0 tcl86-8.6.12 tiff-4.3.0 tk86-8.6.12 vmaf-2.3.1 wayland-1.20.0_2 webp-1.2.2 x265-3.4_2 xorgproto-2021.5 xvid-1.3.7,1
WWW:    https://github.com/facebookresearch/pytorchvideo
Or
Code:
root@molly:/usr/ports # pkg install psearch
Updating dicelan-server repository catalogue...
dicelan-server repository is up to date.
All repositories are up to date.
Checking integrity... done (0 conflicting)
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        psearch: 2.1.0

Number of packages to be installed: 1

Proceed with this action? [y/N]: y
[1/1] Installing psearch-2.1.0...
[1/1] Extracting psearch-2.1.0: 100%
root@molly:/usr/ports # psearch pytorch
math/py-pytorchvideo      Video understanding deep learning library
Thank you,

That is actually PyTorchVideo which is not the same as PyTorch. The thing is that I don't get why a port has been developed for PyTorchVideo when i believe PyTorch is needed to use that library.

Or is it perhaps something that I am missing?

I've tired the search function but since the user yurivict made so many posts about making a port on GitHub I made a last effort try and ask the question here.
 
Hello & Welcome to this FreeBSD community!


I actually had to look into this a few weeks ago. As per my knowledge, there is no port available. Therefore, I started compiling from source. I got libtorch to compile successfully after a few very minor patches and careful selection of configuration options. I can share that if you like.
While I was also able to build a program using libtorch it is still encountering some runtime issue that I have to figure out.

What I mean to say is this: Compiling the source on FreeBSD does appear to be an option without too much hassle. Therefore, creating a port would be too but I don't know much about the python side.
Thank you!

It would be awesome if i could get some help at getting closer to run the package PyTorch on my server.

I am very new to using FreeBSD so I might not understand everything but I will try my best!
 

SirDice

Administrator
Staff member
Administrator
Moderator
That is actually PyTorchVideo which is not the same as PyTorch.
The question was, " if there exits a port of PyTorch". To which I showed you how you can look this up. And the search showed no such port exist, thus answering your question and showing you how to figure this out on your own.
 

SirDice

Administrator
Staff member
Administrator
Moderator
The thing is that I don't get why a port has been developed for PyTorchVideo when i believe PyTorch is needed to use that library.
Looks like PyTorch is simply embedded in the PyTorchVideo code. It cannot have a dependency on PyTorch because that port doesn't exist.

That said, if there's a port for PyTorchVideo (which uses PyTorch), it should be fairly straight-forward to create a 'stand-alone' port for PyTorch itself.
 
Looks like PyTorch is simply embedded in the PyTorchVideo code. It cannot have a dependency on PyTorch because that port doesn't exist.

That said, if there's a port for PyTorchVideo (which uses PyTorch), it should be fairly straight-forward to create a 'stand-alone' port for PyTorch itself.
Does that mean that if I can install PyTorchVideo I can use PyTorch since it is embedded in it?

Sorry for the dumb questions.
 

SirDice

Administrator
Staff member
Administrator
Moderator
May I ask how you determined that?
Because the README explained it used the PyTorch code and the port has no dependency on it. It's also not loaded as an additional distfile. So it must be included with the PyTorchVideo source code itself.
 
The README states that:

Based on PyTorch: Built using PyTorch. Makes it easy to use all of the PyTorch-ecosystem components.
In my opinion this does not clearly indicate that it includes PyTorch itself.
The repository also seems way to lean for this.

I'm not saying you are wrong. I am asking because I might have misunderstood this and I have a pending project where PyTorch on FreeBSD will have to be a thing. If I can avoid jumping through some extra hoops I'd gladly do that.
From what I understand PyTorchVideo does not include/contains/ship a "reusable PyTorch" library or components.
The more likely possibilities I see is that this either requires PyTorch itself to be present to actually use the PyTorchVideo components (which would therefore be portable to FreeBSD without a dependency to PyTorch itself as the PyTorchVideo port "doesn't care" or it is a packaged deal where you don't get the full (or reusable) PyTorch but rather just what is needed for PyTorchVideo. The latter would be alining with previous experiences I made when working with ML/AI stuff from Google, Facebook & co.

Again: I'm trying to gather information, not to have an argument. I would prefer to be wrong about this.
 

SirDice

Administrator
Staff member
Administrator
Moderator
The more likely possibilities I see is that this either requires PyTorch itself to be present to actually use the PyTorchVideo components (which would therefore be portable to FreeBSD without a dependency to PyTorch itself as the PyTorchVideo port "doesn't care" or it is a packaged deal where you don't get the full (or reusable) PyTorch but rather just what is needed for PyTorchVideo.
Port wouldn't be allowed if it had runtime/library dependencies some 'external' code (or portions of something) that isn't in the ports tree. All dependencies must be met for the port to be added. It's also not allowed for build scripts to fetch additional code, everything must be downloaded during the 'fetch' phase of the port. During the 'build' phase poudriere actually disables the network for that jail.

That said, I honestly don't know how PyTorch works. But is it possible PyTorch was used to generate code? And this generated code is what is embedded in PyTorchVideo?
 
claudec as requested here is at least some information regarding my efforts.

On my stable/13 machine, I managed to get libtorch.so to build successfully using the following configuration:
Code:
cmake \
    -B build \
    -DBUILD_SHARED_LIBS:BOOL=ON \
    -DBUILD_BINARY:BOOL=ON \
    -DBUILD_CAFFE2=ON \
    -DCMAKE_BUILD_TYPE:STRING=Release \
    -DBUILD_PYTHON=OFF \
    -DBUILD_TEST=OFF \
    -DONNX_ML=OFF \
    -DUSE_OPENCV=ON \
    -DUSE_CUDA=OFF \
    -DUSE_CUDNN=OFF \
    -DUSE_BREAKPAD=OFF \
    -DUSE_FBGEMM=OFF \
    -DUSE_KINETO=OFF \
    -DBUILD_CUSTOM_PROTOBUF=OFF \
    \
    -DCMAKE_CXX_FLAGS="-Wno-error=bitwise-instead-of-logical -Wno-error=unused-but-set-variable -Wno-error=unused-parameter -Wno-error=sign-compare -pthread"

As usual, select your compiler(s) and Generator as needed. You most likely also want to use a compiler cache for this one because it's 1'700 files of beefy stuff. Took quite a while to build.
This shouldn't matter but I'd rather state it upfront: I used devel/llvm14 rather than clang from base (for unrelated reasons).

The only quirky part in that configuration is the manual setting of the C++ flags. This is due to this "bug": https://github.com/pytorch/FBGEMM/issues/957
This should most likely not be the final configuration tho. I'm especially eyeballing -pthread as this should definitely be handled by CMake itself but as I stated initially I was only spending an hour or two on trying to get this to build.

You definitely want to use the latest `master` branch too. The latest release came with a few more issues that have been addressed in the meantime. On that note, USE_BREAKPAD doesn't seem to be a CMake option anymore in the current master but as stated prior, I did not invest the time needed to get this to build "properly" yet.

Other caveats apply as usual such as the fact that CUDA will most likely not work on FreeBSD (as per my understanding). I also had some issues regarding ONNX but decided to just disable it until I have a working system.

As far as dependencies go: The only one I had to install was devel/py-typing-extensions. I assume everything else was already present on my system but dependencies should be comparably easy to handle (especially in case of python packages).

After successfully building `libtorch.so` you'll most likely run into some compiler & runtime errors when actually trying to link & load it. This will almost definitely also propagate to the python wrappers/bindings/interfaces but those are of no interest to me.
 
Top