Solved Port Makefile Causing Infinite Loop Of Folders

Maybe I worded that poorly. I don't know if the NO_ARCH from before was due to me messing with the ports system too much and mucking things up. I just thought it was worth mentioning since I saw this come up again with a clean install. It may have nothing to do with that module, but lead to a clue in the failures of my own code. If that came across wrong, my apologies.
NO_ARCH does not make a port immune to the effects of circular dependencies. Don't select all options for all ports.

Case in point: security/krb5 includes an LDAP option. This causes security/krb5 to add net/openldap*-server as a build/run dependency. net/openldap*-server includes a GSSAPI option. Enabling it results in net/openldap*-server depending on security/krb5. So, when security/krb5 starts to build it tries to build net/openldap*-server first. Then openldap*-server tries to build krb5 before it. You have a circular dendency. I've had to deal with a number of PRs regarding this very issue. And, a good number of times the person reporing the issue simply doesn't understand the concept of circular dependency and so we're stuck. One person not understanding the concept of circular dependencies simply wanting it to work and the developer telling them it's a chicken and egg scenario.

Enabling all options in all ports is a surefire way to the proverbial chicken and egg scenario in ports.
 
This should help move you along...

Makefile:

Makefile:
PORTNAME=       theZoo
PORTVERSION=    0.6.0.20231026
CATEGORIES=     security python
PKGNAMEPREFIX=  ${PYTHON_PKGNAMEPREFIX}

USE_GITHUB=     yes
GH_ACCOUNT=     ytisf
GH_TAGNAME=     398d83938838ae6f92117d9dbcd07833dd1dcaaf

USES=           python
USE_PYTHON=     autoplist concurrent distutils

RUN_DEPENDS=    ${PYTHON_PKGNAMEPREFIX}urllib3>0:net/py-urllib3@${PY_FLAVOR} \
                ${PYTHON_PKGNAMEPREFIX}pyminizip>0:archivers/py-pyminizip@${PY_FLAVOR}

post-extract:
        cp ${PATCHDIR}/setup.py ${WRKSRC}/

.include <bsd.port.mk>

Run make makesum to fetch the distfile and make the distinfo file.

At this point, make will fail because there is no setup.py. You can either write a do-install to copy the files, or write setup.py. I opted to write setup.py, as you can see referenced in the post-extract target. It will need to live under files/, and look something like:

Python:
from setuptools import setup, find_packages

setup(
    name='theZoo',
    version='0.6.0',
    packages=['imports'],
    install_requires=[
        'urllib3',
        'pyminizip'
    ],
    package_data={'': ['theZoo.py'], 'imports': ['*.py']}
)

I don't work with Python, so I don't really know... but that seems to have been a start.

In the end, you should have the following files:
  • security/py-theZoo/Makefile
  • security/py-theZoo/distinfo
  • security/py-theZoo/pkg-descr
  • security/py-theZoo/files/setup.py
With a working setup.py, you should be good to go. Then you should consider submitting setup.py as an upstream patch so you don't have to include it in the port.

Well, it seems you removed py-sqlite3 from RUN_DEPENDS in the Makefile you just proposed and that inspired me to just try the dependencies in a different order. In order to pass portlint I adjusted the Makefile to this:

PORTNAME= theZoo
PORTVERSION= 0.60
CATEGORIES= security
MASTER_SITES= https://teamsloth.net/

MAINTAINER= notmyemail@gmail.com
COMMENT= Malware Research
WWW= https://thezoo.morirt.com

LICENSE= GPLv3

RUN_DEPENDS= ${PYTHON_PKGNAMEPREFIX}urllib3>0:net/py-urllib3@${PY_FLAVOR} \
${PYTHON_PKGNAMEPREFIX}sqlite3>0:databases/py-sqlite3@${PY_FLAVOR} \
${PYTHON_PKGNAMEPREFIX}pyminizip>0:archivers/py-pyminizip@${PY_FLAVOR}

USES= python
USE_PYTHON= autoplist concurrent distutils

post-extract:
${CP} ${PATCHDIR}/setup.py ${WRKSRC}/

.include <bsd.port.mk>


I do have a setup.py file at
/usr/ports/security/theZoo/work-py39/theZoo-0.60/setup.py. I'm no expert at python or making a port with Python, but from what I read, this should be the correct location for it. So the post-extract line above fails like this:

# make
===> License GPLv3 accepted by the user
===> theZoo-0.60 depends on file: /usr/local/sbin/pkg - found
===> Fetching all distfiles required by theZoo-0.60 for building
===> Extracting for theZoo-0.60
=> SHA256 Checksum OK for theZoo-0.60.tar.gz.
cp -f /usr/ports/security/theZoo/files/setup.py /usr/ports/security/theZoo/work-py39/theZoo-0.60/
cp: /usr/ports/security/theZoo/files/setup.py: No such file or directory
*** Error code 1


I adjusted this to the following to try and reflect the correct location:

post-extract:
${CP} ${PORTSDIR}/security/theZoo/work-${PY_FLAVOR}/${PORTNAME}-${PORTVERSION}/theZoo.py ${LOCALBASE}/bin/theZoo


This seems to have worked better than anything so far, but after a lot of output it eventually failed with the following:

...
Copying theZoo.egg-info to /usr/ports/security/theZoo/work-py39/stage/usr/local/lib/python3.9/site-packages/theZoo-0.60-py3.9.egg
-info
running install_scripts
copying build/scripts-3.9/theZoo.py -> /usr/ports/security/theZoo/work-py39/stage/usr/local/bin
changing mode of /usr/ports/security/theZoo/work-py39/stage/usr/local/bin/theZoo.py to 755
writing list of installed files to '/usr/ports/security/theZoo/work-py39/.PLIST.pymodtmp'
===> Creating unique files: Move MAN files needing SUFFIX
===> Creating unique files: Move files needing SUFFIX
Move: bin/theZoo.py --> bin/theZoo.py-3.9
Link: @bin/theZoo.py --> bin/theZoo.py-3.9
Makefile error: UNIQUE (suffix): bin/theZoo not found
*** Error code 1


This did allow /usr/local/bin/theZoo to be put in place, but it seems that setup.py did not complete as it does not work:

# theZoo
Traceback (most recent call last):
File "/usr/local/bin/theZoo", line 23, in
from imports.update_handler import Updater
ModuleNotFoundError: No module named 'imports'


It seems from the previous Make output that bin/theZoo.py was trying to be copied to bin/theZoo.py-3.9, which wouldn't make sense. At least if you look at the code I posted for setup.py earlier. Any suggestions welcome.
 
NO_ARCH does not make a port immune to the effects of circular dependencies. Don't select all options for all ports.

Case in point: security/krb5 includes an LDAP option. This causes security/krb5 to add net/openldap*-server as a build/run dependency. net/openldap*-server includes a GSSAPI option. Enabling it results in net/openldap*-server depending on security/krb5. So, when security/krb5 starts to build it tries to build net/openldap*-server first. Then openldap*-server tries to build krb5 before it. You have a circular dendency. I've had to deal with a number of PRs regarding this very issue. And, a good number of times the person reporing the issue simply doesn't understand the concept of circular dependency and so we're stuck. One person not understanding the concept of circular dependencies simply wanting it to work and the developer telling them it's a chicken and egg scenario.

Enabling all options in all ports is a surefire way to the proverbial chicken and egg scenario in ports.

You may have missed I'm on a fresh install of FreeBSD 14 now and NO_ARCH is no longer an issue. I've replied here a LOT, so I can fault anyone for missing that. Thanks though.
 
The original problem of the port seemingly 'looping' was simply because the Makefile is included in the original theZoo-0.60.tar.gz distfile.

I'll show you. I've set up your port:
Code:
dice@fbsd-test:~/Sources/forum-port % ll
total 9
-rw-r--r--  1 dice  dice  1432 Oct 24 23:23 Makefile
-rw-r--r--  1 dice  dice   156 Oct 24 22:48 distinfo
make fetch fetches the file, so I have it:
Code:
dice@fbsd-test:~/Sources/forum-port % ls -al /usr/ports/distfiles/theZoo-0.60.tar.gz
-rw-r--r--  1 root  wheel  919235180 Oct 17 15:13 /usr/ports/distfiles/theZoo-0.60.tar.gz
make extract will extract the file in the 'work' directory.
Code:
dice@fbsd-test:~/Sources/forum-port % make extract
===>  License GPLv3 accepted by the user
===>   theZoo-0.60 depends on file: /usr/local/sbin/pkg - found
===> Fetching all distfiles required by theZoo-0.60 for building
===>  Extracting for theZoo-0.60
=> SHA256 Checksum OK for theZoo-0.60.tar.gz.
dice@fbsd-test:~/Sources/forum-port % ls -al work/
total 10
drwxr-xr-x  3 dice  dice   4 Oct 27 21:46 .
drwxr-xr-x  3 dice  dice   5 Oct 27 21:46 ..
-rw-r--r--  1 dice  dice   0 Oct 27 21:46 .extract_done.theZoo._usr_local
drwxr-xr-x  5 dice  dice  13 Oct 17 15:12 theZoo-0.60
Code:
dice@fbsd-test:~/Sources/forum-port % ls -al work/theZoo-0.60/
total 55
drwxr-xr-x  5 dice  dice    13 Oct 17 15:12 .
drwxr-xr-x  3 dice  dice     4 Oct 27 21:46 ..
-rw-r--r--  1 dice  dice  6039 Oct 10 20:48 CODE-OF-CONDUCT.md
-rw-r--r--  1 dice  dice  2561 Oct 10 20:48 CONTRIBUTING.md
-rw-r--r--  1 dice  dice   471 Oct 10 20:48 LICENSE.md
-rw-r--r--  1 dice  dice  1129 Oct 17 15:12 Makefile
-rw-r--r--  1 dice  dice  7041 Oct 10 20:48 README.md
drwxr-xr-x  2 dice  dice     5 Oct 16 19:22 conf
drwxr-xr-x  8 dice  dice    22 Oct 16 19:29 imports
drwxr-xr-x  4 dice  dice     4 Oct 10 20:48 malware
-rw-r--r--  1 dice  dice  3436 Oct 10 20:48 prep_file.py
-rw-r--r--  1 dice  dice    18 Oct 10 20:48 requirements.txt
-rwxr-xr-x  1 dice  dice  3652 Oct 16 01:32 theZoo.py
There is a Makefile there. The 'default' action of a port Makefile is to extract the distfile(s) and run make in the ${WRKSRC} directory. Which would blissfully start to build the Makefile in the work/theZoo-0.60/ directory, this is more or less a copy of the port's Makefile. So it will try to fetch and extract the distfile, etc.

Regarding RUN_DEPENDS, I had to fiddle a lot with spaces and tabs. Somehow copy/pasting seem to have introduced a lot of extra spaces at the end of the lines. make(1) is very finicky about spaces and tabs.
Code:
RUN_DEPENDS=    ${PYTHON_PKGNAMEPREFIX}urllib3>=1.26.0:net/py-urllib3@${PY_FLAVOR} \
                ${PYTHON_PKGNAMEPREFIX}pyminizip>0:archivers/py-pyminizip@${PY_FLAVOR} \
                ${PYTHON_PKGNAMEPREFIX}sqlite3>0:databases/py-sqlite3@${PY_FLAVOR}
Make sure there are no spaces after the \ at the end of the lines. Make sure there is no extra whitespace at the end of any line actually.
 
The original problem of the port seemingly 'looping' was simply because the Makefile is included in the original theZoo-0.60.tar.gz distfile.

I'll show you. I've set up your port:
Code:
dice@fbsd-test:~/Sources/forum-port % ll
total 9
-rw-r--r--  1 dice  dice  1432 Oct 24 23:23 Makefile
-rw-r--r--  1 dice  dice   156 Oct 24 22:48 distinfo
make fetch fetches the file, so I have it:
Code:
dice@fbsd-test:~/Sources/forum-port % ls -al /usr/ports/distfiles/theZoo-0.60.tar.gz
-rw-r--r--  1 root  wheel  919235180 Oct 17 15:13 /usr/ports/distfiles/theZoo-0.60.tar.gz
make extract will extract the file in the 'work' directory.
Code:
dice@fbsd-test:~/Sources/forum-port % make extract
===>  License GPLv3 accepted by the user
===>   theZoo-0.60 depends on file: /usr/local/sbin/pkg - found
===> Fetching all distfiles required by theZoo-0.60 for building
===>  Extracting for theZoo-0.60
=> SHA256 Checksum OK for theZoo-0.60.tar.gz.
dice@fbsd-test:~/Sources/forum-port % ls -al work/
total 10
drwxr-xr-x  3 dice  dice   4 Oct 27 21:46 .
drwxr-xr-x  3 dice  dice   5 Oct 27 21:46 ..
-rw-r--r--  1 dice  dice   0 Oct 27 21:46 .extract_done.theZoo._usr_local
drwxr-xr-x  5 dice  dice  13 Oct 17 15:12 theZoo-0.60
Code:
dice@fbsd-test:~/Sources/forum-port % ls -al work/theZoo-0.60/
total 55
drwxr-xr-x  5 dice  dice    13 Oct 17 15:12 .
drwxr-xr-x  3 dice  dice     4 Oct 27 21:46 ..
-rw-r--r--  1 dice  dice  6039 Oct 10 20:48 CODE-OF-CONDUCT.md
-rw-r--r--  1 dice  dice  2561 Oct 10 20:48 CONTRIBUTING.md
-rw-r--r--  1 dice  dice   471 Oct 10 20:48 LICENSE.md
-rw-r--r--  1 dice  dice  1129 Oct 17 15:12 Makefile
-rw-r--r--  1 dice  dice  7041 Oct 10 20:48 README.md
drwxr-xr-x  2 dice  dice     5 Oct 16 19:22 conf
drwxr-xr-x  8 dice  dice    22 Oct 16 19:29 imports
drwxr-xr-x  4 dice  dice     4 Oct 10 20:48 malware
-rw-r--r--  1 dice  dice  3436 Oct 10 20:48 prep_file.py
-rw-r--r--  1 dice  dice    18 Oct 10 20:48 requirements.txt
-rwxr-xr-x  1 dice  dice  3652 Oct 16 01:32 theZoo.py
There is a Makefile there. The 'default' action of a port Makefile is to extract the distfile(s) and run make in the ${WRKSRC} directory. Which would blissfully start to build the Makefile in the work/theZoo-0.60/ directory, this is more or less a copy of the port's Makefile. So it will try to fetch and extract the distfile, etc.

Regarding RUN_DEPENDS, I had to fiddle a lot with spaces and tabs. Somehow copy/pasting seem to have introduced a lot of extra spaces at the end of the lines. make(1) is very finicky about spaces and tabs.
Code:
RUN_DEPENDS=    ${PYTHON_PKGNAMEPREFIX}urllib3>=1.26.0:net/py-urllib3@${PY_FLAVOR} \
                ${PYTHON_PKGNAMEPREFIX}pyminizip>0:archivers/py-pyminizip@${PY_FLAVOR} \
                ${PYTHON_PKGNAMEPREFIX}sqlite3>0:databases/py-sqlite3@${PY_FLAVOR}
Make sure there are no spaces after the \ at the end of the lines. Make sure there is no extra whitespace at the end of any line actually.

The original problem of the port seemingly 'looping' was resolved a long time ago after many changes to the Makefile in /usr/ports/security/theZoo. I can't fault you or anyone for missing that as we are many replies deep now. However, there being a Makefile in /usr/ports/security/theZoo/work-py39/theZoo was not the issue either. Originally it had an error there was no Makefile there. I put one there and it wanted a distinfo file and so on until it became recursive. That was resolved a while ago if you read back through these replies. I understand its a lot though. You can even remove /usr/ports/security/theZoo/work-py39/theZoo/Makefile as its using the setup.py there now instead. You may have to refresh my code, maybe 'rm /usr/ports/distfiles/theZoo-0.60.tar.gz', but you should now get the same error with or without the /usr/ports/security/theZoo/work-py39/theZoo/Makefile. The new Makefile, distinfo, etc can be found here too if that helps.

As far as the spaces and tabs, I have this working when I run portlint or port test . I might have messed up while copying and pasting or maybe the formatting got lost. The problem with the dependencies noted in RUN_DEPENDS also got resolved. If you run the newer code I mentioned you should see the error discussed in reply #27. No disrespect, just want to explain and be clear where I'm at now. Thanks
 
The original problem of the port seemingly 'looping' was resolved a long time ago after many changes to the Makefile in /usr/ports/security/theZoo.
I know. USE_PYTHON= autoplist concurrent distutils instructs the port system to use a setup.py in the project.

However, there being a Makefile in /usr/ports/security/theZoo/work-py39/theZoo was not the issue either.
It is. It's not being used any more because you've instructed the ports system to use setup.py. It's still there though. It only confuses now, just remove it.
Originally it had an error there was no Makefile there.
Yes. Because the standard port Makefile 'action' tried to execute make(1) in the ${WRKSRC} (that's work/theZoo-0.60/) directory. So you get an error, something along the lines of make: no target to make.. Just remove it, it was never needed and it's not being used now any way.

cy@ was talking about dependency loops. Which could indeed be a problem, but it's a different kind of 'looping' that happened here. I was trying to explain where the original 'looping' issue came from.

A 'simple' way to test your dependencies is make run-depends-list. Another useful one; make install-missing-packages. The dependencies I copied worked for me.

Code:
dice@fbsd-test:~/Sources/forum-port % make run-depends-list
/usr/ports/net/py-urllib3
/usr/ports/archivers/py-pyminizip
/usr/ports/databases/py-sqlite3
/usr/ports/lang/python39
 
I know. USE_PYTHON= autoplist concurrent distutils instructs the port system to use a setup.py in the project.


It is. It's not being used any more because you've instructed the ports system to use setup.py. It's still there though. It only confuses now, just remove it.

Yes. Because the standard port Makefile 'action' tried to execute make(1) in the ${WRKSRC} (that's work/theZoo-0.60/) directory. So you get an error, something along the lines of make: no target to make.. Just remove it, it was never needed and it's not being used now any way.

cy@ was talking about dependency loops. Which could indeed be a problem, but it's a different kind of 'looping' that happened here. I was trying to explain where the original 'looping' issue came from.

A 'simple' way to test your dependencies is make run-depends-list. Another useful one; make install-missing-packages. The dependencies I copied worked for me.

Code:
dice@fbsd-test:~/Sources/forum-port % make run-depends-list
/usr/ports/net/py-urllib3
/usr/ports/archivers/py-pyminizip
/usr/ports/databases/py-sqlite3
/usr/ports/lang/python39

Well, my apologies. I misunderstood and thought you might have missed what was said since your last reply. I just uploaded that content again without /usr/ports/security/theZoo/work-py39/theZoo/Makefile in the code. I also removed /usr/ports/distfiles/theZoo-0.60.tar.gz to be safe. This is the new distinfo file to use:

TIMESTAMP = 1698428060
SHA256 (theZoo-0.60.tar.gz) = a944e8dedb8be95e17d6508b8fb8e5de8282fffe3a492e8a61fcf6288a0912f5
SIZE (theZoo-0.60.tar.gz) = 919283247


I get this with run-depends-list:

# make run-depends-list
/usr/ports/net/py-urllib3
/usr/ports/databases/py-sqlite3
/usr/ports/archivers/py-pyminizip
/usr/ports/devel/py-setuptools
/usr/ports/lang/python39


Running install-missing-packages returns nothing now as all the dependencies are now installed. This is the output when I try to build now:

# make install clean
....
writing theZoo.egg-info/PKG-INFO
writing dependency_links to theZoo.egg-info/dependency_links.txt
writing top-level names to theZoo.egg-info/top_level.txt
listing git files failed - pretending there aren't any
reading manifest file 'theZoo.egg-info/SOURCES.txt'
adding license file 'LICENSE.md'
writing manifest file 'theZoo.egg-info/SOURCES.txt'
Copying theZoo.egg-info to /usr/ports/security/theZoo/work-py39/stage/usr/local/lib/python3.9/site-packages/theZoo-0.60-py3.9.egg-info
running install_scripts
copying build/scripts-3.9/theZoo.py -> /usr/ports/security/theZoo/work-py39/stage/usr/local/bin
changing mode of /usr/ports/security/theZoo/work-py39/stage/usr/local/bin/theZoo.py to 755
writing list of installed files to '/usr/ports/security/theZoo/work-py39/.PLIST.pymodtmp'
===> Creating unique files: Move MAN files needing SUFFIX
===> Creating unique files: Move files needing SUFFIX
Move: bin/theZoo.py --> bin/theZoo.py-3.9
Link: @bin/theZoo.py --> bin/theZoo.py-3.9
Makefile error: UNIQUE (suffix): bin/theZoo not found
*** Error code 1

Stop.
make: stopped in /usr/ports/security/theZoo


Seems like I'm still on the same error unless I've missed something.
 
I do have a setup.py file at
/usr/ports/security/theZoo/work-py39/theZoo-0.60/setup.py. I'm no expert at python or making a port with Python, but from what I read, this should be the correct location for it
It’s not the correct location for you to put it. make clean removes work/, so anything you put in there will be wiped away. The point of ports is to produce a repeatable build. If you need to supply any files or patches, you put them in files/ and reference them in Makefile using ${PATCHDIR}.

I gave you the answer, as far as writing the port. Use what I gave you as a starting point, get setup.py right, and you’re done. Or do something else ;)
 
It’s not the correct location for you to put it. make clean removes work/, so anything you put in there will be wiped away. The point of ports is to produce a repeatable build. If you need to supply any files or patches, you put them in files/ and reference them in Makefile using ${PATCHDIR}.

I gave you the answer, as far as writing the port. Use what I gave you as a starting point, get setup.py right, and you’re done. Or do something else ;)

I think I misunderstood your previous reply. I thought files/ was just a random name or place holder for work-py or work-py/theZoo-0.60. I just tried the exact Makefile you provided last time minus the RUN_DEPENDS as I do it from setup.py now to reduce some warnings when I build. I also made a folder at /usr/ports/security/theZoo/files and put setup.py inside it with the suggestions you had also made for it. I tried it with setup.py still inside /usr/ports/security/theZoo/work-py39/theZoo-0.60 and without it in there, which is what the Makefile will fetch now. Both times I came back to the same error as I got before:


adding license file 'LICENSE.md'
writing manifest file 'theZoo.egg-info/SOURCES.txt'
Copying theZoo.egg-info to /usr/ports/security/theZoo/work-py39/stage/usr/local/lib/python3.9/site-packages/theZoo-0.60-py3.9.egg
-info
running install_scripts
copying build/scripts-3.9/theZoo.py -> /usr/ports/security/theZoo/work-py39/stage/usr/local/bin
changing mode of /usr/ports/security/theZoo/work-py39/stage/usr/local/bin/theZoo.py to 755
writing list of installed files to '/usr/ports/security/theZoo/work-py39/.PLIST.pymodtmp'
===> Creating unique files: Move MAN files needing SUFFIX
===> Creating unique files: Move files needing SUFFIX
Move: bin/theZoo.py --> bin/theZoo.py-3.9
Link: @bin/theZoo.py --> bin/theZoo.py-3.9
Makefile error: UNIQUE (suffix): bin/theZoo not found
*** Error code 1

Stop.
make: stopped in /usr/ports/security/theZoo


I'll admit at first I completely misunderstood, but now I think I'm still missing something. If you want to double-check my Makefile, files/setup.py and grab the new distinfo I uploaded them here again. It does make sense that 'make clean' is going to remove work-py39. Maybe my setup.py is just off.

EDIT: Right after posting this I tried running files/setup.py and see the paths its looking for are off. I'll post back if it succeeds or not after fixing that.

EDIT RESPONSE: I adjusted the paths and 'setup.py build' and setup.py install' inside the files directory seemed to work. However, however, when running 'make install clean' inside /usr/ports/security/theZoo failed due to the paths, so I reverted things back. For now I'm still looking at the same error as the last few replies.
 
Well, I started thinking about that error I was having and eventually concluded it was due to the pkg-plist file. I removed it and the port began working on a box I had FreeBSD 13 on. I went to confirm this on a fresh install of FreeBSD 14 and found I still needed to put RUN_DEPENDS back in the Makefile and append this line to the end of post-extract in the Makefile too:

${CP} ${PORTSDIR}/security/theZoo/work-${PY_FLAVOR}/${PORTNAME}-${PORTVERSION}/theZoo.py ${LOCALBASE}/bin/theZoo

This allowed the port to build successfully. I don't remember how to make a thread SOLVED here, but if someone will tell me I won't have to ask again. Thank you to everyone for your help and keeping me motivated!
 
Back
Top