Solved Port Makefile Causing Infinite Loop Of Folders

I haven't posted here in a while. I took a break from BSD and came back on fire for it. Last week I re-wrote parts of an old Linux app to work with FreeBSD and Python 3 vs 2. I was very happy how it turned out and wondered if I could make it into a port. I had never tired anything like this before, but made a folder at /usr/ports/security/theZoo on my local machine. Inside it I have this in distinfo:

Code:
TIMESTAMP = 1697549012                                                                                                              
SHA256 (theZoo-0.60.tar.gz) = 6d8022e79d7f750ce112bbbb1734c8e82e7a7d6821792d619c507c508249b47d                                      
SIZE (theZoo-0.60.tar.gz) = 919235180

This is my Makefile:

Code:
PORTNAME=       theZoo                                                                                                              
PORTVERSION=    0.60                                                                                                                
CATEGORIES=     security                                                                                                            
MASTER_SITES=   https://teamsloth.net/                                                                                              
                                                                                                                                    
MAINTAINER=     notmyemail@gmail.com                                                                                                
COMMENT=        Malware Research                                                                                                    
WWW=            https://thezoo.morirt.com                                                                                           
                                                                                                                                    
LICENSE=        GPLv3                                                                                                               
                                                                                                                                    
PVERS != pkg info -x python | perl -p -e 's/python3/python3\./g' | awk -F- '{ print $$1}'                                           
FETCH_DEPENDS+=    curl:ftp/curl                                                                                                    
WRKSRC = ${WRKDIR}/${DISTNAME}                                                                                                      
INSTALL_DIR = /usr/local/bin                                                                                                        
LOCALBASE = /usr/local                                                                                                              
PREFIX = /usr/local/etc                                                                                                             
                                                                                                                                    
do-install: all                                                                                                                     
                                                                                                                                    
        pip install --no-warn-script-location -r ${WRKSRC}/requirements.txt                                                         
        install -m 755 ${WRKSRC}/theZoo.py ${INSTALL_DIR}/theZoo                                                                    
        ${MKDIR} ${PREFIX}/theZoo/                                                                                                  
        ${CP} -a ${WRKSRC}/imports ${LOCALBASE}/lib/$(PVERS)/site-packages/                                                         
        ${CP} -a ${WRKSRC}/conf ${PREFIX}/theZoo/                                                                                   
        ${CP} -a ${WRKSRC}/malware ${PREFIX}/theZoo/                                                                                
        ${CP} ${WRKSRC}/*.md ${PREFIX}/theZoo                                                                                       
                                                                                                                                    
deinstall:                                                                                                                          
        ${RM} -f ${INSTALL_DIR}/theZoo                                                                                              
        ${RM} -rf ${WRKDIR}                                                                                                         
        ${RM} -rf ${LOCALBASE}/lib/${PVERS}/site-packages/imports                                                                   
                                                                                                                                    
clean: install                                                                                                                      
        ${RM} -rf work                        

.include <bsd.port.mk>

I haven't added it yet, but sqlite3 will also needed to be installed with pip. The problem is that when I try to build it I get:

Code:
# make                                                                                                                              
===>  License GPLv3 accepted by the user                                                                                            
===>   theZoo-0.60 depends on file: /usr/local/sbin/pkg - found                                                                     
===>   theZoo-0.60 depends on executable: curl - found                                                                              
===> Fetching all distfiles required by theZoo-0.60 for building                                                                    
===>  Extracting for theZoo-0.60                                                                                                    
=> SHA256 Checksum OK for theZoo-0.60.tar.gz.                                                                                       
===>  Patching for theZoo-0.60                                                                                                      
===>  Configuring for theZoo-0.60                                                                                                   
===>  Building for theZoo-0.60                                                                                                      
--- show-dev-warnings ---                                                                                                           
/!\ theZoo-0.60: Makefile warnings, please consider fixing /!\                                                                      
Not validating first entry in CATEGORIES due to being outside of PORTSDIR.                                                          
Please ensure this is proper when committing.                                                                                       
--- check-vulnerable ---                                                                                                            
--- check-license ---                                                                                                               
===>  License GPLv3 accepted by the user                                                                                            
--- pkg-depends ---                                                                                                                 
===>   theZoo-0.60 depends on file: /usr/local/sbin/pkg - found                                                                     
--- fetch-depends ---                                                                                                               
===>   theZoo-0.60 depends on executable: curl - found                                                                              
--- do-fetch ---                                                                                                                    
--- fetch-specials ---                                                                                                              
===> Fetching all distfiles required by theZoo-0.60 for building                                                                    
--- extract-message ---                                                                                                             
===>  Extracting for theZoo-0.60                                                                                                    
--- checksum ---                                                                                                                    
=> No checksum file (/usr/ports/security/theZoo/work/theZoo-0.60/distinfo).                                                         
*** [checksum] Error code 1                                                                                                         
                                                                                                                                    
make[1]: stopped in /usr/ports/security/theZoo/work/theZoo-0.60                                                                     
1 error                                                                                                                             
                                                                                                                                    
make[1]: stopped in /usr/ports/security/theZoo/work/theZoo-0.60                                                                     
===> Compilation failed unexpectedly.                                                                                               
Try to set MAKE_JOBS_UNSAFE=yes and rebuild before reporting the failure to                                                         
the maintainer.                                                                                                                     
*** Error code 1                               

Stop.                                                                                                                               
make: stopped in /usr/ports/security/theZoo

You can probably see from this it starts to look for a Makefile, distinfo, etc. in /usr/ports/security/theZoo/work/theZoo-0.60 instead of /usr/ports/security/theZoo as it should. Once you put all those files in /usr/ports/security/theZoo/work/theZoo-0.60 it creates another work folder in there and restarts the whole process again, leading to an infinite loop of folders and unfinished builds. I tried looking in the online handbook and with portlint, but only saw some warnings that appeared to be unrelated:

Code:
# portlint
WARN: Makefile: [15]: possible direct use of command "awk" found. use ${AWK} instead.
WARN: Makefile: [23]: possible direct use of command "install" found. use ${INSTALL_foobar} instead.
WARN: Makefile: [25]: possible direct use of command "install" found. use ${INSTALL_foobar} instead.
WARN: Makefile: [36]: possible direct use of command "install" found. use ${INSTALL_foobar} instead.
WARN: Makefile: [15]: possible direct use of command "python" found. use ${PYTHON_CMD} instead.
WARN: Makefile: [15]: possible direct use of command "perl" found. use ${PERL} instead.
WARN: Makefile: possible use of absolute pathname "/usr/local/bin".
WARN: Makefile: possible use of absolute pathname "/usr/local".
WARN: Makefile: possible use of absolute pathname "/usr/local/etc".
WARN: Makefile: possible direct use of "INSTALL_DIR = /usr/local" found. if so, use ${PREFIX} or ${LOCALBASE}, as appropriate. 
WARN: Makefile: possible direct use of "LOCALBASE = /usr/local" found. if so, use ${PREFIX} or ${LOCALBASE}, as appropriate.   
WARN: Makefile: possible direct use of "PREFIX = /usr/local" found. if so, use ${PREFIX} or ${LOCALBASE}, as appropriate.      
0 fatal errors and 12 warnings found.

If anyone wants to see the code before downloading it, the source can be found here, but I feel this is just an issue with the Makefile. I feel like if I could just get past this issue I could finish and do more of these. Does anyone know why this is happening? Or how I could fix it?
 
Remove all of this:
Code:
PVERS != pkg info -x python | perl -p -e 's/python3/python3\./g' | awk -F- '{ print $$1}'                                           
FETCH_DEPENDS+=    curl:ftp/curl                                                                                                    
WRKSRC = ${WRKDIR}/${DISTNAME}                                                                                                      
INSTALL_DIR = /usr/local/bin                                                                                                        
LOCALBASE = /usr/local                                                                                                              
PREFIX = /usr/local/etc
I'm not sure what you're trying to do here. But if you need to change the shebang lines, that can be done automatically. Don't set any of WRKSRC, INSTALL_DIR, LOCALBASE, etc. They're set automatically too.

Also bad:
Code:
pip install --no-warn-script-location -r ${WRKSRC}/requirements.txt
Port isn't allowed to fetch stuff, it's the ports system itself that takes care of this. Using the proper dependencies.

And as this appears to be python:

Code:
        install -m 755 ${WRKSRC}/theZoo.py ${INSTALL_DIR}/theZoo                                                                    
        ${MKDIR} ${PREFIX}/theZoo/                                                                                                  
        ${CP} -a ${WRKSRC}/imports ${LOCALBASE}/lib/$(PVERS)/site-packages/                                                         
        ${CP} -a ${WRKSRC}/conf ${PREFIX}/theZoo/                                                                                   
        ${CP} -a ${WRKSRC}/malware ${PREFIX}/theZoo/                                                                                
        ${CP} ${WRKSRC}/*.md ${PREFIX}/theZoo
Things get installed in a 'staging' area nowadays. From this staging area a package is created.
Code:
deinstall:                                                                                                                          
        ${RM} -f ${INSTALL_DIR}/theZoo                                                                                              
        ${RM} -rf ${WRKDIR}                                                                                                         
        ${RM} -rf ${LOCALBASE}/lib/${PVERS}/site-packages/imports
Don't need to do this. It's the package that gets removed when you deinstall something. So the package being created needs to be set up correctly.
 
Thanks for the quick and good answers! This is the Makefile I have now:

Code:
PORTNAME=       theZoo                                                                                                              
PORTVERSION=    0.60                                                                                                                
CATEGORIES=     security                                                                                                            
MASTER_SITES=   https://teamsloth.net/                                                                                              
                                                                                                                                    
MAINTAINER=     notmyemail@gmail.com                                                                                                
COMMENT=        Malware Research                                                                                                    
WWW=            https://thezoo.morirt.com                                                                                           
                                                                                                                                    
LICENSE=        GPLv3                                                                                                               
                                                                                                                                    
RUN_DEPENDS= ${PORTSDIR}/net/py-urllib3:${PORTSDIR}/archivers/py-pyminizip:${PORTSDIR}/databases/py-sqlite3                         
                                                                                                                                    
do-install: all                                                                                                                     
                                                                                                                                    
        install -m 755 /usr/ports/security/theZoo/work/theZoo-0.60/theZoo.py /usr/local/bin/theZoo                                  
        ${MKDIR} /usr/local/etc/theZoo/                                                                                             
        ${CP} -a /usr/ports/security/theZoo/work/theZoo-0.60/imports /usr/local/lib/python3.9/site-packages/                        
        ${CP} -a /usr/ports/security/theZoo/work/theZoo-0.60/conf /usr/local/etc/theZoo/                                            
        ${CP} -a /usr/ports/security/theZoo/work/theZoo-0.60/malware /usr/local/etc/theZoo/                                         
        ${CP} /usr/ports/security/theZoo/work/theZoo-0.60/*.md /usr/local/etc/theZoo                                                
                                                                                                                                    
clean: install                                                                                                                      
        ${RM} -rf work                                                                                                              
                                                                                                                                    
.include <bsd.port.mk>

I took out pip and using this link I added the RUN_DEPS line. I'll do some more testing to ensure this works on a new install, but even with the python modules already installed in my environment, this still fails with that infinite loop of creating more installation folders:

Code:
# make                                                                                                                              
===>  Building for theZoo-0.60                                                                                                      
--- show-dev-warnings ---                                                                                                           
/!\ theZoo-0.60: Makefile warnings, please consider fixing /!\                                                                      
Not validating first entry in CATEGORIES due to being outside of PORTSDIR.                                                          
Please ensure this is proper when committing.                                                                                       
--- check-vulnerable ---                                                                                                            
--- check-license ---                                                                                                               
===>  License GPLv3 accepted by the user                                                                                            
--- pkg-depends ---                                                                                                                 
===>   theZoo-0.60 depends on file: /usr/local/sbin/pkg - found                                                                     
--- fetch-depends ---                                                                                                               
===>   theZoo-0.60 depends on executable: curl - found                                                                              
--- do-fetch ---                                                                                                                    
--- fetch-specials ---                                                                                                              
===> Fetching all distfiles required by theZoo-0.60 for building                                                                    
--- extract-message ---                                                                                                             
===>  Extracting for theZoo-0.60                                                                                                    
--- checksum ---                                                                                                                    
=> No checksum file (/usr/ports/security/theZoo/work/theZoo-0.60/distinfo).                                                         
*** [checksum] Error code 1                                                                                                         
                                                                                                                                    
make[1]: stopped in /usr/ports/security/theZoo/work/theZoo-0.60                                                                     
1 error                                                                                                                             
                                                                                                                                    
make[1]: stopped in /usr/ports/security/theZoo/work/theZoo-0.60                                                                     
===> Compilation failed unexpectedly.                                                                                               
Try to set MAKE_JOBS_UNSAFE=yes and rebuild before reporting the failure to                                                         
the maintainer.                                                                                                                     
*** Error code 1                                                                                                                    
                                                                                                                                    
Stop.                                                                                                                               
make: stopped in /usr/ports/security/theZoo

I'm going to read over these links you sent, but do you know why this is happening?
 
I took out pip and using this link I added the RUN_DEPS line.
Keep in mind that might be an older copy of the Porter's handbook.

But yes, if those modules are required for your code to run, then add them to RUN_DEPENDS. Split them up in lines, that makes it easier to read and follow. Python dependencies are a little more involved though because they can be built with different versions of Python: https://docs.freebsd.org/en/books/porters-handbook/flavors/#flavors-auto-python

Code:
RUN_DEPENDS= ${PYTHON_PKGNAMEPREFIX}urllib3>0:net/py-urllib3@${PY_FLAVOR} \
             ${PYTHON_PKGNAMEPREFIX}pyminizip:archivers/py-pyminizip@${PY_FLAVOR} \
             ${PYTHON_PKGNAMEPREFIX}sqlite3>0:databases/py-sqlite3@${PY_FLAVOR}
Note that you can provide some version restrictions here. If you need at least version 1.0 or 3.5.6 from something you can indicate that here too. The port will fail to build if it cannot satisfy those requirements (if those dependencies are updated for example).

What I usually do is look up the dependency port on Freshports, databases/py-sqlite3 for example. Scroll a little further down the page until you see This port is required by: followed by a bunch of links to other ports that have it as a build or run dependency. Then open one of those port's Makefile and see how it's included there.

Code:
clean: install                                                                                                                       
        ${RM} -rf work
Remove this entire target. A make clean already cleans up the whole work directory by default. Those 'work' directories could be in a WRKDIRPREFIX directory:
Code:
       clean		Remove the expanded source code.  This recurses	to de-
			pendencies unless NOCLEANDEPENDS is defined.
Code:
     WRKDIRPREFIX  Where to create any temporary files.  Useful if PORTSDIR is
                   read-only (perhaps mounted from a CD-ROM).
ports(7)
 
Okay, I now have this in my Makefile:

Code:
PORTNAME=       theZoo                                                                                                 
PORTVERSION=    0.60                                                                                                   
CATEGORIES=     security                                                                                               
MASTER_SITES=   https://teamsloth.net/                                                                                 
                                                                                                                       
MAINTAINER=     notmyemail@gmail.com                                                                                   
COMMENT=        Malware Research                                                                                       
WWW=            https://thezoo.morirt.com                                                                              
                                                                                                                       
LICENSE=        GPLv3                                                                                                  
                                                                                                                       
RUN_DEPENDS= ${PYTHON_PKGNAMEPREFIX}urllib3>0:net/py-urllib3@${PY_FLAVOR}\                                             
             ${PYTHON_PKGNAMEPREFIX}pyminizip:archivers/py-pyminizip@${PY_FLAVOR}\                                     
             ${PYTHON_PKGNAMEPREFIX}sqlite3>0:databases/py-sqlite3@${PY_FLAVOR}                                        
                                                                                                                       
USES=           python:-3.9                                                                                            
                                                                                                                       
do-install: all                                                                                                        
                                                                                                                       
        ${INSTALL_PROGRAM} ${PORTSDIR}/security/theZoo/work/theZoo-0.60/theZoo.py ${LOCALBASE}/bin/theZoo              
        ${MKDIR} ${LOCALBASE}/etc/theZoo/                                                                              
        ${CP} -a ${PORTSDIR}/security/theZoo/work/theZoo-0.60/imports ${LOCALBASE}/lib/python3.9/site-packages/        
        ${CP} -a ${PORTSDIR}/security/theZoo/work/theZoo-0.60/conf ${LOCALBASE}/etc/theZoo/                            
        ${CP} -a ${PORTSDIR}/security/theZoo/work/theZoo-0.60/malware ${LOCALBASE}/etc/theZoo/                         
        ${CP} ${PORTSDIR}/security/theZoo/work/theZoo-0.60/*.md ${LOCALBASE}/etc/theZoo                                
                                                                                                                       
clean:                                                                                                                 
                                                                                                                    
.include <bsd.port.mk>

Surprisingly this comes out perfect with portlint:


Code:
# portlint                                                                                                             
looks fine.

I still get this problem of infinite installation folders:

Code:
# make install clean                                                                                                   
===>  Building for theZoo-0.60                                                                                         
--- show-dev-warnings ---                                                                                              
/!\ theZoo-0.60: Makefile warnings, please consider fixing /!\                                                         
Not validating first entry in CATEGORIES due to being outside of PORTSDIR.                                             
Please ensure this is proper when committing.                                                                          
--- check-vulnerable ---                                                                                               
--- check-license ---                                                                                                  
===>  License GPLv3 accepted by the user                                                                               
--- pkg-depends ---                                                                                                    
===>   theZoo-0.60 depends on file: /usr/local/sbin/pkg - found                                                        
--- fetch-depends ---                                                                                                  
===>   theZoo-0.60 depends on executable: curl - found                                                                 
--- do-fetch ---                                                                                                       
--- fetch-specials ---                                                                                                 
===> Fetching all distfiles required by theZoo-0.60 for building                                                       
--- extract-message ---                                                                                                
===>  Extracting for theZoo-0.60                                                                                       
--- checksum ---                                                                                                       
=> No checksum file (/usr/ports/security/theZoo/work/theZoo-0.60/distinfo).                                            
*** [checksum] Error code 1                                                                                            
                                                                                                                       
make[1]: stopped in /usr/ports/security/theZoo/work/theZoo-0.60                                                        
1 error                                                                                                                
                                                                                                                       
make[1]: stopped in /usr/ports/security/theZoo/work/theZoo-0.60                                                        
===> Compilation failed unexpectedly.                                                                                  
Try to set MAKE_JOBS_UNSAFE=yes and rebuild before reporting the failure to                                            
the maintainer.                                                                                                        
*** Error code 1

I have a pkg-plist and pkg-desc files. I just didn't want to flood these posts with too much information. Interestingly enough, I noticed both a work and work-py39 folders now:

Code:
# ls -1                                                                                                                
Makefile                                                                                                               
distinfo                                                                                                               
pkg-descr                                                                                                              
pkg-plist                                                                                                              
work                                                                                                                   
work-py39

This and the portlint output are the only differences. I wish I could figure out why this is still happening.
 
Last edited by a moderator:
Code:
===> Extracting for theZoo-0.60
--- checksum --- 
=> No checksum file (/usr/ports/security/theZoo/work/theZoo-0.60/distinfo).
*** [checksum] Error code 1
Your distinfo is missing or not correct. The file contains a filename and a hash of the downloaded distfile, it's used to verify the port downloaded the correct file. You don't need to create it by hand, there's a simple command that will download the file, calculate the hash and create a proper distinfo file; make makesum
 
Your distinfo is missing or not correct. The file contains a filename and a hash of the downloaded distfile, it's used to verify the port downloaded the correct file. You don't need to create it by hand, there's a simple command that will download the file, calculate the hash and create a proper distinfo file; make makesum
Hey, I know its been a few days, but let me refresh the problem. There is already a distinfo file in /usr/ports/security/theZoo:

$ cat /usr/ports/security/theZoo/distinfo
TIMESTAMP = 1697549012
SHA256 (theZoo-0.60.tar.gz) = 6d8022e79d7f750ce112bbbb1734c8e82e7a7d6821792d619c507c508249b47d
SIZE (theZoo-0.60.tar.gz) = 919235180


The problem is if you copy /usr/ports/security/theZoo/distinfo to /usr/ports/security/theZoo/work/theZoo-0.60/distinfo like it wants, it will create /usr/ports/security/theZoo/work/theZoo-0.60/work with the source code and then complains there is no /usr/ports/security/theZoo/work/theZoo-0.60/work/distinfo and so on creating an infinitie loop of source code folders. I may not have explained this well before, but that's the real issue that's driving me bonkers.

Thanks
 
Remove that all in your do-install: target, I think that's what causing it to loop. The whole do-install: isn't good actually. You 'install' into a staging area, not directly in the system itself. Never hardcode paths, especially from 'work' area.

To "install" the theZoo executable you use something like this:
Code:
do-install:
                ${INSTALL_PROGRAM} ${WRKSRC}/theZoo.py ${STAGEDIR}${PREFIX}/bin/theZoo

There are some special considerations for Python modules, definitely go through the PythonPortsPolicy patmaddox mentioned.
 
A lot of Python-based ports building has been automated. Look at PythonPortsPolicy for info, and the many Python-based ports as examples.

You can probably replace all of your custom targets with the following, or something close to it:

Code:
USES=        python
USE_PYTHON=    autoplist concurrent distutils
I already have this in my code:

RUN_DEPENDS= ${PYTHON_PKGNAMEPREFIX}urllib3>0:net/py-urllib3@${PY_FLAVOR}\
${PYTHON_PKGNAMEPREFIX}pyminizip:archivers/py-pyminizip@${PY_FLAVOR}\
${PYTHON_PKGNAMEPREFIX}sqlite3>0:databases/py-sqlite3@${PY_FLAVOR}

USES= python:-3.9


Maybe you missed that. I've looked at other ports in python. I haven't found the answer to this issue there.
 
Remove that all in your do-install: target, I think that's what causing it to loop. The whole do-install: isn't good actually. You 'install' into a staging area, not directly in the system itself. Never hardcode paths, especially from 'work' area.

To "install" the theZoo executable you use something like this:
Code:
do-install:
                ${INSTALL_PROGRAM} ${WRKSRC}/theZoo.py ${STAGEDIR}${PREFIX}/bin/theZoo

There are some special considerations for Python modules, definitely go through the PythonPortsPolicy patmaddox mentioned.
If I just remove the "all" from that line, nothing changes and it still tries to do that infinite folder thing. When I changed it to "install" instead of "do-install" I got this:

# make install clean
install -s -m 555 /usr/ports/security/theZoo/work/theZoo-0.60/theZoo.py /usr/local/bin/theZoo
strip: /usr/ports/security/theZoo/work/theZoo-0.60/theZoo.py: file format not recognized
strip: '/usr/local/bin/theZoo': No such file
install: strip command strip failed on /usr/local/bin/theZoo
*** Error code 70

Stop.
make: stopped in /usr/ports/security/theZoo


It seems to think this is a binary now instead of a python script. Also...

# portlint
FATAL: Makefile: direct redefinition of make target "install" discouraged. redefine "do-install" instead.
1 fatal error and 0 warnings found.
 
A lot of Python-based ports building has been automated. Look at PythonPortsPolicy for info, and the many Python-based ports as examples.

You can probably replace all of your custom targets with the following, or something close to it:

Code:
USES=        python
USE_PYTHON=    autoplist concurrent distutils
Actually I just tried adding this to my Makefile:
USES= python
USE_PYTHON= autoplist concurrent distutils

Now I get this:
# make install clean
===> License GPLv3 accepted by the user
===> theZoo-0.60 depends on file: /usr/local/sbin/pkg - found
===> Fetching all distfiles required by theZoo-0.60 for building
===> Extracting for theZoo-0.60
=> SHA256 Checksum OK for theZoo-0.60.tar.gz.
===> Patching for theZoo-0.60
===> theZoo-0.60 depends on package: py39-setuptools>=63.1.0 - found
===> theZoo-0.60 depends on file: /usr/local/bin/python3.9 - found
===> Configuring for theZoo-0.60
Traceback (most recent call last):
File "", line 1, in
FileNotFoundError: [Errno 2] No such file or directory: 'setup.py'
*** Error code 1

Not sure if I'm on the right track. I can make a setup.py for this, but I don't see one in the other python projects you linked.
 
Well, in case anyone is interested, the setup.py does seem more the way to go. My Makefile is no longer creating an infinite loops of folders and I'm getting a lot further with the build now:


# make
===> theZoo-0.60 depends on package: py39-setuptools>=63.1.0 - found
===> theZoo-0.60 depends on file: /usr/local/bin/python3.9 - found
===> Configuring for theZoo-0.60
running config
===> Building for theZoo-0.60
running build
running build_py
running build_scripts
===> Staging for theZoo-0.60
===> theZoo-0.60 depends on package: py39-urllib3>0 - not found
===> License MIT accepted by the user
===> py39-urllib3-1.26.17,1 depends on file: /usr/local/sbin/pkg - found
=> urllib3-1.26.17.tar.gz doesn't seem to exist in /usr/ports/distfiles/.
=> Attempting to fetch https://files.pythonhosted.org/packages/source/u/urllib3/urllib3-1.26.17.tar.gz
urllib3-1.26.17.tar.gz 297 kB 8263 kBps 00s
===> Fetching all distfiles required by py39-urllib3-1.26.17,1 for building
===> Extracting for py39-urllib3-1.26.17,1
=> SHA256 Checksum OK for urllib3-1.26.17.tar.gz.
===> Patching for py39-urllib3-1.26.17,1
===> Applying FreeBSD patches for py39-urllib3-1.26.17,1 from /usr/ports/net/py-urllib3/files
===> py39-urllib3-1.26.17,1 depends on package: py39-setuptools>=63.1.0 - found
===> py39-urllib3-1.26.17,1 depends on file: /usr/local/bin/python3.9 - found
===> Configuring for py39-urllib3-1.26.17,1
running config
===> Building for py39-urllib3-1.26.17,1
.....[OUTPUT SHORTENED FOR SPACE ISSUES]
writing list of installed files to '/usr/ports/archivers/py-pyminizip/work-py39/.PLIST.pymodtmp'
====> Compressing man pages (compress-man)
====> Running Q/A tests (stage-qa)
Warning: 'lib/python3.9/site-packages/pyminizip.cpython-39.so' is not stripped consider trying INSTALL_TARGET=install-strip
or using ${STRIP_CMD}
Error: '/usr/local/lib/python3.9/site-packages/pyminizip.cpython-39.so' is a architecture specific binary file and you have
set NO_ARCH. Either remove NO_ARCH or add 'pyminizip.cpython-39.so' to NO_ARCH_IGNORE.
*** Error code 1

Stop.
make[1]: stopped in /usr/ports/archivers/py-pyminizip
*** Error code 1

Stop.
make: stopped in /usr/ports/security/theZoo


I never had NO_ARCH in my Makefile, but I tried adding 'pyminizip.cpython-39.so' to NO_ARCH_IGNORE like this:

PORTNAME= theZoo
PORTVERSION= 0.60
CATEGORIES= security
MASTER_SITES= https://teamsloth.net/

MAINTAINER= notmyemail@gmail.com
COMMENT= Malware Research
WWW= https://thezoo.morirt.com

LICENSE= GPLv3

RUN_DEPENDS= ${PYTHON_PKGNAMEPREFIX}urllib3>0:net/py-urllib3@${PY_FLAVOR}\
${PYTHON_PKGNAMEPREFIX}pyminizip:archivers/py-pyminizip@${PY_FLAVOR}\
${PYTHON_PKGNAMEPREFIX}sqlite3>0:databases/py-sqlite3@${PY_FLAVOR}

USES= python
USE_PYTHON= autoplist concurrent distutils

NO_ARCH_IGNORE= pyminizip.cpython-39.so

do-install:

${INSTALL_PROGRAM} ${PORTSDIR}/security/theZoo/work/theZoo-0.60/theZoo.py ${LOCALBASE}/bin/theZoo
${MKDIR} ${LOCALBASE}/etc/theZoo/
#${CP} -a ${PORTSDIR}/security/theZoo/work/theZoo-0.60/imports ${LOCALBASE}/lib/python3.9/site-packages/
${CP} -a ${PORTSDIR}/security/theZoo/work/theZoo-0.60/conf ${LOCALBASE}/etc/theZoo/
${CP} -a ${PORTSDIR}/security/theZoo/work/theZoo-0.60/malware ${LOCALBASE}/etc/theZoo/
${CP} ${PORTSDIR}/security/theZoo/work/theZoo-0.60/*.md ${LOCALBASE}/etc/theZoo

clean:

.include <bsd.port.mk>


It still fails with the same error message. Here's my setup.py if it helps:


#!/usr/local/bin/python3.9
# -*- coding: utf-8 -*-"
from setuptools import setup, find_packages
import os
import shutil

base = '/usr/local/etc/theZoo/'

if not os.path.exists(base):
os.makedirs(base)

dirs = ['conf', 'imports', 'malware']

for y in dirs:
new_path = os.path.join(base, y)
if not os.path.exists(new_path):
os.makedirs(new_path)

target_dir = ''

for x in dirs:
source_dir = f'./{x}' # Assumes the directories are in the current working directory

if os.path.exists(source_dir):
for item in os.listdir(source_dir):
source_item = os.path.join(source_dir, item)

if os.path.isfile(source_item):
target_item = os.path.join(base, x, item)
shutil.copy(source_item, target_item)

elif os.path.isdir(source_item):
target_item = os.path.join(base, x, item)
shutil.copytree(source_item, target_item, dirs_exist_ok=True)
else:
print(f"Source directory '{source_dir}' does not exist.")

setup(
name="theZoo",
version="0.60",
packages=find_packages(),
scripts=['theZoo.py'],
test_suite="tests"
)


Anyone know about this?
 
Last edited:
Not sure if anyone is still reading, but I found the previous error was not with my port, but py-pyminizip. Even it wouldn't build from inside /usr/ports/archivers/py-pyminizip until I commented out this line in their Makefile:

NO_ARCH= yes

After that I could compile and install py-pyminizip and let them know about this on github. From there my code went a lot further, but stopped here:

[A lot of output removed for brevity]
no previously-included directories found matching 'zlib-1.2.11/projects'
no previously-included directories found matching 'zlib-1.2.11/qnx'
no previously-included directories found matching 'zlib-1.2.11/win32'
adding license file 'COPYING.txt'
writing manifest file 'pyminizip.egg-info/SOURCES.txt'
Copying pyminizip.egg-info to /usr/ports/archivers/py-pyminizip/work-py39/stage/usr/local/lib/python3.9/site-packages/pymini
zip-0.2.6-py3.9.egg-info
running install_scripts
writing list of installed files to '/usr/ports/archivers/py-pyminizip/work-py39/.PLIST.pymodtmp'
====> Compressing man pages (compress-man)
====> Running Q/A tests (stage-qa)
Warning: 'lib/python3.9/site-packages/pyminizip.cpython-39.so' is not stripped consider trying INSTALL_TARGET=install-strip
or using ${STRIP_CMD}
===> Installing for py39-pyminizip-0.2.6
===> Checking if py39-pyminizip is already installed
===> py39-pyminizip-0.2.6 is already installed
You may wish to ``make deinstall'' and install this port again
by ``make reinstall'' to upgrade it properly.
If you really wish to overwrite the old port of py39-pyminizip
without deleting it first, set the variable "FORCE_PKG_REGISTER"
in your environment or the "make install" command line.
*** Error code 1

Stop.
make[1]: stopped in /usr/ports/archivers/py-pyminizip
*** Error code 1


I found here that FORCE_PKG_REGISTER could be used, but is probably dangerous, or would make adoption of this code difficult. The other option mentioned here is to run make deinstall inside /usr/ports/archivers/py-pyminizip. If I do that I get this when I re-run my build:

# make install clean
===> Staging for theZoo-0.60
===> theZoo-0.60 depends on package: py39-urllib3>0 - found
===> theZoo-0.60 depends on executable: py39-pyminizip - not found
===> Installing for py39-pyminizip-0.2.6
===> Checking if py39-pyminizip is already installed
===> Registering installation for py39-pyminizip-0.2.6 as automatic
Installing py39-pyminizip-0.2.6...
===> theZoo-0.60 depends on executable: py39-pyminizip - not found
*** Error code 1

Stop.
make: stopped in /usr/ports/security/theZoo


So I'm forced to go back and forth between these two and never progress. Does anyone know a work around besides FORCE_PKG_REGISTER?

I have the same setup.py as in my last reply. This is my current Makefile now:


PORTNAME= theZoo
PORTVERSION= 0.60
CATEGORIES= security
MASTER_SITES= https://teamsloth.net/

MAINTAINER= notmyemail@gmail.com
COMMENT= Malware Research
WWW= https://thezoo.morirt.com

LICENSE= GPLv3

RUN_DEPENDS= ${PYTHON_PKGNAMEPREFIX}urllib3>0:net/py-urllib3@${PY_FLAVOR}\
${PYTHON_PKGNAMEPREFIX}pyminizip:archivers/py-pyminizip@${PY_FLAVOR}\
${PYTHON_PKGNAMEPREFIX}sqlite3>0:databases/py-sqlite3@${PY_FLAVOR}

USES= python
USE_PYTHON= autoplist concurrent distutils

do-install:

${INSTALL_PROGRAM} ${PORTSDIR}/security/theZoo/work-${PY_FLAVOR}/theZoo-0.60/theZoo.py ${LOCALBASE}/bin/theZoo
${CP} ${PORTSDIR}/security/theZoo/work/theZoo-0.60/*.md ${LOCALBASE}/etc/theZoo

clean:
${RM} -rf work-${PY_FLAVOR}

.include <bsd.port.mk>
 
I encourage you to build with poudriere so you get a clean environment every time.

Also like I said, you shouldn’t need to define any targets (e.g. do-install or clean) because the Python ports framework automates that. Assuming you have a working setup.py, it should “just work.” Read the wiki page and look at existing ports for examples.
 
I encourage you to build with poudriere so you get a clean environment every time.

Also like I said, you shouldn’t need to define any targets (e.g. do-install or clean) because the Python ports framework automates that. Assuming you have a working setup.py, it should “just work.” Read the wiki page and look at existing ports for examples.

Using poudriere is new to me. I found a link on it here. It seems that there's a lot one could do with it. From cloning the ports tree to operating on ports in a jail. I hate to ask, but is there something specific I should do with it? My setup.py file works when ran from inside /usr/ports/security/theZoo/work-39/theZoo-0.60. I wondered about the do-install and clean in my Makefile. Thanks for clearing that up. Lastly, I've been all over the wiki/handbook lately. I'm sure its there, but I haven't found anything to describe fixing this. Maybe my google-fu is just getting bad. Do you know anything I should look for to refine my search?
 
The point of using poudriere is so that you build with a clean slate, exactly the same way every time. It’s pretty easy to get inconsistent builds when you’re messing around in the ports tree and work dir.

That said, poudriere is an extra layer of complexity (that saves a bunch of time once you set it up) that you can probably get away without for now.

As far as searching… I linked to the Python ports page, as well as to a search query that returns commits for Python ports. There are literally hundreds (maybe thousands) of Python ports examples.
 
The point of using poudriere is so that you build with a clean slate, exactly the same way every time. It’s pretty easy to get inconsistent builds when you’re messing around in the ports tree and work dir.

That said, poudriere is an extra layer of complexity (that saves a bunch of time once you set it up) that you can probably get away without for now.

As far as searching… I linked to the Python ports page, as well as to a search query that returns commits for Python ports. There are literally hundreds (maybe thousands) of Python ports examples.

I wasn't sure how to proceed with poudriere, but what you said about messing around with the ports tree too much made sense. I decided to make a new vm of FreeBSD 14 so the ports tree would be clean. I put my Makefile, distinfo, pkg-plist and pkg-descr at https://teamsloth.net/port.tar.gz. This way I could pull them on to vms faster and maybe somebody can pull all my files if they want to see exactly what I'm doing here. I was surprised to find that I didn't have to edit the Makefile for pyminizip. There was no error about NO_ARCH this time. However, the build still failed with
pyminizip. These were some of the last lines before the error:

no previously-included directories found matching 'zlib-1.2.11/as400'
no previously-included directories found matching 'zlib-1.2.11/examples'
no previously-included directories found matching 'zlib-1.2.11/msdos'
no previously-included directories found matching 'zlib-1.2.11/old'
no previously-included directories found matching 'zlib-1.2.11/projects'
no previously-included directories found matching 'zlib-1.2.11/qnx'
no previously-included directories found matching 'zlib-1.2.11/win32'
adding license file 'COPYING.txt'
writing manifest file 'pyminizip.egg-info/SOURCES.txt'
Copying pyminizip.egg-info to /usr/ports/archivers/py-pyminizip/work-py39/stage/usr/local/lib/python3.9/site-packages/pyminizip-0.2.6-py
3.9.egg-info
running install_scripts
writing list of installed files to '/usr/ports/archivers/py-pyminizip/work-py39/.PLIST.pymodtmp'
====> Compressing man pages (compress-man)
===> Installing for py39-pyminizip-0.2.6
===> Checking if py39-pyminizip is already installed
===> Registering installation for py39-pyminizip-0.2.6 as automatic
Installing py39-pyminizip-0.2.6...
===> theZoo-0.60 depends on executable: py39-pyminizip - not found
*** Error code 1


If I try to build from /usr/ports/archivers/py-pyminizip it succeeds, but when I try to build my port in /usr/ports/security/theZoo it complains again that pyminizip should be reinstalled or deinstalled, thus creating this cycle of an endless back and forth again. I'm not sure if this is related, but I tried 'port test' from inside the directory and a lot of output noticed this error:

running install_scripts
copying build/scripts-3.9/theZoo.py -> /usr/ports/security/theZoo/work-py39/stage/tmp/theZoo-0.60/bin
changing mode of /usr/ports/security/theZoo/work-py39/stage/tmp/theZoo-0.60/bin/theZoo.py to 755
writing list of installed files to '/usr/ports/security/theZoo/work-py39/.PLIST.pymodtmp'
===> Creating unique files: Move MAN files needing SUFFIX
===> Creating unique files: Move files needing SUFFIX
Move: bin/theZoo.py --> bin/theZoo.py-3.9
Link: @bin/theZoo.py --> bin/theZoo.py-3.9
Makefile error: UNIQUE (suffix): bin/theZoo not found
*** Error code 1

Stop.
make: stopped in /usr/ports/security/theZoo
===> Error running make stage
===> Cleaning up


I'm guessing its complaining that /usr/local/bin/theZoo was not found. I would assume that's due to the build failing before that file gets copied, but since I'm unsure, I thought I'd post it here. If this isn't the issue, I find it interesting this failed with the same module on both FreeBSD 13 and a new install of 14. Any suggestions appreciated.
 
Typically ports makefile loops are caused by setting options where port A depends on port B which depends on port C which depends on port D which depends on port A. People usually blame the ports infrastructure where in fact they've shot themselves in the foot.
 
Hello Israel and PatMaddox, I am quite interested seeing the steps you follow, Israel, to reach a resolution to this packaging of python problem. Would be nice to have a document with words Abiword document, screen shots xfce-screenshooter, simple diagrams Dia Diagram Editor, URLs to click on in a PDF file output the user reads, the Kitchen Sink too. Might as well throw that Kitchen Sink inside the package for good measure. Someone might want to wash off all these pieces of dirt, smudged on their fingers by handling the messy creation of a large package. All Kidding aside. To understand all these relationships among Poudriere, Python, Ports, Packaging, Shell Scripts, configuration files that need to be set a certain way to engage a method that operates correctly to produce a final package theZoo-0.60.pkg (tar xzipped txz package file named theZoo-0.60.pkg). Yes that is a great deal of writing, that would explain all the nuances involved. Maybe too much for one person too create.

For my own needs in learning packaging, I just wanted to create a simple package consisting of 4 files: a python file User_Report.py , shell script file User_Report.sh , generate_user_report.desktop, screen_shooter_app.icon . I did not need a Makefile, Nothing to make. Just wanted the basic steps to create a package file generate_user_report-1.0_2.pkg

I have the tar file of these 4 files to be untarred to /tmp/stage to be shared with users to play with on their own computer.

I offer this PDF file here. Looking for Improvement and correction suggestions.
Contact me at https://github.com/wb7odyfred or email fredfinster58@gmail.com

I am interested to write up a document, Isreal, that details in words and pictures and file, your much greater knowledge with your Python packaging problem difficulties. Respectfully Fred Finster
 

Attachments

  • Dead_Simple_Binary_Packaging_How_To_10_6C.pdf
    467.1 KB · Views: 96
  • remove_pdf_suffix_generate_user_report-1.0_2.pkg.pdf
    373.9 KB · Views: 46
Typically ports makefile loops are caused by setting options where port A depends on port B which depends on port C which depends on port D which depends on port A. People usually blame the ports infrastructure where in fact they've shot themselves in the foot.

Maybe I worded that poorly. I don't know if the NO_ARCH from before was due to me messing with the ports system too much and mucking things up. I just thought it was worth mentioning since I saw this come up again with a clean install. It may have nothing to do with that module, but lead to a clue in the failures of my own code. If that came across wrong, my apologies.
 
Hello Israel and PatMaddox, I am quite interested seeing the steps you follow, Israel, to reach a resolution to this packaging of python problem. Would be nice to have a document with words Abiword document, screen shots xfce-screenshooter, simple diagrams Dia Diagram Editor, URLs to click on in a PDF file output the user reads, the Kitchen Sink too. Might as well throw that Kitchen Sink inside the package for good measure. Someone might want to wash off all these pieces of dirt, smudged on their fingers by handling the messy creation of a large package. All Kidding aside. To understand all these relationships among Poudriere, Python, Ports, Packaging, Shell Scripts, configuration files that need to be set a certain way to engage a method that operates correctly to produce a final package theZoo-0.60.pkg (tar xzipped txz package file named theZoo-0.60.pkg). Yes that is a great deal of writing, that would explain all the nuances involved. Maybe too much for one person too create.

For my own needs in learning packaging, I just wanted to create a simple package consisting of 4 files: a python file User_Report.py , shell script file User_Report.sh , generate_user_report.desktop, screen_shooter_app.icon . I did not need a Makefile, Nothing to make. Just wanted the basic steps to create a package file generate_user_report-1.0_2.pkg

I have the tar file of these 4 files to be untarred to /tmp/stage to be shared with users to play with on their own computer.

I offer this PDF file here. Looking for Improvement and correction suggestions.
Contact me at https://github.com/wb7odyfred or email fredfinster58@gmail.com

I am interested to write up a document, Isreal, that details in words and pictures and file, your much greater knowledge with your Python packaging problem difficulties. Respectfully Fred Finster

These pdfs aren't bad. Right now I'm trying to keep it simple with just a command-line port. I do see some stuff I can learn from them though. If I ever figure out what I'm doing wrong here I'll be sure to reach out to you. Thanks
 
When you have theZoo package, tested, installed on fresh system and working. send me the completed product, with a tar gzip tar file So that I might explore all the pieces and files necessary to build this package or port. Well your is definitely a port, Israel.

I ask that you open a text file Thoughts-as-I-port_theZOO.txt And just post your intermediate thoughts as you go through this building a theZOO port creation process. Note in there your confusion thoughts too. as well as the steps that do work. I am interested in the thought process much more than a complete , written up, finished product.
I like that in the above posts , you wrote about your confusion on "where" the process started the circular loop to create an infinite loop.

Fred
 
This should help move you along...

Makefile:

Makefile:
PORTNAME=       theZoo
PORTVERSION=    0.6.0.20231026
CATEGORIES=     security python
PKGNAMEPREFIX=  ${PYTHON_PKGNAMEPREFIX}

USE_GITHUB=     yes
GH_ACCOUNT=     ytisf
GH_TAGNAME=     398d83938838ae6f92117d9dbcd07833dd1dcaaf

USES=           python
USE_PYTHON=     autoplist concurrent distutils

RUN_DEPENDS=    ${PYTHON_PKGNAMEPREFIX}urllib3>0:net/py-urllib3@${PY_FLAVOR} \
                ${PYTHON_PKGNAMEPREFIX}pyminizip>0:archivers/py-pyminizip@${PY_FLAVOR}

post-extract:
        cp ${PATCHDIR}/setup.py ${WRKSRC}/

.include <bsd.port.mk>

Run make makesum to fetch the distfile and make the distinfo file.

At this point, make will fail because there is no setup.py. You can either write a do-install to copy the files, or write setup.py. I opted to write setup.py, as you can see referenced in the post-extract target. It will need to live under files/, and look something like:

Python:
from setuptools import setup, find_packages

setup(
    name='theZoo',
    version='0.6.0',
    packages=['imports'],
    install_requires=[
        'urllib3',
        'pyminizip'
    ],
    package_data={'': ['theZoo.py'], 'imports': ['*.py']}
)

I don't work with Python, so I don't really know... but that seems to have been a start.

In the end, you should have the following files:
  • security/py-theZoo/Makefile
  • security/py-theZoo/distinfo
  • security/py-theZoo/pkg-descr
  • security/py-theZoo/files/setup.py
With a working setup.py, you should be good to go. Then you should consider submitting setup.py as an upstream patch so you don't have to include it in the port.
 
Back
Top