Perl Which Perl Modules and Plugins to use?

[max21]

I’m new to perl and I like it. I don’t want to over do it but I do want to be fully equipped for other common use. What are the most recommended modules and plugins that one would use for webpage manipulation? Right now, all I have is perl installed. I just recently learn how to use perl strings functions. How about if one want to do some serious page scraping and downloading of html source. There are thousands of modules; however, I selected these few because they sound like they may be needed someday once I get the hang of things. Are these OK or have I over done it? What is missing or should be replaced? Why would I want to use sockets when all I want to do mostly is page scraping and build friendly robots? Which one of these actually provides the LWP:: simple variable or include. Is there a way to install a whole bunch at one time thru ports?

www/p5-libwww
# …………………………………………….
www/p5-LWP-Authen-Negotiate
www/p5-LWP-Authen-OAuth
www/p5-LWP-Authen-Wsse
www/p5-LWP-ConnCache-MaxKeepAliveRequests
www/p5-LWP-MediaTypes
www/p5-LWP-Online
www/p5-LWP-Protocol-PSGI
www/p5-LWP-Protocol-connect
www/p5-LWP-Protocol-http10
www/p5-LWP-Protocol-https
www/p5-LWP-Protocol-socks
www/p5-LWP-UserAgent-Determined
www/p5-LWP-UserAgent-POE
www/p5-LWP-UserAgent-WithCache
www/p5-LWPx-ParanoidAgent
www/p5-LWPx-TimedHTTP
www/p5-Markup-Perl
# …………………………………………….
www/p5-WWW-Plurk
www/p5-WWW-Robot
www/p5-WWW-RobotRules
www/p5-WWW-RobotRules-Parser
# …………………………………………….
www/p5-WWW-Mixi-Scraper
www/p5-Web-Scraper
www/p5-Web-Scraper-Config
www/p5-Web-Simple
# …………………………………………….
www/p5-WWW-Search
www/p5-WWW-Search-AltaVista
www/p5-WWW-Search-Google
www/p5-WWW-Search-MSN
# …………………………………………….
devel/p5-IO-AIO
devel/p5-IO-All
devel/p5-IO-All-LWP
# …………………………………………….
net/p5-IO-Interface
net/p5-IO-Socket-INET6
net/p5-IO-Socket-IP
net/p5-IO-Socket-Multicast
net/p5-IO-Socket-Multicast6
net/p5-IO-Socket-Socks
net/p5-IO-Socket-Timeout
net/p5-IO-Socket-UNIX-Util
# …………………………………………….
www/p5-Any-Template
www/p5-Any-URI-Escape
www/p5-AnyEvent-HTTP
www/p5-AnyEvent-HTTP-LWP-UserAgent
www/p5-AnyEvent-HTTPD
www/p5-AnyEvent-ReverseHTTP
# …………………………………………….
www/p5-JSON-API

 

[SirDice]

What are the most recommended modules and plugin that one would use for webpage manipulation?

Start with www/p5-libwww. It contains most of the functionality you need. More than enough to get you started. You can always add specific modules later on when it's more clear what you need.

 

[max21]

Start with www/p5-libwww. It contains most of the functionality you need. More than enough to get you started. You can always add specific modules later on when it's more clear what you need.

There is one thing I need to be sure of and that is how to get LWP::simple. Do it comes with www/p5-libwww or from something else? The reason why, I installed lots of those things but only got a blank page with 0 return stating success (and that was from the most simpless examples out here). This time I'm doing a clean install to learn what works or not. It could have be my sloppy machine. I'm intalling from ports only now.

 

[ShelLuser]

Why not just take a look if you want to know more about a Port / package?

Anyway, I think it does:

peter@zefiris:/usr/ports/www/p5-libwww $ grep -i simple pkg-plist
%%SITE_PERL%%/LWP/Simple.pm
%%PERL5_MAN3%%/LWP::Simple.3.gz

 

[SirDice]

There is one thing I need to be sure of and that is how to get LWP::simple. Do it comes with www/p5-libwww or from something else?

Yes, it includes LWP::Simple.

The reason why, I installed lots of those things but only got a blank page with 0 return stating success (and that was from the most simpless examples out here).

Learn to use the builtin Perl debugger. It will allow you to single step through your code and watch variables. I often use it when my code does something wrong or unexpected.

 

[max21]

Why not just take a look if you want to know more about a Port / package?

ShelLuser, I did by way of Freshports. Things seems to have change such as what use to be in LWP or perl is now in a module. It was only one but I did read about it last week and forgot already. It’s not hard to get confuse. I think even a location of a group of files is difference than what is reported at Freshports.

Anyway, I always wanted to say this - - I see no further ado

Every example now works.

I’m back to ports! This is the order:

lang/perl5.26
www/p5-libwww
www/p5-LWP-Protocol-https * it's a must.

Thanks guys!

It’s in there.

 

[Chris_H]

If I were you. I'd simply look (do a search) on CPAN for things that look like they accomplish what you're interested in getting accomplished. Then simply perform the install. Like any normal Perl module installation. I should add. That you will want to search that they aren't already available in the ports tree first. Even if they aren't. Don't worry. The (FreeBSD) Perl/ports framework knows how to handle your CPAN version, and will incorporate your built, installed Perl module into your ports/package DB, like magic!
Good choice, BTW. I'm a big fan of Perl, myself.

HTH, and happy Perl programming, max21 !


--Chris
EDIT
OH forgot to mention. The beauty of Perl, and the ports framework, is that all your dependencies will automatically be handled.