Fetchmail configuration for IMAP directory

[Deleted member 9563]

I'm trying to download a single directory from an IMAP server, but am having no luck. All I want to do is download the sent directory on the server to a specific directory on my computer. Here is my .fetchmailrc

poll mail.server.com protocol imap
username "not" password "xxxxxxxxxx"
 folder sent -k -a

Since fetchmail is apparently hooked into local mail I've also put

not: /home/ole/MAIL/not/sent

in the /etc/mail/aliases file. However, it appears that it's the fetchmail connection to the server which is failing since the output ends with this:

fetchmail: Query status=3 (AUTHFAIL)

Any suggestions for what I'm doing wrong? Am I using the wrong tool? Can I use telnet for specific IMAP directories?

 

[trh411]

Did you recreate the /etc/mail/aliases.db after modifying /etc/mail/aliases? You might also check out the fetchmail web site. They have lots of info on using fetchmail with other MTAs, etc.

 

[Deleted member 9563]

Thanks @trh411. I didn't know that was required. I've read as much as I can absorb about fetchmail and basically realized that I was mistaken and that it was not really intended for what I want, although I've managed to download mail with it some years back and was hoping to get a command line e-mail situation happening. Anyway, I'm giving up on fetchmail for now since I've spend most of a night on trying it.

However, perhaps I quit too soon since I've just spent over 12 hours trying all the possible permutations of Sylpheed on a number of accounts on several servers and can report that it definitely does not do IMAP on this machine. (Deleting .sylpheed-2.0/ and reinstalling in between.) POP3 works perfectly though. Perhaps it's the servers that I'm accessing, since a relatively similar machine in this house uses IMAP for one of its accounts.

I'm guessing that IMAP requires something which is broken or missing on this install.

FreeBSD SCO.lan.cgs 9.2-RELEASE FreeBSD 9.2-RELEASE #0 r255898: Thu Sep 26 22:50:31 UTC 2013     root@bake.isc.freebsd.org:/usr/obj/usr/src/sys/GENERIC  amd64

Is there a utility that will download individual IMAP directories and also will run on FreeBSD? I can not find any in the ports.

 

[trh411]

You should be able to use fetchmail for what you want to do.

In retrospect, having glossed over it in your original post, I would have liked to see more of the fetchmail log that contained the

fetchmail: Query status=3 (AUTHFAIL)

error. That message suggests nothing more than a simple authentication issue rather than a configuration issue. More of the log file could have confirmed that. Sounds like it's too late now.

Sorry, but I know nothing about mail/sylpheed.

 

[Deleted member 9563]

You should be able to use fetchmail for what you want to do.

In retrospect, having glossed over it in your original post, I would have liked to see more of the fetchmail log that contained the

fetchmail: Query status=3 (AUTHFAIL)

error. That message suggests nothing more than a simple authentication issue rather than a configuration issue. More of the log file could have confirmed that. Sounds like it's too late now.

Sorry, but I know nothing about mail/sylpheed.

Thanks for your consideration. It may not be too late. This is causing me quit a bit of aggravation since I need to deal with some years of mail, and fairly soon. I have little experience with mail programs other than scripting my own system in DOS - however, this needs to be done on my FreeBSD system.

Sylpheed seems to be fine and runs on KDE, but since I cannot get IMAP to work on it either (consistent authentication failure), I think that going back to trying fetchmail may be educational and have long term practically as well. I can then use Sylpheed as an interface later if I like. For now, let me just get the files. So ....

Running fetchmail results in the following:

Enter password for not@mail.netopit.com: 
fetchmail: Server certificate verification error: unable to get local issuer certificate
fetchmail: Broken certification chain at: /C=IL/O=StartCom Ltd./OU=Secure Digital Certificate Signing/CN=StartCom Certification Authority
fetchmail: This could mean that the server did not provide the intermediate CA's certificate(s), which is nothing fetchmail could do anything about.  For details, please see the README.SSL-SERVER document that ships with fetchmail.
fetchmail: This could mean that the root CA's signing certificate is not in the trusted CA certificate location, or that c_rehash needs to be run on the certificate directory. For details, please see the documentation of --sslcertpath and --sslcertfile in the manual page.
fetchmail: Server certificate verification error: certificate not trusted
fetchmail: Warning: the connection is insecure, continuing anyways. (Better use --sslcertck!)
fetchmail: Authorization failure on not@mail.superb.net
fetchmail: For help, see http://www.fetchmail.info/fetchmail-FAQ.html#R15
fetchmail: Query status=3 (AUTHFAIL)

PS: I'm just making test accounts to try for now.

 

[trh411]

Here is what the fetchmail site says for this error: For help see http://www.fetchmail.info/fetchmail-FAQ.html#R15

R15. Help, I'm getting Authorization failure!

First, try upgrading to fetchmail 6.3.18 or newer. Release 6.3.18 has received a considerable number of bug fixes for the authentication feature (AUTH, AUTHENTICATE, SASL). Most notably, fetchmail aborts SASL authentication attempts properly with an asterisk if it detects that it cannot make progress with a particular authentication scheme. This fixes issues where GSSAPI-enabled fetchmail cannot authenticate against Microsoft Exchange 2007 and 2010. Note that this is a bug in old fetchmail versions!

Fetchmail by default attempts to authenticate using various schemes. Fetchmail tries these schemes in order of descending security, meaning the most secure schemes are tried first.

However, sometimes the server offers a secure authentication scheme that is not properly configured, or an authentication scheme such as GSSAPI that requires credentials to be acquired externally. In some situations, fetchmail cannot know that the scheme will fail beforehand, without trying it. In most cases, fetchmail should proceed to the next authentication scheme automatically, but this sometimes does not work.

Solution: Configure the right authentication scheme explicitly, for instance, with --auth cram-md5 or --auth password on the command line or auth "cram-md5" or auth "password" in the rcfile. Details can be found in the manual page.
Note that auth password should only be used across secure links (see the sslcertck and ssl/sslproto options).

Yikes! I'm going to have to read that several times for it to sink in I think.

 

[ljboiler]

I successfully use fetchmail to pull down mail from my ISP via IMAP (SSL connection).

poll mobile.charter.net via mobile.charter.net auth password
  with proto IMAP
  user 'xxxxxxx@charter.net' there with password 'yyyyyy' is 'jimmy' here 
  options keep ssl sslcertck sslcertfile /usr/local/share/certs/ca-root-nss.crt

This drops my mail into /var/mail/jimmy. I remember having authentication problems some time ago when a new version of fetchmail was released, and the solution turned out to be that

auth password

part; the default authorization protocol that fetchmail uses got changed to that GSSAPI thing and you have to explicitly tell it to use the simple "password" mode for remote servers that require it.

 

[kpa]

The error message is about the trust of the issuer of the SSL certificate presented by the IMAP mail server. I haven't used mail/fetchmail myself so I don't know things are configured there but there should be an option to tell it to use a specific file or directory for the SSL CA certificates. The CA certificate bundle is in port security/ca_root_nss and all the certificates in file /usr/local/share/certs/ca-root-nss.crt.

 

[Deleted member 9563]

As @ljboiler suggested, I now have

auth password

in my ~/.fetchmailrc. That's the way I prefer to see it, so I'll keep that, but the server is still balking.

The error message is about the trust of the issuer of the SSL certificate presented by the IMAP mail server. I haven't used mail/fetchmail myself so I don't know things are configured there but there should be an option to tell it to use a specific file or directory for the SSL CA certificates. The CA certificate bundle is in port security/ca_root_nss and all the certificates in file /usr/local/share/certs/ca-root-nss.crt.

Thanks @kpa, that really has helped me get some perspective. Also, my net search suggests that the problem indeed lies there. So far no solution though. The idea of running c_rehash sounds promising, but it doesn't do anything.

I have a feeling that there is some mis-configuration of openssl (I didn't do it!), or I need to tell fetchmail where some certificate or other resides. I confirmed that I have security/ca_root_nss installed.

In case it helps someone come up with a suggestion, I have eliminated the idea of aliases for now so as to simplify the situation. My ~/.fetchmailrc file now looks like this:

poll mail.netopit.com protocol imap
auth password
username "not"
folder sent -k -a

 

[ljboiler]

From the fetchmail output in your original post, you are having two issues:

• cannot make a secure SSL connection
• cannot log in to check for mail

The SSL connection failure does not stop fetchmail from trying to log in; it just switches to a non-encrypted connection:

fetchmail: Server certificate verification error: certificate not trusted
fetchmail: Warning: the connection is insecure, continuing anyways. (Better use --sslcertck!)

Where is the password for user "not" in your changed ~/.fetchmailrc? It is still needed:

poll mail.netopit.com protocol imap
auth password
username "not" password "xxxxxxxxxx"
folder sent -k -a

and since you must have a password in that file, you will want to protect it

chmod 600 ~/.fetchmailrc

For the SSL problem, first make sure you have the CA certificate bundle installed as pointed out by @kpa, and then you must tell fetchmail where to find it by adding the

ssl sslcertck sslcertfile...

part from my original post to your fetchmailrc. As long as the SSL certificate used by the remote mail server is signed by one of the standard authorities, this should give you an encrypted connection.

 

[Deleted member 9563]

Thanks @ljboiler. Password seems to work fine. Putting auth password in the file causes fetchmail to prompt the user for a password.

As for the SSL certificate the problem may be here:

ssl sslcertck sslcertfile...

I don't know what the syntax is. I've been trying various ones - also as a command line option to make changing easy. Other sites suggest this:

sslcertck  /usr/local/share/certs/ca-root-nss.crt

Which results in this line:

fetchmail:/home/ole/.fetchmailrc:3: syntax error at /usr/local/share/certs/ca-root-nss.crt

Without the path it gives this:

fetchmail:/home/ole/.fetchmailrc:3: syntax error at ca-root-nss.crt

 

[ljboiler]

The fetchmail() man page has all the syntax and several examples, but I use the following 3 options:

• ssl - to tell fetchmail I want an encrypted connection; without this it makes an un-encrypted connection
• sslcertck - to tell fetchmail to check that the mail server's SSL certificate is signed by a trusted authority
• sslcertfile <path to CA cert bundle> - the CA bundle with my trusted certificates (/usr/local/share/certs/ca-root-nss.crt)

 

[Deleted member 9563]

I've got the fetchmail man page and three other howtos open beside this window so I got that much. The reason I'm saying that I don't understand the syntax is that what you (and they) suggest is not working and gives the syntax error that I quote above. I guess I'm really missing something here.

First, the authorization failure. I commented out the lines with ssl and put

sslproto ''

in the .fetchmailrc file. I also put the password in, although it will ask for it if I don't. The result is:

/home/ole SCO%  fetchmail
fetchmail: Authorization failure on not@mail.superb.net
fetchmail: Query status=3 (AUTHFAIL)

With the

sslproto ''

removed and the ssl lines back in, my .fetchmailrc looks like this:

poll mail.netopit.com protocol imap
auth password
username "not" password "xxxxxxxx"
ssl
slcertfile /usr/local/share/certs/ca-root-nss.crt
sslcertck
folder sent -k -a

and the output is this:

/home/ole SCO%  fetchmail
fetchmail: Server CommonName mismatch: *.superb.net != mail.netopit.com
fetchmail: OpenSSL reported: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
fetchmail: SSL connection failed.
fetchmail: socket error while fetching from not@mail.netopit.com
fetchmail: Query status=2 (SOCKET)

I understand that the "CommonName mismatch" can be ignored. However the connection still failed.

 

[ljboiler]

I think the sslcertck option is causing the SSL connection failure here, since the common names do not match; the option tells fetchmail to "strictly check the server certificate". Try removing it.

I am confused with the authorization failure you got when you commented out the SSL options and put in

sslproto ''

(am I correct - a pair of single quotes with nothing in between?). That should result in a non-encrypted connection, and it appears that the connection was successful. An authorization error in that case can only mean that

• not@mail.superb.net is not a valid username to use to log in on that remote mail server to get messages via IMAP, or
• you have the wrong password (which I doubt)

 

[Deleted member 9563]

I think the sslcertck option is causing the SSL connection failure here, since the common names do not match; the option tells fetchmail to "strictly check the server certificate". Try removing it.

That results in this:

/home/ole SCO%  fetchmail
fetchmail: Warning: the connection is insecure, continuing anyways. (Better use --sslcertck!)
fetchmail: Authorization failure on not@mail.superb.net
fetchmail: Query status=3 (AUTHFAIL)

I am confused with the authorization failure you got when you commented out the ssl options and put in

sslproto ''

(am I correct - a pair of single quotes with nothing in between?). That should result in a non-encrypted connection, and it appears that the connection was successful.

Yes, single quotes with nothing in between.

An authorization error in that case can only mean that

• not@mail.superb.net is not a valid username to use to log in on that remote mail server to get messages via IMAP, or
• you have the wrong password (which I doubt)

Indeed "not@mail.superb.net" is not a valid username. How do I avoid it? I need to put in the address for the mail server, which is "mail.netopit.com", but the e-mail address of the account is "not@netopit.com". Surely this is a common situation for commercial e-mail servers. It is interesting to note that since I started this thread I have been able to connect to another server with the sylpheed program using IMAP and SSL, though not with fetchmail.

I'm going to put in a ticket with the provider and see if they can give me some insight. They respond very fast so should know something soon.

 

[ljboiler]

The server name that immediately follows the "poll" command in the .fetchmailrc needs to be the name of the server that fetchmail should connect to; the username following the "username" option needs to be whatever account name needed to log in on that server. So, I believe your .fetchmailrc should look like this:

poll mail.netopit.com protocol imap
auth password
username "not@netopit.com" there with password "xxxxxxxx" is "not" here
ssl
slcertfile /usr/local/share/certs/ca-root-nss.crt
folder sent keep all

This tells fetchmail to connect to mail.netopit.com, log in as not@netopit.com with password xxxxxxxxxx, and check the folder named sent for any new mail, which should then be given to the local mail delivery agent for the local user not (the is "not" here part). I have gotten syntax errors putting command-line option flags in .fetchmailrc, so I put in the full keep and all option names.

 

[Deleted member 9563]

The only thing the provider had to tell me for now is that it looks like I should have used "not@netopit.com" - as you're suggesting now. I tried that early on, but must have had some other mistakes.

Here is the output from the script you provided (adding the missing "s"):

fetchmail:/home/ole/.fetchmailrc:8: syntax error at all

Next I tried with only "folder sent" on that line:

fetchmail: Warning: the connection is insecure, continuing anyways. (Better use --sslcertck!)
fetchmail: mailbox selection failed
fetchmail: client/server synchronization error while fetching from not@netopit.com@mail.netopit.com
fetchmail: Query status=7 (ERROR)

Then I tried without the 3 SSL lines:

fetchmail: Server certificate verification error: unable to get local issuer certificate
fetchmail: Broken certification chain at: /C=IL/O=StartCom Ltd./OU=Secure Digital Certificate Signing/CN=StartCom Certification Authority
fetchmail: This could mean that the server did not provide the intermediate CA's certificate(s), which is nothing fetchmail could do anything about.  For details, please see the README.SSL-SERVER document that ships with fetchmail.
fetchmail: This could mean that the root CA's signing certificate is not in the trusted CA certificate location, or that c_rehash needs to be run on the certificate directory. For details, please see the documentation of --sslcertpath and --sslcertfile in the manual page.
fetchmail: Server certificate verification error: certificate not trusted
fetchmail: Warning: the connection is insecure, continuing anyways. (Better use --sslcertck!)
7 messages (4 seen) for not@netopit.com at mail.netopit.com.

It hangs there.
Again, but adding

sslproto ''"

7 messages (4 seen) for not@netopit.com at mail.netopit.com.

And it hangs there.

 

[Deleted member 9563]

Some headway! I found the real name for the "sent" directory by hoovering my pointer over the sent director in the web mail access. It is "INBOX.Sent". So, here is a revised .fetchmailrc file without SSL:

poll mail.netopit.com protocol imap
auth password
username "not@netopit.com" there with password "xxxxxxxx" is "not" here
sslproto ''
folder INBOX.Sent  keep

I realized that the -a or all meant all folders, so didn't make sense in the context of only wanting one folder. The result of this file is a success:

/home/ole SCO%  fetchmail
2 messages (2 seen) for not@netopit.com at mail.netopit.com (folder INBOX.Sent).

However, I don't know where the messages went. Did I get them? Since I used "keep", I can repeat the command and get the same result.

Adding the SSL lines back in, I get:

fetchmail: Warning: the connection is insecure, continuing anyways. (Better use --sslcertck!)
2 messages (2 seen) for not@netopit.com at mail.netopit.com (folder INBOX.Sent).

It looks like I'm getting close. So I then made one little change. I added "sslcertch" - like this:

poll mail.netopit.com protocol imap
auth password
username "not@netopit.com" there with password "xxxxxxxx" is "not" here
ssl
sslcertfile /usr/local/share/certs/ca-root-nss.crt
sslcertck
folder INBOX.Sent  keep

However, the result is this:

fetchmail: Server CommonName mismatch: *.superb.net != mail.netopit.com
fetchmail: OpenSSL reported: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
fetchmail: SSL connection failed.
fetchmail: socket error while fetching from not@netopit.com@mail.netopit.com
fetchmail: Query status=2 (SOCKET)

I read somewhere that the name mismatch is not a problem and is indeed common with this kind of server setup. Still, it didn't work. Fetchmail is not happy about something.

 

[ljboiler]

The sslcertck option tells fetchmail to abort the connection if the SSL certificate can't be verified; without that option, it will continue on with a insecure connection. Sorry, but I can't help you debug that further.

As far as to "where the messages went. Did I get them?": if you still have this in /etc/mail/aliases

not:   /home/ole/Mail/not/sent

Run this as root: newaliases. Then create the empty file for that alias like this as user ole:

mkdir -p /home/ole/Mail/not
touch /home/ole/Mail/not/sent
chmod u+s /home/old/Mail/not/sent

and the messages should end up in /home/ole/Mail/not/sent when you run fetchmail.

Oh, and if you read the fetchmail(1) man page, the all option does not mean "all folders", it means "all messages, even if they have been flagged as being seen". which is something you might have to do one time after you get the alias and local file set up.

 

[Deleted member 9563]

The sslcertck option tells fetchmail to abort the connection if the SSL certificate can't be verified; without that option, it will continue on with a insecure connection. Sorry, but I can't help you debug that further.

That's OK, you've done a great job with the fetchmail.

As far as to "where the messages went. Did I get them?": if you still have this in /etc/mail/aliases

not:   /home/ole/Mail/not/sent

Run this as root: newaliases. Then create the empty file for that alias like this as user ole:

mkdir -p /home/ole/Mail/not
touch /home/ole/Mail/not/sent
chmod u+s /home/old/Mail/not/sent

and the messages should end up in /home/ole/Mail/not/sent when you run fetchmail.

I had actually removed the original aliases entry so as to simply the situation and make it easier to diagnose. I had thought that mail would end up in /var/mail by default. In any case, I just did what you said above and the mail ended up in /home/ole/Mail/not/sent. So that's all good then.

Oh, and if you read the fetchmail(1) man page, the all option does not mean "all folders", it means "all messages, even if they have been flagged as being seen". which is something you might have to do one time after you get the alias and local file set up.

I finally understand. Thanks. One thing that was confusing is the there was an error with all, as well as -a, regardless of where I put it in the .fetchmailrc file.

fetchmail:/home/ole/.fetchmailrc:16: syntax error at all

Removing all cured that problem. There seems to be some discrepancy with the man page in that regard and I was thinking it was because the -all didn't make sense. As it turns out, putting the newer command fetchall in the .fetchmailrc file completely solved that problem.

In retrospect, I can see that the change in fetchmail's protocol with -all (or -a) has been a major confusion for me in figuring this out. I will read some of the information in the /usr/local/share/doc/fetchmail directory and see if I can figure out how to get fetchmail to use SSL. For now I will be able to do what I most need and will worry about security for future and continued use. Thanks for your help.