Solved Local archive of IMAP server

Greetings all,

thanks to bleedwood's and Lamia's links to tutorial, I have set up the mail/mbsync to one of my accounts. Regarding configuration, the terms "Master" and "Slave" have been deprecated, but it was a trivial correction.

It appears that the folders are being synchronized, but form time to time a message:
Code: 
Maildir notice: no UIDVALIDITY, creating new.
appears in-between the cryptic:
Code: 
C: 0/1  B: 1/10  F: +0/0 *0/0 #0/0  N: +0/0 *1/1 #0/0.
There is not much information about this, but as best understood an UID is a persistent unique identifier across sessions when syncing mail, which is validated by the UIDVALIDITY. However, since this is a first session, the explanation makes no sense.

I also found some indication, that the UIDVALIDITY message may be caused by empty folders with no messages. This makes more sense to me, as there are no folders on the Near machine, they are being created upon download. However, I would have thought that the mail/mbsync should be aware of that.

Kindest regards,

M
 
mefizto said:
It appears that the folders are being synchronized
Click to expand...
This means, if you delete on the server, it will delete in local, but in local you want an archive, i.e. not to delete.
Also in the other direction will delete. You will have to experiment with the configuration. Note for example
the "Sync All". See the man page.

I pointed out to this problem before.
 
mefizto said:
Hi Machiaveli,

thank you for the suggestion. No, I have not tried it. I am rather ignorant about the e-mail workings. However, recently I had been made aware of need to preserve some mail, so I started inquiring.

Just to make sure, it is retrieval not synchronization application, correct?

Kindest regards,

M
Click to expand...
Hey mefizto
It's just a mail fetcher and does not do any synchronization. It can retrieve all of emails or just the new ones. Option to keep retrieved mails or not, etc.

https://www.fetchmail.info/

It's an old tool, well tested and does the work no matter what mail protocol you're using (be it POP, IMAP, etc.).
Let me know if you need directions on how to use it.
 
Machiaveli said:
It's just a mail fetcher and does not do any synchronization.
Click to expand...
The problem he may have, is, that if he reads directly from the server and hence marks the mail as
read, fetchmail will not download it anymore.

The issue must be investigated more carefully.

There is no way to consciously using a computer, to being aware of what one is doing.

In that sense, reading with a normal imap client and downloading and saving locally what one
wants is a more secure way. But it means that one archive only few, important things.
 
hruodr said:
The problem he may have, is, that if he reads directly from the server and hence marks the mail as
read, fetchmail will not download it anymore.

The issue must be investigated more carefully.

There is no way to consciously using a computer, to being aware of what one is doing.

In that sense, reading with a normal imap client and downloading and saving locally what one
wants is a more secure way. But it means that one archive only few, important things.
Click to expand...
Indeed, if such a case doing a fetch all is the only option but it may be network intensive as it download again all of the emails.

getmail can be used there instead of fetchmail as it check and download emails based on the server's message id rather than on the read/unread flag.
 
Hi hruodr, Machiaveli,

thank you very much for the continued discussion, it brings important point(s) that I have to consider. I have looke at the mbsync(1), which recites in part:
Expunge {None|Master|Slave|Both}
Permanently remove all messages [on the Master/Slave] marked for
deletion. (Global default: None)
Click to expand...
It is rather ambiguous. For example, if one were to select "Master", all the messages marked for deletion on the server will be deleted. However, what will happen when upon mbsync -a? Will it delete the all the messages marked for deletion on the server? If not, is it not a solution?

On a different note, my download/synchronization appear to have finished successfully, so I have all of my e-mail form one of the providers on a local machine. Thus, I can now at least archive the mail whose topics are no longer active.

Hi Mjölnir,

thank you for the reply. As this had been suggested, I have been contemplating the solution also. Would it not force me to use the client all the time? What will happen when I travel to a location, where I cannot use my laptop and make changes via the web interface?

Kindest regards,

M
 
mefizto said:
thank you for the reply. As this had been suggested, I have been contemplating the solution also. Would it not force me to use the client all the time? What will happen when I travel to a location, where I cannot use my laptop and make changes via the web interface?
Click to expand...
No, not @all -- that's maybe the biggest advantage to use IMAP: the mail stays on the server (which is usually professionally backup'ed etc.pp.). I.e. I can handle my mail with KMail/Claws Mail/Evolution/whatever, and when I'm @friend or internet café etc. w/o my laptop, I can still access my mail via my e-mail provider's web interface. Remember that IMAP allows to set the option "keep mail on server", which implies all the folders I create to sort my mail are created on the server, and I have local copies of all mails that I decide to read. For those I do not read, only the headers (subject, sender etc.) are downloaded. EDIT To make it clear: the local client checks for any changes, i.e. when I change s/th via the web interface, my local mail client (MUA) will get these changes once it connects to the IMAP mail server (MTA).
 
Hi Mjölnir,

thank you very much for the detailed an clear explanation. I gather that the "keep mail on the server" is set at the client. I will look at some of them.

Kindest regards,

M
 
mefizto said:
It is rather ambiguous. For example, if one were to select "Master", all the messages marked for deletion on the server will be deleted. However, what will happen when upon mbsync -a? Will it delete the all the messages marked for deletion on the server? If not, is it not a solution?
Click to expand...
I think yes, it will delete them if you select master. And not expunging is not a solution, because at one
time you will want to delete mails.

mbsync seems to work with IDs and has this advantage over fetchmail and similar.
It seems to be a nice program, but its purpose is handling mail, not archiving them. Using it for both may be
risky. I think one must strictly separate an archive from the collections of mails you are working in.

Just try mail/alpine. It is for reading directly from the server, not from a local copy
done for example with mbsync, but you can save mails in local unix folders, as I
explained in a posting above, and read also them with alpine or any mail client able
to read unix mailboxes.

Alpine does not support out of the box the Maildir format created by mbsync. If you
insist using mbsync, you find other programs, but also in that case is better to separate
the archive from the Maildir.
 
Mjölnir said:
EDIT To make it clear: the local client checks for any changes, i.e. when I change s/th via the web interface, my local mail client (MUA) will get these changes once it connects to the IMAP mail server (MTA).
Click to expand...
It does the work of mbsync and the work of a non imap client combined. I like the idea of separation
of the tasks.
 
H hruodr,

thank you for the reply. As I explained, until now I had been using e-mail without giving it any thoughts, and started the investigation only recently, prompted by the need to have some e-mail accessible long term. Thus, I am just stumbling about. I do not insist on any application, I tried the mail/mbsync based on the several references and nice tutorials posted by the gentlemen.

I, too, would prefer to separate the archiving and managing. But, despite everybody's advises, I am still not clear how to do it efficiently. In that regards, it is unclear to me, how your proposed use of mail/alpine is different form what Mjölnir suggested.

One inelegant solution I contemplated would be to separate the archive-able e-mail, store it at several locations and delete it form the server and local mailbox. Then, just do daily snapshots on the mailbox, thus preserving any changes.
Kindest regards,
M
 
And I like to click with the mouse or on a touchsceen to an icon to do what I want NOW instead of reading 12 manpages to handle cryptic config files & invest 12 days with a plethora of CLI tools for what a single user-friendly GUI tool can do by default in 10 minutes...
 
mefizto said:
until now I had been using e-mail without giving it any thoughts, and started the investigation only recently, prompted by the need to have some e-mail accessible long term. Thus, I am just stumbling about. I do not insist on any application,
Click to expand...

Not insisting on any application is correct: long term archiving should be independent of the perhaps short
living application. The application you use should follow standards, so that you can change the application.
The way how the application stores locally the mail plays here role. mbsync stotres it as Maildir,
each mail in a directory, alpine stores it as unix mbox, all mails concatenated in a file. Of
course you can have many mbox-es containing only one or few mails with the filenames you want ordered
in directories). For Maildir and mbox you will find enough programs, some support both.

mefizto said:
One inelegant solution I contemplated would be to separate the archive-able e-mail, store it at several locations and delete it form the server and local mailbox. Then, just do daily snapshots on the mailbox, thus preserving any changes.
Click to expand...

That is the way. The server store or the "local mailbox", namely the local image of it if you synchronize,
are not for archiving. You delete from these places when your current work ended, but the mail should
remain archived. But for archiving you may use the same format, namely a different local mailbox. See
the formats here:

Mbox - Wikipedia

en.wikipedia.org en.wikipedia.org

en.wikipedia.org

Maildir - Wikipedia

en.wikipedia.org en.wikipedia.org
 
Mjölnir said:
And I like to click with the mouse or on a touchsceen to an icon to do what I want NOW instead of reading 12 manpages to handle cryptic config files & invest 12 days with a plethora of CLI tools for what a single user-friendly GUI tool can do by default in 10 minutes...
Click to expand...

Read exactly what you wrote: because you do not want to learn how to use CLI tools, you expend every time
you use your software 10 minutes for things that the CLI user does in less than 1 minute.
 
I find this thread interesting, because also I have the need to archive mails, I do it in a primitive way,
and many times I felt the need to do it more systematically, specially when searching a mail from my
"archive".

As user of alpine, I think a directory of hierarchies with mboxes at the leaves would be a good
way. alpine easily allows to build it by creating new directories/mboxes and putting mails to
be saved there. Then I would write a small program that takes as argument the root of such hierarchy and
makes or updates a sqlite3 database containing the values of some headers as fields, eventually a field
with the whole text part, but that would inflate the db. Or perhaps there exist such program?
 
hruodr said:
Read exactly what you wrote: because you do not want to learn how to use CLI tools, you expend every time
you use your software 10 minutes for things that the CLI user does in less than 1 minute.
Click to expand...
Well, you may have already realized from some of my posts that neither I refused to learn CLI usage & sh(1)ell & awk(1) script sorcery, nor - generally - fail to set up my UNIX boxes to provide a comfortable environment in both worlds; of course occasionally I fail, then I find cure here :).

The point is: for the average mortal noob, all that bit fiddling is simply far too non-intuitive, and a good GUI turns a computer to be a useful tool for them. Besides that, doing CLI is much more errorprone (due to typos) that clicking some intuitive icons, no matter whether you're a CLI wizzard or a noob. Of course this doesn't mean all the GUI tools available are of reasonably good quality & their UI is designed by following proven, intuitive design principles; sadly, many FLOSS GUI tools are not, but some are extraordinarily good (e.g. the mentioned deskutils/kdepim). And I do urge & try to encourage with all my answers, all non-techies to develop some basic understanding of what happens behind the scenes when they do Clickybunti.
 
Hi hruodr,

thank you for the reply.

Regarding the mailboxes, I read about the different formats, and really like the fact that the Maildir separates the messages automatically. So, I will try to find an application that works with those.

Regarding the archiving, you appear to approve my suggestion, as well as propose a different scheme using database. Could you elaborate? I had actually thought about it more, and found an application that has a port mail/notmuch. It allows to assign a tag to a message. The idea I have is to assign the same tag to the group of messages, and copy the messages with the same tag to an archive. I am still not sure how the tagging works, so more research is needed.

Hi Mjölnir,

I do understand your position that clicking on an icon is potentially less error prone than typing a command. On the other hand, I find the CLI rather intuitive and flexible. I found it rather ingenious like Microsoft's VBA allowed one to write a command/macro and then build a GUI and assign an icon that executed the command/macro.

Kindest regards,

M
 
mefizto said:
Regarding the archiving, you appear to approve my suggestion, as well as propose a different scheme using database. Could you elaborate?
Click to expand...

Nothing special, it is what everyone would do.

You have first a directory hierarchy tree whose leaves are files, namely mboxes containing many mails.
This is the first step organizing your archive: you decide the hierarchy, you decide the names of the
directories and of the mailbox files, you decide what mail goes into what mailbox file, or what mail will not
be archived. This is easily done with mail/alpine.

But for searching, I would write a script with tcl and sqlite3 that generates a sqlite3 database containing
one record for each mail, in each record fields for many metadata you found in the headers of the mails
and perhaps the body, and also medata for finding your mail in the above directory hierarchy and corresponding mailbox. For example fileds for: path to mailbox, position of first byte of the mail in
mailbox, length of the mail, MessageID, References, Date, Subject, From, perhaps a "normalized" form
of the body of the mail, etc. You can also associate tags to mails with a sql db. You search on this db
either with sql commands, or with cli programs you write, also for retrieving a copy of a mail found,
for example giving the rowid that a search should deliver.

I think, it is not difficult to write it, but some care is needed.

BTW. The mail/notmuch you found looks interesting.
 
Hi hruodr,

if I understand your reply, you are proposing one mbox per message. However, does this not contradict your desire to be standard compliant, since the mbox format concatenates all the messages into a single mbox? And, as a consequence, will this not be a problem when/if you are forced to switch the e-nail application?

I have misunderstood your use of the database, I thought it was for archiving.

BTW, if you figure out how to use the mail/notmuch to accomplish what you intended with your - I presume auomatic - database generation, please let me know. I have been reading and reading, but still cannot grasp the concept of automatic tagging that some people proposed.

Kindest regards,

M
 
mefizto said:
if I understand your reply, you are proposing one mbox per message
Click to expand...
No, many mails pro mbox, and many mboxes pro directory.

mbox concatenates, but you may have many mboxes, and alpine may create for you the
many mboxes. You can also put only one mail in a mailbox, but it is a waste of inodes, and grouping
many related mails in one mbox should be seen as an advantage.

I do not consider Maildir with its strange filenames an advantage when archiving.

The directory hierarchy with mboxes is the archive with the original mails and should not be touched.
The database, that can be updated, modified, deleted, recreated is only for searching.

Yes, I will look at mail/notmuch.
 
Hi hruodr,

I have the false impression that I start understanding. You intend to create a mailbox per related e-mail(s). That should yield an easy archiving. But, what is an e-mail should belong to a multiple mailboxes?

In any event, please keep me informed when/if you implement the scheme, so that I can shamelessly ride on you co-tails.

Kindest regards,

M
 
mefizto said:
But, what is an e-mail should belong to a multiple mailboxes
Click to expand...
You can put the same mail in many mboxes, but I would preffer to associate them with tags in the/a db.

I have a lot of projects pending / interrupted. No idea when I will have the calm to begin again coding.

I had a similar project, a small C program to be put in /etc/mail/aliases for inserting arriving mails in a sqlite3
db, filling some fields according to headers. And a CGI script for viewing these mails from the db. The idea
was: sqlite3 db instead of mbox. It seems to work, but I never put it in /etc/aliases because I do not consider
it ready. That was C, because it was to be put in that critical place and I wanted it so fast and meager
as possible. In any case, I got some experience with mail to help implementing this new idea.

But perhaps is there a professional programmer that do it faster than I?
 
Hi hruodr,

yeah, there is still the tension/relationship between the mailboxes and tags, that I do not fully comprehend, hence my liking the single mailbox of Maildir.

Kindest regards,

M
 
You must log in or register to reply here.
Back
Top