| View previous topic :: View next topic |
| Author |
Message |
carpman
Advocate
Joined: 20 Jun 2002
Posts: 2202
Location: London - UK
|
 Posted: Sun May 27, 2007 7:00 pm Post subject: |
 |
|
| adsmith wrote: | OK, this is no longer an archivemail problem, but a combination of bash and cron.
First, you should go google for "Advanced bash scripting guide" for some pointers on scripting.
some cobination of archivemail, chown, and maybe sudo will do what you want, I think.
Although... I'm not sure why you want root to cycle other user's mailboxes, since users generally want complete control of their own mailboxes. The users could always run their own cron job... |
Ok but i am crap at scripting so that is going to be a pain?
Why do i want root to archive users mail? Because i am (newbie) sysadmin and users can't even put mail in folders let alone archive it, plus this a remote imap server so am not going to let them any where near things such cron. Also as a business all email must be kept for X amount of years so just getting them to delete it is not going work, which they have not done that for years and now some maildir are GB in size?
I currently use cron and home wild card to rub SA learn on multiple user home maildir so thought archive might do this?
| Code: |
*/30 * * * * sa-learn --dir --spam /home/*/.maildir/.spam > /dev/null 2>&1
*/30 * * * * sa-learn --dir --ham /home/*/.maildir/ > /dev/null 2>&1
|
_________________
Work Station - 64bit
Gigabyte GA X48-DQ6 Core2duo E8400
8GB GSkill DDR2-1066
SATA Areca 1210 Raid
BFG OC2 8800 GTS 640mb
--------------------------------
Notebook
Samsung Q45 7100 4gb |
|
| Back to top |
|
 |
adsmith
Veteran
Joined: 26 Sep 2004
Posts: 1386
Location: NC, USA
|
| Posted: Sun May 27, 2007 7:18 pm Post subject: |
|
|
No, it will be easy.
I would do something like a simple "do" loop running over the users homedirs Here is pseudo-code bash outline-- do not try to run it or anything:
| Code: |
#!/bin/bash
## archive everyone's mail
find /home/ -maxdepth 1 -type d | while read USERDIR; do
archivemail --(whatever) -o (whereever) ${USERDIR}/.maildir
chown $(basename ${USERDIR}) [the output location]
end do;
|
I think the chown might be necessary in case archivemail will make the archive owned by root. |
|
| Back to top |
|
|
carpman
Advocate
Joined: 20 Jun 2002
Posts: 2202
Location: London - UK
|
| Posted: Sun May 27, 2007 7:29 pm Post subject: |
|
|
| adsmith wrote: | No, it will be easy.
I would do something like a simple "do" loop running over the users homedirs Here is pseudo-code bash outline-- do not try to run it or anything:
| Code: |
#!/bin/bash
## archive everyone's mail
find /home/ -maxdepth 1 -type d | while read USERDIR; do
archivemail --(whatever) -o (whereever) ${USERDIR}/.maildir
chown $(basename ${USERDIR}) [the output location]
end do;
|
I think the chown might be necessary in case archivemail will make the archive owned by root. |
Thanks for vote of confidence, still no wiser but if try i now have somewhere to start, besides i still have to get archivemail working, but that is another thread 
_________________
Work Station - 64bit
Gigabyte GA X48-DQ6 Core2duo E8400
8GB GSkill DDR2-1066
SATA Areca 1210 Raid
BFG OC2 8800 GTS 640mb
--------------------------------
Notebook
Samsung Q45 7100 4gb |
|
| Back to top |
|
|
BitJam
Advocate
Joined: 12 Aug 2003
Posts: 2513
Location: Silver City, NM
|
| Posted: Fri May 09, 2008 4:03 pm Post subject: |
|
|
Here is the code for my maildir archive program. Use it at your own risk!
| Code: | #!/usr/bin/perl
#
# archive-maildir -- A Perl program for archiving messages
# in a maildir format to another maildir.
use strict;
use Getopt::Long;
my $CHUNK_SIZE = 100; #-- how many files to move in one go
my $FIND_DIRS = q(find '%s' -type d -and \\( -name cur -or -name new -or -name tmp \\) -print);
my $DIR_EXT = ".archive";
my $PRETEND;
my $VERBOSE;
my ($LIST_MODE, $NEWER, $OLDER, $DIRS, $MKDIRS, $COPY);
my $ME = $0; $ME =~ s{.*/}{};
my $USAGE = <<USAGE;
Usage: $ME [options] maildir [new_maildir]
Options:
-c --copy Copy files instead of moving them.
-d --dirs Read directory list from STDIN
-h --help Print this help.
-l --list Just list the directories found.
-m --mkdir Make destination dirs even if empty.
-n --newer DAYS Moves files newer than DAYS
-o --older DAYS Moves files older than DAYS
-p --pretend Don't really move anything.
-v --verbose Print what we do as we do it.
You can specify both --older and --newer.
If a new_maildir is not specified, maildir$DIR_EXT is used.
USAGE
GetOptions(
"copy" => \$COPY,
"dirs" => \$DIRS,
"help" => sub { print $USAGE; exit},
"pretend" => \$PRETEND,
"verbose" => \$VERBOSE,
"older=i" => \$OLDER,
"newer=i" => \$NEWER,
"list" => \$LIST_MODE,
"mkdir" => \$MKDIRS,
) or die "\n$USAGE";
my $Move_Word = $COPY ? "copy" : "move";
my $Move_Wordc = ucfirst $Move_Word;
my $Move_Worded = $COPY ? "copied" : "moved";
#$OLDER or $NEWER or $LIST_MODE or
# die "$ME: Cowardly refusing to archive mail without --older or --newer set\n";
@ARGV or die "$ME: Must specify a maildir directory.\n";
my $MAILDIR = shift;
$MAILDIR =~ s{/+$}{}; #-- strip trailing /'s
my $NEW_DIR = shift || $MAILDIR . $DIR_EXT;
$NEW_DIR =~ s{/+$}{};
@ARGV and die "$ME: Extra command line arguments: @ARGV\n";
my @dirs;
if ($DIRS) {
@dirs = <>;
}
else {
my $find_dirs = sprintf $FIND_DIRS, $MAILDIR;
#$VERBOSE and print "$find_dirs\n";
@dirs = `$find_dirs`;
}
chomp @dirs;
$LIST_MODE and do {
print map {"$_\n"} @dirs;
exit;
};
my $file_cnt;
for my $dir (@dirs) {
my $new_dir = $dir;
$new_dir =~ s{^$MAILDIR/}{$NEW_DIR/} or do {
warn "directory '$dir' does not start with '$MAILDIR'. Skipping.\n";
next;
};
#print "$dir\n$new_dir\n\n";
my $cmd = "find '$dir' -type f -and ";
my @opts;
$OLDER and push @opts, "-mtime +$OLDER";
$NEWER and push @opts, "-not -mtime +$NEWER";
@opts and $cmd .= join " ", "\\(", join(" -and ", @opts), "\\)";
$cmd .= " -print";
#print "$cmd\n";
my @files = `$cmd`;
chomp(@files);
@files or do {
$MKDIRS and my_mkdir($new_dir);
next;
};
$file_cnt += @files;
my $s = @files == 1 ? "" : "s";
print "$Move_Wordc (@{[scalar @files]}) file$s to $new_dir\n" if $PRETEND or $VERBOSE;
move_files($new_dir, \@files);
}
print "($file_cnt) files to $Move_Word.\n" if $file_cnt and $PRETEND;
print "($file_cnt) files should have been $Move_Worded.\n" if $file_cnt and not $PRETEND;
#--- End of Main --------------------------------------------------------------
sub move_files {
my ($dest, $files) = @_;
my_mkdir($dest) or return;
$PRETEND and return;
while (@$files) {
my @chunk = splice(@$files, 0, $CHUNK_SIZE);
@chunk or last;
my $cmd = $COPY ? "cp" : "mv";
my @cmd = ("mv", qq{--target-directory=$dest});
#print join(" ", @cmd, @chunk), "\n\n";
#next;
system(@cmd, @chunk);
}
}
sub my_mkdir {
my $dir = shift;
return 1 if -d $dir;
print "mkdir: $dir\n" if $VERBOSE;
return 1 if $PRETEND;
system("mkdir", "-p", $dir);
-d $dir and return 1;
warn "$ME: Warning, could not create directory '$dir'\n";
return;
}
|
|
|
| Back to top |
|
|
redwood
Guru
Joined: 27 Jan 2006
Posts: 306
|
| Posted: Tue Dec 15, 2009 12:00 am Post subject: |
|
|
I though I might give this a try with my courier-imap maildir folders.
I have a lvm2 partition for backups "/mnt/mail/archives/"
and I created a backup folder for someuser:
# maildirmake -f Archives ~someuser/.maildir
Then moved the maildir folder:
# mv ~someuser/.maildir/.Archives /mnt/mail/archives/someuser
Then created a mount point, and mounted as bind:
# mkdir ~someuser/.Archives
# mount -t bind /mnt/mail/archives/someuser ~someuser/.maildir/.Archives
When I make a backup with the script,
archive-mail --list ~someuser/.maildir/ | grep -v Archives | \
bin/archive-mail -v --older 180 --pretend ~someuser/.maildir ~someuser/.maildir/.Archives
the script re-creates the dir structure of ~someuser/.maildir
e.g.
~someuser/.maildir/.Archives/{.INBOX, .FAX, ...}
which results in folders which aren't accessible with courier-imap.
Subfolders of the topdir .maildir must be created using periods instead of /
e.g.
~someuser/.maildir/{.Archives.INBOX, .Archives.FAX.From, .Archives.FAX.To}
I also tried creating a shared-folder
# maildirmake /mnt/mail/shared
which creates /mnt/mail/shared/cur/,new/,tmp/ folders}
# maildirmake -S /mnt/mail/shared/archives/
which creates /mnt/mail/shared/archives{cur/,new/.tmp/}
# maildirmake -s write -f someuser /mnt/mail/shared/archives
which creates /mnt/mail/shared/archives/.someuser{cu/r,new/,tmp/,maildirfolder}
# su someuser -
# maildirmake --add Archives=/mnt/mail/shared/archives/ $HOME/.maildir
which creates shared-maildir file containing name(s) of sharedfolders
and creates ~someuser/.maildir/shared-folders/archives/someuser{cur/,new/,tmp/,
shared->/mnt/mail/shared/archives/.someuser, courierimapkeywords/, courierimapuiddb,shared-timestamp}
which results in similar folder-name hell.
Anyhow, the script does seem to work, ie. it finds all files older than some time, and copies them into
a new place, preserving the directory structure. But browsing the new maildir folder presents a problem if using
courier-imap.
There is a CTAN module, MailFolder, for creating proper maildir folders/filenames, which might be the best way to go. |
|
| Back to top |
|
|
BitJam
Advocate
Joined: 12 Aug 2003
Posts: 2513
Location: Silver City, NM
|
| Posted: Tue Dec 15, 2009 1:26 am Post subject: |
|
|
| Quote: | | Anyhow, the script does seem to work, ie. it finds all files older than some time, and copies them into a new place, preserving the directory structure. But browsing the new maildir folder presents a problem if using courier-imap. |
It was difficult for me to follow exactly what you want to do. It appears that you want the archive-mail program to translate from one format to another which is not what it was designed to do. Did the archive-mail program create your browsing problem or did the problem already exist with the folders you were started with? |
|
| Back to top |
|
|
redwood
Guru
Joined: 27 Jan 2006
Posts: 306
|
| Posted: Tue Dec 15, 2009 4:22 pm Post subject: |
|
|
Sorry for not being more clear.
I have a small office network.
I run a courier-imap+postfix mail server (ala various Gentoo guides)
My users mostly use kmail (and occasionally squirrelmail) to access
their individual courier-imap accounts as well as a general office courier-imap account.
I also run a hylafax server which emails faxes to fax@mydomain which then get
sorted by procmail into a FAX subfolder of the main office account.
The volume of faxes+emails is probably a couple hundred a week, which over time
becomes thousands of email messages. And since many of the emails contain attachments
such as scans of contracts, orders, etc., the ~office/.maildir folder has grown to gigabytes.
I can either buy more disk storage, and move ~office/.maildir to a bigger logical volume,
or ideally, automatically backup the folder to somewhere else, then delete the old emails.
But I want to be able to access the old emails/faxes if necessary. My regular backup is a
complete rsnapshot (rsync hardlinks) backup to /mnt/backups/{daily.?, weekly.?}
But if I need to find a particular email message from
/mnt/backups/weekly.4/www.mydomain.com/home/office/.maildir/.FAX/cur/
a file browser doesn't help because email messages with names like
1260889743.23851_0.www.mydomain.com:2,S
are meaningless without a mail program to display the From, To, Subject, etc
fields from the mail db.
Since my users use kmail with multiple imap accounts
(eg. user1@mydomain, office@mydomain, spamtrap@mydomain)
I thought I might create an Archive folder under the office@mydomain imap account
and routinely backup old emails to this folder, then delete the originals.
If I backup ~office/.maildir/.FAX to ~office/.maildir/.Archives
with your archive-mail script, it creates the folder
~office/.maildir/.Archives/.FAX into which it puts the emails,
but this folder is not visible with kmail because it is improperly named
for an imap subfolder.
The folder should instead be ~office/.maildir/.Archives.FAX
and similarly for other folders, eg.
~office/.maildir/.Archives.Contracts.2008
rather than ~office/.maildir/.Archives/.Contracts.2008
I suppose I could create an imap account archives@mydomain
and then backup ~office/.maildir to ~archives/.maildir
which would create correctly named folders.
I realize I could also use archivemail to backup my imap maildir folders to mbox files,
but then how to view the .mbox file? I think if I open a .mbox file with kmail,
it automatically imports the mail back into my INBOX.
I suppose I haven't thought through this issue thoroughly yet,
which is why I was looking in the forums for ideas on archiving emails. |
|
| Back to top |
|
|
BitJam
Advocate
Joined: 12 Aug 2003
Posts: 2513
Location: Silver City, NM
|
| Posted: Tue Dec 15, 2009 5:49 pm Post subject: |
|
|
Thanks for the crystal clear explanation. I now understand what you want and why you want it. Makes sense. I will think about it. In the meantime I think this is the easiest course of action:
| Quote: | I suppose I could create an imap account archives@mydomain
and then backup ~office/.maildir to ~archives/.maildir
which would create correctly named folders. |
but there is no need to create a separate account. KMail lets you select the top maildir folder so it might be easier to just change that instead of logging in as a different user. I see why it would be more convenient for you to have working ~/.maildir/ARCHIVE directory. |
|
| Back to top |
|
|
redwood
Guru
Joined: 27 Jan 2006
Posts: 306
|
|
| Back to top |
|
|
carpman
Advocate
Joined: 20 Jun 2002
Posts: 2202
Location: London - UK
|
| Posted: Wed Jan 27, 2010 12:30 am Post subject: |
|
|
Having issues with out pf space message but is plenty of space on target backup partition?
It is archiving 5gb worth or email
| Code: |
archivemail -d400 -s '' -o /home/backup /home/user1/.maildir/.Archive_Sent/
archivemail: Warning - changing effective user id: this automatic feature is deprecated and will be removed from later versions.
Traceback (most recent call last):
File "/usr/bin/archivemail", line 1602, in <module>
main()
File "/usr/bin/archivemail", line 702, in main
archive(mailbox_path)
File "/usr/bin/archivemail", line 1154, in archive
_archive_dir(mailbox_name, final_archive_name, "maildir")
File "/usr/bin/archivemail", line 1297, in _archive_dir
archive.write(msg)
File "/usr/bin/archivemail", line 383, in write
self.mbox_file.write(body)
File "/usr/lib/python2.6/gzip.py", line 197, in write
self.fileobj.write( self.compress.compress(data) )
IOError: [Errno 28] No space left on device
Exception IOError: (28, 'No space left on device') in <bound method GzipFile.__del__ of <gzip on 0x9b892ac>> ignored
|
_________________
Work Station - 64bit
Gigabyte GA X48-DQ6 Core2duo E8400
8GB GSkill DDR2-1066
SATA Areca 1210 Raid
BFG OC2 8800 GTS 640mb
--------------------------------
Notebook
Samsung Q45 7100 4gb |
|
| Back to top |
|
|
redwood
Guru
Joined: 27 Jan 2006
Posts: 306
|
| Posted: Wed Jan 27, 2010 2:18 pm Post subject: |
|
|
What does 'df -h /home/backup' report?
What does 'du -h /home/backup' report?
What format filesystem is /home/backup?
Are you perhaps running out of inodes? |
|
| Back to top |
|
|
carpman
Advocate
Joined: 20 Jun 2002
Posts: 2202
Location: London - UK
|
| Posted: Wed Jan 27, 2010 5:10 pm Post subject: |
|
|
| redwood wrote: | What does 'df -h /home/backup' report?
What does 'du -h /home/backup' report?
What format filesystem is /home/backup?
Are you perhaps running out of inodes? |
Out is
| Code: |
df -h /home/backup
Filesystem Size Used Avail Use% Mounted on
/dev/sda10 123G 76G 47G 62% /home
|
format is reiserfs
once backup is done then are moved to external drive.
_________________
Work Station - 64bit
Gigabyte GA X48-DQ6 Core2duo E8400
8GB GSkill DDR2-1066
SATA Areca 1210 Raid
BFG OC2 8800 GTS 640mb
--------------------------------
Notebook
Samsung Q45 7100 4gb |
|
| Back to top |
|
|
redwood
Guru
Joined: 27 Jan 2006
Posts: 306
|
| Posted: Wed Jan 27, 2010 6:47 pm Post subject: |
|
|
Are
/home/backup and
/home/user1/.maildir/.Archive_Sent/
on the same partition?
The output of 'df -h /home/backup' says that backup is a folder in /home partition, right?
And you say that 'du -ch /home/user1/.maildir/.Archive_Sent/' reports ~5G?
I don't know if archivemail uses /tmp, but you could check 'df -h /tmp' |
|
| Back to top |
|
|
carpman
Advocate
Joined: 20 Jun 2002
Posts: 2202
Location: London - UK
|
| Posted: Wed Jan 27, 2010 7:22 pm Post subject: |
|
|
| redwood wrote: | Are
/home/backup and
/home/user1/.maildir/.Archive_Sent/
on the same partition?
The output of 'df -h /home/backup' says that backup is a folder in /home partition, right?
And you say that 'du -ch /home/user1/.maildir/.Archive_Sent/' reports ~5G?
I don't know if archivemail uses /tmp, but you could check 'df -h /tmp' |
yes they are on same partition but as i said after back the file are moved to external drive, don't do straight onto external as is slow usb drive.
I was wondering myself if it used /tmp and if it does this is going to be an issue as /tmp is is only 1.9gb
| Code: |
df -h /tmp
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 1.9G 33M 1.9G 2% /tmp
|
I have one account that is going to require 10gb plus archive
_________________
Work Station - 64bit
Gigabyte GA X48-DQ6 Core2duo E8400
8GB GSkill DDR2-1066
SATA Areca 1210 Raid
BFG OC2 8800 GTS 640mb
--------------------------------
Notebook
Samsung Q45 7100 4gb |
|
| Back to top |
|
|
redwood
Guru
Joined: 27 Jan 2006
Posts: 306
|
| Posted: Wed Jan 27, 2010 7:43 pm Post subject: |
|
|
I don't know python very well,
but it looks like a temp_dir is created/used at line 1130
of /usr/bin/archivemail
and prints out what temp_dir is being used:
vprint("set tempfile directory to '%s'" % new_temp_dir) |
|
| Back to top |
|
|
carpman
Advocate
Joined: 20 Jun 2002
Posts: 2202
Location: London - UK
|
| Posted: Wed Jan 27, 2010 7:55 pm Post subject: |
|
|
| redwood wrote: | I don't know python very well,
but it looks like a temp_dir is created/used at line 1130
of /usr/bin/archivemail
and prints out what temp_dir is being used:
vprint("set tempfile directory to '%s'" % new_temp_dir) |
it does but no option or clue as to where this tmp dir is?
_________________
Work Station - 64bit
Gigabyte GA X48-DQ6 Core2duo E8400
8GB GSkill DDR2-1066
SATA Areca 1210 Raid
BFG OC2 8800 GTS 640mb
--------------------------------
Notebook
Samsung Q45 7100 4gb |
|
| Back to top |
|
|
redwood
Guru
Joined: 27 Jan 2006
Posts: 306
|
|
| Back to top |
|
|
boerKrelis
Apprentice
Joined: 01 Jul 2003
Posts: 241
Location: The Netherlands
|
| Posted: Thu Jan 28, 2010 12:16 am Post subject: |
|
|
I have not checked, but I hope all of us realize (and I'm sure that at least some of us do) that using to find file by date spec does not fly very well on maildir. The file creation/modification date don't say much about the date the e-mail was received. Maybe the user moved some e-mail from 2007 inter-imap-account. Wham, according to find, the mail is now from 2010. Maybe the user changed the 'read' flag on the mail, causing the file's mtime to change.
You need tools that look into the maildir files for determining their arrival date. Archivemail is probably doing so ;-)
Most maildir implementations use a timestamp as the first part of the file, though, so you can still get creative using file utils if you insist, but those timestamps have no meaning to the message the MUA sees. |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Networking & Security
