Until two weeks ago I was using an IMAP server (based on Zimbra) for my work email, but the date for migration to Gmail arrived with no choice to postpone…

I was very tied to using my current setup, where:

  • offlineimap was downloading all the email to a local maildir folder,
  • imapfilter classified the email into folders based on local options
  • mutt accessed the maildir folder for working with the emails
  • a script to remove duplicate emails from disk imapdedup.py before next sync

But, with the change and peculiarities for Gmail, it was no longer working… I was trying several times with different combinations of folder translations but each one took approximately one day to sync all emails, to find out the next issue with the folder translation.

After this, I tried to just skip using my old setup and switch to web interface… I was already using it for personal email for years, but with lot of mailing lists and lots of emails, didn’t worked the way I wanted.

I finally went the intermediate way: using imapfilter directly against Gmail and customize a bit the inbox folders.

This allowed me to adapt the filtering, so that I can still have a readable INBOX with all the relevant mails and keep at the same time the mails sorted as I used to.

My inbox is now configured as multiple inbox with the following filters:

  • is:starred
  • (is:unread AND NOT label:Tags/_pending)

Those show below my regular INBOX folder, allowing to still feature some emails for tracking, and all the others filtered into folders that are not in the pending sort are also displayed.

This is powered with some filters (Gmail side), that keep my inbox clean:

  • to:(-myemail1 -myemail2) -> Skip Inbox, Apply label "Tags/_pending"
  • to:(myemail or myemail2) (invite.ics OR invite.vcs) has:attachment -> Delete it
  • from:([email protected]) -> Skip Inbox, Apply label "Tags/_pending"

Now, the imapfilter part via this .imapfilter/config.lua file:

---------------
--  Options  --
---------------

options.timeout = 60
options.subscribe = true
options.create = true
options.expunge = true

----------------
--  Accounts  --
----------------

-- Connects to "imap1.mail.server", as user "user1" with "secret1" as
-- password.
MAILSERVER = IMAP {
    server = 'imap.gmail.com',
    username = 'MYEMAIL',
    password = 'MYAPPPASSWORD',
    ssl = "tls1"
}
-- My email
myuser = 'MYUSERINEMAILS'

function mine(messages)
    email=messages:contain_cc(myuser)+messages:contain_to(myuser)+messages:contain_from(myuser)
    return email
end

function filter(messages,email,destination)
    messages:contain_from(email):move_messages(destination)
    messages:contain_to(email):move_messages(destination)
    messages:contain_cc(email):move_messages(destination)
    messages:contain_field('sender', email):move_messages(destination)
    messages:contain_field('List-ID', email):move_messages(destination)
end

function deleteold(messages,days)
    todelete=messages:is_older(days)-mine(messages)
    todelete:move_messages(MAILSERVER['[Gmail]/Papelera'])
end

function deleteoldcases(messages,days)
    todelete=messages:is_older(days)
    todelete:move_messages(MAILSERVER['[Gmail]/Papelera'])
end

function markread(messages)
    toread=messages:select_all()
    toread:mark_seen()
end

-- Define the msgs we're going to work on

-- Move sent messages to INBOX to later sorting
-- sent = MAILSERVER['sent']:select_all()
-- sent:move_messages(MAILSERVER['INBOX'])

INBOX = MAILSERVER['INBOX']:select_all()
pending = MAILSERVER['Tags/_pending']:select_all()
todos =  pending + INBOX

todos:contain_subject('Undelivered Mail Returned to Sender'):move_messages(MAILSERVER['[Gmail]/Papelera'])
todos:contain_subject('[sbr-stack] Report: OSP - Stale cases: '):move_messages(MAILSERVER['[Gmail]/Papelera'])
todos:contain_subject('[ sbr-stack-emea ] Cron <root@'):contain_from('(cron Daemon)'):move_messages(MAILSERVER['[Gmail]/Papelera'])
todos:contain_from('[email protected]'):move_messages(MAILSERVER['[Gmail]/Papelera'])

todos:contain_subject('Supplier Remittance Advice- Autogenerated please do not respond to this mail'):move_messages(MAILSERVER['Tags/Privado/Gastos'])

filter(todos:is_seen(),'[email protected]',MAILSERVER['Tags/Lists/OpenStack/gerrit'])

-- Mark as read messages sent from my user
markread(todos:contain_from(myuser))

-- Delete google calendar forwards
todos:contain_to('[email protected]'):move_messages(MAILSERVER['[Gmail]/Papelera'])

filter(todos:contain_subject('[PNT] '),'[email protected]',MAILSERVER['[Gmail]/Papelera'])

-- Filter CPG
filter(todos:contain_subject('Red Hat - Group '),'[email protected]',MAILSERVER['Tags/WORK/Customers/CPG'])

-- Delete messages about New accounts created
usercreated=todos:contain_subject('New Red Hat user account created')*todos:contain_from('[email protected]')
usercreated:move_messages(MAILSERVER['[Gmail]/Papelera'])

-- Search messages from CPG's
cpg = MAILSERVER['Tags/WORK/Customers/CPG']:select_all()
cpg:contain_subject('Red Hat - Group  - '):move_messages(MAILSERVER['[Gmail]/Papelera'])
cpg:contain_subject('(Unpublished)'):move_messages(MAILSERVER['[Gmail]/Papelera'])
cpg:contain_subject(': Where is '):move_messages(MAILSERVER['[Gmail]/Papelera'])

-- Move bugzilla messages
filter(todos:contain_subject('] New:'),'[email protected]',MAILSERVER['Tags/WORK/_bugzilla/new'])
filter(todos,'[email protected]',MAILSERVER['Tags/WORK/_bugzilla'])
bz = MAILSERVER['Tags/WORK/_bugzilla']:select_all()

-- Move unseen requests or answers
reqans=bz:contain_subject('needinfo requested:'):is_unseen() + bz:contain_subject('needinfo canceled:'):is_unseen()
reqans:move_messages(MAILSERVER['Tags/WORK/_bugzilla/reqans'])

reqans=MAILSERVER['Tags/WORK/_bugzilla/reqans']:is_seen()
reqans:move_messages(MAILSERVER['Tags/WORK/_bugzilla'])

-- Clasify on product
bz:contain_field('X-Bugzilla-Product', 'Red Hat Customer Portal'):move_messages(MAILSERVER['Tags/WORK/_bugzilla/portal'])
bz:contain_field('X-Bugzilla-Product', 'Red Hat Enterprise Linux'):move_messages(MAILSERVER['Tags/WORK/_bugzilla/rhel'])
bz:contain_field('X-Bugzilla-Product', 'Red Hat Satellite'):move_messages(MAILSERVER['Tags/WORK/_bugzilla/rhn'])
bz:contain_field('X-Bugzilla-Product', 'Red Hat Enterprise Virtualization Manager'):move_messages(MAILSERVER['Tags/WORK/_bugzilla/rhev'])
bz:contain_field('X-Bugzilla-Product', 'Fedora'):move_messages(MAILSERVER['Tags/WORK/_bugzilla/fedora'])
bz:contain_field('X-Bugzilla-Product', 'Red Hat Ceph Storage'):move_messages(MAILSERVER['Tags/WORK/_bugzilla/ceph'])
bz:contain_field('X-Bugzilla-Product', 'Red Hat Gluster Storage'):move_messages(MAILSERVER['Tags/WORK/_bugzilla/gluster'])

bz:contain_field('X-Bugzilla-Product', 'Red Hat OpenStack'):move_messages(MAILSERVER['Tags/WORK/_bugzilla/rhosp'])
bz = MAILSERVER['Tags/WORK/_bugzilla/rhosp']:select_all()

-- Clasify on component for OSP
bz:contain_field('X-Bugzilla-Component', 'openstack-ceilometer'):move_messages(MAILSERVER['Tags/WORK/_bugzilla/rhosp/ceilometer'])
bz:contain_field('X-Bugzilla-Component', 'openstack-cinder'):move_messages(MAILSERVER['Tags/WORK/_bugzilla/rhosp/cinder'])
bz:contain_field('X-Bugzilla-Component', 'openstack-designate'):move_messages(MAILSERVER['Tags/WORK/_bugzilla/rhosp/designate'])
bz:contain_field('X-Bugzilla-Component', 'openstack-glance'):move_messages(MAILSERVER['Tags/WORK/_bugzilla/rhosp/glance'])
bz:contain_field('X-Bugzilla-Component', 'openstack-heat'):move_messages(MAILSERVER['Tags/WORK/_bugzilla/rhosp/heat'])
bz:contain_field('X-Bugzilla-Component', 'openstack-ironic'):move_messages(MAILSERVER['Tags/WORK/_bugzilla/rhosp/ironic'])
bz:contain_field('X-Bugzilla-Component', 'openstack-keystone'):move_messages(MAILSERVER['Tags/WORK/_bugzilla/rhosp/keystone'])
bz:contain_field('X-Bugzilla-Component', 'openstack-neutron'):move_messages(MAILSERVER['Tags/WORK/_bugzilla/rhosp/neutron'])
bz:contain_field('X-Bugzilla-Component', 'openstack-nova'):move_messages(MAILSERVER['Tags/WORK/_bugzilla/rhosp/nova'])
bz:contain_field('X-Bugzilla-Component', 'rhel-osp-director'):move_messages(MAILSERVER['Tags/WORK/_bugzilla/rhosp/director'])
bz:contain_field('X-Bugzilla-Component', 'rhosp-director'):move_messages(MAILSERVER['Tags/WORK/_bugzilla/rhosp/director'])

bz = MAILSERVER['Tags/WORK/_bugzilla/rhev']:select_all()
bz:contain_field('X-Bugzilla-Component', 'vdsm'):move_messages(MAILSERVER['Tags/WORK/_bugzilla/rhev/vdsm'])

-- Move support messages once read into the Customer/cases folder
filter(todos:contain_subject('Case '):is_seen(),'[email protected]',MAILSERVER['Tags/WORK/Customers/cases'])

MAILSERVER['[Gmail]/Papelera']:select_all():mark_seen()

support = MAILSERVER['Tags/WORK/Customers/cases']:select_all()
-- Restart the search only for messages in Other to also process if we have new rules

support:contain_subject('is about to breach its SLA'):move_messages(MAILSERVER['[Gmail]/Papelera'])
support:contain_subject('has breached its SLA'):move_messages(MAILSERVER['[Gmail]/Papelera'])
support:contain_subject(' has had no activity in '):move_messages(MAILSERVER['[Gmail]/Papelera'])
markread(support:contain_subject('(WoC)'))
markread(support:contain_subject('(Closed)'))

-- Only work on already read messages
support = MAILSERVER['Tags/WORK/Customers/cases']:select_all():is_seen()

-- Process all remaining messages in INBOX + all read messages in pending-sort for mailing lists and move to lists folder
notminelistas=todos-mine(todos)
notminelistas:contain_field('List-ID','<'):move_messages(MAILSERVER['Tags/Lists'])
filter(notminelistas,'list', MAILSERVER['Tags/Lists'])
filter(todos:is_seen(),'list', MAILSERVER['Tags/Lists'])
filter(todos:is_seen(),'googlegroups.com', MAILSERVER['Tags/Lists'])
filter(todos,'bounces',MAILSERVER['Tags/Lists'])

-- Add RH lists, INBOX and _pending and Fedora default bin for reprocessing in case a new list has been added
lists = todos:is_seen() + MAILSERVER['Tags/Lists']:select_all() + MAILSERVER['Tags/Lists/Fedora']:select_all()

todos:contain_subject('unsubcribe'):move_messages(MAILSERVER['[Gmail]/Papelera'])
lists:contain_subject('unsubcribe'):move_messages(MAILSERVER['[Gmail]/Papelera'])

-- Mailing lists
filter(INBOX:is_seen(),'[email protected]',MAILSERVER['Tags/Lists/WORK/Tech/coderepos'])
filter(lists,'[email protected]',MAILSERVER['Tags/Lists/WORK/SysEng/CNV/kubevirt-dev'])
filter(lists,'[email protected]',MAILSERVER['Tags/Lists/WORK/SysEng/CNV/metalkube-dev'])

-- Fedora
filter(lists,'kickstart-list',MAILSERVER['Tags/Lists/Fedora/kickstart'])
filter(lists,'[email protected]',MAILSERVER['Tags/Lists/Fedora/Ambassador'])
filter(lists,'[email protected]',MAILSERVER['Tags/Lists/Fedora/infra'])
filter(lists,'[email protected]',MAILSERVER['Tags/Lists/Fedora/announce'])
filter(lists,'lists.fedoraproject.org',MAILSERVER['Tags/Lists/Fedora'])

-- OSP
filter(lists,'[email protected]',MAILSERVER['Tags/Lists/OpenStack'])
filter(lists,'[email protected]',MAILSERVER['Tags/Lists/OpenStack/Operators'])
filter(lists,'[email protected]',MAILSERVER['Tags/Lists/OpenStack/es'])
filter(lists,'[email protected]',MAILSERVER['Tags/Lists/OpenStack/rdo'])

filter(lists,'@bugs.launchpad.net',MAILSERVER['Tags/Lists/OpenStack/launchpad'])
lists:contain_field('X-Launchpad-Notification-Type', 'bug'):move_messages(MAILSERVER['Tags/Lists/OpenStack/launchpad'])

filter(lists,'[email protected]',MAILSERVER['Tags/Lists/Others/pgsql-hackers'])
filter(lists,'[email protected]',MAILSERVER['Tags/Lists/Others/pgsql-hackers'])

-- Filter  messages not filtered back to INBOX
pending:move_messages(MAILSERVER['INBOX'])

-- Start processing of messages older than:
maxage=365

-- Delete old messages from mailing lists

maxage=180
deleteold(MAILSERVER['Tags/Lists'],maxage)
deleteold(MAILSERVER['Tags/Lists/Fedora'],maxage)
deleteold(MAILSERVER['Tags/Lists/Fedora/Ambassador'],maxage)
deleteold(MAILSERVER['Tags/Lists/Fedora/announce'],maxage)
deleteold(MAILSERVER['Tags/Lists/Fedora/infra'],maxage)
deleteold(MAILSERVER['Tags/Lists/Fedora/kickstart'],maxage)
deleteold(MAILSERVER['Tags/Lists/OpenStack'],maxage)
deleteold(MAILSERVER['Tags/Lists/OpenStack/es'],maxage)
deleteold(MAILSERVER['Tags/Lists/Others/pgsql-hackers'],maxage)
deleteold(MAILSERVER['Tags/Lists/WORK/SysEng/CNV/kubevirt-dev'],maxage)
deleteold(MAILSERVER['Tags/Lists/WORK/SysEng/CNV/metalkube-dev'],maxage)



-- Delete old BZ tickets
maxage=30
deleteoldcases(MAILSERVER['Tags/WORK/_bugzilla/rhosp'],maxage)
deleteoldcases(MAILSERVER['Tags/WORK/_bugzilla/rhosp/ceilometer'],maxage)
deleteoldcases(MAILSERVER['Tags/WORK/_bugzilla/rhosp/cinder'],maxage)
deleteoldcases(MAILSERVER['Tags/WORK/_bugzilla/rhosp/designate'],maxage)
deleteoldcases(MAILSERVER['Tags/WORK/_bugzilla/rhosp/glance'],maxage)
deleteoldcases(MAILSERVER['Tags/WORK/_bugzilla/rhosp/heat'],maxage)
deleteoldcases(MAILSERVER['Tags/WORK/_bugzilla/rhosp/ironic'],maxage)
deleteoldcases(MAILSERVER['Tags/WORK/_bugzilla/rhosp/keystone'],maxage)
deleteoldcases(MAILSERVER['Tags/WORK/_bugzilla/rhosp/neutron'],maxage)
deleteoldcases(MAILSERVER['Tags/WORK/_bugzilla/rhosp/nova'],maxage)
deleteoldcases(MAILSERVER['Tags/WORK/_bugzilla/ceph'],maxage)

maxage=30
deleteoldcases(MAILSERVER['Tags/WORK/_bugzilla'],maxage)
deleteoldcases(MAILSERVER['Tags/WORK/_bugzilla/portal'],maxage)
deleteoldcases(MAILSERVER['Tags/WORK/_bugzilla/rhel'],maxage)
deleteoldcases(MAILSERVER['Tags/WORK/_bugzilla/rhn'],maxage)
deleteoldcases(MAILSERVER['Tags/WORK/_bugzilla/fedora'],maxage)
deleteoldcases(MAILSERVER['Tags/WORK/_bugzilla/gluster'],maxage)
deleteoldcases(MAILSERVER['Tags/WORK/_bugzilla/security'],maxage)


-- delete old cases
maxage=30

-- for each in $(cat .imapfilter/config.lua|grep -i cases|tr " ,()" "\n"|grep cases|sort|uniq|grep -v ":" );do echo "deleteoldcases($each,maxage)";done
deleteoldcases(MAILSERVER['Tags/WORK/Customers/cases'],maxage)

-- Empty trash every 7 days
maxage=7
old=MAILSERVER['[Gmail]/Papelera']:is_older(maxage)
old:move_messages(MAILSERVER['[Gmail]/Papelera'])

I think that is more or less self-explanatory, but in short, first sorts mails into folders based on the sender, recipient, keywords and finally, applies expiration policy to the folders to remove old emails that might not be relevant anymore.

In this case, it also keeps messages in which I was directly involved (removing, via the function to delete messages that are in the variable ‘mine’)

During the next days I’ll try to get back email in mutt, but this at least makes the usage in the meantime more bearable…

Enjoy! (and if you do, you can Buy Me a Coffee )