nixspamsum
author Pascal Volk <user@localhost.localdomain.org>
Sun, 11 Apr 2010 19:00:44 +0000
changeset 9 3a392067015c
parent 8 7f131cf431bc
child 10 07b9fe5c6fcf
permissions -rwxr-xr-x
Count also naked IPv4 addresses. Beautified NiXSapmSum:RE_PF.
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
0
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
     1
#!/usr/bin/env python
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
     2
# -*- coding: utf-8 -*-
7
83c4798e291d Small code cleanups. Added current year to the copyright notices.
Pascal Volk <user@localhost.localdomain.org>
parents: 6
diff changeset
     3
# Copyright 2009 - 2010 Pascal Volk
5
65590f05bb97 Added COPYING to the repository
Pascal Volk <user@localhost.localdomain.org>
parents: 4
diff changeset
     4
# See COPYING for distribution information.
0
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
     5
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
     6
__author__ = 'Pascal Volk'
4
ae58d9881be5 showResult(): count matches; break if no matches were found
Pascal Volk <user@localhost.localdomain.org>
parents: 3
diff changeset
     7
__version__ = '0.1.2'
ae58d9881be5 showResult(): count matches; break if no matches were found
Pascal Volk <user@localhost.localdomain.org>
parents: 3
diff changeset
     8
__date__ = '2009-07-03'
0
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
     9
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
    10
import os
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
    11
import re
8
7f131cf431bc Switched to fileinput module. This allows to read also gzip/bzip2
Pascal Volk <user@localhost.localdomain.org>
parents: 7
diff changeset
    12
import fileinput
0
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
    13
6
723b4c4e15cb Set/updated descriptions and error messages.
Pascal Volk <user@localhost.localdomain.org>
parents: 5
diff changeset
    14
class NiXSapmSum(object):
723b4c4e15cb Set/updated descriptions and error messages.
Pascal Volk <user@localhost.localdomain.org>
parents: 5
diff changeset
    15
    """
723b4c4e15cb Set/updated descriptions and error messages.
Pascal Volk <user@localhost.localdomain.org>
parents: 5
diff changeset
    16
    Small log parser class to parse and summarize NiX Spam DNSBL lookup
723b4c4e15cb Set/updated descriptions and error messages.
Pascal Volk <user@localhost.localdomain.org>
parents: 5
diff changeset
    17
    based rejects from a mail log file.
723b4c4e15cb Set/updated descriptions and error messages.
Pascal Volk <user@localhost.localdomain.org>
parents: 5
diff changeset
    18
    """
723b4c4e15cb Set/updated descriptions and error messages.
Pascal Volk <user@localhost.localdomain.org>
parents: 5
diff changeset
    19
    __slots__ = ('_doms', '_mxs', '_repo')
0
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
    20
9
3a392067015c Count also naked IPv4 addresses. Beautified NiXSapmSum:RE_PF.
Pascal Volk <user@localhost.localdomain.org>
parents: 8
diff changeset
    21
    RE_FQDN = '(?:[a-z0-9-]{1,63}\.){1,}[a-z]{2,6}'
3a392067015c Count also naked IPv4 addresses. Beautified NiXSapmSum:RE_PF.
Pascal Volk <user@localhost.localdomain.org>
parents: 8
diff changeset
    22
    RE_IPv4 = '(?:[\d]{1,3}\.){3}[\d]{1,3}'
0
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
    23
    """Regular expression pattern for mail logs from Postfix"""
9
3a392067015c Count also naked IPv4 addresses. Beautified NiXSapmSum:RE_PF.
Pascal Volk <user@localhost.localdomain.org>
parents: 8
diff changeset
    24
    RE_PF = r'''^[\w\s:-]{17,80}\spostfix\/smtpd\[[\d]{1,5}\]:\sNOQUEUE:
3a392067015c Count also naked IPv4 addresses. Beautified NiXSapmSum:RE_PF.
Pascal Volk <user@localhost.localdomain.org>
parents: 8
diff changeset
    25
                \sreject:.*blocked\susing\six.dnsbl.manitu.net;
3a392067015c Count also naked IPv4 addresses. Beautified NiXSapmSum:RE_PF.
Pascal Volk <user@localhost.localdomain.org>
parents: 8
diff changeset
    26
                \sSpam\ssent\sto\sthe\smailhost\s(%s|%s)
3a392067015c Count also naked IPv4 addresses. Beautified NiXSapmSum:RE_PF.
Pascal Volk <user@localhost.localdomain.org>
parents: 8
diff changeset
    27
                \swas\sdetected\sby\sNiX\sSpam.*$''' % (RE_FQDN, RE_IPv4)
3a392067015c Count also naked IPv4 addresses. Beautified NiXSapmSum:RE_PF.
Pascal Volk <user@localhost.localdomain.org>
parents: 8
diff changeset
    28
0
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
    29
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
    30
    def __init__(self):
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
    31
        self._doms = {}
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
    32
        self._mxs  = {}
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
    33
        self._repo = None
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
    34
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
    35
    def setLogFormat(self, format='postfix'):
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
    36
        if format == 'postfix':
9
3a392067015c Count also naked IPv4 addresses. Beautified NiXSapmSum:RE_PF.
Pascal Volk <user@localhost.localdomain.org>
parents: 8
diff changeset
    37
            self._repo = re.compile(NiXSapmSum.RE_PF, re.VERBOSE)
0
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
    38
        else:
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
    39
            raise Exception('MTA/Logformat not supported yet.')
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
    40
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
    41
    def parseLog(self, filehandle):
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
    42
        for l in filehandle:
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
    43
            mo = self._repo.match(l)
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
    44
            if mo:
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
    45
                mx = mo.group(1)
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
    46
                try:
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
    47
                    self._mxs[mx] += 1
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
    48
                except KeyError:
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
    49
                    self._mxs[mx]  = 1
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
    50
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
    51
    def countByDom(self):
9
3a392067015c Count also naked IPv4 addresses. Beautified NiXSapmSum:RE_PF.
Pascal Volk <user@localhost.localdomain.org>
parents: 8
diff changeset
    52
        ipv4po = re.compile(NiXSapmSum.RE_IPv4)
0
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
    53
        for mx in self._mxs.keys():
9
3a392067015c Count also naked IPv4 addresses. Beautified NiXSapmSum:RE_PF.
Pascal Volk <user@localhost.localdomain.org>
parents: 8
diff changeset
    54
            mo = ipv4po.match(mx)
3a392067015c Count also naked IPv4 addresses. Beautified NiXSapmSum:RE_PF.
Pascal Volk <user@localhost.localdomain.org>
parents: 8
diff changeset
    55
            if mo:
3a392067015c Count also naked IPv4 addresses. Beautified NiXSapmSum:RE_PF.
Pascal Volk <user@localhost.localdomain.org>
parents: 8
diff changeset
    56
                dom = mo.group(0)
3a392067015c Count also naked IPv4 addresses. Beautified NiXSapmSum:RE_PF.
Pascal Volk <user@localhost.localdomain.org>
parents: 8
diff changeset
    57
            else:
3a392067015c Count also naked IPv4 addresses. Beautified NiXSapmSum:RE_PF.
Pascal Volk <user@localhost.localdomain.org>
parents: 8
diff changeset
    58
                dom = '.'.join(mx.split('.')[-2:])
0
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
    59
            try:
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
    60
                self._doms[dom] += self._mxs[mx]
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
    61
            except KeyError:
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
    62
                self._doms[dom]  = self._mxs[mx]
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
    63
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
    64
    def getDomains(self):
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
    65
        return self._doms
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
    66
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
    67
    def getMXs(self):
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
    68
        return self._mxs
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
    69
8
7f131cf431bc Switched to fileinput module. This allows to read also gzip/bzip2
Pascal Volk <user@localhost.localdomain.org>
parents: 7
diff changeset
    70
0
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
    71
def getOptionParser():
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
    72
    from optparse import OptionParser
6
723b4c4e15cb Set/updated descriptions and error messages.
Pascal Volk <user@localhost.localdomain.org>
parents: 5
diff changeset
    73
    description = 'NiX Spam DNSBL lookup based rejects summarizer'
0
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
    74
    usage  = 'usage: %prog [options] maillog [maillog [...]]'
1
7d5cee19c20a Renamed class NiXSpamPlot to NiXSapmSum. Added getDomLen(). Rewrote showResult()
Pascal Volk <user@localhost.localdomain.org>
parents: 0
diff changeset
    75
    version = '%prog '+__version__
7d5cee19c20a Renamed class NiXSpamPlot to NiXSapmSum. Added getDomLen(). Rewrote showResult()
Pascal Volk <user@localhost.localdomain.org>
parents: 0
diff changeset
    76
    parser = OptionParser(description=description,usage=usage,version=version)
0
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
    77
    parser.add_option('-d', action='store_true', dest='countByDom',
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
    78
            default=False, help='summarize all MX by domain')
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
    79
    parser.add_option('-m', action='store_false', dest='countByDom',
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
    80
            help='count per MX host [default]')
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
    81
    parser.add_option('-o', dest='oFormat', default='table',metavar='FORMAT',
7
83c4798e291d Small code cleanups. Added current year to the copyright notices.
Pascal Volk <user@localhost.localdomain.org>
parents: 6
diff changeset
    82
            choices=('csv', 'table'),
0
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
    83
            help='the output format: table or csv [default: %default]')
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
    84
    parser.add_option('-p', action='store_true', dest='percent', default=False,
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
    85
            help='show also percentages in table output [default: %default]')
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
    86
    parser.add_option('-s', dest='order', default='name', metavar='SORTBY',
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
    87
            help='arrange output by: name or count [default: %default]')
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
    88
    parser.add_option('-t', dest='format', default='postfix',metavar='MTA',
7
83c4798e291d Small code cleanups. Added current year to the copyright notices.
Pascal Volk <user@localhost.localdomain.org>
parents: 6
diff changeset
    89
            choices=('postfix',),
0
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
    90
            help='MTA that generated the maillog [default: %default]')
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
    91
    return parser
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
    92
7
83c4798e291d Small code cleanups. Added current year to the copyright notices.
Pascal Volk <user@localhost.localdomain.org>
parents: 6
diff changeset
    93
8
7f131cf431bc Switched to fileinput module. This allows to read also gzip/bzip2
Pascal Volk <user@localhost.localdomain.org>
parents: 7
diff changeset
    94
def check_files(log_files):
7f131cf431bc Switched to fileinput module. This allows to read also gzip/bzip2
Pascal Volk <user@localhost.localdomain.org>
parents: 7
diff changeset
    95
    """Checks that all files from *log_files* exist and all of them are
7f131cf431bc Switched to fileinput module. This allows to read also gzip/bzip2
Pascal Volk <user@localhost.localdomain.org>
parents: 7
diff changeset
    96
    readable.
7f131cf431bc Switched to fileinput module. This allows to read also gzip/bzip2
Pascal Volk <user@localhost.localdomain.org>
parents: 7
diff changeset
    97
7f131cf431bc Switched to fileinput module. This allows to read also gzip/bzip2
Pascal Volk <user@localhost.localdomain.org>
parents: 7
diff changeset
    98
    If a file doesn't exist or is not readable, it will be removed from
7f131cf431bc Switched to fileinput module. This allows to read also gzip/bzip2
Pascal Volk <user@localhost.localdomain.org>
parents: 7
diff changeset
    99
    *log_files* set.
7f131cf431bc Switched to fileinput module. This allows to read also gzip/bzip2
Pascal Volk <user@localhost.localdomain.org>
parents: 7
diff changeset
   100
7f131cf431bc Switched to fileinput module. This allows to read also gzip/bzip2
Pascal Volk <user@localhost.localdomain.org>
parents: 7
diff changeset
   101
    This function will return *True*, if at least one file has passed the
7f131cf431bc Switched to fileinput module. This allows to read also gzip/bzip2
Pascal Volk <user@localhost.localdomain.org>
parents: 7
diff changeset
   102
    checks. Otherwise *False* will be returned. And the *log_files* set
7f131cf431bc Switched to fileinput module. This allows to read also gzip/bzip2
Pascal Volk <user@localhost.localdomain.org>
parents: 7
diff changeset
   103
    will be emptied.
7f131cf431bc Switched to fileinput module. This allows to read also gzip/bzip2
Pascal Volk <user@localhost.localdomain.org>
parents: 7
diff changeset
   104
7f131cf431bc Switched to fileinput module. This allows to read also gzip/bzip2
Pascal Volk <user@localhost.localdomain.org>
parents: 7
diff changeset
   105
    :param log_files: set of file names
7f131cf431bc Switched to fileinput module. This allows to read also gzip/bzip2
Pascal Volk <user@localhost.localdomain.org>
parents: 7
diff changeset
   106
    :type log_files: set
7f131cf431bc Switched to fileinput module. This allows to read also gzip/bzip2
Pascal Volk <user@localhost.localdomain.org>
parents: 7
diff changeset
   107
    :rtype: bool"""
7f131cf431bc Switched to fileinput module. This allows to read also gzip/bzip2
Pascal Volk <user@localhost.localdomain.org>
parents: 7
diff changeset
   108
    assert isinstance(log_files, set), 'log_files argument must be a set'
7f131cf431bc Switched to fileinput module. This allows to read also gzip/bzip2
Pascal Volk <user@localhost.localdomain.org>
parents: 7
diff changeset
   109
    lf_copy = log_files.copy()
7f131cf431bc Switched to fileinput module. This allows to read also gzip/bzip2
Pascal Volk <user@localhost.localdomain.org>
parents: 7
diff changeset
   110
    for lf in lf_copy:
7f131cf431bc Switched to fileinput module. This allows to read also gzip/bzip2
Pascal Volk <user@localhost.localdomain.org>
parents: 7
diff changeset
   111
        if not os.path.isfile(lf):
7f131cf431bc Switched to fileinput module. This allows to read also gzip/bzip2
Pascal Volk <user@localhost.localdomain.org>
parents: 7
diff changeset
   112
            os.sys.stderr.write('Warning: No such file: %r\n' % lf)
7f131cf431bc Switched to fileinput module. This allows to read also gzip/bzip2
Pascal Volk <user@localhost.localdomain.org>
parents: 7
diff changeset
   113
            log_files.remove(lf)
7f131cf431bc Switched to fileinput module. This allows to read also gzip/bzip2
Pascal Volk <user@localhost.localdomain.org>
parents: 7
diff changeset
   114
        elif not os.access(lf, os.R_OK):
7f131cf431bc Switched to fileinput module. This allows to read also gzip/bzip2
Pascal Volk <user@localhost.localdomain.org>
parents: 7
diff changeset
   115
            os.sys.stderr.write('Warning: Cannot read file: %r\n' % lf)
7f131cf431bc Switched to fileinput module. This allows to read also gzip/bzip2
Pascal Volk <user@localhost.localdomain.org>
parents: 7
diff changeset
   116
            log_files.remove(lf)
7f131cf431bc Switched to fileinput module. This allows to read also gzip/bzip2
Pascal Volk <user@localhost.localdomain.org>
parents: 7
diff changeset
   117
7f131cf431bc Switched to fileinput module. This allows to read also gzip/bzip2
Pascal Volk <user@localhost.localdomain.org>
parents: 7
diff changeset
   118
    if log_files:
7f131cf431bc Switched to fileinput module. This allows to read also gzip/bzip2
Pascal Volk <user@localhost.localdomain.org>
parents: 7
diff changeset
   119
        return True
7f131cf431bc Switched to fileinput module. This allows to read also gzip/bzip2
Pascal Volk <user@localhost.localdomain.org>
parents: 7
diff changeset
   120
    return False
0
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
   121
1
7d5cee19c20a Renamed class NiXSpamPlot to NiXSapmSum. Added getDomLen(). Rewrote showResult()
Pascal Volk <user@localhost.localdomain.org>
parents: 0
diff changeset
   122
2
a72ea07394cc NiXSapmSum.getTotal(): deleted
Pascal Volk <user@localhost.localdomain.org>
parents: 1
diff changeset
   123
def buildTable(output, domains, percent, orderBy):
a72ea07394cc NiXSapmSum.getTotal(): deleted
Pascal Volk <user@localhost.localdomain.org>
parents: 1
diff changeset
   124
    k = 0 if orderBy == 'name' else 1
a72ea07394cc NiXSapmSum.getTotal(): deleted
Pascal Volk <user@localhost.localdomain.org>
parents: 1
diff changeset
   125
    doms = sorted(domains.items(), lambda d,c: cmp(d[k],c[k]), reverse=k)
7
83c4798e291d Small code cleanups. Added current year to the copyright notices.
Pascal Volk <user@localhost.localdomain.org>
parents: 6
diff changeset
   126
    dlen = len(max(domains.iterkeys(), key=len)) + 1
2
a72ea07394cc NiXSapmSum.getTotal(): deleted
Pascal Volk <user@localhost.localdomain.org>
parents: 1
diff changeset
   127
    clen = len(str(max(domains.values())))
a72ea07394cc NiXSapmSum.getTotal(): deleted
Pascal Volk <user@localhost.localdomain.org>
parents: 1
diff changeset
   128
    total = sum(domains.values())
a72ea07394cc NiXSapmSum.getTotal(): deleted
Pascal Volk <user@localhost.localdomain.org>
parents: 1
diff changeset
   129
    if percent:
3
6b0d09cdfbdb buildTable(): small cosmetics/enhancements
Pascal Volk <user@localhost.localdomain.org>
parents: 2
diff changeset
   130
        format = ' %%%ds  %%%dd  %%6.2f %%%%\n' % (-dlen, clen)
2
a72ea07394cc NiXSapmSum.getTotal(): deleted
Pascal Volk <user@localhost.localdomain.org>
parents: 1
diff changeset
   131
        for d, c in doms:
a72ea07394cc NiXSapmSum.getTotal(): deleted
Pascal Volk <user@localhost.localdomain.org>
parents: 1
diff changeset
   132
            dfrac = 100./total*c
a72ea07394cc NiXSapmSum.getTotal(): deleted
Pascal Volk <user@localhost.localdomain.org>
parents: 1
diff changeset
   133
            output.write(format % (d, c, dfrac))
3
6b0d09cdfbdb buildTable(): small cosmetics/enhancements
Pascal Volk <user@localhost.localdomain.org>
parents: 2
diff changeset
   134
        output.write('%s\n' % ((clen+dlen+14)*'-'))
6b0d09cdfbdb buildTable(): small cosmetics/enhancements
Pascal Volk <user@localhost.localdomain.org>
parents: 2
diff changeset
   135
        output.write(format % ('total', total, 100))
2
a72ea07394cc NiXSapmSum.getTotal(): deleted
Pascal Volk <user@localhost.localdomain.org>
parents: 1
diff changeset
   136
    else:
3
6b0d09cdfbdb buildTable(): small cosmetics/enhancements
Pascal Volk <user@localhost.localdomain.org>
parents: 2
diff changeset
   137
        format = ' %%%ds  %%%dd\n' % (-dlen, clen)
2
a72ea07394cc NiXSapmSum.getTotal(): deleted
Pascal Volk <user@localhost.localdomain.org>
parents: 1
diff changeset
   138
        for d in doms:
a72ea07394cc NiXSapmSum.getTotal(): deleted
Pascal Volk <user@localhost.localdomain.org>
parents: 1
diff changeset
   139
            output.write(format % d)
3
6b0d09cdfbdb buildTable(): small cosmetics/enhancements
Pascal Volk <user@localhost.localdomain.org>
parents: 2
diff changeset
   140
        output.write('%s\n' % ((clen+dlen+4)*'-'))
6b0d09cdfbdb buildTable(): small cosmetics/enhancements
Pascal Volk <user@localhost.localdomain.org>
parents: 2
diff changeset
   141
        output.write(format % ('total', total))
2
a72ea07394cc NiXSapmSum.getTotal(): deleted
Pascal Volk <user@localhost.localdomain.org>
parents: 1
diff changeset
   142
1
7d5cee19c20a Renamed class NiXSpamPlot to NiXSapmSum. Added getDomLen(). Rewrote showResult()
Pascal Volk <user@localhost.localdomain.org>
parents: 0
diff changeset
   143
def showResult(nixspamsum, options):
0
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
   144
    if options.countByDom:
1
7d5cee19c20a Renamed class NiXSpamPlot to NiXSapmSum. Added getDomLen(). Rewrote showResult()
Pascal Volk <user@localhost.localdomain.org>
parents: 0
diff changeset
   145
        nixspamsum.countByDom()
7d5cee19c20a Renamed class NiXSpamPlot to NiXSapmSum. Added getDomLen(). Rewrote showResult()
Pascal Volk <user@localhost.localdomain.org>
parents: 0
diff changeset
   146
        domains = nixspamsum.getDomains()
0
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
   147
    else:
1
7d5cee19c20a Renamed class NiXSpamPlot to NiXSapmSum. Added getDomLen(). Rewrote showResult()
Pascal Volk <user@localhost.localdomain.org>
parents: 0
diff changeset
   148
        domains = nixspamsum.getMXs()
7
83c4798e291d Small code cleanups. Added current year to the copyright notices.
Pascal Volk <user@localhost.localdomain.org>
parents: 6
diff changeset
   149
    if not domains:
6
723b4c4e15cb Set/updated descriptions and error messages.
Pascal Volk <user@localhost.localdomain.org>
parents: 5
diff changeset
   150
        print "No NiX Spam DNSBL rejects found"
4
ae58d9881be5 showResult(): count matches; break if no matches were found
Pascal Volk <user@localhost.localdomain.org>
parents: 3
diff changeset
   151
        return
ae58d9881be5 showResult(): count matches; break if no matches were found
Pascal Volk <user@localhost.localdomain.org>
parents: 3
diff changeset
   152
ae58d9881be5 showResult(): count matches; break if no matches were found
Pascal Volk <user@localhost.localdomain.org>
parents: 3
diff changeset
   153
    from cStringIO import StringIO
ae58d9881be5 showResult(): count matches; break if no matches were found
Pascal Volk <user@localhost.localdomain.org>
parents: 3
diff changeset
   154
    output = StringIO()
2
a72ea07394cc NiXSapmSum.getTotal(): deleted
Pascal Volk <user@localhost.localdomain.org>
parents: 1
diff changeset
   155
    # build the table
1
7d5cee19c20a Renamed class NiXSpamPlot to NiXSapmSum. Added getDomLen(). Rewrote showResult()
Pascal Volk <user@localhost.localdomain.org>
parents: 0
diff changeset
   156
    if options.oFormat == 'table':
2
a72ea07394cc NiXSapmSum.getTotal(): deleted
Pascal Volk <user@localhost.localdomain.org>
parents: 1
diff changeset
   157
        buildTable(output, domains, options.percent, options.order)
a72ea07394cc NiXSapmSum.getTotal(): deleted
Pascal Volk <user@localhost.localdomain.org>
parents: 1
diff changeset
   158
    # generate comma separated values
1
7d5cee19c20a Renamed class NiXSpamPlot to NiXSapmSum. Added getDomLen(). Rewrote showResult()
Pascal Volk <user@localhost.localdomain.org>
parents: 0
diff changeset
   159
    elif options.oFormat == 'csv':
4
ae58d9881be5 showResult(): count matches; break if no matches were found
Pascal Volk <user@localhost.localdomain.org>
parents: 3
diff changeset
   160
        order = domains.keys()
ae58d9881be5 showResult(): count matches; break if no matches were found
Pascal Volk <user@localhost.localdomain.org>
parents: 3
diff changeset
   161
        order.sort()
ae58d9881be5 showResult(): count matches; break if no matches were found
Pascal Volk <user@localhost.localdomain.org>
parents: 3
diff changeset
   162
        for d in order:
ae58d9881be5 showResult(): count matches; break if no matches were found
Pascal Volk <user@localhost.localdomain.org>
parents: 3
diff changeset
   163
            output.write("'%s',%d\n" % (d, domains[d]))
2
a72ea07394cc NiXSapmSum.getTotal(): deleted
Pascal Volk <user@localhost.localdomain.org>
parents: 1
diff changeset
   164
    # should never be reached
1
7d5cee19c20a Renamed class NiXSpamPlot to NiXSapmSum. Added getDomLen(). Rewrote showResult()
Pascal Volk <user@localhost.localdomain.org>
parents: 0
diff changeset
   165
    else:
2
a72ea07394cc NiXSapmSum.getTotal(): deleted
Pascal Volk <user@localhost.localdomain.org>
parents: 1
diff changeset
   166
        print "Oops, error in function showResult() happend"
a72ea07394cc NiXSapmSum.getTotal(): deleted
Pascal Volk <user@localhost.localdomain.org>
parents: 1
diff changeset
   167
    # show the result
a72ea07394cc NiXSapmSum.getTotal(): deleted
Pascal Volk <user@localhost.localdomain.org>
parents: 1
diff changeset
   168
    print output.getvalue()
0
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
   169
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
   170
def main():
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
   171
    parser = getOptionParser()
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
   172
    opts, args = parser.parse_args()
8
7f131cf431bc Switched to fileinput module. This allows to read also gzip/bzip2
Pascal Volk <user@localhost.localdomain.org>
parents: 7
diff changeset
   173
    if not args:
6
723b4c4e15cb Set/updated descriptions and error messages.
Pascal Volk <user@localhost.localdomain.org>
parents: 5
diff changeset
   174
        parser.error('No log file specified')
8
7f131cf431bc Switched to fileinput module. This allows to read also gzip/bzip2
Pascal Volk <user@localhost.localdomain.org>
parents: 7
diff changeset
   175
    # drop duplicates
7f131cf431bc Switched to fileinput module. This allows to read also gzip/bzip2
Pascal Volk <user@localhost.localdomain.org>
parents: 7
diff changeset
   176
    log_files = set(args)
7f131cf431bc Switched to fileinput module. This allows to read also gzip/bzip2
Pascal Volk <user@localhost.localdomain.org>
parents: 7
diff changeset
   177
    # remove inexistent/unreadable files
7f131cf431bc Switched to fileinput module. This allows to read also gzip/bzip2
Pascal Volk <user@localhost.localdomain.org>
parents: 7
diff changeset
   178
    if not check_files(log_files):
7f131cf431bc Switched to fileinput module. This allows to read also gzip/bzip2
Pascal Volk <user@localhost.localdomain.org>
parents: 7
diff changeset
   179
        os.sys.stderr.write('No readable log files found\n')
7f131cf431bc Switched to fileinput module. This allows to read also gzip/bzip2
Pascal Volk <user@localhost.localdomain.org>
parents: 7
diff changeset
   180
        return 1
7f131cf431bc Switched to fileinput module. This allows to read also gzip/bzip2
Pascal Volk <user@localhost.localdomain.org>
parents: 7
diff changeset
   181
1
7d5cee19c20a Renamed class NiXSpamPlot to NiXSapmSum. Added getDomLen(). Rewrote showResult()
Pascal Volk <user@localhost.localdomain.org>
parents: 0
diff changeset
   182
    nixss = NiXSapmSum()
7d5cee19c20a Renamed class NiXSpamPlot to NiXSapmSum. Added getDomLen(). Rewrote showResult()
Pascal Volk <user@localhost.localdomain.org>
parents: 0
diff changeset
   183
    nixss.setLogFormat(opts.format)
8
7f131cf431bc Switched to fileinput module. This allows to read also gzip/bzip2
Pascal Volk <user@localhost.localdomain.org>
parents: 7
diff changeset
   184
7f131cf431bc Switched to fileinput module. This allows to read also gzip/bzip2
Pascal Volk <user@localhost.localdomain.org>
parents: 7
diff changeset
   185
    fi = fileinput.FileInput(log_files, openhook=fileinput.hook_compressed)
7f131cf431bc Switched to fileinput module. This allows to read also gzip/bzip2
Pascal Volk <user@localhost.localdomain.org>
parents: 7
diff changeset
   186
    nixss.parseLog(fi)
7f131cf431bc Switched to fileinput module. This allows to read also gzip/bzip2
Pascal Volk <user@localhost.localdomain.org>
parents: 7
diff changeset
   187
    fi.close()
1
7d5cee19c20a Renamed class NiXSpamPlot to NiXSapmSum. Added getDomLen(). Rewrote showResult()
Pascal Volk <user@localhost.localdomain.org>
parents: 0
diff changeset
   188
    showResult(nixss, opts)
8
7f131cf431bc Switched to fileinput module. This allows to read also gzip/bzip2
Pascal Volk <user@localhost.localdomain.org>
parents: 7
diff changeset
   189
    return 0
0
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
   190
2d97e75f16cf initial commit: »don't fear the nervous delete finger«
Pascal Volk <user@localhost.localdomain.org>
parents:
diff changeset
   191
if __name__ == '__main__':
8
7f131cf431bc Switched to fileinput module. This allows to read also gzip/bzip2
Pascal Volk <user@localhost.localdomain.org>
parents: 7
diff changeset
   192
    os.sys.exit(main())