pflogsumm-1.1.13/0000755000175000017500000000000015076454367014143 5ustar jseymourjseymourpflogsumm-1.1.13/pflogsumm-faq.txt0000644000175000017500000007247415013372212017453 0ustar jseymourjseymour FAQ for Pflogsumm - A Log Summarizer/Analyzer for the Postfix MTA Introduction I wouldn't have believed it. What started out mostly as a light- hearted exercise in improving my facility with Perl--with the hope that something useful would come out of it as well--has turned out to be a somewhat popular utility. And as more Admins find out about postfix, and more end up trying pflogsumm, many of the questions, suggestions, and enhancement requests are becoming "frequently asked". So odd as it seems (to me, at any rate), it looks like it's time for a FAQ. Index of pflogsumm Frequently Asked Questions (in no particular order) 1. Project Status 2. "Could You Make" or "Here's A Patch To Make" Pflogsumm Do ... 3. Requires Date:Calc Module 4. Built-In Support for Compressed Logs 5. Processing Multiple Log Files 6. Time-Based Reporting and Statistics 7. By-domain Listings 8. Reject, Deferred and Bounced Detail Info 9. "Orphaned" (no size) Messages 10. Pflogsumm misses/mis-diagnoses/mis-reports, etc. 11. Pflogsumm is generating lots of "uninitialized value" warnings 12. Pflogsumm just doesn't work or doesn't report anything 13. Postfix Rejects Pflogsumm Reports Because Of Body Checks 14. Pflogsumm Reports Double Traffic When Anti-Virus Scanner Used 15. Pflogsumm's numbers don't add up 16. Hourly stats for reports run without "-d" option are halved 17. How Do I Get Pflogsumm To Email Reports To Me Daily/Weekly/etc.? 18. How Can I View Pflogsumm's Reports In My Web Browser? 19. New Red Hat install - Pflogsumm no longer works! 20. How can I best format my custom reject messages for display in Pflogsumm's output? 21. Pflogsumm doesn't understand my log file format 22. Why Isn't There Any Mention Of "Monkey Butler" In The FAQ? 23. Translating Pflogsumm (Support for Internationalization) 24. Pflogsumm may sometimes calculate "yesterday" incorrectly 25. Sending Logfile Samples 26. From Where Can I Obtain Pflogsumm? 1. Project Status New work on Pflogsumm is sporadic. It pretty much does everything I need it to do and, so far as I can tell, pretty much what most other people need it to do. And my time is limited. I'll still take bug reports. I'll still fix bugs. (But I promise no time-line.) I'll still answer questions (as time allows). And I *may* add the occasional enhancement or whatever--as the mood strikes--but Pflogsumm is pretty much a "finished work" as far as I'm concerned. 2. "Could You Make" or "Here's A Patch To Make" Pflogsumm Do ... Unless it's a *bug* fix, please see: "1. Project Status" To the argument "But it's a patch, all you have to do is...," the answer is: "Not quite." Every time I make a change to Pflogsumm I have to run it through a series of regression checks to make sure the change didn't break something. Then there's the commit, documentation, web page update, etc. cycle. I'm particularly unlikely to add code to Pflogsumm to account for non-standard Postfix log entries. "Non-standard" being defined as "other than what Wietse's code does." Or additional stats gathering that nobody else has requested and strikes *me* as of limited interest or use. In addition to the development cycle, there's the issue of "code bloat." Pflogsumm already takes enough (too much?) time and memory on busy machines with large logs. I'm not prone to make this worse for the sake of these things. See Also: 21. Pflogsumm doesn't understand my log file format 3. Requires Date::Calc Module Pflogsumm requires the Date::Calc module. You can download and install the Date::Calc module from CPAN. It can be found at: http://search.cpan.org/search?module=Date::Calc Or you can remove the code that's dependent on the Date::Calc module. For the convenience of folks that would prefer to take this approach, I've "fenced" all such code like this: # ---Begin: SMTPD_STATS_SUPPORT--- . . . . . . # ---End: SMTPD_STATS_SUPPORT--- However, if you do this you will lose support for --smtpd-stats. Later versions of the Pflogsumm distribution include a script to semi-automate removing smtpd stats support, if you so-desire. As of Pflogsumm-1.1.1, the presence of Date::Calc is optional. If you don't want to use the Pflogsumm options that depend upon it, you neither need Date::Calc, nor is it necessary to manually remove the code that depends upon it. 4. Built-In Support for Compressed Logs I took a look at this. There is a Perl module (which I downloaded, built, and installed here) to interface to libz, but after considering the changes that would be necessary--and the fact that those changes would require that potential users have to download/build/install libz (and of the correct version) and the additional Perl module, I decided to forego this enhancement. I could just open a pipe within Pflogsumm and use zcat/gunzip/gzip. That would depend upon a) them being there [probably a safe bet-- considering the logs somehow got into that format :-), but...] and b) one of these either being in the path or having an environment variable or a script variable or... The thing is, in the latter case there's really no "savings" over simply piping into Pflogsumm in the first place. Multiple processes get spawned, pipes opened, etc. either way. It would add a little convenience, is all. So I could do it. And there are a couple of ways I could do it. And my mind is certainly still open on the issue. I'm just not convinced there's a good reason to do it, is all. And I'd like to avoid "creeping over-feature-itis" if I can. My position is *not* set in stone on this issue. In the mean-time: zcat /var/log/maillog.0.gz |pflogsumm or gunzip should do the trick quite nicely for you. If you've a complex situation, for example: your logs aren't rotated exactly at midnight, you might try something like: (zcat /var/log/maillog.0.gz; cat /var/log/maillog) \ |pflogsumm -d yesterday See Also: 5. Processing Multiple Log Files 17. How Do I Get Pflogsumm To Email Reports To Me Daily/Weekly/etc.? 5. Processing Multiple Log Files When processing multiple log files (say: an entire weeks worth of logs that are rotated daily), it is important that Pflogsumm be fed them in chronological order. For performance and memory conservation reasons, Pflogsumm relies on log messages "arriving" in the order in which they were created. If you do something like this: pflogsumm /var/log/maillog* you might not get what you expect! Instead, try something like: pflogsumm `ls -rt /var/log/maillog*` A more complex example, where compressed logs are involved: (zcat `ls -rt /var/log/maillog.*.gz`; cat /var/log/maillog) \ |pflogsumm Obviously, this depends on the file modification times for your logs being reflective of their chronological order. If that can't be trusted, you're gonna have to get ugly. Like in enumerating each file, or as in: (for each in 3 2 1 0; do zcat "/var/log/maillog.$each.gz" done cat /var/log/maillog) |pflogsumm or (somewhat more efficiently--by running zcat only once): (zcat `for ea in 3 2 1 0; do echo "/var/log/maillog.$ea.gz"; done`; cat /var/log/maillog) |pflogsumm [Note: I didn't actually run these. So you would be well-advised to double-check them.] See Also: 4. Built-In Support for Compressed Logs 17. How Do I Get Pflogsumm To Email Reports To Me Daily/Weekly/etc.? 6. Time-Based Reporting and Statistics There has been a small assortment of requests for different time statistics reporting. And adding this would be relatively straight- forward. (Just have to reach a consensus on exactly *what* should be reported, and how. This could easily get out of hand!) There's only one *small* problem. Ironically, it's time. I've experimented with Pflogsumm grokking the log timestamps. As a matter-of-fact: the enhancement added in the 19990110-05 version required that I do some of this. My first pass was to use the Perl timelocal() function to convert those sub-strings to an integer for subsequent comparison operations. Imagine my surprise when performance of the resulting code was a factor of five (5) times slower than that of its predecessor. After a "remove the statements until it got fast again" exercise, I found that the culprit was timelocal(). As of version 19990321-01, Pflogsumm does by-domain stats reporting of average and maximum delivery time by host/domain. And an even earlier version added by-hour and by-day message count reporting. Anything much beyond these is going to get "expensive." If/when any additional time-based stats reporting is added: I think they are definitely going to be optional. One way you can make up for Pflogsumm's deficiency in this respect is to use good ol' Unix tools like "grep" to pre-process your log files before feeding them to Pflogsumm. E.g.: grep "Feb 9" /var/log/maillog |pflogsumm what_ever_args Note that single-digit days-of-the-month have an additional leading space in the logfiles, where the digit for two-digit dates would be. 7. By-domain Listings I figured on the desire for this one from the start. There are many possibilities: 1) A single report, split by domain 2) An option to limit reporting to a particular domain This issue is kind of tricky. The popularity of Unix amongst SysAdmins is testimony to the beauty of being able to wire- together small, simple tools so that one can generate output to ones taste. Anything I do is likely to make some Admins happy and others wishing I'd done it "the other way". One thought that occurred is to perhaps provide a couple of options that would allow one to limit a particular report to sender=regular_expression and/or recipient=regular_expression The problem with this solution is that an Admin desiring to emit custom reports for multiple domains would have to re-process the same log multiple times--once for each desired domain. So I'm still thinking about this one. 8. Reject, Deferred and Bounced Detail Info I've actually only received one query about this so far, but there are bound to be more. So... The "detailed" information in the "Reject", "Deferred" and "Bounced" reports is a compromise. Just take a stroll through your postfix logs some day and observe the variation in how the "reason" for a particular reject, defer, or bounce is reported. Without putting a lot of static comparisons for each-and-every case into the analyzer, I have absolutely no hope is doing this very well. Emitting the entire "reason" is not good, either. The entire "reason" string can be very long. Depending on what somebody is using to display Pflogsumm's output, the lines may well wrap-- producing output that is no more readable than just grepping the logs. And anything more I do to this end may soon be rendered moot. After Wietse gets most of the more important functional stuff out of the way, Postfix logging is going to be completely re-written. (Oh boy, won't that be fun!) I'm hoping I'll be able to get some input into the process so the formatting is more amenable to automated processing. Wietse has indicated that such would be the case. Also, please note my primary objective behind Pflogsumm (besides the entertainment value): "just enough detail to give the administrator a ``heads up'' for potential trouble spots." It's not *supposed* to do away with manual inspection entirely. For those that really want all that extra detail in the log summary reports, specify the "--verbose-msg-detail" switch. See Also: 25. Sending Logfile Samples 9. "Orphaned" (no size) Messages The Problem: Message size is reported only by the queue manager. The message may be delivered long-enough after the (last) qmgr log entry that the information is not in the log(s) processed by a particular run of pflogsumm. The Result: "Orphaned" messages. These are reported by Pflogsumm as "Messages with no size data." This, of course, throws off "Recipients by message size" and the total for "bytes delivered." ("bytes in messages" in earlier versions.) The Solution: "Memorize" message sizes by queue i.d. Easy in theory. Difficult in practice. At least at the moment. You see, if Pflogsumm's going to "memorize" message sizes, it has to have some definitive way to know when to delete a no- longer-needed reference. Otherwise the memory file will just grow forever. As with the "Reject, Deferred and Bounced Detail Info" issue above, I'm hoping the get some input into future changes in logging issues. In any event: maybe whatever comes out of the logging redesign will provide a solution. As of Pflogsumm version 1.0.12, the "Messages with no size data" report can be turned off. 10. Pflogsumm misses/mis-diagnoses/mis-reports, etc. Are you using a real old version of VMailer? As of pflogsumm version 19990220-06, versions of VMailer prior to 19981023 are no longer supported. Sorry. Pflogsumm-19990121-01.pl will be made permanently available from now on for those with out-of-date versions of VMailer prior to 19981023. Are you processing your log files in chronological order? See item "5: "Processing Multiple Log Files". Pflogsumm is being developed by me on my rather small-scale server at home. There are only two users on the system. And I do no mail-forwarding. So the log samples I have to work with are commensurately limited. If there's something that Pflogsumm is not doing, or not doing right, let me know what it is, what you think it ought to do, and send me a representative sample of *real* log entries with which to work. See Also: 5. Processing Multiple Log Files 12. Pflogsumm just doesn't work or doesn't report anything 15. Pflogsumm's numbers don't add up 19. New Red Hat install - Pflogsumm no longer works! 21. Pflogsumm doesn't understand my log file format 25. Sending Logfile Samples 11. Pflogsumm is generating lots of "uninitialized value" warnings Are you using a version of Perl lower than 5.004_04? Perhaps with a "beta" version of pflogsumm? If so, try turning off the "-w" switch. Pflogsumm as of 19990413-02beta appeared to work correctly with Perl 5.003 in spite of the warnings. (Those warnings didn't appear with Perl 5.004.) I don't guarantee that I'll remember to test future versions of pflogsumm against 5.003, but I'll try to :-). You really should consider upgrading your Perl to 5.004 or later. 12. Pflogsumm just doesn't work or doesn't report anything Did you *download* Pflogsumm as opposed to grabbing it by "copy-and-paste" from a browser? Copy-and-paste can result in lines being unintentionally wrapped and hard-tabs being converted to spaces. This will break Pflogsumm. Also, I've received a couple of reports by people downloading Pflogsumm with Lynx that the download has long lines wrapped. Naturally, this breaks Pflogsumm. See Also: 10. Pflogsumm misses/mis-diagnoses/mis-reports, etc. 19. New Red Hat install - Pflogsumm no longer works! 21. Pflogsumm doesn't understand my log file format 13. Postfix Rejects Pflogsumm Reports Because Of Body Checks You configure Postfix to do body checks, Postfix does its thing, Pflogsumm reports it and Postfix catches the the same string in the Pflogsumm report. There are several solutions to this. Wolfgang Zeikat contributed this: #!/usr/bin/perl use MIME::Lite; ### Create a new message: $msg = MIME::Lite->new( From => 'your@send.er', To => 'your@recipie.nt', # Cc => 'some@other.com, some@more.com', Subject => 'pflogsumm', Date => `date`, Type => 'text/plain', Encoding => 'base64', Path =>'/tmp/pflogg', ); $msg->send; Where "/tmp/pflogg" is the output of Pflogsumm. This puts Pflogsumm's output in a base64 MIME attachment. In a follow-up to a thread in the postfix-users mailing list, Ralf Hildebrandt noted: "mpack does the same thing." The canonical FTP site for mpack is ftp.andrew.cmu.edu:pub/mpack/ The solution I came up with is to modify the body_checks statements to ignore the strings when they're in a Pflogsumm report, as follows: Bounce anything with 6 or more "$"s in a row... /\${6,}/ REJECT Which, of course, catches the line in the Pflogsumm report too. So... /^(?!\s+[0-9]+\s+).*?\${6,}/ REJECT which reads "anything with 6 or more '$'s in a row that is not a line beginning with one or more whitespace characters, followed by one or more digits, followed by one or more whitespace characters." (This is using PCRE's, btw.) Note that my solution will be more computationally expensive, by a *long* way, than encoding Pflogsumm's output into a format that body_checks won't catch. Robert L Mathews suggested the following solution /^ {6,11}[[:digit:]]{1,6}[ km] / OK Placed at the beginning of a body_checks file, this will "pre-approve" lines in Pflogsumm's output that might otherwise get caught. That's a POSIX regexp version. A PCRE version of the same thing would be: /^ {6,11}\d{1,6}[ km] / OK 14. Pflogsumm Reports Double Traffic When Anti-Virus Scanner Used Sadly, there's absolutely nothing I can do about this :-(. The problem arises because of the way in which anti-virus scanning is handled by Postfix. Basically, Postfix "delivers" each email to the anti-virus scanner and the anti-virus scanner re-sends it through Postfix. So each email really is received twice and sent/delivered twice. And yes, I tried. I really, really tried. If I recall correctly, I spent some two days mucking-about with this problem. Actually thought I had it once or twice. But the results inevitably failed regression testing. At the end of this, and with some more careful thought, I realized it just wasn't possible. If you think you can prove me wrong, please do so. I'd be quite pleased to be proven wrong on this one. johnfawcett at tiscali-dot-it believes he's done it. You may find prefiltering your log with his "prepflog" does it for you. You can find it at . Note: Because of the way John's pre-processing script works, which, given my own experiments, is probably the way it *has* to work to work correctly, integrating his code into Pflogsumm would be difficult, at best, if it's even possible within Pflogsumm's current structure. 15. Pflogsumm's numbers don't add up Pflogsumm reports more "delivered" than "received" Naturally. A single email message can have multiple recipients. Pflogsumm reports more "rejected" than "received" Why doesn't delivered + deferred + bounce + rejected = received? Some rejects (header and body checks, for example) happen in "cleanup," after alias lists are expanded. Thus a single received message will be rejected multiple times: once for each recipient. The "size=" fields, multiplied by their "nrcpt=" fields, when added-up yields a total higher than Pflogsumm's "bytes delivered" total. Pflogsumm doesn't count something delivered until it actually *is* delivered. Nrcpt only suggests the number of intended recipients, not how many are actually deliverable. Only if there were no bounces, rejects, defers or other undeliverables for everything that was received would a calculation such as that above yield the proper value. Pflogsumm's "% rejected" doesn't add up The "percent rejected" and "percent discarded" figures are only approximations. They are calculated as follows (example is for "percent rejected"): percent rejected = (rejected / (delivered + rejected + discarded)) * 100 Given the issues discussed above, this is really the best that can be hoped-for, IMO. I consistently see more "delivered" than "received." How is that possible? Any message that's got multiple recipients in the "To:," "Cc:," and "Bcc:" fields will result in a single "received" with multiple "delivered"s, as well as, possibly, multiple "rejects" (or reject warnings, discards or holds), depending on where in Postfix' processing the rule was that resulted in the reject, etc. See Also: 10. Pflogsumm misses/mis-diagnoses/mis-reports, etc. 16. Hourly stats for reports run without "-d" option are halved Scenario: On day #1 of a fresh logfile, you run Pflogsumm with "-d today" and the next day you run it with no "-d" option. The "Per-Hour Traffic" statistics are approximately halved. How can this be? Note that when you run Pflogsumm on a logfile that contains multi-day logfile entries, Pflogsumm automatically changes the per-hour stats to daily traffic averages. If there's even *one* logfile entry from another day, all of the per-hour stats will be divided by two. Unless you rotate logfiles *precisely* at midnight--and it's unlikely you can guarantee that happening--there's no way to prevent this. 17. How Do I Get Pflogsumm To Email Reports To Me Daily/Weekly/etc.? Excuse me? You're running a mailserver and you don't know how to use cron to run things on a scheduled basis and pipe the output to something that'll email it to you? Oh. My. Lord. *sigh* Here's my crontab entries: 10 0 * * * /usr/local/sbin/pflogsumm -d yesterday /var/log/syslog \ 2>&1 |/usr/bin/mailx -s "`uname -n` daily mail stats" postmaster 10 4 * * 0 /usr/local/sbin/pflogsumm /var/log/syslog.0 \ 2>&1 |/usr/bin/mailx -s "`uname -n` weekly mail stats" postmaster (Those are actually each a single line. I line-broke them [and signified that with the "\"s] for readability.) The first generates stats for the previous day and is run *after* midnight. The second is run against the previous week's entire log. (I rotate my logs weekly.) If you rotate your logs on a different schedule, want monthly reports, etc., I leave it as an exercise to you, the reader, to figure out how to concatenate several logs to stdout and feed that to Pflogsumm. See Also: 4. Built-In Support for Compressed Logs 5. Processing Multiple Log Files The Unix manual pages for "cron," "crontab," "cat," "zcat," "gzip," "gunzip," "mail," "mailx," etc. 18. How Can I View Pflogsumm's Reports In My Web Browser? Just direct Pflogsumm's output to a file, call it "something.txt" or whatever, and look at it with your browser :). If you want to get fancy, create a post-processing shell script that'll create a date-tagged file, like "mailstats-20030216.txt". It's easy. See Also: Pflogsumm Through A Browser, on Pflogsumm's home page. 19. New Red Hat install - Pflogsumm no longer works! From some email exchanges with a couple of people that reported this... "It appears the Pflogsumm is broken with RedHat9. I can take the same log file and run it under Solaris9/RedHat 7.3 (perl 5.8 on both) without a problem, but it breaks on RH9." "Oops. Sorry about the false alarm. This is an issue with some of the other Perl scripts that are out there due to Red Hat 8/9 using LANG=en_US.UTF-8 Changing the locale to "POSIX" fixes this... LANG=C Note that Pflogsumm works fine when run through cron.daily, as cron has different environment settings." "Ah, the good old RH8/9 UTF-8 strikes again. I should have known. Setting LANG to either en_US or C fixes the problem." What the above means is that you have to change the "LANG" environment variable from "en_US.UTF-8" to "en_US" or "C". E.g.: LANG="en_US" export LANG in your shell. Or you could add these commands to your login profile. (I.e.: $HOME/.bash_profile, if you're using bash.) Or set the system-wide default in /etc/sysconfig/i18n. My RH boxes have LANG set to "en_US" there, and everything seems to work fine. (If you set it in your profile or the system-wide default, you'll need a fresh login for it to take effect, obviously.) See Also: 10. Pflogsumm misses/mis-diagnoses/mis-reports, etc. 12. Pflogsumm just doesn't work or doesn't report anything 21. Pflogsumm doesn't understand my log file format 20. How can I best format my custom reject messages for display in Pflogsumm's output? Reject reason strings found in the mail log will be truncated at the first comma (","), colon (":") or semi-colon (";"). If you want a "clause" in your reject message to appear in Pflogsumm's output, without having to specify --verbose-msg-detail, use a punctuation mark other than one of those three, such as a dash ("-"). 21. Pflogsumm doesn't understand my log file format I've received several requests to modify Pflogsumm's log file format regular expression matching to accommodate "non-standard" log file formats. I'm not inclined to honour such requests. The regexp that identifies Postfix' log file entries is nearly incomprehensible as it is. If your log file format has extra fields (e.g.: FreeBSD syslogd with "-v -v" specified), or, as in one case (metalog), is lacking fields, and you insist on doing things that way, I recommend you code-up a little pre-filter to mung the format into a standard one. See Also: 10. Pflogsumm misses/mis-diagnoses/mis-reports, etc. 12. Pflogsumm just doesn't work or doesn't report anything 19. New Red Hat install - Pflogsumm no longer works! 22. Why Isn't There Any Mention Of "Monkey Butler" In The FAQ? A friend of mine asked me if I'd put the phrase "monkey butler" in the FAQ. The answer is no. Pflogsumm is used by some rather large corporations. There are credibility issues. Sorry. :) 23. Translating Pflogsumm (Support for Internationalization) Unfortunately, Pflogsumm doesn't currently have i18n support. It wasn't until at least Perl 5.6 that i18n was included as part of the base distribution. Since, last time I looked, 5.005* was still the most widely-used version of Perl (that's what I'm still running everywhere, too), I can't put i18n in without chancing breaking things right-and-left for the majority of my customers. Even with Perl 5.6 and above, it was mentioned in postfix-users, by Liviu Daia, that "Perl 5.6+ has locales. Locales can give you localized dates, charsets, standard error messages etc., but it won't automatically switch languages of the strings defined in your program. For that, you still need gettext or something equivalent." So I'm not clear on the future of i18n support in Pflogsumm. But I'm keeping an eye on things. Proper i18n support has long been one of the top things on my own wish list! Prospective translators are urged to translate *only* the stable/production versions. Beta and Alpha versions can sometimes change rapidly. If you do translate Pflogsumm, let me know and I'll put a link to it on Pflogsumm's main web page. 24. Pflogsumm may sometimes calculate "yesterday" incorrectly As Wieland Chmielewski aptly noted: Subroutine get_datestr incorrectly assumes that each day of the year comprises 24 hours. In those countries which participate in Daylight Saving Time policy there is one day with 23 hours and one day with 25 hours. So, chances are (for 1 hour within those days) that get_datestr actually returns either "the day before yesterday" or "today" instead of "yesterday" as requested. Right you are, Wieland, and thanks for the catch. Problem is, of course, there's really no clean, easy, certain fix. The work-around is to stay well clear of DST never-never land with your cron jobs. 25. Sending Logfile Samples Here's the deal with whatever you may send me in the way of log samples: . Obfuscate them if you want. But take care not alter them in such a manner that they're not accurate wrt the "realism" of the data, make sure the field formatting is not altered, and that the order of the log entries is not altered. . The world is an unsafe place for your data, no matter where it might reside. But I'll do my level best to ensure that your data does not fall into the hands of others. . If you want, I'll PGP-encrypt the data when it's not in use. . You can PGP-encrypt it when you send it to me if you're concerned. My PGP public key can be found on my Web site and at the PGP public key servers. . If you want, I'll delete the sample data when the work is done. But I would *like* to keep it around for future regression-testing. It's your call. Let me know. 26. From Where Can I Obtain Pflogsumm? http://jimsun.LinxNet.com/postfix_contrib.html Created: 15 Feb., 1999 / Last updated: 22 March, 2010 pflogsumm-1.1.13/ChangeLog0000644000175000017500000011112415076453663015713 0ustar jseymourjseymourChangeLog for pflogsumm [Notes: Let me know if you would like to be notified as new versions are released. The latest released version can always be found at http://jimsun.LinxNet.com/postfix_contrib.html. As of 2025-07-30 the pflogsumm project page, above, supports an Atom feed for update notifications.] rel-1.1.13 20251023 Bug Fixes Corrected client-info extraction which sometimes extracted the wrong bits from log lines. (Bug introduced in v1.1.12.) Restored --zero-fill option accidentally dropped in v1.1.12 Cleanup/Maintenance Removed redundant parentheses from several regex capture assignments, improving readability without altering behavior. Simplified IPv4 octet parsing expression. Simplified reject/hold/discard capture assignment. Normalized unparsed deferral handling: now labeled as `(unknown host)` instead of a fabricated sentence. Strip trailing port suffixes from hostnames/IP addresses for cleaner aggregation. Improved domain normalization in all summaries for better host/domain aggregation. rel-1.1.12 20250819 *** Breaking Changes *** Date::Calc now Required due to date(-range) enhancements. UI Changes Options using underscores have been restored (reversing removal in v1.1.8). Their use now results in a prominent "deprecated" message in the report. Support for underscores will be removed after a suitable deprecation period. (N.B: This was accomplished without re-introducing Bugzilla bug 1931403.) Renamed option --unprocd to --unprocd-file for better clarity. (--unprocd was introduced in v1.1.11.) Changed --pscrn-detail option to behave like all other report-limiting options: Full detail unless this option is specified to limit or suppress it. (--pscrn-detail was introduced in v1.1.11.) Added support for a config file via --config. Command-line arguments override config file for non-boolean options. Note: This option requires the Config::Simple Perl module. (Only required if --config is used.) Added long-form options --date-range, --extended-detail, --host-cnt, --quiet, and --user-cnt to facilitate more user- friendly configuration files. Short-form options are retained for command-line use. Added --dump-config as a configuration creation/debugging aid. Expanded -d/--date-range options to include today, yesterday, this/last week/month, and specific date and date ranges in ISO 8601/RFC 3339 format. N.B.: Sadly, the extended date range options come with an unavoidable processing performance penalty. This is because the only reasonable way to do it was to convert everything into decimal values. Added --dow0mon (day of week 0 is Monday) for use in conjunction with this/last week date ranges. (Default is Sunday.) Added support for log entries with RFC 5424/3164-style fields and optional syslog version (e.g., "<123>1 ..."). Reports now display a date range at the top when processing multi-day or multi-date logs. Set default output column width to 80 columns, consistent with documentation. Squashed another, hopefully the last, bug that would sometimes lead to inaccurate tallying of messages received. Belated note: --rej-add-from/--rej-add-to always include "<" and ">", respectively, since v1.1.7. rel-1.1.11 20250530 Now requires Perl v5.10.0 minimum. Added support for postscreen summary and detail data: --pscrn-stats - display postscreen summary stats --pscrn-detail [cnt] - emit detailed postscreen data, optionally to the top [cnt] events. Improved host/domain/IP address normalization. Now normalizes IPv6 addresses. Bugfix: Potential undefined variable condition in get_smh. (Derives seconds, minutes, and hours from seconds.) Cosmetic: Fixed displayed single-digit day-of-month in report heading when "-d " specified. (Broken in v1.1.6) Added day-of-week to report heading when "-d " specified. Now always displays "Postfix Log Summaries" heading. Minor code optimizations to employ defined-or ("//") and reduce from List::Util. Added -x (debug) option. Debugging emitted to STDERR Added --unprocd option. Unprocessed log lines written to specified filename. rel-1.1.10 20250529 Bugfix: Messages rejected in latter SMTP processing, after they'd already been assigned a queue i.d., were incorrectly counted as received. Bugfix: postscreen reject detail processing was inadvertently hobbled in 1.1.7 as a side-effect of streamlined host/domain/ip-address parsing. Fixed. rel-1.1.9 20250527 Bugfix: Messages rejected in cleanup were incorrectly counted as received. Bugfix: In the unlikely (?) event a qid was used more than once in the same logfile input only the size of the message with the first occurrence of that qid would be counted. Bugfix: Eliminated errant debugging message that escaped my attention in versions 1.1.7 and 1.1.8. Documentation bugfix: "Addresses" misspelled as "adresses." Thanks and a tip o' the hat to Sven Hoexter (sven-at-stormbind-dot-net) for the heads up. rel-1.1.8 20250525 Removed the following deprecated options --no_bounce_detail --no_deferral_detail --no_reject_detail --no_smtpd_warnings pflogsumm no longer accepts underscores ("_") as part of option names. (Note: this finally addresses four-year-old Bugzilla bug 1931403.) Code cleanup: Changed some variable constants to true constants. Added test for reject type "BDAT" (Thanks to Maxim, admin-at-modum-dot-by, for the heads up.) Fixed unitialized $hostID value bug-reported to OpenSUSE. Thanks to Sven Uebelacker (uebelhacker) for the bug report. I *suspect* that, with an earlier fix I applied, that undefined value will no longer occur, but this fix certainly won't hurt anything—just in case. Added -srs-mung option. Thanks and a tip o' the hat to Tom Hendrikx (tom-at-whyscreem-dot-net) for the contribution. rel-1.1.7 20250524 Added parsing for "Recipient address triggers...," as in DISCARD on recipient actions. Improved "deferred" and "bounce" detail Added --use-orig-to switch, which causes the value of the "orig_to" field to be reported, rather than that of the "to" field, when "orig_to" is present. (This was long-ago requested by Tony Earnshaw and was overlooked by me. My apologies, Tony.) Added --rej-add-to switch, which adds the target email address to sender and client reject reports. (This was also long-ago requested by Tony Earnshaw.) For client rejects this could have created an ambiguity: If only either --rej-add-from or --rej-add-to was specified, you'd have no way of knowing, after-the-fact, which address was reported. So "<" and ">" have been added, to indicate "from" and "to," respectively. If both are specified, you get "from -> to." Added --colwidth switch, to allow adjustment of report's output width. (Default remains at 80 columns.) N.B.: --verbose-msg-detail overrides Now condenses deferrals in a manner similar to bounces. Now condenses warnings in a manner similar to reject reports. Made count reporting more consistent. Now each reporting section shows the item count in parenthesis (e.g.: "(nn)") or, if the count is limited and less than full count, as "(top nn of nn)". Suggestion by Michael Rasmussen (michael-at-michaelsnet-dot-us), tho not done in the way he suggested. Fixed policyd-spf-perl log line detection. Fixed conversation/lost connection while sending/receiving deferral detection Improved "said:" string trimmer to be a mite less aggressive when it didn't have to be. Improved reject report consolidation. In Host/Domain Summary: Message Delivery: Changed "sent cnt" to "msg cnt" since we're talking about messages delivered, not necessarily "sent," per se. Fixed bug in smtpd "disconnect from" parsing and streamlined. Removed superfluous additional "bounced" processing Caught another warning message Catch "Server configuration error" lines. Enhanced FQDN/IP addr/domain name parsing (in gimme_domain()): . More exhaustive IPv6 address check . Handles IPv6-mapped-IPv4 addresses . Properly handles the newer, longer TLDs . Treats in-addr.arpa and ip6.arpa FQNS like "unknown"s . More streamlined Added pffrombyto and pftobyfrom command-line utilities to the distribution. (Actually in 1.1.6.) rel-1.1.6 20250522 Renamed from "pflogsumm.pl" to "pflogsumm" Merged Debian patches through 1.1.5-8. Following are excerpted from Debian changelog. Fix regex in milter-rejects patch. Kudos to Andreas Jaggi for the report and patch. (Closes: #1027829) Update postscreen-rejects patch with a fix for IPv6 addresses. Kudos to Juri Haberland for the fix. (Closes: #955627) Add patch to count milter rejects provided by Matus Uhlar. Import postscreen support patch provided by Matus Uhlar d/patches/postscreen-rejects (Closes: #861402) Now matches "traditional" log date strings with either leading space or leading zero on single-digit days. (Debian bug report 1068425) Improvement to "unprocessed" debugging code—used for maintenance only. rel-1.1.5 20120205 Fixed RFC 3339 support. Releases 1.1.3 and 1.1.4 were badly broken in this respect. Thanks and a tip o' the hat to Sven Hoexter (sven-at-timegate-dot-de) for the help. ETA (2025-05-21): From Debian changelog: Thanks to Mihai Stan for the bugreport and the initial patch. rel-1.1.4 20120201 Modified for compatibility with -o syslog_name=blurfl/submission and -o syslog_name=blurfl/smtps set in master.cf. (These are the defaults in Postfix 2.9 and beyond.) N.B.: This doesn't mean you'll get submission and smtps broken-out separately from plain old smtp, it simply means the presence of the new sub-strings won't break Pflogsumm. Changed "_"s (underscores) in option switches to "-"s (dashes). (Underscores are still accepted.) Thanks and a tip o' the hat to David Landgren (david-at-landgren-dot-net) for the suggestion. Removed switches deprecated in 1.1.3 from the docs. Improved ISO timestamp parsing to account for optional fractional seconds part. (This is thrown-away by Pflogsumm.) Minor updates to the FAQ. Replaced "depreciated" with "deprecated" throughout. Thanks and a tip o' the hat to Rob Arista for the heads-up. Fixed bug in host normalization function that was broken for IPv4 addresses. rel-1.1.3 20100320 Added long-awaited switches to optionally reduce detail reporting: --bounce_detail=N, --deferral_detail=N, --reject_detail=N, --smtp_detail=N, smtpd_warning_detail=N, and --detail=N. Setting any of them to 0 suppresses that detail entirely. --detail=N sets the default for all of them, as well as for -u=N and -h=N. With the above enhancements, the following switches are deprecated, and will eventually be removed: --no_bounce_detail, --no_deferral_detail, --no_reject_detail and --no_smtpd_warnings. They are replaced by setting the desired --*_detail=0. They still work, but using them generates a warning. Added support for parsing logs with RFC 3339 timestamps. Thanks and a tip o' the hat to sftf-at-yandrex-dot-ru for the heads-up and the code contribution. (N.B.: My code does not require a command-line switch. The format is detected automatically.) Fixed some --ignore-case inconsistincies. Thanks and a tip o' the hat to Richard Blanchet (richard-dot-blanchet-at-free-dot-fr) for the heads-up and the diff. Fixed parsing bug that resulted in attempts to treat kind-of-IPv4-looking strings as IPv4 addresses. (I really need to improve reject/defer/etc. "reason" parsing to fix this properly.) Thanks to Joseph Vit (jvit-at-certicon-dot-cz) for the bug report. rel-1.1.2 20080629 Fixed bug with calculating yesterday's date in vicinity of DST changes. (Thanks and a tip o' the hat to Wieland Chmielewski for bringing the problem to my attention.) Added missing "underlining" to some (sub-)section titles for consistency. rel-1.1.1 20070406 Fixed to parse Postfix-2.3 (and beyond) logfiles. Thanks to whomever contributed to http://bugs.gentoo.org/show_bug.cgi?id=144236 Removed support for vmailer. Removed "SMTPD_STATS_SUPPORT" "fences" in code in favour of code to automatically detect the availability of Date::Calc. If --smtpd_stats is specified and Date::Calc is not installed, now bails-out with friendly message. (Adapted from suggestion and examples provided by David Landgren . Thanks!) Removed rem_smtpd_stats_supp.pl utility from distribution. (No longer needed.) Memory footprint improvement: Pflogsumm no longer stores data for reports that are supressed via --no_ switches. Removed extraneous arguments in two calls to print_nested_hash that would result in the "quiet" flag being ignored. Thanks to Pavel Urban (pupu-at-pupu-dot-cz) for bringing that to my attention. Added notes to FAQ about translations and i18n, about mismatching "received"/"delivered" counts, about bug in calculating "yesterday," and about John Fawcett's "prepflog." rel-1.1.0 20031212 Promoted 1.0.18 (Beta) to "production/stable" version release. rel-1.0.18 20031204 Fixed reject parsing for "DATA" smtpd rejects. rel-1.0.17 20031129 Fixed reject parsing to properly recognize bare "User unknown". (Thanks to J.D. Bronson" for the bug-report and sample logfile lines.) rel-1.0.16 20031128 Re-worked "to" and "from" field parsing in reject report handling to make it more robust in pathological cases. (Thanks to Paul Brooks and Lars Hecking for the bug-reports and sample logfile lines.) Fixed warnings resulting from non-standard, extraneous syslog input. (Thanks to Mathias Behrle for the report.) Fixed reject parsing to account for really atrocious garbage in HELO strings, sender addresses and recipient addresses. (Thanks to Lars Hecking for the bug-report and sample logfile lines.) Fixed reject parsing to properly recognize "CONNECT" smtpd rejects. (Thanks to Mike Vanecek for the bug-report.) Fixed reject parsing to properly recognize "User unknown in relay recipient table." (Thanks to Lars Hecking for the bug-report and sample logfile lines.) Some code optimization resulting in 3-5% performance improvement. rel-1.0.15 20030914 Pflogsumm *should* now properly parse and handle log entries with IPv6 addresses in them. (Adapted from idea and code submitted by Stefan `Sec` Zehl .) Fixed "User unknown in local recipient table" reject reports to show target recipient address, rather then sending domain, to be consistent with other "recipient" reports. (Thanks to WC Jones for the suggestion.) Fixed parsing of "Recipient address rejected" for recipient address verification (RAV) rejects. (Thanks to Len Conrad for the suggestion.) FAQ additions regarding recommendations on how to format custom reject reports for "best" results in Pflogsumm's output and note regarding "non-standard" syslogd's. rel-1.0.14 20030909 Fixed bug in parsing for "Host/Domain Summary: Messages Received" report improvement (rel-1.0.13) that resulted from (unexpected, to me) lines such as ... postfix/smtpd[31430]: E02DDB04E: client=blurfl[1.2.3.4], sasl_method=LOGIN, sasl_username=phred rel-1.0.13 20030907 The "Host/Domain Summary: Messages Received" report would show simply "from=<>", for the host/domain, for postmaster bounces. Pflogsumm now substitutes the client hostname or IP address for these, unless it's from the pickup daemon, in which case "from=<>" is retained. (Note that "Senders by message count/size" reports are unaffected by this change.) "Senders by message count" and "Recipients by message count" reports are now secondarily sorted by domain, host and user parts. (As a side-effect: So are "Senders by message size" and "Recipients by message size" but, being as the odds are against numerous senders and recipients having the same total message sizes, this change hasn't much effect with those.) rel-1.0.12 20030902 Rejects, warns, etc. now print sub-category totals. E.g.: message reject detail --------------------- RCPT Relay access denied (total: 6) (Adapted from idea and code submitted by blake7-at-blake7-dot-org.) Reject, warning, etc. reports are now sorted by 2nd column (e.g.: IP address, domain, etc.) within count. (Adapted from idea and code submitted by David Landgren .) Added --no_smtpd_warnings (report) option. Added --no_no_msg_size (report) option. A couple of minor improvements to reject parsing/reporting. rel-1.0.11 20030617 This is a bug-fix release. There was a problem in the way pflogsumm-1.0.8 through 1.0.10 handled the --syslog_name option: When --syslog_name was specified, some log entries with the default "postfix" name would be missed. This revision may introduce incompatibilities if you're logging two or more instances of Postfix to the same log. See the docs included in the tarball for details. rel-1.0.10 20030219 Re-worked "% rejected" calculation to include messages discarded and added "% discarded" calculation/display. rel-1.0.9 20030217 Bugfix: If Perl's -w is specified at run-time and there were no messages delivered or rejected, uninitialized variable warnings would be issued in the percent rejected calculation code. Thanks for Larry Hansford (and many others since!) for the bug report. rel-1.0.8 20030216 Bugfix: Fixed problem with "orig_to=" being parsed as "to=". This resulted in *very* wrong output. Thanks to Bjorn Swift for the report. Added "% rejected" to Grand Totals "rejected" figure. This is calculated as: rejected / (delivered + rejected). (I did this purely because it amuses me.) Bugfix: Fix, in reject processing, for truncated overly-long "to" fields. Thanks to Rick Troxel for reporting the problem. Added --syslog_name option. Thanks to Ben Rosengart for the suggestion. rel-1.0.7 20021231 Corrected and improved message reject/reject warn/hold/discard parsing. Again. (Thanks to Peter Santiago for reporting the problem that initiated these improvements.) rel-1.0.6 20021230 Added support for reporting message reject warnings, holds and discards. Note: Message rejects, reject warnings, holds and discards are all reported under the "rejects" column for the Per-Hour and Per-Day traffic summaries. More aggressive verp munging (again). (Prompted, in part, by a suggestion from Casey Peel. Thanks!) Verp munging now applied to sender addresses in smtpd reject reports. WARNING: Please note that verp munging is highly experimental! Pflogsumm distribution changed to gzip'd tarball format. Tightened-up parsing. Thanks for Ralf Hildebrandt for noting and reporting the problem. Docs at the top of pflogsumm.pl changed to POD format for automated manpage generation. README added. Automatically-generated manpage added. "To Do" moved out of ChangeLog into separate file. Package now includes convenience Perl script for removing smtpd stats support for those who don't have Date::Calc, don't want to install it and don't care about smtpd stats reporting. Belated thanks to Len Conrad in regards to the Sender Address Verification work in 1.0.5. rel-1.0.5 20021222 Fixed to parse smtpd rejects for Postfix versions as of 20021026. (Retained compatibility with older versions of Postfix.) Note: smtpd and header-/body-checks warn, hold and discard messages are *not* currently parsed/reported. I'll need to get some logfile entries. Fixed parsing to handle the new "sender address verification" lines. Added "--zero_fill" option to put zeros in columns that might otherwise be blank in some reports. (Suggestion by Matthias Andree). Fixed "Message size exceeds fixed limit" parsing for reject reporting. rel-1.0.4 20020224 Added "--no_*_detail" options. (Suppresses some of the "detail" reports.) Added "--version" option. (Thanks to "Guillaume") Improved handling of "[ID nnnnnn some.thing]" stuff (Thanks to Blake Dunmire) Repaired and optimized some of the "reject" parsing. Added processing and report of smtp delivery failures. Added --rej_add_from option: For those reject reports that list IP addresses or host/domain names: append the email from address to each listing. (Note: does not apply to "Improper use of SMTP command pipelining" report.) rel-1.0.3 20010520 Minor re-work of "reject: RCPT" parsing to account for Yet Another Change in the logfile format. (Semi-colon changed to a comma in "blocked using rbl.maps.vix.com,".) rel-1.0.2 20010519 Took another whack at "verp" munging. *sigh* Added code to summarize "Improper use of SMTP command pipelining" rejects by client. rel-1.0.1 20010518 Modified to catch "reject: header" log entries changed as of postfix release-20010228 (?). Prior versions of postfix had the string "warning: " (where the qid normally is). Thanks to Glen Eustace , Len Conrad , Daniel Roesen , Milivoj Ivkovic and j_zuilkowski@hotmail.com (Jon Zuilkowski) for reports and/or patches. Fixed a couple of "uninitialized variable" problems. Committed (actually starting with 20000925-01beta) to CVS. 20000925-01 Added a line to compensate for (new?) "[ID nnnnnn some.thing]" sub-strings that appear in logfile entries under Sun Solaris 8. (At least. Others?) Note: Upon being committed to CVS, this became rel-0.9.0. 20000916-01 Forgot to add "--problems_first" to the "usage" output and in the synopsis at the top of the comments. 20000908-01beta Re-did what 20000907-02beta was *supposed* to be! To wit: replaced missing "--ignore_case" bugfix, "panic" entry processing, improvements to "fatal" and "warning" message reporting and missing "--mailq" option. (Obviously: 20000907-02beta was derived from the wrong code base.) 20000907-02beta Fixed bug in ISO date formatting that caused the month to be off by one. Thanks to Kurt Andersen for the report and the patch. Fixed overflow of connect time reporting into days. (Can happen during weekly summaries for sites with large volumes of email.) Thanks again to Kurt Andersen for the report and the fix. Improved "rejects" reporting *again*. Thanks to Thomas Parmelan for the patch. Added "--problems_first" option to print "problem" reports such as bounces, defers, warnings, fatal errors, etc. before "normal" reports. 19991219-02 Fixed bug in code that prevented "--ignore_case" from actually doing anything. Thanks to Nadeem Hasan for reporting this and supplying the fix. 19991219-01beta Added the following caveat to the "Notes" section of Pflogsumm: ------------------------------------------------------------- IMPORTANT: Pflogsumm makes no attempt to catch/parse non- postfix/vmailer daemon log entries. (I.e.: Unless it has "postfix/" or "vmailer/" in the log entry, it will be ignored.) ------------------------------------------------------------- Added reporting of "panic" log messages. This was missed until now! Increased reporting detail of "fatal" and "warning" entries. (Actually, "warning" detail was increased in 19991120-01beta. Neglected to note it then.) 19991123-01 (unreleased) Added "--mailq" option. (Convenience factor.) Runs Postfix's "mailq" command at the end of the other reports. ------------------------------------------------------- NOTE: If Postfix's "mailq" command isn't in your $PATH, you'll have to edit the "$mailqCmd" variable located near the top of pflogsumm to path it explicitly. ------------------------------------------------------- 19991120-01 Tried once again to improve parsing of reject log entries. Specifically: those associated with "RCPT" rejects. 19991016-01 (not generally released) Added --smtpd_stats. Generates smtpd connection statistics. --------------------------------------------------------------- NOTE: Support for --smtpd_stats requires the Date::Calc module (available from CPAN). If you don't want to go to the trouble of fetching & installing that module, and you don't want smtpd stats anyway, *carefully* identify all of the code sections delimited by "# ---Begin: SMTPD_STATS_SUPPORT---" and "# ---End: SMTPD_STATS_SUPPORT---" and remove them. --------------------------------------------------------------- 19990909-01 (not generally released) Added -i and --ignore_case options. Causes entire email address to be lower-cased instead of just the host/domain part. Added "use locale". (This means that the sorting order within reports may be different from before--depending on how you have your machine's locale set.) 19990904-03 Improved "reason" parsing and reporting for bounced and deferred messages. Added parsing of "cleanup" reject lines to catch PCRE/regexp rejects. Added "reject" stats to per-hour and (on multi-day reports) per- day reports. Improved "warnings" report to show details. A single message deferred multiple times showed up as multiple deferrals--implying that multiple messages were deferred. Now shows "how many messages were deferred" and "how many deferrals" as separate stats. Changed display of "Grand Totals" to make it a bit more readable (IMO). Added "automatic perl finder" line for those systems that don't support the "#!" notation. By popular demand: added note to comments as to where pflogsumm home page could be found :-). 19990413-02 Fixed problem with last octet of IP address getting truncated in reports when IP address used in place of unknown hosts. Changed the way a few internal variables were handled to be compatible with Perl 5.003. Don't run it under Perl 5.003 with the "-w" perl switch, tho! It will issue lots of warnings. All tests I performed indicated that it produces the correct output, however. ------------------------------------------------------------ NOTE: While this version was tested to work with Perl 5.003, I recommend that you upgrade to 5.004 or later. I will not guarantee that I'll remember to do the full regression- testing that I usually do with 5.003 as well. ------------------------------------------------------------ 19990411-01 NOTICE: As of this version of pflogsumm.pl, the "-c" switch is GONE! (As per the previous notice.) Added "--help" option to emit short usage message and bail out. Added "--iso_date_time" switch to change displays of dates and times to ISO 8601 standard formats (CCYY-MM-DD and HH:MM), rather than "Month-name Day-number CCYY" and "HHMM" formats (the default). Added "--verbose_msg_detail" switch. This causes the full "reason" to be displayed for the message deferral, bounce and reject summaries. (Note: this can result in quite long lines in the report. Also note that there have been a couple of subtle changes in the "reason" parsing/reporting in the default mode (no "--verbose_msg_detail".) Added "--verp_mung" option. The problem this addresses is "VERP" generated (??? so far as I can tell!) addresses (?) of the form: "list-return-NN-someuser=some.dom@host.sender.dom" These result in mail from the same "user" and site to look like it originated from different users, when in fact it originates from the same "user." There are presently two "levels" of address munging available. With no numeric argument (or any value less than 2), the above address will be converted to: "list-return-ID-someuser=some.dom@host.sender.dom" In other words: the numeric value will be replaced with "ID". By specifying "--verp_mung=2", the munging is more "aggressive", converting the above address to something like: "list@host.sender.dom" Which looks more "normal." (Actually: specifying anything less than 2 does the "simple" munging and anything greater than 1 results in the more "aggressive" hack being applied.) Added "--uucp_mung" switch for consistence with "--verp_mung". 19990321-01 NOTICE: As of this version of pflogsumm.pl, versions of VMailer prior to 19981023 are no longer supported. Sorry. Pflogsumm-19990121-01.pl will be made permanently available from now on for those with out-of-date versions of VMailer prior to 19981023. NOTICE: As of this version of pflogsumm.pl, the "-c" switch is DEPRECATED. This version is transitional and retains it. The next version will not have it. Subsequent versions may re-use it for another purpose. Use the "-h" and "-u" switches instead. Added "-h" and "-u" switches to provide finer-grained control over report output. Deprecated "-c". Added "deferred" and "bounced" to "Grand Totals", "by-day" and "by- hour" reports. Added "by-host/domain" reports. For sent (delivered) and received messages: lists message count, total size of messages and host/domain. For delivered messages: also lists number of deferred messages and average and maximum delivery time. Both reports sorted by message count in descending order. Grand totals also now list number of recipient and sender hosts/domains. Re-wrote "by-user" data collection storage to reduce memory consumption by in-memory hashes. Moved "credits" from pflogsumm.pl to this file. 19990121-01 Now accounts for forwarded messages. Side-effects of the above: . Total messages size now broken-out into total bytes received and total bytes delivered. . Count of forwarded messages now reported. . Postfix-internally-generated messages (e.g.: Postmaster notifications of bounces) are no longer counted as "received". (They do, however, show up as "delivered".) . Forwarded addresses no longer show up as "recipients" (just as with aliases and mailing lists). Note that "delivered" will exceed "received" when messages are forwarded because of additional header lines. 19990116-01 Added processing for "reject" log entries. Expanded detail of "deferred" and "bounced" log entries to include "reason". 19990110-05 Added "messages received/delivered by hour" and "messages received/delivered by day" reports. See the "Notes" section in the documentation for details on how these behave. Broke-out total message count to "messages received" and "messages delivered". (For the above two enhancements: "postfix/pickup" and "postfix/smtpd" lines are now processed. They used to be discarded.) Renamed "summary" report to "Grand Totals". Added code to parse date & time stamps from log entries. This was needed, in part, for the "messages per-hour/day" reports. It would have been necessary for future enhancements in the way of date- & time-based processing anyway. Added "Notes" section to docs at top of code. 19990109-01 Improved display of large integer values. 19990107-01 Bugfix only. Data for "extended detail" listing was being built even if "-e" not specified. This resulted in unexpected excessive memory consumption when crunching large amounts of data. Added warning about memory consumption when "-e" option specified. 19990103-01 Further improvement to "accuracy" of by-domain-then-logname sort. (Presently used only by "extended detail" listing). For comparison purposes: mungs "machine(s).host.dom" into "host.dom.machine(s)" so sort is keyed on "base" domain name before machines within the domain. Does *not* attempt to reverse the order of the "machine(s)" - so within a particular "base" domain, may not come out in quite the right order. ("foo.bar.some.dom" will come out before "sales.acme.some.dom", for example.) Also works for 2x2-style domain names. (I.e.: "some.do.co") 19990102-01 (never released) Added "mung UUCP-style bang-paths" switch (-m). Improved performance and "accuracy" of by-domain-then-logname sort used by (only at present) "extended detail" listing. 19990101-02 Added "extended detail" option (-e). At present this includes only a per-message detail listing, which lists per-message detail sorted by sender domain, then sender username, then by queue i.d. Improved docs a bit. 19990101-01 Replaced warning message when message size unavailable in favor of producing a report of these, sorted by queue i.d. Unlike the other reports, this report header is not emitted at all if there are none of these. (Always acts as if the -q switch had been specified). 19981231.01 Added experimental code to lower-case all domain names so that "user@foo.dom" and "user@FOO.DOM" will come out the same. Added test for existence of message size value when "to=" records are being processed. This was necessary for cases in which the logfile entry containing the "status=sent" record is not processed at the same time as the logfile containing the "size=nnnn" record. Note that this will produce a summary that will show recipient counts without matching recipient sizes. The only way to cure this would be to create a separate disk file to "memorize" message sizes. (Which would introduce a whole new raft of problems.) Added warning message (emitted to stderr) when the situation above is detected. Fixed "usage" message to indicate you can specify files on command line Wrapped a couple of long lines in the comments and code. Added (temporary) version numbering scheme. Started this log. Other changes/enhancements since previous un-version-numbered versions: deals with log entries for VMailer as well as Postfix, more robust parsing of "to=" and "from=" fields (now handles spaces in these), eliminated double-counting of message sizes (happened when delivery was deferred), re-structured parsing to be more robust (not- to-mention correct!), added "grand summary" report at top (total messages, total size, number of senders and recipients). Credits [Note: The credits reflect suggestions and code contributions that have actually been added. If your contribution doesn't appear here, it may simply mean that it hasn't been added yet. (In which case it should be on the list above.) On the other hand: if I failed to credit you for something that *has* been added, please let me know!] Paul D. Robertson For much testing and patience and many good suggestions on how pflogsumm could be improved. Simon J Mudd For the following code contributions: Add "deferred" and "bounced" to "by hour" reports. (I also added these to "by day" reports and "Grand Totals".) "VERP" (?) address munger (less-agressive version) Suggestion for "by domain" delivery delay report. For the --smtpd_stats suggestion. Anders Arnholm For pointing out the problem with forwarded messages. Walcir Fontanini For pointers to changes to make for Perl 5.003 compatibility. (Added to 19990413-02beta.) (Which I will *try* to keep in mind!) Eric Cholet For the --ignore_case patch. Kurt Andersen For the ISO date formatting month-off-by-one patch and the connect time overflow fix. Thomas Parmelan For improved "rejects" reporting patch. Glen Eustace Patch to fix "reject: header" matching after Wietse changed the logfile format. pflogsumm-1.1.13/pffrombyto0000755000175000017500000001544715013664267016264 0ustar jseymourjseymour#!/usr/bin/perl -w eval 'exec perl -S $0 "$@"' if 0; =head1 NAME pffrombyto - List "from" addresses by "to" whom in Postfix log file Copyright (C) 2007-2025 by James S. Seymour, Release 1.2 =head1 SYNOPSIS pffrombyto -[bchrRv] [mailfile] If no file(s) specified, reads from stdin. Output is to stdout. =head1 DESCRIPTION pffrombyto parses Postfix log files to generate a list of "from" addresses, based on a specified "to" address or address fragment. =head1 OPTIONS -b Include bounces -c Include client hostnames/addrs, as well -h Emit help message and exit -r Include rejects -R Hard rejects only -v Emit version and exit =head1 RETURN VALUE pffrombyto doesn't return anything of interest to the shell. =head1 ERRORS Error messages are emitted to stderr. =head1 EXAMPLES Generate a list of all the senders of email to the recipient "username@example.com" pffrombyto username@example.com /var/log/maillog As a convenience, pffrombyto tries to intelligently determine how to handle regexp meta-characters. If it's passed a search expression that does NOT contain meta-character escapes ("\"), it will assume that "." and "+" are literals, and will escape them for you. In the example above, the "." in the FQDN part of the search term would've been automatically escaped for the user. Likewise: pffrombyto username+foo@example.com /var/log/maillog would have the "+" and "." escaped. If you wanted to find all plussed targets for "username," you'd have to do: pffrombyto 'username\+.+@example\.com' /var/log/maillog =head1 SEE ALSO pflogsumm, pftobyfrom =head1 NOTES All search terms and searched fields are lower-cased. The pffrombyto Home Page is at: http://jimsun.LinxNet.com/postfix_contrib.html =head1 REQUIREMENTS Perl =head1 LICENSE This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You may have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. An on-line copy of the GNU General Public License can be found http://www.fsf.org/copyleft/gpl.html. =cut use strict; use Getopt::Std; (my $progName = $0) =~ s/^.*?\///o; my $usageMsg = "Usage: $progName -[bchrRv] [mailfile] -b Include bounces -c Include client hostnames/addrs, as well -h Emit this help message and exit -r Include rejects -R Hard rejects only (4xx rejects ignored) -v Emit version and exit"; my $revision = '1.2'; use vars qw($opt_b $opt_c $opt_h $opt_r $opt_R $opt_v); getopts('bchrRv') || die "$usageMsg\n"; $opt_r = 1 if($opt_R); if($opt_h || $opt_v) { print "$progName $revision\n" if($opt_v); print "$usageMsg\n" if($opt_h); exit; } my ($fromQid, $fromWhom, %fromQids, %accList, %rejList, %bncList); my ($clientQid, $clientID, %clientQids); my ($toWhom); die "$usageMsg\n" unless($toWhom = shift @ARGV); my $doEscapes = !($toWhom =~ /\\/); # Escape "."s and "+"s? $toWhom =~ s/([\.\+])/\\$1/g if($doEscapes); while(<>) { if($opt_c) { if(($clientQid, $clientID) = /:(?: \[ID \d+ mail\.info\])? ([A-F0-9]+): client=(.+)$/o) { # print "dbg: $clientQid, $clientID\n"; $clientQids{$clientQid} = $clientID; next; } } if(($fromQid, $fromWhom) = /:(?: \[ID \d+ mail\.info\])? ([A-F0-9]+): from=<([^>]+)>/o) { # print "dbg: $fromQid, $fromWhom\n"; $fromQids{$fromQid} = $fromWhom; if($opt_c && $clientQids{$fromQid}) { $fromQids{$fromQid} .= " $clientQids{$fromQid}"; } }elsif($opt_r && (my ($respCode, $fromWhom, $toWhomFull) = /: NOQUEUE: reject: RCPT from \S+: (\d+) .+from=<([^>]+)> to=<(.*$toWhom[^>]*)>/oi)) { ++$rejList{lc $toWhomFull}{"$fromWhom"} unless($opt_R && $respCode == 450); }elsif((my $toQid, $toWhomFull) = /:(?: \[ID \d+ mail\.info\])? ([A-F0-9]+): to=<([^>]*$toWhom[^>]*)>, .+ status=sent/oi) { if($fromQids{$toQid} && ! $opt_R) { # print "dbg: $fromQids{$toQid} $toWhomFull\n"; ++$accList{lc $toWhomFull}{$fromQids{$toQid}}; } }elsif($opt_b && (($toQid, $toWhomFull) = /:(?: \[ID \d+ mail\.info\])? ([A-F0-9]+): to=<(.*$toWhom[^>]*)>, .+ status=bounced/oi)) { if($fromQids{$toQid}) { # print "dbg: $fromQids{$toQid} $toWhomFull\n"; ++$bncList{lc $toWhomFull}{$fromQids{$toQid}}; } } } if(%accList) { print "\nDelivered:\n"; walk_nested_hash(\%accList, 0); } if($opt_r && %rejList) { print "\nRejected:\n"; walk_nested_hash(\%rejList, 0); } if($opt_b && %bncList) { print "\nBounced:\n"; walk_nested_hash(\%bncList, 0); } print "\n"; # "walk" a "nested" hash sub walk_nested_hash { my ($hashRef, $level) = @_; $level += 2; my $indents = ' ' x $level; my ($keyName, $hashVal) = each(%$hashRef); if(ref($hashVal) eq 'HASH') { foreach (sort keys %$hashRef) { print "$indents$_"; # If the next hash is finally the data, total the # counts for the report and print my $hashVal2 = (each(%{$hashRef->{$_}}))[1]; keys(%{$hashRef->{$_}}); # "reset" hash iterator unless(ref($hashVal2) eq 'HASH') { my $cnt = 0; $cnt += $_ foreach (values %{$hashRef->{$_}}); print " (total: $cnt)"; } print "\n"; walk_nested_hash($hashRef->{$_}, $level); } } else { really_print_hash_by_cnt_vals($hashRef, 0, $indents); } } # *really* print hash contents sorted by numeric values in descending # order (i.e.: highest first), then by IP/addr, in ascending order. sub really_print_hash_by_cnt_vals { my($hashRef, $cnt, $indents) = @_; foreach (map { $_->[0] } sort { $b->[1] <=> $a->[1] || $a->[2] cmp $b->[2] } map { [ $_, $hashRef->{$_}, normalize_host($_) ] } (keys(%$hashRef))) { printf "$indents%6d %s\n", $hashRef->{$_}, $_; last if --$cnt == 0; } } # Normalize IP addr or hostname # (Note: Makes no effort to normalize IPv6 addrs. Just returns them # as they're passed-in.) sub normalize_host { # For IP addrs and hostnames: lop off possible " (user@dom.ain)" bit my $norm1 = (split(/\s/, $_[0]))[0]; if((my @octets = ($norm1 =~ /^(\d+)\.(\d+)\.(\d+)\.(\d+)$/o)) == 4) { # Dotted-quad IP address return(pack('C4', @octets)); } else { # Possibly hostname or user@dom.ain #return(join( '', map { lc $_ } reverse split /[.@]/, $norm1 )); return lc $_[0]; } } pflogsumm-1.1.13/pftobyfrom0000755000175000017500000001447515013664267016264 0ustar jseymourjseymour#!/usr/bin/perl -w eval 'exec perl -S $0 "$@"' if 0; =head1 NAME pftobyfrom - List "to" addresses by "from" whom in Postfix log file Copyright (C) 2007-2025 by James S. Seymour, Release 1.3 =head1 SYNOPSIS pftobyfrom -[bhrRv] [mailfile] If no file(s) specified, reads from stdin. Output is to stdout. =head1 DESCRIPTION pftobyfrom parses Postfix log files to generate a list of "to" addresses, based on a specified "from" address or address fragment. =head1 OPTIONS -b Include bounces -h Emit help message and exit -r Include rejects -R Hard rejects only -v Emit version and exit =head1 RETURN VALUE pftobyfrom doesn't return anything of interest to the shell. =head1 ERRORS Error messages are emitted to stderr. =head1 EXAMPLES pftobyfrom example.com /var/log/maillog Generates a list of all the recipients of email from any senders in "example.com" As a convenience, pftobyfrom tries to intelligently determine how to handle regexp meta-characters. If it's passed a search expression that does NOT contain meta-character escapes ("\"), it will assume that "." and "+" are literals, and will escape them for you. In the example above, the "." in the FQDN part of the search term would've been automatically escaped for the user. Likewise: pftobyfrom username+foo@example.com /var/log/maillog would have the "+" and "." escaped. If you wanted to find all plussed senders for "username," you'd have to do: pftobyfrom 'username\+.+@example\.com' /var/log/maillog =head1 SEE ALSO pflogsumm, pffrombyto =head1 NOTES All search terms and searched fields are lower-cased. The pftobyfrom Home Page is at: http://jimsun.LinxNet.com/postfix_contrib.html =head1 REQUIREMENTS Perl =head1 LICENSE This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You may have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. An on-line copy of the GNU General Public License can be found http://www.fsf.org/copyleft/gpl.html. =cut use strict; use Getopt::Std; (my $progName = $0) =~ s/^.*?\///o; my $usageMsg = "Usage: $progName -[bhrRv] [mailfile] -b Include bounces -h Emit this help message and exit -r Include rejects -R Hard rejects only -v Emit version and exit"; my $revision = '1.3'; use vars qw($opt_b $opt_h $opt_r $opt_R $opt_v); getopts('bhrRv') || die "$usageMsg\n"; $opt_r = 1 if($opt_R); if($opt_h || $opt_v) { print "$progName $revision\n" if($opt_v); print "$usageMsg\n" if($opt_h); exit; } my ($fromQid, %fromQids, %accList, $fromWhomFull, %rejList, %bncList); my ($fromWhom); die "$usageMsg\n" unless($fromWhom = shift @ARGV); my $doEscapes = !($fromWhom =~ /\\/); # Escape "."s and "+"s? $fromWhom =~ s/([\.\+])/\\$1/g if($doEscapes); while(<>) { if(($fromQid, $fromWhomFull) = /:(?: \[ID \d+ mail\.info\])? ([A-F0-9]+): from=<(.*$fromWhom[^>]*)>/oi) { # print "dbg: from: $fromQid $fromWhomFull\n"; $fromQids{$fromQid} = lc $fromWhomFull; next; }elsif($opt_r && (my ($respCode, $fromWhomFull, $to) = /: NOQUEUE: reject: RCPT from \S+: (\d+) .+from=<(.*$fromWhom[^>]*)> to=<([^>]+)>/oi)) { ++$rejList{lc $fromWhomFull}{"$to"} unless($opt_R && $respCode == 450); }elsif((my $toQid, $to) = /:(?: \[ID \d+ mail\.info\])? ([A-F0-9]+): to=<([^>]+)>, .+ status=sent/o) { # print "dbg: to: $toQid $to\n"; if($fromQids{$toQid} && ! $opt_R) { # print "dbg: match!\n"; ++$accList{$fromQids{$toQid}}{$to}; } }elsif($opt_b && (($toQid, $to) = /:(?: \[ID \d+ mail\.info\])? ([A-F0-9]+): to=<([^>]+)>, .+ status=bounced/o)) { # print "dbg: to: $toQid $to\n"; if($fromQids{$toQid}) { # print "dbg: match!\n"; ++$bncList{$fromQids{$toQid}}{$to}; } } } if(%accList) { print "\nDelivered:\n"; walk_nested_hash(\%accList, 0); } if($opt_r && %rejList) { print "\nRejected:\n"; walk_nested_hash(\%rejList, 0); } if($opt_b && %bncList) { print "\nBounced:\n"; walk_nested_hash(\%bncList, 0); } print "\n"; # "walk" a "nested" hash sub walk_nested_hash { my ($hashRef, $level) = @_; $level += 2; my $indents = ' ' x $level; my ($keyName, $hashVal) = each(%$hashRef); if(ref($hashVal) eq 'HASH') { foreach (sort keys %$hashRef) { print "$indents$_"; # If the next hash is finally the data, total the # counts for the report and print my $hashVal2 = (each(%{$hashRef->{$_}}))[1]; keys(%{$hashRef->{$_}}); # "reset" hash iterator unless(ref($hashVal2) eq 'HASH') { my $cnt = 0; $cnt += $_ foreach (values %{$hashRef->{$_}}); print " (total: $cnt)"; } print "\n"; walk_nested_hash($hashRef->{$_}, $level); } } else { really_print_hash_by_cnt_vals($hashRef, 0, $indents); } } # *really* print hash contents sorted by numeric values in descending # order (i.e.: highest first), then by IP/addr, in ascending order. sub really_print_hash_by_cnt_vals { my($hashRef, $cnt, $indents) = @_; foreach (map { $_->[0] } sort { $b->[1] <=> $a->[1] || $a->[2] cmp $b->[2] } map { [ $_, $hashRef->{$_}, normalize_host($_) ] } (keys(%$hashRef))) { printf "$indents%6d %s\n", $hashRef->{$_}, $_; last if --$cnt == 0; } } # Normalize IP addr or hostname # (Note: Makes no effort to normalize IPv6 addrs. Just returns them # as they're passed-in.) sub normalize_host { # For IP addrs and hostnames: lop off possible " (user@dom.ain)" bit my $norm1 = (split(/\s/, $_[0]))[0]; if((my @octets = ($norm1 =~ /^(\d+)\.(\d+)\.(\d+)\.(\d+)$/o)) == 4) { # Dotted-quad IP address return(pack('C4', @octets)); } else { # Possibly hostname or user@dom.ain #return(join( '', map { lc $_ } reverse split /[.@]/, $norm1 )); return lc $_[0]; } } pflogsumm-1.1.13/pflogsumm0000755000175000017500000027170515076174043016104 0ustar jseymourjseymour#!/usr/bin/perl -w eval 'exec perl -S $0 "$@"' if 0; =head1 NAME pflogsumm - Produce Postfix MTA logfile summary Copyright (C) 1998-2025 by James S. Seymour, Release 1.1.13 =head1 SYNOPSIS pflogsumm [--config ] [--bounce-detail ] [--colwidth ] [--deferral-detail ] [--detail ] [-d ] [--dow0mon] [-e] [-h ] [-i] [--iso-date-time] [--mailq] [-m] [--no-no-msg-size] [--problems-first] [--pscrn-detail ] [--pscrn-stats] [-q] [--rej-add-from] [--rej-add-to] [--reject-detail ] [--smtp-detail ] [--smtpd-stats] [--smtpd-warning-detail ] [--srs-mung] [--syslog-name=string] [-u ] [--unprocd-file ] [--use-orig-to] [--verbose-msg-detail] [--verp-mung[=]] [-x] [--zero-fill] [file1 [filen]] pflogsumm --[dump-config|help|version] Note: Where both long- and short-form options exist only the latter are shown above. See man page for long-form equivalents. If no file(s) specified, reads from stdin. Output is to stdout. Errors and debug to stderr. =head1 DESCRIPTION Pflogsumm is a log analyzer/summarizer for the Postfix MTA. It is designed to provide an over-view of Postfix activity, with just enough detail to give the administrator a "heads up" for potential trouble spots. Pflogsumm generates summaries and, in some cases, detailed reports of mail server traffic volumes, rejected and bounced email, and server warnings, errors and panics. =head1 OPTIONS --bounce-detail Limit detailed bounce reports to the top . 0 to suppress entirely. --config Path to a configuration file containing pflogsumm options. Supports all standard command-line options (without the leading "-" or "--"). Options like "config", "dump-config", "help", and "version" technically work here, too, though they're not particularly useful in this context. Command-line arguments override config file values except for boolean options. --colwidth Maximum report output width. Default is 80 columns. 0 = unlimited. N.B.: --verbose-msg-detail overrides -d --date-range Limits the report to the specified date or range. Accepted values: today yesterday "this week" / "last week" "this month" / "last month" YYYY-MM[-DD] "YYYY-MM[-DD] YYYY-MM[-DD]" These options do what they suggest, with one important caveat: ISO 8601 / RFC 3339-style dates and ranges may not yield accurate results when used with traditional log formats lacking year information ("month day-of-month"). In such cases, pflogsumm assumes log entries are from the current year. For example, if the current month is April and a log contains "Apr NN" entries from the previous year, they will be interpreted as from the *current* April. As such, date-based filtering is only reliable for entries less than ~365 days old for old-/traditional-style logfiles. Arguments containing spaces must be quoted! This/last week/month arguments can take underscores, rather than spaces, to avoid quoting: E.g.: --date-range last_week ISO 8601/RFC 3339 date ranges may optionally use a hyphen or the word "to" for readability. E.g.: "2025-08-01 to 2025-08-08" If an optional day (DD) is omitted, the range becomes the full month. E.g.: 2025-08 == 2025-08-01 through 2025-08-31 "2025-07 - 2025-08" == 2025-07-01 - 2025-08-31 --dow0mon First day of the week is Monday, rather than Sunday. (Used only for this/last week calculations.) --deferral-detail Limit detailed deferral reports to the top . 0 to suppress entirely. --detail Sets all --*-detail, -h and -u to . Is over-ridden by individual settings. --detail 0 suppresses *all* detail. --dump-config Dump the config to STDOUT and exit. This can be used as both a debugging aid and as a way to develop your first config file. For the latter: Simply run your usual pflogsumm command line, adding --dump-config to it, and redirect STDOUT to a file. To make it cleaner: Remove unset configs: pflogsumm --dump-config |grep -v ' = $' -e --extended-detail Extended (extreme? excessive?) detail Emit detailed reports. At present, this includes only a per-message report, sorted by sender domain, then user-in-domain, then by queue i.d. WARNING: the data built to generate this report can quickly consume very large amounts of memory if a lot of log entries are processed! -h --host-cnt top to display in host/domain reports. 0 = none. See also: "-u" and "--*-detail" options for further report-limiting options. --help Emit short usage message and bail out. (By happy coincidence, "-h" alone does much the same, being as it requires a numeric argument :-). Yeah, I know: lame.) -i --ignore-case Handle complete email address in a case-insensitive manner. Normally pflogsumm lower-cases only the host and domain parts, leaving the user part alone. This option causes the entire email address to be lower- cased. --iso-date-time For summaries that contain date or time information, use ISO 8601 standard formats (CCYY-MM-DD and HH:MM), rather than "Mon DD CCYY" and "HHMM". -m modify (mung?) UUCP-style bang-paths This is for use when you have a mix of Internet-style domain addresses and UUCP-style bang-paths in the log. Upstream UUCP feeds sometimes mung Internet domain style address into bang-paths. This option can sometimes undo the "damage". For example: "somehost.dom!username@foo" (where "foo" is the next host upstream and "somehost.dom" was whence the email originated) will get converted to "foo!username@somehost.dom". This also affects the extended detail report (-e), to help ensure that by- domain-by-name sorting is more accurate. See also: --uucp-mung --mailq Run "mailq" command at end of report. Merely a convenience feature. (Assumes that "mailq" is in $PATH. See "$mailqCmd" variable to path thisi if desired.) --no-no-msg-size Do not emit report on "Messages with no size data". Message size is reported only by the queue manager. The message may be delivered long-enough after the (last) qmgr log entry that the information is not in the log(s) processed by a particular run of pflogsumm. This throws off "Recipients by message size" and the total for "bytes delivered." These are normally reported by pflogsumm as "Messages with no size data." --problems-first Emit "problems" reports (bounces, defers, warnings, etc.) before "normal" stats. --pscrn-detail Limit postscreen detail reporting to top lines of each event. 0 to suppress entirely. Note: Postscreen rejects are collected and reported in any event. --pscrn-stats Collect and emit postscreen summary stats. Note: Postscreen rejects are collected and reported in any event. --rej-add-from For those reject reports that list IP addresses or host/domain names: append the email from address to each listing. (Does not apply to "Improper use of SMTP command pipelining" report.) -q --quiet quiet - don't print headings for empty reports note: headings for warning, fatal, and "master" messages will always be printed. --rej-add-to For sender reject reports: Add the intended recipient address. --reject-detail Limit detailed smtpd reject, warn, hold and discard reports to the top . 0 to suppress entirely. --smtp-detail Limit detailed smtp delivery reports to the top . 0 to suppress entirely. --smtpd-stats Generate smtpd connection statistics. The "per-day" report is not generated for single-day reports. For multiple-day reports: "per-hour" numbers are daily averages (reflected in the report heading). --smtpd-warning-detail Limit detailed smtpd warnings reports to the top . 0 to suppress entirely. --srs-mung Undo SRS address munging. If your postfix install has an SRS plugin running, many addresses in the report will contain the SRS-formatted email addresses, also for non-local addresses (f.i. senders). This option will try to undo the "damage". Addresses of the form: SRS0=A6cv=PT=sender.example.com=support@srs.example.net will be reformatted to their original value: support@sender.example.com --syslog-name=name Set syslog-name to look for for Postfix log entries. By default, pflogsumm looks for entries in logfiles with a syslog name of "postfix," the default. If you've set a non-default "syslog_name" parameter in your Postfix configuration, use this option to tell pflogsumm what that is. See the discussion about the use of this option under "NOTES," below. -u --user-cnt top to display in user reports. 0 == none. See also: "-h" and "--*-detail" options for further report-limiting options. --unprocd-file Emit unprocessed logfile lines to file --use-orig-to Where "orig_to" fields are found, report that in place of the "to" address. --uucp-mung modify (mung?) UUCP-style bang-paths See also: -m --verbose-msg-detail For the message deferral, bounce and reject summaries: display the full "reason", rather than a truncated one. Note: this can result in quite long lines in the report. --verp-mung --verp-mung=2 Do "VERP" generated address (?) munging. Convert sender addresses of the form "list-return-NN-someuser=some.dom@host.sender.dom" to "list-return-ID-someuser=some.dom@host.sender.dom" In other words: replace the numeric value with "ID". By specifying the optional "=2" (second form), the munging is more "aggressive", converting the address to something like: "list-return@host.sender.dom" Actually: specifying anything less than 2 does the "simple" munging and anything greater than 1 results in the more "aggressive" hack being applied. See "NOTES" regarding this option. --version Print program name and version and bail out. -x Enable debugging to STDERR --zero-fill "Zero-fill" certain arrays so reports come out with data in columns that that might otherwise be blank. =head1 RETURN VALUE Pflogsumm doesn't return anything of interest to the shell. =head1 ERRORS Error messages are emitted to stderr. =head1 EXAMPLES Produce a report of previous day's activities: pflogsumm -d yesterday /var/log/maillog A report of prior week's activities: pflogsumm -d last_week /var/log/maillog.0 What's happened so far today: pflogsumm -d today /var/log/maillog Crontab entry to generate a report of the previous day's activity at 10 minutes after midnight: 10 0 * * * /usr/local/sbin/pflogsumm -d yesterday /var/log/maillog 2>&1 |/usr/bin/mailx -s "`uname -n` daily mail stats" postmaster Crontab entry to generate a report for the prior week's activity. 10 4 * * 0 /usr/local/sbin/pflogsumm -d "last week" /var/log/maillog.0 2>&1 |/usr/bin/mailx -s "`uname -n` weekly mail stats" postmaster (The two crontab examples, above, must actually be a single line each. They're broken-up into two-or-more lines due to page formatting issues.) Using a config file: pflogsumm --config /usr/local/etc/pflogusmm/daily.conf Using a config file, overriding a config file options on the command line: pflogsumm --config /usr/local/etc/pflogsumm/daily.conf --detail 30 This would override *all* detail settings in the config file, setting them all to 30. pflogsumm --config /usr/local/etc/pflogsumm/daily.conf --detail 30 --host-cnt 10 This would override all detail settings in the config file, setting them all to 30, with the global detail setting in turn being overridden to 10 for host count. =head1 SEE ALSO pffrombyto, pftobyfrom The pflogsumm FAQ: pflogsumm-faq.txt. =head1 NOTES Some options, such as date range, have both short-form and long-form names. In the interest of brevity, only the short-form options are shown in the SYNOPSIS and in pflogsumm's "help" output. Pflogsumm makes no attempt to catch/parse non-Postfix log entries. Unless it has "postfix/" in the log entry, it will be ignored. It's important that the logs are presented to pflogsumm in chronological order so that message sizes are available when needed. For display purposes: integer values are munged into "kilo" and "mega" notation as they exceed certain values. I chose the admittedly arbitrary boundaries of 512k and 512m as the points at which to do this--my thinking being 512x was the largest number (of digits) that most folks can comfortably grok at-a-glance. These are "computer" "k" and "m", not 1000 and 1,000,000. You can easily change all of this with some constants near the beginning of the program. "Items-per-day" reports are not generated for single-day reports. For multiple-day reports: "Items-per-hour" numbers are daily averages (reflected in the report headings). Message rejects, reject warnings, holds and discards are all reported under the "rejects" column for the Per-Hour and Per-Day traffic summaries. Verp munging may not always result in correct address and address-count reduction. Verp munging is always in a state of experimentation. The use of this option may result in inaccurate statistics with regards to the "senders" count. UUCP-style bang-path handling needs more work. Particularly if Postfix is not being run with "swap_bangpath = yes" and/or *is* being run with "append_dot_mydomain = yes", the detailed by-message report may not be sorted correctly by-domain-by-user. (Also depends on upstream MTA, I suspect.) The "percent rejected" and "percent discarded" figures are only approximations. They are calculated as follows (example is for "percent rejected"): percent rejected = (rejected / (delivered + rejected + discarded)) * 100 There are some issues with the use of --syslog-name. The problem is that, even with Postfix' $syslog_name set, it will sometimes still log things with "postfix" as the syslog_name. This is noted in /etc/postfix/sample-misc.cf: # Beware: a non-default syslog_name setting takes effect only # after process initialization. Some initialization errors will be # logged with the default name, especially errors while parsing # the command line and errors while accessing the Postfix main.cf # configuration file. As a consequence, pflogsumm must always look for "postfix," in logs, as well as whatever is supplied for syslog_name. Where this becomes an issue is where people are running two or more instances of Postfix, logging to the same file. In such a case: . Neither instance may use the default "postfix" syslog name and... . Log entries that fall victim to what's described in sample-misc.cf will be reported under "postfix", so that if you're running pflogsumm twice, once for each syslog_name, such log entries will show up in each report. The Pflogsumm Home Page is at: http://jimsun.LinxNet.com/postfix_contrib.html =head1 REQUIREMENTS Requires Perl 5.10, minimum, and Date::Calc For --config, Pflogsumm requires the Config::Simple module. Both of the above can be obtained from CPAN at http://www.perl.com or from your distro's repository. Pflogsumm is currently written and tested under Perl 5.38. As of version 19990413-02, pflogsumm worked with Perl 5.003, but future compatibility is not guaranteed. =head1 LICENSE This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You may have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. An on-line copy of the GNU General Public License can be found http://www.fsf.org/copyleft/gpl.html. =cut require v5.10.0; use strict; use locale; use feature 'state'; use Getopt::Long; use List::Util qw(reduce); use Time::Local; use Date::Calc qw(Add_Delta_Days Week_of_Year Delta_DHMS Day_of_Week Monday_of_Week Days_in_Month); use POSIX qw(strftime); eval { require Config::Simple }; my $haveConfigSimple = $@ ? 0 : 1; my $mailqCmd = "mailq"; my $release = "1.1.13"; # Variables and constants used throughout pflogsumm our ( $progName, $usageMsg, @monthNames, %monthNums, $thisYr, $thisMon, @dowNames, %fromDate, %thruDate, %qidTracker ); # Some constants used by display routines. I arbitrarily chose to # display in kilobytes and megabytes at the 512k and 512m boundaries, # respectively. Season to taste. use constant { DIV_BY_ONE_K_AT => 524288, # 512k DIV_BY_ONE_MEG_AT => 536870912, # 512m ONE_K => 1024, # 1k ONE_MEG => 1048576, # 1m }; # Constants used throughout pflogsumm @monthNames = qw(Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec); %monthNums = qw( Jan 0 Feb 1 Mar 2 Apr 3 May 4 Jun 5 Jul 6 Aug 7 Sep 8 Oct 9 Nov 10 Dec 11); @dowNames = qw("" Mon Tue Wed Thu Fri Sat Sun); ($thisMon, $thisYr) = (localtime(time()))[4,5]; $thisYr += 1900; # # Variables used only in main loop # # Per-user data my (%recipUser, $recipUserCnt); my (%sendgUser, $sendgUserCnt); # Per-domain data my (%recipDom, $recipDomCnt); # recipient domain data my (%sendgDom, $sendgDomCnt); # sending domain data # Indexes for arrays in above use constant { MSG_CNT_I => 0, # message count MSG_SIZE_I => 1, # total messages size MSG_DFRS_I => 2, # number of defers MSG_DLY_AVG_I => 3, # total of delays (used for averaging) MSG_DLY_MAX_I => 4, # max delay }; my ( $cmd, $qid, $addr, $orig_to, $size, $relay, $status, $delay, $strtDate, $endDate, %panics, %fatals, %warnings, %masterMsgs, %deferred, %bounced, %noMsgSize, %msgDetail, $msgsRcvd, $msgsDlvrd, $sizeRcvd, $sizeDlvrd, $msgMonStr, $msgMon, $msgDay, $msgTimeStr, $msgHr, $msgMin, $msgSec, $msgYr, $revMsgDateStr, $dayCnt, %msgsPerDay, %rejects, $msgsRjctd, %warns, $msgsWrnd, %discards, $msgsDscrdd, %holds, $msgsHld, %rcvdMsg, $msgsFwdd, $msgsBncd, $msgsDfrdCnt, $msgsDfrd, %msgDfrdFlgs, %connTime, %smtpdPerDay, %smtpdPerDom, $smtpdConnCnt, $smtpdTotTime, %pscrnConnTime, %pscrnPerDay, %pscrnPerIP, $pscrnConnCnt, $pscrnTotTime, %smtpMsgs, $sizeDataExists, @deprecated ); $dayCnt = $smtpdConnCnt = $smtpdTotTime = 0; # Init total messages delivered, rejected, and discarded $msgsDlvrd = $msgsRjctd = $msgsDscrdd = 0; # Init messages received and delivered per hour my @rcvPerHr = (0) x 24; my @dlvPerHr = @rcvPerHr; my @dfrPerHr = @rcvPerHr; # defers per hour my @bncPerHr = @rcvPerHr; # bounces per hour my @rejPerHr = @rcvPerHr; # rejects per hour my $lastMsgDay = 0; # Init "doubly-sub-scripted array": cnt, total and max time per-hour my @smtpdPerHr; for (0 .. 23) { $smtpdPerHr[$_] = [0,0,0]; } # # Postscreen # my @pscrnPerHr; for (0 .. 23) { $pscrnPerHr[$_] = [0,0,0]; } my @pscrnRegexs = ( { 'expr' => '(ALLOWLIST VETO) \[(.+)\]:(\d+)' }, { 'expr' => '(BARE NEWLINE) from \[(.+)\]:(\d+) after .+' }, { 'expr' => '(BDAT without valid RCPT) from \[(.+)\]:(\d+)' }, { 'expr' => '(COMMAND COUNT LIMIT) from \[(.+)\]:(\d+) after .+' }, { 'expr' => '(COMMAND LENGTH LIMIT) from \[(.+)\]:(\d+) after .+' }, { 'expr' => '(COMMAND PIPELINING) from \[(.+)\]:(\d+) after .+: .+' }, { 'expr' => '(COMMAND TIME LIMIT) from \[(.+)\]:(\d+) after .+' }, { 'expr' => '(CONNECT) from \[(.+)\]:(\d+) to \[.+\]:\d+' }, { 'expr' => '(ENFORCE) \[(.+)\]:(\d+), (PSC_CLIENT_ADDR_PORT.+;)' }, { 'expr' => '(DATA without valid RCPT) from \[(.+)\]:(\d+)' }, { 'expr' => '(DISCONNECT) \[(.+)\]:(\d+)' }, { 'expr' => '(DROP) \[(.+)\]:(\d+), (PSC_CLIENT_ADDR_PORT.+;)' }, { 'expr' => '(DNSBL rank \d+) for \[(.+)\]:(\d+)' }, { 'expr' => '(FAIL) \[(.+)\]:(\d+), (PSC_CLIENT_ADDR_PORT.+;)' }, { 'expr' => '(HANGUP) after .+ from \[(.+)\]:(\d+) in ' }, { 'expr' => '(NON-SMTP COMMAND) from \[(.+)\]:(\d+) after .+: .+' }, { 'expr' => '(NOQUEUE: reject: CONNECT) from \[(.+)\]:(\d+): (all server ports busy)' }, { 'expr' => '(NOQUEUE: reject: CONNECT) from \[(.+)\]:(\d+): (too many connections)' }, { 'expr' => '(NOQUEUE: reject: RCPT) from \[(.+)\]:(\d+): \d+ ' }, { 'expr' => '(PASS .+) \[(.+)\]:(\d+)$' }, { 'expr' => '(PASS .+) \[(.+)\]:(\d+)(?:, )(PSC_CLIENT_ADDR_PORT.+;)' }, { 'expr' => '(PREGREET) .+ after .+ from \[(.+)\]:(\d+): .+' }, { 'expr' => '(SKIP) \[(.+)\]:(\d+), (PSC_CLIENT_ADDR_PORT.+;)' }, { 'expr' => '(reject: connect) from \[(.+)\]:(\d+): (all screening ports busy)' }, { 'expr' => '(\b\w+LISTED) \[(.+)\]:(\d+)' }, { 'expr' => '(\b\w+LIST VETO) \[(.+)\]:(\d+)' }, { 'expr' => '(UNFAIL) \[(.+)\]:(\d+), (PSC_CLIENT_ADDR_PORT.+;)' }, { 'expr' => '(UNPASS) \[(.+)\]:(\d+), (PSC_CLIENT_ADDR_PORT.+;)' }, ); # FIXME: Not certain what to do with this one # { 'expr' => '(\[(.+)\]:\d+: replacing command \\".+\\" with \\".+\\")' }, my %pscrnHits; ($progName = $0) =~ s/^.*\///; $usageMsg = "usage: $progName [--config ] [--bounce-detail ] [--colwidth ] [--deferral-detail ] [--detail ] [-d ] [--dow0mon] [-e] [-h ] [-i] [--iso-date-time] [--mailq] [-m] [--no-no-msg-size] [--problems-first] [--pscrn-detail ] [--pscrn-stats] [-q] [--rej-add-from] [--rej-add-to] [--reject-detail ] [--smtp-detail ] [--smtpd-stats] [--smtpd-warning-detail ] [--srs-mung] [--syslog-name=string] [-u ] [--unprocd-file ] [--use-orig-to] [--verbose-msg-detail] [--verp-mung[=]] [-x] [--zero-fill] [file1 [filen]] $progName --[dump-config|help|version] Note: Where both long- and short-form options exist only the latter are shown above. See man page for long-form equivalents."; # # Central options specifications. This allows us to create a unified set # of arguments to GetOpts, for processing Config::Simple, and for dumping # the configuration. # # type: s = string, i = integer, b = boolean, f = float (validated manually) # Notes: "i" and "s" are used in the GetOpts hash. "f" is translated to "s". # Short options are ignored by Config::Simple processing. my %optionSpec = ( 'bounce-detail' => { type => 'i' }, 'colwidth' => { type => 'i' }, 'config' => { type => 's' }, # not exposed as CLI short option 'date-range' => { type => 's', short => 'd' }, 'debug' => { type => 'b', short => 'x' }, 'deferral-detail' => { type => 'i' }, 'detail' => { type => 'i' }, 'dow0mon' => { type => 'b' }, 'dump-config' => { type => 'b' }, 'extended-detail' => { type => 'b', short => 'e' }, 'help' => { type => 'b' }, 'host-cnt' => { type => 'i', short => 'h' }, 'ignore-case' => { type => 'b', short => 'i' }, 'iso-date-time' => { type => 'b' }, 'mailq' => { type => 'b' }, 'no-no-msg-size' => { type => 'b' }, 'problems-first' => { type => 'b' }, 'pscrn-detail' => { type => 'i' }, # optional arg 'pscrn-stats' => { type => 'b' }, 'quiet' => { type => 'b', short => 'q' }, 'rej-add-from' => { type => 'b' }, 'rej-add-to' => { type => 'b' }, 'reject-detail' => { type => 'i' }, 'smtp-detail' => { type => 'i' }, 'smtpd-stats' => { type => 'b' }, 'smtpd-warning-detail' => { type => 'i' }, 'srs-mung' => { type => 'b' }, 'syslog-name' => { type => 's' }, 'unprocd-file' => { type => 's' }, 'use-orig-to' => { type => 'b' }, 'user-cnt' => { type => 'i', short => 'u' }, 'uucp-mung' => { type => 'b', short => 'm' }, 'verbose-msg-detail' => { type => 'b' }, 'verp-mung' => { type => 'i' }, # optional arg 'version' => { type => 'b' }, 'zero-fill' => { type => 'b' }, ); # Storage for actual values our %opts; # Dynamically build GetOptions argument list my @getopt_args; for my $long (sort keys %optionSpec) { my $type = $optionSpec{$long}->{type}; my $short = $optionSpec{$long}->{short}; my $opt_string = $long; if ($type eq 'f') { $opt_string .= "=s"; } elsif ($type ne 'b') { $opt_string .= "=$type"; } push @getopt_args, $opt_string => \$opts{$long}; if (defined $short) { my $short_string = $short; if ($type eq 'f') { $short_string .= "=s"; } elsif ($type ne 'b') { $short_string .= "=$type"; } push @getopt_args, $short_string => \$opts{$long}; } } # Ok, this is kind of ugly, but it solves a problem: We don't want to # *require* Config::Simple, but we also don't want to warn about it if it's # not needed, so... my $configFile; for (my $i = 0; $i < @ARGV; $i++) { if($ARGV[$i] eq '--config' && defined $ARGV[$i + 1]) { $configFile = $ARGV[$i + 1]; splice @ARGV, $i, 2; # Remove from ARGV last; } } if($haveConfigSimple) { # manually import the Config::Simple routines we want no warnings 'once'; *ConfigSimpleNew = sub { Config::Simple->new(@_) }; *ConfigSimpleVars = *Config::Simple::vars; *ConfigSimpleError = *Config::Simple::error; } elsif(defined($configFile)) { # If user specified --config but doesn't have Config::Simple # installed, die with friendly help message. die < length($b) ? $a : $b } keys %opts; # indent a little... my $fmtStr = sprintf "%%%ds =", length($longestKey) + 2; foreach my $key (sort keys(%opts)) { next if $key eq 'dump-config'; if($optionSpec{$key}{'type'} eq 'b') { printf "${fmtStr} %s\n", $key, defined($opts{$key})? "true" : ""; } elsif(looks_like_number($opts{$key})) { # internally: 0 == none, undefined == -1 == all my $val = $opts{$key} == 0? "none" : ($opts{$key} == -1? "all" : $opts{$key}); printf "${fmtStr} $val\n", $key; } else { printf "${fmtStr} %s\n", $key, defined($opts{$key})? $opts{$key} : ""; } } exit 0; } # debugging my $unProcd; if($opts{'unprocd-file'}) { open($unProcd, "> $opts{'unprocd-file'}") || die "couldn't open \"$opts{'unprocd-file'}\": $!\n"; } while(<>) { s/: \[ID \d+ [^\]]+\] /: /; # lose "[ID nnnnnn some.thing]" stuff my $logRmdr; next unless((($msgYr, $msgMon, $msgDay, $msgHr, $msgMin, $msgSec, $logRmdr) = line_matches_dates($_, $strtDate, $endDate)) == 7); # Snag first date seen ($fromDate{'yr'}, $fromDate{'mon'}, $fromDate{'day'}) = ($msgYr, $msgMon, $msgDay) unless($fromDate{'mon'}); # Snag last date seen ($thruDate{'yr'}, $thruDate{'mon'}, $thruDate{'day'}) = ($msgYr, $msgMon, $msgDay); unless((($cmd, $qid) = $logRmdr =~ m#^(?:postfix|$syslogName)(?:/(?:smtps|submission))?/([^\[:]*).*?: ([^:\s]+)#o) == 2 || (($cmd, $qid) = $logRmdr =~ m#^((?:postfix)(?:-script)?)(?:\[\d+\])?: ([^:\s]+)#o) == 2) { print $unProcd "[01]: $_" if $unProcd; next; } chomp; # the following test depends on one getting more than one message a # month--or at least that successive messages don't arrive on the # same month-day in successive months :-) unless($msgDay == $lastMsgDay) { $lastMsgDay = $msgDay; $revMsgDateStr = sprintf "%d%02d%02d", $msgYr, $msgMon, $msgDay; ++$dayCnt; if(defined($opts{'zero-fill'})) { ${$msgsPerDay{$revMsgDateStr}}[4] = 0; } } # regexp rejects happen in "cleanup" if($cmd eq "cleanup" && (my($rejSubTyp, $rejReas, $rejRmdr) = $logRmdr =~ /\/cleanup\[\d+\]: .*?\b((?:milter-)?reject|warning|hold|discard): (header|body|END-OF-MESSAGE) (.*)$/) == 3) { $rejRmdr =~ s/( from \S+?)?; from=<.*$// unless($opts{'verbose-msg-detail'}); # FIXME: In retrospect: I've no idea where I came up with the magic numbers I pass to this function. $rejRmdr = string_trimmer($rejRmdr, 64); if($rejSubTyp eq "reject" or $rejSubTyp eq "milter-reject") { ++$rejects{$cmd}{$rejReas}{$rejRmdr} unless($opts{'reject-detail'} == 0); ++$msgsRjctd; if($opts{'debug'}) { push(@{$qidTracker{$qid}{'status'}}, "\$cmd: $cmd, \$rejSubTyp: $rejSubTyp, --\$msgsRcvd"); ++$qidTracker{$qid}{'lateRejects'}; print STDERR "dbg: late reject \$rcvdMsg{$qid}{'size'}: $rcvdMsg{$qid}{'size'}\n" if $rcvdMsg{$qid}{'size'}; } --$msgsRcvd; # Late Reject: It will have already been counted as "Received," even though it ultimately is not } elsif($rejSubTyp eq "warning") { ++$warns{$cmd}{$rejReas}{$rejRmdr} unless($opts{'reject-detail'} == 0); ++$msgsWrnd; } elsif($rejSubTyp eq "hold") { ++$holds{$cmd}{$rejReas}{$rejRmdr} unless($opts{'reject-detail'} == 0); ++$msgsHld; } elsif($rejSubTyp eq "discard") { push(@{$qidTracker{$qid}{'status'}}, "\$cmd: $cmd, \$rejSubTyp: $rejSubTyp") if $opts{'debug'}; ++$discards{$cmd}{$rejReas}{$rejRmdr} unless($opts{'reject-detail'} == 0); ++$msgsDscrdd; } delete($rcvdMsg{$qid}); # We're done with this ++$rejPerHr[$msgHr]; ++${$msgsPerDay{$revMsgDateStr}}[4]; } elsif($qid eq 'warning') { (my $warnReas = $logRmdr) =~ s/^.*warning: //; unless($opts{'verbose-msg-detail'}) { # Condense smtpd and other warnings $warnReas =~ s/^(Unable to look up (?:MX|NS) host) for .+(: Host not found(?:,try again)?)/$1$2/ || $warnReas =~ s/^(hostname ).+ (does not resolve to address) [0-9A-F:\.]+$/$1$2/ || $warnReas =~ s/^(hostname ).+ (does not resolve to address) .+(: hostname nor servname provided, or not known)$/$1$2$3/ || $warnReas =~ s/^(Unable to look up (?:MX|NS) host ).+ (for (?:Sender address|Client host|Helo command)) .+(: (?:hostname nor servname provided, or not known|No address associated with hostname))$/$1$2$3/ || $warnReas =~ s/^(malformed domain name in resource data of MX record) for .*$/$1/ || $warnReas =~ s/^(numeric domain name in resource data of (?:MX|NS) record) for .*$/$1/ || $warnReas =~ s/^(numeric hostname): .*$/$1/ || $warnReas =~ s/^(valid_hostname: invalid character) .*$/$1/ || $warnReas =~ s/^[0-9A-F:\.]+ (address not listed for hostname) .*$/$1/ || $warnReas =~ s/^[0-9A-F]+: (queue file size limit exceeded)$/$1/ || $warnReas =~ s/^[^:]+: (SASL (?:LOGIN|PLAIN|CRAM-MD5) authentication failed(?:: Invalid authentication mechanism)?).*$/$1/ || $warnReas =~ s/^(Illegal address syntax )from .+ (in (?:MAIL|RCPT) command): .*$/$1$2/ || $warnReas =~ s/^(non-SMTP command) from .+?(: \S+) .*$/$1$2/ || $warnReas =~ s/^(Connection concurrency limit exceeded: \d+ )from \S+ (for service .+)$/$1$2/ || $warnReas =~ s/^[0-9A-F:\.]+ (hostname ).+ (verification failed: No address associated with hostname)$/$1$2/ || $warnReas =~ s/^[\w\.-]+: (RBL lookup error: Host or domain name not found. Name service error )for name=[\w\.-]+ (type=.+: Host not found, try again)$/$1$2/ || $warnReas =~ s/^.+((?:postfix-)?policyd-spf-perl: process )id \d+: (command time limit exceeded)$/$1$2/ || $warnReas =~ s/(process .+) pid \d+ (exit status \d+)/$1 $2/; } $warnReas = string_trimmer($warnReas, 66); unless($cmd eq "smtpd" && $opts{'smtpd-warning-detail'} == 0) { ++$warnings{$cmd}{$warnReas}; } } elsif($qid eq 'fatal') { (my $fatalReas = $logRmdr) =~ s/^.*fatal: //; $fatalReas = string_trimmer($fatalReas, 66); ++$fatals{$cmd}{$fatalReas}; } elsif($qid eq 'panic') { (my $panicReas = $logRmdr) =~ s/^.*panic: //; $panicReas = string_trimmer($panicReas, 66); ++$panics{$cmd}{$panicReas}; } elsif($qid eq 'reject') { proc_smtpd_reject($logRmdr, \%rejects, \$msgsRjctd, \$rejPerHr[$msgHr], \${$msgsPerDay{$revMsgDateStr}}[4]); } elsif($qid eq 'reject_warning') { proc_smtpd_reject($logRmdr, \%warns, \$msgsWrnd, \$rejPerHr[$msgHr], \${$msgsPerDay{$revMsgDateStr}}[4]); } elsif($qid eq 'hold') { proc_smtpd_reject($logRmdr, \%holds, \$msgsHld, \$rejPerHr[$msgHr], \${$msgsPerDay{$revMsgDateStr}}[4]); } elsif($qid eq 'discard') { proc_smtpd_reject($logRmdr, \%discards, \$msgsDscrdd, \$rejPerHr[$msgHr], \${$msgsPerDay{$revMsgDateStr}}[4]); } elsif($cmd eq 'master') { ++$masterMsgs{(split(/^.*master.*: /, $logRmdr))[1]}; } elsif($cmd eq 'smtpd' || $cmd eq 'postscreen') { if((my ($clientInfo)) = $logRmdr =~ /\[\d+\]: \w+: client=(.+?)(?:,|$)/) { # # Warning: this code in two places! # ++$rcvPerHr[$msgHr]; ++${$msgsPerDay{$revMsgDateStr}}[0]; if($opts{'debug'}) { push(@{$qidTracker{$qid}{'status'}}, "\$cmd: $cmd, ++\$msgsRcvd"); ++$qidTracker{$qid}{'rcvdCnt'}; } ++$msgsRcvd; $rcvdMsg{$qid}{'whence'} = gimme_domain($clientInfo); # Whence it came } elsif(my($rejSubTyp) = $logRmdr =~ /\[\d+\]: \w+: (reject(?:_warning)?|hold|discard): /) { if($rejSubTyp eq 'reject') { proc_smtpd_reject($logRmdr, \%rejects, \$msgsRjctd, \$rejPerHr[$msgHr], \${$msgsPerDay{$revMsgDateStr}}[4]); # Experimental unless($qid eq 'NOQUEUE') { if($opts{'debug'}) { push(@{$qidTracker{$qid}{'status'}}, "\$cmd: $cmd, \$rejSubTyp: $rejSubTyp, --\$msgsRcvd"); ++$qidTracker{$qid}{'lateRejects'}; print STDERR "dbg: late reject \$rcvdMsg{$qid}{'size'}: $rcvdMsg{$qid}{'size'}\n" if $rcvdMsg{$qid}{'size'}; } --$msgsRcvd # Late reject: It's been counted as received already } delete($rcvdMsg{$qid}) if($rcvdMsg{$qid}); # Late Reject: If it's rejected later in the game } elsif($rejSubTyp eq 'reject_warning') { proc_smtpd_reject($logRmdr, \%warns, \$msgsWrnd, \$rejPerHr[$msgHr], \${$msgsPerDay{$revMsgDateStr}}[4]); } elsif($rejSubTyp eq 'hold') { push(@{$qidTracker{$qid}{'status'}}, "\$cmd: $cmd, \$rejSubTyp: $rejSubTyp") if $opts{'debug'}; proc_smtpd_reject($logRmdr, \%holds, \$msgsHld, \$rejPerHr[$msgHr], \${$msgsPerDay{$revMsgDateStr}}[4]); } elsif($rejSubTyp eq 'discard') { push(@{$qidTracker{$qid}{'status'}}, "\$cmd: $cmd, \$rejSubTyp: $rejSubTyp") if $opts{'debug'}; proc_smtpd_reject($logRmdr, \%discards, \$msgsDscrdd, \$rejPerHr[$msgHr], \${$msgsPerDay{$revMsgDateStr}}[4]); } } else { if($cmd eq 'smtpd') { next unless(defined($opts{'smtpd-stats'})); if($logRmdr =~ /: connect from /) { $logRmdr =~ /\/smtpd\[(\d+)\]: /; @{$connTime{$1}} = ($msgYr, $msgMon + 1, $msgDay, $msgHr, $msgMin, $msgSec); } elsif($logRmdr =~ /: disconnect from /) { my ($pid, $hostID) = $logRmdr =~ /\/smtpd\[(\d+)\]: disconnect from (.+?)( unknown=\d+\/\d+)?( commands=\d+\/\d+)?$/; if(exists($connTime{$pid}) && ($hostID = gimme_domain($hostID))) { my($d, $h, $m, $s) = Delta_DHMS(@{$connTime{$pid}}, $msgYr, $msgMon + 1, $msgDay, $msgHr, $msgMin, $msgSec); delete($connTime{$pid}); # dispose of no-longer-needed item my $tSecs = (86400 * $d) + (3600 * $h) + (60 * $m) + $s; ++$smtpdPerHr[$msgHr][0]; $smtpdPerHr[$msgHr][1] += $tSecs; $smtpdPerHr[$msgHr][2] = $tSecs if($tSecs > $smtpdPerHr[$msgHr][2]); unless(${$smtpdPerDay{$revMsgDateStr}}[0]++) { ${$smtpdPerDay{$revMsgDateStr}}[1] = 0; ${$smtpdPerDay{$revMsgDateStr}}[2] = 0; } ${$smtpdPerDay{$revMsgDateStr}}[1] += $tSecs; ${$smtpdPerDay{$revMsgDateStr}}[2] = $tSecs if($tSecs > ${$smtpdPerDay{$revMsgDateStr}}[2]); if($hostID){ unless(${$smtpdPerDom{$hostID}}[0]++) { ${$smtpdPerDom{$hostID}}[1] = 0; ${$smtpdPerDom{$hostID}}[2] = 0; } ${$smtpdPerDom{$hostID}}[1] += $tSecs; ${$smtpdPerDom{$hostID}}[2] = $tSecs if($tSecs > ${$smtpdPerDom{$hostID}}[2]); } ++$smtpdConnCnt; $smtpdTotTime += $tSecs; } } } elsif($cmd eq 'postscreen' && (defined $opts{'pscrn-stats'} || $opts{'pscrn-detail'})) { my ($pscrnAct, $clientIP, $clientPort, $pscrnAddl, $capCnt); print STDERR "\n" if($opts{'debug'}); print STDERR "\$opts{'pscrn-stats'}: " . ($opts{'pscrn-stats'} // 0) .", \$opts{'pscrn-detail'}: $opts{'pscrn-detail'}\n" if($opts{'debug'}); foreach my $regEx (@pscrnRegexs) { print STDERR "\$regEx->{'expr'}: \"$regEx->{'expr'}\"\n" if($opts{'debug'}); if(($capCnt = (($pscrnAct, $clientIP, $clientPort, $pscrnAddl) = $logRmdr =~ /$regEx->{'expr'}/)) >= 3) { ++$regEx->{'cnt'}; # Not (currently?) used if($opts{'debug'}) { foreach ($pscrnAct, $clientIP, $clientPort, $pscrnAddl) { print STDERR "capt: \"$_\"\n" if(defined $_ ); } } last; } } print STDERR "\$capCnt: $capCnt\n\$logRmdr: \"$logRmdr\"\n" if($opts{'debug'}); my $bump_capt_cnt = sub { if($capCnt == 4) { print STDERR "Bumping \$pscrnHits{\"$pscrnAct $pscrnAddl\"}{\"$clientIP\"} on \$logRmdr: \"$logRmdr\"\n" if($opts{'debug'}); ++$pscrnHits{"$pscrnAct $pscrnAddl"}{$clientIP} if($opts{'pscrn-detail'}); print STDERR "\$cmd: \"$cmd\", \$logRmdr: \"$logRmdr\"\n" if($opts{'debug'}); } else { print STDERR "Bumping \$pscrnHits{\"$pscrnAct\"}{\"$clientIP\"} on \$logRmdr: \"$logRmdr\"\n" if($opts{'debug'}); ++$pscrnHits{$pscrnAct}{$clientIP} if($opts{'pscrn-detail'}); print STDERR "\$cmd: \"$cmd\", \$logRmdr: \"$logRmdr\"\n" if($opts{'debug'}); } }; if($capCnt == 3) { if($pscrnAct eq 'CONNECT') { @{$connTime{"$clientIP:$clientPort"}} = ($msgYr, $msgMon + 1, $msgDay, $msgHr, $msgMin, $msgSec); print STDERR "\@{\$connTime{\"$clientIP:$clientPort\"}}: " . join(' / ', @{$connTime{"$clientIP:$clientPort"}}) . "\n" if($opts{'debug'}); } elsif($pscrnAct =~ /^(DISCONNECT|HANGUP|PASS (NEW|OLD))$/) { print STDERR "DISCO: \$pscrnAct: \"$pscrnAct\", \$clientIP: \"$clientIP\", \$clientPort: \"$clientPort\"\n" if($opts{'debug'}); if(exists($connTime{"$clientIP:$clientPort"})) { my($d, $h, $m, $s) = Delta_DHMS(@{$connTime{"$clientIP:$clientPort"}}, $msgYr, $msgMon + 1, $msgDay, $msgHr, $msgMin, $msgSec); delete($connTime{"$clientIP:$clientPort"}); # dispose of no-longer-needed item my $tSecs = (86400 * $d) + (3600 * $h) + (60 * $m) + $s; print STDERR "DISCONNECT: \$tSecs: $tSecs\n" if($opts{'debug'}); ++$pscrnPerHr[$msgHr][0]; $pscrnPerHr[$msgHr][1] += $tSecs; $pscrnPerHr[$msgHr][2] = $tSecs if($tSecs > $pscrnPerHr[$msgHr][2]); unless(${$pscrnPerDay{$revMsgDateStr}}[0]++) { ${$pscrnPerDay{$revMsgDateStr}}[1] = 0; ${$pscrnPerDay{$revMsgDateStr}}[2] = 0; } ${$pscrnPerDay{$revMsgDateStr}}[1] += $tSecs; ${$pscrnPerDay{$revMsgDateStr}}[2] = $tSecs if($tSecs > ${$pscrnPerDay{$revMsgDateStr}}[2]); unless(${$pscrnPerIP{$clientIP}}[0]++) { ${$pscrnPerIP{$clientIP}}[1] = 0; ${$pscrnPerIP{$clientIP}}[2] = 0; } ${$pscrnPerIP{$clientIP}}[1] += $tSecs; ${$pscrnPerIP{$clientIP}}[2] = $tSecs if($tSecs > ${$pscrnPerIP{$clientIP}}[2]); ++$pscrnConnCnt; $pscrnTotTime += $tSecs; # Want the per-postscreen-action stats? $bump_capt_cnt->() if($opts{'pscrn-detail'} && $pscrnAct =~ /^PASS (NEW|OLD)$/); } } else { $bump_capt_cnt->() if($opts{'pscrn-detail'}); # Want the per-postscreen-action stats? } } elsif($capCnt == 4) { $bump_capt_cnt->() if($opts{'pscrn-detail'}); # Want the per-postscreen-action stats? } else { print $unProcd "[02]: $_\n" if($unProcd && (defined $opts{'pscrn-stats'} || $opts{'pscrn-detail'})); } } } } else { my $toRmdr; if((($addr, $size) = $logRmdr =~ /from=<([^>]*)>, size=(\d+)/) == 2) { ++$sizeDataExists; # Flag for orphan rcvdMsg cleanup: Older logs won't have size data next if($rcvdMsg{$qid}{'size'}); # avoid double-counting! if($addr) { if($opts{'uucp-mung'} && $addr =~ /^(.*!)*([^!]+)!([^!@]+)@([^\.]+)$/) { $addr = "$4!" . ($1? "$1" : "") . $3 . "\@$2"; } $addr =~ s/(@.+)/\L$1/ unless($opts{'ignore-case'}); $addr = lc($addr) if($opts{'ignore-case'}); $addr = verp_mung($addr); $addr = srs_mung($addr); } else { $addr = "from=<>"; } $rcvdMsg{$qid}{'size'} = $size; push(@{$msgDetail{$qid}}, $addr) if($opts{'extended-detail'}); # Avoid counting forwards if($rcvdMsg{$qid}{'whence'}) { # Get the domain out of the sender's address. If there is # none (e.g.: "from=<>"): Use the client domain/IP-address my $domAddr; if($addr eq 'from=<>' || ($domAddr = fold_domain($addr)) eq '') { $domAddr = $rcvdMsg{$qid}{'whence'}; } ++$sendgDomCnt unless(${$sendgDom{$domAddr}}[MSG_CNT_I]); ++${$sendgDom{$domAddr}}[MSG_CNT_I]; ${$sendgDom{$domAddr}}[MSG_SIZE_I] += $size; ++$sendgUserCnt unless(${$sendgUser{$addr}}[MSG_CNT_I]); ++${$sendgUser{$addr}}[MSG_CNT_I]; ${$sendgUser{$addr}}[MSG_SIZE_I] += $size; $sizeRcvd += $size; } } elsif((($addr, $orig_to, $relay, $delay, $status, $toRmdr) = $logRmdr =~ /to=<([^>]*)>, (?:orig_to=<([^>]*)>, )?relay=([^,]+), (?:conn_use=[^,]+, )?delay=([^,]+), (?:delays=[^,]+, )?(?:dsn=[^,]+, )?status=(\S+)(.*)$/) >= 4) { $addr = $orig_to if($opts{'use-orig-to'} && $orig_to); if($opts{'uucp-mung'} && $addr =~ /^(.*!)*([^!]+)!([^!@]+)@([^\.]+)$/) { $addr = "$4!" . ($1? "$1" : "") . $3 . "\@$2"; } $addr =~ s/(@.+)/\L$1/ unless($opts{'ignore-case'}); $addr = lc($addr) if($opts{'ignore-case'}); $relay = lc($relay) if($opts{'ignore-case'}); my $domAddr = fold_domain($addr); # get domain only if($status eq 'sent') { # was it actually forwarded, rather than delivered? if(my ($newQid) = $toRmdr =~ /\(forwarded as ([^\)]+)\)/) { push(@{$qidTracker{$qid}{'status'}}, "\$cmd: $cmd, \$status: $status, forwarded as new qid $1, ++\$msgsFwdd") if $opts{'debug'}; ++$msgsFwdd; delete($rcvdMsg{$qid}); # We're done with this next; } ++$recipDomCnt unless(${$recipDom{$domAddr}}[MSG_CNT_I]); ++${$recipDom{$domAddr}}[MSG_CNT_I]; ${$recipDom{$domAddr}}[MSG_DLY_AVG_I] += $delay; if(! ${$recipDom{$domAddr}}[MSG_DLY_MAX_I] || $delay > ${$recipDom{$domAddr}}[MSG_DLY_MAX_I]) { ${$recipDom{$domAddr}}[MSG_DLY_MAX_I] = $delay } ++$recipUserCnt unless(${$recipUser{$addr}}[MSG_CNT_I]); ++${$recipUser{$addr}}[MSG_CNT_I]; ++$dlvPerHr[$msgHr]; ++${$msgsPerDay{$revMsgDateStr}}[1]; if($opts{'debug'}) { push(@{$qidTracker{$qid}{'status'}}, "\$cmd: $cmd, \$status: $status, ++\$msgsDlvrd"); ++$qidTracker{$qid}{'dlvrdCnt'}; } ++$msgsDlvrd; if($rcvdMsg{$qid}{'size'}) { ${$recipDom{$domAddr}}[MSG_SIZE_I] += $rcvdMsg{$qid}{'size'}; ${$recipUser{$addr}}[MSG_SIZE_I] += $rcvdMsg{$qid}{'size'}; $sizeDlvrd += $rcvdMsg{$qid}{'size'}; } else { ${$recipDom{$domAddr}}[MSG_SIZE_I] += 0; ${$recipUser{$addr}}[MSG_SIZE_I] += 0; $noMsgSize{$qid} = $addr unless($opts{'no-no-msg-size'}); push(@{$msgDetail{$qid}}, "(sender not in log)") if($opts{'extended-detail'}); # put this back later? mebbe with -v? # msg_warn("no message size for qid: $qid"); } push(@{$msgDetail{$qid}}, $addr) if($opts{'extended-detail'}); } elsif($status eq 'deferred') { unless($opts{'deferral-detail'} == 0) { my ($deferredReas) = $logRmdr =~ /, status=deferred \(([^\)]+)/; if(!defined($opts{'verbose-msg-detail'})) { my ($host, $reason, $moreReason); # More ugliness :/ unless((($host, $reason) = $deferredReas =~ /^host (\S+) (?:said|refused to talk to me): ([^(]+)/) || (($host, $reason) = $deferredReas =~ /^(?:delivery temporarily suspended: )?connect to (.+?(?::\d+)?): ([^\)]+)$/) || (($host, $reason) = $deferredReas =~ /^cannot (?:append to file|update mailbox) ([^:.]+)[:.] (.+)$/) || (($reason, $host, $moreReason) = $deferredReas =~ /^.*(Name service error )for (?:domain |name=)?([^: ]+):? (.+)$/) || (($reason, $host, $moreReason) = $deferredReas =~ /^((?:conversation|lost connection) )with (\S+) ((?:timed out )?while (receiving|sending) .+)$/) || (($reason, $host, $moreReason) = $deferredReas =~ /^(delivery temporarily suspended: )connect to ([^:]+): (.+)$/) ) { $host = "(unknown host)"; $reason = $deferredReas; } $host =~ s/:\d{2,3}$//; # Strip trailing port numbers $reason .= $moreReason if($moreReason); # ick # Finally... $reason = said_string_trimmer($reason, 66); ++$deferred{$cmd}{$host}{$reason}; } else { ++$deferred{$cmd}{$deferredReas}; } } ++$dfrPerHr[$msgHr]; ++${$msgsPerDay{$revMsgDateStr}}[2]; push(@{$qidTracker{$qid}{'status'}}, "\$cmd: $cmd, \$status: $status, ++\$msgsDfrd") if $opts{'debug'}; ++$msgsDfrdCnt; ++$msgsDfrd unless($msgDfrdFlgs{$qid}++); ++${$recipDom{$domAddr}}[MSG_DFRS_I]; if(! ${$recipDom{$domAddr}}[MSG_DLY_MAX_I] || $delay > ${$recipDom{$domAddr}}[MSG_DLY_MAX_I]) { ${$recipDom{$domAddr}}[MSG_DLY_MAX_I] = $delay } } elsif($status eq 'bounced') { unless($opts{'bounce-detail'} == 0) { my ($bounceReas) = $logRmdr =~ /, status=bounced \((.+)\)/; unless(defined($opts{'verbose-msg-detail'})) { $bounceReas = said_string_trimmer($bounceReas, 66); } ++$bounced{$relay}{$bounceReas}; } ++$bncPerHr[$msgHr]; ++${$msgsPerDay{$revMsgDateStr}}[3]; push(@{$qidTracker{$qid}{'status'}}, "\$cmd: $cmd, \$status: $status, ++\$msgsBncd") if $opts{'debug'}; ++$msgsBncd; } else { print $unProcd "[03]: $_\n" if $unProcd; } } elsif($cmd eq 'pickup' && $logRmdr =~ /: (sender|uid)=/) { # # Warning: this code in two places! # ++$rcvPerHr[$msgHr]; ++${$msgsPerDay{$revMsgDateStr}}[0]; if($opts{'debug'}) { push(@{$qidTracker{$qid}{'status'}}, "\$cmd: $cmd, ++\$msgsRcvd"); ++$qidTracker{$qid}{'rcvdCnt'}; } ++$msgsRcvd; $rcvdMsg{$qid}{'whence'} = "pickup"; # Whence it came } elsif($cmd eq 'smtp' && $opts{'smtp-detail'} != 0) { # Was an IPv6 problem here if($logRmdr =~ /.* connect to (\S+?): ([^;]+); address \S+ port.*$/) { ++$smtpMsgs{lc($2)}{$1}; } elsif($logRmdr =~ /.* connect to ([^[]+)\[\S+?\]: (.+?) \(port \d+\)$/) { ++$smtpMsgs{lc($2)}{$1}; } else { print $unProcd "[04]: $_\n" if $unProcd; } } elsif($cmd =~ /^n?qmgr$/ && $logRmdr =~ /\bremoved$/) { delete($rcvdMsg{$qid}); # We're done with this } else { print $unProcd "[05]: $_\n" if $unProcd; } } } # Experimental heuristic: # # If messages were "received" but undelivered, unforwarded, and not # rejected in cleanup, odds are nothing was ever really received—not # even a 0-length message. # # N.B.: This may result in wonky outcomes for older Postfix logs # where some of the data in newer logs isn't availble. # if(my $noSizeCnt = scalar grep { !exists $rcvdMsg{$_}{'size'} } keys %rcvdMsg) { foreach my $qid (keys %rcvdMsg) { push(@{$qidTracker{$qid}{'status'}}, "No \$rcvdMsg{$qid}{'size'} at end of processing: --\$msgsRcvd") if $opts{'debug'}; } $msgsRcvd -= $noSizeCnt if $sizeDataExists; } # Extensive queue I.D. lifetime tracking if($opts{'debug'} && scalar keys %qidTracker) { my ($qidCnt, $rcvdDlvrd, $dlvrdCnt, $addlDlvr, $multiDlvrCnt, $noSizeCnt, $addlRcvd, $multiRcvdCnt, $noRcvdCnt, $lateRejects) = ((0) x 10); foreach my $qid (sort keys %qidTracker) { ++$qidCnt; print STDERR "qid: $qid\n"; if(exists $qidTracker{$qid}{'dlvrdCnt'}) { ++$rcvdDlvrd; $dlvrdCnt += $qidTracker{$qid}{'dlvrdCnt'}; if($qidTracker{$qid}{'dlvrdCnt'} > 1) { $addlDlvr += $qidTracker{$qid}{'dlvrdCnt'} - 1; ++$multiDlvrCnt; } print STDERR " delivered cnt: $qidTracker{$qid}{'dlvrdCnt'}\n" } else { print STDERR " delivered cnt: 0\n"; } if(! $qidTracker{$qid}{'rcvdCnt'}) { print STDERR " received cnt: 0\n"; ++$noRcvdCnt; } elsif($qidTracker{$qid}{'rcvdCnt'} > 1) { $addlRcvd += $qidTracker{$qid}{'rcvdCnt'} - 1; ++$multiRcvdCnt; print STDERR " received cnt: $qidTracker{$qid}{'rcvdCnt'}\n"; } $lateRejects += $qidTracker{$qid}{'lateRejects'} if $qidTracker{$qid}{'lateRejects'}; foreach my $event (@{$qidTracker{$qid}{'status'}}) { print STDERR " $event\n"; } if(exists $rcvdMsg{$qid} && ! exists $rcvdMsg{$qid}{'size'}) { print STDERR " no size data\n"; ++$noSizeCnt; } } printf STDERR "\n %6d%s qids\n", adj_int_units($qidCnt); printf STDERR " %6d%s qids delivered\n", adj_int_units($rcvdDlvrd); printf STDERR " %6d%s qids w/multi-deliveries\n", adj_int_units($multiDlvrCnt); printf STDERR " %6d%s total add'l deliveries\n", adj_int_units($addlDlvr); printf STDERR " %6d%s qids w/multi-received\n", adj_int_units($multiRcvdCnt); printf STDERR " %6d%s total add'l received\n", adj_int_units($addlRcvd); printf STDERR " %6d%s qids w/no received count\n", adj_int_units($noRcvdCnt); printf STDERR " %6d%s forwarded\n", adj_int_units($msgsFwdd); printf STDERR " %6d%s delivered by cnt\n", adj_int_units($dlvrdCnt); printf STDERR " %6d%s discarded\n", adj_int_units($msgsDscrdd); printf STDERR " %6d%s qids w/no size data\n", adj_int_units($noSizeCnt); printf STDERR " %6d%s late rejects (rec'd but not dlvrd)\n", adj_int_units($lateRejects); } # debugging if($unProcd) { close($unProcd) || warn "problem closing \"$opts{'unprocd-file'}\": $!\n"; } # Calculate percentage of messages rejected and discarded my $msgsRjctdPct = 0; my $msgsDscrddPct = 0; if(my $msgsTotal = $msgsDlvrd + $msgsRjctd + $msgsDscrdd) { $msgsRjctdPct = int(($msgsRjctd/$msgsTotal) * 100); $msgsDscrddPct = int(($msgsDscrdd/$msgsTotal) * 100); } print "Postfix Log Summaries"; if (defined($thruDate{'mon'}) && defined($thruDate{'day'})) { # We can safely assume that if we've a thruDate we've a fromDate my $monName = $monthNames[ $fromDate{'mon'}]; my $day = $fromDate{'day'}; my $yr = $fromDate{'yr'} // $thisYr; # st00pid Day_of_Week requires months indexed from 1, not 0 my $dowIdx = Day_of_Week($yr, $fromDate{'mon'} + 1, $day); my $dowStr = $dowNames[$dowIdx]; $day =~ s/^0//; print " for $dowStr, $monName $day $yr"; # One or both of these could be undefined, so... my $fromYr = $fromDate{'yr'} // $thisYr; my $thruYr = $thruDate{'yr'} // $thisYr; unless($fromDate{'mon'} == $thruDate{'mon'} && $fromDate{'day'} == $thruDate{'day'} && $fromYr == $thruYr) { my $monName = $monthNames[ $thruDate{'mon'}]; my $day = $thruDate{'day'}; my $yr = $thruDate{'yr'} // $thisYr; my $dowIdx = Day_of_Week($yr, $thruDate{'mon'} + 1, $day); my $dowStr = $dowNames[$dowIdx]; $day =~ s/^0//; print " through $dowStr, $monName $day $yr"; } } print "\n"; # Did they use any deprecated "_" options? if(scalar @deprecated) { print "\n"; print "$_\n" foreach (@deprecated); } print_subsect_title("Grand Totals"); print "messages\n\n"; printf " %6d%s received\n", adj_int_units($msgsRcvd); printf " %6d%s delivered\n", adj_int_units($msgsDlvrd); printf " %6d%s forwarded\n", adj_int_units($msgsFwdd); printf " %6d%s deferred", adj_int_units($msgsDfrd); printf " (%d%s deferrals)", adj_int_units($msgsDfrdCnt) if($msgsDfrdCnt); print "\n"; printf " %6d%s bounced\n", adj_int_units($msgsBncd); printf " %6d%s rejected (%d%%)\n", adj_int_units($msgsRjctd), $msgsRjctdPct; printf " %6d%s reject warnings\n", adj_int_units($msgsWrnd); printf " %6d%s held\n", adj_int_units($msgsHld); printf " %6d%s discarded (%d%%)\n", adj_int_units($msgsDscrdd), $msgsDscrddPct; print "\n"; printf " %6d%s bytes received\n", adj_int_units($sizeRcvd); printf " %6d%s bytes delivered\n", adj_int_units($sizeDlvrd); printf " %6d%s senders\n", adj_int_units($sendgUserCnt); printf " %6d%s sending hosts/domains\n", adj_int_units($sendgDomCnt); printf " %6d%s recipients\n", adj_int_units($recipUserCnt); printf " %6d%s recipient hosts/domains\n", adj_int_units($recipDomCnt); if(defined($opts{'smtpd-stats'})) { print "\nsmtpd\n\n"; printf " %6d%s connections\n", adj_int_units($smtpdConnCnt); printf " %6d%s hosts/domains\n", adj_int_units(int(keys %smtpdPerDom)); printf " %6d avg. connect time (seconds)\n", $smtpdConnCnt > 0? ($smtpdTotTime / $smtpdConnCnt) + .5 : 0; { my ($sec, $min, $hr) = get_smh($smtpdTotTime); printf " %2d:%02d:%02d total connect time\n", $hr, $min, $sec; } } if(defined($opts{'pscrn-stats'})) { print "\npostscreen\n\n"; printf " %6d%s connections\n", adj_int_units($pscrnConnCnt); printf " %6d%s IP addresses\n", adj_int_units(int(keys %pscrnPerIP)); printf " %6d avg. connect time (seconds)\n", ($pscrnConnCnt && $pscrnConnCnt > 0)? ($pscrnTotTime / $pscrnConnCnt) + .5 : 0; { my ($sec, $min, $hr) = get_smh($pscrnTotTime); printf " %2d:%02d:%02d total connect time\n", $hr, $min, $sec; } } print "\n"; print_problems_reports() if(defined($opts{'problems-first'})); print_per_day_summary(\%msgsPerDay) if($dayCnt > 1); print_per_hour_summary(\@rcvPerHr, \@dlvPerHr, \@dfrPerHr, \@bncPerHr, \@rejPerHr, $dayCnt); print_recip_domain_summary(\%recipDom, $opts{'host-cnt'}); print_sending_domain_summary(\%sendgDom, $opts{'host-cnt'}); if(defined($opts{'smtpd-stats'})) { print_per_day_smtpd(\%smtpdPerDay, $dayCnt) if($dayCnt > 1); print_per_hour_smtpd(\@smtpdPerHr, $dayCnt); print_domain_smtpd_summary(\%smtpdPerDom, $opts{'host-cnt'}); } print_user_data(\%sendgUser, "Senders by message count", MSG_CNT_I, $opts{'user-cnt'}, $opts{'quiet'}); print_user_data(\%recipUser, "Recipients by message count", MSG_CNT_I, $opts{'user-cnt'}, $opts{'quiet'}); print_user_data(\%sendgUser, "Senders by message size", MSG_SIZE_I, $opts{'user-cnt'}, $opts{'quiet'}); print_user_data(\%recipUser, "Recipients by message size", MSG_SIZE_I, $opts{'user-cnt'}, $opts{'quiet'}); print_hash_by_key(\%noMsgSize, "Messages with no size data", 0, 1); print_problems_reports() unless(defined($opts{'problems-first'})); print_detailed_msg_data(\%msgDetail, "Message detail", $opts{'quiet'}) if($opts{'extended-detail'}); # Print "problems" reports sub print_problems_reports { unless($opts{'deferral-detail'} == 0) { print_nested_hash(\%deferred, "message deferral detail", $opts{'deferral-detail'}, $opts{'quiet'}); } unless($opts{'bounce-detail'} == 0) { print_nested_hash(\%bounced, "message bounce detail (by relay)", $opts{'bounce-detail'}, $opts{'quiet'}); } unless($opts{'reject-detail'} == 0) { print_nested_hash(\%rejects, "message reject detail", $opts{'reject-detail'}, $opts{'quiet'}); print_nested_hash(\%warns, "message reject warning detail", $opts{'reject-detail'}, $opts{'quiet'}); print_nested_hash(\%holds, "message hold detail", $opts{'reject-detail'}, $opts{'quiet'}); print_nested_hash(\%discards, "message discard detail", $opts{'reject-detail'}, $opts{'quiet'}); } unless($opts{'smtp-detail'} == 0) { print_nested_hash(\%smtpMsgs, "smtp delivery failures", $opts{'smtp-detail'}, $opts{'quiet'}); } unless($opts{'smtpd-warning-detail'} == 0) { print_nested_hash(\%warnings, "Warnings", $opts{'smtpd-warning-detail'}, $opts{'quiet'}); } print_nested_hash(\%pscrnHits, "postscreen actions", $opts{'pscrn-detail'}, $opts{'quiet'}) if($opts{'pscrn-detail'}); print_nested_hash(\%fatals, "Fatal Errors", 0, $opts{'quiet'}); print_nested_hash(\%panics, "Panics", 0, $opts{'quiet'}); print_hash_by_cnt_vals(\%masterMsgs,"Master daemon messages", 0, $opts{'quiet'}); } if($opts{'mailq'}) { # flush stdout first cuz of asynchronousity $| = 1; print_subsect_title("Current Mail Queue"); system($mailqCmd); } # print "per-day" traffic summary # (done in a subroutine only to keep main-line code clean) sub print_per_day_summary { my($msgsPerDay) = @_; my $value; print_subsect_title("Per-Day Traffic Summary"); print < $b } keys(%$msgsPerDay)) { my ($msgYr, $msgMon, $msgDay) = unpack("A4 A2 A2", $_); if($opts{'iso-date-time'}) { printf " %04d-%02d-%02d ", $msgYr, $msgMon + 1, $msgDay } else { my $msgMonStr = $monthNames[$msgMon]; printf " $msgMonStr %2d $msgYr", $msgDay; } foreach $value (@{$msgsPerDay->{$_}}) { printf " %6d%s", adj_int_units($value // 0); } print "\n"; } } # print "per-hour" traffic summary # (done in a subroutine only to keep main-line code clean) sub print_per_hour_summary { my ($rcvPerHr, $dlvPerHr, $dfrPerHr, $bncPerHr, $rejPerHr, $dayCnt) = @_; my $reportType = $dayCnt > 1? 'Daily Average' : 'Summary'; my ($hour, $value); print_subsect_title("Per-Hour Traffic $reportType"); print < 0 && $rptCnt > $cnt? "top $cnt of " : "") . "$rptCnt)"; my $avgDly; print_subsect_title("Host/Domain Summary: Message Delivery $topCnt"); print <{$_}}[MSG_CNT_I]) { $avgDly = (${$hashRef->{$_}}[MSG_DLY_AVG_I] / ${$hashRef->{$_}}[MSG_CNT_I]); } else { $avgDly = 0; } printf " %6d%s %6d%s %6d%s %5.1f %s %5.1f %s %s\n", adj_int_units(${$hashRef->{$_}}[MSG_CNT_I]), adj_int_units(${$hashRef->{$_}}[MSG_SIZE_I]), adj_int_units(${$hashRef->{$_}}[MSG_DFRS_I]), adj_time_units($avgDly), adj_time_units(${$hashRef->{$_}}[MSG_DLY_MAX_I]), $_; last if --$cnt == 0; } } # print "per-sender-domain" traffic summary # (done in a subroutine only to keep main-line code clean) sub print_sending_domain_summary { use vars '$hashRef'; local($hashRef) = $_[0]; my($cnt) = $_[1]; return if($cnt == 0); my $rptCnt = keys %{$hashRef}; my $topCnt = "(" . ($cnt > 0 && $rptCnt > $cnt? "top $cnt of " : "") . "$rptCnt)"; print_subsect_title("Host/Domain Summary: Messages Received $topCnt"); print <{$_}}[MSG_CNT_I]), adj_int_units(${$hashRef->{$_}}[MSG_SIZE_I]), $_; last if --$cnt == 0; } } # print "per-user" data sorted in descending order # order (i.e.: highest first) sub print_user_data { my($hashRef, $title, $index, $cnt, $quiet) = @_; my $dottedLine; return if($cnt == 0); my $rptCnt = keys %{$hashRef}; $title .= " (" . ($cnt > 0 && $rptCnt > $cnt? "top $cnt of " : "") . "$rptCnt)"; return unless($rptCnt || ! $quiet); $title .= "\n" . "-" x length($title) if($rptCnt); printf "\n$title\n"; foreach (map { $_->[0] } sort { $b->[1] <=> $a->[1] || $a->[2] cmp $b->[2] } map { [ $_, $hashRef->{$_}[$index], normalize_host($_) ] } (keys(%$hashRef))) { printf " %6d%s %s\n", adj_int_units(${$hashRef->{$_}}[$index]), $_; last if --$cnt == 0; } } # print "per-hour" smtpd connection summary # (done in a subroutine only to keep main-line code clean) sub print_per_hour_smtpd { my ($smtpdPerHr, $dayCnt) = @_; my ($hour, $value); if($dayCnt > 1) { print_subsect_title("Per-Hour SMTPD Connection Daily Average"); print <[0] || next; my $avg = int($smtpdPerHr[$hour]->[0]? ($smtpdPerHr[$hour]->[1]/$smtpdPerHr[$hour]->[0]) + .5 : 0); if($dayCnt > 1) { $smtpdPerHr[$hour]->[0] /= $dayCnt; $smtpdPerHr[$hour]->[1] /= $dayCnt; $smtpdPerHr[$hour]->[0] += .5; $smtpdPerHr[$hour]->[1] += .5; } my($sec, $min, $hr) = get_smh($smtpdPerHr[$hour]->[1]); if($opts{'iso-date-time'}) { printf " %02d:00-%02d:00", $hour, $hour + 1; } else { printf " %02d00-%02d00 ", $hour, $hour + 1; } printf " %6d%s %2d:%02d:%02d", adj_int_units($smtpdPerHr[$hour]->[0]), $hr, $min, $sec; if($dayCnt < 2) { printf " %6ds %6ds", $avg, $smtpdPerHr[$hour]->[2]; } print "\n"; } } # print "per-day" smtpd connection summary # (done in a subroutine only to keep main-line code clean) sub print_per_day_smtpd { my ($smtpdPerDay, $dayCnt) = @_; print_subsect_title("Per-Day SMTPD Connection Summary"); print < $b } keys(%$smtpdPerDay)) { my ($msgYr, $msgMon, $msgDay) = unpack("A4 A2 A2", $_); if($opts{'iso-date-time'}) { printf " %04d-%02d-%02d ", $msgYr, $msgMon + 1, $msgDay } else { my $msgMonStr = $monthNames[$msgMon]; printf " $msgMonStr %2d $msgYr", $msgDay; } my $avg = (${$smtpdPerDay{$_}}[1]/${$smtpdPerDay{$_}}[0]) + .5; my($sec, $min, $hr) = get_smh(${$smtpdPerDay{$_}}[1]); printf " %6d%s %2d:%02d:%02d %6ds %6ds\n", adj_int_units(${$smtpdPerDay{$_}}[0]), $hr, $min, $sec, $avg, ${$smtpdPerDay{$_}}[2]; } } # print "per-domain-smtpd" connection summary # (done in a subroutine only to keep main-line code clean) sub print_domain_smtpd_summary { use vars '$hashRef'; local($hashRef) = $_[0]; my($cnt) = $_[1]; return if($cnt == 0); my $rptCnt = keys %{$hashRef}; my $topCnt = "(" . ($cnt > 0 && $rptCnt > $cnt? "top $cnt of " : "") . "$rptCnt)"; my $avgDly; print_subsect_title("Host/Domain Summary: SMTPD Connections $topCnt"); print <{$_}}[1]/${$hashRef->{$_}}[0]) + .5; my ($sec, $min, $hr) = get_smh(${$hashRef->{$_}}[1]); printf " %6d%s %2d:%02d:%02d %6ds %6ds %s\n", adj_int_units(${$hashRef->{$_}}[0]), $hr, $min, $sec, $avg, ${$hashRef->{$_}}[2], $_; last if --$cnt == 0; } } # print hash contents sorted by numeric values in descending # order (i.e.: highest first) sub print_hash_by_cnt_vals { my($hashRef, $title, $cnt, $quiet) = @_; my $dottedLine; $title = sprintf "%s%s", $cnt? "top $cnt " : "", $title; unless(%$hashRef) { return if($quiet); $dottedLine = ": none"; } else { $dottedLine = "\n" . "-" x length($title); } printf "\n$title$dottedLine\n"; really_print_hash_by_cnt_vals($hashRef, $cnt, ' '); } # print hash contents sorted by key in ascending order sub print_hash_by_key { my($hashRef, $title, $cnt, $quiet) = @_; my $dottedLine; $title = sprintf "%s%s", $cnt? "first $cnt " : "", $title; unless(%$hashRef) { return if($quiet); $dottedLine = ": none"; } else { $dottedLine = "\n" . "-" x length($title); } printf "\n$title$dottedLine\n"; foreach (sort keys(%$hashRef)) { printf " %s %s\n", $_, $hashRef->{$_}; last if --$cnt == 0; } } # print "nested" hashes sub print_nested_hash { my($hashRef, $title, $cnt, $quiet) = @_; my $dottedLine; unless(%$hashRef) { return if($quiet); $dottedLine = ": none"; } else { $dottedLine = "\n" . "-" x length($title); } printf "\n$title$dottedLine\n"; walk_nested_hash($hashRef, $cnt, 0); } # "walk" a "nested" hash sub walk_nested_hash { my ($hashRef, $cnt, $level) = @_; $level += 2; my $indents = ' ' x $level; my ($keyName, $hashVal) = each(%$hashRef); if(ref($hashVal) eq 'HASH') { foreach (sort keys %$hashRef) { print "$indents$_"; # If the next hash is finally the data, total the # counts for the report and print my $hashVal2 = (each(%{$hashRef->{$_}}))[1]; keys(%{$hashRef->{$_}}); # "reset" hash iterator unless(ref($hashVal2) eq 'HASH') { my $rptLines = keys %{$hashRef->{$_}}; my $rptCnt = reduce { $a + $b } 0, values %{$hashRef->{$_}}; print " ("; print "top $cnt of " if($cnt > 0 && $rptLines > $cnt); print "$rptCnt)"; } print "\n"; walk_nested_hash($hashRef->{$_}, $cnt, $level); } } else { really_print_hash_by_cnt_vals($hashRef, $cnt, $indents); } } # print per-message info in excruciating detail :-) sub print_detailed_msg_data { use vars '$hashRef'; local($hashRef) = $_[0]; my($title, $quiet) = @_[1,2]; my $dottedLine; unless(%$hashRef) { return if($quiet); $dottedLine = ": none"; } else { $dottedLine = "\n" . "-" x length($title); } printf "\n$title$dottedLine\n"; foreach (sort by_domain_then_user keys(%$hashRef)) { printf " %s %s\n", $_, shift(@{$hashRef->{$_}}); foreach (@{$hashRef->{$_}}) { print " $_\n"; } print "\n"; } } # *really* print hash contents sorted by numeric values in descending # order (i.e.: highest first), then by IP/addr, in ascending order. sub really_print_hash_by_cnt_vals { my($hashRef, $cnt, $indents) = @_; foreach (map { $_->[0] } sort { $b->[1] <=> $a->[1] || $a->[2] cmp $b->[2] } map { [ $_, $hashRef->{$_}, normalize_host($_) ] } (keys(%$hashRef))) { printf "$indents%6d%s %s\n", adj_int_units($hashRef->{$_}), $_; last if --$cnt == 0; } } # Print a sub-section title with properly-sized underline sub print_subsect_title { my $title = $_[0]; print "\n$title\n" . "-" x length($title) . "\n"; } # Normalize IP addr or hostname sub normalize_host { my ($host) = @_; # Strip off possible " (user@dom.ain)" bit my $norm1 = (split(/\s/, $host))[0]; # Helper function to normalize IPv6 address sub normalize_ipv6 { my ($addr) = @_; # Convert to lowercase and remove leading/trailing whitespace $addr = lc($addr); $addr =~ s/^\s+|\s+$//g; # Split into blocks my @blocks = split /:/, $addr; my $double_colon_index = -1; my $block_count = @blocks; # Find double colon (::) if it exists for my $i (0 .. $#blocks) { if ($blocks[$i] eq '') { $double_colon_index = $i; last; } } # Handle double colon abbreviation if ($double_colon_index >= 0) { my $missing_blocks = 8 - $block_count + 1; # +1 for the empty block my @zeroes = ('0000') x $missing_blocks; splice @blocks, $double_colon_index, 1, @zeroes; } # Pad each block with leading zeros to 4 digits @blocks = map { sprintf("%04s", $_) } @blocks; # Ensure exactly 8 blocks if (@blocks < 8) { push @blocks, ('0000') x (8 - @blocks); } elsif (@blocks > 8) { die "Invalid IPv6 address: $addr\n"; } return join '', @blocks; } if ((my @octets = $norm1 =~ /^(\d{1,3})\.(\d{1,3})\.(\d{1,3})\.(\d{1,3})$/) == 4) { # Dotted-quad IPv4 address # Validate each octet is in range 0-255 for my $octet (@octets) { return "0_$norm1" if $octet > 255; # Treat invalid as hostname } # Convert to 8-character hex (2 chars per octet) my $hex = sprintf("%02x%02x%02x%02x", @octets); return "2_$hex"; } elsif ($norm1 =~ /^[\dA-Fa-f]+:/) { # IPv6 address return "3_" . normalize_ipv6($norm1); } else { # Hostname or user@dom.ain return "0_" . join('', map { lc $_ } reverse split /[.@]/, $norm1); } } # subroutine to sort by domain, then user in domain, then by queue i.d. # Note: mixing Internet-style domain names and UUCP-style bang-paths # may confuse this thing. An attempt is made to use the first host # preceding the username in the bang-path as the "domain" if none is # found otherwise. sub by_domain_then_user { # first see if we can get "user@somedomain" my($userNameA, $domainA) = split(/\@/, ${$hashRef->{$a}}[0]); my($userNameB, $domainB) = split(/\@/, ${$hashRef->{$b}}[0]); # try "somedomain!user"? ($userNameA, $domainA) = (split(/!/, ${$hashRef->{$a}}[0]))[-1,-2] unless($domainA); ($userNameB, $domainB) = (split(/!/, ${$hashRef->{$b}}[0]))[-1,-2] unless($domainB); # now re-order "mach.host.dom"/"mach.host.do.co" to # "host.dom.mach"/"host.do.co.mach" $domainA =~ s/^(.*)\.([^\.]+)\.([^\.]{3}|[^\.]{2,3}\.[^\.]{2})$/$2.$3.$1/ if($domainA); $domainB =~ s/^(.*)\.([^\.]+)\.([^\.]{3}|[^\.]{2,3}\.[^\.]{2})$/$2.$3.$1/ if($domainB); # oddly enough, doing this here is marginally faster than doing # an "if-else", above. go figure. $domainA = "" unless($domainA); $domainB = "" unless($domainB); if($domainA lt $domainB) { return -1; } elsif($domainA gt $domainB) { return 1; } else { # disregard leading bang-path $userNameA =~ s/^.*!//; $userNameB =~ s/^.*!//; if($userNameA lt $userNameB) { return -1; } elsif($userNameA gt $userNameB) { return 1; } else { if($a lt $b) { return -1; } elsif($a gt $b) { return 1; } } } return 0; } # Subroutine used by host/domain reports to sort by count, then size. # We "fix" un-initialized values here as well. Very ugly and un- # structured to do this here - but it's either that or the callers # must run through the hashes twice :-(. sub by_count_then_size { ${$hashRef->{$a}}[MSG_CNT_I] = 0 unless(${$hashRef->{$a}}[MSG_CNT_I]); ${$hashRef->{$b}}[MSG_CNT_I] = 0 unless(${$hashRef->{$b}}[MSG_CNT_I]); if(${$hashRef->{$a}}[MSG_CNT_I] == ${$hashRef->{$b}}[MSG_CNT_I]) { ${$hashRef->{$a}}[MSG_SIZE_I] = 0 unless(${$hashRef->{$a}}[MSG_SIZE_I]); ${$hashRef->{$b}}[MSG_SIZE_I] = 0 unless(${$hashRef->{$b}}[MSG_SIZE_I]); return(${$hashRef->{$a}}[MSG_SIZE_I] <=> ${$hashRef->{$b}}[MSG_SIZE_I]); } else { return(${$hashRef->{$a}}[MSG_CNT_I] <=> ${$hashRef->{$b}}[MSG_CNT_I]); } } # Get range of dates to parse sub get_dates { my ($range, $day0mon, $currTime) = @_; my ($startYr, $startMon, $startDay, $endYr, $endMon, $endDay); $currTime //= time(); my ($sec, $min, $hour, $day, $mon, $yr) = localtime($currTime); $yr += 1900; $mon += 1; # Normalize $range =~ s/_/ /g; if ($range eq 'today') { ($startYr, $startMon, $startDay) = ($yr, $mon, $day); ($endYr, $endMon, $endDay) = ($yr, $mon, $day); } elsif ($range eq 'yesterday') { ($startYr, $startMon, $startDay) = Add_Delta_Days($yr, $mon, $day, -1); ($endYr, $endMon, $endDay) = ($startYr, $startMon, $startDay); } elsif ($range eq 'this week' or $range eq 'last week') { # 1) Get local calendar date for "now" my ($sec,$min,$hour,$d,$mo,$y) = localtime($currTime); my $midnight_now = timelocal(0,0,0, $d, $mo, $y); # local midnight of "now" # 2) Day-of-week at local midnight (0=Sun..6=Sat) my $dow = (localtime($midnight_now))[6]; # 3) Days since start-of-week (Sun-start vs Mon-start) my $since_start = $day0mon ? ($dow == 0 ? 6 : $dow - 1) : $dow; # 4) Convert to Y-M-D, then use calendar math only my ($ny,$nmon,$nday) = ($y + 1900, $mo + 1, $d); my $offset = -$since_start + ($range eq 'last week' ? -7 : 0); my ($sy,$sm,$sd) = Add_Delta_Days($ny, $nmon, $nday, $offset); # start (Y-M-D) my ($ey,$em,$ed) = Add_Delta_Days($sy, $sm, $sd, 6); # end (Y-M-D) # 5) Back to epochs at local midnight (isdst auto-handled) my $start_epoch = timelocal(0,0,0, $sd, $sm-1, $sy-1900); my $end_epoch = timelocal(0,0,0, $ed, $em-1, $ey-1900); ($startYr,$startMon,$startDay) = ($sy,$sm,$sd); ($endYr, $endMon, $endDay) = ($ey,$em,$ed); } elsif ($range eq 'this month') { ($startYr, $startMon, $startDay) = ($yr, $mon, 1); ($endYr, $endMon, $endDay) = Add_Delta_Days($yr, $mon, 1, Days_in_Month($yr, $mon) - 1); } elsif ($range eq 'last month') { my ($lastYr, $lastMon) = ($mon == 1) ? ($yr - 1, 12) : ($yr, $mon - 1); ($startYr, $startMon, $startDay) = ($lastYr, $lastMon, 1); ($endYr, $endMon, $endDay) = Add_Delta_Days($lastYr, $lastMon, 1, Days_in_Month($lastYr, $lastMon) - 1); } elsif ($range =~ /^(\d{4})-(\d{2})(?:-(\d{2}))?$/) { ($startYr, $startMon, $startDay) = ($1, $2, $3); unless(defined($startDay)) { $startDay = 1; ($endYr, $endMon, $endDay) = ($1, $2, Days_in_Month($startYr, $startMon)); } else { ($endYr, $endMon, $endDay) = ($1, $2, $3); } } elsif ($range =~ /^(\d{4}-\d{2}(?:-\d{2})?)\s+(?:(?:to|-)\s+)?(\d{4}-\d{2}(?:-\d{2})?)$/) { my ($s, $e) = ($1, $2); ($startYr, $startMon, $startDay) = split(/-/, $s); $startDay = 1 unless($startDay); ($endYr, $endMon, $endDay) = split(/-/, $e); $endDay = Days_in_Month($endYr, $endMon) unless($endDay); } else { die "Invalid date range format: '$range'\n"; } my $start_time = timelocal(0, 0, 0, $startDay, $startMon - 1, $startYr - 1900); my $end_time = timelocal(0, 0, 0, $endDay, $endMon - 1, $endYr - 1900); die "End date precedes start date: '$range'" if $end_time < $start_time; return ($start_time, $end_time); } # # If a line matches the desired date (range): Return the year, month, day, hour, minutes, seconds, and log remainder # # N.B.: Year is returned adj. to +1900 # Month is returned as 0-11 # sub line_matches_dates { my ($line, $startEpoch, $endEpoch) = @_; my $now = time(); my ($epoch, $msgYr, $msgMon, $msgDay, $msgHr, $msgMin, $msgSec, $logRmdr); # Try RFC 3339 / ISO 8601 first if (scalar (($msgYr, $msgMon, $msgDay, $msgHr, $msgMin, $msgSec, $logRmdr) = ($line =~ /^(?:<\d{1,3}>(?:[1-9]\d*\s+|\s*))?(\d{4})-(\d{2})-(\d{2})T(\d{2}):(\d{2}):(\d{2})(?:\.\d+)?(?:[\+\-](?:\d{2}):(?:\d{2})|Z)? \S+ (.+)$/)) == 7) { # RFC 3339 months start at "1", we index from 0 --$msgMon; return ($msgYr, $msgMon, $msgDay, $msgHr, $msgMin, $msgSec, $logRmdr) unless(defined($startEpoch) && defined($endEpoch)); $epoch = eval { timelocal(0, 0, 0, $msgDay, $msgMon, $msgYr - 1900) }; return ($epoch >= $startEpoch && $epoch <= $endEpoch) ? ($msgYr, $msgMon, $msgDay, $msgHr, $msgMin, $msgSec, $logRmdr) : (undef); } # Try traditional syslog format my $monStr; if(scalar (($monStr, $msgDay, $msgHr, $msgMin, $msgSec, $logRmdr) = ($line =~ /^(?:<\d{1,3}>(?:[1-9]\d*\s+|\s*))?(\w{3}) {1,2}(\d{1,2}) (\d{2}):(\d{2}):(\d{2}) \S+ (.+)$/)) == 6) { return (undef) unless defined($msgMon = $monthNums{$monStr}); my ($currMon, $currYr) = (localtime($now))[4,5]; # If month in logfile line is > current month the logfile line must be from last year --$currYr if($msgMon > $currMon); return ($currYr + 1900, $msgMon, $msgDay, $msgHr, $msgMin, $msgSec, $logRmdr) unless (defined($startEpoch) && defined($endEpoch)); $epoch = eval { timelocal(0, 0, 0, $msgDay, $msgMon, $currYr) }; return (defined $epoch && $epoch >= $startEpoch && $epoch <= $endEpoch) ? ($currYr + 1900, $msgMon, $msgDay, $msgHr, $msgMin, $msgSec, $logRmdr) : (undef); } return (undef); # Not a parsable line } # if there's a real hostname/domain: uses that. Otherwise uses # the IP addr. # # N.B.: in-addr.arpa and ip6.arpa FQDNs return IP addrs # # Lower-cases returned domain name. # # Handles IPv4, IPv6, and IPv6-mapped-IPv4 addrs, current-style # Postfix bracketed IP addrs, and old-style slash-separated IP addrs # # N.B.: IP addr checking is not exhaustive # sub gimme_domain { my $line = $_[0]; # Treat letters, digits, dot, hyphen, and underscore as "hostname-ish". # Start only if we're NOT preceded by one of those (or we're at BOL). state $HBOUND = qr/(?"s? # sub fold_domain { my ($fqdn) = @_; state %original_tlds; return '' unless defined $fqdn; # Strip brackets, "user@", leading and trailing whitespace, lowercase (my $domain = $fqdn) =~ s/^\s*]+)>?\s*$/\L$1/; # Strip trailing "." (PTRs, etc.) $domain =~ s/\.$//; return $domain if $domain !~ /\./; unless(%original_tlds) { %original_tlds = map { $_ => 1 } qw(com net org gov mil edu); } my @parts = split /\./, $domain; my ($sld,$tld) = @parts[-2,-1]; return "$sld.$tld" if $original_tlds{$tld}; return join('.', @parts[1..$#parts]) if @parts > 3; # elide leftmost once return $domain; } # Return (value, units) for integer sub adj_int_units { my $value = $_[0]; my $units = ' '; $value = 0 unless($value); if($value > DIV_BY_ONE_MEG_AT) { $value /= ONE_MEG; $units = 'm' } elsif($value > DIV_BY_ONE_K_AT) { $value /= ONE_K; $units = 'k' } return($value, $units); } # Return (value, units) for time sub adj_time_units { my $value = $_[0]; my $units = 's'; $value = 0 unless($value); if($value > 3600) { $value /= 3600; $units = 'h' } elsif($value > 60) { $value /= 60; $units = 'm' } return($value, $units); } # Trim a "said:" string, if necessary. Add elipses to show it. # FIXME: This sometimes elides The Wrong Bits, yielding # summaries that are less useful than they could be. sub said_string_trimmer { my($trimmedString, $maxLen) = @_; $trimmedString =~ s/^\d{3}([ -]#?\d\.\d\.\d)? //; while(length($trimmedString) > $maxLen) { if($trimmedString =~ /^.* said:( \d{3}([ -]#?\d\.\d\.\d)?)? /) { $trimmedString =~ s/^.* said:( \d{3}([ -]#?\d\.\d\.\d)?)? //; } elsif($trimmedString =~ /(delivery error|Requested action not taken|Transaction failed|RCPT TO): */) { $trimmedString =~ s/^.*?: *//; } else { $trimmedString = substr($trimmedString, 0, $maxLen - 3) . "..."; last; } } return $trimmedString; } # Trim a string, if necessary. Add elipses to show it. sub string_trimmer { my($trimmedString, $maxLen) = @_; unless($opts{'colwidth'} == 0) { $maxLen += $opts{'colwidth'} - 80 if($opts{'colwidth'} > 0); if(length($trimmedString) > $maxLen) { $trimmedString = substr($trimmedString, 0, $maxLen - 3) . "..."; } } return $trimmedString; } # Get seconds, minutes and hours from seconds sub get_smh { my $sec = shift @_; my ($min, $hr); if($sec) { $hr = int($sec / 3600); $sec -= $hr * 3600; $min = int($sec / 60); $sec -= $min * 60; } else { ($sec, $min, $hr) = (0, 0, 0); } return($sec, $min, $hr); } # Process smtpd rejects sub proc_smtpd_reject { my ($logLine, $rejects, $msgsRjctd, $rejPerHr, $msgsPerDay) = @_; my ($rejTyp, $rejFrom, $rejRmdr, $rejReas); my ($from, $to); ++$$msgsRjctd; ++$$rejPerHr; ++$$msgsPerDay; # Hate the sub-calling overhead if we're not doing reject details # anyway, but this is the only place we can do this. return if($opts{'reject-detail'} == 0); # This could get real ugly! # First: get everything following the "reject: ", etc. token # Was an IPv6 problem here ($rejTyp, $rejFrom, $rejRmdr) = $logLine =~ /^.* \b(?:reject(?:_warning)?|hold|discard): (\S+) from (\S+?): (.*)$/; print STDERR "\$rejTyp: \"$rejTyp\", \$rejReas: \"$rejReas\"\n" if($opts{'debug'} && defined $rejTyp && defined $rejReas); # Next: get the reject "reason" $rejReas = $rejRmdr; unless(defined($opts{'verbose-msg-detail'})) { if($rejTyp eq "RCPT" || $rejTyp eq "DATA" || $rejTyp eq "CONNECT" || $rejTyp eq "BDAT") { # special treatment :-( # If there are "<>"s immediately following the reject code, that's # an email address or HELO string. There can be *anything* in # those--incl. stuff that'll screw up subsequent parsing. So just # get rid of it right off. $rejReas =~ s/^(\d{3} (?:\d\.\d\.\d )?<).*?(>:)/$1$2/; $rejReas =~ s/^(?:\d{3} \d\.\d\.\d )(Protocol error);.*$/$1/; # from Deb 1.1.15-8 $rejReas =~ s/^.*?[:;] (?:\[[^\]]+\] )?([^;,]+)[;,].*$/$1/; $rejReas =~ s/^((?:Sender|Recipient) address rejected: [^:]+):.*$/$1/; $rejReas =~ s/(client|Client host|Sender address) .+? blocked/blocked/; # from Deb 1.1.15-8 $rejReas =~ s/^\d{3} \d\.\d\.\d (Server configuration (?:error|problem));.+$/$1/; # Condense fqrdns.pcre reports $rejReas =~ s/^Unverified (Client host rejected: Generic - Please relay via ISP).*$/$1/; } elsif($rejTyp eq "MAIL") { # *more* special treatment :-( grrrr... $rejReas =~ s/^\d{3} (?:<.+>: )?([^;:]+)[;:]?.*$/$1/; } elsif($rejTyp eq "connect") { # and still *more* special treatment :-( *sigh*... $rejTyp = 'CONNECT'; } else { $rejReas =~ s/^(?:.*[:;] )?([^,]+).*$/$1/; } } # Snag recipient address # Second expression is for unknown recipient--where there is no # "to=" field, third for pathological case where recipient # field is unterminated, forth when all else fails. (($to) = $rejRmdr =~ /to=<([^>]+)>/) || (($to) = $rejRmdr =~ /\d{3} <([^>]+)>: User unknown /) || (($to) = $rejRmdr =~ /to=<(.*?)(?:[, ]|$)/) || ($to = "<>"); $to = lc($to) if($opts{'ignore-case'}); # Snag sender address (($from) = $rejRmdr =~ /from=<([^>]+)>/) || ($from = "<>"); if(defined($from)) { $from = verp_mung($from); $from = srs_mung($from); $from = lc($from) if($opts{'ignore-case'}); } # stash in "triple-subscripted-array" if($rejReas =~ m/^Sender address rejected:/) { # Sender address rejected: Domain not found # Sender address rejected: need fully-qualified address my $rejData = $from; $rejData .= " ($to)" if($opts{'rej-add-to'} && $to); ++$rejects->{$rejTyp}{$rejReas}{$rejData}; } elsif($rejReas =~ m/^(Recipient address rejected:|User unknown( |$))/) { # Recipient address rejected: Domain not found # Recipient address rejected: need fully-qualified address # User unknown (in local/relay recipient table) #++$rejects->{$rejTyp}{$rejReas}{$to}; my $rejData = $to; if($opts{'rej-add-from'}) { $rejData .= " (" . ($from? $from : gimme_domain($rejFrom)) . ")"; } ++$rejects->{$rejTyp}{$rejReas}{$rejData}; } elsif($rejReas =~ s/^.*?\d{3} (Improper use of SMTP command pipelining);.*$/$1/) { # Was an IPv6 problem here my ($src) = $logLine =~ /^.+? from (\S+?):.*$/; ++$rejects->{$rejTyp}{$rejReas}{$src}; } elsif($rejReas =~ s/^.*?\d{3} (Message size exceeds fixed limit);.*$/$1/) { my $rejData = gimme_domain($rejFrom); $rejData .= " ($from)" if($opts{'rej-add-from'}); ++$rejects->{$rejTyp}{$rejReas}{$rejData}; } elsif($rejReas =~ s/^.*?\d{3} (Server configuration (?:error|problem));.*$/(Local) $1/) { my $rejData = gimme_domain($rejFrom); $rejData .= " ($from)" if($opts{'rej-add-from'}); ++$rejects->{$rejTyp}{$rejReas}{$rejData}; } elsif($rejReas =~ m/^Helo command rejected: Invalid name$/) { my $rejData = gimme_domain($rejFrom); $rejData .= " ($from)" if($opts{'rej-add-from'}); ++$rejects->{$rejTyp}{$rejReas}{$rejData}; } else { print STDERR "dbg: unknown/un-enumerated reject reason: \$rejReas: \"$rejReas\", \$rejTyp: \"$rejTyp\", \$rejFrom: \"$rejFrom\"!\n" if($opts{'debug'}); my $rejData = gimme_domain($rejFrom); if($opts{'rej-add-from'} && $opts{'rej-add-to'} && $to) { $rejData .= " ($from -> $to)"; } elsif($opts{'rej-add-from'}) { $rejData .= " (< $from)"; } elsif($opts{'rej-add-to'} && $to) { $rejData .= " (> $to)"; } ++$rejects->{$rejTyp}{$rejReas}{$rejData}; } } # Hack for VERP (?) - convert address from somthing like # "list-return-36-someuser=someplace.com@lists.domain.com" # to "list-return-ID-someuser=someplace.com@lists.domain.com" # to prevent per-user listing "pollution." More aggressive # munging converts to something like # "list-return@lists.domain.com" (Instead of "return," there # may be numeric list name/id, "warn", "error", etc.?) sub verp_mung { my $addr = $_[0]; if(defined($opts{'verp-mung'})) { $addr =~ s/((?:bounce[ds]?|no(?:list|reply|response)|return|sentto|\d+).*?)(?:[\+_\.\*-]\d+\b)+/$1-ID/i; if($opts{'verp-mung'} > 1) { $addr =~ s/[\*-](\d+[\*-])?[^=\*-]+[=\*][^\@]+\@/\@/; } } return $addr; } sub srs_mung { my $addr = $_[0]; if(defined($opts{'srs-mung'})) { $addr =~ s/^SRS(?:[01])(?:[=+-])(?:[^=]+=[\w\.]+==)*(?:[^=]+=[^=]+=)([\w\.]+)=(.+)@[\w\.]+$/$2\@$1/i; } return $addr; } ### ### Warning and Error Routines ### # Emit warning message to stderr sub msg_warn { warn "warning: $progName: $_[0]\n"; } pflogsumm-1.1.13/pftobyfrom.10000644000175000017500000001110015076454367016405 0ustar jseymourjseymour.\" -*- mode: troff; coding: utf-8 -*- .\" Automatically generated by Pod::Man 5.01 (Pod::Simple 3.43) .\" .\" Standard preamble: .\" ======================================================================== .de Sp \" Vertical space (when we can't use .PP) .if t .sp .5v .if n .sp .. .de Vb \" Begin verbatim text .ft CW .nf .ne \\$1 .. .de Ve \" End verbatim text .ft R .fi .. .\" \*(C` and \*(C' are quotes in nroff, nothing in troff, for use with C<>. .ie n \{\ . ds C` "" . ds C' "" 'br\} .el\{\ . ds C` . ds C' 'br\} .\" .\" Escape single quotes in literal strings from groff's Unicode transform. .ie \n(.g .ds Aq \(aq .el .ds Aq ' .\" .\" If the F register is >0, we'll generate index entries on stderr for .\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index .\" entries marked with X<> in POD. Of course, you'll have to process the .\" output yourself in some meaningful fashion. .\" .\" Avoid warning from groff about undefined register 'F'. .de IX .. .nr rF 0 .if \n(.g .if rF .nr rF 1 .if (\n(rF:(\n(.g==0)) \{\ . if \nF \{\ . de IX . tm Index:\\$1\t\\n%\t"\\$2" .. . if !\nF==2 \{\ . nr % 0 . nr F 2 . \} . \} .\} .rr rF .\" ======================================================================== .\" .IX Title "PFTOBYFROM 1" .TH PFTOBYFROM 1 2025-05-22 1.1.13 "User Contributed Perl Documentation" .\" For nroff, turn off justification. Always turn off hyphenation; it makes .\" way too many mistakes in technical documents. .if n .ad l .nh .SH NAME pftobyfrom \- List "to" addresses by "from" whom in Postfix log file .PP Copyright (C) 2007\-2025 by James S. Seymour, Release 1.3 .SH "SYNOPSIS pftobyfrom \-[bhrRv] [mailfile]" .IX Header "SYNOPSIS pftobyfrom -[bhrRv] [mailfile]" .Vb 1 \& If no file(s) specified, reads from stdin. Output is to stdout. .Ve .SH DESCRIPTION .IX Header "DESCRIPTION" .Vb 2 \& pftobyfrom parses Postfix log files to generate a list of "to" addresses, \& based on a specified "from" address or address fragment. .Ve .SH OPTIONS .IX Header "OPTIONS" .Vb 1 \& \-b Include bounces \& \& \-h Emit help message and exit \& \& \-r Include rejects \& \& \-R Hard rejects only \& \& \-v Emit version and exit .Ve .SH "RETURN VALUE" .IX Header "RETURN VALUE" .Vb 1 \& pftobyfrom doesn\*(Aqt return anything of interest to the shell. .Ve .SH ERRORS .IX Header "ERRORS" .Vb 1 \& Error messages are emitted to stderr. .Ve .SH EXAMPLES .IX Header "EXAMPLES" .Vb 1 \& pftobyfrom example.com /var/log/maillog \& \& Generates a list of all the recipients of email from any senders \& in "example.com" \& \& As a convenience, pftobyfrom tries to intelligently determine how to \& handle regexp meta\-characters. If it\*(Aqs passed a search expression \& that does NOT contain meta\-character escapes ("\e"), it will assume \& that "." and "+" are literals, and will escape them for you. In the \& example above, the "." in the FQDN part of the search term would\*(Aqve \& been automatically escaped for the user. Likewise: \& \& pftobyfrom username+foo@example.com /var/log/maillog \& \& would have the "+" and "." escaped. If you wanted to find all \& plussed senders for "username," you\*(Aqd have to do: \& \& pftobyfrom \*(Aqusername\e+.+@example\e.com\*(Aq /var/log/maillog .Ve .SH "SEE ALSO" .IX Header "SEE ALSO" .Vb 1 \& pflogsumm, pffrombyto .Ve .SH NOTES .IX Header "NOTES" .Vb 1 \& All search terms and searched fields are lower\-cased. \& \& The pftobyfrom Home Page is at: \& \& http://jimsun.LinxNet.com/postfix_contrib.html .Ve .SH REQUIREMENTS .IX Header "REQUIREMENTS" .Vb 1 \& Perl .Ve .SH LICENSE .IX Header "LICENSE" .Vb 4 \& This program is free software; you can redistribute it and/or \& modify it under the terms of the GNU General Public License \& as published by the Free Software Foundation; either version 2 \& of the License, or (at your option) any later version. \& \& This program is distributed in the hope that it will be useful, \& but WITHOUT ANY WARRANTY; without even the implied warranty of \& MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the \& GNU General Public License for more details. \& \& You may have received a copy of the GNU General Public License \& along with this program; if not, write to the Free Software \& Foundation, Inc., 59 Temple Place \- Suite 330, Boston, MA 02111\-1307, \& USA. \& \& An on\-line copy of the GNU General Public License can be found \& http://www.fsf.org/copyleft/gpl.html. .Ve pflogsumm-1.1.13/pffrombyto.10000644000175000017500000001121415076454367016413 0ustar jseymourjseymour.\" -*- mode: troff; coding: utf-8 -*- .\" Automatically generated by Pod::Man 5.01 (Pod::Simple 3.43) .\" .\" Standard preamble: .\" ======================================================================== .de Sp \" Vertical space (when we can't use .PP) .if t .sp .5v .if n .sp .. .de Vb \" Begin verbatim text .ft CW .nf .ne \\$1 .. .de Ve \" End verbatim text .ft R .fi .. .\" \*(C` and \*(C' are quotes in nroff, nothing in troff, for use with C<>. .ie n \{\ . ds C` "" . ds C' "" 'br\} .el\{\ . ds C` . ds C' 'br\} .\" .\" Escape single quotes in literal strings from groff's Unicode transform. .ie \n(.g .ds Aq \(aq .el .ds Aq ' .\" .\" If the F register is >0, we'll generate index entries on stderr for .\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index .\" entries marked with X<> in POD. Of course, you'll have to process the .\" output yourself in some meaningful fashion. .\" .\" Avoid warning from groff about undefined register 'F'. .de IX .. .nr rF 0 .if \n(.g .if rF .nr rF 1 .if (\n(rF:(\n(.g==0)) \{\ . if \nF \{\ . de IX . tm Index:\\$1\t\\n%\t"\\$2" .. . if !\nF==2 \{\ . nr % 0 . nr F 2 . \} . \} .\} .rr rF .\" ======================================================================== .\" .IX Title "PFFROMBYTO 1" .TH PFFROMBYTO 1 2025-05-22 1.1.13 "User Contributed Perl Documentation" .\" For nroff, turn off justification. Always turn off hyphenation; it makes .\" way too many mistakes in technical documents. .if n .ad l .nh .SH NAME pffrombyto \- List "from" addresses by "to" whom in Postfix log file .PP Copyright (C) 2007\-2025 by James S. Seymour, Release 1.2 .SH "SYNOPSIS pffrombyto \-[bchrRv] [mailfile]" .IX Header "SYNOPSIS pffrombyto -[bchrRv] [mailfile]" .Vb 1 \& If no file(s) specified, reads from stdin. Output is to stdout. .Ve .SH DESCRIPTION .IX Header "DESCRIPTION" .Vb 2 \& pffrombyto parses Postfix log files to generate a list of "from" addresses, \& based on a specified "to" address or address fragment. .Ve .SH OPTIONS .IX Header "OPTIONS" .Vb 1 \& \-b Include bounces \& \& \-c Include client hostnames/addrs, as well \& \& \-h Emit help message and exit \& \& \-r Include rejects \& \& \-R Hard rejects only \& \& \-v Emit version and exit .Ve .SH "RETURN VALUE" .IX Header "RETURN VALUE" .Vb 1 \& pffrombyto doesn\*(Aqt return anything of interest to the shell. .Ve .SH ERRORS .IX Header "ERRORS" .Vb 1 \& Error messages are emitted to stderr. .Ve .SH EXAMPLES .IX Header "EXAMPLES" .Vb 2 \& Generate a list of all the senders of email to the recipient \& "username@example.com" \& \& pffrombyto username@example.com /var/log/maillog \& \& As a convenience, pffrombyto tries to intelligently determine how to \& handle regexp meta\-characters. If it\*(Aqs passed a search expression \& that does NOT contain meta\-character escapes ("\e"), it will assume \& that "." and "+" are literals, and will escape them for you. In the \& example above, the "." in the FQDN part of the search term would\*(Aqve \& been automatically escaped for the user. Likewise: \& \& pffrombyto username+foo@example.com /var/log/maillog \& \& would have the "+" and "." escaped. If you wanted to find all \& plussed targets for "username," you\*(Aqd have to do: \& \& pffrombyto \*(Aqusername\e+.+@example\e.com\*(Aq /var/log/maillog .Ve .SH "SEE ALSO" .IX Header "SEE ALSO" .Vb 1 \& pflogsumm, pftobyfrom .Ve .SH NOTES .IX Header "NOTES" .Vb 1 \& All search terms and searched fields are lower\-cased. \& \& The pffrombyto Home Page is at: \& \& http://jimsun.LinxNet.com/postfix_contrib.html .Ve .SH REQUIREMENTS .IX Header "REQUIREMENTS" .Vb 1 \& Perl .Ve .SH LICENSE .IX Header "LICENSE" .Vb 4 \& This program is free software; you can redistribute it and/or \& modify it under the terms of the GNU General Public License \& as published by the Free Software Foundation; either version 2 \& of the License, or (at your option) any later version. \& \& This program is distributed in the hope that it will be useful, \& but WITHOUT ANY WARRANTY; without even the implied warranty of \& MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the \& GNU General Public License for more details. \& \& You may have received a copy of the GNU General Public License \& along with this program; if not, write to the Free Software \& Foundation, Inc., 59 Temple Place \- Suite 330, Boston, MA 02111\-1307, \& USA. \& \& An on\-line copy of the GNU General Public License can be found \& http://www.fsf.org/copyleft/gpl.html. .Ve pflogsumm-1.1.13/README0000644000175000017500000000236415013372212015003 0ustar jseymourjseymour Pflogsumm README There's not much to installing pflogsumm, so it's all manual. 1. Unpack the distribution (if you're reading this, you've already gotten that far) 2. Copy or move pflogsumm to some directory from which you'll want to execute it. Maybe rename it to just "pflogsumm." Watch the ownerships and permissions. Make sure it's executable. E.g.: cp pflogsumm /usr/local/bin/pflogsumm chown bin:bin /usr/local/bin/pflogsumm chmod 755 /usr/local/bin/pflogsumm 3. If there's a manual page available (pflogsumm.1), copy that to /usr/local/man/man1 or wherever you stash local manpages. Make sure it's world-readable. E.g.: cp pflogsumm.1 /usr/local/man/man1/pflogsumm.1 chown bin:bin /usr/local/man/man1/pflogsumm.1 chmod 644 /usr/local/man/man1/pflogsumm.1 4. Read the man page (or the top of pflogsumm itself) for usage. 5. Check the FAQ (pflogsumm-faq.txt) 6. Configure your cron jobs if you're going to run pflogsumm on an automatic, scheduled basis. There are tips in the manpage and the FAQ. That's about it. As the manpage and FAQ both note: pflogsumm requires the Date::Calc Perl module if you want to use --smtpd_stats. pflogsumm-1.1.13/pflogsumm.10000644000175000017500000005600715076454367016246 0ustar jseymourjseymour.\" -*- mode: troff; coding: utf-8 -*- .\" Automatically generated by Pod::Man 5.01 (Pod::Simple 3.43) .\" .\" Standard preamble: .\" ======================================================================== .de Sp \" Vertical space (when we can't use .PP) .if t .sp .5v .if n .sp .. .de Vb \" Begin verbatim text .ft CW .nf .ne \\$1 .. .de Ve \" End verbatim text .ft R .fi .. .\" \*(C` and \*(C' are quotes in nroff, nothing in troff, for use with C<>. .ie n \{\ . ds C` "" . ds C' "" 'br\} .el\{\ . ds C` . ds C' 'br\} .\" .\" Escape single quotes in literal strings from groff's Unicode transform. .ie \n(.g .ds Aq \(aq .el .ds Aq ' .\" .\" If the F register is >0, we'll generate index entries on stderr for .\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index .\" entries marked with X<> in POD. Of course, you'll have to process the .\" output yourself in some meaningful fashion. .\" .\" Avoid warning from groff about undefined register 'F'. .de IX .. .nr rF 0 .if \n(.g .if rF .nr rF 1 .if (\n(rF:(\n(.g==0)) \{\ . if \nF \{\ . de IX . tm Index:\\$1\t\\n%\t"\\$2" .. . if !\nF==2 \{\ . nr % 0 . nr F 2 . \} . \} .\} .rr rF .\" ======================================================================== .\" .IX Title "PFLOGSUMM 1" .TH PFLOGSUMM 1 2025-10-22 1.1.13 "User Contributed Perl Documentation" .\" For nroff, turn off justification. Always turn off hyphenation; it makes .\" way too many mistakes in technical documents. .if n .ad l .nh .SH NAME pflogsumm \- Produce Postfix MTA logfile summary .PP Copyright (C) 1998\-2025 by James S. Seymour, Release 1.1.13 .SH SYNOPSIS .IX Header "SYNOPSIS" .Vb 10 \& pflogsumm [\-\-config ] [\-\-bounce\-detail ] \& [\-\-colwidth ] [\-\-deferral\-detail ] [\-\-detail ] \& [\-d ] [\-\-dow0mon] [\-e] [\-h ] [\-i] \& [\-\-iso\-date\-time] [\-\-mailq] [\-m] [\-\-no\-no\-msg\-size] \& [\-\-problems\-first] [\-\-pscrn\-detail ] [\-\-pscrn\-stats] \& [\-q] [\-\-rej\-add\-from] [\-\-rej\-add\-to] [\-\-reject\-detail ] \& [\-\-smtp\-detail ] [\-\-smtpd\-stats] [\-\-smtpd\-warning\-detail ] \& [\-\-srs\-mung] [\-\-syslog\-name=string] [\-u ] \& [\-\-unprocd\-file ] [\-\-use\-orig\-to] [\-\-verbose\-msg\-detail] \& [\-\-verp\-mung[=]] [\-x] [\-\-zero\-fill] [file1 [filen]] \& \& pflogsumm \-\-[dump\-config|help|version] \& \& Note: Where both long\- and short\-form options exist only the \& latter are shown above. See man page for long\-form equivalents. \& \& If no file(s) specified, reads from stdin. Output is to stdout. Errors \& and debug to stderr. .Ve .SH DESCRIPTION .IX Header "DESCRIPTION" .Vb 4 \& Pflogsumm is a log analyzer/summarizer for the Postfix MTA. It is \& designed to provide an over\-view of Postfix activity, with just enough \& detail to give the administrator a "heads up" for potential trouble \& spots. \& \& Pflogsumm generates summaries and, in some cases, detailed reports of \& mail server traffic volumes, rejected and bounced email, and server \& warnings, errors and panics. .Ve .SH OPTIONS .IX Header "OPTIONS" .Vb 1 \& \-\-bounce\-detail \& \& Limit detailed bounce reports to the top . 0 \& to suppress entirely. \& \& \-\-config \& \& Path to a configuration file containing pflogsumm \& options. \& \& Supports all standard command\-line options (without the \& leading "\-" or "\-\-"). Options like "config", "dump\-config", \& "help", and "version" technically work here, too, though \& they\*(Aqre not particularly useful in this context. \& \& Command\-line arguments override config file values except \& for boolean options. \& \& \-\-colwidth \& Maximum report output width. Default is 80 columns. \& 0 = unlimited. \& \& N.B.: \-\-verbose\-msg\-detail overrides \& \& \-d \& \-\-date\-range \& \& Limits the report to the specified date or range. \& \& Accepted values: \& \& today \& yesterday \& "this week" / "last week" \& "this month" / "last month" \& YYYY\-MM[\-DD] \& "YYYY\-MM[\-DD] YYYY\-MM[\-DD]" \& \& These options do what they suggest, with one \& important caveat: \& \& ISO 8601 / RFC 3339\-style dates and ranges may \& not yield accurate results when used with \& traditional log formats lacking year information \& ("month day\-of\-month"). \& \& In such cases, pflogsumm assumes log entries \& are from the current year. For example, if the \& current month is April and a log contains "Apr \& NN" entries from the previous year, they will \& be interpreted as from the *current* April. \& \& As such, date\-based filtering is only reliable \& for entries less than ~365 days old for \& old\-/traditional\-style logfiles. \& \& Arguments containing spaces must be quoted! \& \& This/last week/month arguments can take underscores, \& rather than spaces, to avoid quoting: E.g.: \& \& \-\-date\-range last_week \& \& ISO 8601/RFC 3339 date ranges may optionally use a \& hyphen or the word "to" for readability. E.g.: \& \& "2025\-08\-01 to 2025\-08\-08" \& \& If an optional day (DD) is omitted, the range becomes \& the full month. E.g.: \& \& 2025\-08 == 2025\-08\-01 through 2025\-08\-31 \& \& "2025\-07 \- 2025\-08" == 2025\-07\-01 \- 2025\-08\-31 \& \& \-\-dow0mon \& First day of the week is Monday, rather than Sunday. \& \& (Used only for this/last week calculations.) \& \& \-\-deferral\-detail \& \& Limit detailed deferral reports to the top . 0 \& to suppress entirely. \& \& \-\-detail \& Sets all \-\-*\-detail, \-h and \-u to . Is \& over\-ridden by individual settings. \-\-detail 0 \& suppresses *all* detail. \& \& \-\-dump\-config \& Dump the config to STDOUT and exit. \& \& This can be used as both a debugging aid and as a way \& to develop your first config file. For the latter: \& Simply run your usual pflogsumm command line, adding \& \-\-dump\-config to it, and redirect STDOUT to a file. \& \& To make it cleaner: Remove unset configs: \& \& pflogsumm \-\-dump\-config |grep \-v \*(Aq = $\*(Aq \& \& \-e \& \-\-extended\-detail \& \& Extended (extreme? excessive?) detail \& Emit detailed reports. At present, this includes \& only a per\-message report, sorted by sender domain, \& then user\-in\-domain, then by queue i.d. \& \& WARNING: the data built to generate this report can \& quickly consume very large amounts of memory if a \& lot of log entries are processed! \& \& \-h \& \-\-host\-cnt \& \& top to display in host/domain reports. \& 0 = none. \& \& See also: "\-u" and "\-\-*\-detail" options for further \& report\-limiting options. \& \& \-\-help Emit short usage message and bail out. \& \& (By happy coincidence, "\-h" alone does much the same, \& being as it requires a numeric argument :\-). Yeah, I \& know: lame.) \& \& \-i \& \-\-ignore\-case \& Handle complete email address in a case\-insensitive \& manner. \& \& Normally pflogsumm lower\-cases only the host and \& domain parts, leaving the user part alone. This \& option causes the entire email address to be lower\- \& cased. \& \& \-\-iso\-date\-time \& \& For summaries that contain date or time information, \& use ISO 8601 standard formats (CCYY\-MM\-DD and HH:MM), \& rather than "Mon DD CCYY" and "HHMM". \& \& \-m modify (mung?) UUCP\-style bang\-paths \& \& This is for use when you have a mix of Internet\-style \& domain addresses and UUCP\-style bang\-paths in the log. \& Upstream UUCP feeds sometimes mung Internet domain \& style address into bang\-paths. This option can \& sometimes undo the "damage". For example: \& "somehost.dom!username@foo" (where "foo" is the next \& host upstream and "somehost.dom" was whence the email \& originated) will get converted to \& "foo!username@somehost.dom". This also affects the \& extended detail report (\-e), to help ensure that by\- \& domain\-by\-name sorting is more accurate. \& \& See also: \-\-uucp\-mung \& \& \-\-mailq Run "mailq" command at end of report. \& \& Merely a convenience feature. (Assumes that "mailq" \& is in $PATH. See "$mailqCmd" variable to path thisi \& if desired.) \& \& \-\-no\-no\-msg\-size \& \& Do not emit report on "Messages with no size data". \& \& Message size is reported only by the queue manager. \& The message may be delivered long\-enough after the \& (last) qmgr log entry that the information is not in \& the log(s) processed by a particular run of \& pflogsumm. This throws off "Recipients by message \& size" and the total for "bytes delivered." These are \& normally reported by pflogsumm as "Messages with no \& size data." \& \& \-\-problems\-first \& \& Emit "problems" reports (bounces, defers, warnings, \& etc.) before "normal" stats. \& \& \-\-pscrn\-detail \& \& Limit postscreen detail reporting to top lines of \& each event. 0 to suppress entirely. \& \& Note: Postscreen rejects are collected and reported \& in any event. \& \& \-\-pscrn\-stats \& Collect and emit postscreen summary stats. \& \& Note: Postscreen rejects are collected and reported \& in any event. \& \& \-\-rej\-add\-from \& For those reject reports that list IP addresses or \& host/domain names: append the email from address to \& each listing. (Does not apply to "Improper use of \& SMTP command pipelining" report.) \& \& \-q \& \-\-quiet \& quiet \- don\*(Aqt print headings for empty reports \& \& note: headings for warning, fatal, and "master" \& messages will always be printed. \& \& \-\-rej\-add\-to \& For sender reject reports: Add the intended recipient \& address. \& \& \-\-reject\-detail \& \& Limit detailed smtpd reject, warn, hold and discard \& reports to the top . 0 to suppress entirely. \& \& \-\-smtp\-detail \& \& Limit detailed smtp delivery reports to the top . \& 0 to suppress entirely. \& \& \-\-smtpd\-stats \& Generate smtpd connection statistics. \& \& The "per\-day" report is not generated for single\-day \& reports. For multiple\-day reports: "per\-hour" numbers \& are daily averages (reflected in the report heading). \& \& \-\-smtpd\-warning\-detail \& \& Limit detailed smtpd warnings reports to the top . \& 0 to suppress entirely. \& \& \-\-srs\-mung \& Undo SRS address munging. \& \& If your postfix install has an SRS plugin running, many \& addresses in the report will contain the SRS\-formatted \& email addresses, also for non\-local addresses (f.i. \& senders). This option will try to undo the "damage". \& \& Addresses of the form: \& \& SRS0=A6cv=PT=sender.example.com=support@srs.example.net \& \& will be reformatted to their original value: \& \& support@sender.example.com \& \& \-\-syslog\-name=name \& \& Set syslog\-name to look for for Postfix log entries. \& \& By default, pflogsumm looks for entries in logfiles \& with a syslog name of "postfix," the default. \& If you\*(Aqve set a non\-default "syslog_name" parameter \& in your Postfix configuration, use this option to \& tell pflogsumm what that is. \& \& See the discussion about the use of this option under \& "NOTES," below. \& \& \-u \& \-\-user\-cnt \& \& top to display in user reports. 0 == none. \& \& See also: "\-h" and "\-\-*\-detail" options for further \& report\-limiting options. \& \& \-\-unprocd\-file \& \& Emit unprocessed logfile lines to file \& \& \-\-use\-orig\-to \& \& Where "orig_to" fields are found, report that in place \& of the "to" address. \& \& \-\-uucp\-mung \& modify (mung?) UUCP\-style bang\-paths \& \& See also: \-m \& \& \-\-verbose\-msg\-detail \& \& For the message deferral, bounce and reject summaries: \& display the full "reason", rather than a truncated one. \& \& Note: this can result in quite long lines in the report. \& \& \-\-verp\-mung \& \-\-verp\-mung=2 \& Do "VERP" generated address (?) munging. Convert \& sender addresses of the form \& "list\-return\-NN\-someuser=some.dom@host.sender.dom" \& to \& "list\-return\-ID\-someuser=some.dom@host.sender.dom" \& \& In other words: replace the numeric value with "ID". \& \& By specifying the optional "=2" (second form), the \& munging is more "aggressive", converting the address \& to something like: \& \& "list\-return@host.sender.dom" \& \& Actually: specifying anything less than 2 does the \& "simple" munging and anything greater than 1 results \& in the more "aggressive" hack being applied. \& \& See "NOTES" regarding this option. \& \& \-\-version Print program name and version and bail out. \& \& \-x Enable debugging to STDERR \& \& \-\-zero\-fill "Zero\-fill" certain arrays so reports come out with \& data in columns that that might otherwise be blank. .Ve .SH "RETURN VALUE" .IX Header "RETURN VALUE" .Vb 1 \& Pflogsumm doesn\*(Aqt return anything of interest to the shell. .Ve .SH ERRORS .IX Header "ERRORS" .Vb 1 \& Error messages are emitted to stderr. .Ve .SH EXAMPLES .IX Header "EXAMPLES" .Vb 1 \& Produce a report of previous day\*(Aqs activities: \& \& pflogsumm \-d yesterday /var/log/maillog \& \& A report of prior week\*(Aqs activities: \& \& pflogsumm \-d last_week /var/log/maillog.0 \& \& What\*(Aqs happened so far today: \& \& pflogsumm \-d today /var/log/maillog \& \& Crontab entry to generate a report of the previous day\*(Aqs activity \& at 10 minutes after midnight: \& \& 10 0 * * * /usr/local/sbin/pflogsumm \-d yesterday /var/log/maillog \& 2>&1 |/usr/bin/mailx \-s "\`uname \-n\` daily mail stats" postmaster \& \& Crontab entry to generate a report for the prior week\*(Aqs activity. \& \& 10 4 * * 0 /usr/local/sbin/pflogsumm \-d "last week" /var/log/maillog.0 \& 2>&1 |/usr/bin/mailx \-s "\`uname \-n\` weekly mail stats" postmaster \& \& (The two crontab examples, above, must actually be a single line \& each. They\*(Aqre broken\-up into two\-or\-more lines due to page \& formatting issues.) \& \& Using a config file: \& \& pflogsumm \-\-config /usr/local/etc/pflogusmm/daily.conf \& \& Using a config file, overriding a config file options on the command \& line: \& \& pflogsumm \-\-config /usr/local/etc/pflogsumm/daily.conf \& \-\-detail 30 \& \& This would override *all* detail settings in the config \& file, setting them all to 30. \& \& pflogsumm \-\-config /usr/local/etc/pflogsumm/daily.conf \& \-\-detail 30 \-\-host\-cnt 10 \& \& This would override all detail settings in the config \& file, setting them all to 30, with the global detail \& setting in turn being overridden to 10 for host count. .Ve .SH "SEE ALSO" .IX Header "SEE ALSO" .Vb 1 \& pffrombyto, pftobyfrom \& \& The pflogsumm FAQ: pflogsumm\-faq.txt. .Ve .SH NOTES .IX Header "NOTES" .Vb 4 \& Some options, such as date range, have both short\-form and \& long\-form names. In the interest of brevity, only the \& short\-form options are shown in the SYNOPSIS and in \& pflogsumm\*(Aqs "help" output. \& \& Pflogsumm makes no attempt to catch/parse non\-Postfix log \& entries. Unless it has "postfix/" in the log entry, it will be \& ignored. \& \& It\*(Aqs important that the logs are presented to pflogsumm in \& chronological order so that message sizes are available when \& needed. \& \& For display purposes: integer values are munged into "kilo" and \& "mega" notation as they exceed certain values. I chose the \& admittedly arbitrary boundaries of 512k and 512m as the points at \& which to do this\-\-my thinking being 512x was the largest number \& (of digits) that most folks can comfortably grok at\-a\-glance. \& These are "computer" "k" and "m", not 1000 and 1,000,000. You \& can easily change all of this with some constants near the \& beginning of the program. \& \& "Items\-per\-day" reports are not generated for single\-day \& reports. For multiple\-day reports: "Items\-per\-hour" numbers are \& daily averages (reflected in the report headings). \& \& Message rejects, reject warnings, holds and discards are all \& reported under the "rejects" column for the Per\-Hour and Per\-Day \& traffic summaries. \& \& Verp munging may not always result in correct address and \& address\-count reduction. \& \& Verp munging is always in a state of experimentation. The use \& of this option may result in inaccurate statistics with regards \& to the "senders" count. \& \& UUCP\-style bang\-path handling needs more work. Particularly if \& Postfix is not being run with "swap_bangpath = yes" and/or *is* being \& run with "append_dot_mydomain = yes", the detailed by\-message report \& may not be sorted correctly by\-domain\-by\-user. (Also depends on \& upstream MTA, I suspect.) \& \& The "percent rejected" and "percent discarded" figures are only \& approximations. They are calculated as follows (example is for \& "percent rejected"): \& \& percent rejected = \& \& (rejected / (delivered + rejected + discarded)) * 100 \& \& There are some issues with the use of \-\-syslog\-name. The problem is \& that, even with Postfix\*(Aq $syslog_name set, it will sometimes still \& log things with "postfix" as the syslog_name. This is noted in \& /etc/postfix/sample\-misc.cf: \& \& # Beware: a non\-default syslog_name setting takes effect only \& # after process initialization. Some initialization errors will be \& # logged with the default name, especially errors while parsing \& # the command line and errors while accessing the Postfix main.cf \& # configuration file. \& \& As a consequence, pflogsumm must always look for "postfix," in logs, \& as well as whatever is supplied for syslog_name. \& \& Where this becomes an issue is where people are running two or more \& instances of Postfix, logging to the same file. In such a case: \& \& . Neither instance may use the default "postfix" syslog name \& and... \& \& . Log entries that fall victim to what\*(Aqs described in \& sample\-misc.cf will be reported under "postfix", so that if \& you\*(Aqre running pflogsumm twice, once for each syslog_name, such \& log entries will show up in each report. \& \& The Pflogsumm Home Page is at: \& \& http://jimsun.LinxNet.com/postfix_contrib.html .Ve .SH REQUIREMENTS .IX Header "REQUIREMENTS" .Vb 1 \& Requires Perl 5.10, minimum, and Date::Calc \& \& For \-\-config, Pflogsumm requires the Config::Simple module. \& \& Both of the above can be obtained from CPAN at http://www.perl.com \& or from your distro\*(Aqs repository. \& \& Pflogsumm is currently written and tested under Perl 5.38. \& As of version 19990413\-02, pflogsumm worked with Perl 5.003, but \& future compatibility is not guaranteed. .Ve .SH LICENSE .IX Header "LICENSE" .Vb 4 \& This program is free software; you can redistribute it and/or \& modify it under the terms of the GNU General Public License \& as published by the Free Software Foundation; either version 2 \& of the License, or (at your option) any later version. \& \& This program is distributed in the hope that it will be useful, \& but WITHOUT ANY WARRANTY; without even the implied warranty of \& MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the \& GNU General Public License for more details. \& \& You may have received a copy of the GNU General Public License \& along with this program; if not, write to the Free Software \& Foundation, Inc., 59 Temple Place \- Suite 330, Boston, MA 02111\-1307, \& USA. \& \& An on\-line copy of the GNU General Public License can be found \& http://www.fsf.org/copyleft/gpl.html. .Ve pflogsumm-1.1.13/ToDo0000644000175000017500000000505415051123406014713 0ustar jseymourjseymour To Be Done (Maybe) Fix parsing for "451 4.3.5 Server configuration error;" date ranges, "lastweek", etc. (Done) (options for?) break-down by local vs. non-local?, further "drill-downs" to sender/recipient domains? Separate reports by-domain? (Would require that pflogsumm write files instead of emitting to stdout? Or maybe that would be an option?) Separate options for sender domain and receiver domain? Or perhaps sender address and receiver address regex "filtering" options? lower-case domains (Done) don't k-ize msg counts? (Or do so at a higher boundary?) (Done) implement proper version numbering (interim version numbering in place) (use SCCS or RCS?) (Done) add changelog (Done) Expand docs (in code?) Re-do docs to POD format? (Done) Improve UUCP-style bang-path handling Add option to use disk-based hash files instead of in-memory so processing logfiles with lots of postfix log entries doesn't run the machine out of memory? (This would really whack performance!) (Maybe only needed with "-e" option.) Add another "step" to integer processing: "g"? (Display formatting will presently mess up at values exceeding 999999m.) Internationalization? Necessary? There's English-language month abbreviations hard-coded in pflogsumm. I'm admittedly real weak in this area. (Covered by ISO 8601 option [below]?) Add option to lower-case entire addresses, rather than only the domains. (Or make that the default, and add an option to make it just domains?) (The RFCs say username parts shouldn't be touched!) (Done) Add percentage-of-total to all (?) reports? Add ability to handle compressed files within pflogsumm? (Unlikely) SMTP logging by host (msgs, connects) (add to new "by-domain" reports?) (Done) UUCP logging of some sort? (At least what is sent to what host/gateway. Since rmail receives incoming, don't know as I can do anything there.) Option for ISO 8601 standard date & time formats. (Done) Option for specifying "host" (for multi-host logfiles)? Add POSIX regexp body_checks "bypass" docs to FAQ. (Liviu Daia, Noel Jones) Add "helper" program to distribution based on Wolfgang Zeikat's MIME::Lite suggestion for the body_checks issue? (Will want to make it read from stdin, securely create tempfile and clean up afterward.) Expand on SAV reporting? (Len Conrad) Add "reject details" on "too many errors after " log lines? (Len Conrad)