Problem with SMTPHandler + logging.conf

[PaulStat]

I'm trying to use pythons logging.handlers.SMTPHandler with a configuration file (so that I don't have to have passwords etc. inside of the script)

Now the guide I'm following is here, now the RotatingFileHandler is working, but I never receive an email, or an error for the SMTPHandler.

Anyway here's the python code

import logging
import logging.config

logDir = "./logs/"

logging.config.fileConfig(logDir+'logging.conf')
logging.getLogger('email')

logging.debug('THIS IS A DEBUG MESSAGE')
logging.error('THIS IS AN ERROR')

And here's the config file (xx's denote passwords and stuff which I've removed for the sake of posting here)

[loggers]
keys=root,email

[logger_root]
level=DEBUG
handlers=rotatingFileHandler

[logger_email]
level=ERROR
handlers=email
qualname=email

[formatters]
keys=emailFormatter,rotatingFormatter

[formatter_emailFormatter]
format=%(asctime)s %(name)-12s %(levelname)-8s %(message)s

[formatter_rotatingFormatter]
format=%(asctime)s %(name)-12s %(levelname)-8s %(message)s
datefmt=%m-%d %H:%M

[handlers]
keys=email,rotatingFileHandler

[handler_email]
class=handlers.SMTPHandler
level=ERROR
formatter=emailFormatter
args=('xx','xx','xx','ERROR!',['xx','xx'])
host=xx
from=xx
to=xx
subject=ERROR!

[handler_rotatingFileHandler]
class=handlers.RotatingFileHandler
level=DEBUG
formatter=rotatingFormatter
args=('./logs/log.out', 'maxBytes=1000000', 'backupCount=5')

Reading the link I posted above it says

The args entry, when eval()uated in the context of the logging package’s namespace, is the list of arguments to the constructor for the handler class. Refer to the constructors for the relevant handlers, or to the examples below, to see how typical entries are constructed.

Now the constructor for the SMTPHandler takes the form

class logging.handlers.SMTPHandler(mailhost, fromaddr, toaddrs, subject[, credentials])¶

    Returns a new instance of the SMTPHandler class. The instance is initialized with the from and to addresses and subject line of the email. The toaddrs should be a list of strings. To specify a non-standard SMTP port, use the (host, port) tuple format for the mailhost argument. If you use a string, the standard SMTP port is used. If your SMTP server requires authentication, you can specify a (username, password) tuple for the credentials argument.

But looking at their example for SMTPHandler they have their args SMTPHandler set like so

[handler_hand07]
class=handlers.SMTPHandler
level=WARN
formatter=form07
args=('localhost', 'from@abc', ['user1@abc', 'user2@xyz'], 'Logger Subject')

Note how the args aren't in the same order as the constructor specifies and it's missing the to field, I've tried using the same format as above in my config file but I still receive no email and no errors.

The arguments I've defined for SMTP are correct as I previously had the SMTPHandler directly instantiated in the code and it worked.

Can someone please help me figure out how to get it working using a config file :)

[Gribouillis]

You may have mismatched the arguments format. The 3rd argument (the 'to' field) is a list of addresses like ['user1@abc', 'user2@xyz'] . Your args should probably look like

args=('xx','xx',['myemailaddress',],'ERROR!',('myusername','mypassword'))

Note the list and the tuple in the arglist.

[PaulStat]

Ah ok thanks I did misread it, I tried with the format you provided above, but I'm still not receiving any email.

Is there anyway to debug this type of issue, seeing as I'm receiving no error.

[Gribouillis]

Ah ok thanks I did misread it, I tried with the format you provided above, but I'm still not receiving any email.

Is there anyway to debug this type of issue, seeing as I'm receiving no error.

I copied the SMTPHandler.emit method (in my python 2.6 distribution). This method send the email using the smtplib module. You could write this in your code before using SMTPHandler

from logging.handlers import SMTPHandler

def get_emit():
    def emit(self, record):
        """
        Emit a record.

        Format the record and send it to the specified addressees.
        """
        try:
            import smtplib
            try:
                from email.utils import formatdate
            except ImportError:
                formatdate = self.date_time
            port = self.mailport
            if not port:
                port = smtplib.SMTP_PORT
            smtp = smtplib.SMTP(self.mailhost, port)
            msg = self.format(record)
            msg = "From: %s\r\nTo: %s\r\nSubject: %s\r\nDate: %s\r\n\r\n%s" % (
                            self.fromaddr,
                            string.join(self.toaddrs, ","),
                            self.getSubject(record),
                            formatdate(), msg)
            if self.username:
                smtp.login(self.username, self.password)
            smtp.sendmail(self.fromaddr, self.toaddrs, msg)
            smtp.quit()
        except (KeyboardInterrupt, SystemExit):
            raise
        except:
            self.handleError(record)
    return emit

SMTPHandler.emit = get_emit()

Now the SMTPHandler class uses the copied emit function. You could add print statements in this function to see if the sender uses the correct username, password, fromaddr, toaddrs, msg, etc.

[PaulStat]

I must be doing something wrong, I changed the python code but the print statements don't work?

import logging
import logging.config

from logging.handlers import SMTPHandler

def get_emit():
   def emit(self, record):
      """
      Emit a record.
      Format the record and send it to the specified addressees.
      """
      try:
         import smtplib
         try:
            from email.utils import formatdate
         except ImportError:
            formatdate = self.date_time
         port = self.mailport
         if not port:
            port = smtplib.SMTP_PORT
         smtp = smtplib.SMTP(self.mailhost, port)
         msg = self.format(record)
         msg = "From: %s\r\nTo: %s\r\nSubject: %s\r\nDate: %s\r\n\r\n%s" % (
                        self.fromaddr,
                        string.join(self.toaddrs, ","),
                        self.getSubject(record),
                        formatdate(), msg)
         print "Mailhost: " + self.mailhost+" From: " +self.fromaddr +" To: " + self.toaddrs +" User: " +self.username +" Password: " +self.password
         if self.username:
            smtp.login(self.username, self.password)
         smtp.sendmail(self.fromaddr, self.toaddrs, msg)
         print "DONE"
         smtp.quit()         
      except (KeyboardInterrupt, SystemExit):
         print "EXIT"
         raise
      except:
         print "ERROR"
         self.handleError(record)
   return emit

logDir = "./logs/"

SMTPHandler.emit = get_emit()

logging.config.fileConfig(logDir+'logging.conf')
logging.getLogger('email')

logging.debug('THIS IS A DEBUG MESSAGE')
logging.error('THIS IS AN ERROR')

[Gribouillis]

Yes, I don't know where the problem comes from. Perhaps from a question of shared ressource, because the logging module uses threads. You could try this after the loggers and handlers are created

mylogger = logging.getLogger('email')
import functools
for h in mylogger.handlers:
    if isinstance(h, SMTPHandler):
        h.emit = functools.partial(get_emit(), h)

[PaulStat]

well it still doesn't print anything from emit, but If I change your code above to

mylogger = logging.getLogger('email')
import functools
for h in mylogger.handlers:
    if isinstance(h, SMTPHandler):
        print "We have a SMTPHandler"
        h.emit = functools.partial(get_emit(), (h,))

It does print "We have a SMTPHandler"

[Gribouillis]

Ok, let's try something even worse: define a new class

class MyHandler(SMTPHandler):
    def emit(self, record):
        """
        Emit a record.

        Format the record and send it to the specified addressees.
        """
        try:
            import smtplib
            try:
                from email.utils import formatdate
            except ImportError:
                formatdate = self.date_time
            port = self.mailport
            if not port:
                port = smtplib.SMTP_PORT
            smtp = smtplib.SMTP(self.mailhost, port)
            msg = self.format(record)
            msg = "From: %s\r\nTo: %s\r\nSubject: %s\r\nDate: %s\r\n\r\n%s" % (
                            self.fromaddr,
                            string.join(self.toaddrs, ","),
                            self.getSubject(record),
                            formatdate(), msg)
            if self.username:
                smtp.login(self.username, self.password)
            smtp.sendmail(self.fromaddr, self.toaddrs, msg)
            smtp.quit()
        except (KeyboardInterrupt, SystemExit):
            raise
        except:
            self.handleError(record)

mylogger = logging.getLogger('email')
import functools
for h in mylogger.handlers:
    if isinstance(h, SMTPHandler):
        print "We have a SMTPHandler"
        h.__class__ = MyHandler

[PaulStat]

Still only "We have a SMTPHandler", I tried changing the logging.conf file to say class=MyHandler instead of class=handlers.SMTPHandler but it didn't like that

[Gribouillis]

Last solution I see: save a copy of the file logging/handlers.py in your python library (call it save_handler.py for example), then put the print statements in handlers.py and when everything is debugged, reinstall your saved copy as handler.py (it is not very dangerous because the python source code is available from python.org).

[PaulStat]

I had considered doing that, but I assumed the python executable would be using the compiled version of the code i.e. the .pyc

Anyway I put some print statements inside of the emit method for SMTPHandler of C:\Python26\Lib\logging\handlers.py, still nothing. I then out of curiosity put a print "hello" inside of the __init__ and that works! So emit() isn't being called from what I can tell.

Edit: Further to that I modified the __init__ to print out the mailhost, from, to etc. And they're all correct.

[Gribouillis]

I had considered doing that, but I assumed the python executable would be using the compiled version of the code i.e. the .pyc

Anyway I put some print statements inside of the emit method for SMTPHandler of C:\Python26\Lib\logging\handlers.py, still nothing. I then out of curiosity put a print "hello" inside of the __init__ and that works! So emit() isn't being called from what I can tell.

Edit: Further to that I modified the __init__ to print out the mailhost, from, to etc. And they're all correct.

If emit is not being called, it looks like a very good reason why you don't receive any email, so the question now is to find why this emit is not called (by the way, when a .py has been modified after the .pyc was created, the .pyc is rewritten by python and the more recent .py is used).

[PaulStat]

hmmm yes, any suggestions how I go about that? :)

[Gribouillis]

hmmm yes, any suggestions how I go about that? :)

Thanks to google, I discovered this module which offers new mailing handlers for the logging module http://pypi.python.org/pypi/mailinglogger. Install it with easy_install. I think it's the best I can do (I never logged through email) !

In fact, I'm not sure that it will solve your problem: it seems to declare a subclass of SMTPHandler witha new emit method.

[PaulStat]

It turns out the issue was to do with

logging.getLogger('email')

logging.debug('THIS IS A DEBUG MESSAGE')
logging.error('THIS IS AN ERROR MESSAGE')

logging relates to the root logger and because the email logger was a child of the root logger the error message never gets sent to the email logger. Whereas if I changed it to the following, it correctly uses the email logger as well as propagating to the root logger

logger = logging.getLogger('email')

logger.debug('THIS IS A DEBUG MESSAGE')
logger.error('THIS IS AN ERROR MESSAGE')

[Gribouillis]

It turns out the issue was to do with

logging.getLogger('email')

logging.debug('THIS IS A DEBUG MESSAGE')
logging.error('THIS IS AN ERROR MESSAGE')

logging relates to the root logger and because the email logger was a child of the root logger the error message never gets sent to the email logger. Whereas if I changed it to the following, it correctly uses the email logger as well as propagating to the root logger

logger = logging.getLogger('email')

logger.debug('THIS IS A DEBUG MESSAGE')
logger.error('THIS IS AN ERROR MESSAGE')

Nice find. We should have seen that in the source code :).