Decorators
Decorators are intuitive and extremely useful. To demonstrate, we’ll look at a simple example. Let’s say we’ve got some function that sums all numbers 0 to n:
def sum_0_to_n(n):
count = 0
while n > 0:
count += n
n -= 1
return count
and we’d like to time the performance of this function. Of course we could just modify the function like so:
import time
def sum_0_to_n(n):
start = time.time()
count = 0
while n > 0:
count += n
n -= 1
end = time.time()
print "function took %f seconds" % (end-start)
return count
but this is poor practice because every time we want to time a new function we would need to repeat ourselves by modifying it with the time variables. Instead, let’s make a new function that can take our sum_0_to_n function as its argument:
def timethis(func):
'''
This function wraps other functions and reports the execution time.
'''
def wrapper(n):
start = time.time()
result = func(n)
end = time.time()
print "function took %f seconds" % (end-start)
return result
return wrapper
Great! Now we can just call our sum function as the argument of our time function:
sum_0_to_n_timed = timethis(sum_0_to_n) sum_0_to_n_timed(100)
Now, if you’ve followed this argument and are comfortable with higher-order functions, then you already understand decorators. In fact, decorators are just syntactic sugar for passing functions to higher-order functions. Instead of this:
sum_0_to_n_timed = timethis(sum_0_to_n)
We can just do this to our original sum function:
@timethis
def sum_0_to_n(n):
count = 0
while n > 0:
count += n
n -= 1
return count
And get the same result as we did when explicitly passing in our sum function to the time function. Decorators are very handy tools to keep repetition at a minimum and to distribute properties to functions on a large scale – when you want a new function to be timed, just add the decorator at the top.
Let’s say we do indeed want to start timing all of our functions with the timethis decorator. For example, let’s add it to our very useful sum+m function:
@timethis
def sum_0_to_n_plus_m(n, m):
'''
Sum to n plus m.
'''
count = 0
while n > 0:
count += n
n -= 1
return count + m
However, when we run it we get a type error:
TypeError Traceback (most recent call last) in () ----> 1 sum_0_to_n_plus_m(10, 11111) TypeError: wrapper() takes exactly 1 argument (2 given)
It looks like our timethis function needs a modification to handle functions with more than one argument. This is where the * operator comes in: it tells our function to expect any number of arguments:
def timethis(func):
'''
This function wraps other functions and reports the execution time.
'''
def wrapper(*args, **kwargs):
'''
This is the wrapper documentation.
'''
start = time.time()
result = func(*args, **kwargs)
end = time.time()
print end-start
return result
return wrapper
OK! One last modification to our decorator function. Let’s say that sum_0_to_n_plus_m has been implemented and distributed in a widely used library for inelegant mathematicians. One such inelegant mathematician is having trouble with the function so calls the function documentation:
print sum_0_to_n_plus_m.__doc__
But instead sees this:
This is the wrapper documentation.
The docstring and function attributes are lost in the wrapper. So now we need a way to preserve this information, which is where functools wraps comes in: by adding the @wraps decorator to our own decorators, the function attributes are preserved:
from functools import wraps
def timethis(func):
'''
This function wraps other functions and reports the execution time.
'''
@wraps(func)
def wrapper(*args, **kwargs):
'''
This is the wrapper documentation.
'''
start = time.time()
result = func(*args, **kwargs)
end = time.time()
print end-start
return result
return wrapper
That’s right, we put a decorator inside of our decorator. At the time of writing, the Xzibit meme has already fallen out of fashion and is omitted; its inclusion would would only confuse and dishearten posterity. Anyway, @wraps is not terribly important, but it’s good practice to use it and it comes up frequently.
At this point we have seen the basic structure of robust decorator: a higher order function that can take (nearly) any other function as argument and will preserve the argument metadata. Decorators can be used in all kinds of ways, and can be extended to take arguments, take adjustable user attributes, work in and out of classes with different inheritance properties, etc. For example, here’s a decorator that lets us optionally call timethis from the wrapped function itself:
def optional_timethis(func):
@wraps(func)
def wrapper(time_me=False, *args, **kwargs):
if time_me:
start = time.time()
result = func(*args, **kwargs)
end = time.time()
print end-start
return result
else:
return func(*args, **kwargs)
return wrapper
So that we can just add the (time_me=True) param if interested in the performance of our function. [Note that code above is for Python 3.3; Python 2.7 doesn’t seem to support *args and **kwargs when passing optional arguments to the wrapped function.]
Metaprogramming
Now that we’ve been passing functions as objects to other functions, a question naturally comes up: can we do the same thing with classes? Let’s try it:
def Dog_maker():
class Dog(object):
def __init__(self, name):
self.name = name
self.toys = []
def add_toy(self, toy):
self.toys.append(toy)
return Dog
Dog = Dog_maker()
Fido = Dog("Fido")
Fido.add_toy("frisbee")
Fido.toys
['frisbee']
Interesting. A class is also an object. So what’s the type of class? In our example, the type of Fido is Dog, so what’s the type of Dog? It turns out that the class of Dog and all other classes is “type.” This is our introduction to the “type” metaclass, which is just a special object that creates classes.
Type is a constructor that takes:
- name – name of the class
- bases – parent classes of the class
- dictionary – attributes and methods of the class
Interestingly enough, classes can be written in this format to explicitly call the type constructor; the two blocks below are equivalent:
class Animal(object):
status = "good boy"
class Dog(Animal):
def get_status(self):
return self.status
Animal = type('Animal', (), dict(status="good boy"))
Dog = type('Dog', (Animal,), dict(get_status = lambda self: self.status))
The interesting part of this new type interface for class construction is that it opens up the ability to generate classes programmatically. If we need to generate a number of new classes with different names, parent classes, methods, and attributes we can simply iterate these over something like generate_subclass:
def generate_subclasses(description, my_dict, *up_class):
return type(description, (up_class), my_dict)
It’s time to ‘fess up. While decorators are extremely useful, metaclasses – not so much. I’ve never needed them (though you can always bend the problem to your will), and based on what others have written they are rarely the best design choice, not least because no one reading your code will know how to use them. However knowing about metaclasses and understanding classes as objects has freed me up to consider new approaches I might not have considered otherwise. And, of course, a deeper understanding of the language is a reward in itself.
Sources:
Post by Jake Vanderplas (core sklearn developer with some really interesting posts)
Python Patterns, Recipes, and Idioms
Python documentation
Nick Ryan 