Python trick

The following tricks I find pretty useful in my daily Python work. I also added a few I stumbled upon lately.

1. Use collections

This really makes your code more elegant and less verbose, a few examples I absorbed this week:

Named tuples:

>>> Point = collections.namedtuple('Point', ['x', 'y'])
>>> p = Point(x=1.0, y=2.0)
>>> p
Point(x=1.0, y=2.0)

Now you can index by keyword, much nicer than offset into tuple by number (less readable)

>>> p.x
1.0
>>> p.y

Elegantly used when looping through a csv:

with open('stock.csv') as f:
    f_csv = csv.reader(f)
 headings = next(f_csv)
 Row = namedtuple('Row', headings)
 for r in f_csv:
 row = Row(*r) # note the star extraction
 # ... process row ...

I like the unpacking star feature to throw away useless fields:

line = 'nobody:*:-2:-2:Unprivileged User:/var/empty:/usr/bin/false'
>>> uname, *fields, homedir, sh = line.split(':')
>>> uname
'nobody'
>>> homedir
'/var/empty'
>>> sh
'/usr/bin/false'

Superconvenient: the defaultdict:

from collections import defaultdict
rows_by_date = defaultdict(list)
for row in rows:
    rows_by_date[row['date']].append(row)",

Before I would init the list each time which leads to needless code:

if row['date'] not in rows_by_date:

 rows_by_date[row['date']] = []

You can use OrderedDict to leave the order of inserted keys:

>>> import collections
>>> d = collections.OrderedDict()
>>> d['a'] = 'A'
>>> d['b'] = 'B'
>>> d['c'] = 'C'
>>> d['d'] = 'D'
>>> d['e'] = 'E'
>>> for k, v in d.items():
...     print k, v
... 
a A
b B
c C
d D
e E

Another nice one is Counter:

from collections import Counter

words = [
  'look', 'into', 'my', 'eyes', 'look', 'into', 'my', 'eyes',
  'the', 'eyes', 'the', 'eyes', 'the', 'eyes', 'not', 'around', 'the',
  'eyes', ""don't"", 'look', 'around', 'the', 'eyes', 'look', 'into',
  'my', 'eyes', ""you're"", 'under'
]
word_counts = Counter(words)
top_three = word_counts.most_common(3)
print(top_three)
# Outputs [('eyes', 8), ('the', 5), ('look', 4)]",

Again, before I would write most_common manually. Not necessary, this is all done already somewhere in the stdlib :)

2. sorted() accepts a key arg which you can use to sort on something else

Here for example we sort on surname:

>>> sorted(names, key=lambda name: name.split()[-1].lower())
['Ned Batchelder', 'David Beazley', 'Raymond Hettinger', 'Brian Jones']

3. Create XMl from dict

Creating XML tags manually is usually a bad idea, I bookmarked this simple dict_to_xml helper:

from xml.etree.ElementTree import Element
def dict_to_xml(tag, d):
    '''
    Turn a simple dict of key/value pairs into XML
    '''
    elem = Element(tag)
    for key, val in d.items():
        child = Element(key)
        child.text = str(val)
        elem.append(child)
    return elem"

4. Oneliner to see if there are any python files in a particular directory

Sometimes ‘any’ is pretty useful:

import os
files = os.listdir('dirname')
if any(name.endswith('.py') for name in files):
5. Use set operations to match common items in lists
>>> a = [1, 2, 3, 'a']
>>> b = ['a', 'b', 'c', 3, 4, 5]
>>> set(a).intersection(b)
{3, 'a'}

6. Use re.compile

If you are going to check a regular expression in a loop, don’t do this:

for i in longlist:
  if re.match(r'^...', i)
yet define the regex once and use the pattern:
p = re.compile(r'^...')
for i in longlist: 
  if p.match(i)

7. Printing files with potential bad (Unicode) characters

The book suggested to print filenames of unknown origin, use this convention to avoid errors:

def bad_filename(filename):
    return repr(filename)[1:-1]
try:
    print(filename)
except UnicodeEncodeError:
    print(bad_filename(filename))

Handling unicode chars in files can be nasty because they can blow up your script. However the logic behind it is not that hard to grasp. A good snippet to bookmark is the encoding / decoding of Unicode:

>>> a
'pýtĥöñ is awesome\n'
>>> b = unicodedata.normalize('NFD', a)
>>> b.encode('ascii', 'ignore').decode('ascii')
'python is awesome\n'

O’Reilly has a course on Working with Unicode in Python.

8. Print is pretty cool (Python 3)

I am probably not the only one writing this kind of join operations:

>>> row = ["1", "bob", "developer", "python"]
>>> print(','.join(str(x) for x in row))
1,bob,developer,python

Turns out you can just write it like this:

>>> print(*row, sep=',')
1,bob,developer,python
Note again the * unpacking.

9. Functions like sum() accept generators / use the right variable type

I wrote this at a conference to earn me a coffee mug ;)

sum = 0
for i in range(1300):
    if i % 3 == 0 or i % 5 == 0:
        sum += i
print(sum)

Returns 394118, while handing it in I realized this could be written much shorter and efficiently:

>>> sum(i for i in range(1300) if i % 3 == 0 or i % 5 == 0)
394118

A generator:

lines = (line.strip() for line in f)

is more memory efficient than:

lines = [line.strip() for line in f] # loads whole list into memory at once

And concatenating strings is inefficient:

s = "line1\n"
s += "line2\n"
s += "line3\n"
print(s)

Better build up a list and join when printing:

lines = []
lines.append("line1")
lines.append("line2")
lines.append("line3")
print("\n".join(lines))
Another one I liked from the cookbook:
portfolio = [
  {'name':'GOOG', 'shares': 50},
  {'name':'YHOO', 'shares': 75},
  {'name':'AOL', 'shares': 20},
  {'name':'SCOX', 'shares': 65}
]
min_shares = min(s['shares'] for s in portfolio)

One line to get the min of a numeric value in a nested data structure.

10. Enumerate lines in for loop

You can number lines (or whatever you are looping over) and start with 1 (2nd arg), this is a nice debugging technique

for lineno, line in enumerate(lines, 1): # start counting at 0
    fields = line.split()
    try:
        count = int(fields[1])
        ...
    except ValueError as e:
        print('Line {}: Parse error: {}'.format(lineno, e))

11. Pandas

Import pandas and numpy:

import pandas as pd
import numpy as np

12. Make random dataframe with three columns:

df = pd.DataFrame(np.random.rand(10,3), columns=list('ABC'))
Select:
# Boolean indexing (remember the parentheses)
df[(df.A < 0.5) & (df.B > 0.5)]
# Alternative, using query which depends on numexpr
df.query('A < 0.5 & B > 0.5')
Project:
# One columns
df.A
# Multiple columns
# there may be another shorter way, but I don't know it
df.loc[:,list('AB')]

• Often used snippets
• Dates

13. Difference (in days) between two dates:

from datetime import date

d1 = date(2013,1,1)
d2 = date(2013,9,13)
abs(d2-d1).days
directory-of-script snippet
os.path.dirname(os.path.realpath(__file__))
# combine with
os.path.join(os.path.dirname(os.path.realpath(__file__)), 'foo','bar','baz.txt')

14. PostgreSQL-connect-query snippet

import psycopg2
conn = psycopg2.connect("host='localhost' user='xxx'  password='yyy' dbname='zzz'")
cur = conn.cursor()
cur.execute("""SELECT * from foo;""")
rows = cur.fetchall()
for row in rows:
  print "   ", row[0]
conn.close()
Input parsing functions

15. Expand input-file args:

# input_data: e.g. 'file.txt' or '*.txt' or 'foo/file.txt' 'bar/file.txt'
filenames = [glob.glob(pathexpr) for pathexpr in input_data]
filenames = [item for sublist in filenames for item in sublist]

15. Parse key-value pair strings like ‘x=42.0,y=1’:

kvp = lambda elem,t,i: t(elem.split('=')[i])
parse_kvp_str = lambda args : dict([(kvp(elem,str,0), kvp(elem,float,1)) for elem in args.split(',')])
parse_kvp_str('x=42.0,y=1')

Postgres database functions

16. Upper case in Python (just for example):

-- create extension plpythonu;
CREATE OR REPLACE FUNCTION python_upper
(
  input text
) RETURNS text AS
$$
  return input.upper()
$$ LANGUAGE plpythonu STRICT;

17. Convert IP address from text to integer:

CREATE FUNCTION ip2int(input text) RETURNS integer
LANGUAGE plpythonu
AS $$
if 'struct' in SD:
    struct = SD['struct']
else:
    import struct
    SD['struct'] = struct
if 'socket' in SD:
    socket = SD['socket']
else:
    import socket
    SD['socket'] = socket
return struct.unpack("!I", socket.inet_aton(input))[0]
$$;
Convert IP address from integer to text:
CREATE FUNCTION int2ip(input integer) RETURNS text
LANGUAGE plpythonu
AS $$
if 'struct' in SD:
    struct = SD['struct']
else:
    import struct
    SD['struct'] = struct
if 'socket' in SD:
    socket = SD['socket']
else:
    import socket
    SD['socket'] = socket
return socket.inet_ntoa(struct.pack("!I", input))
$$;

18. Commandline options

optparse-commandline-options snippet
from optparse import OptionParser
usage = "usage: %prog [options] arg "
parser = OptionParser(usage=usage)
parser.add_option("-x", "--some-option-x", dest="x", default=42.0, type="float",
                  help="a floating point option")
(options, args) = parser.parse_args()
print options.x
print args[0]

19. print-in-place (progress bar) snippet

import time
import sys
for progress in range(100):
  time.sleep(0.1)
  sys.stdout.write("Download progress: %d%%   \r" % (progress) ) 
  sys.stdout.flush()

Packaging snippets

20. poor-mans-python-executable trick

Learned this trick from voidspace. The trick uses two files (__main__.py and hashbang.txt):

__main__.py:
print 'Hello world'
hashbang.txt (adding a newline after ‘python2.6’ is important):
#!/usr/bin/env python2.6
Build an “executable”:
zip main.zip __main__.py
cat hashbang.txt main.zip > hello
rm main.zip
chmod u+x hello
Run “executable”:
$ ./hello
Hello world

21. import-class-from-file trick

Import class MyClass from a module file (adapted from stackoverflow):

import imp
mod = imp.load_source('name.of.module', 'path/to/module.py')
obj = mod.MyClass()

22. Occusional-usage snippets

Extract words from string

words = lambda text: ''.join(c if c.isalnum() else ' ' for c in text).split()
words('Johnny.Appleseed!is:a*good&farmer')
# ['Johnny', 'Appleseed', 'is', 'a', 'good', 'farmer']

23. IP address to integer and back

import struct
import socket
def ip2int(addr):                                                               
    return struct.unpack("!I", socket.inet_aton(addr))[0]                       
def int2ip(addr):                                                               
    return socket.inet_ntoa(struct.pack("!I", addr))

24. Fluent Python Interface

Copied from riaanvddool.

# Fluent Interface Definition
class sql:
  class select:
    def __init__(self, dbcolumn, context=None):
      self.dbcolumn = dbcolumn
      self.context = context
    def select(self, dbcolumn):
      return self.__class__(dbcolumn,self)
# Demo
q = sql.select('foo').select('bar')
print q.dbcolumn  #bar
print q.context.dbcolumn  #foo
Flatten a nested lists
def flatten(elems):
 """
 [['a'], ['b','c',['d'],'e',['f','g']]]
 """
 stack = [elems]
 top = stack.pop()
 while top:  
  head, tail = top[0], top[1:]
  if tail: stack.append(tail)
  if not isinstance(head, list): yield head   
  else: stack.append(head)
  if stack: top = stack.pop()
  else: break
snap rounding
EPSILON = 0.000001
snap_ceil = lambda x: math.ceil(x) if abs(x - round(x)) > EPSILON else round(x)
snap_floor = lambda x: math.floor(x) if abs(x - round(x)) > EPSILON else round(x)
merge-two-dictionaries snippet
x = {'a': 42}
y = {'b': 127}
z = dict(x.items() + y.items())
# z = {'a': 42, 'b': 127}

25. anonymous-object snippet

Adapted from stackoverflow:

class Anon(object):
    def __new__(cls, **attrs):
        result = object.__new__(cls)
        result.__dict__ = attrs
        return result

26. Alternative:

class Anon(object):
    def __init__(self, **kwargs):
        self.__dict__.update(kwargs)
    def __repr__(self):
        return self.__str__()
    def __str__(self):
        return ", ".join(["%s=%s" % (key,value) for key,value in self.__dict__.items()])

27. generate-random-word snippet

Function that returns a random word (could also use random.choicewith this list of words):

import string, random
randword = lambda n: "".join([random.choice(string.letters) for i in range(n)])
setdefault tricks
Increment (and initialize) value:
d = {}
d[2] = d.setdefault(2,39) + 1
d[2] = d.setdefault(2,39) + 1
d[2] = d.setdefault(2,39) + 1
d[2] # value is 42

29. Append value to (possibly uninitialized) list stored under a key in dictionary:

d = {}
d.setdefault(2, []).append(42)
d.setdefault(2, []).append(127)
d[2] # value is [42, 127]

Binary tricks

30. add-integers-using-XOR snippet

Swap two integer variables using the XOR swap algorithm:

x = 42
y = 127
x = x ^ y
y = y ^ x
x = x ^ y
x # value is 127
y # value is 42

This is a true story that happened in Japan.

In order to renovate the house, a Japanese man breaks open the wall. Japanese houses normally have a hollow space between the wooden walls.

When tearing down the walls, he found that there was a lizard stuck there, because a nail from outside had been hammered into one of its feet.

He sees this, feels pity, & at the same time curiousity. When he checks the nail, it was nailed 5 years ago, when the house was built!!!

*What happened?*

The lizard had survived in such a position for 5 yrs!!!

In a dark wall partition for 5 years without moving, it is impossible and mind-boggling.

Then he wondered how this lizard had survived for 5 years without moving a single inch, since its foot was impaled!

So he stopped his work and observed the lizard, to see what it has been doing, and how it has been eating

Later, not knowing from where it came, appears another lizard, with food in it's mouth.

Ah! He was stunned and touched deeply.

For the lizard that was impaled by the nail, another lizard had been feeding it for the past 5 years.

Imagine? It has been doing that untiringly for 5 long years, without giving up hope on its partner.

Imagine what a small creature can do, that a creature blessed with a brilliant mind cannot do.

• Please , never abandon your loved ones. Don't turn your back on people because they are in difficulty. Don't see yourself as more intelligent or the best. You may be a bit luckier today but life is uncertain and can change things tomorrow....

Before you say something just remember...

• It takes a moment to break, but an entire lifetime to make .........
• God created space within our fingers so that we can hold hands......
• Be there for somebody today.

Love one another!!!

  
TAKE A SHOWER

A long and cold if possible.

It doesn't matter if you have already taken one in the morning.

The next time you are

• Feeling lost - Take a shower!
• Feeling guilty - Take a shower!
• Feeling unproductive - Take a shower!
• Feeling stressed - Take a shower!
• Feeling bored - Take a shower!
• Feeling depressed - Take a shower!
• Feeling lazy/ sleepy - Take a shower!
• Feeling sweaty - Take a shower!😳

 🙂

LESSON FOR LIFE

1. If you are the smartest in the room, you are in the wrong room.
2. Be nice to people on the way up or it is real shitty on the way down
3. Don’t tell anybody about the people you find annoying.
4. Don’t criticize work or anyone in the hallways.
5. Everything communicates. How you dress, how you stand, how you speak, etc. matters.
6. Do not confuse your personal identity with your employment. So, have a life outside corporate life.

न में अगले साल चुनाव होने वाले हैं, जिसकी तैयारी जोर-शोर से चल रही है। आतंकी संगठन ‘जमात-उद-दावा’ के सरगना हाफ़िज़ सईद ने भी इस बार चुनाव लड़ने का एलान किया है। इसी सिलसिले में उन्होंने कल अपनी पार्टी का घोषणा-पत्र जारी कर दिया। इस घोषणा-पत्र में वादा किया गया है कि “अगर उनकी सरकार आती है, पाकिस्तान में कोई भी बेरोजगार नहीं रहेगा, और हर हाथ को काम (AK-47) दिया जाएगा। ये योजना देश में हमेशा चालू रहेगी, और जो भी इसके तहत काम मांगेगा उसे तुरंत AK-47 पकड़ा के काम पर लगा दिया जाएगा