Petro Verkhogliad @ Experiment Room 4

Decision Regions for Two-Dimensional Gaussian Data in Python

written by Petro, on Dec 1, 2009 4:29:00 PM.

There is an example on page 44 of “Pattern Classification” by Duda, Hart and Stork that shows how to calculate the decision boundary for the two-category two-dimensional data. Extending this to the multi-dimensional case is not difficult.

A couple of weeks ago I wrote the code for this, but till now I’ve been putting off posting it. Well, here is the code and some discussion.

The first thing we want to do is create some random vectors with given means and covariance matrices. There are easier ways of doing this. For example, the gauss function returns Gaussian random values. This could be done with the standard libraries.

While we’re at it we might as well define the means and the covariances (and do the imports).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
# std library imports
import random

from operator import attrgetter
from collections import defaultdict

# matplotlib imports
import matplotlib.pyplot as plt
from matplotlib.font_manager import fontManager, FontProperties
from mpl_toolkits.mplot3d import Axes3D
from matplotlib import rc
font= FontProperties(size='x-small');

# numpy/scipy imports
import numpy as np
from scipy import *
from scipy.linalg import *

DIMENIONS = 2


mu1 = np.array([3,6])
mu2 = np.array([3, -2])
covar1 = np.array([[0.5, 0], 
                   [0,   2]])
covar2 = np.array([[2, 0],
                   [0, 2]])

def random_vectors(mu1, m2, covar1, covar2, num=200, dim=DIMENSIONS):
    """
    Generate a given number of randoms of the specified dimension 
    given the means and the covariance matrices for the two classes.

    @param mu1: Mean for class omega1
    @type mu1: np.array
    @param mu2: Mean for class omega2
    @type mu2: np.array
    @param covar1: Covariance matrix for class omega1
    @type covar1: np.array
    @param covar2: Covariance matrix for class omega2
    @type covar2: np.array
    @rtype: tuple
    @return: A 2-tuple. The list of vectors for the first mean and covariance is
             at the first index. The list of vectors for the second mean 
             and covariance is at the second index.
    """
    def gauss():
        return (sum([random.random() for i in xrange(12)]) - 6)

    # store the vectors here
    result1 = list()
    result2 = list()

    # get the eigenvalues and eigenvectors for each of the covariance matrices
    evals1, evec1 = eig(covar1)
    evals2, evec2 = eig(covar2)

    evals1 = np.diag([np.power(val.real,0.5) for val in evals1])
    evals2 = np.diag([np.power(val.real,0.5) for val in evals2])

    for i in xrange(num):
        vec1 = np.array([gauss() for i in range(dim)])
        vec2 = np.array([gauss() for i in range(dim)])

        vec1 = mu1 + dot(dot(evec1, evals1), vec1)
        vec2 = mu2 + dot(dot(evec2, evals2), vec2)

        result1.append(vec1)
        result2.append(vec2)
    return result1, result2

In the next step we want get our lists of our random vectors

1
o1, o2 = random_vectors(mu1, mu2, covar1, covar2)

Now we can calculate the boundary parameters.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
def boundary_parameters(mu1, mu2, covar1, covar2):
    """
    Calculate A, B and C parameters for the discriminant boundary function.
    @param mu1: The mean of the first distribution
    @type mu1:  numpy.array
    @param mu2: The mean of the second distribution
    @type mu2: numpy.array
    @param covar1: The covariance matrix for the first distribution
    @param covar2: The covariance matrix for the second distribution
    @return: Tuple of parameters where the first parameter is the matrix A, the 
             second parameter is the matrix B, and the third parameter is a
             scalar value C.
    @rtype: tuple
    
    """

    a1 = 0.5*(inv(covar2) - inv(covar1))
    b1 = (dot(mu1.T,inv(covar1)) - dot(mu2.T,inv(covar2)))
    c1 = 0.5*(dot(dot(mu2.T,inv(covar2)), mu2) - dot(dot(mu1.T, inv(covar1)), mu1)) + np.log(p1/p2) + np.log(det(covar2) / det(covar1))
    return a1,b1,c1

a,b,c = boundary_parameters(mu1, mu2, covar1, covar2)

Now for each of the vectors we can calculate the boundary value in a given range.

1
2
3
4
5
6
x = list()
y = list()
for v in xrange(-5,12, 1):
    val = boundary(v, a, b, c)
    x.append(v)
    y.append(val)

If we were work the solution by hand we would arrive at the following boundary function. So, let’s write it as a lambda.

1
f = lambda x: 3.514 - 1.125*x + 0.187*(x**2)

Finally, we can plot the whole thing.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
fig = plt.figure(figsize=(10,7))
ax = fig.add_subplot(111)
ax.scatter(*zip(*o1), color='r')
ax.scatter(*zip(*o2), color='g')
# the line that we plotted by solving the quadratics 
ax.plot(x,y)
# the true line
ax.plot(np.arange(-5,12),[f(x) for x in np.arange(-5,12)], color='c')
ax.grid(True)
plt.show()

The output looks like this:

The complete source code here: qda.py

Animated bar graphs with Matplotlib and Qt4

written by Petro, on Dec 1, 2009 3:29:00 PM.

The new Matplotlib book shows how to animate line drawing. What I really needed was bar graphs. I didn’t see an example of animating bar graphs in the book (probably because it is follows naturally from the line animation example). The bar graph documentation for matplotib shows how to create a set of bar graphs. The code for animated bars is below.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
import sys
from PyQt4 import QtGui

from matplotlib.figure import Figure
from matplotlib.backends.backend_qt4agg import FigureCanvasQTAgg as FigureCanvas

import numpy as np


class Monitor(FigureCanvas):
    def __init__(self):
        self.fig = Figure()
        self.ax = self.fig.add_subplot(111)

        FigureCanvas.__init__(self, self.fig)

        # the file is 5 float values separated by spaces on every line, like this:
        #0.28 0.18 0.18 0.18 0.18
        #0.352 0.162 0.162 0.162 0.162
        #0.4168 0.1458 0.1458 0.1458 0.1458
        #0.37512 0.23122 0.13122 0.13122 0.13122
        self.data = [np.array(map(float, line.strip().split())) for line in open('data.txt').readlines()]
        self.counter = 1
        
        # the width of the bars
        self.width = 0.8

        # the locations of the bars
        self.locs = np.arange(len(self.data[0]))

        # the first set of bars
        self.bars = self.ax.bar(self.locs, self.data[0], self.width, color='#6a7ea6')

        # set up the lables for the bars
        self.ax.set_xticks(self.locs+0.5)
        self.ax.set_xticklabels(['Red', 'Green', 'Black', 'Orange', 'Yellow'])

        # set the limit for the x and y
        self.ax.set_xlim(0., len(self.data[0]))
        self.ax.set_ylim(0., 1.,)

        # draw the canvas
        self.fig.canvas.draw()

        # start the timer
        self.timer = self.startTimer(1000)


    def timerEvent(self, evt):
        # update the height of the bars, one liner is easier
        [bar.set_height(self.data[self.counter][i]) for i,bar in enumerate(self.bars)]

        # force the redraw of the canvas
        self.fig.canvas.draw()

        # update the data row counter
        self.counter += 1

if __name__ == "__main__":
    app = QtGui.QApplication(sys.argv)
    w = Monitor()
    w.setWindowTitle("Convergence")
    w.show()
    sys.exit(app.exec_())

Matplotlib animation with PyQt4

written by Petro, on Nov 29, 2009 8:49:00 PM.

The code is fairly self-explanatory. We create a single line in the figure and then update it on every timer tick.

Edit: The code below is inspired by/borrowed from the new Matplotlib book by Sandro Tosi.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
import sys
import random
from PyQt4 import QtGui

from matplotlib.figure import Figure
from matplotlib.backends.backend_qt4agg \
        import FigureCanvasQTAgg as FigureCanvas

class Monitor(FigureCanvas):
    def __init__(self):
        self.fig = Figure()
        self.ax = self.fig.add_subplot(111)

        # initialize the figure canvas
        FigureCanvas.__init__(self, self.fig)

        # set up the display limits for the figure
        self.ax.set_xlim(0,30)
        self.ax.set_ylim(0,100)

        # turn off autoscaling
        self.ax.set_autoscale_on(False)

        # this is the value that will be updated
        self.value = []
        # this is the line that will be animated
        self.line_value, = self.ax.plot([], self.value, label="Values")

        # shwo the legend
        self.ax.legend()
        self.fig.canvas.draw()

        # start the Qt timer
        self.timerEvent(None)
        self.timer = self.startTimer(1000)


    def timerEvent(self, evt):
        """
        This code will be executed on every timer tick.
        """
        # get the new data value
        self.value.append(random.randrange(0,30))
        # update the line with the new values
        self.line_value.set_data(range(len(self.value)), self.value)

        # force the redraw of the canvas
        self.fig.canvas.draw()

        
if __name__ == "__main__":
    # set up the qt application
    app = QtGui.QApplication(sys.argv)
    # initialize the widget
    w = Monitor()
    # set the title of the window
    w.setWindowTitle('Updating in real time')
    # show the widged on the screen
    w.show()
    # start the main application loop
    sys.exit(app.exec_())

Deleting words in bash and IPython

written by Petro, on Oct 14, 2009 2:05:21 PM.

Finally figured out how to make bash delete words the same way Vim does. For starters I enable vi editing mode in readline. So, when i press Ctrl-W I expect bash to delete up to the previous word delimiter. By default bash/readline deletes up to the previous space. So, to enable the use of punctuation as a word delimiter while using vi editing mode one has to use the following in the ~/.inputrc

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
set editing-mode vi
set keymap vi

# turn off the stupid bell
set bell-style none

$if mode=vi
    set keymap vi-command
    "gg": beginning-of-history
    "G": end-of-history

    set keymap vi-insert
    "\C-w": backward-kill-word
    "\C-p": history-search-backward
$endif
Since IPython respects readline, this will take effect in the IPython shell. :)

Epydoc style docstrings with snippetsEmu

written by Petro, on Oct 5, 2009 12:08:00 PM.

snippetsEmu is a great Vim plugin that attempts to emulate some of the behaviour of snippets from Textmate. It has support for multiple languages. I primarily use the Python, Latex, Javascript and HTML snippets.

At the same time, when I write Python code I use Epydoc for documentation. By default snippetsEmu outputs the following docstring for a function definition:

1
2
3
4
def hello(varname):
    """
    varname -- <{}>
    """

This is pretty good, but what I want is something like this:

1
2
3
4
5
def hello(varname):
    """
    @param varname: <{}>
    @type varname: <{}>
    """

This can be done by changing a single line in the ~/.vim/plugin/after/ftplugin/python_snippets.vim file. There is a function there called PyGetDocstringFromArgs. The line we want to change is this:

1
let docvars = map(split(text), 'v:val." -- ".st.et')

We want to change it to:

1
let docvars = map(split(text), '"\@param ".v:val."\: ".st.et."\n".indent."@type ".v:val."\: ".st.et')
Now whenever I do def and type the function parameters they are generated in Epydoc style.

Working with Vim and IPython

written by Petro, on Oct 4, 2009 9:03:00 PM.

Note: the text below is not for the fainthearted. Setting up Vim to communicate with IPython is currently not very straightforward. The situation will improve soon.

When I write Python code I tend to use IPython quite a bit to test and experiment. Currently this means quite a bit of reloading and writing test code in the IPython shell. I’ve always been a little jealous of the Emacs folks for being able to communicate with the Python shell with ease. Vim has a different philosophy. We use the editor to edit stuff and the shell to run it. With that in mind, one can set up Vim to talk to IPython.

There are several steps for this:

  1. Ensure that Vim is compiled with +python
  2. Get the code that enables communication with IPython
  3. Write code to send code from Vim to the server
Note: this assumes that you have Python, IPython and Twisted installed.

1. Ensure that vim is compiled with +python

Open your favorite version of Vim and do
1
:version

In there look for “+python”. If it is not there, you need to somehow get a version that has “+python”. It is also important to check which version of Python your Vim was compiled against. For this, execute the following command in vim:

1
:python import sys; print sys.version
The output should be the same as your primary version of Python. For example, I am on OS X 10.5.8. By default it ships with Apple Python 2.5. I prefer to use Python 2.6 from MacPorts. Because of this I had to build a version of Macvim from source that pointed to my installation of Python 2.6. This deserves its own post.

2. Getting the code for communicating with IPython

Matt Foster has written a plugin for Textmate called ipythontm-bundle. This Textmate bundle uses the ipy_textmateserver.py to create a Twisted server to which stuff is sent for evaluation from Vim.

This file is can be downloaded from the textmate-server branch of IPython. For Vim we only need this file. Why only this file will be explained at the end of this post. Once the file is downloaded put it in your IPython/Extensions directory. For me this directory is:

1
/opt/local/Library/Frameworks/Python.framework/Versions/Current/lib/python2.6/site-packages/IPython/Extensions

3. The Vim side of life

At the moment, Vim has no idea that there will be a server running. Therefore, we have to write a plugin. The following (very very simple) code server as said plugin.
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
import vim
import socket

def runner(content):
    s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
    try:
        # make sure to chage the directory here
        s.connect("/YOUR/HOME/DIR/.ipython/IPYS")
        s.send(content)
        s.close()
    except:
        print "Error: could not connect to iPython server."

# --- Functions for running code in iPython --------------------------------- #

def run_current_line():
    runner(vim.current.line)

def run_selection():
    runner("\n".join(vim.current.range))

def run_buffer():
    runner("\n".join(vim.current.buffer))

# --- Clean up function

def remove_sockets():
    import os
    import sys
    os.remove("/Users/petrov/.ipython/IPYS")

Essentially, the server will create a UNIX socket. By default the socket is:

1
$HOME/.ipython/IPYS

It is possible to rename the socket or use different ones for different Vim windows. For now, I wanted to get the basics going. The Python code above should go into

1
$HOME/.vim/plugin

4. Using the code

Now that the code is in place we can start using it. Start IPython and execute the following code:
1
2
3
4
5
6
import ipy_textmateserver
# create the server
s = ipy_textmateserver.TextMateServer()
# start the server
s.start()
# to stop the server do: s.stop()
Now the server is running in that instance of the shell. Open a Python file in Vim. Create a map for executing the current line in Vim:
1
:map ,el :python run_current_line()<cr>

Put the cursor on the line of code you want to eval and do: ,el. This should send the line to IPython and you should see the results of execution in the shell.
The other two functions in on the plugin side can also be mapped to whatever you want.

Question and Answer

This is very complicated!
Yes, this is not the easiest thing in the world. But it let’s me work the way I want to work. In the future this will be packaged better as a proper Vim plugin.

Why not use the client that comes with the ipythontm-bundle.
I did try to use that code. However, it seems that Vim and Textmate do things differently. The Vim client that comes with ipythontm-bundle uses a Twisted reactor to send code to the server. This is great. However, once the reactor is stopped it can’t be restarted. Any attempts to do so result in a blocked thread. When this happens from inside Vim, it kills the whole session. This sucks. My guess is that Textmate reloads the whole client file. Vim doesn’t seem to do this. Therefore, I opted to use simple Python sockets.

Why not use ipy_vimserver.py?
I tried this as well. However, it uses socket code that doesn’t work on my machine. If you’re on Linux you should be able to use ipy_vimserver.py without any problems.

PyQt4 splashscreen

written by Petro, on Jul 19, 2009 6:57:00 PM.

Note to self: the code below will show a splash screen before the main window loads

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
import sys
from time import sleep

from PyQt4.QtCore import *
from PyQt4.QtGui import *


LOADING_MESSAGE = "Loading stuff"
# set this to point to .png file somewhere on your disk. For example
# SPLASH_IMAGE = "/Users/username/Desktop/image.png"
SPLASH_IMAGE_PATH = ""

class Box(QDialog):
    def __init__(self, parent=None):
        super(Box, self).__init__(parent)


if __name__ == "__main__":
    qapp = QApplication(sys.argv)

    splash = QSplashScreen(QPixmap(SPLASH_IMAGE_PATH))
    splash.showMessage(LOADING_MESSAGE)
    splash.show()

    qapp.processEvents()

    for i in xrange(10):
        sleep(1)
        print i

    box = Box()
    box.show()
    splash.finish(box)
    sys.exit(qapp.exec_())

That is all.

Recently read/watched.

written by Petro, on Jun 29, 2009 2:02:00 PM.

In an interesting coincidence I finished reading A Madman Dreams of Turing Machines the other day. Interesting because Turing’s birthday was just a few days ago. Doubly interesting because a great follow up to the book turned up in the Google Tech Talks YouTube channel, The Church-Turing Thesis: Story and Recent Progress.

The book is good, even if a little light on the specifics of Godel’s or Turing’s work. The video is excellent in both presentation and content.

One of the reasons I like Python

written by Petro, on Jun 25, 2009 12:01:00 PM.

Here is one of the reasons why I like Python:
1
2
res = dict([(item_list.count(r), r) for r in list(set(item_list))])
print {max(res.keys()): res[max(res.keys())]}
“item_list” is a list of strings that may contain duplicates. The first line removes the duplicates and builds a dictionary of items in the list keyed on the number of times the items appears in the list. The second line prints the item with the highest count.

Nice colorscheme for Vim

written by Petro, on May 4, 2009 1:50:00 PM.

By accident I came across a really nice colorscheme for Vim called Mustang. The link takes you to a page with a screenshot, alternatively you can download here.