Decision Regions for Two-Dimensional Gaussian Data in Python
There is an example on page 44 of “Pattern Classification” by Duda, Hart and Stork that shows how to calculate the decision boundary for the two-category two-dimensional data. Extending this to the multi-dimensional case is not difficult.
A couple of weeks ago I wrote the code for this, but till now I’ve been putting off posting it. Well, here is the code and some discussion.
The first thing we want to do is create some random vectors with given means and covariance matrices.
There are easier ways of doing this. For example, the gauss function returns Gaussian random values.
This could be done with the standard libraries.
While we’re at it we might as well define the means and the covariances (and do the imports).
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 | # std library imports import random from operator import attrgetter from collections import defaultdict # matplotlib imports import matplotlib.pyplot as plt from matplotlib.font_manager import fontManager, FontProperties from mpl_toolkits.mplot3d import Axes3D from matplotlib import rc font= FontProperties(size='x-small'); # numpy/scipy imports import numpy as np from scipy import * from scipy.linalg import * DIMENIONS = 2 mu1 = np.array([3,6]) mu2 = np.array([3, -2]) covar1 = np.array([[0.5, 0], [0, 2]]) covar2 = np.array([[2, 0], [0, 2]]) def random_vectors(mu1, m2, covar1, covar2, num=200, dim=DIMENSIONS): """ Generate a given number of randoms of the specified dimension given the means and the covariance matrices for the two classes. @param mu1: Mean for class omega1 @type mu1: np.array @param mu2: Mean for class omega2 @type mu2: np.array @param covar1: Covariance matrix for class omega1 @type covar1: np.array @param covar2: Covariance matrix for class omega2 @type covar2: np.array @rtype: tuple @return: A 2-tuple. The list of vectors for the first mean and covariance is at the first index. The list of vectors for the second mean and covariance is at the second index. """ def gauss(): return (sum([random.random() for i in xrange(12)]) - 6) # store the vectors here result1 = list() result2 = list() # get the eigenvalues and eigenvectors for each of the covariance matrices evals1, evec1 = eig(covar1) evals2, evec2 = eig(covar2) evals1 = np.diag([np.power(val.real,0.5) for val in evals1]) evals2 = np.diag([np.power(val.real,0.5) for val in evals2]) for i in xrange(num): vec1 = np.array([gauss() for i in range(dim)]) vec2 = np.array([gauss() for i in range(dim)]) vec1 = mu1 + dot(dot(evec1, evals1), vec1) vec2 = mu2 + dot(dot(evec2, evals2), vec2) result1.append(vec1) result2.append(vec2) return result1, result2 |
In the next step we want get our lists of our random vectors
1 | o1, o2 = random_vectors(mu1, mu2, covar1, covar2) |
Now we can calculate the boundary parameters.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 | def boundary_parameters(mu1, mu2, covar1, covar2): """ Calculate A, B and C parameters for the discriminant boundary function. @param mu1: The mean of the first distribution @type mu1: numpy.array @param mu2: The mean of the second distribution @type mu2: numpy.array @param covar1: The covariance matrix for the first distribution @param covar2: The covariance matrix for the second distribution @return: Tuple of parameters where the first parameter is the matrix A, the second parameter is the matrix B, and the third parameter is a scalar value C. @rtype: tuple """ a1 = 0.5*(inv(covar2) - inv(covar1)) b1 = (dot(mu1.T,inv(covar1)) - dot(mu2.T,inv(covar2))) c1 = 0.5*(dot(dot(mu2.T,inv(covar2)), mu2) - dot(dot(mu1.T, inv(covar1)), mu1)) + np.log(p1/p2) + np.log(det(covar2) / det(covar1)) return a1,b1,c1 a,b,c = boundary_parameters(mu1, mu2, covar1, covar2) |
Now for each of the vectors we can calculate the boundary value in a given range.
1 2 3 4 5 6 | x = list() y = list() for v in xrange(-5,12, 1): val = boundary(v, a, b, c) x.append(v) y.append(val) |
If we were work the solution by hand we would arrive at the following boundary function. So, let’s write it as a lambda.
1 | f = lambda x: 3.514 - 1.125*x + 0.187*(x**2) |
Finally, we can plot the whole thing.
1 2 3 4 5 6 7 8 9 10 | fig = plt.figure(figsize=(10,7)) ax = fig.add_subplot(111) ax.scatter(*zip(*o1), color='r') ax.scatter(*zip(*o2), color='g') # the line that we plotted by solving the quadratics ax.plot(x,y) # the true line ax.plot(np.arange(-5,12),[f(x) for x in np.arange(-5,12)], color='c') ax.grid(True) plt.show() |
The output looks like this:
The complete source code here: qda.py