Context

Earlier this year I performed a review of OpenCourseware courses from three different institutions: MIT, Carnegie Mellon, and The Open University. I evaluated three courses from each of the three institutions.

Process

As I reviewed the courses, I kept a log of features that I recognized as valuable. At the end of my review I codified the results by creating a master list of features, and then organizing the features into five categories:

1. Site Structure and Features
2. Coruse Structure and Features
3. Types of Content
4. Efficacy of Content
5. Community of Learners

After creating the list, I went back through all nine courses and marked whether or not each feature existed in each course. I found that a binary answer wasn’t always satisfactory. In some cases, the feature existed, but wasn’t complete or fully developed. In this resulting chart, green checks represent “fully present”; black squares represent “partially developed.”

My next step was to assign a relative value for each feature. It was obvious during my review that different institutions valued different features. To serve my purposes, I needed to determine the value that my organization (Center for Distance Education) places on each item in the list. I collaborated with a colleague, Chris Lott, to place a value of 1, 2, or 3 on each item in the list (three being high).

After designating a value for each item on the master list, I entered everything into a spreadsheet and created a series of formulas to calculate relative weights for each of the five categories. As one product of my research, I wanted a way to visually compare the strengths of institutions as well as individual courses, using CDE’s perceived values as a metric. My first thought was to create a treemap (view the treemap attempt), but the data wasn’t as visually intuitive as I’d hoped in this format. After a bit of trial and error, I settled on Bubble Charts as a more effective type of display for this particular data. In the first chart below you’ll see a comparison between the three institutions. In the second chart, a more detailed view comparing all nine courses against the CDE ideal.

Note that the size of the bubble indicates the total number of features employed, while the colors represent the distribution within categories. It’s readily evident that Carnegie Mellon (OLI) and The Open University (OpenLearn) implement more of the features from the list than MIT. It’s also clear that The Open University has placed more emphasis on learning community features than the other two institutions.

Meta

Data Visualizations were created using IBM’s Many Eyes site. I stored some of my pdf documents on Issuu. (This site creates an interesting way to view and expand pdf documents.) For convenience in linking from this blog, the pie chart and bubble charts were uploaded to Flickr.