One of the standard questions asked when analyzing data collected in an experiment is, "is there a difference between conditions?".  Perhaps you might want to compare two slightly different screen layouts and you want to know where the differences arise in the data?  For example, you might want to see what sort of effect moving information from one part of the screen to another has on the location of eye-gaze measured for multiple study participants.  One very useful way to evaluate the effect of such a modification is to employ a comparison measure called the Cohen's-d.  It provides a metric to describe the 'effect size' of the manipulation made to the stimuli in your experiment.  It is a measure of how two distributions of data overlap.  A link that illustrates how the Cohen's-d changes with respect to the overlap of a distribution of data, click here.  

We're applying this measure to evaluate how eye-gaze heat maps differ between experimental conditions.  For example, we want to know if the locations at which people look differ when the relevant information for a behavioural task are moved around.  To see how looking differs between conditions we apply a Cohens-d comparison of our two conditions of interest to each pixel defining our heat map.  This efficiently coveys information about the overlap of distributions of gaze across study participants.  It is far more informative than simply subtracting the difference between the average heat map of our two experimental conditions.

Something to consider, when applying Cohen's-D analysis to evaluate differences in looking, is that there must be enough data at each 'pixel of looking'.  When there isn't enough data, the 'noise' in the data begins to have an impact on your Cohen's-d result and this can be misleading.  Hence, we've dropped our analysis resolution from 800 x 600 pixels to 60 x 80.  This provides us with enough pixels for analysis to differentiate our two experimental conditions while also binning enough samples so that we can make a reliable Cohen's-d evaluation of the data.

One of the major challenges when using 'prototype code' to analyze EEG data from different projects is avoiding breaking the code when making small changes to suit new data.  Our EEG data mining methods is based on a learning algorithm originally written to analyze small data sets (less than 1 gigabyte).  Our new learning algorithm is suitable to mine EEG data sets that are greater than 6 gigabytes in size.  We have to be sure that all parts of this large dataset are appropriately examined by our algorithm and that we don't miss any fine, low variance detail that can help improve our data mining result.  Hence learning can not take place using large jumps in the learning rate. However, we do want the algorithm to come to an answer as quickly as possible.  I've applied a few tricks to get best learning without wasting too many iterations on diverging solutions.

Another challenge is adapting our existing code to be able to handle missing data.  In all the EEG analysis I've every attempted, there has only been 1 case when I didn't have to deal with missing data because I had the option of throwing away datasets with missing parts.  Every other time, we have had to retain as much data as possible and it has taken huge amounts of time to tweak our code to compensate for the missing data.  Unfortunately, our current experiment requires that we retain as much data as possible, and of course, there are some incomplete datasets to contend with.  What I usually do in cases such as these is to go through all of the data mining, validation, and volume estimation steps using the complete datasets (to make sure the code works correctly and provides reasonable results).  Once I've established correct operation of the code, I then make the necessary changes to the code so that I can include the incomplete datasets in the analysis.  This usually has the effect of 'improving' the data mining results (because I have more data).

We have a sophisticated validation algorithm that identifies which of the data mining results are likely artefacts caused by cable movement, electrode movement, muscle activity, or are poorly separated brain activities.  The existing validation algorithm provides a reasonably objective means to decide which data mining results actually tell us something about the activities of particular parts of the brain and which we can ignore in subsequent analysis steps.  Since I'm a creative person, I though of some ways to improve the existing algorithm.  Hence, in the current experiment, I've decided to use an additional dimension by which to separate our good brain activity estimates from artefacts to further improve the validation algorithm.

October 14, 2009 - Improving our Existing Validation Algorithm