Qualitative Data Analysis: Card Sorting

So you have some qualitative data -- maybe from interviews, maybe from an observation session -- and you want to do some data analysis. You know what you are looking for (i.e. causes of miscommunication or misunderstanding, resolution strategies), but have little to no hypothetical support for any themes amongst your data. One qualitative data analysis technique you can use is card sorting. 

This blog post is written in collaboration with my colleague Justin Smith; it is based on our experiences doing research in our research group (Developer Liberation Front) and my time at Microsoft Research this past summer working with Tom Zimmermann in the Empirical Software Engineering group and what we have found to be most efficient. For those who are interested in how to do a card sort, or how others do card sorting, I'm going to talk about how to do a card sort based on our experiences.

First thing's first: before you can do a card sort, even before you look at your data, you should have an idea of what it is you're looking for in your data. For example, with the study I am working on, I'm interested in what makes tool output difficult to interpret; more specifically I want to identify the areas where there is either miscommunication, risk of miscommunication, or misunderstanding and see what causes each.

• Recommendation #1: Come up with some criteria that can be used when extracting quotes. Your criteria should be based on whatever it is you're searching for in the transcript. For example, with Justin's study, we wanted to find implicit and explicit questions developers need answered when resolving security vulnerabilities; therefore, one trivial criteria was that the text should be an explicit question posed by the developer. The more defined and specific what you're looking for is, the easier it will be to extract data for your card sort.
• Recommendation #2: Have at least two people extract quotes (including yourself obviously); you should all be using the criteria you put together. This will help validate your criteria as well as increase the validity of the quotes you extract. If you're using two people, both should extract quotes from all transcripts; once that's done, the two of you should sit down and determine where you agree and disagree. Two is usually enough, but if you decide to use three people, we recommend having 2 people work on different sets of data (i.e. person 1 and 2 on a set, person 1 and 3 on a set...); once finished, each couple will work out their disagreements. This is something you will want to report when you attempt to publish your findings :) (it's called inter-rater reliability).

Once you have your set of quotes from your transcript, you need to put them on notecards. Advice from someone who's been there: although having physical, paper notecards is a necessity for a card sort, we highly recommend having an electronic copy of your quotes (I put mine in Excel).

For our first card sorts, we manually made physical note cards -- they look nice and the card sort itself was okay, but keeping track of themes and sub-themes during the card sort, and after, was not trivial and sometimes led to confusion and the need for back tracking.

A pointer regarding using spreadsheets to store your data:

• Anything you would include on your notecard should be a column in the spreadsheet; for example, the columns in my spreadsheet are Participant (P1, P2...), Tool Being Used, and Quote. I also have a column for a unique identifier for each quote (Card Number) and the Emergent Themes from each round of the card sort. It might also be beneficial to include a Timestamp column; this way, you can have an approximate location in your media to find the quote if needed.

There are a number of advantages to having your quotes in a spreadsheet, specifically Excel:

1. You don't have to worry (as much) about water or a random fire ruining your data. It's also harder to lose your data if it's electronically stored.
2. Spreadsheets are searchable; paper notecards are not.
3. Typically card sorts are often done in iterations, and you want to be able to report anything that happens (i.e. cards moving from one theme to another); this is MUCH easier if you can put the themes for each round of the card sort into a spreadsheet. 
4. Organizing paper notecards can be tedious and error-prone (i.e. trying to find a card and messing up an entire pile); having an electronic copy you can easily organize your data. If you decide to organize your notecards, you at least know you have a proxy of how the data was in your electronic copy.
5. Actually making notecards is a time consuming process, especially when making them by hand. Instead, if you have your quotes in an Excel spreadsheet, and label the columns as mentioned earlier, you can use Mail Merge to create your notecards electronically :). Details regarding how to do this can be found in the following supplementary blog post.

Now, it's time to complete the card sort. This can be done in one phase, however, we recommend doing it in multiple phases. Minimally, there's a phase 1 for preliminary sorting into themes and phase 2 for sorting each theme into high-level themes. This is particularly useful for large datasets where you can wind up with a large number of themes after the first phase; typically there are common themes amongst those separate themes, thereby warranting another phase of sorting. You may also want to include a validation phase once you have determined all the low level emergent themes (after phase 1). This phase is to ensure that all quotes have been sorted into the best possible theme.

• Recommendation #1: Include others in the card sort process; this lessens the bias behind the themes you find (and from our experience helps come up with distinct themes with clear definitions, which is super important when working with qualitative data). One thing to be aware of is the more cooks in the kitchen, the more time it might take to complete (more potential for disagreement and need for discussion), so plan accordingly. For our most recent card sort, 2 hours was our max at one time so, with a little over 300 notecards, we did four 2-hour sessions. Previously I've had shorter sessions; the longer time for this study, we believe, is a product of the type of data we're working with (non-interview).
• Recommendation #2: As you're doing you're card sort, keep track of important information as it changes (i.e. how you define your themes/sub-themes). Also, keep track of quotes that you and your sorters believe best represent each theme. Doing these things will make reporting your findings much easier. 

Once you've done all this, you're ready to start thinking about what your paper is going to look like and where the interesting stories are in your themes (the fun part). With that being said, good luck fellow qualitative researchers! :)

Thanks again to Justin for helping me put this together and the Developer Liberation Front and Tom for the experiences!

Creating Notecards using Microsoft Word Mail Merge

This blog post provides step-by-step instructions for creating notecards (possibly for a card sort) using Microsoft Excel and Word. Thanks to Tom Zimmermann of Microsoft Research who taught me this nifty trick :D.

This tutorial assumes you already have your data in a spreadsheet, as discussed in another blog post.

1. Save your excel spreadsheet -- make sure your data columns are labeled.
2. Open or create a Word template with the number of notecards you would like on one page. I created my own using the Create Table command in Word; either way, make sure you know what size each notecard is (you'll need it later).
3. Once you have the template/document open, you will need to put in the various components of your notecard where you'd like them to be. As an example, my template I created based on my spreadsheet columns can be found here.
4. Next, select the Mailings tab > Start Mail Merge > Email Messages. It technically doesn't matter what you select here; I choose Email Messages because it makes life easier :).
   
5. Next, select Select Recipients >  Use An Existing List...
   Browse to and select the spreadsheet with your data. If your quotes are in a workbook, make sure you select the correct sheet.
   
   
6. Now you want to map the different "fields" in your document to the columns in your spreadsheet. This is done using the Insert Merge Field menu.
   
   The list that comes is populated with the columns in your spreadsheet -- insert each field into one notecard (i.e. replace <<participant>> with Participant merge field) then copy and paste all fields into each notecard. Once finished, it will look something like this:
   

   
   
   
7. If you click Preview Results you can see what your notecards will look like. If you click it now, each notecard will have the same information on it. This is because you have to tell Mail Merge to go to the next record in the spreadsheet.
   
   To tell Word you want to go to the next record for each notecard, you need to add a rule; this is done by going to the Rules menu and selecting Next Record. You need the <<Next Record>> field on each notecard; it should look something like this:
   

   
   
   

   Now when you click Preview Results you should see different data for each notecard.

   
   
8. Now you're ready to Finish & Merge!

   

   I typically select Edit Individual Documents... so I can make sure everything is copesthetic.
9. The final step is optional, but recommended: because quotes vary in length, depending on the template you used for your notecards you may want to remove any extra space added for the short quotes due to the long quotes. It's a tedious process but helps save paper :).

Once you have your notecards ready, you can print, cut, then you're ready to get to sortin'! :D

IBM University Day 2015 - Women in Education and Research

I attended IBM University Day for the first time this past Friday. I didn't know what to expect. I had never been to this event, which apparently had been going on for some time now. Also, I've always heard (and sometimes felt) a stigma behind IBM that made me weary of an event they would have titled "Women in Education and Research". Aside from the fact that I've always associated IBM with conservative, older white men, as an African American woman I'm always looking for someone like me doing extraordinary thing at events like this and am most often disappointed. I just knew this event would be no different...except I was surprisingly and so thankfully wrong. Of course there were all women speakers, all with different backgrounds and areas of work. On top of this diversity, 4 of the 12 speakers were African American -- that's 33%! Compared to the ~3% of us that make up the entire tech industry, that's incredibly refreshing! Although not everyone built their career in a technical field, most of them came up in STEM which makes it even more refreshing. I always love to see women succeed but it is especially helpful for me to see the variety that was exhibited at this event.

Aside from attending out of curiosity, the event included a "poster session" (see me above at poster session :D). I use "" because I think I spent a total of 15 minutes at my poster where there was traffic in that area. That's the one complaint I have about the event -- if I hear poster session I'm thinking I'm gonna have some time to flex. I had almost none. Aside from that, however, the research I am currently working on require developers. And being IBM codes in Java, it seems fitting that I take the opportunity to meet folks and make connections I could use to recruit developers. Fortunately, despite the little to no time I had at my poster to explain my research, I was able to chat with some folks and get them interested in helping me out. I can't say I'd attend to "present" a poster again, however, it was not a total loss :).

As for the bulk of the event, there was a series of talks given by females with various backgrounds in education and research. The major take-aways I got from the series are:

• Always be you; it’s okay to be different! From different comes change, and change is almost always good.
• Nothing is set in stone; don’t be afraid to try different careers. Sometimes that’s what it takes find your passion/niche.
• Adversity is almost inevitable, especially as the minority (women, AA, Latino/a); deal with it in stride and know you’re not alone.
• Women make incredible contributions everyday - let’s keep the trend going and bring our contributions to the forefront where they deserve to be!

For those who are curious, or wanted to attend but missed it, here's the line up for your exploring pleasure :)

Fran O'Sullivan
IBM Senior State Executive for NC and General Manager of Systems Strategy and Operations

Fran O'Sullivan's talk focused on her history at IBM as a woman and lessons she learned along the way. This was a dominant theme among the talks. One interesting part of her timeline, which began in the 80s, is that the first women appeared in her managerial chain in 2014 -- two years ago. Unfortunately, this wasn't super surprising; especially for IBM. Another interesting story was the "Frank" story. One of her bosses called her Frank on a note; she went to his office and asked "Who's Frank?". Of course he was flustered by his mistake, but she made light of the situation. One piece of advice she had for the audience was not to take everything so serious. She ended her talk with call to action to get and keep women in STEM (see photo).

Dr. Tashni Ann Dubroy
President, Shaw University

Dr. Dubroy spoke on her background, experiences, and why she feels we need more women in education and research; more specifically, the need for more some in STEM. She was born in Jamaica, adding to the diversity I spoke of, and decided at a young age she wanted to study chemistry. She spoke on quite a few things I, as well as others, could relate to. For example, she spoke of her difficulties with chemistry when she took her first course and how someone told her she had a "mental block" that was preventing her learning. She eventually overcame this mental block, but I think this is something that happens in CS as well; there are mental blocks regarding concepts that seem difficult when really it's just a matter of relating what you're learning to something you know or understand. She also spoke on how a positive outlook leads to positive direction in life (even when you don't immediately realize it) and how being an "all arounder", or someone with various aspects to their background outside their main area, is a pro not a con.
Dr. Dubroy is also an entrepreneur (part of being an "all arounder" :D), having co-founded the Brilliant and Beautiful Foundation and a hair care line called Tea and Honey Blends. How cool is that?!

Dr. Terri Lomax
Executive Vice President, Discovery-Science-Technology at RTI (Research Triangle Institute)

The theme of Dr. Lomax's presentation was "change is good". I was sold before she began, but if I wasn't she was a great example of why change is good (i.e. not scary, typically for the better). She went through lots of changes in her journey to where she is today, but the most relevant that I think anyone considering getting their PhD should know, is her advisor horror story. She had an advisor that refused to be helpful; whether a personal problem her or just his way, it didn't benefit her. She changed advisors and completed her PhD with a supporting advisor. I know too many people who have had similar issues (and stuck with that advisor for much longer than I could have) -- so know, change is good. Often change, especially in the context of this example, can be the difference between you finishing your PhD in 5 years and 10! One of her changes even brought her to the wonderful NC State :). She also talked about her initiatives to make CS relatable and more appealing to younger audiences by having forensic weekends where they can "do science and meet people". 

Susan Kellogg

Associate Vice Chancellor and Deputy Chief Information Officer in ITS at UNC

Susan Kellogg's, as did the rest, spoke on her journey - more specifically she focused on why she chose academia and advice for career decision making. And she did so without any slides (which is ideal if done well -- and she did a good job). Two major points came from her talk: 1) Pay attention to the fine print and 2) Be true to you. There was an interesting story behind each of these pieces of advice, however, the be true to you was the one that stuck for me. Especially considering I myself am somewhat of a pariah in what I do for various reasons (being an African American woman, my love for tattoos, piercings, and fashion, etc.). Her story centered around her pants suits; yes, pants suits. She didn't realize it as she was doing it, but just by being herself she changed the culture of one of the companies in her career path. Coming into the company, she was the only female to wear pants suits -- rather than changing who she was or shying away from it, she owned it. Before she knew it, more women were wearing pants suits rather than skirts and dresses. Small wins.


Dr. Wanda Lester
Interim Dean of the School of Business,  NCCU

This one hit close to home, as Dr. Lester is from Tallahassee, Florida -- same city as the love of my life :). One of the more experienced speakers of the day, Dr. Lester spoke on her experiences as an African American woman building her career in a time when racial tensions were worse than they are right now (also without slides). Despite any changes, trials, and tribulations she encountered, she kept her head held high and spent many years in educations building her career. She spoke on the importance of mentoring, something I harp on regularly both on here and in person with others I meet. She talked about long-term mentors but she also brought up something I had never thought about, which she called "momentary mentors". These are people who may not always around as a mentor but have or will serve a specific purpose on your career path. Although I have always considered Dr. Bowring to be my mentor, as I think about what a momentary mentor is I know I've had those along the way and continue to meet more as I work towards my degree. 

Ana Biazetti
Chair of IBM NC TEC (North Carolina Technical Experts Council)

I was trying to make the most of the little bit of time I had at my poster, so I missed the first part of this talk.  Based on the portion I saw, there was the similar theme of here's my journey and advice I have based on my experiences. The first slide I saw was "how to be an effective technical leader" (which I took a photo of but the background makes it hard to read). Though I didn't get to hear her talk about it, I can see how her advice can apply to any career, such as collaboration outside your team and focusing on execution for impact.

Dr. Veena Misra
Director of NC State ASSIST

Dr. Misra is a 3 time NC State graduate (BS, MS, PhD) and is now a professor at NC State; one of those rare stores, like that of my co-advisor Sarah Heckman. She discussed with us her work in ASSIST, an NSF center for research on wearables and sensors. I actually got a chance to chat with one of the students working in the ASSIST center and I must say, pretty cool research going on :). She also focused specifically on gifts in her personal journey and challenges. A gift that stood out as something I can only agree with because I've myself experienced it, and that's unexpected opportunities. I take it as a general rule of thumb that unexpected opportunities (hell, any opportunity) should be taken advantage of, especially if they benefit your path or career. On the challenges side, one of the typical challenges discussed is work/life balance, which of course she mentioned. However, she also mentioned some challenges that I never would have thought of myself (but have experienced); those are "dealing with negative news" and "leading while being you". Both are important to your sanity in grad school -- negative news is inevitable so you want to be conscious of how you deal with it and as Susan also said, you never want to lose yourself in anything you do. Always be you, because you are awesome!

Dr. Donna Grant
Associate Professor/ CIS Department Chair, NCCU

Dr. Grant's talks may have been one of my favorites...she was energetic and focused her talk on her journey in STEM and how she learned to soar (something we all want to do). One of the most interesting facts she brought up in her talk is the fact that she got her PhD from DePaul University in 2007...and was the first AA person to ever do so. Not female. Person. This is similar to the PhD program in CS at NC State - the first African American woman to get her PhD was in 2006. For universities like NC State where the PhD program has been around for over 40 years this is craziness. Dr. Grant also discussed her journey from corporate to academia, where she used her corporate background to inform her lessons -- also one of my teaching philosophies I'm developing. Just one advantage to going into industry then back to academia :). She also mentioned quotes that inspire and motivate her, including the serenity prayer and a quote about fear of being powerful.

Dr. Alisha Malloy
Associate Professor, Former CIS Department Chair, NCCU

Sharing a time slot with Dr. Grant, Dr. Malloy (formerly holding the role Dr. Grant has) spoke on her experiences and how her military background informed and led to her decision to pursue a PhD and build a career in academia. Her desire to get a PhD and educate others came from her experience in the military, where she was 1 of 2 women and of course in the minority as an African American. She said something about this discrepancy in numbers and figured she could either complain about the change or be the change. She decided to be the change -- so she got her PhD and moved to academia where she could pay it forward. One of my favorite life philosophies: I got, so now I should give back.

Dr. Susan Rodger
Head of the NC Alice program,  Professor of the Practice, Duke

I've had the privilege of meeting Dr. Rodger and hearing her speak at previous events, so I already knew going in a good bit about her current work. She gave some insights into her background and how she got where she is today, giving advice along the way. One interesting piece of advice came from her discussion on how she chose to attend graduate school. She wasn't sure whether she wanted to go to graduate school or into industry after graduating undergrad, so she put in applications for both. When she decided that grad school was the route for her, she already had job offers. Rather than turning them down outright, she asked each about the opportunity for summer internships -- brilliant! I wish I would have thought of this but hopefully this piece of information will benefit someone :). She also spoke on some of her initiatives, including Alice In Schools and Notable Women in CS.

Dr. Rada Chirkova
Director of NC State STEED

Dr. Chirkova, a professor at NC State (CSC), didn't spend much time talking; however, she made some great points in the short presentation given. She spent some time talking about the STEED ( Science of Technologies for End-to-End Enablement of Data) group's research but spent most of her time talking about the people who have supported her since her start as a professor at NC State, both from the university and individuals. Although I'm a student, not a member of faculty, I can relate to the feeling of support she has felt. Not everyone is as helpful as others, but there are many people that are truly here to help. The major piece of advice she wanted to pass along was to not try to do everything yourself, something else I can relate to from experience. "Listen and delegate" were her exact words. Working for your PhD as well as what comes after can be stressful - the more help and support you can get the less stressed you'll be. 

Dr. Rachana Gupta
Associate Director of NC State ECE Senior Design

Dr. Gupta, another member of the NC State faculty (ECE), also gave a fairly short talk -- possibly because she was last and the event was already past time by the time she came up to bat. She spent most of her time talking about her research and work she does with the ECE Senior Design. She also discussed how to make yourself marketable to companies. One piece of advice she wanted to pass along at the end of her talk -- something I've said many times and heard from others -- is that the PhD really isn't for everyone. It's important to know you want to do the PhD and have some motivation to finish; it gets rough, so you need that intrinsic motivation to keep you going when times get rough (this I know from experience).