Qualitative Data Analysis: Card Sorting

So you have some qualitative data -- maybe from interviews, maybe from an observation session -- and you want to do some data analysis. You know what you are looking for (i.e. causes of miscommunication or misunderstanding, resolution strategies), but have little to no hypothetical support for any themes amongst your data. One qualitative data analysis technique you can use is card sorting. 

This blog post is written in collaboration with my colleague Justin Smith; it is based on our experiences doing research in our research group (Developer Liberation Front) and my time at Microsoft Research this past summer working with Tom Zimmermann in the Empirical Software Engineering group and what we have found to be most efficient. For those who are interested in how to do a card sort, or how others do card sorting, I'm going to talk about how to do a card sort based on our experiences.

First thing's first: before you can do a card sort, even before you look at your data, you should have an idea of what it is you're looking for in your data. For example, with the study I am working on, I'm interested in what makes tool output difficult to interpret; more specifically I want to identify the areas where there is either miscommunication, risk of miscommunication, or misunderstanding and see what causes each.

• Recommendation #1: Come up with some criteria that can be used when extracting quotes. Your criteria should be based on whatever it is you're searching for in the transcript. For example, with Justin's study, we wanted to find implicit and explicit questions developers need answered when resolving security vulnerabilities; therefore, one trivial criteria was that the text should be an explicit question posed by the developer. The more defined and specific what you're looking for is, the easier it will be to extract data for your card sort.
• Recommendation #2: Have at least two people extract quotes (including yourself obviously); you should all be using the criteria you put together. This will help validate your criteria as well as increase the validity of the quotes you extract. If you're using two people, both should extract quotes from all transcripts; once that's done, the two of you should sit down and determine where you agree and disagree. Two is usually enough, but if you decide to use three people, we recommend having 2 people work on different sets of data (i.e. person 1 and 2 on a set, person 1 and 3 on a set...); once finished, each couple will work out their disagreements. This is something you will want to report when you attempt to publish your findings :) (it's called inter-rater reliability).

Once you have your set of quotes from your transcript, you need to put them on notecards. Advice from someone who's been there: although having physical, paper notecards is a necessity for a card sort, we highly recommend having an electronic copy of your quotes (I put mine in Excel).

For our first card sorts, we manually made physical note cards -- they look nice and the card sort itself was okay, but keeping track of themes and sub-themes during the card sort, and after, was not trivial and sometimes led to confusion and the need for back tracking.

A pointer regarding using spreadsheets to store your data:

• Anything you would include on your notecard should be a column in the spreadsheet; for example, the columns in my spreadsheet are Participant (P1, P2...), Tool Being Used, and Quote. I also have a column for a unique identifier for each quote (Card Number) and the Emergent Themes from each round of the card sort. It might also be beneficial to include a Timestamp column; this way, you can have an approximate location in your media to find the quote if needed.

There are a number of advantages to having your quotes in a spreadsheet, specifically Excel:

1. You don't have to worry (as much) about water or a random fire ruining your data. It's also harder to lose your data if it's electronically stored.
2. Spreadsheets are searchable; paper notecards are not.
3. Typically card sorts are often done in iterations, and you want to be able to report anything that happens (i.e. cards moving from one theme to another); this is MUCH easier if you can put the themes for each round of the card sort into a spreadsheet. 
4. Organizing paper notecards can be tedious and error-prone (i.e. trying to find a card and messing up an entire pile); having an electronic copy you can easily organize your data. If you decide to organize your notecards, you at least know you have a proxy of how the data was in your electronic copy.
5. Actually making notecards is a time consuming process, especially when making them by hand. Instead, if you have your quotes in an Excel spreadsheet, and label the columns as mentioned earlier, you can use Mail Merge to create your notecards electronically :). Details regarding how to do this can be found in the following supplementary blog post.

Now, it's time to complete the card sort. This can be done in one phase, however, we recommend doing it in multiple phases. Minimally, there's a phase 1 for preliminary sorting into themes and phase 2 for sorting each theme into high-level themes. This is particularly useful for large datasets where you can wind up with a large number of themes after the first phase; typically there are common themes amongst those separate themes, thereby warranting another phase of sorting. You may also want to include a validation phase once you have determined all the low level emergent themes (after phase 1). This phase is to ensure that all quotes have been sorted into the best possible theme.

• Recommendation #1: Include others in the card sort process; this lessens the bias behind the themes you find (and from our experience helps come up with distinct themes with clear definitions, which is super important when working with qualitative data). One thing to be aware of is the more cooks in the kitchen, the more time it might take to complete (more potential for disagreement and need for discussion), so plan accordingly. For our most recent card sort, 2 hours was our max at one time so, with a little over 300 notecards, we did four 2-hour sessions. Previously I've had shorter sessions; the longer time for this study, we believe, is a product of the type of data we're working with (non-interview).
• Recommendation #2: As you're doing you're card sort, keep track of important information as it changes (i.e. how you define your themes/sub-themes). Also, keep track of quotes that you and your sorters believe best represent each theme. Doing these things will make reporting your findings much easier. 

Once you've done all this, you're ready to start thinking about what your paper is going to look like and where the interesting stories are in your themes (the fun part). With that being said, good luck fellow qualitative researchers! :)

Thanks again to Justin for helping me put this together and the Developer Liberation Front and Tom for the experiences!

Creating Notecards using Microsoft Word Mail Merge

This blog post provides step-by-step instructions for creating notecards (possibly for a card sort) using Microsoft Excel and Word. Thanks to Tom Zimmermann of Microsoft Research who taught me this nifty trick :D.

This tutorial assumes you already have your data in a spreadsheet, as discussed in another blog post.

1. Save your excel spreadsheet -- make sure your data columns are labeled.
2. Open or create a Word template with the number of notecards you would like on one page. I created my own using the Create Table command in Word; either way, make sure you know what size each notecard is (you'll need it later).
3. Once you have the template/document open, you will need to put in the various components of your notecard where you'd like them to be. As an example, my template I created based on my spreadsheet columns can be found here.
4. Next, select the Mailings tab > Start Mail Merge > Email Messages. It technically doesn't matter what you select here; I choose Email Messages because it makes life easier :).
   
5. Next, select Select Recipients >  Use An Existing List...
   Browse to and select the spreadsheet with your data. If your quotes are in a workbook, make sure you select the correct sheet.
   
   
6. Now you want to map the different "fields" in your document to the columns in your spreadsheet. This is done using the Insert Merge Field menu.
   
   The list that comes is populated with the columns in your spreadsheet -- insert each field into one notecard (i.e. replace <<participant>> with Participant merge field) then copy and paste all fields into each notecard. Once finished, it will look something like this:
   

   
   
   
7. If you click Preview Results you can see what your notecards will look like. If you click it now, each notecard will have the same information on it. This is because you have to tell Mail Merge to go to the next record in the spreadsheet.
   
   To tell Word you want to go to the next record for each notecard, you need to add a rule; this is done by going to the Rules menu and selecting Next Record. You need the <<Next Record>> field on each notecard; it should look something like this:
   

   
   
   

   Now when you click Preview Results you should see different data for each notecard.

   
   
8. Now you're ready to Finish & Merge!

   

   I typically select Edit Individual Documents... so I can make sure everything is copesthetic.
9. The final step is optional, but recommended: because quotes vary in length, depending on the template you used for your notecards you may want to remove any extra space added for the short quotes due to the long quotes. It's a tedious process but helps save paper :).

Once you have your notecards ready, you can print, cut, then you're ready to get to sortin'! :D

IBM University Day 2015 - Women in Education and Research

I attended IBM University Day for the first time this past Friday. I didn't know what to expect. I had never been to this event, which apparently had been going on for some time now. Also, I've always heard (and sometimes felt) a stigma behind IBM that made me weary of an event they would have titled "Women in Education and Research". Aside from the fact that I've always associated IBM with conservative, older white men, as an African American woman I'm always looking for someone like me doing extraordinary thing at events like this and am most often disappointed. I just knew this event would be no different...except I was surprisingly and so thankfully wrong. Of course there were all women speakers, all with different backgrounds and areas of work. On top of this diversity, 4 of the 12 speakers were African American -- that's 33%! Compared to the ~3% of us that make up the entire tech industry, that's incredibly refreshing! Although not everyone built their career in a technical field, most of them came up in STEM which makes it even more refreshing. I always love to see women succeed but it is especially helpful for me to see the variety that was exhibited at this event.

Aside from attending out of curiosity, the event included a "poster session" (see me above at poster session :D). I use "" because I think I spent a total of 15 minutes at my poster where there was traffic in that area. That's the one complaint I have about the event -- if I hear poster session I'm thinking I'm gonna have some time to flex. I had almost none. Aside from that, however, the research I am currently working on require developers. And being IBM codes in Java, it seems fitting that I take the opportunity to meet folks and make connections I could use to recruit developers. Fortunately, despite the little to no time I had at my poster to explain my research, I was able to chat with some folks and get them interested in helping me out. I can't say I'd attend to "present" a poster again, however, it was not a total loss :).

As for the bulk of the event, there was a series of talks given by females with various backgrounds in education and research. The major take-aways I got from the series are:

• Always be you; it’s okay to be different! From different comes change, and change is almost always good.
• Nothing is set in stone; don’t be afraid to try different careers. Sometimes that’s what it takes find your passion/niche.
• Adversity is almost inevitable, especially as the minority (women, AA, Latino/a); deal with it in stride and know you’re not alone.
• Women make incredible contributions everyday - let’s keep the trend going and bring our contributions to the forefront where they deserve to be!

For those who are curious, or wanted to attend but missed it, here's the line up for your exploring pleasure :)

Fran O'Sullivan
IBM Senior State Executive for NC and General Manager of Systems Strategy and Operations

Fran O'Sullivan's talk focused on her history at IBM as a woman and lessons she learned along the way. This was a dominant theme among the talks. One interesting part of her timeline, which began in the 80s, is that the first women appeared in her managerial chain in 2014 -- two years ago. Unfortunately, this wasn't super surprising; especially for IBM. Another interesting story was the "Frank" story. One of her bosses called her Frank on a note; she went to his office and asked "Who's Frank?". Of course he was flustered by his mistake, but she made light of the situation. One piece of advice she had for the audience was not to take everything so serious. She ended her talk with call to action to get and keep women in STEM (see photo).

Dr. Tashni Ann Dubroy
President, Shaw University

Dr. Dubroy spoke on her background, experiences, and why she feels we need more women in education and research; more specifically, the need for more some in STEM. She was born in Jamaica, adding to the diversity I spoke of, and decided at a young age she wanted to study chemistry. She spoke on quite a few things I, as well as others, could relate to. For example, she spoke of her difficulties with chemistry when she took her first course and how someone told her she had a "mental block" that was preventing her learning. She eventually overcame this mental block, but I think this is something that happens in CS as well; there are mental blocks regarding concepts that seem difficult when really it's just a matter of relating what you're learning to something you know or understand. She also spoke on how a positive outlook leads to positive direction in life (even when you don't immediately realize it) and how being an "all arounder", or someone with various aspects to their background outside their main area, is a pro not a con.
Dr. Dubroy is also an entrepreneur (part of being an "all arounder" :D), having co-founded the Brilliant and Beautiful Foundation and a hair care line called Tea and Honey Blends. How cool is that?!

Dr. Terri Lomax
Executive Vice President, Discovery-Science-Technology at RTI (Research Triangle Institute)

The theme of Dr. Lomax's presentation was "change is good". I was sold before she began, but if I wasn't she was a great example of why change is good (i.e. not scary, typically for the better). She went through lots of changes in her journey to where she is today, but the most relevant that I think anyone considering getting their PhD should know, is her advisor horror story. She had an advisor that refused to be helpful; whether a personal problem her or just his way, it didn't benefit her. She changed advisors and completed her PhD with a supporting advisor. I know too many people who have had similar issues (and stuck with that advisor for much longer than I could have) -- so know, change is good. Often change, especially in the context of this example, can be the difference between you finishing your PhD in 5 years and 10! One of her changes even brought her to the wonderful NC State :). She also talked about her initiatives to make CS relatable and more appealing to younger audiences by having forensic weekends where they can "do science and meet people". 

Susan Kellogg

Associate Vice Chancellor and Deputy Chief Information Officer in ITS at UNC

Susan Kellogg's, as did the rest, spoke on her journey - more specifically she focused on why she chose academia and advice for career decision making. And she did so without any slides (which is ideal if done well -- and she did a good job). Two major points came from her talk: 1) Pay attention to the fine print and 2) Be true to you. There was an interesting story behind each of these pieces of advice, however, the be true to you was the one that stuck for me. Especially considering I myself am somewhat of a pariah in what I do for various reasons (being an African American woman, my love for tattoos, piercings, and fashion, etc.). Her story centered around her pants suits; yes, pants suits. She didn't realize it as she was doing it, but just by being herself she changed the culture of one of the companies in her career path. Coming into the company, she was the only female to wear pants suits -- rather than changing who she was or shying away from it, she owned it. Before she knew it, more women were wearing pants suits rather than skirts and dresses. Small wins.


Dr. Wanda Lester
Interim Dean of the School of Business,  NCCU

This one hit close to home, as Dr. Lester is from Tallahassee, Florida -- same city as the love of my life :). One of the more experienced speakers of the day, Dr. Lester spoke on her experiences as an African American woman building her career in a time when racial tensions were worse than they are right now (also without slides). Despite any changes, trials, and tribulations she encountered, she kept her head held high and spent many years in educations building her career. She spoke on the importance of mentoring, something I harp on regularly both on here and in person with others I meet. She talked about long-term mentors but she also brought up something I had never thought about, which she called "momentary mentors". These are people who may not always around as a mentor but have or will serve a specific purpose on your career path. Although I have always considered Dr. Bowring to be my mentor, as I think about what a momentary mentor is I know I've had those along the way and continue to meet more as I work towards my degree. 

Ana Biazetti
Chair of IBM NC TEC (North Carolina Technical Experts Council)

I was trying to make the most of the little bit of time I had at my poster, so I missed the first part of this talk.  Based on the portion I saw, there was the similar theme of here's my journey and advice I have based on my experiences. The first slide I saw was "how to be an effective technical leader" (which I took a photo of but the background makes it hard to read). Though I didn't get to hear her talk about it, I can see how her advice can apply to any career, such as collaboration outside your team and focusing on execution for impact.

Dr. Veena Misra
Director of NC State ASSIST

Dr. Misra is a 3 time NC State graduate (BS, MS, PhD) and is now a professor at NC State; one of those rare stores, like that of my co-advisor Sarah Heckman. She discussed with us her work in ASSIST, an NSF center for research on wearables and sensors. I actually got a chance to chat with one of the students working in the ASSIST center and I must say, pretty cool research going on :). She also focused specifically on gifts in her personal journey and challenges. A gift that stood out as something I can only agree with because I've myself experienced it, and that's unexpected opportunities. I take it as a general rule of thumb that unexpected opportunities (hell, any opportunity) should be taken advantage of, especially if they benefit your path or career. On the challenges side, one of the typical challenges discussed is work/life balance, which of course she mentioned. However, she also mentioned some challenges that I never would have thought of myself (but have experienced); those are "dealing with negative news" and "leading while being you". Both are important to your sanity in grad school -- negative news is inevitable so you want to be conscious of how you deal with it and as Susan also said, you never want to lose yourself in anything you do. Always be you, because you are awesome!

Dr. Donna Grant
Associate Professor/ CIS Department Chair, NCCU

Dr. Grant's talks may have been one of my favorites...she was energetic and focused her talk on her journey in STEM and how she learned to soar (something we all want to do). One of the most interesting facts she brought up in her talk is the fact that she got her PhD from DePaul University in 2007...and was the first AA person to ever do so. Not female. Person. This is similar to the PhD program in CS at NC State - the first African American woman to get her PhD was in 2006. For universities like NC State where the PhD program has been around for over 40 years this is craziness. Dr. Grant also discussed her journey from corporate to academia, where she used her corporate background to inform her lessons -- also one of my teaching philosophies I'm developing. Just one advantage to going into industry then back to academia :). She also mentioned quotes that inspire and motivate her, including the serenity prayer and a quote about fear of being powerful.

Dr. Alisha Malloy
Associate Professor, Former CIS Department Chair, NCCU

Sharing a time slot with Dr. Grant, Dr. Malloy (formerly holding the role Dr. Grant has) spoke on her experiences and how her military background informed and led to her decision to pursue a PhD and build a career in academia. Her desire to get a PhD and educate others came from her experience in the military, where she was 1 of 2 women and of course in the minority as an African American. She said something about this discrepancy in numbers and figured she could either complain about the change or be the change. She decided to be the change -- so she got her PhD and moved to academia where she could pay it forward. One of my favorite life philosophies: I got, so now I should give back.

Dr. Susan Rodger
Head of the NC Alice program,  Professor of the Practice, Duke

I've had the privilege of meeting Dr. Rodger and hearing her speak at previous events, so I already knew going in a good bit about her current work. She gave some insights into her background and how she got where she is today, giving advice along the way. One interesting piece of advice came from her discussion on how she chose to attend graduate school. She wasn't sure whether she wanted to go to graduate school or into industry after graduating undergrad, so she put in applications for both. When she decided that grad school was the route for her, she already had job offers. Rather than turning them down outright, she asked each about the opportunity for summer internships -- brilliant! I wish I would have thought of this but hopefully this piece of information will benefit someone :). She also spoke on some of her initiatives, including Alice In Schools and Notable Women in CS.

Dr. Rada Chirkova
Director of NC State STEED

Dr. Chirkova, a professor at NC State (CSC), didn't spend much time talking; however, she made some great points in the short presentation given. She spent some time talking about the STEED ( Science of Technologies for End-to-End Enablement of Data) group's research but spent most of her time talking about the people who have supported her since her start as a professor at NC State, both from the university and individuals. Although I'm a student, not a member of faculty, I can relate to the feeling of support she has felt. Not everyone is as helpful as others, but there are many people that are truly here to help. The major piece of advice she wanted to pass along was to not try to do everything yourself, something else I can relate to from experience. "Listen and delegate" were her exact words. Working for your PhD as well as what comes after can be stressful - the more help and support you can get the less stressed you'll be. 

Dr. Rachana Gupta
Associate Director of NC State ECE Senior Design

Dr. Gupta, another member of the NC State faculty (ECE), also gave a fairly short talk -- possibly because she was last and the event was already past time by the time she came up to bat. She spent most of her time talking about her research and work she does with the ECE Senior Design. She also discussed how to make yourself marketable to companies. One piece of advice she wanted to pass along at the end of her talk -- something I've said many times and heard from others -- is that the PhD really isn't for everyone. It's important to know you want to do the PhD and have some motivation to finish; it gets rough, so you need that intrinsic motivation to keep you going when times get rough (this I know from experience).

VL/HCC Day 3

I will state again, I love VL/HCC :)
If ever given the opportunity to go (if you do any human centric or visual language research), GO! It's valuable in so many ways...even the talks you think you're not interested in you end up loving. Like today's keynote...I thought, not another blocks talk...and then it happened. And I was glad I was there.


The keynote today was titled "Taking Stock of Blocks: Promises and Challenges of Blocks Programming Languages" given by Franklyn Turbak of Wellesley College Computer Science Department. I was going to write a post about it, but Felienne already did a great job, so check it out!

One thing I found interesting is that there is a long history of blocks-based programming. Here is the lineage discussed in the talk:

Blox (Glinert, 1986) - first time puzzle pieces used to represent code
LogoBlocks (Begel, 1996)
Alice (Pausch et al., 2001) - 3D animations; evolved from Python to drag and drop
PicoBlocks (Bonta, Silverman, et al., 2006) - microprocessor for robotics; passes the "Lucite Test" - imagine constructing out of physical blocks; also has extension language
Scratch (Resnick et al., 2007) - best of Alice and PicoBlocks
StarLogo TNG (Roque, Wendel, et al., 2007) - created OpenBlocks frameworks for other to build language
BYOB/Snap! (Harvey et al., 2008) - have first class functions
App Inventor Classic - clunky
Blockly (Fraser, 2012) - javascript based (embedded in web browser); mutators = edit blocks with other mini block language
App Inventor 2 (2013) - local variables, improved parameters
PencilCode (Bau 2013) - toggle between blocks and text in interesting way
Droplets (Bau 2014) - '' ''

Languages with physical blocks (cool!)
Robot Park
Tangible Kindergarten

As the conference comes to an end, I think about how grateful I am for the experiences I've had and continue to have as PhD student. I met and connected with one of my research/blogging idols (Felienne - we even had drinks together! :D), made some new unexpected connections with other researchers that I should definitely know (Mark Guzdial, Caitlin Kelleher,  Ronald Metoyer to name a few), and even made the realization that I've advanced in my field/area as I know more and more people that I encounter at these conferences (and they actually like my research!!). It's venues like VL/HCC where I feel like I get the most value as a researcher -- I'm walking away more ready and confident than I came. And that's alright :).

Favorites talks (from first session -- took second half to visit family :D):

A Syntax-Directed Keyboard Extension for Writing Source Code on Touchscreen Devices
Islam Almusaly and Ronald Metoyer
(screenshot)

Adapting Higher-order List Operators for Blocks Programming
Soojin Kim and Franklyn Turbak
PHOLOs - "Pseduo-Higher-Order Operators"

Hub Map: A new approach for visualizing traffic data sets with multi-attribute link data
Andrew Simmons, Iman Avazpour, Hai L. Vu, Rajesh Vasa
perfect for venue location (ATL known for traffic)


Interesting papers (missed the talk):
Natural Language Programming: Designing Effective Environments for Novices
Judith Good and Katherine Howland

A Principle Evaluation for a Principled Idea Garden
Will Jerigan, Amber Horvath, Michael Lee, Margaret Burnett, Taylor Cuilty, Sandeep Kuttal, Anicia Peters, Irwin Kwan, Faezeh Bahmani, Andrew Ko

Enabling Independent Learning of Programming Concepts through Programming Completion Puzzles
Kyle J. Harms, Noah Rowlett, Caitlin Kelleher

VL/HCC Day 2 - Keynote and other memorable talks

Another great day at VL/HCC :).

To start, there was an amazing keynote given by Georgia Tech's own Mark Guzdial, a CS Education legend, titled "Requirements for a Computing-Literate Society". The focus of the talk was the challenges while working towards a computing-literate society and how we can re-invent CS education to accomplish this goal. I must say, as an African American female from South Carolina, where CS education is almost non-existent in K-12 education and almost completely unrelatable beyond that, I could completely understand and relate to everything he said. Because I loved it SO much, I took some notes to share some of the major points.

WHY SHOULD WE CARE?
    Mark made a few great points as to why we should care about the advancement and spreading of CS education and knowledge. The two predominant ones (that for sure stood out) are that 1) CS is study of process/problem solving, which impacts everyone and 2) learning and understanding computer science provides the ability for people to express themselves in ways they couldn't without programs/automation.

WHAT ARE THE CHALLENGES TO ACCOMPLISHING THIS GOAL?
   One of the obvious and most talked about challenges is access to computer science courses or resources for learning CS and accommodations for diversity -- though there are initiatives, such as CODE2040 aimed at increasing diversity and access to computing resources, there is still a gap leading to lower participation. This is especially true for underrepresented minorities like myself.

Another challenge, which I never thought about, was what Mark called the "inverse lake wobegon effect" -- in other words, we think we know more than we do. This theory suggests that the way things are currently done, we only know the top half. The top half, which I would NOT include myself in, would be for example students with CS courses in high school. Those with access are most privileged meaning they are the ones that get noticed by universities or even at the university level getting noticed by professors and other students as being the "real deal" while people like me fall by the waist-side. Thank goodness for my mentor who wouldn't let that happen - #thepowerofmentors

The last challenge he discussed was the unanswered questions that policy-makers continue to ask. To segway into this discussion, Mark starts with a discussion of initiatives such as "Georgia Computes!" and CAITE that aim to inspire students to study CS. He also eludes to some of the differences across states that make it difficult to make concrete, uniform improvements -- these differences also relate to to unanswered questions that could affect how we improve and facilitate CS education for the masses. For example, states vary on opinions of AP CS courses (apparently to some, access to advanced education is considered 'elitist'), what CS is, and whether to require CS. From this, Mark notes other questions that are unanswered, and extremely relevant, such as are the CS requirements really CS. For this he used South Caroline as an example, which hit home for me. I was shocked when I heard that the SC curriculum requires CS for graduation. It was a few years back when I graduated and although I did take the one and only programming class offered by my high school, I knew nothing of a CS requirement or anything close to it. According to some of his findings, one of the questions that need to be answered is 'what kind of CS can we teach to everyone?' -- especially considering how important a high school degree is. We don't want to have people not graduating because they can't pass a CS course but at the same time, typing is not Computer Science >.>.

One of the more interesting findings presented is that although initiatives like "Georgia Computes!" seems to be effective overall, it has different affects on different minorities/populations. But why?? For example, when observing the affects for black in CS (graph above), there is almost a completely flat line, meaning there is no increase in participation. This all showcases the need for more research and exploration into why minorities, such as African Americans, choose to study CS and how we can get them engaged and keep them engaged. Fortunately, Mark had some ideas for that as well :)

WHAT ARE SOME POSSIBLE SOLUTIONS?

   Mark discussed two possible solutions, each of which take on a different aspect of the CS education problem.

1) The Role of Context - For many, the problem is that CS seems irrelevant to them or their lives. I can say for me, if I hadn't discovered I could fiddle with my MySpace page and that be considered  CS, I might have thought the same thing! But many are asking, how can computing be considered irrelevant? Easy. The context in which we teach/introduce CS is critical, especially for younger audiences who may have preconceived (incorrect) notions of what CS is. For example, some of the problems we are first confronted with when learning CS are tower of hanoi or fibonnacci sequence. How many of us can say that what we do now is at all relevant or related to either of these?? I know I sure can't. Increasing interest could be as simple as teaching in a relevant context (i.e. robots and digital media).
An example of this that he spoke about is called Glitch Game Testers, started by Betsy DiSalvo and Amy Bruckman. This program hires African American males as game testers and have them think about computing deeper than the curriculum. Turns out, all of them finished high school and over half continued on to take computing classes post-secondary. This further showcases the importance of making CS relatable and relevant.

2) Understanding CS Teachers Needs - An important foundation for CS education is of course the educators! Mark suggests that teachers need sense of identity, which takes the form of confidence in their ability, as well as a sense of community with role models to look up to (which I relate to completely). Disciplinary Commons is a group that was started to bring together CS teachers to talk about classes; it's not big, but it's a step in the right direction. There is also a need for more professional learning on how to teach CS; too often we assume that to teach someone about CS, you have to be damn near a software developer. But is that true? Mark posed this question, and it's another one of those things you don't think about until someone else says it. An example he used is that one of the most successful CS teachers actually focus less on coding and more on writing assignments. Because the goal is for students to learn and understand what CS is. If it's something their interested in, the rest will follow.

Although my research area isn't CS Education, I was extremely moved by this talk and I hope the work continues and gets the publicity and backing it needs to really make a difference for states like my home.


As I did yesterday, here are some of the more memorable talks from today (again, in my opinion :D):

Supporting Exploratory Data Analysis with Live Programming
Robert DeLine and Danyel Fisher
Related:
Tempe -- web app for live exploration and analysis of data

Tempe: Live Scripting for Live Data (short paper on technology from full paper above)
Robert DeLine, Danyel Fisher, Badrish Chandramouli, Jonathan Goldstein, Michael Barnett, James Terwilliger, and John Wernsing

Jeeves – A Visual Programming Environment for Mobile Experience Sampling
Daniel Rough, Aaron Quigley
replacing paper "diaries" and expensive or difficult apps with Jeeves (using visual programming)
great/engaging slide set!

Detecting Problematic Lookup Functions in Spreadsheets
Felienne Hermans, Efthimia Aivaloglou, Bas Jansen
discusses usage of and problems with lookup functions in Excel
<3 presentation style (less than 1 minute summary at end)

Interactive Visual Machine Learning in Spreadsheets
Advait Sarkar, Mateja Jamnik, Alan Blackwell and Martin Spott
BrainCel v0.2 - spreadsheets and visualizations to help end users use and understand machine learning

Extending Scratch: New Pathways into Programming 
Sayamindu Dasgupta, Shane Clements, Abdulrahman Y. Idlbi, Chris Willis-Ford and Mitchel Resnick
Scratch Extension System, toolkit for anyone to extend Scratch language and capabilities
Resources:
http://scratchx.org
http://wiki.scratch.mit.edu/wiki/Scratch_Extension_Protocol_(2.0)
Considerations:
maintaining low barrier to entry, consistency with other blocks/conventions, and right level of abstraction, choosing the right extension

Strengthening Collaborative Groups Through Art-Mediated Self-Expression
Mengyao Zhao, Yi Wang, David Redmiles
Building interpersonal relationships between local and remote team members via art with Doodled "Us" - collaborative doodle system

Understanding Triggers for Clarification Requests in Community-Based Software Help Forums
Nathaniel Hudson, Parmit K. Chilana, Xiaoyu Guo, Jason Day, and Edmund Liu
What causes people to ask clarifying questions to improve Q&A site experiences --> design interventions to make Q&A sites more efficient



I will try to post as much as I can tomorrow - it's my last day so I really want to visit family that's down here. One of the downsides of the PhD is you really don't get to see family and friends as much as you like so advantage I will take of this :)
Until next time!

VL/HCC 2015 - Atlanta, GA (GC & Day 1)

Whew!

After traveling to Houston for Grace Hopper (post to come soon about the happenings there), and now to ATL for VL/HCC (IEEE Symposium on Visual Languages and Human Centric Computing) I'm pretty exhausted. BUT not too exhausted to share some of the greatness that has gone down since I've gotten here for the conference :).

To start, the GC (Graduate Consortium for those who don't know) was amazing. I found myself comparing it to last year, and I can honestly say it just keeps getting better. Now, this may be because my research gets more refined, which makes it more suitable to feedback, but I feel like I really got some ideas that are gonna push me in the right direction. On top of the great feedback I got, I met and connected with some AMAZING PhD students at various stages of their career. Despite all having unique research interests and directions, we were all able to provide insights to improve each others' work (and even potentially reference each others' work).

Now, for the first day of the conference. The theme this year is "Computational Thinking and Computer Science Education"...one thing I love about VL/HCC is that it's a smaller venue, so it's a lot more personal. I walked in for the intro and first session and found a seat next to a friendly looking female...of course I asked if the seat was taken and then proceeded to introduce myself. Come to find out, I was sitting next to none other than Felienne Hermans of Delft University in the Netherlands! The cool thing about it is I've been virtually stalking her ever since she wrote a blog post about my first conference paper/presentation 2 years ago...and now I've met her face to face. And it feels awesome. The most awesome part is I got to talk to her about my dissertation research, and she loved the idea! We did some brainstorming and she even had some work she's going to pass my way related to it. Yet another boost of confidence for my dissertation research :).

It would have been hard to ruin my day after how it started; fortunately, I didn't have to worry about that. The day was filled with interesting talks related to computational thinking and computer science education. Although all the talks were great, here are some of my favorites from the day:

Tutorons: Generating Context-Relevant, On-DemandExplanations and Demonstrations of Online Code - context relevant explanations of code (browser add-on)
Andrew Head, Codanda Appachu, Marti A. Hearst, Bjorn Hartmann
Resources:
http://www.tutorons.com/


Codepourri: Creating Visual Coding TutorialsUsing A Volunteer Crowd Of Learners - crowdsourced visual tutorials for learning to program (using crowd of learners)
Mitchell Gordon and Philip J. Guo

Personality and Intrinsic Motivational Factors in EUP - model that predicts motivation based on personality profiles
Saeed Aghaee, Alan F. Blackwell, David Stillwell, Michal Kosinki
Terms:
1. bricoleurism - like to build things, tinkering with stuff
2. technophilia - love of tech/new tech
3. artistry - enjoy experimenting with creative ideas
Resources:
MyPersonality dataset
Big Five Inventory Personality Test
Related:
Facebook 'likes' to predict personality profiles
Computer better predictor of personality than people

Scientists Tell Stories About Seeking Help with Programming - "war stories" to determine help seeking behaviors of EU scientists [qualitative study]
Brian Frey and Carolyn Seaman

Facilitating Testing and Debugging of Markov Decision Processes with Interactive Visualizations
Sean McGregor, Hailey Buckingham, Thomas G. Dietterich, Rachel Houtman, Claire Montgomery, Ronald Metoyer
Resources:
MDPVis.github.io

A Study of Interactive Code Annotation for Access Control Vulnerabilities
Tyler Thomas, Bill Chu, Heather Lipford, Justin Smith, Emerson Murphy-Hill
Memorable Quote: "Ain't nobody wanna be hacked"

Codechella: program for interactive and collaborative tutoring/building mental models - simulate in-person help in an online platform
Philip J. Guo, Jeffery White, Renan Zanelatto
Resources:
Demos: pgbovine.net/rosetta
Live: pythontutor.com

Semantic Zooming of Code Change History
Youngseok Yoon and Brad A. Myers


That's all for now - I'll try to post again for day 2 and will definitely be posting about GHC soon. Until next time!

JGit Growing Pains

So, I've been using JGit (I believe a previous blog post talks about this) for about a year now...I need someone to explain to me why I'm just learning how the revert command works for the API!?

Let me break it down...JGit allows you to manipulate repositories from Java. Cool shit right? However, either from my lack of experience with JGit or my (at the time) noob-ish knowledge of Git, I am just learning that revert does not do what I would expect. Take the following code for example...

for (RevCommit rev: revisions){

if (ObjectId.toString(rev.getId()).equals(currentHash)){

git.revert().include(rev).call();

}

}

Now, as most developers do, I went to the internet to find out how to revert a repository from Java, and this is what I found. Not much more documentation than what you're looking at right here...As I could recall, the way revert worked was that you can revert your repository to any given revision -- because of this I made the silly assumption that the code above would revert the repository to the RevCommit I passed in. Except I was wrong. And I didn't realize I was wrong until the revision I was analyzing did not match up with the revision I was diffing in my code from a file add...

Today, one year later, I have been scouring the internet to find out how this sequence of method calls work together and how, specifically, the include(RevCommit)method works. Does include mean this is the revision I want to revert to? Does it mean I want to revert the changes I made at this revision to the previous revision? Do I need to use more than one include call to revert back more than one revision? All of these are questions I have that unfortunately the interwebs is doing not so great a job of answering.

For those curious, revert (in this context) actually works by reverting the changes made to the RevCommit passed in. So whatever revision gets passed in is reverted to the revision before...which meant, due to my misunderstanding, I was basically doing everything backwards. I'm almost embarrassed it took me so long to realize this problem; until I realize that if there was some clear documentation out there that told me this information I could have discovered this issue earlier (or prevented it all together). For those of you thinking "that couldn't have made that big of a difference"...once I got the revert process correct, I had to restructure the way my code works for some of the new detectors I implemented as well as to detect any sort of pattern removal from the repository.

So my word of advice? If you're using JGit for analyzing source code at each revision in a repository and care about at which revision source code was introduced, know that the revert command should always come AFTER analysis if you care at all about analysis and revisions matching up (which I do). This makes it sound intuitive...but is it? 

Hopefully this helps someone else struggling with this particular feature of JGit. Until next time!