Research Methods for HCI Book Club

On March 8–16, 2012 Chat will host a 9-day intensive book club on research methods for HCI. The goal of this is to bump up the scientific standard of our chair. (Since we are already awesome on the engineering side, why not make the science great as well?)


Format & content

On each weekday, we will read specific book chapters, papers or watch videos. The materials will cover topics such as:
  • research process: step-by-step
  • qualitative methods: observation, ethnography
  • quantitative methods: survey
  • experimental research: design, implementation, and evaluation
  • statistics
Participants will also spend some time thinking and making connections between the material and their research interest.

In parallel to reading and thinking, the participants will practice statistical analysis from Jacob Wobbrock’s self-study handout: Practical Statistics for HCI.

Participants should expect to spend roughly 8 hours a day (including weekends) to cover all materials. Depending on the background, one might need less time for familiar material.

In the morning of each weekday, we will meet and discuss interesting points for an hour to hash out difficult points and discuss interesting implications before we jump into the next step.


Note

  • This is not a formal course, so the participants will not gain any credits. (On a plus side, there is no exam!)
  • This is also not a job, so no HiWi hours should be spent for that. (On a plus side, knowledge is for free!)
  • This is invitation-only because a small number of participants leads to more active and intense discussion.


Place & time

Weekdays meeting: 9:00–10:00 at Project Space 1
Weekend: Saturday 13:00–14:00 on Skype



Literature

Main:
Supplements:
Scientific writing references:

Schedule (Spring break 2012)

The following sections are the tentative schedule of the club. The order will be updated on-the-fly, depending on the interest of members which will be determined during the daily meetings.

* New material that is added after the book club are indicated with red asterisks.


March, 8: Basic toolbox and experimental research (part 1)
…discoveries are not the fruit of outstanding talent, but rather of common sense enhanced and strengthened by technical education and a habit of thinking about scientific problems.—Santiago Ramón y Cajal
Live activities
  • Elevator pitch: How to tell people about yourself and your research in 2 minutes.
  • Six level of redundancy in papers.

Overview of research methods
  • Lazar: Table of content
    • Overview of established research methods
    • Guiding question: For each method, write down a scenario or research statement that is related to your work that could benefit from using that method.
  • Cozby: 1.3 Goal of Science
    • Three goals of science (which are different from engineering or usability testing)
  • Field: Section 1.2–1.4 Research framework
    • Main framework in research, viewing from the side of quantitative research
  • Lazar: 11.1–11.3 Qualitative research framework
    • ''Alternative framework, viewing from the side of qualitative research"
  • Cozby: 2.1 Hypotheses and predictions (Optional)
    • Term definitions
  • Cozby: 2.3 Source of ideas
    • Where does the research questions come from?
  • Lazar: 1.3 Inherent conflicts in HCI (Optional)
    • Faster or less error does not mean usable
  • Lazar: 1.4 Interdisciplinary nature of HCI research (Optional)
    • You are not alone. People from different expertise contribute different skills
  • Lazar: 1.6 Research and usability testing (Optional)
    • What are the differences?

Literature research skill: reading
  • Adler: pp. 46: The essence of active reading: the four basic questions a reader asks
  • Adler: pp. 48: How to make a book your own, the three kinds of note-making
  • Cozby: 2.5 Anatomy of research article
  • Adler: pp. 32: Inspectional reading I: systematic skimming or pre-reading
  • Adler: pp. 392: Exercise in the second level of reading (part 2) *
  • Zobel: 10.5: Reading (Optional)
  • William G. Griswold: How to Read an Engineering Research Paper (Optional) *

Experimental research overview
  • Lazar: Chapter 2: Experimental research
    • Overview of experimental research and basic term definitions

Visualizing your data
  • Field: 4.2.1 What makes a good graph?
  • Griffiths: 1. Visualizing information
    • Which graph should you use?
  • Griffiths: Appendix 1.1: Other ways of presenting data
  • JMP Help: Tutorials > Graph Builder Tutorial

Descriptive statistics and basic concepts
  • Lazar: 4.1–4.4 Data preparation and descriptive statistics
  • Wobbrock: 01 Concepts & definitions
  • Wobbrock: 02 Understanding a data table

Your research: How do others test their theories, what data is used to support which arguments?
  1. Pick 10 papers that are related to your research direction
  2. For each paper, roughly identify the sequence of methods that is used (e.g., qualitative or quantitative).
  3. For each method, identify arguments why this research method is used.
  4. For each method, identify what the measured data is (i.e., dependent variable in experimental research)
  5. Without considering the outcome, does the measured data support the arguments?
  6. For quantitative data, what visualizations are used to present the data.
  7. Did the author try to deceive you with the graph?

March, 9: Experimental research (part 2)
To call in the statistician after the experiment is done may be no more than asking him to perform a post-mortem examination: he may be able to say what the experiment died of.—Ronald Fisher
Experimental designs
  • Lazar: Chapter 3: Experimental designs
  • Cozby: Chapter 4 Studying behaviors
    • Look for additional types of variables that you should be aware of
  • Cozby: 8.1 Confounding and Internal Validity
  • Cozby: 8.3 Assigning participants to experimental conditions
  • Cozby: Appendix D: constructing a Latin Square (I wrote a Python script for that)
  • Field: 1.5 What to measure
    • Types of variables, errors, validity & reliability
  • (video) Ben Goldacre: Battling bad science

Statistical analysis overview
Fitting linear model and ANOVA
  • (video) Field: Statistical Models on YouTube
  • (videos) Scott E Page: Model thinking L06: linear & nonlinear models *
  • Field: 7.2.(1–3) Introduction to regression (Video on YouTube)
  • Field: 10.2.3–9 ANOVA (One-way, independent) (Video on YouTube)
  • Field: 10.2.11–12 Planned contrasts and Post hoc procedures (Video on YouTube)
  • Wobbrock 03: Introduction to ANOVA

March, 10: Experimental research (part 3) & Experiment protocol reconstruction
No amount of experimentation can ever prove me right; a single experiment can prove me wrong.—Albert Einstein
Practices in experimental research
Common measures in HCI
Statistical analysis
  • Wobbrock: 04 Statistics tools
  • (video) Field: Two-way independent ANOVA on YouTube
  • (video) Field: Repeated measures ANOVA on YouTube
  • (video) Field: Mixed ANOVA on YouTube

Graphing and confidence intervals
  • (Cumming & Finch, 2005) Inference by Eye
    • Easy-to-read introduction about CI
  • (Masson & Loftus, 2003) Using confidence interval for graphically based data interpretation
    • More examples on CI
  • (Tryon, 2001) Evaluating statistical difference, equivalence, and indeterminacy using inferential confidence intervals
    • Tells you what to do if you want to prove equivalence; I'm not sure about its acceptance in CHI community though.

Statistics cautionary tales
  • (Utts, 2003) What Educated Citizens Should Know about Statistics and Probability
    • Common misunderstandings about statistics
  • (Mogie, 2004) In support of null hypothesis significance testing

Side notes
Your research: Reconstructing an experimental protocol.
  1. Pick one experimental research paper that you are interested in.
  2. Try to write an experimental protocol from the description of the experiment in the paper
  3. With the protocol written, try asking yourself which details that are necessary for reconstructing the experiment are missing
  4. Think about your research, sketch an idea of a possible experiment. (You will work on this tomorrow.)

March, 11: Sketching your experiment
A theory can be proved by experiment; but no path leads from experiment to the birth of a theory.—Manfred Eigen
Your research: Sketch an experiment for your research
  1. Based on the idea from yesterday, sketch out important components of your experiments (IV, DV, users, designs, etc.)
  2. Write down an experimental protocol
  3. Debug your experimental protocol: what assumptions have you made that were not explicit
  4. On Monday, your peer will review your experimental protocol

March, 12: Survey
"An approximate answer to the right problem is worth a good deal more than an exact answer to an approximate problem." —John Tukey
Live activities
  • Discussion on experiment protocols

Experimental protocol check list
  1. Is the research question stated clearly?
  2. Is there any alternative interpretation of the question?
  3. Suppose you can prove the stated hypotheses, does it contribute to the understanding of the research question?
  4. Are variables defined clearly on the operational level?
  5. Is there more than one possible interpretation for the variables?
  6. Is the experimental design chosen carefully with consideration of the trade-offs?
  7. Are the statistical methods specified?
  8. Are the resources needed to conduct the experiment stated?
  9. Is the duration of the experiment appropriate?
  10. Ultimate question: If you had no idea about the experiment before, could you pick up this protocol, set up, and conduct the experiment? (Replicability)

Survey
  • Lazar: Chapter 5: Surveys
  • Cozby: Chapter 7: from 'Sampling from a population' to the end of the chapter
  • (Grandhi et al., CHI'05) Sharing the Big Apple
    • You'll revisit the statistics part later, so ignore it for now

Statistics
  • Wobbrock 05: Repeated measures
  • Field 5.2–5.3: What are assumptions? & Assumptions of parametric data
  • Field pp. 98 (Jane Superbrain 4.1): What is an outlier?
  • Field pp. 102 (Jane Superbrain 4.2): Using z-score to find outliers
    • Focus only theoretical part
  • Field 5.7.1: Dealing with outliers
  • Field 5.7.2: Dealing with non-normality and unequal variances
  • Wobbrock 06: Transformations
  • Why is this quote partially right and partially wrong?
    • If your experiment needs statistics, you ought to have done a better experiment - Ernest Rutherford

Tips from the field
  • Make your data table readable by itself
    • Introduce redundancy in data table (human-readable vs machine coded column). It is useful for checking the recorded data. It also helps you to recall the context if you get back to the table later.
  • Data cleaning usually takes more time than the analysis itself
    • Use marginal summary (count, mean, sum) to check the data after you capture it or after any data transformation.

March, 13: Ethnography
"And you know, until you got ice cream spilled on you, you're not doing a field work"—Randy Pausch
Ethnography
  • Lazar Chapter 9: Ethnography
  • David Travis: The 5 Habits of Highly Effective Field Researchers
  • (video) Ethnographic study of projector use from Xerox PARC
  • (Nomura et al., CSCW'06) The uses of paper in commercial airplane flight operations
    • Skim the paper and take a close look at major findings. What evidence is used to support them? How are the evidences presented in text, tables, and figures? How is the story of the paper structured?
  • Optional reading: (Cabtree, CHI'09) Ethnography considered harmful
    • Criticizes an emerging trend of (mis)using ethnography in HCI.

Statistics

March, 14: Interview and Qualitative data analysis
"All research ultimately has a qualitative grounding"–Donald Campbell
Live activities
  • Guest: Christian Corsten—Analyzing data from photo diary

Interview
  • Lazar Chapter 8: Interview and focus groups
  • (secondary) Margolis: 16 Interviewing tips

Qualitative data analysis
  • Lazar Chapter 11: Analyzing qualitative data
  • (Furniss et al., CHI'11) Confessions from a Grounded Theory PhD
    • Concise example of research based on grounded theory and some practical tips
  • (Grandhi et al., CHI'11) Telling Calls
    • Notice how the qualitative data and quantitative data is used to complement each other
    • Note down any questions that you might want to ask Suki tomorrow!

Statistics (secondary)
  • JMP Help > Books > DOE Guide > Chapter 14 Prospective Sample Size and Power (from the beginning until k-Sample Means)
    • Shows how to use JMP to calculate sample size from estimated parameters
  • Optional reading about mixed model: Field 19.3.2: Fixed and random coefficients
  • Optional reading about REML: Field SPSS TIP 19.1 (pp. 746): ML vs. REML

March, 15: Sketching your research expedition
"It is important that your focus be on problems and not on techniques or specialized tools. The latter come and go and as a researcher you want to be able to shift your approaches as needed to solve the more fundamental problems.—Peter Feibelman"

Live activities
  • Field trip: Suki will share her experience about qualitative study

What is the kind of research that the CHI/UIST community expects?
The following articles reflect and criticize the publication process in the HCI area. Knowing the publication process inside-out will help you shape your publication in the way that the community expects. Note that many pieces are just opinions, and there are subcommunities in HCI which might use slightly different practices and values.

Your research: two-month research plan Use the quantitative and qualitative methods you learned so far to sketch a skeleton of your research project. Based on the limited literature that you already know, sketch a plan for two months for conducting the research. Do not include literature review in the plan.

Prepare a short 10 minutes non-slide presentation for tomorrow. You may prepare some diagrams on paper to show the overview and your ideas.

Keep in mind that this plan is based on what you have done so far. The literature might be incomplete, the method might be flawed. It's OK. This is the first practice run for your thesis anyway.

Hint: try to use descriptive, correlative, and experimental research style to test your research questions. (If it suits, of course)


March, 16: Wrap-up and outlook
"The only way of discovering the limited of the possible is to venture a little way past them into the impossible."—Author C. Clarke

Live activities
  • Discussion about presentation
  • 2-month research plan presentation
  • Real-research example of an experimental protocol

Conference
Long-term research goal

Internal resources
Attachments:
File Description File size Downloads Last modified
latinSquare.py Python 1 kB 1944 2012-03-09 09:11
Details

We use cookies on our website. Some of them are essential for the operation of the site, while others help us to improve this site and the user experience (tracking cookies). You can decide for yourself whether you want to allow cookies or not. Please note that if you reject them, you may not be able to use all the functionalities of the site.

Ok
More information