We want to dispel this notion that the CHI review process is an adversarial one, and instead characterize the process as one where authors and reviewers work TOGETHER to create the best quality conference possible. Toward that end, we want to communicate to prospective authors (and to reviewers of CHI papers) what makes a successful paper submission to CHI. By "successful", we mean a submission that presents the work with all the information needed to get a fair review. If the work is innovative and interesting to the CHI community, then "successful" submissions should be accepted. But, if the review process finds that the work does not meet the quality threshold for accepted papers, then a "successful" submission will still generate constructive feedback to the authors.
Our observations are based on a study that was conducted on the reviews of CHI '95 papers. We looked at the reviews of papers that were not accepted to the conference to identify the problems cited by reviewers. We hope that making a global statement about CHI's values for papers and making authors aware of common problems will lead to a more successful submission and review process. Specific comments about good submissions are included for each type of paper submission.
A useful place to start discussing successful CHI submissions is the template that reviewers use to review the submission. Although the wording of the review form changes from year, the review form essentially asks:
Because CHI is a multi-disciplinary and international conference, it is important to take a global perspective on "what's new". It's easy to get caught up in your particular specialty in your corner of the world, but successful submissions help people in other disciplines and locales appreciate what is new and significant about your work.
You should also put your work in the context of related previous work. Citing related work and describing how your work differs is a good start towards demonstrating what's new about your work. There's no need (or space) to overdo it, but reviewers frequently complained about the lack of connection with relevant work.
Sometimes authors bury the new and significant parts of the work among the many other issues crammed into the paper. With the limits of eight pages (and about 20 minutes of presentation) to describe your work, it's important to choose an appropriate focus for the paper. In general, it is difficult to make more than three major points in a paper of that length, so think carefully about the issues you want to highlight. Keep in mind that you want the reader to be able to easily identify your contribution and why it is important.
Reviewers often mentioned that an obvious or important issue with the work was not addressed by the authors. They also criticized papers for not providing enough evidence or sound reasoning for their claims. A similar concern was not justifying the design choices and not explaining why certain design features were included. In summary, you should not only explain what you did, but why you did it so that readers (including reviewers) can be convinced that you made appropriate choices. Explaining your choices can also stimulate more research by helping others see alternative approaches.
One way to support your ideas, one that CHI strongly prefers, is to include some evaluation or application of the ideas. CHI is not especially interested in descriptions of new technology without any evaluation of its usefulness to people. If it is not possible to conduct an evaluation with real users, then find some other way to indicate how your work benefits people. CHI is a conference about people using technology, and reviewers are looking for serious consideration of this central issue.
Finally, it's important to pick an appropriate evaluation methodology for the task and to conduct it appropriately. Reviewers often cited inappropriate methodology choices (e.g., using a quantitative method for something that has qualitative effects) and improper uses of the method (e.g., not enough subjects, inappropriate statistical analyses).
You would be surprised at the number of reviewer comments that indicate that the work was not adequately described. Usually the authors simply did not carefully describe what they did or how they studied what they did. Describing the work involves not only good writing, but also a solid structure that helps the reader follow the explanation. It also helps to support the text with figures, tables, even videos that are clear and easy to understand. Authors should: clearly describe what was done and/or how it was studied, write clearly and concisely (avoiding jargon), organize their paper to flow logically and smoothly, provide the right level of detail, and make good use of figures to support the text.
Although all presentations at CHI are made using English, CHI is a conference with an international audience (and an international panel of reviewers). Keeping a global perspective also means using the English language in a way that effectively communicates across cultural boundaries. For authors who are not native speakers of English, reviewers try to assess the quality of the work independent of language issues. However, it is in your best interest to communicate as clearly and effectively as you can in English. If a native English speaker is available, it is a good idea to ask them to proofread your paper before you submit it. For those whose first language is English, it helps to keep in mind that non-native English speakers will be reading and reviewing the paper. It is best to avoid long, complex sentences as well as regional colloquialisms, jokes, or puns that are difficult for someone outside your culture to understand.
It's important to note that CHI reviews papers on an "as-is" basis. The review process does not include enough time for a second review after the author has made requested changes, so reviewers must make a decision whether the submission in its current form is an acceptable CHI publication. (However, reviewers often do make suggestions, and authors of accepted papers are encouraged to revise their work before a final draft is required.) Describing future work (or work expected to be completed before the conference) is often interesting, but you cannot rely on any unfinished work to gain acceptance to the conference. Reviewers often commented that a submission was premature and should be resubmitted when more of the work was completed.
Empirical papers focus on data collected on the use or design of an HCI artifact. Empirical Papers need to use a sound methodology, so that their conclusions are believable and that readers can understand the situations to which those conclusions apply. CHI has a diverse empirical research tradition ranging from the very qualitative to the very quantitative. Some reviewers will NOT share the same assumptions about what makes good research and you will need to justify the appropriateness of your techniques and the significance of the results (not just statistical significance, but the significance to the CHI community). On the other hand, one or more reviewers WILL be steeped in your research tradition and will expect you to apply the appropriate methodological techniques from that tradition. Try to anticipate where expert reviewers might have specific concerns about your approach and provide enough information to reassure them of your methodological rigor.
Experience papers describe how HCI methods, theory, or tools were applied to the design or development of an HCI artifact. Experience Papers are judged on the value of the experience to practitioners -- will someone doing HCI work in a development organization learn something from this paper that they can apply to their practice? If you are describing something that has been done before, you need to have significant added value -- e.g., a combination of methods that is worth more than the individual ones, or ironing out some practical problems in an academic method. It is important to focus on and draw out the information useful to practitioners, rather than just describing your experience.
Systems papers describe the design and implementation of a tool or application that has a novel UI component. The new idea must be original, of interest to a reasonably broad subset of the CHI community, and be based on sound reasoning. Systems papers should be extremely clear about what has and has not been implemented. They should also provide enough information that an experienced researcher could implement a similar system. It is also important to show at least informal evidence that the system is usable and useful to its intended users. This is particularly important if a reviewer's intuitions about usability and usefulness are not consistent with yours -- its hard to argue against solid data that users liked the system or were more productive with it.
Theory papers describe principles, concepts, or models on which HCI work can be based. Theories range from constructs about physical and perceptual behavior to more cognitive and organizational issues. Again, CHI draws from many theoretical traditions. Your paper has to make sense both to those who are highly knowledgeable about your traditions and to those who want to begin using your theoretical constructs and predictions in their own work. You will improve your chances of a fair review if you are very clear about how your paper builds on or contrasts with related work (particularly work in this area that leads to different conclusions than your work). You will also need to make clear the relevance of your results to research and practice in HCI; don't assume that the connections will be obvious to the reviewers.
Methodology papers describe a novel method for the design or evaluation of an HCI artifact. Methodology Papers are judged very similarly to Theory Papers. The additional criterion is that the methodology must be described so that a reader can judge its usefulness. One way to help readers make that judgement is to describe the methodology in detail, which may be difficult to do in eight pages. If you cannot do so, you are obliged to tell the reader how to find the appropriate supplementary materials to use this method. Another approach is to include a study of people using the methodology that demonstrates how it improved the process it was intended to address.
New this year are Opinion papers. Sometimes papers are submitted to CHI that do not meet any of the submission types above, but are provocative essays on some area of interest to the CHI community. Such papers are a challenge to write well, but when a good one comes along, it is often the most discussed paper at the conference. For an Opinion Paper to be accepted, it must cover a topic of interest to a relatively broad segment of the CHI audience, have well supported arguments (including data from research or practice), and be expected to have a stimulating effect on the CHI community. Particularly high standards will be expected of an Opinion Paper -- it may be harder to get an Opinion Paper accepted than any other type of submission.