Version of 31 July 2007
The quality of reviews is very important to any conference community. The Bled Conference benefits from the efforts of c. 200 people who review papers each year. Although the conference is fortunate in having such a large community of committed scholars, and the quality of reviews is generally high, it is important that the quality be sustained and where necessary improved.
So reviewers are requested to ensure that their reviews reflect the guidance provided in this document.
The purpose of reviewing a paper is to assure quality, in both the negative sense of filtering out low-quality papers, and the positive sense of encouraging the Author(s) to deliver a better version of the paper in question, and better papers in the future.
What represents 'quality' differs between journals and conferences, and between conferences. The Bled community is special, because it comprises both academics and practitioners. The community wants contributions that make a difference, and whose significance they can grasp.
In short, the Bled Community is interested in papers that satisfy the following all-too-rare quality factors (bad wordplay intended):
|E ase of Access|
In brief, the Program Chair, the paper's Author(s), and the Bled Conference Community.
The foreground purpose is to provide the Program Chair with information about the paper's qualities, and its acceptability for the conference. The background purpose is to inform the Author(s) what needs to be done to improve the paper. All members of the Bled Community stand to gain from a professional approach to reviewing, because it enhances the quality of the papers that get accepted into the Program.
Here are the key characteristics of a review that meets the needs of the Program Chair, the Author(s) and the Bled Conference Community alike:
1. Demonstrated Understanding of the Paper. Briefly summarise the paper, or convey by other means that you've grasped what the author was trying to do. If the readers of a review aren't satisfied that the reviewer understood the paper, they will de-value the comments
2. Positive Features of the Paper. This should be brief, but it should be present. Criticism is more likely to be effective if it is cushioned by some recognition of the author's worth. It's unusual for a paper to reach a reviewer without at least some redeeming features
3. Critique, Expressed Constructively. This is the central feature: a good review must lead somewhere. The Program Chair needs to make a decision; and the Author needs suggestions about what they can do about the features the reviewer doesn't like. If your opinion is that the paper isn't appropriate for Bled, is there some other venue where it might find a home? Or are there steps that the author could take to improve their chances next time they submit a paper?
4. Value-Add. Performing a review is an act of collaboration, not competition. In addition to identifying and helping to eradicate inadequacies, offer additional perspectives and references
Here are some traps that reviewers need to avoid. The first few are corollaries of the four quality indicators discussed above:
• Don't be vague about the paper's weaknesses. It helps neither the Program Chair nor the Author to say that 'this paper makes too limited a contribution to be considered for this venue', 'the method is weak', 'the literature search is too narrow', or 'the implications are insufficiently developed'. Comments need to be specific, and supported by evidence
• Don't be emotive, aggressive, sarcastic or demeaning. This shakes the confidence of junior or otherwise insecure authors, and in any case reduces the effectiveness of the advice provided
• Don't criticise without 'actionable advice'. This term is Allen Lee's. He argues that a criticism shouldn't even be made unless it includes a suggestion about what might be done about it
Other issues are ethical in nature:
• Don't delegate unless you exercise control over the quality of the review. It is unfair to the Program Chair and the Author(s) to assign the task to a junior colleague or a student, unless you review their draft comments and the paper itself, adapt the draft as necessary, and take personal responsibility for the review that is submitted in your name
• Don't fail to declare a conflict of interest. Examples include an association with the research reported on, or with one of the authors of the paper. (This need not disqualify you from doing the review; but the relationship needs to be known)
• Don't fail to declare the limits of your expertise. A partial lack of expertise (e.g. limited knowledge of the research domain or the specific sub-area or the particular research method) does not disqualify you from doing the review (otherwise some papers would never find a reviewer!). But the scope of the problem needs to be declared to the Program Chair and Author(s) alike
• Don't take premature advantage of content. Prior to publication, don't provide copies to a colleague or student who works in the same area (unless you approach the Program Chair and get the Author's consent to early distribution)
Directly relevant articles and notes from within the IS discipline include:
Bieber M. (1997) 'How to Review'
Davison R.M., de Vreede G-J. & Briggs R.O. (2006) 'On Peer Review Standards For The Information Systems Literature' Commun. AIS 16, 49 (2005) 967-980
Drawing on the literature, Davison et al. (2006, p. 15) identifies attributes of a good reviewer as being humaneness, competence, openness, unbiassedness and absence of prejudice, ethicality, timeliness, persuasiveness and diligence
Koh C. (2003) 'IS journal review process: a survey on IS research practices and journal review issues' Infor. & Mngt 40 (2003) 743–756
Lee A.S. (1995) 'Reviewing a Manuscript for Publication' Invited Note in J. Ops Mngt 13, 1 (July 1995) 87-92
Zmud R. (1998) 'A Personal Perspective on the State of Journal Refereeing' MIS Qtly 22, 2 (September 1998)
A couple of relevant publications from other disciplines are:
Black N., van Rooyen S., Godlee F., Smith R. & Evans S. (1998) 'What Makes a Good Reviewer and a Good Review for a General Medical Journal' J. Am. Med. Assoc. 280, 3 (1998) 231-233
Finney D.J. (1997) 'The Responsible Referee' Biometrics 53, 2 (June 1997) 715-719
Hames I. (2007) 'Peer Review and Manuscript Management in Scientific Journals' Blackwell/ALPSP, 2007, from Blackwells
Smith, A.J. (1990) 'The task of the referee' IEEE Computer 23, 4 (April 1990) 65-71