The Bled Conference Guidelines for Paper Reviewers

Version of 31 July 2007, rev. 27 September 2024

© Xamax Consultancy Pty Ltd, 2006-07

The work is intended for copying and adaptation, subject to the liberal terms of the CC-BY licence

The licence requires attribution to this source, and display of and link to the CC-BY licence

Introduction

The quality of reviews is very important to any conference community. The Bled Conference benefits from the efforts of c. 200 people who review papers each year. Although the conference is fortunate in having such a large community of committed scholars, and the quality of reviews is generally high, it is important that the quality be sustained and where necessary improved.

So reviewers are requested to ensure that their reviews reflect the guidance provided in this document.

The Function of Reviewing

The purpose of reviewing a paper is to assure quality, in both the negative sense of filtering out low-quality papers, and the positive sense of encouraging the Author(s) to deliver a better version of the paper in question, and better papers in the future.

What represents 'quality' differs between journals and conferences, and between conferences. The Bled community is special, because it comprises both academics and practitioners. The community wants contributions that make a difference, and whose significance they can grasp.

In short, the Bled Community is interested in papers that satisfy the following all-too-rare quality factors (bad wordplay intended):

R elevance
A mbition
R igour
E ase of Access

Who is the Reviewer Working For?

In brief, the Program Chair, the paper's Author(s), and the Bled Conference Community.

The foreground purpose is to provide the Program Chair with information about the paper's qualities, and its acceptability for the conference. The background purpose is to inform the Author(s) what needs to be done to improve the paper. All members of the Bled Community stand to gain from a professional approach to reviewing, because it enhances the quality of the papers that get accepted into the Program.


Indicators of Quality in a Review

Here are the key characteristics of a review that meets the needs of the Program Chair, the Author(s) and the Bled Conference Community alike:

1. Demonstrated Understanding of the Paper. Briefly summarise the paper, or convey by other means that you've grasped what the author was trying to do. If the readers of a review aren't satisfied that the reviewer understood the paper, they will de-value the comments

2. Positive Features of the Paper. This should be brief, but it should be present. Criticism is more likely to be effective if it is cushioned by some recognition of the author's worth. It's unusual for a paper to reach a reviewer without at least some redeeming features

3. Critique, Expressed Constructively. This is the central feature: a good review must lead somewhere. The Program Chair needs to make a decision; and the Author needs suggestions about what they can do about the features the reviewer doesn't like. If your opinion is that the paper isn't appropriate for Bled, is there some other venue where it might find a home? Or are there steps that the author could take to improve their chances next time they submit a paper?

4. Value-Add. Performing a review is an act of collaboration, not competition. In addition to identifying and helping to eradicate inadequacies, offer additional perspectives and references


What Not To Do

Here are some traps that reviewers need to avoid. The first few are corollaries of the four quality indicators discussed above:

Don't be vague about the paper's weaknesses. It helps neither the Program Chair nor the Author to say that 'this paper makes too limited a contribution to be considered for this venue', 'the method is weak', 'the literature search is too narrow', or 'the implications are insufficiently developed'. Comments need to be specific, and supported by evidence

Don't be emotive, aggressive, sarcastic or demeaning. This shakes the confidence of junior or otherwise insecure authors, and in any case reduces the effectiveness of the advice provided

Don't criticise without 'actionable advice'. This term is Allen Lee's. He argues that a criticism shouldn't even be made unless it includes a suggestion about what might be done about it

Other issues are ethical in nature:

Don't delegate to a junior colleague or a student unless you are warranting to the Program Chair that the person is an appropriate reviewer, and that person submits in their own name (whether or not you review the draft prior to completion). [ Note: This item was amended to its current, improved expression in September 2024 ]

Don't fail to declare a conflict of interest. Examples include an association with the research reported on, or with one of the authors of the paper. (This need not disqualify you from doing the review; but the relationship needs to be known)

Don't fail to declare the limits of your expertise. A partial lack of expertise (e.g. limited knowledge of the research domain or the specific sub-area or the particular research method) does not disqualify you from doing the review (otherwise some papers would never find a reviewer!). But the scope of the problem needs to be declared to the Program Chair and Author(s) alike

Don't take premature advantage of content. Prior to publication, don't provide copies to a colleague or student who works in the same area (unless you approach the Program Chair and get the Author's consent to early distribution)


Some Examples of Good and Bad Content in Reviews

Bad

Good


Bibliography

Directly relevant articles and notes from within the IS discipline include:

Bieber M. (1997) 'How to Review'

Davison R.M., de Vreede G-J. & Briggs R.O. (2006) 'On Peer Review Standards For The Information Systems Literature' Commun. AIS 16, 49 (2005) 967-980

Drawing on the literature, Davison et al. (2006, p. 15) identifies attributes of a good reviewer as being humaneness, competence, openness, unbiassedness and absence of prejudice, ethicality, timeliness, persuasiveness and diligence

Koh C. (2003) 'IS journal review process: a survey on IS research practices and journal review issues' Infor. & Mngt 40 (2003) 743–756

Lee A.S. (1995) 'Reviewing a Manuscript for Publication' Invited Note in J. Ops Mngt 13, 1 (July 1995) 87-92

Zmud R. (1998) 'A Personal Perspective on the State of Journal Refereeing' MIS Qtly 22, 2 (September 1998)

A couple of relevant publications from other disciplines are:

Black N., van Rooyen S., Godlee F., Smith R. & Evans S. (1998) 'What Makes a Good Reviewer and a Good Review for a General Medical Journal' J. Am. Med. Assoc. 280, 3 (1998) 231-233

Finney D.J. (1997) 'The Responsible Referee' Biometrics 53, 2 (June 1997) 715-719

Hames I. (2007) 'Peer Review and Manuscript Management in Scientific Journals' Blackwell/ALPSP, 2007, from Blackwells

Smith, A.J. (1990)  'The task of the referee' IEEE Computer 23, 4 (April 1990) 65-71