Roger Clarke's Web-Site
© Xamax Consultancy Pty Ltd, 1995-2024
Infrastructure
& Privacy
Matilda
Roger Clarke's Web-Site© Xamax Consultancy Pty Ltd, 1995-2024 |
|
|||||
| HOME | eBusiness |
Information Infrastructure |
Dataveillance & Privacy |
Identity Matters | Other Topics | |
| What's New |
Waltzing Matilda | Advanced Site-Search | ||||
Final Version of 25 March 2012
Submitted for the 25th Bled eConference Special Issue
© Xamax Consultancy Pty Ltd, 2011-12
Available under an AEShareNet 
licence or a Creative
Commons 
licence.
This document is at http://www.rogerclarke.com/EC/Bled25TA.html
The Bled eConference is the longest-running themed conference associated with the Information Systems discipline. The themes addressed by the conference have migrated progressively from Electronic Data Interchange (EDI) via Inter-Organisational Systems (IOS) and eCommerce to encompass all aspects of the use of networking facilities in industry and government, and more recently by individuals, groups and society as a whole. The conference has contributed over 1,000 papers to the literature. It has supported the development of large numbers of graduate students, and many international research collaborations have been brought to fruition beside a Slovenian lake.
The Bled eConference was first held in June 1988 and has been held annually since then, always on the edge of the lake of Bled, beside the Julian Alps, in north-western Slovenia. It has been organised throughout by the Faculty of Organizational Sciences at the University of Maribor, located at nearby Kranj. It has attracted international participants since 1989, has been a fully international event since 1990, and has included a refereed research stream since 1995. The Proceedings include over 1,000 papers, with 773 since 1995 having been fully-refereed.
The information systems (IS) discipline can be reasonably regarded as having come into existence in 1965-67 (Clarke 2008c). In the IS discipline and associated research domains, only a very few conference series are of longer standing than Bled - HICSS, since 1968, the IS Research Seminar in Scandinavia (IRIS) - since 1978, and ICIS - since 1980. Two other major series emerged in the years following the establishment of Bled - the Australasian Conference in IS (ACIS) - since 1990 and ECIS - since 1993. All of those events, however, are generic IS conferences. So Bled is very probably the longest-running thematic conference series in or associated with the IS discipline. Hence an examination of the conference's themes and sub-themes is warranted by its historical and substantve significance, not merely for symbolic or celebratory reasons.
This paper reports on the results of an assessment of the topics addressed by papers that have been presented at the conferences and published in the Proceedings. It commences with a brief history, a depiction of the nature of the conference-series and community, and descriptive statistics. This is followed by analyses firstly of the themes addressed by papers during the first 24 years, and secondly of indicators of the conferences' impact.
This section briefly presents the origins of the series and the nature of the conference as a whole. It then provides data about the Research Stream since 1995, which is the primary focus of this paper.
Until the second half of 1991, Slovenia was part of Yugoslavia, and hence part of the Soviet bloc. The 1988 conference was local, but in 1989 it heavily involved three leading US academics, Milt Jenkins (University of Baltimore), Doug Vogel (then of the University of Arizona, subsequently of the City University of Hong Kong), and Don McCubbrey (University of Denver). The 1990 and 1991 conferences attracted increasing numbers of academics from Europe and beyond, particularly Australia.
The country's brief war of independence was fought a few weeks after the 1991 conference. From 1992 onwards, conference chair Joze Gricar created a focal point for many communities of interest. EDI in international trade, including in ports and in Customs, was complemented by the application of electronic tools in local and national governments, in regional economies, and along trade routes. Until 1994, plenary sessions featured parallel translations between English and Slovene; but Slovenes are multi-lingual, and the conference language has been English since then. Through the 1990s, the Bled eConference played a significant role in the process of Slovenian democratisation, economic change, and entry as a full member of the European Union.
A distinctive feature of the conference, throughout its history, has been the successful marriage of Industry and Research Streams. Academics from around the world have always found themselves rubbing shoulders with Eurocrats from Paris and elsewhere, and IT and management professionals from industry and government in Slovenia and the nearby region. The conference has also had several other innovative features, which are discussed below.
As the scope of 'electronic' moved beyond data interchange (EDI), and as communications matured from private and value-added networks (VANs) to the Internet, and as the 'e' prefix was prepended to commerce, and then to government, and to services, and to publishing, and onwards to voting and democracy, Gricar moved the Bled conference with it. The community of Bled participants expanded, and the countries whose academics contributed to the conference multiplied.
The Conference has always been held under the auspices of the Faculty of Organizational Sciences of the University of Maribor at Kranj. It has enjoyed strong support from successive Deans, from the University as a whole, from successive Slovenian governments, and from the European Commission. Faculty-members, staff and senior students have invested a great deal of time, effort and imagination into the planning, preparation, logistics and management of the conference, in 25 successive years, so far. The corporate sponsors include several that have been supporters over an extended period of time, including, in alphabetical order, Adria Airlines, Gorenski Glas, IBM, Microsoft, Oracle, the Slovenian Chamber of Commerce, and Union Beer.
Chairmanship of the Conference has always been performed by a senior academic of the Faculty. Gricar was founder and Conference Chair from 1988 to 2008. Commencing in 2009, the most recent four conferences have been chaired by Andreja Pucihar.
The Conference has featured an Industry Stream throughout its life, in recent years described as a Business and Government Panel Track. Papers with a research orientation were included from the beginning, and a community of academics developed who welcomed the interleaving of research, professional and executive perspectives on electronic communications.
In 1995, the Research Stream was formalised into a fully refereed conference. The c. 250 formal papers published between 1988 and 1994 have been succeeded by 773 refereed papers during the 17 years 1995 to 2011 and will be joined by 42 more at the 25th conference in June 2012.
For many years, the style was that of a community symposium, intended to be inclusive. For the last 5 years, since 2008, the threshold for acceptance has been raised, and the format tightened to 14 sessions of 3 papers each, with a maximum of 2 parallel research sessions, complemented by several academic panel sessions, a Student ePrototype Bazaar, a Graduate Student Consortium, plenaries and 2-4 parallel industry sessions.
The policy was adopted at the outset to seek diversity in the Chairmanship of the Research Stream. The Stream Chair is appointed two years ahead, so that the current year's chair has a colleague working with them. This has had the desirable effect of achieving continuity in flavour, style and values, while ensuring that changes in the conference community's interests have been detected and reflected. During the 18 years, 16 different individuals have performed the Program Chair role, 9 of them male and 7 female. See Appendix 1. The Chairs' affiliations at the time have been with universities in 9 different countries. Of the 19 occasions 1995-2013, the Chair has been provided 10 times by a European country (5 times by Germany and once each by Greece, the UK, the Netherlands, Finland and Switzerland), with 4 from the USA, 4 from Australia, and 1 from Hong Kong.
There has been a pronounced trend away from sole-authored papers (from 48% down to 14%) and towards teams of 3 or more authors (12% to 48%). Across the years, 21% of papers have had a sole author, 42% have been dual-authored, 26% have had three authors, 8% have had four authors, and 3% (25 papers) have had 5, 6 or 7 authors. Of the 13 papers that have won the Outstanding Paper Award to date, only two have been sole-authored, and they were very early in the Award's history.
Authorship and community participation have had a strongly international flavour since 1990, and particularly so since 1995. Exhibit 1 summarises the national and institutional affiliations of authors during the period. On average, 17 countries have been represented on each program (range 11 to 23), together with 55 universities (range 24 to 94), with a further 9 authors having an affiliation other than a university (range 2 to 19).
Papers | Authors | Countries | Universities | Other Affiliations | |
1995 | 25 | 41 | 11 | 26 | 2 |
1996 | 32 | 70 | 12 | 24 | 3 |
1997 | 33 | 79 | 15 | 41 | 9 |
1998 | 43 | 105 | 15 | 58 | 4 |
1999 | 44 | 92 | 16 | 48 | 4 |
2000 | 48 | 119 | 21 | 61 | 5 |
2001 | 50 | 109 | 22 | 62 | 16 |
2002 | 49 | 119 | 23 | 57 | 13 |
2003 | 71 | 151 | 21 | 94 | 6 |
2004 | 52 | 114 | 20 | 66 | 13 |
2005 | 45 | 105 | 17 | 60 | 8 |
2006 | 52 | 112 | 20 | 55 | 11 |
2007 | 60 | 148 | 20 | 76 | 11 |
2008 | 45 | 89 | 13 | 50 | 16 |
2009 | 42 | 96 | 15 | 61 | 9 |
2010 | 41 | 92 | 14 | 57 | 4 |
2011 | 42 | 108 | 15 | 39 | 19 |
A total of 41 countries have been represented on the program during the 17 years. At least 10 further countries have been represented on the delegates lists, taking the total past 50 countries in all. Authors' countries of affiliation have included 26 European countries, including all Western and Central European countries except for 4 very small republics and 2 islands. Many Eastern Europe countries have also contributed. The USA, Canada, Australia and New Zealand have been strongly represented. From East Asia, 7 nations have been represented, plus India and Oman. To date, however, the only African nations on the program have been South Africa and Tunisia. The national affiliations that can be seen most often among the almost 1800 authors' names on the 773 papers have been Australia 389 times (22%), The Netherlands 260 (15%), Germany 234 (13%), UK 100, Finland 95, USA 71, Ireland 68, Greece 66 and Switzerland 61 times.
A further important international and multi-cultural element has been the diversity of graduate students who have participated. René Wagenaar brought students with him for conference experience as early as 1992, and invited students from the University of Maribor back to his then institution, Erasmus University in Rotterdam. Paula and Paul Swatman in particular brought many students across the world from Australia during the following years. Similarly, Niels Christian Juul brought students from the Copenhagen Business School. There were also fruitful collaborations among students from the University of Maribor with US universities, especially Baltimore, and with Australian universities, including Deakin.
This section documents the themes that the Bled eConference has addressed, commencing with the overall conference titles, and then delving down into content analysis of paper titles, abstracts and full-text. There is only a limited IS literature relating to thematic analysis of conference-series, but see Galliers & Whitley (2002, 2007).
The broad themes that the series has encompassed are shown in Exhibit 2, and Appendix 2 lists the title of each conference.
Title | Years | No. of Conferences |
| Electronic Data Interchange (EDI) | 1988-1992 | 5 |
| EDI and Inter-Organizational Systems | 1993-1995 | 3 |
| Electronic Commerce | 1996-2004 | 9 |
| eConference | 2005-2012 | 8 |
The conference's initial focus was on Electronic Data Interchange (EDI). This was an endeavour to achieve reliable and efficient batch transfer of transaction data between organisations. After five years, the title was broadened to encompass Inter-Organizational Systems (IOS) as a whole. This period coincided with the liberalisation of access to the Internet. During the following three years, networked applications broadened in scope, and hence the nine conferences between 1996 and 2004 ran under variants of the term Electronic Commerce. By 2005, electronic applications were moving well beyond organisational boundaries to involve individuals at one level, and societies at another. To accommodate this, the most recent eight conferences have been under the catch-phrase 'eConference'.
Tag-clouds are tools intended to assist in visualising the frequency of words within a collection of text. Utilising the tag-cloud approach, the Conference Organisers produced the image in Exhibit 3 as a way of encapsulating the intended scope of the Bled eConference 1988-2012.
Manual analysis of the paper titles during the early, unrefereed period, 1988-94, showed that the term 'EDI' dominated, although 'commerce' was emergent during the early 1990s. This sub-section reports on a series of studies of the terms in the titles and abstracts of the papers published during the refereed period, 1995-2011. The source materials are in Appendix 3.
In Exhibit 4, the tag-clouds are displayed for all titles during the period since the Research Stream became fully refereed. The number of occurrences of the string in the titles of the 773 papers is shown in the tag-cloud. The cut-off was arbitrarily set at 15 occurrences - in order to achieve inclusion of the terms 'government' and 'consumer'.
The results show the strengths and weaknesses of the tag-cloud notion. There is a need to cope with phrases (e.g. 'electronic commerce' and 'social media'), to declare equivalence (ecommerce, e-commerce, electronic commerce), to exclude generic terms (e.g. 'information' and 'system'), and to allow for multiple usages of the same term (such as 'model' and 'framework').
Between them, the terms business (94), commerce (84), industry (42), SMEs (29) and enterprises (23) total 272 uses of tags related to organisational interests. This has completely dominated government (16), but also customer (23), public (16), consumer (15) and social aspects (19), which total only 89 - although the term services (59) does extend across both sectors. The counts for the terms Internet and mobile were very similar (both 60). As the following analysis confirms, however, the periods during which they were of major interest were different.
It would be challenging to grasp change across a set of 25 or even a dozen such clouds. In order to reduce the data to a manageable scale, the proceedings since 1995 were grouped into 3-year sets. The tagcrowd.com site was used to display the most-frequent 20-30 terms in the titles within each group. The resulting tag-clouds are provided in Appendix 4. The primary tags identified among the 773 refereed papers during the refereed period, 1995-2011, are summarised in Exhibit 5. Generic terms such as management, system and framework were ignored. The peak-years for each tag are highlighted using bold-face type.
Tag 1995-96 1997-99 2000-02 2003-05 2006-08 2009-11
EDI 23 9 . . . .
electronic 21 68 32 16 9 7
commerce 15 38 24 12 . .
business 8 17 11 28 22 8
Internet 4 17 20 9 . .
case 3 9 16 17 15 9
Web . . 10 . . 7
mobile . . 9 20 23 8
trust . . 10 11 . .
online . . 11 14 . .
ebusiness . . 9 15 . .
eCommerce . . 6 12 . .
SME . . 12 7 8 .
adoption . . . 15 7 9
health . . . . . 9
social . . . . . 15
An analysis of the papers' Abstracts was undertaken, in order to produce a set of tag-clouds. The resulting tag-clouds are provided in Appendix 5. There proved to be little new information in the tag-clouds for the Abstracts in comparison with those for the Titles in Appendix 4. One important reason is that the process counts occurrences of the word in the entire set, not occurrences of instances of Abstracts in which the word occurs. For many purposes, this constitutes noise rather than information.
To complement the automated approach, manual analyses were undertaken. These suffer the disadvantages of unreliability, observer bias and unrepeatability. There is a key advantage over the automated approach, however. This is the ability to utilise knowledge of the IS discipline and the relevant research-domains in order to filter out common words, to cope with multi-word terms, to recognise approximate synonyms, and to distinguish distinct uses of the same term.
Consideration was given to using an existing classification scheme. Candidates include those of Barki & Rivard (1993) and Banker & Kauffman (2004). Such schemes are, by intent, comprehensive, and therefore relevant to an assessment of a generic venue such as ECIS (Galliers & Whitley (2002, 2007). On the other hand, the Bled conference is themed. In particular, a large proportion of papers would inevitably be classified under Banker & Kauffman's and Galliers & Whitley's 'Electronic Markets' category.
No readily-available set of sub-categories within 'Electronic Markets' was identified. In any case, the topics at Bled conferences have been dynamic, with noticeable changes year by year - as demonstrated by the tag-analysis in Exhibit 5 above. The analysis was therefore undertaken without the benefit, and constraint, of a pre-determined set of categories.
Visual inspection was undertaken of the titles of the 773 papers in the refereed Proceedings for the period 1995-2011. A total of 63 keywords were identified, with 972 mentions, for an average of 1.25 keywords per paper. The keywords were then clustered, informally, on the basis of the author's familiarity with the subject-matter and his particular world-view. No authoritative basis for the clustering is, or can be, claimed. This gave rise to 34 keyword-clusters within 3 major groups, and provided the basis for the detailed counts in Appendix 6.
The 15 clusters with the largest number of papers are listed in Exhibit 6, together with the periods in which they they were most popular. Almost 60% of keyword-occurrences were Categories of eBusiness, with EDI giving way early in the period to eCommerce - which was easily the largest cluster-count, with 14% of the total. That in turn gave way to eGovernment, and then Mobile, with marketing and then health dominating very recently. Corporate Perspectives accounted for 22% of keyword-occurrences, with interest focussed on IOS, then supply chain, and then business models and business strategy factors. The remaining 19% of keyword-occurrences encompassed a range of Research Topics, of which the adoption and trust clusters were the most prominent.
Keyword-Clusters Main Period No. of
Mentions
Categories of eBusiness 571
EDI 1995-1998 41
eCommerce 1996-2001 140
eMarkets, Directories, Auctions 1998-2002 51
eGovernment 2004-2008 35
SMEs 1998-2002 53
MCommerce, Mobile Apps 2002-2009 56
eMarketing, CRM, Consumer Behaviour 2003-2011 55
eHealth 2006, 2011-11 32
Other (8 clusters) 108
Corporate Perspectives 216
Inter-Organisational Systems 1995-1998 24
Supply Chain, ECR, Intermediaries 1998-2003 37
Business Models 2003-2005, 2009 29
BPR, Transformation, alignment, integration 2003-2007 42
Strat Alliance, Bus Networks, Virtual Orgns 2004, 2007-2009 37
Other (4 clusters) 47
Research Topics 185
Adoption, Impediments, Success Factors 2001-2007 52
Trust, Reputation, Risk 2001-2004, 2010-11 37
Other (7 clusters) 96
The effort involved in manual inspection of the papers' Abstracts was well in excess of the available resources, and was not undertaken.
The full-text of papers from 1988 to 1997 were not available in machine-readable format. This includes the first three refereed conferences. Searches by keyword-in-fulltext were feasible on the Bled eConference site for 1998-2011, and in the AIS eLibrary for 2001-10.
Content analysis of the full-text was desirable, but beyond the time and resources available during the preparation of this paper. Some preliminary experiments were performed, however. For example, a search using <smartphone> identifies 7 papers, 1 paper in each of 2005-11 except for 2008 (0) and 2011 (2), i.e. 1, 1, 1, 0, 1, 1, 2.
A search using <Facebook> indicates a surge in interest. It detects the first paper in 2007, followed by 4 in 2008, 4 in 2009, 8 in 2010 and 12 in 2011, i.e. 1, 4, 4, 8, 12. This represents 10%, 10%, 19% and a remarkable 28% of the papers in the last four conferences.
A search using <Google>, followed by inspection of the papers' titles, gave rise to the following inferences:
It appears that particular services may be of interest to Bled authors only during a brief window, and that window may be some time after the services have been launched and achieved market penetration.
The period of currency of now-expired or passé products, standards and memes can also be analysed. For example, Digicash had two mentions in 1998 and 3 in 1999, plus 1 retrospective mention in each of 2003 and 2006. ebXML peaked in 2002-03 (3 and 5 papers), but also had mentions earlier, in 2000-01 (1 each), and later, in 2004 (1), 2006 (3) and 2008 (1). EDIFACT was an important topic prior to the period available for full-text analysis, but interest in it fell after 1998-99 (7 and 10 papers), with some mentions every year until 2010 (3, 3, 3, 4, 3, 2, 4, 1, 2, 1, 1), but - probably for the first time ever - none in 2011.
These experiments have confirmed that undirected analysis by means of visual examination is very challenging and time-consuming, and the results are likely to be subject to considerable variation depending on the researchers and their preoccupations and motivations. A more promising approach may be computer-supported analysis, such as that conducted by Heinz Dreher, and intended for inclusion in this Special Section.
This section utilises available sources to gauge the impact that the conference has had. The first sub-section identifies a variety of channels through which the conferences have had an impact on the ecommerce and related research domains. The second sub-section exploits the few available metrics.
A conference-series makes contributions to the discipline, to the profession, and to individuals, and Bled has done so through multiple channels.
From 1988 to 2000 (13 years), the Proceedings were published in hard-copy form only. In some years, documents relating to the parallel Industry Stream were published in the same Volume, and in some years as a separate Volume. Since 2001, the Proceedings have been published on CD in PDF format (12 years). The 11 years 2001-11 are all also accessible online, in two locations, open-content licensed from the conference web-site, and behind a subscription-based paywall in the AIS eLibrary.
Program Chairs have worked with the authors of selected papers to produce Special Sections in leading journals. This has resulted in 53 of the 773 papers (7%) being subsequently upgraded to journal level.
During 1995-2006, a collaboration with the leading International Journal of Electronic Commerce resulted in 9 Special Sections and 36 papers. The journal Electronic Markets published a Special Section of 3 papers developed from the 1999 Bled conference, and since 2007, a collaboration with that journal has resulted in a further 4 Sections and 14 papers to 2010, with another scheduled for 2012. The issues and papers are identified in Appendix 9, and access to indexes of the Sections is facilitated from the Research Stream web-page.
Many Bled papers are known to have been improved by their authors and submitted elsewhere (not least because some authors have declined the invitation to submit revised versions for Conference-derived Special Sections). A reasonable speculation might be that articles developed from an additional 40-80 conference papers (5-10%) have been published in journals.
PhD Consortia have gradually become mainstream features of a range of conferences. The Bled Graduate Student Consortium, led by Milt Jenkins, commenced in 1990, and ran for over a decade until 2002, supported by the Slovenian company KRKA. It contributed greatly to the development of early-career researchers. The Consortium series was revived in 2011 under Doug Vogel.
Another conference feature that has benefited generations of graduate students is the Student ePrototype Bazaar. This combines a 'trade fair' and 'poster session', and enables both graduate and advanced undergraduate students to present their prototypes and discuss their ideas with conference participants. This ran from 1997 to 1999 in the Slovene language only, and has run annually since 2000 in English. It has been named for René Wagenaar since 2007, following his premature death. Its coordinators in recent years have been Harry Bouwman from Delft University of Techology and Johan Versendaal from the University of Applied Sciences Utrecht.
Research collaboration has been a feature throughout Bled's history, with many research-teams formed, and many projects conceived and developed beside the lake. The conference has also provided a convenient meeting-venue for many existing project teams, and for Editorial Boards, in particular of the journal Electronic Markets.
In addition, many papers presented at the Conference have featured authors from multiple institutions. As shown in Exhibit 7, on average 11 papers at each conference (c. 25%) have been multi-institution, with a range from 4 to 21 (13%-36%).
The international nature of the Research Stream is confirmed by the fact that about two-thirds of the multi-institutional teams have involved authors from different countries (c. 15%), with a range from 1 to 13 (4%-26%). Within international authorial teams, enormous diversity is evident. Authors have come from 30 different countries, most commonly Germany (53 times), Australia (33), USA (21), The Netherlands (18), Switzerland (17), UK (17), Ireland (12) and Denmark (10). The contributions to an appreciation of cultural differences across economies, societies and language groups has been enormous.
Papers | Papers with Authors in Different
Institutions | % | Papers with Authors in Different
Countries | % | ||||
1995 | 25 |
4
| 16 |
2
| 8 | |||
1996 | 32 | 5 | 16 | 1 | 3 | |||
1997 | 33 | 11 | 33 | 6 | 18 | |||
1998 | 43 | 13 | 30 | 7 | 16 | |||
1999 | 44 | 7 | 16 | 4 | 9 | |||
2000 | 48 | 12 | 25 | 7 | 15 | |||
2001 | 50 | 13 | 26 | 11 | 22 | |||
2002 | 49 | 10 | 20 | 2 | 4 | |||
2003 | 71 | 21 | 30 | 11 | 15 | |||
2004 | 52 | 13 | 25 | 8 | 15 | |||
2005 | 45 | 12 | 27 | 9 | 20 | |||
2006 | 52 | 7 | 13 | 6 | 12 | |||
2007 | 60 | 15 | 25 | 13 | 22 | |||
2008 | 45 | 8 | 18 | 6 | 13 | |||
2009 | 42 | 15 | 36 | 11 | 26 | |||
2010 | 41 | 14 | 34 | 9 | 22 | |||
2011 | 42 | 11 | 26 | 5 | 12 | |||
Avge | 45 | 11 | 24 | 7 | 16 |
The conference has always valued quality papers. The first concrete step taken to encourage and assure ongoing improvements in quality was the move to full refereeing of submitted papers, in 1995. The program was expanded in some years, with a particular aim of assisting authors from countries with a less well-established research tradition to gain experience in the conduct, preparation and presentation of research. In recent years, however, the threshold for acceptance has been raised, by limiting the available slots, resulting in acceptance rates in the range of 30% to 50%.
Two further measures have been adopted to encourage quality in papers submitted to the Conference.
One was the establishment of an Outstanding Paper Award. Since 1999, the opening session at each conference has re-affirmed the criteria applied in selecting the winner, as a means of signalling the desirable characteristics of Bled papers. The winners of the first 13 awards, 1999-2011, are listed on the conference web-site and in Appendix 7. The authors' affiliations are irrelevant to the decision made each year, but considerable diversity has been evident, with the 40 authors of the first 13 award-papers having their affiliations in 9 different countries.
The other initiative has been the provision of guidelines for reviewers, formalised in Clarke (2007). These stress the importance of such aspects as clarity, constructive tone, and 'actionable advice'. Especial weight is placed on the quality factors that are important in papers for a conference that is instrumentalist in its orientation and features both research and industry streams:
There are few readily-available ways to measure the impact of even a single conference, let alone a long series of them. This sub-section presents data from two analyses, and considers a third, experimental approach.
A conventional approach to impact measures is to count the number of citations each paper receives. Citations may be subject to more or less strict rules regarding the nature of the venue in which the citation arises, e.g. any publication, papers in refereed venues only, or papers in selected refereed venues only (Clarke 2008a, 2008b).
The Thomson/ISI/Web of Science collection has long been unfriendly to the IS discipline, and, although the inherent bias has recently been relaxed, it has been omitted from the study. Google Scholar is much more inclusive than Thomson, and although its catchment is somewhat indiscriminate, it is a far more useful tool for the present purpose.
Google Scholar was accessed in June 2011, and again in November 2011, December 2011 and March 2012. The 'permanent beta' nature of Google generally, and Google Scholar in particular, was evident from the analyses that were conducted. There appear to have been further increases in Google's catchment during the second half of 2011, although this may have stabilised by the end of the year. The service is highly unstable, only limited documentation is available, repeats of experiments seldom produce the same results, and search methods have to be continually changed in order to re-discover references that had previously been found. The level of confidence in the following comments is therefore only moderately high.
In June 2011, Google Scholar showed about 2,300 citations of Bled eConference papers. See Appendix 8. By November, this had grown to about 2,900 citations, by December 2011 to around 3,250 citations, and by March 2012 to about 3,400. After allowing for increases in Google's catchment, the accretion-rate from newly-published papers would appear to be about 400 p.a. or about 10-15% p.a.
In common with academic papers generally, most papers achieve very few citations, and only a small percentage receive an appreciable number. About 230 of the 773 papers (30%) appeared to have achieved at least 1 citation. The top 3 papers in March 2012 showed citation-counts of 644, 421 and 109, or about one-third of the total. A further 8 papers had counts in the range 40-65, 30 in the range 20-40 and 40 in the range 10-20. The top-50 list was heavily biased towards older papers, c. 1998-2004.
Based on a comparative analysis conducted on June 2007 data, Clarke (2008a) suggested that, at that time, 500 citations indicated a 'classic' article, 75 a 'high-impact article', and 40 a 'significant-impact article'. Growth has to be allowed for since then, from both subsequently-published articles and increases in Google Scholar's catchment area. In order to assess Bled conference papers against these scales, updated thresholds of 600, 90 and 50 were postulated, and it was assumed that the patterns for the eCommerce research domain are broadly comparable with those for the IS discipline.
On that basis, 1 Bled conference paper was just in the 'classic' range (> 600 citations), 1 was high in the high-impact range (> 400), 1 was low in the high-impact range (> 90), 3 were in the 'significant-impact' range (> 50), and a further dozen papers had at least 35 citations. On one hand, this appears low. On the other, the IS discipline and many other disciplines active in the eCommerce research domain have a strong tendency to cite articles published in journals rather than in conference proceedings.
An analysis was undertaken of the Google citation counts of the 53 papers that originated as Bled conference papers, but revised versions of which were published in 9 Special Sections in the International Journal of Electronic Commerce (IJEC) and 5 Special Sections in Electronic Markets (EM). See Appendix 9. In August 2011, this showed a total of 1,875 further citations. Only the top 3 papers had achieved the (arbitrary) threshold of 90 for a high-impact article, with 285, 200 and 134 citations. A further 8 of the 53 had achieved the 'significant-impact' threshold of 50. (A repeat search for a sample of the papers in December 2011 showed that they had gained 20-25% in their citation-counts during the intervening 4 months). The most recent paper in the top 3 is from 2006. This is consistent with the notion of an indicative 5-year lag before an article accumulates an appreciable list of citations.
In only 6 of the 53 cases have the conference papers underlying the journal articles attracted 10 or more citations, for 110 citations in total. In one case, the journal article has 11 citations and the conference paper 20, but in all others the journal article has the higher count. The first-ranking article shows 285 citations for the journal article but only 2 for the conference paper; whereas the second-ranking journal article has 200 citations compared with 33. One factor in this relationship may be the relatively short exposure of the conference paper alone, which has varied between 1 and 2 years.
The second readily-available metric is the Download Count from the AIS eLibrary (AISeL). This service had only been available for 3 years at the time the statistics were extracted in mid-2011, and only the papers for 10 of the 24 conferences had been loaded at that stage. The papers therefore reflect between 9 months and almost 3 years of downloads. The total to that date was about 10,000 downloads.
Across the whole of the AIS eLibrary, the top 1% of papers were downloaded 352 or more times and the top 5% were downloaded 167 or more times. An arbitrary cut-off-point of 50 downloads was selected, in order to focus attention on those papers that, on this basis, were the most impactful. This identified 40 papers, of which the the top 2 are in AISeL's top 1%, with 693 and 519 downloads, and the next 3 are within its top 5%, with 269, 247 and 169 downloads. See Appendix 10.
As shown in Exhibit 8, there is a scattered pattern across the 10 years 2001-10. All but 8 of the 506 papers had been downloaded at least once, and the average was almost 20 downloads per paper. The average download per paper across the whole of the AIS eLibrary collection is almost 47; but many others have been accessible for much longer periods than have the Bled Proceedings, and journal articles generally average considerably higher download-counts than conference papers.
| Year | Date Loaded | Months Available | Downloads |
| 2001 | Aug 2008 | 34 | 1,110 |
| 2002 | Aug 2008 | 34 | 442 |
| 2003 | Aug 2008 | 34 | 1,782 |
| 2004 | Aug 2008 | 34 | 574 |
| 2005 | Aug 2008 | 34 | 745 |
| 2006 | Feb 2009 | 28 | 536 |
| 2007 | Sep 2009 | 21 | 1,247 |
| 2008 | Sep 2009 | 21 | 814 |
| 2009 | Sep 2009 | 21 | 1,844 |
| 2010 | Oct 2010 | 9 | 684 |
| TOTAL | 9,778 |
Interestingly, the two stars from the Google Scholar Citation Count study (both of which were published in 2002) have had only modest downloads from the AIS eLibrary. The third-ranking citation-count paper (published in 2003) is the highest-ranking of any Bled paper based on AIS eLibrary downloads. The fourth-, fifth- and sixth-ranking papers by citation-count again do not score highly on AIS eLibrary downloads. From the other perspective, the second-, third- and fourth-ranking by AIS eLibrary downloads (published in 2003, 2009 and 2009) do not score highly on citation-count.
An inference that might be drawn is that the two metrics are measuring impacts of quite different kinds. Citations counted by Google Scholar are indicative of use by researchers, whereas AIS eLibrary downloads may be predominantly for teaching purposes rather than research. Alternatively, the long lag-time from research to publication may explain some of the variance. In that case, citations counts for Bled papers could be expected to lift during 2012-15. Another possibility is that older papers' time may have passed, and yet another that the the 'citation stars' are valued, and cited, by people outside the IS discipline.
Papers are accessible in a variety of repositories, including:
The third and fourth of these present an additional possible impact metric: hit-counts on employer and personal web-sites. Few university repositories appear to publish hit-counts. This section accordingly focusses on authors' own web-sites.
Authors have the legal capacity to publish copies of the vast majority of their research papers. In some cases, they own, or co-own, the copyright in their papers. In other cases, they have an at least implied licence, and in many cases, as a result of the open access movement, an explicit licence from the copyright-owner to do so (Clarke & Kingsley 2009).
On the one hand, many authors may regard the reticulation of their articles as being the responsibility of other parties, or they may see the establishment and maintenance of a rich personal web-site as being too onerous. On the other hand, many mid-career researchers have a substantial corpus of works, and both mid-career and early-career researchers with ambition seek opportunities to promote their work and their capabilities.
The author was an early mover in the personal web-site arena, having launched his site in 1994-95. Since then, the site has accumulated over 35 million hits. The author has had 16 papers published in 15 of the 24 Bled conference Proceeedings to date, all of them also published on his web-site. An experimental investigation was undertaken, seeking an understanding of the impact of self-publication of scholarly papers on authors' own web-sites. The raw data is provided in Appendix 11, and interpretations of the data are in Appendix 12. These show that the author's repository of 16 Bled papers gains well in excess of 10,000 hits p.a., and has a cumulative hit-count (excluding hits by search-engine robots) of over 230,000, with several individual papers well in excess of 20,000.
For reasons discussed in Appendix 12, great care is needed in drawing inferences from the data. However, it is clear that the number of hits in the personal web-repository is far greater than the download-counts within the AIS eLibrary - by 15-20 times. The sources of the hits include not only accesses by researchers but also by enrolled and informal students, and by people in business and government. Depending on the intended audience, the value of accesses by people other than academic researchers may need to be heavily discounted - although less so in the case of a venue with avowedly instrumentalist intent such as the Bled eConference.
This single-case study has shown that at least these 16 of the 773 Bled papers have had considerable additional impact through exposure on a personal web-repository. It is unclear how prevalent personal web-sites are among IS researchers in general and Bled authors in particular. In order to provide some insight into the extent to which Bled authors operate their own web-repositories, searches were conducted in relation to the authors of the 'top five' Bled papers as measured by Google citations for conference papers, by Google citations for journal articles developed from Bled conference papers, and by AIS Downloads. Mere lists of publications, and even access to Abstracts, were excluded from the count. The sample is intentionally biased, in that it comprised successful authors - and in most cases senior authors. Such authors would appear to be more likely than others to maintain their own repositories - although contrary factors exist, such as the likelihood that younger authors are more technically-capable and that they perceive the need to try hard to gain exposure. The results are summarised in Appendix 13.
Of the 29 authors in the sample, 7 were found to operate repositories that were readily located and that contained copies of at least a considerable proportion of each author's papers. In all cases, they were pages or sub-sites within a university domain, which could be expected to have a moderately high page-rank with search-engines. Given that the papers would rank reasonably highly in searches, these authors' Bled papers may have gained considerable numbers of downloads from the authors' own web-repositories.
This paper's first purpose was to summarise the nature of the Bled eConference. It is an international event, located in an Eastern European member-state of the European Union. Its organisers have actively pursued European themes, but have also attracted consistent participation from elsewhere, particularly Australia and North America. A great deal of collaborative activity has been stimulated and supported, across institutions, and across national borders.
The second purpose was to present analyses of the themes that have been nominated by Program Chairs and that have been pursued by authors. Syntactic analysis using both manual and simple electronic tools has identified the emergence, rise and fall of many topic-areas. The early focus on EDI and then eCommerce has been progressively replaced by a much broader of applications of electronic tools. Similarly, the early dominance of business interests and economic dimensions has been progressively complemented by a still-growing emphasis on the social dimensions.
Experimentation showed that it is already feasible to apply automated full-text word and string searching to support specific purposes, such as establishing the extent to which a particular topic has been previously addressed in the Bled literature. It appears that more generalised tools with greater capacity for semantic analysis may be emerging. These capabilities augur well for the gradual, if greatly belated, emergence of a 'cumulative tradition' (Keen 1980).
A third intention was to evaluate the conference's impacts. A considerable number of channels has been identified through which the first 25 years of the conference have had impacts on research, on researchers, on students, and on business and government. Over 5,000 citations of the 773 papers and 53 journal articles published to date have been supplemented by over 10,000 downloads from the AIS repository, and hundreds of thousands of hits on the papers in the authors' own repositories. The strategies of open-content licensing and multiple channels ensure that these counts will continue to grow.
The mechanics of citation- and download-counts are of significance. On the other hand, the Bled eConference is above all a community. Generations of participants have migrated from graduate student, to early-career academic, to senior member of the profession. They have considered a broad range of information technologies applied to business, government and society, have appreciated cultural differences, and have developed international linkages and collaborations. The first 25 years have delivered enormous personal and community value, and laid the foundation for the next decades of change.
Banker R.D. & Kauffman R.J. (2004) 'The Evolution of Research on Information Systems: A Fiftieth-Year Survey of the Literature in Management Science' Management Science 50, 2 (March 2004) 281-298
Barki, H. & Rivard S. (1993) 'A Keyword Classification Scheme for IS Research Literature: An Update' MIS Quarterly 17,2 (June 1993)
Clarke R. (2007) 'The Bled Conference Guidelines for Paper Reviewers' Xamax Consultancy Pty Ltd, July 2007, at http://www.rogerclarke.com/EC/BledRevGuide.html
Clarke R. (2008a) 'An Exploratory Study of Information Systems Researcher Impact' Commun. AIS 22, 1 (January 2008 ) 1-32, PrePrint at http://www.rogerclarke.com/SOS/Cit-CAIS.html
Clarke R. (2008b) 'A Citation Analysis of Australian Information Systems Researchers: Towards a New ERA?' Australasian Journal of Information Systems 15, 2 (June 2008), at http://dl.acs.org.au/index.php/ajis/article/view/465/436
Clarke R. (2008c) 'A Retrospective on the Information Systems Discipline in Australia' Chapter 2 of Gable G.G. et al. (2008) 'The Information Systems Discipline in Australia' ANU e-Press, 2008, pp. 47-107, at http://epress.anu.edu.au/info_systems_aus/pdf_instructions.html, PrePrint at http://www.rogerclarke.com/SOS/AISHist.html
Clarke R. & Kingsley D.A. (2009) 'Open Access to Journal Content as a Case Study in Unlocking IP' SCRIPTed 6, 2 (August 2009), at http://www.law.ed.ac.uk/ahrc/script-ed/vol6-2/clarke.pdf
Galliers R.D. & Whitley E.A. (2002) 'An Anatomy of European Information Systems Research - ECIS 1993 - ECIS 2002: Some Initial Findings' Proc. ECIS 2002, June 2002, Gdask, Poland, PrePrint at http://personal.lse.ac.uk/whitley/allpubs/ECIS2002.pdf
Galliers R.D. & Whitley E.A. (2007) 'Vive les differences? Developing a profile of European information systems research as a basis for international comparisons' Euro. J. of Infor. Syst. 16, 1 (2007) 20-35
Keen P.G.W. (1980) 'MIS Research: Reference Disciplines and a Cumulative Tradition' Proc. 1st Int'l Conf. Infor. Syst. (ICIS), Philadelphia, 1980
Appendix 1: Research Stream Chairs (RTF format)
Appendix 2: Conference Titles (RTF format)
Appendix 3: Paper Titles, Authors and Abstracts 1995-2011 (XLS format)
Appendix 4: Tag-Clouds for Bled Paper Titles 1995-2011 (.doc format)
Appendix 5: Tag-Clouds for Bled Paper Abstracts 1995-2011 (.doc format)
Appendix 6: Keyword-in-Title Analysis 1995-2011 (XLS format)
Appendix 7: Outstanding Paper Award Winners (RTF format)
Appendix 8: Google Scholar Citation Counts (RTF format)
Appendix 9: Journal Article Citation Counts (RTF format)
Appendix 10: AIS eLibrary Downloads (RTF format)
Appendix 11: Analysis of Hit-Counts on a Personal Web-Site (XLS format)
Appendix 12: Analysis of Personal Web-Sites (HTML format)
Appendix 13: Author Repositories (RTF format)
The assistance is gratefully acknowledged of Gregor Lenart of the University of Maribor, who provided extracts from the Bled Conference database, and of Andy Schwarz, VP Technology for AIS, who provided statistical information on downloads of Bled papers from the AIS eLibrary. The comments of the reviewers were very helpful in tightening the paper's focus and refining the analyses.
Roger Clarke is Principal of Xamax Consultancy Pty Ltd, Canberra. He is also a Visiting Professor in the Cyberspace Law & Policy Centre at the University of N.S.W., and a Visiting Professor in the Research School of Computer Science at the Australian National University.
| Personalia |
Photographs Presentations Videos |
Access Statistics |
 |
The content and infrastructure for these community service pages are provided by Roger Clarke through his consultancy company, Xamax. From the site's beginnings in August 1994 until February 2009, the infrastructure was provided by the Australian National University. During that time, the site accumulated close to 30 million hits. It passed 65 million in early 2021. Sponsored by the Gallery, Bunhybee Grasslands, the extended Clarke Family, Knights of the Spatchcock and their drummer |
Xamax Consultancy Pty Ltd ACN: 002 360 456 78 Sidaway St, Chapman ACT 2611 AUSTRALIA Tel: +61 2 6288 6916 |
Created: 28 August 2011 - Last Amended: 25 March 2012 by Roger Clarke - Site Last Verified: 15 February 2009
This document is at www.rogerclarke.com/EC/Bled25TA.html
Mail to Webmaster - © Xamax Consultancy Pty Ltd, 1995-2022 - Privacy Policy