The Effects of Item Ordering on Reproducibility in Information Systems Online Survey Research

2021 | Communications of the Association for Information Systems | Citations: 2

Authors: Wilson, Vance; Srite, Mark; Loiacono, Eleanor

Abstract: Online survey applications offer various options for administering items, such ... Expand

Abstract: Online survey applications offer various options for administering items, such as approaches that completely or partially randomize the order in which they present items to each subject. Vendors claim individual randomization eliminates key sources of method bias that can impact reproducibility. However, little empirical evidence exists to directly support this claim, and it is difficult to evaluate based on existing research because such research has underreported item-ordering methodologies and the reporting that does occur frequently lacks clarity. In this paper, we investigate the effect that item ordering has on reproducibility in IS online survey research via comprehensively comparing five prominent item-ordering approaches: 1) individually randomized, 2) static grouped by construct, 3) static intermixed, 4) individually randomized grouped-by-construct blocks containing static items, and 5) static grouped-by-construct blocks containing individually randomized items. We found significant, overarching differences among these approaches that can threaten research findings' reproducibility. These differences appear across the measures we studied, which included item and construct means, reliability and construct validity statistics, serial effects, and subjects' fatigue and frustration that resulted from the survey-taking process. Our findings support a call for several key changes in how researchers report and use item-ordering approaches that pertain particularly to IS online survey research.Serial mechanisms accompany item placement in a survey (i.e., the position where one statically places items in a survey). In reviewing the literature, Krosnick and Presser (2010) theorized that serial order effects result from systematic changes in subjects' motivation, learning, and fatigue that occur as they progress through a survey.Research has long established that individually randomized item ordering removes the opportunity for both semantic and serial mechanisms to systematically bias survey results. Perreault (1975, p. 545) noted:Although such order effect is commonly recognized as a source of bias in survey research, it is frequently ignored. This lack of rigor is explained by two considerations. First, models for statistical estimations of order effect are complicated and, more important, appropriate only in restrictive cases. If order bias is not treated statistically, it should be controlled by randomizing its effect across respondents. This type of control requires that the researcher produce different questionnaires, comprised of random orderings of relevant items (while maintaining proper overall sequence) for each respondent. The initial expense and clerical difficulties of producing many different questionnaires are obvious and are further compounded by the problems of coding the data from the completed questionnaires into a consistent (machine-interpretable) form.We need to recognize that individually randomized item ordering does not eliminate semantic or serial effects for individual subjects who one may anticipate will interpret item meanings and experience motivation, learning, and fatigue at varying levels depending on the particular item ordering they view. However, individual randomization will randomly distribute the error representing the combined effects of these interpretations and experiences across subjects and, thus, appear as random error rather than as systematically concentrated in the survey response data. Collapse

Topics: website job satisfaction online learning system

Methods: survey structural equation modeling parametric test survey design one-way ANOVA

Beautiful is Good and Good is Reputable: Multiple-Attribute Charity Website Evaluation and Initial Perceptions of Reputation Under the Halo Effect

2019 | Journal of the Association for Information Systems | Citations: 0

Authors: Kwak, Dong-Heon; Ramamurthy, K. (Ram); Nazareth, Derek

Abstract: The halo effect has been extensively used to understand how people make judgment ... Expand

Abstract: The halo effect has been extensively used to understand how people make judgments about the quality of an object. Also, the halo effect has been known to occur when people evaluate multiattribute objects. Although websites consist of multiple attributes and dimensions, prior research in information systems has paid little attention to how people evaluate multi-attribute websites and associated halos. Furthermore, research investigating how initial evaluations of reputation are formed toward unknown objects under the halo effect is scarce. Based on these two research gaps, the purposes of this study are to identify whether there is evidence of salient halos in the evaluation of multi-attribute websites and to theorize initial perceptions of reputation. To accomplish these objectives, we introduce a framework for classifying halos based on attributes and dimensions. Also, this study employs charity websites as a multi-attribute donation channel consisting of three attributes of information content quality (mission information, financial information, and donation information) and four attributes of system quality (navigability, download speed, visual aesthetics, and security). Based on the proposed framework, this study proposes four types of halos that are relevant to charity website evaluation—collective halo (attribute-to-attribute), aesthetics halo (attribute-to-dimension), reciprocal-quality halo (dimension-to-dimension), and quality halo (dimension-to-dimension). The results of structural equation modeling and other analyses provide evidence of the various proposed halos. Collapse

Topics: website system quality data quality service quality website design

Methods: experiment theory development survey factorial experiment descriptive statistic

ME, YOU OR AI? HOW DO WE FEEL ABOUT DELEGATION

2019 | European Conference On Information Systems | Citations: 0

Authors: Leyer, Michael; Schneider, Sabrina

Abstract: Driven by the growing availability of data and information paired with the human ... Expand

Abstract: Driven by the growing availability of data and information paired with the human brain’s limited processing capabilities, Artificial Intelligence (AI) has become an increasingly relevant actor in strategic decision making. To leverage the potential AI provides for strategic decision making, firms rely on managers that are willing to transfer authority and control to AI-based decision systems. By analyzing the willingness to delegate strategic decisions and emotional responses to outcomes of delegated decisions, this research contributes to a better understanding of human behavior in decision delegation to AI in the context of strategic decisions. Based on a policy-capturing study, we find that humans are less likely to delegate a strategic decision to AI, than to another person. Further, our findings reveal that emotional responses to the outcomes of delegated decisions are more intense when responsibility was delegated to another human being, rather than to an AI-enabled system. Collapse

Topics: artificial intelligence decision making decision support system organizational context IT career

Methods: computational algorithm experimental group multivariate analysis of variance statistical hypothesis test Games-Howell test

Theories: attachment theory

Can Privacy and Security Be Friends? A Cultural Framework to Differentiate Security and Privacy Behaviors on Online Social Networks

2017 | HICSS | Citations: 2

Authors: Dincelli, Ersin; Goel, Sanjay

Abstract: The boundaries between online privacy and security behaviors in the literature s ... Expand

Abstract: The boundaries between online privacy and security behaviors in the literature seem blurred. Although these two behaviors are conceptually related, we argue that one does not necessarily imply the other. In this study we aimed to (1) explore the subtle differences between online privacy and security behaviors, and (2) examine how users’ cultural characteristics and a group of multilevel factors exert different effects on the two behaviors. To achieve these two goals, we created a framework by coupling the grid-group theory and INDCOL scale to segment individuals into four categories based on autonomy (individualist vs. collectivist) and acceptance of control (hierarchy vs. equality). The results of one-way ANOVA and path analysis partially confirmed that the underlying mechanisms of online privacy and security behaviors were inherently different. This study provides a basis for creating contextualized security trainings and warnings based on individual differences to promote better privacy and security behaviors. Collapse

Topics: privacy privacy behavior digital literacy password Facebook

Methods: path analysis survey parametric test descriptive statistic ANOVA

Tapping into the Collective Creativity of the Crowd: The Effectiveness of Key Incentives in Fostering Creative Crowdsourcing

2017 | HICSS | Citations: 0

Authors: Literat, Ioana

Abstract: To better understand the conditions that most effectively stimulate creative par ... Expand

Abstract: To better understand the conditions that most effectively stimulate creative participation online, a crowdsourcing project was implemented on Amazon’s Mechanical Turk, collecting 4200 written and visual submissions from online participants. An experimental research design tested the impact of specific incentive structures (i.e. financial rewards, bonuses, specification of project purpose, attribution of authorship credit) on the outcomes of creative participation (quantity of submissions, quality of submissions, time spent on task). The study found that extrinsic rewards (i.e. higher pay and bonuses) are effective in encouraging participants to accept the creative task, whereas the strategies that boost the creativity of the submissions are: offering a bonus, mentioning a charitable purpose, and giving contributors authorship credit. These findings help illuminate the factors that have the greatest impact on the quality and quantity of online creative participation, thus making a vital contribution to our understanding of digital creativity. Collapse

Topics: crowdsourcing internet technology accounting

Methods: Games-Howell test Tukey HSD for post poc analysis experimental design post-hoc analysis experimental group

Social media management strategies for organizational impression management and their effect on public perception

2016 | Journal of Strategic Information Systems | Citations: 0

Authors: Benthaus, Janek; Risius, Marten; Beck, Roman

Abstract: With the growing importance of social media, companies increasingly rely on soci ... Expand

Abstract: With the growing importance of social media, companies increasingly rely on social media management tools to analyze social media activities and to professionalize their social media engagement. In this study, we evaluate how social media management tools, as part of an overarching social media strategy, help companies to positively influence the public perception among social media users. A mixed methods approach is applied, where we quantitatively analyze 15 million user-generated Twitter messages containing information about 45 large global companies highly active on Twitter, as well as almost 160 thousand corresponding messages sent from these companies via their corporate Twitter accounts. Additionally, we conducted interviews with six social media experts to gain complementary insights. By these means, we are able to identify significant differences between different social media management strategies and measure the corresponding effects on the public perception. Collapse

Topics: Twitter social media word of mouth application programming interface user interface

Methods: qualitative interview mixed method sentiment analysis computational algorithm design methodology

Theories: management theory

Strategies for Establishing Service Oriented Design in Organizations

2012 | International Conference on Information Systems | Citations: 0

Authors: Aier, Stephan

Abstract: Service orientation is a broadly discussed design paradigm for information syste ... Expand

Abstract: Service orientation is a broadly discussed design paradigm for information systems engineering. Only recently there have been several contributions to individual dimensions of service orientation like service definition, service modeling, service management or service governance. However, a combination of useful solutions for partial problems might not constitute an effective overall approach to service orientation. We argue that the establishment of service orientation is a wicked [design] problem, i.e. poorly formulated, confusing, and permeated with conflicting values of many decision makers or other stakeholders. Therefore this paper focuses on understanding the design problem of establishing service orientation in organizations in its design dimensions. The paper identifies four strategies for establishing service orientation and derives four propositions on the development paths which may connect these strategies. Collapse

Topics: enterprise resource planning enterprise application integration critical success factor IT governance systems design

Methods: survey data transformation factor analysis design artifact ANOVA

Theories: punctuated equilibrium theory service-dominant logic

Trust perceptions of online travel information by different content creators: Some social and legal implications

2011 | Information Systems Frontiers | Citations: 14

Authors: Burgess, Stephen; Sellitto, Carmine; Cox, Carmen; Buultjens, Jeremy

Abstract: Consumers are increasingly turning to the online environment to provide informat ... Expand

Abstract: Consumers are increasingly turning to the online environment to provide information to assist them in making purchase decisions related to travel products. They often rely on travel recommendations from different sources, such as sellers, independent experts and, increasingly, other consumers. A new type of online content, user-generated content (UGC), provides a number of legal and social challenges to providers and users of that content, especially in relation to areas such as defamation, misrepresentation and social embarrassment. This paper reports research that examined the level of trustworthiness of online travel information from these different sources. The study used a survey of Australian travel consumers (n = 12,000) and results support the notion that there are differences in the level of trust for online travel information from different sources. Respondents ‘tended to agree’ that they trusted information provided by travel agents, information from commercial operators and comments made by travellers on third party websites. However, the highest level of trust was afforded to information provided on State government tourism websites. These results suggest that greater trust is placed in online travel comments when they are on a specific travel website than when they are on a more generic social networking website. However, respondents were ‘not sure’ that they trusted comments made by travellers on weblogs and on social networking sites. Some 88% of respondents that had not visited UGC websites (or were unsure if they had) indicated that they thought that UGC would be useful in the future – suggesting that they feel that any concerns they may have in relation to legal and social problems resulting from its use will be resolved. Collapse

Topics: website electronic mail social network database system internet technology

Methods: survey survey design Mann Whitney U test Games-Howell test

Publication Opportunities in Premier Business Outlets: How Level Is the Playing Field?

2006 | Information Systems Research | Citations: 19

Authors: Valacich, Joseph S.; Fuller, Mark A.; Schneider, Christoph; Dennis, Alan R.

Abstract: This paper reports an analysis of the proportion of faculty publishing articles ... Expand

Abstract: This paper reports an analysis of the proportion of faculty publishing articles in premier business journals (i.e., the ratio of authors of premier business journal articles to total faculty of a discipline) across the disciplines of accounting, finance, management, marketing, and information systems (IS) for the years 1994─2003. This analysis revealed that over this period the management discipline had on average the highest proportion of faculty publishing in premier journals (12.7 authors per 100 management faculty), followed by finance (9.4 authors per 100 faculty), marketing (9.2 authors per 100 faculty), IS (5.5 authors per 100 faculty), and accounting (4.8 authors per 100 faculty). A further analysis examined these ratios for the different disciplines over time, finding that the ratios of authors to faculty have actually decreased for the disciplines of marketing and IS over this time period but have remained stable for the disciplines of accounting, management, and finance. Given steady growth in faculty size of all disciplines, the proportion of faculty publishing articles in premier journals in 2003 for all disciplines is lower than their 10-year averages, with IS having the lowest proportion in 2003. A sensitivity analysis reveals that without substantial changes that would allow more IS faculty to publish in the premier journals (e.g., by increasing publication cycles, number of premier outlets, and so on), IS will continue to lag far below the average of other disciplines. The implications of these findings for IS researchers, for institutions and administrators of IS programs, and for the IS academic discipline are examined. Based on these implications, recommendations for the IS discipline are presented. Collapse

Topics: accounting marketing management IS discipline electronic business IS management

Methods: literature sample sensitivity analysis parametric test ANOVA survey