**** Data and analysis underlying the research into The Artificial-Social-Agent Questionnaire: Establishing the long and short questionnaire versions. ****
Author: Siska Fitrianie, Merijn Bruijnes, Fengxiang Li, Amal Abdulrahman, Willem-Paul Brinkman
Corresponding author: Siska Fitrianie, s.fitrianie@tudelft.nl
https://doi.org/10.4121/19758436

Interactive Intelligence Group
Department of Intelligent Systems
P.O. Box 5031,
2600 GA Delft,
The Netherlands

Description of the dataset
------------------------------------------------------------------------------------------------------------------------------------
The Archive.zip is a complementary material of the paper: Siska Fitrianie, Merijn Bruijnes, Fengxiang Li, Amal Abdulrahman, and Willem-Paul Brinkman. 2022. The Artificial-Social-Agent Questionnaire: Establishing the long and short questionnaire versions. In ACM International Conference on Intelligent Virtual Agents (IVA’22), September, 2022, Faro, Portugal. ACM, New York, NY, USA. https://doi.org/10.1145/3514197.3549612
It contains a technical report, data, scripts and analysis based on an empirical study (n = 532). To access the content, one can follow the table of content in index.html, which is corresponding to the table of content (outline) of the paper.


We present a technical report, data and analysis underlying the study on establishing the Artificial Social Agent (ASA) Questionnaire. The ASA Questionnaire is an instrument for evaluating human interaction with an ASA, resulted from multi-year efforts involving more than 100 Intelligent Virtual Agent (IVA) researchers worldwide within the OSF work-group of Artificial Social Agent Evaluation Instrument (https://osf.io/6duf7/). It has 19 measurement constructs constituted by 90 items, which capture more than 80% of the constructs identified in empirical studies published in the IVA conference 2013-2018.  This paper reports on construct validity analysis, specifically convergent and discriminant validity of initial 131 instrument items that involved 532 crowd-workers who were asked to rate human interaction with 14 different ASAs. The analysis included several factor analysis models, and resulted in the selection of 90 items for inclusion of the long version of the ASA questionnaire. In addition, a representative item of each construct or dimension was select to create a 24-item short version of the ASA questionnaire. Whereas the long version is suitable for a comprehensive evaluation of human-ASA interaction, the short version allows quick analysis and description of the interaction with the ASA. To support reporting ASA questionnaire results, we also put forward an ASA chart. The chart provides a quick overview of agent profile. 

This study is approved by the Human research Ethics Committee TU Delft date 18-12-2020 and registered at Open Science Framework https://doi.org/10.17605/OSF.IO/KZ8V4

Abstract of the paper Fitrianie et al. (2022)
------------------------------------------------------------------------------------------------------------------------------------
We present the ASA Questionnaire, an instrument for evaluating human interaction with an artificial social agent (ASA), resulted from multi-year efforts involving more than 100 Intelligent Virtual Agent (IVA) researchers worldwide. It has 19 measurement constructs constituted by 90 items, which capture more than 80% of the constructs identified in empirical studies published in the IVA conference 2013-2018.  This paper reports on construct validity analysis, specifically convergent and discriminant validity of initial 131 instrument items that involved 532 crowd-workers who were asked to rate human interaction with 14 different ASAs. The analysis included several factor analysis models, and resulted in the selection of 90 items for inclusion of the long version of the ASA questionnaire. In addition, a representative item of each construct or dimension was selected to create a 24-item short version of the ASA questionnaire. Whereas the long version is suitable for a comprehensive evaluation of human-ASA interaction, the short version allows quick analysis and description of the interaction with the ASA. To support reporting ASA questionnaire results, we also put forward an ASA chart. The chart provides a quick overview of agent profile.

Requirements
------------------------------------------------------------------------------------------------------------------------------------
1. Internet Browser
2. Pdf reader to read .pdf file
3. Text editor to read .csv file
4. Microsoft Excel 2003 or higher to read .xlsx file
5. R (v4.0.4) with factor analysis libraries from the package pscyh (v2.1.3) and lavaan (v0.6-8) to run R codes in:
   construct_validity/r_script

Quick Instruction
------------------------------------------------------------------------------------------------------------------------------------
1. Unzip Archive.zip
2. Run index.html on an internet browser
3. Follow the table of content that is corresponding with the outline of the paper Fitrianie et al. (2022)
4. Use provided links to access the technical report, codes, data and analysis results

Revision
------------------------------------------------------------------------------------------------------------------------------------
01/09/2023
- Fix typeO on TechnicalReport_ASAEvalInst-TR08.pdf related to the ASA Questionnaire ITEMID=C01D02Q10 (HLB2) and ITEMID=C18D03Q0 (UEP1)
- Fix typeO on 131_questionnaire_items.xlxs  related to the ASA Questionnaire ITEMID=C01D02Q10 (HLB2) and ITEMID=C18D03Q0 (UEP1)