Using PIAAC Log Files and Process Data: What can the process data tell us about the process of test taking?
| Summary | How to Apply & Requirements |
| About the Workshop | Dates & Location |
| Agenda | Workshop Evaluation |
About the Workshop
The PIAAC 2012 study was the first international large-scale assessment to implement a computer-based measurement of adults’ key competencies. On average, about 77 percent of the populations in all the participating countries were able to complete the cognitive assessment on the computer, with 84 percent in the United States and 82 percent in Germany (Mamedova and Pawlowski 2018). Hence, most of the respondents’ actions were automatically recorded by the interview software during the direct assessment and stored with time stamps in separate so-called “log files.” The log files contain process data, such as which buttons and tabs they clicked and how many times, for each participant taking the literacy, numeracy, and problem solving in technology-rich environments (PS-TRE) assessments on the computer. In 2017, the OECD released the log-file data as public use files for 17 countries participating in Round 1 of PIAAC 2012.
The log files expand the PIAAC databases and offer new research opportunities. For instance, they allow researchers to:
analyze timing behavior (e.g., total time per item and components thereof) and interactions with the item (e.g., selecting and changing a response option);
explore the process components of a successful item solution (e.g., number and sequence of switching environment);
identify groups of test takers with similar solution behaviors (e.g., groups with more or less efficient strategies); and
analyze the relationship between success in item completion and indicators of solution strategies (e.g., navigation patterns).
This is the second training workshop that aims to widen the use of the PIAAC log-file data; the first workshop was organized by the GESIS in Mannheim, Germany in the spring of 2017. Drawing on theoretical lectures and hands-on exercises, this training focuses on the following goals:
Introduce a conceptual background on how to interpret and use log data; and discuss the merits and challenges of data analysis enriched with log data, using examples from previous research.
Familiarize participants with the accessibility, structure, and content of the PIAAC log files (documentation available for reference); and describe how the log files can be analyzed in conjunction with the released PIAAC public use files.
Familiarize participants with the PIAAC LogDataAnalyzer (LDA) Windows-based tool, which provides raw PIAAC log data as well as predefined generic and task-specific process indicators derived from the raw log data; and how to further analyze these data in standard statistical software such as the R environment.
Prepare participants to derive and/or use process indicators and select appropriate statistical methods based on a research question (e.g., modeling and explaining test-taking engagement by means of item response times), taking into account the complex data structure of the PIAAC data; examples will be given for both exploratory and hypothesis-testing data analyses.
