Series:

Thibault Clérice and Matthew Munson

Abstract

Much attention has been given in the last several years to the task of automatically extracting semantic information from raw textual data and the best algorithms and the best parameters to achieve this task. But the accomplishment of this task need not be an end in itself. Instead, the data that is produced by these processes can be used to answer questions outside of purely linguistic studies. This paper aims to help make these algorithmic processes more accessible to other humanities disciplines by considering how one can qualitatively assess the results returned by such algorithms. It will introduce an effective and yet easy-to-understand metric for parameter choice which we call Gap Score. Then it will use Gap Score to analyze three distinct sets of results produced by two different algorithmic processes to discover what type of information they return and, thus, for which types of hermeneutical tasks they may be useful. Our purpose in doing this is to demonstrate that the accuracy of an algorithm on a specific test, or even a range of tests, does not tell the user everything about that algorithm. Gap Score introduces a qualitative aspect to the assessment of algorithmic processes that recognizes that an algorithm that might score lower on a certain standardized test may actually be better for certain hermeneutical tasks than a better scoring algorithm.