psychometric forensics, conference on test security, data forensics, test fraud

Can we call it Psychometric Forensics?

An emerging sector in the field of psychometrics is the area devoted to analyzing test data to find cheaters and other illicit or invalid testing behavior. We lack a generally agreed-upon and marketable term for that sort of work, and I’d like to suggest that we use Psychometric Forensics.

While research on this topic is more than 50 years old, the modern era did not begin until Wollack published his paper on the Omega index in 1997. Since then, the sophistication and effectiveness of methodology in the field has multiplied, and many more publications focus on it than in the pre-Omega era. This is evidenced by not one but three recent books on the subject:

  1. Wollack, J., & Fremer, J. (2013).  Handbook of Test Security.
  2. Kingston, N., & Clark, A. (2014).  Test Fraud: Statistical Detection and Methodology.
  3. Cizek, G., & Wollack, J. (2016). Handbook of Quantitative Methods for Detecting Cheating on Tests.

In addition, there was the annual Conference on the Statistical Detection of Test Fraud, which recently changed its name to the Conference on Test Security.

I’m excited about the advances being made, but one thing has always bugged me about this: the name, or lack thereof. There is no name that’s relatively accepted for the topic of statistical detection. Test Security is a good name for the broader topic, which also includes things like security policies for test centers. But not for the data side. You can see that the 2014 and 2016 books as well as the original conference title all have similarity, but don’t agree, which I think in part is because the names are too long and lack the modern marketing oomph. Just like the dreadful company names of the 1960s and 1970s… stuff like Consolidated International Business Logistics Incorporated. No wonder so many companies changed to acronyms in the 1980s.

As I thought about this, I began by laying down my requirements for what I would want in the name:

  1.  Short enough to not be a mouthful – preferably two words max
  2. Descriptive enough to provide some initial idea what it is
  3. Not already taken elsewhere (the use of “certification management” in the testing industry is a great example of this failure)
  4. Wide enough to cover cases that are not fraud/malicious but still threats to validity

Statistical Detection of Test Fraud or configurations of it pass Requirements 2 and 3, but certainly fail 1 and 4. Conversely, Data Forensics or similar terms would pass Requirements 1 and 4, but fail 2 and 3 quite badly. I’m also guilty, by naming my software program Software for Investigating Fraud in Testing – mostly because I thought the acronym SIFT fit so well.

I’d like to suggest the term Psychometric Forensics. This meets all 4 requirements, at least within the testing industry (we’re still fighting that uphill battle to get the rest of the world to even understand what psychometrics is). A quick Google search does not find hits, and only finds two instances of psychometric data forensics, which are buried in technical reports that primarily use the term data forensics.

However, I’m not completely sold on it.  Any other suggestions?  I’d love to hear them!

The following two tabs change content below.

Nathan Thompson, PhD

Chief Product Officer at ASC
I am a psychometrician, software developer, author, and researcher, currently serving as Chief Product Officer for Assessment Systems Corporation (ASC). My mission is to elevate the profession of psychometrics by using software to automate the menial stuff like job analysis and Angoff studies, so we can focus on more innovative work. My core goal is to improve assessment throughout the world. I was originally trained as a psychometrician, doing an undergrad at Luther College in Math/Psych/Latin and then a PhD in Psychometrics at the University of Minnesota. I then worked multiple roles in the testing industry, including item writer, test development manager, essay test marker, consulting psychometrician, software developer, project manager, and business leader. Research and innovation are incredibly important to me. In addition to my own research, I am cofounder and Membership Director at the International Association for Computerized Adaptive Testing, You can often find me at other important conferences like ATP, ICE, CLEAR, and NCME. I've published many papers and presentations, and my favorite remains http://pareonline.net/getvn.asp?v=16&n=1.

Latest posts by Nathan Thompson, PhD (see all)

0 replies

Leave a Reply

Want to join the discussion?
Feel free to contribute!

Leave a Reply