CSE702 Seminar: Analyzing Cognitive Tendencies From Chess Data

Spring 2025, Reg # 21364

Dr. Kenneth W. Regan

Topics

Artificial intelligence; machine learning; predictive analytic modeling; chess; decision theory; psychometrics/performance evaluation; cheating detection; cognitive tendencies; applied Turing tests; connections to LLMs.

General Description

My predictive analytic model of move-choice at chess(*) and simpler associated scripts operate in a data-rich environment. They show definite human tendencies and foibles, some of which are used to distinguish human from computer play and catch instances of human players cheating via computer assistance. The focus will be on making larger conclusions, with the following among multiple possible objectives:

  1. How can learning growth of young players be measured? The pandemic brought this need to the fore while official ratings were largely frozen, and the international chess rating system on the whole is still far from recovered.
  2. Following on from (1), where does the notorious gender gap in chess begin, and what are implications for STEM education?
  3. Can notions of difficulty of chess games crafted in the model be carried over to yield internal means of evaluating the difficulty of standardized tests? The mathematics of chess ratings (which are based on logistic curves) and of formulas in item response theory and other areas of psychometrics have commonalities.
  4. What is the relationship between the quality of decisions and the time to make them---both time available and time consumed?
  5. Can this model be extended to offer a simple frequentist alternative to neural methods of ChatGPT (etc.) cheat detection, noting how the "best next word" architecture relates to "best next move" in game play?
  6. How does the model supplement other forms of statistical analysis?
  7. What external lessons are there for statistical practice? (See these two recent talks.)

The seminar will follow a lecture-and-reading pattern for the first month-plus to bring us up to speed.

Intro slides. Background on chess programs.

Piazza Page for 2025.

Notes For 2025

Week 2: Elo Ratings

Week 2+3: Elo and concordance metrics.

Week 3: Monday. Tuesday.

Week 4: Monday,

Last year's edition made important progress on items 3 and 4, and one project group reflected item 5 by starting the design of a transformer to play chess. Some of this new material was included in latter slides of these presentations:

The findings regarding thinking time are surprising. The model needs some further tuning to get a more-precise fix on the magnitude of these phenomena.

In this edition, I wish to devote more emphasis to item 5---to try to exploit mathematical correspondences to carry over some of this work to the analysis of Large Language Models. But there is a wide range of possible directions.

Posted Notes From 2024 Meetings not listed were demo and discussion sessions.

Tue. 1/30: Elo Ratings from the ground up.

Tue. 2/06: Raw Metrics Versus Ratings (before and after Sonas correction).

Tue. 2/13: Screening and Normality; Brief user guide.

Thu. 2/15 & 2/22: Main Analyzer Workflow.

Tue. 2/27: Model Particulars and Principles.

Thu. 2/29: Core Predictive Component.

Tue. 3/05 & Thu. 3/7: Cross-Validation and Experiments

Tue. 3/12 & Thu. 3/14: Model Issues and ToDos

After spring break, the seminar went into presentation discussion mode. These particular notes are labeled for weeks "8" and "9" for continuity:

Week 8: Difficulties with Measuring Difficulty

Week 9: IPRs and Error Bars

(*) The model is called "Fidelity" and is developed in refereed publications thru 2015 and more-recent articles on the GLL blog, of which the most important is "Predicting Chess and Horses". Other general information includes a two-page description and a longer overview of the research, the latter with some mathematical details. My homepage links my public anti-cheating site, papers, talks, media articles and podcasts, and other pages.

Expectations

As with past years, requirements will be (1) participation in discussions and little experiments, in class and/or on Piazza, (2) learning and applying computational statistical and charting tools, and (3) presenting a "mini-project" or paper (perhaps teamed). No background in chess or other computer-treated strategy games is assumed (enough will be covered early on) and also there are no other prerequisites.

Students in the seminar will be given access to private data and sites where experimentation is done. The last section of the overview includes some possible seminar topics and projects within this research, but students will be equally welcome to give presentations relating it to machine-learning related topics they have had in other courses. The larger goal will be to provide experience with data-analysis basics including regressions, confidence testing, conducting numerical experiments, and drawing and reporting inferences.

Grading is S/U, 1--3 credits. Weekly meeting times (two 80-minute sessions per week is my preferred system) will be determined around schedules of those who register or otherwse express interest.