Log and Report File Format

The log file consists mostly of data clipped from chess engines, which is date and time stamped. The procedure followed on this site:

1. Make a text file of the game to be tested. Find the first non-book move, or the first move in the range to be tested. Insert several empty lines around each move to be tested, to leave ample room for pasting engine output.
2. At the top record hardware used, engine used, hash table size and other relevant parameter settings, and basic methodology.
3. Paste data after the played move being tested, unless the moves are kept in pairs like 15.Nd5 e6 and the player being tested is White, in which case the move-15 test data may come before it.
4. Following "zig-zag" method, first paste MLM at 12/12, or if k lines are already done (shown by turning from grey to black) to extension depth e by the time data is clipped, label it "12/e +k".
5. Then paste SLM at d/d. This does not need a hand-typed label, since (unlike with MLM) the depth is recorded with the engine output. Optionally, delete lines for depths below 11, and it is OK to continue beyond depth d if there is a compelling reason. (See notes on "scripting" below.)
6. Then paste MLM at depth d_L, labeling it "end of depth d_L" or "d_L+1/d_L+1"---or "d_L+1/e + k" if k lines in the depth-(d_L + 1) round are already done.
To record the outcome of the test, type a fresh line of the form

{Name} bbbbbbb {type} MATCH (#.move played), delta[2] = {value} [comments]

Here {Name} is a tag identifying the player being tested, and "bbbbbbb" is a string of (d - d_min + 1) characters with:
• 1 for a match at that ply
• 0 for a non-match at that ply
• T (optional) for a tie at that ply depth among 2 or more moves. However, SLM always identifies a single principal move at each depth, and the fact of matching it is relevant even in case of ties. Hence the SLM/"zig-zag" runs do not use T---instead, ties are noted in [comments].
• "-" in case no data was recorded at that ply depth.
The "type" of match is
• NON-MATCH, if all "b"s are 0
• FULL MATCH, if all "b"s are 1
• MAIN MATCH, if any of the final three "b"s is a 1---this range centers on the bit deemed most likely to be used by the choice of d.
• PART MATCH, otherwise.
7. In case of a FULL or MAIN MATCH, after the move played write "delta = " with {value} = the difference between the MLM eval of the move played and the MLM eval of the highest alternative. This difference can be negative, owing to differences between SML and MLM depth, or possibly even to hash collisions and other causes of experimental variation. The character "=" should not occur earlier on the line.
8. In case of a PART or NON-MATCH, after the move played write "delta2 = " with {value} = the MLM eval of the move played minus the MLM eval of the engine's top move (which in the zig-zag method will usually equal its prior SLM eval). This difference can be positive.
9. In case all alternatives lead to mate, write "delta = 9.99". In case the move played was a repetition, or the result should be considered invalid for some other reason, write "delta = NaN". Do not write a leading "+" sign before a value.
10. When d has been increased on reaching simpler positions, note the first such move in the file, and use "+bb" to indicate the extra bit(s). Judgment of MAIN matches should reflect the new d, while tallies of matches at given ply depths should not(?).

Extracting Results Files

A simple results file---not indicative of the full statistical method---starts by pulling off all lines with "MATCH" from the log file. These will come initially in sorted order by game move. Sorting on the field after the "=" sign (sort -r -t= -k2 {file} in UNIX/Linux) gives a preliminary ordering by significance. The totals of moves, matches of each kind and non-matches, and matches at each ply depth, are also recorded in results-files on this site.

More sophisticated extraction needs to read all of the computed MLM values for alternative moves. Customized scripts for doing this reading may require more-detailed format specifications than given here.

Scripting These Tests

The UCI, XBoard/WinBoard, and CECP protocols do not have fine enough controls for scripting tests of this kind. The built-in analysis modes of ChessBase/Fritz GUI engines run only in reverse from the end of games, which is not realistic for anti-cheating testing since evals of later positions are stored in hash. Buttons and options needed to script these tests unattended include:

• Move Range m1..m2, options forward/backward
• Last Ply Searched at min/max of depth d, time t.
• Search Type: S only/M only/S then M/M then S/SMS/MSM (...)
• NumLines for MLM: L
• reset maxSearchDepth = d
• clip at depths d1,d2,...,d
• clip at times t1,t2,...,t
• Special variables time and lastEval can be used in forming conditions.

A typical script could be: reset maxPlyDepth = 17
for m in 14..53 {
  reset numLines = 10  //does not clear hash
  m.startAnalysis()
  for d in 11..11 {
    clip at depths d  //gets MLM at 12/12
  }
  reset numLines = 1   //goes into SLM without clearing hash, pauses
  for d in 12..maxSearchDepth {
    if
    

Remaining Methodology

This covers the gathering and presentation of the basic data. Rules for statistical analysis, and for drawing conclusions according to recognized significance levels, will be detailed in our main papers.