KWR's Screening Tests with Houdini 3 of Several Famous "Mercurial Performances" ------------------------------------------------------------------------------- Using Single-PV fixed-depth 17, 512MB or 256MB hash, forward mode in Arena GUI First, here is Borislav Ivanov from the 2012 Zadar Open. The Single-PV test shows a much more extreme result than the Multi-PV test with Houdini. The results for depths 19 and 21 are actually less extreme (about 70% matching) Right-hand column for other players shows move-match percentages against them. AE is "Scaled Average Error" from the Regan-Haworth-Macieja ICGA 2011 paper. *Very very rough* guide: intrinsic performance = 3550 - 1.5*AE where you read the AE without the decimal point. It is *rough* because it does *not* account for the difficulty of the particular positions the player faced, which the IPR regression *does*. The difference is often 100--200 Elo points or more. For Ivanov, the rough indicator does 1.5*287 = 431 to put him about 3120 (actual 3089), and 1.5*545 = 818 to put his opponents about 2730 (actual 2476). Since the game with Jovanic had exceedingly long stretches with quiet, equal positions, the rough AE measure may give both players too much credit (Jovanic especially since he didn't blunder) which my IPR procedure discounts. ------------------------------------------------------------------------------- Report from IvanovZadarOpenHou3d17, excluding repeats and |prev-eval| > 3 First-line matches to Houdini_3_Standard_w32 1-cpu at depth 17: Player Name Matches/Turns = Pctg., AE Opponents' figures ------------------------------------------------------------------------------- Ivanov, Bor BUL: 203/ 276 = 73.60%, 0.0287 135/ 278 = 48.60%, 0.0545 Jovanic, O. : 53/ 98 = 54.10%, 0.0152 66/ 95 = 69.50%, 0.0213 Kozul, Z. : 12/ 26 = 46.20%, 0.0982 22/ 26 = 84.60%, 0.0044 Kuljasevic, D. : 13/ 22 = 59.10%, 0.0357 16/ 21 = 76.20%, 0.0365 Kurajica, B. : 8/ 16 = 50.00%, 0.0526 16/ 16 = #####%, 0.0000 Predojevic, B. : 10/ 23 = 43.50%, 0.0677 11/ 24 = 45.80%, 0.1224 Saric, Iv : 9/ 25 = 36.00%, 0.1279 17/ 25 = 68.00%, 0.0462 Schachinger, M.: 7/ 20 = 35.00%, 0.0672 17/ 21 = 81.00%, 0.0024 Sumets, A. : 17/ 34 = 50.00%, 0.0438 24/ 33 = 72.70%, 0.0241 Zelcic, R. : 6/ 14 = 42.90%, 0.1363 14/ 15 = 93.30%, 0.0046 Totals for all players in IvanovZadarOpenHou3d17: 338 / 554 = 61.01% Aggregate difference in IvanovZadarOpenHou3d17: 23.0714 / 554 = 0.0416 ------------------------------------------------------------------------------- The Single-PV test comes with no statement of odds, doesn't "prove" anything, and I haven't even run large Houdini comparison data like with Rybka 3. Since 100% matching overflows the Perl script's report field, it's easy to spot. The following lightning-in-a-bottle 2600+ performances by 2200-rated players have been cited for comparison, so I have run them with the same settings. ------------------------------------------------------------------------------ Report from SofiaPolgarRome1989Hou3d17, excluding repeats and |prev-eval| > 3 First-line matches to Houdini_3_Pro_x64 1-cpu at depth 17: Player Name Matches/Turns = Pctg., AE Opponents' figures ------------------------------------------------------------------------------- Cardinali, Marc: 18/ 32 = 56.30%, 0.1354 17/ 32 = 53.10%, 0.1302 Chernin, Alexan: 13/ 20 = 65.00%, 0.1563 13/ 21 = 61.90%, 0.0580 D'Amore, Carlo : 21/ 40 = 52.50%, 0.1156 23/ 40 = 57.50%, 0.0874 Dolmatov, Serge: 4/ 10 = 40.00%, 0.0595 5/ 11 = 45.50%, 0.0569 Mrdja, Milan : 17/ 32 = 53.10%, 0.1610 18/ 32 = 56.30%, 0.1229 Palatnik, Semon: 28/ 41 = 68.30%, 0.0745 23/ 42 = 54.80%, 0.0530 Polgar, Sofia : 158/ 283 = 55.80%, 0.0763 142/ 278 = 51.10%, 0.1100 Rabczewski, Ada: 10/ 21 = 47.60%, 0.1675 13/ 22 = 59.10%, 0.1038 Razuvaev, Yuri : 20/ 60 = 33.30%, 0.0703 34/ 60 = 56.70%, 0.0454 Suba, Mihai : 11/ 22 = 50.00%, 0.0897 12/ 23 = 52.20%, 0.0404 Totals for all players in SofiaPolgarRome1989Hou3d17: 300 / 561 = 53.48% Aggregate difference in SofiaPolgarRome1989Hou3d17: 52.1983 / 561 = 0.0930 ------------------------------------------------------------------------------ Still hallowed after all these years: 8.5/9, FIDE performance about 2900, but by my rough measure "only" 2400, with her opponents playing rather generously. To be sure, winning a game depends on more than the abstract quality of moves. The overlooked 31...Rxc2! shot in Anand-Kasparov 1995 game 11 is trivial for a computer but is one of the game's great sporting moments, created by Garry by applying pressure on the heels of the Game 10 masterpiece. But this point is in the opposite direction from any expressed concern about computer use. ------------------------------------------------------------------------------- Report from TateCroatia2010Hou3d17, excluding repeats and |prev-eval| > 3 First-line matches to Houdini_3_Pro_x64 1-cpu at depth 17: Player Name Matches/Turns = Pctg., AE Opponents' figures ------------------------------------------------------------------------------- Brkic, Ante : 35/ 53 = 66.00%, 0.0340 35/ 54 = 64.80%, 0.0708 Cvitan, Ognjen : 3/ 5 = 60.00%, 0.0812 3/ 6 = 50.00%, 0.0443 Drazic, Sinisa : 7/ 14 = 50.00%, 0.1092 4/ 14 = 28.60%, 0.0616 Lenic, Luka : 30/ 52 = 57.70%, 0.0261 31/ 53 = 58.50%, 0.0214 Loncar, Robert : 9/ 23 = 39.10%, 0.1735 13/ 23 = 56.50%, 0.0452 Nikolov, Sasho : 27/ 61 = 44.30%, 0.0938 28/ 63 = 44.40%, 0.0622 Predojevic, Bor: 24/ 32 = 75.00%, 0.0508 15/ 31 = 48.40%, 0.0619 Tate, Alan : 150/ 291 = 51.50%, 0.0615 156/ 287 = 54.40%, 0.0835 Yilmaz Mustafa : 12/ 29 = 41.40%, 0.1206 8/ 29 = 27.60%, 0.1138 Zufic, Miroslav: 9/ 18 = 50.00%, 0.2242 13/ 18 = 72.20%, 0.0900 Totals for all players in TateCroatia2010Hou3d17: 306 / 578 = 52.94% Aggregate difference in TateCroatia2010Hou3d17: 41.8509 / 578 = 0.0724 ------------------------------------------------------------------------------- Tate's opponents---his only loss was to Predojevic---matched more than he did. Roughly 2625, opponents 2300. ------------------------------------------------------------------------------- Report from HermanMilanOpen2010Hou3d17, excluding repeats and |prev-eval| > 3 First-line matches to Houdini_3_Pro_x64 1-cpu at depth 17: Player Name Matches/Turns = Pctg., AE Opponents' figures ------------------------------------------------------------------------------- Agrifoglio, Fab: 4/ 8 = 50.00%, 0.2687 5/ 9 = 55.60%, 0.0362 Borgo, Giulio : 30/ 53 = 56.60%, 0.0654 31/ 54 = 57.40%, 0.0567 David, Alberto : 8/ 23 = 34.80%, 0.2625 10/ 24 = 41.70%, 0.1896 Garano, Nicola : 14/ 29 = 48.30%, 0.1279 20/ 30 = 66.70%, 0.0805 Herman, Matthew: 145/ 262 = 55.30%, 0.0683 134/ 259 = 51.70%, 0.1007 Manca, Federico: 23/ 35 = 65.70%, 0.0338 20/ 34 = 58.80%, 0.0616 Marin, Mihail : 3/ 7 = 42.90%, 0.0628 2/ 7 = 28.60%, 0.0210 Saric, Sinisa : 15/ 41 = 36.60%, 0.1049 24/ 41 = 58.50%, 0.0471 Solodovnichenko: 17/ 32 = 53.10%, 0.0456 20/ 31 = 64.50%, 0.0335 Tratar, Marko : 20/ 31 = 64.50%, 0.1080 13/ 32 = 40.60%, 0.0726 Totals for all players in HermanMilanOpen2010Hou3d17: 279 / 521 = 53.55% Aggregate difference in HermanMilanOpen2010Hou3d17: 43.9766 / 521 = 0.0844 ------------------------------------------------------------------------------- Roughly 2525, opponents only 2000 on raw figures, but notice the weirdly huge figures from F. Agrifolglio and Alberto David. My IPR regression smooths those out. Plus it runs on all 9 games at once.