STS revisited

with

STS-SF15

STS-LC0

STS stands for Strategic Test Suite. From the CPW we read -


Strategic Test Suite, (STS) a series of themed test suites by Dann Corbit and Swaminathan Natarajan, designed to evaluate chess engine's long term understanding of strategical and positional concepts. More recently, the positions were revised and converted by Ferdinand Mosca to be used by an analysis tool, MEA.


Created in 2008 and containing 1500 positions in total. Each position was analyzed with the then strongest engines giving points for the four best moves varying from 1 to 10 points. Examples:


1r6/1R2bk2/P3p1pp/4Pp2/1r1P3P/1N2B1P1/5PK1/8 w - - bm Rxb4; c0 "Rxb4=10, Rxb8=9, Na5=7, Nc5=2";
1r6/3n4/p2b3p/2nN1kpP/1BP1p3/3rP3/3NKP2/1R1R4 w - - bm Bxc5; c0 "Bc3=10, Bxc5=8, Rdc1=0, Re1=0";


As such people were able to more or less create their own rating list based on these 1500 positions, not as good the real rating lists, but not bad either. Anno 2022 the set is completely outdated, current top engines play ~500 elo stronger than those in 2008.


We made an effort to give STS a major update by re-analyzing the 1500 positions with nowadays strongest engines, Stockfish 15 and Lc0. And with MEA you can produce your own alternative rating list. See [ SF15 ] [ LC0 ] or as text file [ SF15 ] [ LC0 ].


INSTALLATION

The STS package comes in 2 folders, STS-SF15 and STS-LC0, the 1500 positions in each folder are analyzed at 15 seconds per move, SF15 using 4 threads and Lc0 using version 28.0 with the advised 130Mb net for that time control.


OPERATION

Operation goes via the provided sts.bat file. For a first smell double click on sts.bat and it will run the 6 pre-installed engines and produce a html (automatically updated in your browser) and a text file mea-result.txt for posting results.


Adding new engines - open the file sts.bat with a text editor.

set MT=100
set HASH=64
set THREADS=1
set PROTOCOL=uci
set EPD=epd\sts-sf15.epd
set MRL=mea_results


set EXE=engines\Koivisto_8.0.exe
set NAME=Koivisto_8.0


set MT stands for "move-time", here you can change the average time in milliseconds, 1000 is one second.

set EXE defines the place of the engine, engines typically are stored in the engines folder.

set NAME is the name you want the engine named in the output.


Add as many engines as you wish in one run, see the 6 engines example.


Hints

1. Don't change the blue listed paramters unless you know what you are doing.

2. Several engines don't handle low time controls (such as 100ms) well, resulting in bad results. Advice use 1000ms.

_________________________________________________________________________________________________


Delete engines - results are kept in the file mea-results.csv and here you can make changes or delete engines.


make-html.bat - double click for new html or text output, for instance if you have made changes to mea-results.csv.



Credits

Ferdinand Mosca

The programmers for the fair use of their engines