PGN Annotator

Update version 1.2

Changes marked with the yellow new icon.


Chess-Artist is a very nice program to annotate PGN games. Annotate follows a different path, it stores and keeps the analysis results in a multi functional database, the Polyglot book format. The current main Polyglot book database contains over 51 million positions with the scores and depths from top engines and with that we can annotate (your) PGN games in a blink of an eye. Consider the immortal game of Anderssen vs Kieseritzky and the 3 output modes of Annotate.


1. Blunder check

2. Full annotation

3. Full annotation with long engine name

__________________________________________________________________________________________________

Operation

Annotate PGN - Select a PGN, there after the Book Database and choose the output mode. For example select morphy.pgn and annotator.bin and all games of Morphy are stored in the folder pgn\output.pgn


Basically this is what the program is about, unless you want more, for instance:


  1. Analyze and store your own games with any UCI engine to improve your play, a personal example.
  2. Analyze and store the games of your favorite (or own) engine to improve its play.
  3. Extend the current Book Database with more analyzed positions.
  4. Create your own analyzed Book Database.

Analyze your own games or engine games


Make Analyzed Book - Select the PGN and wait till the engine menu arrives, manage the engine settings (see example) and when ready press "S". For example, select "ProDeo.pgn", press "S" and the Book Database "ProDeo.bin" will be created. Then run Annotate PGN and figure out where ProDeo 2.2 (or its opponent) went wrong.


Merge Books - Will merge 2 books into one, the output is always book.bin


Make Engine Book - Create a Book Database from annotated engine-engine matches (with depth and score). Example: select ccrl-3500.pgn (only 32 games), a book ccrl-3500.bin will be created and you are prompted to make an engine name selection. Type a part of the engine name (Stoc for Stockfish or Dragon for Komodo Dragon) is suffiecient to store all the scores and depths of that particular engine in the book. Type a * in case you want all games.


Make Player PGN - Create a specific Player PGN. For instance, create a PGN with the games of Kasparov from MillionBase 3.5, type Kasparov and kasparov.pgn will be created. Use the PGN (for instance) with Make Analyzed Book.


Make Players PGN - Does the same as above but for multiple players at the same

time. Consider the file players.txt in the root folder, 156 top players from the past and present. Using this function with MillionBase 3.5 will create a (large) PGN with the games these players played against each other. Example, in case you only want the games of Kasparov vs Karpov create a small players.txt file with only their names.


Find Position - also direct reachable via the [F3] function key is a powerful feature to 1) instantly find a position (as EPD taken from the clipboard) in the Book Database, 2) analyse this position with your favorite UCI chess engine (default is Stockfish 12) and 3) post the outcome on a (chess) forum with a simple copy&paste. Example. To configure (change) engines press [F1]. Note, if the position is not in the Book Database the annalysis and forum post functions still work.

__________________________________________________________________________________________________


Database management


Engine coding - With such a large Book Database (currently 51 million positions) analyzed by various engines it's important to classify each position. This is maintained in the text file engines.txt by giving each position a number. As one can see the Book Database mostly contains Stockfish 12 analysis. It's adviced after an analysis job (Make Analyzed Book and Make Engine Book) to code the new created book with the engine it has been analyzed with this function, the engine.txt will show up to make the decision easy. The engine code is used in the main function Annotate PGN to list the analysis engine. Positions without an engine code are listed with a * (star) only. Of course you are totally free to create your own Book Database with its own engine.txt file.


Book Statistics - Shows the internals of Book Database, the supplied 51 million Book Database as an example.


Remove Book voids - This option is reachable via the [F2] function key and weeds out zero weight book entries (see Book Statistics) in order to save disk space. Zero weight book entries are positions which are never played and thus unusable. A new book is create as books\new.bin.

________________________________________________________________________________________________

Distributed Examples

as found in the download

PGN

Games

Description

Alekhine, Alexander

1.596

Most of his games.

AlphaZero vs Stockfish 8

10

The 10 cherry-picked games DeepMind provided.

Anderssen, Adolf

681

Most known from his immortal game againsts Lionel Kieseritzk in 1851.

DeepBlue-Kasparov

12

The 12 games (1996 and 1997) that took the chess world by surprise.

Immortal games

68

The best chess has to offer.

Morphy, Paul

211

The legend of romantic chess.

To try yourself, choose Annotate -> Alekhine.pgn -> Alekhine.bin -> Start

___________________________________________________________________________________________________


A few examples to highlight the Find Position | Analyze | Post option.


2r2rk1/pp3pp1/4bb1p/q2p1P1Q/3P4/2N5/PP4PP/1K1R1B1R b - - bm Rxc3; c1 "Pillsbury - Lasker 1896";  [ view ]

b2r3r/k4p1p/p2q1np1/NppP4/3p1Q2/P4PPB/1PP4P/1K1RR3 w - - bm Rxd4; c1 "Kasparov - Topalov 1999"; [ view ]

2rq2k1/4bppp/p1rp4/1p1NpP2/4P3/2PQ4/PP4PP/3R1R1K w - - bm Ra1!; c1 "Fischer - Olicio Gadia 1960"; [ view ]

Remarks


1. There are 5 annotation symbols.

?! - when score drops with 0.50

?  - when score drops with 1.00

?? - when score drops with 2.00 or more

!  - when score increases with 1.00

!! - when score increases with 2.00 or more


2. When you have an egine running in Find Position don't abort the program with ctrl_c or ctrl_break. Stop the engine first, else the engine keeps on running.


3. The big Book Database can be used as a Polyglot book but lacks the tuning of POLY 1.7


4. On some Windows systems starting Annotate will show up much to small, it's explained in 2 screenshots how to fix that permanently. [ shot-1 ] [ shot-2 ]

Download section


Two downloads, the full version with the big 51 million Book Database (size 443 Mb) and the same version with a small Book Database of 6.3 million positions (size 76 Mb).

Annotator Light

76 Mb

Credits

Annotator 1.0

443 Mb

Ferdinand Mosca

for MEA

Tord Romstad, Marco Costalba and Joona Kiiski

Fabien Letouzey

for Polyglot

for the use of PGN's.

_________________________________________________________________________________________________


Version 1.2

Overview Function-Keys [ F1 ] ... [ F11 ]

KEY

Description

Status

F1

Engine Configuration - select analysis engine, number of threads, hash table size.


F2

Remove Book voids - remove positions that are not analyzed and thus have little value.


F3

Find Position - copy a valid EPD in the clipboard and press [F3].


F4

Online help - moves you to this page.


F5

Quickly count the games in a PGN.


F6

Create selected opening books - from a large PGN create a new PGN with the opening moves of your choice. For instance, create (or open) the textfile opening.txt and type: "e2e4 e7e5 g1f3 d7d6 ", save and press [F6] select the PGN and all the games of the Pirc-Defence are stored in opening.pgn


F7

Create an analyzed Polyglot opening book from analyzed EPD files - as long as the EPD's are valid and contain the bm | ce  and acd tags this will work. Press [F7] select the EPD and a corresponding Polyglot *.bin book is created, i.e. test.epd becomes test.bin.


F8

Remove Blunders - remove obvious blunders from a Polyglot database. Based on the highest engine score remove those moves that score xx centipawns less. xx by default is set to 200 centipawns. For example: using a value of 50 will remove many moves (1.g4 | 1.b4 etc.) from the start position. A NEW.BIN database will be created without making changes to the original database, safety first.


Important - For Polyglot databases larger than 320Mb use the separate utility poly-blunders available in the download below. It can handle Polyglot books of 3Gb.


F9

Remove Engines - Suppose your current Polyglot Database looks like this example and you want to remove the Houdini-6 analysis from the database. Type "3" and a NEW.BIN database will be created without the Houdini-6 analysis without making changes to the original database, safety first.


F10

Rename Engines - Suppose your current Polyglot Database looks like this example and you want to combine the Stockfish 12 and Stockfish 13 analysis as one Stockfish. Type "10" (Stockfish 13) and thereafter "1" (Stockfish 12) and a NEW.BIN database will be created where the Stockfish 13 analysis is combined with the Stockfish 12 analysis. And of course without making changes to the original database, safety first.


Then do a "Book Statistic" from the menu and check if everything went to your expectation before you rename NEW.BIN.


F11

Create opening sets for engine-engine testing. From a good PGN create flexible opening sets. Included in the Annotator-Basic version (download below) are 2 PGN's (gm-2600.pgn and gm-2700.pgn) with only games between 2600 and 2700 elo rated grandmasters. Press [F11], select the PGN and type the number of moves and a corresponding PGN opening set will be created. In the gm-2600 and gm-2700 folder are the 2-moves.pgn .... 10-moves.pgn already created. Examples [ 1 ] [ 2 ] [ 3 ]


Instead of a new large download only replace the old annotate.exe with the new one or download the basic version.

Annotator 1.2 Basic

44 Mb

_________________________________________________________________________________________________

Book Database updates


The main database is frequently updated with the latests analysis, since late february 2021 with Stockfish 13.

Update

Size

Games used

Positions

Date

Content

34 Mb

76.348

4.006.433

2021-03-03

57 Mb

240.605

6.990.662

2021-03-14

regular.7z


To update the main database annotator.bin run Merge Books from the menu and combine it with an update file, for instance players.bin (see the documentation above). Note that Merge Books takes considerable time to finish. A more easy way is to download the (below) latest updated version of annotator.bin and/or use it in combination with ProDeo Knowledge.

players.7z - contains the full games of 40 legendary players from Morphy till Carlsen analyzed with Stockfish 12 and since the recent release with Stockfish 13. In total 4 million positions.


gambits.7z - contains many unorthodox openings and gambits as listed on the Wikepedia. The purpose of these openings is that they hardly get attention in the standard books of engines but may serve as a starting point for correspondence chess. Books are created from MillionBase 3.45 and Correspondence 2.0 and are analyzed with Stockfish 13 at depth=20 about ~3300 elo.


regular.7z - contains popular openings played. Books are created from MillionBase 3.45 and Correspondence 2.0 and are analyzed with Stockfish 13 at depth=20 about ~3300 elo.

________________________________________________________________________________________________

Gambits and unorthodox openings

Statistically the white score is 54%


Book

Positions

Score - white pov

Moves

334.473

58.72%

1.d4 d5 2.e4 dxe4

105.160

54.34%

1.e4 e5 2.Nf3 d5

1.078.737

54.92%

1.d4 Nf6 2.c4 c5 3.d5 b5

6.184

37.71%

1.e4 Nf6 2.Bc4 Nxe4 3.Bxf7+

721.004

47.8%

1.f4

1.504.434

55.83%

1. d4 f5

222.540

58.54%

1.d4 d5 2.c4 e5

237.553

55.3%

1.e4 e5 2.Nf3 Nc6 3.Bc4 Bc5 4.b4

84.232

54.87%

1.e4 e5 2.d4 exd4 3.c3

19.489

45.5%

1.e4 e5 2.d4 exd4 3.f4

739.071

49.54%

1.e4 e5 2.f4 exf4

74.192

53.76%

1.d4 Nf6 2.c4 e6 3.Nf3 c5 4.d5 b5

532.675

58.1%

1.d4 Nf6 2.c4 e5

21.759

48.55%

1.e4 e6 2.d4 d5 3.e5 c5 4.Qg4

62.219

52.51%

1.e4 e6 2.Nf3 d5 3.e5 c5 4.b4

58.366

46.82%

1.e4 e6 2.f4

75.361

56.67%

1.d4 e5

14.411

56.73%

1.e4 e5 2.Nf3 Nc6 3.Bc4 Nd4

6.204

56.13%

1.e4 e5 2.Nf3 Nc6 3.Bc4 f5

171.875

55.38%

1.e4 e5 2.f4 d5

314.117

58.8%

1.e4 e5 2.Nf3 f5

156.533

52.3%

1.e4 c5 2.Nf3 d6 3.d4 cxd4 4.Nxd4 Nf6 5.Nc3 a6 6.Bg5 e6 7.f4 Qb6 8.Qd2 Qxb2

57.575

49.54%

1.e4 e5 2.Nf3 Nf6 3.Nxe5 d6 4.Nxf7

111.611

55.67%

1.e4 e5 2.Nf3 Nc6 3.d4 exd4 4.c3

332.602

52.33%

1.e4 e5 2.Nf3 Nc6 3.d4 exd4 4.Bc4

__________________________________________________________________________________________________


Regular openings

Total games used : 1.729.754

Book

Positions

Games used

Moves

598.161

19.200

1.e4 c5 2.Nf3 d6 3.d4 cxd4 4.Nxd4 e5

1.116.467

54.880

1.e4 c5 2.Nf3 Nc6 3.d4 cxd4 4.Nxd4 Nf6 5.Nc3 e5 6.Ndb5 d6

365.313

11.638

1.e4 c5 2.Nf3 d6 3.d4 cxd4 4.Nxd4 Nf6 5.Nc3 e6

167.164

5.044

1.e4 c5 2.Nf3 d6 3.d4 cxd4 4.Nxd4 Nf6 5.Nc3 a6 6.g3

251.200

8.245

1.e4 c5 2.Nf3 d6 3.d4 cxd4 4.Nxd4 Nf6 5.Nc3 a6 6.f4

510.383

17.441

1.e4 c5 2.Nf3 d6 3.d4 cxd4 4.Nxd4 Nf6 5.Nc3 a6 6.Bc4

1.391.532

60.416

1.e4 c5 2.Nf3 d6 3.d4 cxd4 4.Nxd4 Nf6 5.Nc3 a6 6.Bg5

1.329.161

53.198

1.e4 c5 2.Nf3 d6 3.d4 cxd4 4.Nxd4 Nf6 5.Nc3 g6

476.516

21.075

1.e4 e5 2.Nf3 d6

582.184

28.008

1.c4 c5

1.245.604

54.654

1.c4 e5

221.028

12.616

1.e4 e5 2.Nf3 Nc6 3.Bb5 a6 4.Bxc6

427.582

18.428

1.e4 e5 2.Nf3 Nc6 3.Bb5 a6 4.Ba4 Nf6 5.O-O Nxe4

113.295

3.481

1.e4 e5 2.Nf3 Nc6 3.Bb5 a6 4.Ba4 Nf6 5.O-O b5 6.Bb3 Bb7

180.980

13.559

1.e4 e5 2.Nf3 Nc6 3.Bb5 a6 4.Ba4 Nf6 5.O-O Be7 6.Re1 b5 7.Bb3 O-O 8.c3 d5

100.266

3.184

1.e4 e5 2.Nf3 Nc6 3.Bb5 a6 4.Ba4 Nf6 5.O-O Be7 6.Re1 b5 7.Bb3 O-O 8.a4, h3, d4, d3

562.369

21.546

1.e4 e5 2.Nf3 Nc6 3.Bb5 a6 4.Ba4 Nf6 5.O-O Be7 6.Re1 b5 7.Bb3 d6

949.226

41.679

1.d4 Nf6 2.Bg5

239.738

18.215

1.d4 Nf6 2.c4 c5 3.d5 e6 4.Nc3 exd5 5.cxd5 d6

708.421

43.203

1.d4 Nf6 2.c4 g6 3.Nc3 d5

607.289

39.087

1.d4 Nf6 2.c4 e6 3.g3

1.649.730

88.836

1.d4 Nf6 2.c4 e6 3.Nf3

242.704

6.472

1.d4 Nf6 2.c4 e6 3.Nc3 Bb4 4.Nf3

179.726

5.311

1.d4 Nf6 2.c4 e6 3.Nc3 Bb4 4.a3

173.002

5.195

1.d4 Nf6 2.c4 e6 3.Nc3 Bb4 4.Bg5

328.108

10.371

1.d4 Nf6 2.c4 e6 3.Nc3 Bb4 4.f3

468.777

26.107

1.d4 Nf6 2.c4 e6 3.Nc3 Bb4 4.Qc2

569.030

31.819

1.d4 Nf6 2.c4 e6 3.Nc3 Bb4 4.e3

288.750

13.307

1.d4 Nf6 2.c4 d6

282.979

15.708

1.d4 Nf6 2.c4 g6 3. Nc3 Bg7 4.e4 d6 5.Be2

249.025

8.778

1.d4 Nf6 2.c4 g6 3. Nc3 Bg7 4.e4 d6 5.f4

447.945

29.070

1.d4 Nf6 2.c4 g6 3. Nc3 Bg7 4.e4 d6 5.f3

399.972

25.097

1.d4 Nf6 2.c4 g6 3. Nc3 Bg7 4.e4 d6 5.Nf3

189.962

10.387

1.c4 Nf6 2.d4 g6 3.g3 Bg7 4.Bg2

264.360

7.675

1.d4 d5 2.c4 Nc6

1.706.149

94.981

1.d4 d5 2.c4 c6

480.650

25.477

1.d4 d5 2.c4 dxc4

537.297

26.611

1.d4 d5 2.c4 e6 3.Nf3

1.024.824

54.297

1.d4 d5 2.c4 e6 3.Nc3

1.070.635

48.418

1.e4 d5

745.969

35.847

1.e4 Nf6

1.216.917

50.840

1.e4 g6

174.799

7.115

1.e4 b6

293.302

11.130

1.e4 Nc6

122.059

5.852

1.e4 c6 2.c4

462.385

24.754

1.e4 c6 2.d4 d5 3.exd5

602.798

27.796

1.e4 c6 2.d4 d5 3.e5

118.698

5.611

1.e4 c6 2.d4 d5 3.Nc3 dxe4 4.Nxe4 Nf6

110.848

6.189

1.e4 c6 2.d4 d5 3.Nc3 dxe4 4.Nxe4 Nd7

285.166

26.495

1.e4 c6 2.d4 d5 3.Nc3 dxe4 4.Nxe4 Bf5

331.053

15.887

1.e4 e6 2.d4 d5 3.exd5

377.253

19.947

1.e4 e6 2.d4 d5 3.e5 c5 4.c3

405.010

28.442

1.e4 e6 2.d4 d5 3.Nd2 c5

305.671

20.945

1.e4 e6 2.d4 d5 3.Nd2 Nf6

197.978

11.296

1.e4 e6 2.d4 d5 3.Nc3 dxe4 4.Nxe4

730.384

39.666

1.e4 e6 2.d4 d5 3.Nc3 Bb4

536.463

30.651

1.e4 e6 2.d4 d5 3.Nc3 Nf6

357.383

14.946

1.e4 e5 2.Nc3

333.300

13.931

1.e4 e5 2.Bc4

215.093

9.825

1.e4 e5 2.d4

319.897

19.076

1.e4 e5 2.Nf3 Nf6 3.Nxe5

230.509

15.360

1.e4 e5 2.Nf3 Nf6 3.d4

57.166

1.622

1.e4 e5 2.Nf3 Nf6 3.Bc4

196.653

5.734

1.e4 e5 2.Nf3 Nc6 3.c3

447.695

23.910

1.e4 e5 2.Nf3 Nc6 3.d4 exd4 4.Nxd4

575.168

30.260

1.e4 e5 2.Nf3 Nc6 3.Bc4 Bc5

108.335

4.190

1.Nf3 d5 2.b3

468.048

20.075

1.Nf3 d5 2.c4

715.313

32.954

1.Nf3 d5 2.g3

443.125

16.986

1.b3

219.190

8.365

1.b4