Blind Tests of Crystal Structure Prediction

A series of blind tests of crystal structure prediction has been organised by the Cambridge Crystallographic Data Centre to evaluate the development of crystal structure prediction tools on an ongoing basis. So far, the tests have taken place in 1999, 2001, 2004, 2007 and 2010. The results of the 2010 blind test will not be discussed here as they have not yet been published. In each test, the molecular structures of three or four compounds are provided to a number of invited participants who have up to six months to submit a maximum of three predicted crystal structures for each compound. The experimental crystal structures have been determined, but are kept hidden from the participants. After the predictions have been submitted to an independent referee, the experimental structures are disclosed and compared to the various predicted structures. The participants then gather at a workshop to discuss the results and a joint publication is produced. 

Second polymorph discovered after blind test 

Not blind (partial structure discovered midway blind test) 

B not part of blind test C not part of blind test 

---

Fig. 4 The molecules used in the blind tests of crystal structure prediction, organized by the Cambridge Crystallographic Data Centre in 1999 and 2001. For each molecule, the success rate is given as x/y, where x is the number of successful predictions, and is the number of groups that submitted (usually) three guesses for the crystal Structure.

Table 2.1 Diagrams of the molecules included as targets in the four blind tests of crystal structure prediction... 

FIGURE 5.4 Molecule XX (benzyl-(4-(4-methyl-5-(p-tolylsulfonyl)-l,3-thiazol-2-yl)phenyl) carbamate) from the fifth blind test of crystal structure prediction, (a) Chemical diagram, (b) overlay of one of the 48 database generated conformations (red) with the conformation in the observed crystal structure, (c) overlay of the CSP global minimum in lattice energy (green) with the observed structure from X-ray diffraction. Source Kazantsev et al. [29]. Reprinted with permission of Elsevier, (see insert for color representation of the figure.)... 

The Cambridge Crystallographic Data Centre is thanked for arranging the blind tests, which advanced the area of crystal structure prediction and provided an objective test of the progress. My postgraduate students. Graeme Day and Theresa Beyer, are thanked for providing data for this article. 

A few things can be said about the overall results of the four blind tests (Table 2.2) there has been some success for rigid molecules, although the predictability of the different category 1 and 2 crystal structures is variable. Molecular flexibility is a serious obstacle for current methods of crystal structure prediction only one category 3 success was achieved in the first three blind tests and, while more success was achieved for the flexible molecule in the latest test, this was partly due to the more restricted molecular flexibility of the molecule chosen that year [19]. 

Unbeknown to the participants of the first blind test, the crystal structures of two polymorphs of molecule I had been determined, a Plfc form and a meta-stable Pbca form. The latter form was predicted correctly by four participants, but the stable Plfc form was not found by any of the participants as one of their three predictions. Subsequent studies using molecular mechanics calculations to compute the lattice energies of the two polymorphs more accurately failed to obtain a satisfactory result for the stable form. Using solid state quantum mechanics, however, it was found that only about 0.1 kcal mol separates the calculated lattice energies of the two polymorphs, indicating that either form may be observed experimentally. ... 

The state of the art in crystal structure calculation is tested every few years in the crystal structure prediction blind tests organised by the Cambridge Crystallographic Data Centre (CCDC). As of the... 

Motherwell WDS, Ammon HE, DunitzJD, Dzyabchenko A, Erk P, Gavezzotti A, Hofmann DWM, Leusen I, LommerseJPM, Mooij WTM, Price SL, Scheraga H, Schweizer B, Schmidt MU, van Eijck BP, Verwem P, and Williams DE. Crystal Structure Prediction of Small Organic Molecules A Second Blind Test. Acta CrystB 2002 B58 647-661. 

Table 1 Overview of programs developed for organic crystal structure prediction by searching for minima in the lattice energy. The types of molecules for which the program was originally developed are given, though all programs with emboldened names were used in the blind tests and so have been used for a wider range of systems (Fig. 4).

Fig. 1 A plot of the lattice energy minima found in the crystal structure prediction search for the rigid CHNO molecule in the 2001 blind test (Fig. 4). in the search by Price. This diagram illustrates the plurality of distinct minima in different space groups found in a relatively sparse search, which did not locate the experimental structure.

Computational assessment of the likelihoods of occurrence and the relative stabilities of polymorphs is not necessarily more effective than the experimental approach. Whilst great advances have been made in the field of ab initio crystal structure prediction (CSP), as documented in five international blind tests spanning the years 1999-2010 [5], it is still not routinely possible to predict whether a molecule is likely to be polymorphic or to confirm whether the most thermodynamically stable structure has been found experimentally, especially for molecules of the complexity of a typical drug. It is possible to compute the polymorph landscape for a specific flexible molecule, but the calculations require considerable expertise, and the timescales and computing resources can render CSP impractical for application to even a limited portfolio of candidate APIs. 

The bottom line from the present status of the blind tests conducted on crystal structure prediction [9] is that there are now a few positive results for rigid molecules, as compared with none only ten years ago but also that the results have sometimes a scientifically unpleasant taste of haphazardness. Hits and misses come and go on differences of fractions of a kJ mol certainly, as the result of chance rather than of accuracy. 

Several examples of the MOLPAK + WMIN structure prediction procedures are given in the next sections. The problem of identifying the correct crystal structure from literally thousands of possible structures remains. The which is the best/correct solution was an important topic during the 1999 and 2001 CCDC sponsored blind tests [12]. Calculated lattice energies are adequate in many cases, but in others small lattice energy differences make it difficult to separate the wheat from the chaff. Other criteria, such as good or bad patterns of intermolecular contacts in comparison with known crystal structures could be helpful. 

---