The Blind Tests

For each blind test, recently determined, but unpublished, crystal structures were collected from crystallographers kind enough to delay publication of their structures. From these, a set of molecules were chosen for the test, and their structural diagrams (Table 2.1) were circulated to the participants. The experimentally determined crystal structures were withheld for 6 months by an independent referee, during which time predictions had to be submitted each participant was asked to propose three crystal structures, in order of preference, for each molecule. A prediction was considered to be successful if one of the three submitted structures was an adequate representation of the experimentally determined crystal structure [7]. 

A few things can be said about the overall results of the four blind tests (Table 2.2) there has been some success for rigid molecules, although the predictability of the different category 1 and 2 crystal structures is variable. Molecular flexibility is a serious obstacle for current methods of crystal structure prediction only one category 3 success was achieved in the first three blind tests and, while more success was achieved for the flexible molecule in the latest test, this was partly due to the more restricted molecular flexibility of the molecule chosen that year [19]. 

Category 2 - Rigid molecules with challenging functional groups  

In judging the success of ranking methods, we should discard those entries that did not produce the structure in the first step. Instead, we ask how many of those that did generate the correct structure ranked it within their top three Category I success rates were 3 of 10 = 30% (IV), 5 of 11 = 45% (Vni), 0 of 4 = 0% (XI) and 4 of 10 = 40% 

---

Conclusive determinations of accuracy and upper age limits cannot be done until more extensive data are available. However, some rough estimates can be made for cotton from the data presented here. The consistency of the trend for the 814-day-old, 47-year-old, and 400-year-old samples indicates that the shifts should be quite consistent from one sample to the next. It is reasonable that these shifts will be repeatable to within 0.01. The slopes for the aging curve can be estimated from a curve drawn through the data, including the blind test datum. The resulting uncertainty in the age determination can then be estimated. The results are shown in Table V. This table indicates that the method... 

Over the life of the project, five samplings were taken after burial of the mines, in environmental conditions ranging from very hot and dry to moderately cold and damp. In every sampling both systems detected the presence of explosive contamination. Even three days after burial of the mines, both systems detected the presence of mines in the blind test area. This was a surprise to the team, because it was expected that there would not have been time for explosives to leach from the mines to the soil surface. In general, there was an increase in contamination of the area with time, with more positive samples being obtained as the time the mines were in the ground increased. 

In the proximity area, samples were taken along and 2 m to each side of 3, 7, and 11-m radii marked around each mine during each sampling event. Fido and the MECHEM canines routinely detected contamination up to 11 metres from the mine centres. Because of the layout of the test field (the mines were only 30 metres apart), it was impossible to determine if contamination spread past 11 metres from the mines. Results from the blind test area suggest that contamination spread more than 11 metres, but it was... 

The goal of these tests was to determine whether both systems were effective in detecting areas that were mined. Based on the test results, it was determined that both systems could detect mined areas. In retrospect, the blind test area probably contained too many mines and did not contain a large area that was free of mines. Because of the large number of mines in the area, contamination of the test area was widespread. Hence, it was not possible to delineate a mined area from a non-mined area. It should be again noted that both systems found the area to be free of contamination prior to emplacement of the mines. 

Table 1 Overview of programs developed for organic crystal structure prediction by searching for minima in the lattice energy. The types of molecules for which the program was originally developed are given, though all programs with emboldened names were used in the blind tests and so have been used for a wider range of systems (Fig. 4).

Fig. 4 The molecules used in the blind tests of crystal structure prediction, organized by the Cambridge Crystallographic Data Centre in 1999 and 2001. For each molecule, the success rate is given as x/y, where x is the number of successful predictions, and is the number of groups that submitted (usually) three guesses for the crystal Structure.

One approach is to assume that these kinetic factors are in some way represented in experimental crystal structures. Thus, analyses using the information in the Cambridge Structural Database were developed.Although there have yet to be successful independent predictions by this method in the blind tests, this approach holds promise for discriminating among the hypothetical crystal structures within the energy range of potential polymorphism and force field uncertainty. 

The Cambridge Crystallographic Data Centre is thanked for arranging the blind tests, which advanced the area of crystal structure prediction and provided an objective test of the progress. My postgraduate students. Graeme Day and Theresa Beyer, are thanked for providing data for this article. 

Table 2.2 Success rates in the blind tests of crystal structure prediction ...

The improvement in results with more accurate electrostatic models can also be seen by studying the blind test results in more detail. For example, the observed crystal structure of molecule VIII contains molecular tapes based on / (8) hydrogen bond dimers... 

The success of crystal structure generation methods for rigid molecules has been quite high across the blind tests, with most of the participants locating the observed structure somewhere in their lists for most molecules in categories 1 and 2. The notable exceptions are molecules XI, IX, and xm (Table 1). 

Figure 6 Target molecules in the first four blind tests of crystal structure prediction. Molecules are numbered according to the Roman numerals used in the blind test publications. Categories are described in the text.

A second polymorph of molecule IV was discovered after the second blind test. ° Since the new polymorph has two independent molecules in the asymmetric unit, it was not predicted by any of the participants because it was indicated at the start of the blind test that the structure would have just one molecule in the asymmetric unit. Two participants predicted the first polymorph correctly. The DFT(d) method ranks the two polymorphs as the first and second lowest energy structures among all the structures predicted by all the participants. ... 

The bottom line from the present status of the blind tests conducted on crystal structure prediction [9] is that there are now a few positive results for rigid molecules, as compared with none only ten years ago but also that the results have sometimes a scientifically unpleasant taste of haphazardness. Hits and misses come and go on differences of fractions of a kJ mol certainly, as the result of chance rather than of accuracy. 

---