Fault injection

The results of the fault injection campaign are interpreted by undertaking the following decisions failure mode acceptance, failure mode handling or failure mode elimination. 

Both developers and testers use a wide variety of tools to help them test their software. Debugging versions of a product typically contain tens of thousands of checked assertion statements. Various program analysis tools have been developed to detect such things as the use of uninitialized variables. Debugging versions also typically contain code that checks for memory allocation errors and corrupted data structures. Yet another technique used is fault injection. Code is added to a system to artificially cause faults to occur in various subsystems and to produce incorrect input parameters to and output results from called functions. 

Since diagnostics are such a critical variable in the calculations, the ability to measure and evaluate the effectiveness of the diagnostics is important. This is done using an extended failure modes and effects analysis technique (Ref. 9) and verified with fault injection testing (Ref. 10 and 11). The techniques were refined to include multiple failure modes (Ref. 12) and today are commonly used to evaluate diagnostic capability and failure mode split (Ref. 13). 

The FMEDA can also be used as a guide when selecting manual fault injection test cases. If 100% testing is not done, the tests should be done on components with the highest failure rate first. These contribute the most to the coverage factor. Test cases are also chosen when the actual result of the FMEDA is in doubt. 

Table 5.1 Percentage of number of error from fault injection results for PODER fault tolerant technique in miniMIPS running the matrix mirltiphcation 66...

Table 5.3 Number of faults injected by simulation fault injection in miniMIPS protected by OCFCM and the percentage of detected errors 67...

Table 7.2 Fault injection by partial reconfiguration in SRAM-based FPGA 84...

This book exposes the reasons behind the concerns surroimding a system s fault tolerance. Some of the most irrrportarrt farrlt tolerance techniques presented in the literature are analyzed arrd three novel hybrid techniques to detect farrlts in processors are presented in detail. It also covers fault injection techniques appUed to the novel techrriques, including farrlt injection by simrrlation, configuration bitstream farrlt injection, and neutron arrd Cobalt-60 beams irradiation experimerrts. 

The book is organized as follows Chap. 2 presents the terminology and general concepts used in this work. Chapter 3 describes existing fault tolerant techniques for processors presented in the literature. Chapter 4 describes the fault tolerant techniques implemented in this work to detect transierrt fairlts in processors, from which two are known software-based and three are new lybrid techniques. Chapter 5 presents experimental fault injection campaigns for the implemented fairlt tolerarrt techniques. Chapter 6 presents the configuration bitstream fairlt injection campaign and results. Chapter 7 presents radiation experiments on some of the proposed techniques. Chapter 8 describes future work and concludes the book. 

Soft errors can be detected and corrected by the system s logic, meaning that it does not require a hard reset to recover from an error. Sections 7.1.2 and 7.2.2 present neutron irradiation experiments simulating the effect of SEE in Flash-base and SRAM-based FPGAs, while Chaps. 5 and 6 present fault injection simulation experiments simulating SEEs at RTL level and in the configuration memoiy bitstream, respectively. In this work, SEUs and SETs will be used to describe transient faults that the proposed techniques can cope with. 

Although the effect of faults is increasing, the rate is not yet sufficient to test fault tolerant techniques at ground level. In order to do so, fault emulation and testing is necessary. In this Section, we will go over a few options to do so, such a software fault injection by simulation, fault injection in the FPGA s memory configuration bitstream and irradiation experiments. 

Fault injection by simulation can be done by injecting faults at logical or electrical levels in conunercial simulators, such as ModelSim from Mentor or iSim... 

FPGA Memory Configuration Bitstream Fault Injection... 

Alternatively, circuit instrumentation can be used for fault injection. Circuit instrumentation consists in inserting extra hardware modules, also called instruments. They can provide external controllability and observabihty to inject a fault and observe its effects. Circuit instrumentation is an automatic process that is basically... 

As mentioned in the previous Chapters, fault injection campaigns can be performed in different levels. The fault injection by simulation is performed by adding upsets when simulating a given system in commercial simulators, such as ModelSim from Mentor or iSim from Xilinx, or open source simulators, such as Spice. They are able to provide the best control over the experiment. This characteristic is because the simulation can be stopped at any simulation time and the internal values from anywhere inside the system can be read and stored for post-analysis. As a consequence, the amount of data is the biggest among the different levels of available fault injection campaigns types. 

As a drawback, fault injection by simulation requires considerable computational power and simulation time, since huge amounts of data must be processed, relative to the size of the circuit and its complexity. As an example, the injection of 100,000 faults in the mimMlPS may take up to a few days to finish, depending on the simulator and the application running on the processor. 

In order to perform the fault injection campaign by simulation and test the fault tolerant techniques from Chap. 4, we used the fault injector proposed and described in Azambuja (2010b), which uses the commercial simulator ModelSim. The main reasons for using this fault injector were accessibility to its source code and the possibility to automate the process that could take up to weeks of computatiom... 

The fault injector tool used in this fault injection campaign has three files as inputs (1) a fault definition file, that contains the number of faults to be injected... 

After each fault injection and application run, results are collected and analyzed by the script. As a result, we have the number of injected faults and their effect on the system. 

In the following, we describe the simulation fault injection campaign for each technique. 

The fault injection campaign was performed automatically. At the end of each execution, the results stored in memory were compared with the expected correct values. If the results matched, the fault was discarded. The amoimt of faults masked by the program is application related and it should not interfere with the analysis. In the end, only faults not masked by the application were considered in the analysis. When 100% signal coverage was achieved and at least 4 faults per signal were detected we normalized the faults, varying from 4 to 5 faults per signal. Those faults were then used to build the test case list. 

The resulting faults were classified by their somce and effect on the system. We defined fom groups of fault sources to inject SEU and SET types of faults data path, control path, register bank and ALU. Program and data memories are assumed to be protected by ED AC and therefore were not upset during the fault injection campaign. 

---