Audits/auditing Safety Case

Audits. Safety case audits tend to look at the process rather than the substance of safety cases. 

Notified Bodies, third-party bodies, which are required to audit safety cases via a contracted independent safety auditor. 

Complete audited safety case (including quantitative safety tolerability levels), conducted by a third-party-independent safety assessor, presented to government for final approval. 

The effectiveness with which the commitments in the safety case are being implemented Monitoring the effectiveness of SMS and operator audits of them The degree to which the work force is involved in implementing the Safety Case Regime. 

A management-risk analysis method would not only inform safety cases, it would also be complimentary to an organisation s audit function. For example, a checklist to inform audits could be made of processes or activities that are deemed particularly risky or that rely on assumptions in which there is limited confidence. In addition, the frequency or thoroughness of audits and the focus of safety assessments may also be increased for parts of a management system that are considered similarly risky, or whose failures could lead to particularly severe consequences. In these ways, both the efficiency and the effectiveness of audits and safety assessments could be improved. 

It would be suitable for selective use in management dedsion-making, and managers could easily be trained to use it for that purpose. It could inform audits and safety assurance, and it would provide input to safety cases. 

The safety case is regarded as an increasingly important audit tool and of greater potential value than off-the-shelf safety audit schemes. Local managers are expected to undertake regular safety audits and are externally audited every three years. Railtrack s safety case is audited by its own auditors and also by the Railway Inspectorate, who assess Railtrack s procedures for auditing the train operating companies compliance with their safety cases (HSE, 1997 13 ff.). 

While privatization and the preparation, validation and audit of safety cases have focused the minds of senior management, the effects lower down are felt to be patchy. An industry representative explained ... 

This additional report commissioned by the Deputy Prime Minister expressed concern that the safety and standards directorate of Railtrack was insufficiently distinct from the company s commercial interests and it recommended separation of the safety departments from commercial interests in the company (HSE, 2000c). This report was also critical of accountability in the new system and recommended that HSE should oversee Railtrack s safety operations much more rigorously and should take a much more active role in checking Railtrack s audits of network users. But beyond this it also recommended that HSE should take over some of the safety responsibilities currently held by Railtrack, in particular its responsibilities for accepting and auditing the safety cases of one group of network users, namely the train operating companies. 

The process summary document then identifies the extent to which the outputs of that process have met the safety objectives. Did all of the tests pass If not, what was the pass rate and where are the failures recorded and analyzed Were all of the tests carried out exactly as specified If not what were the deviations and what are the consequences for the safety objectives Were all of the tests completed If not where is it stated which parts of the software are not tested Since this part of the assessment of evidence is concerned with process compliance, additional backing evidence for the safety case can be provided through quality auditing to check that processes are being properly carried out. 

Life is not perfect and neither is evidence. Software will contain faults tests will fail documentation will contain errors and audits will find deviations in processes. The problem for assessment of the safety case is how much of this actually matters to safety. 

When all platforms are designed and operated to the same standards it is relatively easy to audit them. The auditor simply has to look up the appropriate code or rule in order to determine compliance. Such is not the case with a safety case system, where each platform has its own unique program against which it has to be evaluated. 

The importance of audits and how they are carried out has been discussed in Chapter 4. The audits of Safety Cases follow the same general principles, although the word assessof seems to be used in place of auditor, Either way, they are doing the same thing —they are checking that the Safety Case as implemented is meeting expectations. 

Figure 12.7 illustrates a notional representation of typical safety-relevant data and information flows as would be reviewed in the V-shape safety case process illustrated in Figure 12.4. Note that this is just showing some of the safety information exchanges that would occur within an SMS and is not how an SMS is audited. Of course, the SMS also includes all the management systems that manage these information flows.

The HMRI had the power to reject Railtrack s Safety Case and to question Railtrack on issues within TOC safety cases that they had accepted. It is pertinent to state that as far as this author is aware, no TOC safety case issue accepted by Railtrack has ever been challenged by the regulator. However, the Safety Case has been used by HMRI to test for compliance after an accident, and as it is a legal document, to mount prosecutions on the basis of non-compliance. Railtrack Safety Standards has been carrying out safety case compliance audits as part of its own safety case commitments to control risks imported by TOCs, and HMRI can sample audit the Railtrack audits as a check on their thoroughness. 

Analysis records are used to create an auditable trial of evidence gathering in support of the analysis reports. These are used as evidence, referred to and from the safety case ... 

---