No subject

CDRH's recently upgraded "software forensics lab" marks a substantial
financial and time investment by the center to get at the root cause of
the most troubling postmarket device software failures, but the agency
hopes manufacturers will pick up the tab in the future to prevent
dangerous anomalies before their products hit the market.

In the past year or so, the forensics lab, housed within the Office of
Science and Engineering Laboratories, has used high-power instruments
and techniques to analyze "source code" in roughly a half dozen medical
devices whose software was thought to be responsible for causing serious
adverse events.

The analyses, which employ tools routinely used in the automotive and
aeronautical industries, exposed several software design errors linked
to the adverse events and at least one latent error that had the
potential to cause an injury, according to Al Taylor, the Director of
OSEL's Division of Electrical and Software Engineering.

"The Gray Sheet" visited the lab last week and sat down with DESE's
Deputy Director Brian Fitzgerald.

Software Complexity Challenges CDRH Reviewers

"Traditionally, software has been very, very difficult technology to
penetrate from the perspective of a reviewer," Fitzgerald explained.

That's because it is nearly impossible to test every eventuality that
the software could lead to.

The more complex devices are programmed with tens of thousands of lines
of code, making manual review of the entire system an impractical task.
Software embedded in a pacemaker or implantable defibrillator may have
more than 80,000 lines of code, and Fitzgerald said he had seen an
infusion pump programmed with 170,000 lines.

So rather than review the software itself, in a manner similar to how
one would test the mechanical components of a device, FDA has generally
relied on validating the software design processes in order to mitigate
risk.

Fitzgerald said the results of this approach have been good, but not
perfect. Many software errors, like unintended restarts or incorrect
measurements, do not surface until after the devices are approved and in
use. Flaws are corrected "in the field" with subsequent software
updates.

But about a decade ago, another potential approach emerged. In 1996, a
European space rocket, the Ariane 5, exploded just after launch because
of a computer bug. Scientists made quick use of a sophisticated
programming tool called "static analysis" to figure out what went wrong.

That experience validated the strategy as a useful tool for automating
the search for hard-to-find coding vulnerabilities in complex software.
It has since been embraced by the aeronautical and automotive sectors to
help minimize errors and accidents. Now FDA is trying to bring medical
device manufacturers into the fold.

Lab Limits Tools To Imminent-Threat Recalls

Ultimately, the agency hopes manufacturers will use static analyzers in
their software development processes to reduce the number of bugs that
creep past reviewers and into the market. But for now, DESE has found
the five systems it has recently acquired to be powerful tools in
addressing concerns in the postmarket stage.

A number of recent recalls have been linked to software anomalies. In
October 2006, St. Jude Medical discovered that software flaws in certain
APS III and Merlin PCS programmers caused errors in the battery
indicators of Identity pacemakers (1"The Gray Sheet" Oct. 23, 2006, p.
16).

More serious Class I recalls have plagued infusion pump manufacturers,
including Baxter, which had to recall in June the 4,500 Colleague
systems it had purportedly fixed because a software bug inappropriately
stopped infusion in the updated devices (2"The Gray Sheet" July 23,
2007, p. 4).

A Class I recall of Defibtech's automatic external defibrillators in
February was also software-related.

The software forensics lab has worked with CDRH's Office of Compliance
in "several high-profile compliance cases" to review problematic
software and firms' proposed fixes, according to CDRH's fiscal 2006
annual report.

But static analysis remains a resource-intensive process with a high
learning curve. "Only cases that present an imminent public health
threat warrant the level of effort required to do the analysis," Taylor
noted.

DESE is staffed by 18 people, a third of whom work with the static
analyzers. Fitzgerald said the division hopes to bring in more academics
with software expertise over the next few months.

Sometimes, in addition to validating a manufacturer's corrective actions
for a specific recall, the analysis uncovers software errors that are
unanticipated.

Fitzgerald acknowledged these new capabilities at CDRH can be
intimidating. "No manufacturer wants to be in a position where FDA [is]
finding bugs in his code," he said. "And so we treat these very, very
privately."

Fitzgerald said the unexpected errors are reported as "unresolved
anomalies" and it is up to the manufacturer to determine if they could
cause a device failure. In most cases they are harmless.

A recent review in the forensics lab found 180 "questionable constructs"
in 100,000 lines of code, but only two turned out to be real design
issues, Taylor said.

He also pointed to two other cases where static analysis of the software
did not find any bugs, thus clearing software as a root cause candidate
in the recall investigations.

Premarket Analysis Could Help Reduce Liability

Ideally, static analyzers would be used to eliminate errors before a
product gets to market in the first place.

But Fitzgerald said that as long as user fee review timelines are in
place, CDRH cannot routinely use these tools in premarket evaluation:
"We understand that there are timelines involved in premarket approvals
that we would very probably adversely affect."

"In any case, we would like to put ourselves out of the business because
industry could put themselves into it," he said.

So far a few of the larger device companies have invested in their own
static analyzers for use in software development. But FDA is urging
other firms to get on board.

According to Fitzgerald, market competition between 10-15 software
developers has brought the cost of static analyzers down in recent
years. Many of the tools are now available for between $10,000 and
$20,000, he said. "When you consider the cost of a recall, that's
actually a bargain."

Still, medical device manufacturers have traditionally been conservative
regarding changes in development processes.

Ultimately, Fitzgerald envisions in five or 10 years a premarket
approval process where a company can demonstrate in its product
submission its use of a validated static analyzer to screen embedded
software and reviewers can quickly sign off.

In addition to easier and quicker 510(k)s and PMA supplements, the
result would be fewer, less serious software-related recalls, he
projected.

"These days it's clear that everybody [is] interested in software not
having latent vulnerabilities that are discovered in the marketplace.
That's the last place anymore you want to find out the stuff doesn't
work," Fitzgerald added. "Why should you have to if you can go out and
get a tool that can find a large percentage of those problems?"

"We're hoping that by quietly talking about static analysis tools, by
encouraging static tool vendors to contact medical device manufacturers,
and by medical device manufacturers staying on top of their technology,
that we can introduce this up-to-date vision that we have," he said.

- Chloe Taft