DNA Dilemma, dissecting the FDA

Mary Carmichael (General Editor, Newsweek) has this week been running a very interesting series of articles about the “dilemma” of whether or not to undergo a 23andMe type DTC genetic test. As part of that she interviewed the FDA and with journalistic cunning recorded the whole interview and published the full transcript (here). It was a very good thing too that she has the tapes because the interview was in some places quite revealing.

Mary asked me to comment/dissect the interview and I have done so below – I look forward to seeing the comments of others that Mary has asked on their blogs as well.

In a sentence the interview came over to me as unnecessarily vague and evasive (unless for some reason it was necessary!) – certainly lacking in clarity and not answering some of the questions asked but giving up some useful information.

DNA Dilemma: The Full Interview With the FDA on DTC Genetic Tests

NEWSWEEK: During the recent congressional hearing, 23andMe’s general counsel said that the FDA “encouraged them to proceed” freely in 2008. What specifically changed about their test panel between 2008 and now that caused the FDA to start to take some regulatory action? Alberto, I think you’ve touched on this a little with me before, that the concern was mutations with powerful effects on breast-cancer risk or drug metabolism—
tests that might influence medical decisions. Is that correct?
Alberto Gutierrez: They met with [former FDA commissioner Andrew von Eschenbach]. We were not there. We did meet with them on several occasions, but their claims of what they’re offering have been changing constantly. What was there in 2008 I’m sure looks very different from what is there now. The drug-metabolizing claims were not there previously, and even some of the claims that directly touch on disease states, cardiovascular disease, breast cancer were actually not originally—they’re much bigger now.

KG: “we were not there” – so what?! Is he really saying that anything that went on before has no relevance now he is in the seat? He can’t say “I’m sure it looks different”, it either does or it doesn’t and he should know. The cardio stuff was there, there has always been health related content. It’s true though that it has changed with the addition of the “locked” more serious disease associations like BRCA1 and the pharmacogenetics, which are significant additions

That’s interesting. I think before when we talked, the cardio and breast-cancer stuff was not brought up. So even some of the common disease claims—what is the specific concern with them? My understanding was that a lot of those genes really don’t nudge the risk up or down very much.
AG: The concern actually is with everything. The law requires us to clear devices or approve devices before they go into the marketplace when they make medical claims. The question with 23andMe has been whether their claims were medical claims or not. The original claims they were making really were very much on the edge. We actually told them that we thought they were medical claims, but it was at least possible that you could argue that they were not. Now clearly the claims they are making are medical claims. And to show you this is an issue we actually have been dealing with, we actually brought it up in SACGHS, that this was a point of discussion in 2009, because we kept hearing from 23andMe claiming that they were not making medical claims. We actually thought to have the Secretary’s Advisory Committee at least state whether they considered the claims that they were making medical claims or not.

KG: So confirmation that for the FDA the test IS a medical device that needs to be cleared or approved.

In our previous interview, and this is a direct quote, you said those health claims were “what we would consider to be moderate risk.” I’ve recently seen [the FDA’s director of the Center for Devices and Radiological Health] Jeffrey Shuren quoted as saying these claims are high risk, which I assume could mean stricter regulation. Which is it, moderate or high risk, and has something changed since our initial interview? 
AG: It always depends on what medications they can have, how they’re offering it, we always look at what is the risk of an undetected false result, what would be the medical action taken, and whether there’s high risk there or not. I don’t think Dr. Shuren meant to imply that this would be [pre-market approval, or PMA], but that there might be some claims there that we may consider to be PMA-like, or PMA per encounter ...
Elizabeth Mansfield: In general, though, they have a variety of claims that span our normal risk classification. On a claim-by-claim basis, there are some that might be low, there are some that might be moderate, there are some that might be high. I think Dr. Shuren’s assessment that these were “high risk” was as opposed to “no risk.”

KG: Well that cleared that up then! I’m still totally confused about Shuren’s high risk and Manfield’s low, moderate and high. Why is AG talking about “medications they can have” etc? What’s the relevance? What exactly is an “undetected false result”? If it’s undetected a) how can it be false and b) how can any action be taken?!

Maybe I am being mean, let’s assume he means false negative or false positive, that I can understand is an issue worth evaluating - at least if they are actually referring to a false lab result, a miscalled SNP, rather than a false interpretation. (One point though, the FDA will need also include in the equation the likelihood of a false positive or negative – the analysis that Daniel MacArthur carried out on Mark Henderson’s DNA suggested and amazingly low error rate of “less than one in every 14,000 markers”):

• For most of what 23andMe do (and the others) the false negative/positive is going to be low risk or no risk – this cannot be opposed by most critics who apart from saying that it should be heavily regulated also say that the risk predictions are small and imprecise. Yes, that’s accepted, which means a false result would be low risk or even no risk.
• For the pharmacogenetics aspects this is different. If the false result is used to modify a drug prescription then it could be medium or high risk (I’m not qualified or knowledgeable enough about potential adverse consequences to choose which). A false positive would be the same. If the companies want to continue to claim that the information is for education only and is not intended for clinical use they need to make that very clear next to these results – they need to say that before any clinical decisions are made that these particular gene variants need to be retested by a lab that offers the test as a true regulated medical device. This I can agree with – the 23andMe pharmacogenetic interpretations are correct but that will not be the case with all companies in the future, and it applies to both physician prescribed and DTC tests.
• For serious diseases, the “locked in” sections – where you need to give specific approval via the website before the results are revealed to you I would assume would be medium/high risk. To be wrongly informed of cystic fibrosis carrier status, high risk BRCA variants, etc, would be worrying for the customer. Not life threatening though, as long as explained correctly. I have not seen how they present a positive (as in carrier of risk allele) but again they would have caution that the result is provisional and any further action will require testing via a properly regulated device. Medium or High though, I am not sure, although I would say medium (but I realise a lot would say high) – If I got the information that I had a very high risk for prostate cancer I would be worried, I would straight away get retested by another lab, if that confirmed it, I would want to repeat it again (I have worked in both research and diagnostic labs for many years, I would repeat it!). If it was confirmed then it’s possible 23andEtc would have saved my life – if it was not confirmed, I would have gone through a week or two of anxiety, but I would not have thrown myself off a bridge.

Having just typed the above (“it’s possible 23andEtc would have saved my life”) – I am thinking now that it would be more risky than beneficial if the FDA took action which reduced access to these tests. They are potentially lifesaving and I am thinking that perhaps the “high risk of serious disease” tests are actually more important to go DTC (not as a free for all though, this does not mean NO regulation) and should be encouraged. Let people decide and pay for something if they want to.

If a medical decision is going to be made then I would want the interpretation to be as bullet proof as possible, therefore approved. I cannot see a serious (medium or high risk) medical decision being made without the use of a doctor though so I would only require approval of tests marketed to doctors as tests on which clinical decisions will be made. It’s a sort of paradox here, if it’s DTC it should be clear that it is not for medical use – a doctor should not make a clinical decision based on a 23andMe test. A customer can make a lifestyle decision, that’s different. A paradox because the decision of Pathway to go only via medical doctors should logically make it more liable for regulation. I would assess the 23andMe website as just like any other healthcare site, like WebMD, is that a medical device? WedMD has risk factor calculators, it makes medical interpretations of personal information, I really don’t see the difference as far as regulation is concerned. No doctor would make a clinical decision based on a patients interpretation of what he or she has read on WebMD, no reliable doctor that is.

So the FDA would actually be going through these panels test by test, and looking at each one and seeing, Is this one low risk or moderate risk or high risk, and approving them specifically?
AG: Yes, in a general sense we would go claim by claim. Whether each claim would require a different submission or not—we’d be looking at each of their claims and looking to see that they would have data to credibly back the claims they are making.

KG: good luck to them – Sisyphus

So if one of these companies wanted to add a new test to its panel, it would have to get that specifically approved? 
AG: No. I want to make a distinction between what we would be doing now, and maybe what we would do in the future, and the reason here is that—Dr. Shuren put this on the table [unintelligible]—with these devices maybe it can collect a lot of data that may not be relevant at the moment. If later on data becomes available that shows that the company may be able to make a claim that doesn’t exist now, is there a way for us to do this that wouldn’t require necessarily a submission for each of them, or that we could bring together experts that could make the call that there is now enough evidence to make this kind of claim? That is something that we are discussing, that we are trying to figure out how we’re going to deal with devices that generate a lot of data for which there is no use now but may be in the future, like with whole-genome sequencing …

KG: Again very clear! It will be “claim by claim” (which is a specific thing, it can’t be a “general sense”) – or maybe it won’t be. All he should have said to the previous question is “that we are trying to figure out how”. Also Shuren in Congress seemed to recognise that this would be impossible and suggested a different way, something like overall approval of a company with regular reviews (like CLIA perhaps?)

[When we spoke last time], you had just sent five letters out and Illumina got one. Your concern in that letter, I believe, was that Illumina was selling a “research use only” chip and it was being used for nonresearch purposes by DTC firms. There were several reports I saw this morning indicating that the FDA has “requested” that Illumina stop selling chips to DTC genetic-test providers. Is that true in whole or in part?
EM: We have asked Illumina to work with us to bring their devices into compliance with our regulations. We have not specifically requested that they stop selling them to anybody.

KG: OK… can you “non-specifically” make such a request, with a nudge and a wink perhaps?

So they can continue supplying 23andMe and deCODEme with chips right now?
EM: We’re working with them on that …

KG: Ah yes, looks like you might be able to…

Illumina obviously isn’t the only chip maker out there. Why is it the only one that is getting these letters? Are the other ones going to be called in as well?
AG: You can bet that with the original set of letters in 2009, Illumina was not the only one to receive a letter. 
I’m sorry, I’m not familiar with the 2009 letters. What were those? 
AG: We sent letters to several companies with direct-to-consumer chip arrays to come in and discuss the issue.
And did those companies come in? 
AG: Yes, and I think that’s part of what you saw with the public meeting on June 30 on the array issue and why several companies want to figure out a way to [unintelligible] …
[The chips are labeled “research use only.”] The DTC providers, when they sell these tests, aren’t [necessarily] doing research. Doesn’t that imply that Illumina does have to stop selling the chips to them?
EM: If they continue to label them that way, yes, that would be something that we would most likely take a little further action on. At the moment, they’re working with us and we will see what they can bring to us in a reasonable amount of time without completely blowing up their business or the market or anything …

KG: I really hope that is the case

So I guess if it’s not being used for research only it needs a different type of [label or] approval?
AG: That’s correct…

KG: Label OR approval, what do they want, what would Illumina have to do, how long, how much, what about when new SNPs are added, etc?

Until that happens, though, is it correct that the Illumina chips will remain on the market for direct-to-consumer companies to buy?
AG: [Pause] I think Illumina needs to figure out how they’re going to move forward.

KG: Oh dear, back to the “non-specific” – horses head in the bed?

Okay … I want to move on to whether the issue with direct-to-consumer is actually providing data to people, or is it the interpretation algorithms these companies are using? So, would a company need to be approved just to provide a raw SNP list to people?
EM: They would if they made medical claims about that data. If they don’t make any medical claims about that data, then they’re free to provide information as far as we’re concerned.

KG: Then the FDA need to give clear guidelines about what constitutes a medical claim. But one good thing – there will be NO obstruction to me getting my raw genotype data DTC. So then what happens if I go and read about my SNPs on www.snpedia.com, or pubmed, or OMIM or the excellent “HuGE Navigator”? What if I put my data through Promethease ? What about reading about SNPs and disease risk on WebMD and on and on

What exactly would constitute a “medical claim?” Would pointing people to medical research papers [qualify]?
AG: It depends. There are rules as to how one can do that … Those rules are actually worked out pretty well, and they just would need to make sure they’re staying within the rules.
Are those rules on the Web?
AG: I don’t know where the policy is. I would have to get it for you. It’s an agencywide policy. I would have to find it for you. And it won’t be that easy for people to follow it…

KG: The rules are worked out pretty well, now dammit, if only I could find them… What? “I don’t know”, “I wasn’t there” (see above), “it won’t be easy to follow” – terrible admissions to make

Alberto, I had heard a story, I believe it was in Rob Stein’s article in [The Washington Post], about these women who were having prophylactic oophorectomies on the basis of direct-to-consumer genetic results that were  “questionable.”
AG: That’s a little bit taken out of context in the sense that it’s not based on the direct-to-consumer [model]. The cases we heard of were based on lab-developed tests that were not properly validated. What I did say was that even though 23andMe doesn’t make a direct claim as for ovarian cancer, they do provide information that links what they are giving the consumer to ovarian cancer.

KG: A little bit? It would have to be a lot out of context to have made such a shocking claim. It was shameful (actually it was also unbelievable, for good reason, it wasn’t true)

So that’s an example of where the interpretation is about pointing to a medical paper. 
AG: It’s more than that. They interpret the medical paper for them … Ovarian cancer is called the “silent killer.” People actually don’t know they have it until it’s too late. And women that are high risk a lot of times choose to have their ovaries removed because they are afraid that it may not be caught in time. There’s no diagnostic test for it. So the tests that we have seen—there was one that was taken off the market because of FDA action … We knew that in those cases most women who were being operated on actually were likely to be negative for ovarian cancer … The one we heard most about originally, it was a test called OvaCheck that was going to go to the market, and we believed it was not well characterized and it was going to create problems. It never made it to the market, and then in 2008 another one made it to the market called OvaSure, and the FDA wrote LabCorp a warning letter …

KG: His answer is not at all relevant to your question but is interesting just the same. The “silent killer” – that makes me think, again, that anything that makes people more aware of their health, their possible risks, possible diseases, especially symptom free diseases, etc, should be encouraged, and should be DTC. I was not aware of this information about ovarian cancer, nor was my wife. If it silent then you are not going to go to a doctor about it until it is too late and of course AG knows very very very well that nobody would undergo serious surgery without medical guidance, I suppose it’s impossible unless you really want to do it in an illegal fashion, as in the days of the back street abortion.

Thank you, that’s very helpful to clarify. In general, when the FDA makes decisions about potentially fairly serious new regulation of a device or a test or a drug for that matter, does it do any sort of analysis and take into account the costs that regulation may impose on companies?
AG: This is a more complicated question than you actually think. We have rules we’re supposed to follow that we’re supposed to apply evenhandedly to everybody about what exactly is a new product. The law requires new medical devices to come into the FDA to get approval …That’s what the law is, so we don’t actually get to make a decision that’s based on the economic issues for that new device.

KG: It shouldn’t be complicated – surely they can say to a company “you will need PMA and you can expect it to cost $xyz”? Or do they just enjoy helping to proliferate the “FDA regulations consultant” industry?

I’m a little bit confused. Can we talk about that in the specific context of direct-to-consumer tests? What kind of thinking was there about the costs? 
AG: [Direct-to-consumer tests] are a new medical device, and under the rules of the 1976 law, new medical devices require FDA clearance or approval before they go onto the market.

KG: No doubt again, they ARE a medical device

Right. But as part of that deliberation process, when you guys are considering whether to approve a device or not, do you take into account the cost that would impose on companies or the general impact on the industry?
AG: No. No. Our review does not, no, we don’t take into account cost.
EM: And Mary, that cuts agencywide. That is not considered in any of our reviews.
Okay. Thank you. I thought I had seen some obscure law saying there was a
requirement to take into [account] costs … Yes. Here it is. This is actually on the FDA Web site: “As part of its mission to supply economic analysis to decision makers, the Economics Staff in the Office of Planning conducts economic analyses of all important proposed and final regulations issued by the Food and Drug Administration. Each economic analysis includes an assessment of the costs, benefits, and cost-effectiveness of the action.” So this is right on the Web. How can you not take— 
AG: So that is just, if you look at the first part, that is for all new guidances, for new rules and regulations.
EM: Which this is not.
AG: This is neither a rule nor a regulation.

KG: But once upon a time it was a new rule so an economic assessment should have been done, it should exist and can be adjusted for inflation – what’s the problem??

I see. You’re saying that because this is just like in 1976, these are medical devices, therefore they need approval— you don’t consider this new regulation, it’s just that the old regulations need to apply to these companies?
AG: That’s correct.

KG: Do they actually know what they are talking about?

Okay, thank you. One more question: … We’ve had the hearing and there were obviously some fairly troubling things that came out of the GAO report. I’m wondering what impact the hearing is having on the FDA right now. Are you guys taking those findings into account at all? Is that influencing your decisions?
AG: The findings of the GAO?
And other things that came out of the hearing. 
AG: No, we believed that these devices [should be] under regulation. It hasn’t influenced us. Dr. Shuren did say that we should have been moving faster, and that we need to make these companies come in and get cleared for moving forward and selling their tests on the market. We will proceed with due diligence and try to help the companies come into compliance.   
EM: The GAO report clearly did not undermine our feelings that these tests could present some risks to patients.

KG: I hope AG is right and that the GAO report is not used. EM’s answer though suggest it will be influential and I bet it will be. It should not be because as even the GAO admit, it was not scientific, the GAO did not make all of their information available for scrutiny, it was not independently reviewed, only selected sections and edited conversations were presented in a very theatrical way. I would bet that if all the information was made available it would show that overall the industry, as far as the reputable companies are concerned, is working very well and may even undermine EM’s feelings. It was very obvious that the GAO was making a “case for the prosecution” it was not an even handed survey, it was presented in a way to maximise the impression that the whole field is misleading and unethical.

[Given] Dr. Shuren’s comments that there should have been faster movement, are you guys feeling pressured to move fast right now?
AG: I think we are doing what we are supposed to be doing. We are obviously trying to get the companies to come into compliance and working with them to do so.

KG: OK we’ll see what’s next. Will it be like 2006 where the FDA got a similar rap on the knuckles by the Senate? If so then I suppose we will be back here after 4 more years of limbo to discuss the GAO/FDA/ETC findings on the DTC whole genome sequencing industry.

Overall the interview does nothing to clarify much at all, except that the FDA decided long ago that they are medical devices, even though they don’t know where the rules are (did he really say that? Thank goodness you got the tapes!). It’s all very murky – I was rather cynical in thinking that there are some murky politics behind it but after talking to Mary last night who had actually spoken to them, I think it is more that they simply do not know what to do yet, they don’t know how to regulate it. That is fine, it’s a very complex new situation, it’s not a simple laboratory test – the interpretation is a constantly changing river of information. There is no shame in not knowing what to do but the best response is not to pretend you are in control, it is better to consult widely (and openly, in public) and work closely with the industry you are trying to regulate. It is an opportunity to develop new innovative regulatory methods and not try to apply old rules, designed for simple tests, to this new, complex area.

One thing is clear though – if it makes a medical claim is it a medical device. I believe there should be clear guidelines for medical claims. Look – the nutrition and supplement industry are desperate to make health claims, they have to spend $millions to get them accepted and most of their applications are rejected. Here with DTC the situation seems to be the opposite, the FDA seem to have decided quite easily that these are valid medical claims. A medical claim should be something that would influence a clinical decision. Or a medical claim would be something like “if you take this test and follow our advice, or take our supplements, it will reduce your risk for disease” – if the DTC companies are quite clear that they are providing information that should not be used to make clinical decisions, and are not claiming to actually reduce your risk, are they really making health claims, are they really any different to any of the websites I have mentioned, including many run by the government?

---

Other recent comments on the FDA re personal genetics:

GeneticFuture: The FDA doesn't plan to regulate access to raw genetic data;

FDA Blog: FDA states cost 'not considered in any of our reviews' and In wake of FDA offensive, genomics entrepreneurs look overseas