Thursday, July 22, 2010

DTC, FDA, GAO…2006 and all that

I wrote a post below a few months ago but for various reasons got cold feet about posting it. Yesterday made me angry (like Daniel Macarthur), warmed them up and so after sleeping on it decided to go ahead. It is about events of 4 years ago but it needs hardly anything changed to make it perfect for today. I have made some slight modifications and added an introduction specifically about yesterdays US Congress Committee on Energy and Commerce hearing into the direct-to-consumer genetic testing industry.

The Congress hearing was eerily familiar. There were a few differences, in 2006 it in the Senate and was all out attack by a single senator (a committee of 1), yesterday at least there was balance and some sensible questions from some members of congress. I felt Henry Waxman was OK as was Burgess.
The whole hearing was based around the GAO investigation and this was just a mirror image of what happened in 2006. Just the same – selected evidence, highlight the most headline grabbing bits, even if they are false, uncover some truly bad practices and make it look like they apply to the whole industry – the tactics are clear in the title of the actual report (pdf download here):

Misleading Test Results Are Further Complicated by Deceptive Marketing and Other Questionable Practices

My title would be: Misleading Conclusions Further Complicated by Deceptive Investigation Techniques and Other Questionable Practices. What was really sleazy was the way it was presented. The highest profile companies were required to attend, they were identified during the hearing but none of the other companies were present and Kutz (GAO investogator) only semi-reluctantly revealed that the dodgy supplements company (he said this was fraud) was Genewize. For the rest of the hearing all we heard about was 23andME, deCODE, Pathway and Navigenics, they were in the dock and everything revealed in the report seemed to apply to them and therefore the whole DTC genetics industry must be rotten. This was the clear intention otherwise the other companies would have been identified and the worst practices would have been attributed correctly.

What we went through was rotten, what those present went through yesterday was rotten. Our reaction was to work quietly with FDA and FTC but keep a low media profile. In hindsight I think we should have had much more aggressive PR, to defend ourselves and expose the flaws in the GAO, but we didn’t and it festered.

There is no point in me going over yesterdays meeting because Dan Vorhaus in a a fantastic post yesterday has done the job (even more impressive as it was posted just after the hearing finished). Daniel Macarthur has a different type of description, very emotional - I share his anger and a lot of his pessimism. Mark Henderson of The Times also had a quick post (you will need to pay a sub, currently £1/30d – I know, I know, Murdoch and all that, but worth it just to read Mark who is by far the most balanced journalist on this subject that I have come across, no surprise that he is one of the few that are prominent on the twitter etc conversations on the subject).

Dan Vorhaus I felt was far too kind to the GAO – for me the GAO are the real problem. First of all though, just to be clear, at face value they did uncover some truly bad stuff – I am not surprised at that, it’s all over the internet. Disappointing that some was from Navigenics and Pathway – they need to sort out their sales / customer services people. There is no defence for the bad stuff the GAO found but there is also no defence for the way it was presented. The bad stuff can be fixed where possible and destroyed where necessary, but only if we know what we are dealing with, how bad, how extensive, who?

It is a tragedy that the GAO avoided the opportunity to produce a real balanced fair and USEFUL report, identifying clearly who did what, giving credit where it is due, calling expert help from both sides rather than relying on a clearly anti-DTC person. There is real rubbish out there, real harm being done. Kutz identified Genewize and actually accused them of fraud. Genewize claim to be making $millions per quarter (see Genelink, the parent company), that’s a lot of fraud, why is it kept quiet, why were they not in the dock? Apart from the 4 companies they tested (23andMe, deCODE, Navigenics, Pathway) they also investigated 11 others, who were they?

What now?

1) For me the most serious thing to fix is the difference in risk predictions for the same diseases. Sure, it happens with non-genetic algorithms for complex disease risk but genetics is supposed to be different. The consumer does not care about the explanations however valid they may be. The 4 above (23DNP!) need to get together and sort it out – maybe they should all agree on a standard chip, all do the same SNPs and differentiate themselves on their services, that’s more or less what they are doing now anyway. Pathway needs to get it’s act together and start being more open. It needs to publish full sample reports, details of risk calc, etc. It won’t even provide the raw DNA data to the paying customer for what I am reliably informed (Daniel MacArthur in Times comment) are absurd reasons – have you got something to hide? Would a standard panel be acceptable to them? Why not, soon we’ll be into whole genome sequencing anyway?

2) 23DNP should as soon as possible create an industry body and introduce principles of self-regulation – come on, you’ve been talking about it since the HGC banged your heads together in 2008, get a move on. Basing it on the HGC code of practice would be a good start. If and when FDA act they may find adopting what you develop to be simple and effective.

3) 23DNP should immediately create their own GTR, they know how to do it best so that it will be accessible to the consumer, maybe then when the NIH get ready (should I hold my breath?) they will operate as a joint venture. Hopefully the NIH GTR will be mandatory as well.

4) Call for transparency from the GAO. They investigated 15 companies. Who are they? What methods exactly did they use? How many calls did they make, how many were met with good responses and how many bad? Give us full transcripts of the calls, not just selected (and edited?) clips. They have done a lot of work, with all the data it should be possible to get a real picture of the actual state of the industry, separate the good from the bad, i.e. protect the consumer. Is it possible to demand this information be made public under freedom of information acts??

5) 23DNP should aggressively defend themselves in the media, criticize and expose the GAO methods, be public about the negative things found and what is being done about them.

OK that’s enough for now, here is the original post with my own GAO history:

The recent post by Linda Avey on the motives of the journalist apparently about to write a negative article on 23andme struck a chord. I have spent many years facing down the anti personal genetics brigade and bear the scars. I began way way back in 2002 (when a genome cost around $1 billion…) when I joined a little UK start up company called Sciona. The way I joined them was ironic – sitting in a pub in London reading a newspaper (The Guardian) I saw an attack dog article all about this unethical, misleading, scam job gene-diet company called Sciona. I sat up because one of the co-founders was a friend and former colleague at UCL, Rosalynn Gill (PGP #9!). I immediately got back in touch with “congratulations” on the press coverage, went down to see Sciona and was presented with a ton of publications that underpinned their work. I knew the reputation of the Sciona people and I was not surprised that they were easily able to prove to me that the Guardian article itself was misleading and based on ignorance (or just ignored most of the facts). Sciona had just won some funding and I joined the company, staying on the (very enjoyable, frustrating, eye-opening, teeth-grinding) roller-coaster until the end of 2008 when Lehman went bust, investment funds collapsed and along with it, a few months later, very sadly, also Sciona (by now located in Colorado).

Sciona was the first to offer this type of genetic testing DTC and boy did we get attacked, from so many sides, relentlessly. But with perseverance we did over the years make many friends in the academic world and managed to gain respect – among the informed. We won public research grants from the UK and EU and were very active in NuGO. We ended up with Jose Ordovas (who I am very happy to count as a good friend) on our scientific advisory board and a certain Wally Gilbert as well. The main investors were BASF, DSM, Burrill and Bioventures – companies whose chips are of the bluest hue and who certainly did not write cheques (or checks) without carrying out the most thorough due diligence. We all believed in personal genetics and we all still do.

Anyway, back to motives – the people who attack these new disruptive technologies, what are their motives? With personal genetics we are interested mainly about improving healthcare (NOT medicine, but healthcare, it’s different) so their motives should be a concern for the health of the consumer, concerns that they are not being mislead etc. But are these the real motives? I don’t think so. In the beginning there was Genewatch who ran a campaign against us from 2002 onwards. Lack of any real concern for the consumer is given away by their ideological and dogmatic attacks against every application of genetics? For them it’s all just an excuse for nasty companies to make money. They were so dogmatic and un-analytical that they became a broken record irrelevance.

Did the GAO really put the people first when it conducted it’s “investigation” into us and others in 2006? If so why was their embargoed report leaked to the press the day before the actual hearing (the press re-leaked it to us to get our comments and the articles were published on the morning of the hearing…I was a Brit staying in the Watergate hotel in DC, it was a surreal experience)? If they really cared and wanted a balanced report why did they include their press release, with their conclusions, in the press pack handed out to the audience and participants as they entered the room where the hearing was to take place? Why did the senator (there was only 1 at this “committee” hearing…OK another did appear, asked a question and left 10 minutes later, pity, I was looking forward to meeting Hillary Clinton and Elisabeth Dole) completely ignore all of the testimony from the participants (just about all of his questions were ably rebutted by Rosalynn Gill, one reply left the senator mumbling for words…)? Why did they allow inaccuracies in their report and in their press release to go ahead uncorrected? The most blatant, and headline grabbing, was the claim about one company offering personalised supplements based on the nutrigenetic test – the report stated that the ingredients in the supplements that had an annual cost of approx $1200 could actually be bought in a supermarket for only $35. My goodness, what a rip off, what a scam… But this was completely false – they had only considered the vitamin and mineral components and completely ignored the herbal and other components, which are much more costly (green tea extract, genistein, bioflavonoids, etc). Whatever you may think about the efficacy of the herbals etc, they cost a lot and to buy all of the components from elsewhere would have cost a lot more than $35 and indeed a lot more than $1200. Of course all this was made clear at the hearing, but it made no difference, that headline was just too good to drop so they continued with the press release and the conclusions remained unaltered. It may have been a mistake originally, maybe they were not so thorough in their investigation and they missed the fact the the supplements were more than just vitamins and minerals, but when they allowed the conclusions to stand it was no longer just a mistake. It caused economic damage to a particular company that was doing nothing wrong.

The press reports then and now are almost all just based on the press release and the conclusions therein – it is still trotted out that the GAO found that the products and services were misleading. Well they didn’t, certainly not ours anyway, I can’t speak for the other companies, although then, as now, there were some obvious rogues. The GAO tactics have been questioned by a couple of influential people at least.
One is David Castle, philosopher/ethicist, in a free academic article
Although there are several methodological flaws in the report, the conclusion that at-home genetic tests offered to consumers are snake-oil was uncritically repeated. The flaws in the methodology and conclusions of the GAO report are serious and potentially damaging to private interests in nutrigenomics, as well as public confidence in the Food and Drug Administration (FDA).

Another is Ruth DeBusk in the Washintonian, April, 2010

While geneticist Ruth DeBusk concedes that there are “absolutely legitimate issues of concern” about nutritional genetic testing and many genetics labs, she calls the GAO report “irresponsible.”
“It’s clear the GAO investigators did not understand what nutritional genomics was all about,” DeBusk says. She says that some genetic-testing labs are doing solid research and offering legitimate genetic tests.

However in most articles the job done by the GAO PR is just lazily and uncritically repeated by just about all the journalists including…you guessed it…Andrew Pollack

I wish the GAO had carried out a proper review, along the lines of the HGC in the UK. The HGC were quite sceptical about Sciona and others, but they produced two balanced reports over the years and are still doing excellent work with their promotion of a Code of Practice. Here is a difference, the HGC is working on making things work, the GAO just made its headline then moved on to the next (hatchet?) job. The real pity is that there really are and were then, some highly dubious and really misleading scammers touting DNA tests with about zero evidence, they were, and are, the real villains. That’s it, I’ve got a lot off my chest and feel better for this rant. Good luck to 23andme, deCODE, Navigenics etc, bad luck to the scammers and real snake oil merchants.

The above paragraph I have emphasised because I wrote it back in March 2010 about the events in 2006 – and I don’t need to change a single word to make it perfectly relevant for yesterdays charade.

The situation in 2010 is a lot better than 2002 but there is still a way to go, investors please note, this is not a “5 year exit-strategy” business, if you just want to make money, go elsewhere, if you want to change the world, stay with us.

Tuesday, July 20, 2010

Mark Henderson cubed, comments here but go and read the article, it’s worth the £1 on it’s own

I began this by commenting on Mark Henderson’s Blog about his genetic results, I had just had an espresso (and I live in Italy) & got carried away, it’s Mark’s fault as well, he invited comments via Twitter. The Times has become a subscription site for a few weeks – I really urge you to go and subscribe, it’s worth it for now at least (£1 for 30 days), the science coverage is the best in it’s class. Anyway, here are my comments on his excellent article, which you should go and read at:


Good stuff Mark and glad that others are climbing over the paywall to read it, it feels pretty lonely here! Just a few comments:

1) "Many of these, however, are based on preliminary, unreplicated findings [re 23andMe]" - this is OK and glad that Gutman agreed yesterday that prelim findings/information is OK as long as it is clearly labelled as such. I'm interested for professional and personal reasons in prelim findings, I can get them from the primary source but there is no reason why it should not be possible for anybody to access the same information in a more user friendly way (mostly from research that we have all paid for) - why should it only be accessible to "us". in fact the more public access the better, it could even improve journalism...I'm thinking Indy here, not the Times (well maybe Sunday Times)!

2) Yes interesting that Pathway are now docs only. I have always been disappointed by the lack of information on the Pathway site, what have they got to hide since their competitors have shown everything. They've always been the LibDems of the DTC world and just like with Clegg et al, recent events (Walgreens and following storm) have brought them to prominence. I wouldn't call them losers, and even if their attempt was an "ill-starred initiative" they are now definitely on the map, it brought them loads of publicity (the storm was interesting though, seems that selling through a pharmacy, with professional present, was considered worse than DTC via internet). Anyway, still do not like their opaque website and would not recommend them for that reason, I hope that the NIH Genetic Testing Registry will sort that out

3) On the NIH GTR - I hope that they learn from the DTC companies on how to present information, they will miss a big opportunity if they do not make it extremely user friendly for public and professionals alike

4) Not being able to see the Pathway results (by the way are they copyright protected or can you put them on the Times server for download, if you wanted of course? That would increase subs even more!) - I get the feeling that I agree with you. The "take action" stuff looks like they are presenting the risks as applying to YOU as an individual without making it clear enough that these are population risks. In fact, based on the genetics alone, you may not need to take action. Without the environmental, family history etc info the personal risk cannot be accurate. The gene for "dangerous driving" will only be relevant to you if you actually drive. I believe that 23andMe and Decode communicate this well enough, Pathway seem to be trying too hard to do "personalised" healthcare.

5) Re your glaucoma risk "I am now a confused customer!" Again I hope that the GTR will help (I am maybe hoping for far too much from the GTR) - if it has a forum area where anyone can post to (incl anonymous) then maybe you would get expert help in sorting out the confusion, we know from the blogs that the experts are out there and are happy to help

6) The apparent sampling errors could be due to typing complementary strands, so GG would be the same as CC? I don’t know if that would be the reason for AG → TG though (BTW do 23andme type rs6553078, according to dbSNP this is what rs10390924 has become?). However it would be very interesting to compare the full concordance for all the SNPs in the 3 sets of results, it’s important to know what the accuracies are, maybe you could get help from a friendly blogger (I would offer if I knew a good way to do it!)

7) “I'm still firmly of the view that individuals should be able to buy access to their genomes without medical supervision” – do you think though that they should be regulated in some way or is some sort of strong push for complete transparency enough (eg. GTR, again…)? If they are deemed Class II by FDA that would require premarket review, but of what? The details and interpretation are in continual flux – we’ll end up with a Zeno’s Paradox – if you pre-market review the service at a point in time you won’t see it moving and by the time you get the results it won’t be there anymore. If they want to make it Class II they will have to be innovative with the review process

Sorry, this became far too long, I should not write comments on genetics just after an espresso, but you asked!

If anybody is reading this (I have even fewer readers than The Times) please comment over at Mark’s blog (you can copy here as well, but I insist that you comment at Mark’s as well, please…I get NO commission, just want to help get discussion going at Eureka again)

Friday, July 16, 2010

Response to NIH regarding Genetic Testing Registry (GTR)

The NIH is seeking input and advice on the following items:

Are there any types of genetic tests that should not be included in the GTR?
All tests that make some sort of utility claim should be included, also those that are not overtly connected to health (e.g. Child IQ, sports performance, baldness, etc). One aim of the GTR should be to create confidence in genetic testing and help protect against false or exaggerated claims. To have one single reliable (worldwide I hope) location where anyone, scientist, medic, consumer, journalist, business partner, etc can go to would be very helpful. If we end up with fragmented GTRs for different areas it will be less useful. Although run by NIH the criteria should include more than just strictly health applications

Further, the GTR should go beyond “testing” – it should cover all services, especially interpretation services which will become the main area of “personal genetics” as the testing part itself becomes routine.

What are the potential uses of the GTR for
  1. Researchers – find collaborators and suppliers. ELSI researchers will benefit from a central mass of information
  2. Patients/consumers – find reputable suppliers, education
  3. Health care providers – find reputable suppliers, education
  4. Clinical laboratory professionals – deposit information on own services, find suppliers / partners / customers, information on competitors(!)
  5. Payers – education and information to help make decisions on utility and reimbursement. Maybe they will learn more about the opportunities for prevention and will be kept aware of specific ethical/legal problems regarding health insurance
  6. Genetic testing entities/data submitters – information on available tests & services. Depending on how the database is structured it may offer the possibility of collaborative data sharing
  7. Policy makers – education, awareness of services available, consumer attitudes (if consumer input is allowed)
  8. Electronic health records – depends on the scope of the GTR and the database structure but there is a potentially very interesting opportunity to explore using standard language to enable genetic interpretation information to be incorporated into EHR.

What data elements are critical to include for use by (1) researchers, (2) patients/consumers, (3) health care providers, (4) clinical laboratory professionals, (5) payers, (6) genetic testing entities/data submitters, (7) policy makers, and (8) electronic health records? 
Genes, SNPs, dbSNP rs numbers, claims that are made with clear links to how they arrived at the claims/advice/interpretations from the scientific literature. Full bibliography. It would be useful to have independent assessments of each service as well – either by professionals (like a journal editorial board and peer review) or via moderated comment, or both

What are the potential benefits and risks associated with facilitating public access to information about the:

  1. Availability and accessibility of genetic tests?
  2. Scientific basis and validity of genetic tests?
  3. Utility of genetic tests?

A central reliable source of tests available, with some independent professional assessments of validity and utility will encourage consumer confidence, will allow the choice of relevant tests and will hopefully steer consumers away from untrustworthy services. The information will also help to decide whether healthcare professional involvement may be useful either before or after testing. Healthcare professionals also count as consumers, such a resource would be very helpful to determine what tests to use and how to integrate with clinical decision making.

a) Depends on how the GTR is set up. If not presented well it could increase confusion and damage the uptake of genetics in health. If public access is to be given it will need to be pitched at the various levels expected to use it, from uninformed layperson to genetics expert. It has to be more consumer friendly than (not a criticism of NCBI, which is obviously excellent for professionals!).

b) The way the tests/services are presented, it has to be clear that simply being present on the registry is not an endorsement of the claims of the services offered.

c) A dedicated genetic testing resgistry is required but it should not foster so-called “genetic exceptionalism”. It should not encourage that idea that common genetic variations associated with traits are deterministic of disease independent of environmental variables

What is the best way to distinguish between data fields left blank because of an absence of data/evidence and those left blank for other reasons?  How important is this distinction for enhancing transparency, including for the purpose of identifying research opportunities?   
Best way is to state the reason for the absence of data. There may be no data, it may be restricted for personal or commercial confidentiality, it may be conditionally available, e.g. for research, etc.

To adequately and accurately describe a genetic test, which of the following data elements should be included in the GTR? Are there other data elements that should be added? What information is necessary to represent adequately each data element?

  1. Contact information (e.g., location, name of the laboratory director, and contact information for the laboratory performing the test)
  2. Laboratory certifications (e.g., Federal or State certification of the laboratory that performs the test)
  3. Name of the test  (e.g., common test name, commercial name, marketing materials about the test and/or genetic testing entity, standard identifier (e.g. CPT codes, LOINCii))
  4. Regulatory clearances (e.g., for tests reviewed by the Food and Drug Administration, the 510(k) or premarket approval (PMA) number)
  5. Intended use of the test (e.g., diagnosis, screening, drug response)
  6. Recommended patient population
  7. Limitations of the test (e.g., is the test validated only for certain subpopulations or limited to particular uses such as screening but not diagnostic testing?)
  8. Test methodology
  9. Analyte(s)—What is being measured in the test (e.g., genetic sequence)
  10. Specimen requirements (e.g., blood, saliva, tissue samples, amniotic fluid)
  11. Availability (e.g., is the submitter the sole provider of the test or are there multiple providers?)
  12. Accessibility (e.g., accessible through a health provider, public health mandate, and/or direct-to-consumer)
  13. Performance characteristicsi
    1. Analytical sensitivity
    2. Analytical specificity
    3. Accuracy
    4. Precision
    5. Reportable range of test results
    6. Reference range
    7. Method used for proficiency testing (e.g., formal PT program, alternative assessment) and score
  14. Clinical validityi
    1. Clinical sensitivity
    2. Clinical specificity
    3. Positive and negative predictive value
    4. Prevalence
    5. Penetrance
    6. Modifiers
  15. Utility (e.g., clinical and/or personal utility) or outcomes
    1. Benefits
    2. Harms
    3. Added value, compared with current management without genetic testing
  16. Cost (e.g., price of the test, health insurance coverage)

    All of the above. Accuracy of testing via historical data of control samples, allele frequencies stratified for population etc. Clinical validity to include odds ratios, relative risks, lifetime risks – in a way that is understandable by lay people. Modifiers are important, almost all risk assessments will be modified by environmental parameters and it is important that this is clear.

    Clinical and personal utility is sometimes straightforward but often is hard to quantify. This section needs to be explanatory, figures and percentages will not be very helpful most of the time

    Other info: most tests will involve the use of software in their interpretation. It will be important to give as much information as possible on this. For risk calculations the precise methods need to be supplied, for interpretations of gene x gene and gene x environment interactions and the advice generated, descriptions of the algorithms should be supplied. Also the validation procedures of the software should be detailed – how it is controlled to be sure that the correct advice/results are given for the various input results. What level of standard was used in the creation of the software and in the programming of the rules?

    What types of information might be difficult for test providers to submit and why?
    While some software details should be given it will clearly be difficult to submit full algorithms etc, and other commercial secrets.

    What are the advantages and disadvantages of collecting and providing information on the molecular basis of genetic tests, such as detailed information about what the test detects and the specific methods employed?
    No further comments

    In addition to the data elements, would it be helpful to reference other resources, and if so, which ones (e.g., published studies, recommendations from expert panels such as the Secretary’s Advisory Committee on Heritable Disorders in Newborns and Children, U.S. Preventive Services Task Force, or Evaluation of Genomic Applications in Practice and Prevention Working Group)?
    No comment

    As the GTR is being designed, what are the important processes to consider to make the submission of data as easy as possible for the data provider (e.g., the capability of linking to information that has been submitted to other agencies, such as the Food and Drug Administration and the Centers for Medicare and Medicaid Services, or a master file of data common to particular tests)?
    No comment

    Which potential benefits and risks would be most likely to affect the decisions of researchers, test developers, and manufacturers on whether to submit data to the GTR, and what factors will best encourage submission of complete and accurate data?
    Submission to the GTR should become a “required” step by any service provider. Not necessarily enforced but essential in the sense that if it is not there then it is suspect. Service providers would benefit from a GTR logo that they could use on their websites to link to their own submissions. It should become a commercial risk not to submit to the GTR. With the help of press, social media, conferences etc, the GTR should be widely publicised, easily accessible (and easy to understand/navigate, especially for journalists)

    What are the most effective methods to ensure continued stakeholder input into the maintenance of the GTR?
    Keep it up to date. Ensure an adequate budget, keep it publicised, regularly report on results, benefits, effects, etc of the GTR including in scientific papers

    For what purpose(s) would you use the Registry to support your professional efforts?
    Submitting tests and services, reviewing other tests and services, commenting on them (preferably on the GTR site itself).

    Are there any other issues that NIH should consider in the development of the GTR?
    As mentioned above, a review mechanism would be useful. A sort of peer review as used by scientific journals, with editorial board etc. There should be areas for comments by other users, maybe via comments or a wiki type system. Most service providers will have nothing to fear but there are already too many very dubious services and the GTR can help weed them out either via their absence from the registry or by critical analysis on the site. In addition it would be useful to have space for comment of tests which are available but which are not on the registry. Many of these will likely be dubious and are absent to avoid scrutiny – they should be scrutinised anyway. Others may well be valuable but have not been submitted fir various reasons (time & resources, ignorance of the registry, etc)

    The most valuable thing that the GTR can achieve is clarity and transparency…many services will be direct to consumer but both healthcare professionals and consumers alike will suffer if the genetics/genomics service industry becomes as opaque and exploited as the supplement industry.