It should come as no surprise to anyone that e-discovery is more than anything about trust and transparency. Because the results of any collection or review are likely to end up in court, it is paramount that customers both understand the way a product works and trust that the vendor will be there to back up the results in an honest and straightforward manner. It is for that reason that I am personally blown away by a recent filing I read from Southern District of New York Court System.
The filing is the Declaration of Paul J Neale, CEO of DOAR Litigation Consulting, in support of plaintiffs’ objection to an eDiscovery ruling by Magistrate Judge James Peck in the case Da Silva Moore v. Publicis Groupe. In the filing’s opening salvo Mr. Neale states: “Defendants, along with their experts at Recommind, obfuscate the flaws in the ESI protocol (“the protocol”) adopted by Judge Peck……The protocol’s primary flaw is that it does not include a scientifically supported method for validating the results of the Recommind Axcelerate system’s predictive coding process as modified by the Defendants and accepted by Judge Peck. As it currently stands, the Plaintiffs, the Defendants and the Court will never know whether the Defendants’ predictive coding process met any acceptable standard for the production of documents responsive to Plaintiffs’ document requests.”
In short, as I read it, what Mr. Neale is saying is that Recommind’s Axcelerate solution was sufficiently a black box that opposing counsel was unable to figure out how it worked without a Daubert-type hearing (which the judge did not order – a fact that plaintiffs object to strenuously). However, a special hearing might not have been necessary had the product been more transparent. To be fair many/most of the predictive coding approaches I have seen fall victim to this exact issue, which, in my opinion, is one of the primary inhibitors to that technology. That is not to say predictive coding isn’t useful, but I feel that it is not a substitute for review – it is merely an enhancer that should accelerate the pace of review and serve as a tool to confirm results. However, the means by which it does this should not be hidden from the user. Indeed, while Magistrate Judge Peck has permitted the use of the technology, it is possible that a more transparent solution could have avoided the expense of motions practice in this case.
Also revealed in Mr. Neale’s Declaration is an interesting statement about Recommind’s participation in the 2011 TREC (Text REtrieval Conference) study and alleged attempts to make their technology and process appear more reliable. According to Mr. Neale’s declarations Recommind’s representative Mr Seggebruch’s “statements in his March 7, 2012 declaration (that were also echoed by Recommind in its widely distributed marketing material), which refer to Recommind’s performance in the 2011 TREC study, are misleading and incomplete.” For the complete picture of the ways in which he claims Recommind was “misleading and incomplete” I would refer readers of this blog back to Mr Neale’s declaration, but two of the biggest issues he mentions are:
- Accuracy: According to Mr. Neale, Mr. Seggebruch stated that “in one category, Recommind achieved F1 scores over 60%.” However, Mr. Neale suggests that a “draft version of TREC’s report of the 2011 study indicates that Recommind’s actual F1 scores were significantly lower than their hypothetical F1 scores. For example, for the same run in which Recommind received a 62.3% hypothetical F1 score, their actual F1 score is 24.7% and their recall was 25.8%. In other words, over 74% of all responsive documents were missed.”
- Credibility: Mr. Neale goes on to question Recommind and Mr. Seggebruch’s credibility and general behavior: “Mr. Seggebruch’s statements referring to Recommind’s 2011 TREC results are further undermined by the fact that Recommind has been banned from future participation in TREC studies due to violating their agreement with TREC by publicizing its preliminary results as compared to other participants in 2011 and prior years.”
Mr. Neale’s declaration is well written and does an excellent job of examining the myriad issues surrounding plaintiffs’ objections to the judge’s order. This blog simply isn’t long enough to do it justice. The reason I am bringing it up in this blog is because it is simply another example of why consumers of eDiscovery products need to perform a thorough analysis, including a PoC before choosing a product. The act of purchasing an eDiscovery solution includes the acquisition of a long term partner. You won’t easily be able to divorce yourself from the vendor or the product and there will be times when the credibility of the vendor is of paramount importance. Make sure you work with a company that puts transparency and honesty ahead of all else and I personally would recommend avoiding any company that inflates its claims or utilizes complex, difficult-to-explain black box technologies for anything, especially something like predictive coding. After all you never know when the answer you need will be stuck inside the black box.