By: Kate Bauer,
Technology-Assisted Review: Overcoming the Judicial Double-Standard
Although it has been nearly six years since technology-assisted review (“TAR”) first garnered judicial approval as a permissible form of document review,[1] inconsistent rulings about the amount of disclosure TAR requires have hobbled its adoption. Citing the importance of transparency, courts have tended to look more favorably on parties who agree to exchange the relevant and irrelevant documents used to train the algorithm.[2] When a party refuses to turn over these documents, some courts have regarded TAR usage with skepticism, or even disallowed it.[3] This emphasis on transparency overlooks a crucial point: courts do not have authority to compel parties to turn over irrelevant documents.[4] Further, it makes little sense to hold TAR—which has been shown to be more accurate and more cost-effective than traditional human review[5]—to a higher standard than exhaustive manual review. Indeed, concerns about accuracy are more appropriately leveled at the document review methods parties have traditionally used, which research has shown are notoriously inconsistent.[6] Accordingly, a court should regard a party’s decision to use TAR with at least as much deference as is given to traditional methodologies.
Technology-assisted review (“TAR”) is a method of document review in which attorneys manually review a subset of documents—commonly referred to as a “seed set”—for relevance, then submit those decisions to a computer algorithm.[7] The algorithm then (1) examines the documents the attorneys have coded, (2) identifies conceptually similar documents within the collection, and (3) predicts whether these conceptually similar documents are relevant or irrelevant based on how the attorneys coded similar documents in the seed set.[8] By amplifying attorney decisions on these training documents to similar documents across the document universe, TAR enables attorneys to quickly and accurately categorize documents they have not laid eyes on.[9] As with human reviewers, attorneys ensure they have properly trained the algorithm by reviewing samples of TAR predictions.[10] Where TAR predictions are incorrect, attorneys correct erroneous coding on the original training document that informed the TAR predictions.[11] TAR then adjusts its predictions for all similar documents in the database.[12]
Prior to the development of TAR algorithms, document review was understood to be a linear process in which teams of attorneys reviewed documents one by one. Linear human review is time-consuming, expensive, and inconsistent.[13] Reviewer coding errors amplify these issues because correcting errors requires manual re-review of existing coding decisions. With the volume of electronically-stored information increasing year over year, the expense of this model has become increasingly unsustainable.[14] To reduce the volume of data for review, attorneys frequently rely on keyword searches (the equivalent of CTRL + F); however, this method has also been shown to have substantial shortcomings.[15] The burdens of reviewing large amounts of data motivated recent revisions to the Federal Rules to emphasize “proportional[ity] to the needs of the case” as a factor that could be considered in limiting the scope of discovery.[16]
Despite increasing document volumes and research on the shortcomings of traditional review methods, the perception of linear human review as the “gold standard” of document review has stubbornly persisted.[17] Although empirical data demonstrates that TAR can achieve results as good as or superior to exhaustive manual review,[18] courts hesitated to stray from the accepted linear review model.
In Da Silva Moore v. Publicis Groupe, the first opinion to approve the use of TAR, Judge Andrew Peck lauded the defendant’s decision to turn over the relevant and irrelevant documents it had used to train the algorithm (“seed sets”).[19] Peck felt that this approach “reduced fears about the so-called ‘black box’ of technology,” and “highly recommend[ed] that counsel in future cases be willing to at least discuss, if not agree to, such transparency in the computer-assisted review process.”[20] However, Judge Peck’s praise for the defendant’s decision to disclose its seed set should not be interpreted as a judicial requirement disclosure; indeed, as he himself noted in a later opinion, Da Silva Moore stopped short of any such requirement.[21]
The Federal Rules of Civil Procedure require cooperation in discovery,[22] but they also dictate that irrelevant documents are outside the scope of discovery.[23] In defining scope, the Rules state, “Parties may obtain discovery regarding any nonprivileged matter that is relevant to any party’s claim or defense and proportional to the needs of the case” (emphasis added).[24] Because relevance to a claim or defense is a prerequisite for obtaining discovery, documents which attorneys identify as irrelevant are, by definition, outside the scope of discovery.[25]
While parties are still free to agree to exchange irrelevant documents if they wish, the Rules do not authorize courts to mandate such disclosure.[26] In the context of TAR, one court observed that requests for the whole seed set “reached well beyond the scope of any permissible discovery by seeking irrelevant or privileged documents used to tell the algorithm what not to find. That [an opposing party] has no right to discover irrelevant or privileged documents seems self-evident.”[27]
Good reasons exist for only authorizing courts to compel the production of relevant documents. As one court has observed, without such limitations a party seeking discovery might “obtain a permit to explore the documents in the possession of his adversary in the hope that he may find something which may adversely affect the adversary’s case, or that may prove helpful to the case of the movant.”[28] Even if a fishing expedition would be unlikely to unearth anything of value, the client may still wish to withhold irrelevant materials to safeguard against potential disclosure of sensitive, damaging, or embarrassing information of no consequence to the instant litigation. Because irrelevant materials often run the gamut from fantasy football updates and grocery lists to private medical information and romantic indiscretions, clients have compelling privacy interests at stake in withholding irrelevant materials.[29]
On the other hand, obstructionism in discovery has become an undeniably costly source of satellite litigation.[30] Transparency, it is argued, encourages parties to work together toward the resolution of substantive legal disputes rather than engaging in adversarial discovery disputes.[31] The influential Sedona Conference, for example, advocates for “cooperative, collaborative, transparent” discovery in order to reduce costs.[32] In this vein, the DOJ Antitrust Division requests that parties who utilize TAR provide one or more statistically significant samples of non responsive documents for DOJ review to ensure obviously responsive documents are not omitted. [33] However, while transparency may well help to ensure accuracy and reduce litigation costs, it comes at the expenses of privacy. Therefore, the decision to waive such privacy concerns must repose with the parties themselves, not their judges.[34]
Nevertheless, several courts have opined that parties should provide access to irrelevant documents in the name of transparency.[35] This sympathy may be based on misreading Da Silva Moore as requiring disclosure seed sets,[36] when in fact Peck merely praised and encouraged the defendant’s decision to disclose its seed sets.[37] For example, in one case the court refused to allow a party to use TAR, noting that the party proposing to use it was “unwilling to engage in the type of cooperation and transparency that . . . is needed for a predictive coding protocol to be accepted by the court . . . .”[38] In several other cases where the parties have agreed to exchange seed sets of their own volition, courts have made it clear that such transparency is expected.[39] However, requiring that parties exchange documents outside the scope of discovery is an impermissible exercise of judicial authority.
Judge Peck, in his subsequent Rio Tinto v. Vale opinion, observed that whether disclosure of a seed set is required is an “open question.” He points out that there are alternative methods to evaluate the adequacy of productions besides disclosing seed sets. Peck mentions three alternatives to seed sets that a requesting party might use to evaluate a production it receives: (1) identifying gaps in the production, (2) calculating statistical recall, or (3) performing a quality control of samples of documents identified as not responsive.[40] While identifying gaps can be performed by analyzing the contents of productions a party receives, the other two options—like disclosure of seed sets—would require the opposing party to disclose irrelevant documents in its possession.[41] Here we see one concern motivating these transparency demands: that parties using TAR might be failing—whether by neglect or by design—to disclose relevant documents. However, as with mandatory disclosure of seed sets, a judicial order requiring parties to reveal irrelevant documents as a condition of using TAR would be an ultra vires act.
The concern that relevant documents are being withheld from discovery is nothing new, and it is not specific to TAR. Courts have grappled with discovery disputes for the past century, extensively revising discovery rules in the process.[42] The following process emerged: relevant, unprivileged documents[43] are to be produced upon request,[44] and the producing attorneys must certify that they have made a reasonable inquiry in response to a discovery request.[45] To ensure cooperation, the Rules impose mandatory sanctions for improper certifications[46] and discretionary sanctions for failure to cooperate in discovery.[47] Ethical rules supply an affirmative obligation for attorneys to maintain technological competence[48] and to deal fairly with one another.[49]
If the existing discovery rules have been sufficient for traditional review, they must also be sufficient for TAR because both methods rely on the accuracy of human judgment. Concerns about bad faith aside, when a traditional document production withholds relevant documents, it does so as a result of the inconsistent judgment calls made by humans. Similarly, to the extent that a TAR production omits relevant documents, it does so because human judgments trained it to do so.[50] Document review is only as accurate as the judgments of the reviewers performing the review. Because TAR has been demonstrated to correctly amplify attorney coding decisions more consistently than teams of human reviewers,[51] concerns about the content of a production rest on the same foundation as they ever did: the potential for human error.
Elevated judicial scrutiny of TAR productions is also unwarranted because the rules already require attorneys to certify that they have made a reasonable inquiry in response to a discovery request, regardless of review methodology. [52] Attorneys who certify unreasonably deficient productions face mandatory sanctions,[53] and discretionary sanctions remain available for failure to cooperate.[54] These sanctions supply a powerful deterrent against lax review methodologies, whether TAR or traditional. Because TAR enables attorneys to apply their judgment across the document universe more consistently than manual review, TAR actually reduces deficiencies.[55]
For the reasons discussed, it is unreasonable to hold TAR to a higher standard than manual review methodologies.[56] First, TAR has been shown to be more accurate, cost-effective, and efficient than manual review.[57] Second, courts have no authority to require parties to disclose irrelevant documents.[58] Attempting to impose ultra vires disclosure requirements on the use of TAR forces parties who refuse to sacrifice their privacy to pursue costly, time-consuming manual review. Ironically, this process is likely to be less accurate than using TAR in the first place. Third, procedural safeguards already exist to ensure that attorneys are making reasonable inquiries in response to discovery requests.[59] If these procedural safeguards are sufficient to govern manual review despite its shortcomings, they must also be sufficient when employing a superior methodology such as TAR.
The stark reality is that the volume of discoverable data is continuing to grow. If judges don’t stop imposing unreasonable restraints on the use of TAR, litigation will become a privilege reserved for the few parties who can afford the exploding expense of discovery.
[1] Da Silva Moore v. Publicis Groupe & MSL Grp., 287 F.R.D. 182, 192 (S.D.N.Y. 2012).
[2] See, e.g., Rio Tinto PLC v. Vale S.A., 306 F.R.D. 125, 128-29 (S.D.N.Y. 2015) (Peck, M.J.); Progressive Cas. Ins. Co. v. Delaney, No. 2:11-cv-00678, 2014 U.S. Dist. WL 3563467, at *10 (D. Nev. 2014); In re Biomet M2a Magnum Hip Implant Prod. Liab. Litig., No. 3:12-MD-2391, 2013 U.S. Dist. WL 6405156, at *2 (N.D. Ind. 2013); Da Silva Moore, 287 F.R.D. at 192.
[3] See, e.g., Progressive Cas. Ins. Co., 2014 U.S. Dist. WL 3563467 at *10; Biomet, 2013 U.S. Dist. WL 6405156, at *1 (“An unexplained lack of cooperation in discovery can lead a court to question why the uncooperative party is hiding something, and such questions can affect the exercise of discretion. . . . But I don’t have any discretion in this dispute. I won’t order Biomet to reveal which of the documents it has disclosed were used in the seed set, but I urge Biomet to re-think its refusal.”).
[4] See Fed. R. Civ. P. 26(b)(1).
[5] See generally Maura R. Grossman & Gordon V. Cormack, Technology-Assisted Review in E-Discovery Can Be More Effective and More Efficient Than Exhaustive Manual Review, XVII Rich. J.L. & Tech. 1 (2011).
[6] Ellen M. Voorhees, Variations in Relevance Judgments and the Measurement of Retrieval Effectiveness, 36 Info. Processing & Mgmt. 697, 701 (2000) (concluding that assessors disagree that a document is relevant at least as often as they agree); David C. Blair & M.E. Maron, An Evaluation of Retrieval Effectiveness for a Full-Text Document-Retrieval System, 28 Commc’ns Ass’n Computing Mach. 289, 295-96 (1985) (finding that paralegals who thought they had retrieved 75% of relevant documents using iterative keyword searches had only found 20%).
[7] See Maura R. Grossman & Gordon V. Cormack, The Grossman-Cormack Glossary of Technology-Assisted Review, 7 Fed. Cts. L. Rev. 1, 29 (2013).
[8] See id. at 32.
[9] See id.
[10] Id. at 34.
[11] Id. at 33-34.
[12] Id.
[13] See The Sedona Conference, The Sedona Conference Best Practices Commentary on the Use of Search and Information Retrieval Methods in E-Discovery, 8 Sedona Conf. J. 189, 199 (2007); see also Grossman & Cormack, supra note 5, at 37; Voorhees, supra note 6; Blair & Maron, supra note 6.
[14] See The Sedona Conference, supra note 13 (“Even assuming that the profession had the time and resources to continue to conduct manual review of massive sets of electronic data sets (which it does not), the relative efficacy of that approach versus utilizing newly developed automated methods of review remains very much open to debate.”).
[15] See Blair & Maron, supra note 6, at 295-96 (1985).
[16] Fed. R. Civ. P. 26 advisory committee notes to 2015 amendment.
[17] See The Sedona Conference, supra note 13 (“[T]here appears to be a myth that manual review by humans of large amounts of information is as accurate and complete as possible– perhaps even perfect–and constitutes the gold standard by which all searches should be measured.”).
[18] See Grossman & Cormack, supra note 5, at 37 (reporting that manual reviewers identified between 25% and 80% of relevant documents, while technology-assisted review returned between 67% and 86%). See also Herbert L. Roitblat et al., Document Categorization in Legal Electronic Discovery: Computer Classification vs. Manual Review, J. of Am. Soc’y for Info. Sci. & Tech. 70, 79 (2010) (performing an empirical assessment to “answer the question of whether there was a benefit to engaging in a traditional human review or whether computer systems could be relied on to produce comparable results,” and concluding that “[o]n every measure, the performance of the two computer systems was at least as accurate (measured against the original review) as that of human re-review.”).
[19] Da Silva Moore, 287 F.R.D. at 192.
[20] Id.
[21] Rio Tinto PLC v. Vale S.A., 306 F.R.D. 125, 128-29 (S.D.N.Y. 2015) (Peck, M.J.).
[22] Fed. R. Civ. P. 37.
[23] Fed. R. Civ. P. 26(b)(1).
[24] Id.
[25] Id.
[26] See id.
[27] Biomet, 2013 U.S. Dist. WL 6405156, at *1.
[28] United States v. Becton, Dickinson & Co., 30 F.R.D. 132, 134 (D.N.J. 1962).
[29] Matthew Lynch, Discovery Evolutions Hold Promise for Greater Privacy Benefits for Litigants, IT-Lex Technology Law, Oct. 22, 2013, available at https://www.foley.com/discovery-evolutions-hold-promise-for-greater-privacy-benefits-for-litigants-10-22-2013/.
[30] See The Sedona Conference, The Sedona Conference Cooperation Proclamation, 10 Sedona Conf. J. 331, 331 (2009).
[31] See id.
[32] Id.
[33] Tracy Greer, Technology-Assisted Review and Other Discovery Initiatives at the Antitrust Division, Department of Justice (2014), available at https://www.justice.gov/sites/default/files/atr/legacy/2014/03/27/304722.pdf.
[34] Fed. R. Civ. P. 26(b)(1).
[35] See, e.g., Bridgestone Ams., Inc. v. Int’l Bus. Machs. Corp., No. 3:13-1196, 2014 U.S. Dist. WL 4923014 at *1 (M.D. Tenn. 2014); Progressive Cas. Ins. Co., 2014 U.S. Dist. WL 3563467, at *11; Transcript of Record at 9, 14, Fed. Hous. Fin. Agency v. JPMorgan Chase & Co., No. 1:11-cv-06188 (S.D.N.Y. July 24, 2012).
[36] See Progressive Cas. Ins. Co., 2014 U.S. Dist. WL 3563467, at *11 (declining to allow predictive coding when counsel was “unwilling to engage in the type of cooperation and transparency that . . . is needed for a predictive coding protocol to be accepted by the court . . .”). But see Biomet, 2013 U.S. Dist. WL 6405156, at *2 (holding that, while a party’s failure to disclose seed set fell below Sedona Conference Cooperation Proclamation standard, the Proclamation “can’t provide [the court] with authority to compel discovery of information not made discoverable by the Federal Rules.”).
[37] See Da Silva Moore, 287 F.R.D. at 192. (“[Defendant] confirmed that ‘[a]ll of the documents that are reviewed as a function of the seed set, whether [they] are ultimately coded relevant or irrelevant, aside from privilege, will be turned over to’ plaintiffs. . . . This Court highly recommends that counsel in future cases be willing to at least discuss, if not agree to, such transparency in the computer-assisted review process.”). See also Rio Tinto, 306 F.R.D. at 128 (Peck, M.J.) (“One TAR issue that remains open is how transparent and cooperative the parties need to be with respect to the seed or training set(s).”).
[38] Progressive Cas. Ins. Co., 2014 U.S. Dist. WL 3563467, at *11.
[39] See, e.g., Bridgestone, 2014 U.S. Dist. WL 4923014 at *1 (“[O]penness and transparency in what Plaintiff is doing will be of critical importance. Plaintiff has advised that they will provide the seed documents they are initially using to set up predictive coding. The Magistrate Judge expects full openness in this matter.”); Transcript of Record at 9, 14, Fed. Hous. Fin. Agency (bench decision requiring transparency and cooperation, including giving the plaintiff full access to the seed set’s responsive and non-responsive documents except privileged).
[40] Rio Tinto, 306 F.R.D. at 128-29 (“Requesting parties can insure that training and review was done appropriately by other means, such as statistical estimation of recall at the conclusion of the review as well as by whether there are gaps in the production, and quality control review of samples from the documents categorized as non-responsive.”).
[41] In order to calculate recall, a party must review a mix of relevant and irrelevant documents. The party then codes the documents for relevance, and compares its decisions about relevance against the algorithm’s predictions about relevance. The more alignment between human reviewer and the algorithm’s predictions, the better the recall. See Grossman & Cormack, supra note 7, at 27 (defining “recall” as “[t]he fraction of Relevant Documents that are identified as Relevant by a search or review effort.”).
[42] See, e.g., Fed. R. Civ. P. 26 advisory committee notes to 1946, 1970, 1980, 1983, 1993, 2000, 2006, and 2015 amendments; Fed. R. Civ. P. 37 advisory committee notes to 1970, 1980, 1993, 2000, 2006, and 2015 amendments.
[43] Fed. R. Civ. P. 26(b)(1).
[44] Fed. R. Civ. P. 34(a).
[45] Fed. R. Civ. P. 26(g).
[46] Fed. R. Civ. P. 26(g)(3).
[47] Fed. R. Civ. P. 37(b).
[48] Model Rules of Prof’l Conduct r. 1.1 (Am Bar Ass’n 2016).
[49] Id. at r. 3.4.
[50] See Grossman & Cormack, supra note 7, at 29.
[51] See Grossman & Cormack, supra note 5, at 37.
[52] Fed. R. Civ. P. 26(g).
[53] Id.
[54] Fed. R. Civ. P. 37(b).
[55] See Grossman & Cormack, supra note 5, at 61.
[56] Rio Tinto, 306 F.R.D. at 129 (“[I]t is inappropriate to hold TAR to a higher standard than keywords or manual review. Doing so discourages parties from using TAR for fear of spending more in motion practice than the savings from using TAR for review.”).
[57] See Grossman & Cormack, supra note 5; Voorhees, supra note 6; Blair & Maron, supra note 6.
[58] Fed. R. Civ. P. 26(b)(1).
[59] Fed. R. Civ. P. 26(g)(3); Fed. R. Civ. P. 37(b).
Image Source: http://www.ihi.org/communities/blogs/lean-accounting-how-to-turn-data-overload-into-continuous-improvement.
