DownloadPDF

Cite as: Bennett B. Borden & Jason R. Baron, Finding the Signal in the Noise: Information Governance, Analytics, and the Future of Legal Practice, 20 Rich. J.L. & Tech. 7 (2014), http://jolt.richmond.edu/v20i2/article7.pdf.

 

Bennett B. Borden* and Jason R. Baron**

 

Introduction

[1]        In the watershed year of 2012, the world of law witnessed the first concrete discussion of how predictive analytics may be used to make legal practice more efficient.  That the conversation about the use of predictive analytics has emerged out of the e-Discovery sector of the law is not all that surprising: in the last decade and with increasing force since 2006—with the passage of revised Federal Rules of Civil Procedure that expressly took into account the fact that lawyers must confront “electronically stored information” in all its varieties—there has been a growing recognition among courts and commentators that the practice of litigation is changing dramatically.  What needs now to be recognized, however, is that the rapidly evolving tools and techniques that have been so helpful in providing efficient responses to document requests in complex litigation may be used in a variety of complementary ways to the discovery process itself.

[2]        This Article is informed by the authors’ strong views on the subject of using advanced technological strategies to be better at “information governance,” as defined herein.  If a certain evangelical strain appears to arise out of these pages, the authors willingly plead guilty.  One need not be an evangelist, however, but merely a realist to recognize that the legal world and the corporate world both are increasingly confronting the challenges and opportunities posed by “Big data.”[1]  This Article has a modest aim: to suggest certain paths forward where lawyers may add value in recommending to their clients greater use of advanced analytical techniques for the purpose of optimizing various aspects of information governance.  No attempt at comprehensiveness is aimed for here; instead, the motivation behind writing this Article is simply to take stock of where the legal profession is, as represented by the emerging case law on predictive coding represented by Da Silva Moore,[2] and to suggest that the expertise law firms have gained in this area may be applied in a variety of related contexts.

[3]        To accomplish what we are setting out to do, we will divide the discussion into the following parts: first, a synopsis of why and how predictive coding first emerged against the backdrop of e-Discovery.  This discussion will include a brief overview of predictive coding with references to the technical literature, as the subject has been recently covered exhaustively elsewhere.  Second, we will define what we mean by “Big data,” “analytics,” and “information governance,” for the purpose of providing a proper context for what follows.  Third, we will note those aspects of an information governance program that are most susceptible to the application of predictive coding and related analytical techniques.  Perhaps of most value, we wish to share a few “early” examples of where we as lawyers have brought advanced analytics, like predictive coding, to bear in non-litigation contexts and to assist our clients in creative new ways.  We fully expect that what we say here will be overrun with a multitude of real-life use cases soon to emerge in the legal space.  Armed with the knowledge that we are attempting to catch lightning in a bottle and that law reviews on subjects such as this one have ever decreasing “shelf-lives”[3] in terms of the value proposition they provide, we proceed nonetheless.

A.  The Path to Da Silva Moore

[4]        The Law of Search and Retrieval.  In the beginning, there was manual review.  Any graduate of a law school during the latter part of the twentieth century who found herself or himself employed before the year 2000 at a law firm specializing in litigation and engaged in high-stakes discovery remembers well how document review was conducted: legions of lawyers with hundreds if not thousands of boxes in warehouses, reviewing folders and pages one-by-one in an effort to find the relevant needles in the haystack.[4]  (Some of us also remember “Sheparding” a case to find subsequent citations to it, using red and yellow booklets, before automated key-citing came along.)  Although manual review continues to remain a default practice in a variety of more modest engagements, it is increasingly the case that all of discovery involves “e-Discovery” of some sort—that the world is simply “awash in data”[5] (starting but by no means ending with email, messages and other textual documents of all varieties), and that it will increasingly be the unusual case of any size where documents in paper form still loom large as the principal source of discovery.

[5]        At the turn of the century, the dawning awareness of the need to deal with a new realm of electronically stored information (“ESI”) led to burgeoning efforts on many fronts, including, for example, the creation of The Sedona Conference working group on electronic document retention and production, members of which drafted The Sedona Conference Principles: Addressing Electronic Document Production (2005; 2d ed. 2007) and its “prequel,” The Sedona Guidelines: Best Practice Guidelines and Commentary for Managing Records and Information in the Electronic Age (2005; 2d ed. 2007).  These early commentaries, including a smattering of pre-2006 case law,[6] recognized that changes in legal practice were necessary to accommodate the big changes coming in the world of records and information management within the enterprise.  Subsequent developments would constitute various complementary threads leading to the greater use of analytics in the legal space.

[6]        First, part of that early recognition was that in an inflationary universe of rapidly expanding amounts of ESI, new tools and techniques would be necessary for the legal profession to adapt and keep up with the times.[7]  By the time of adoption of the revised Federal Rules of Civil Procedure in 2006, which expressly added the term “ESI” to supplement “documents” in the rule set applicable to discovery practice, the legal profession was well aware of the need to perform automated searches in the form of keyword searching within large data sets as the only realistically available means for sorting information into relevant and non-relevant evidence in particular engagements, be they litigation or investigations.  So too, it was recognized early on in commentaries[8] and followed by case law[9] that keyword searching, as good a tool as it was, had profound limitations that in the end do not scale well.  At the end of the day, even being able to limit or cull down a large data set to one percent of its original size through the use of keywords leaves the lawyer with the near impossible task of manually reviewing a very large set of documents at great cost.[10]

[7]        Second, in evolving e-Discovery practice after 2006, a growing recognition also occurred around the idea that e-Discovery workflows are an “industrial” process in need of better metrics and measures for evaluating the quality of productions of large data sets.  As recognized in The Sedona Conference Commentary on Achieving Quality in E-discovery (Post-Public Comment Version 2013):

The legal profession has passe
d a crossroads: When faced with a choice between continuing to conduct discovery as it had “always been practiced” in a paper world—before the advent of computers, the Internet, and the exponential growth of electronically stored information (ESI)—or alternatively embracing new ways of thinking in today’s digital world, practitioners and parties acknowledged a new reality and chose progress.  But while the initial steps are completed, cost-conscious clients and over-burdened judges are increasingly demanding that parties find new approaches to solve litigation problems.[11]

 [8]        The Commentary goes on to suggest that the legal profession would benefit from greater

awareness about a variety of processes, tools, techniques, methods, and metrics that fall broadly under the umbrella term “quality measures” and that may be of assistance in handling ESI throughout the various phases of the discovery workflow process.  These include greater use of project management, sampling, machine learning, and other means to verify the accuracy and completeness of what constitutes the “output” of e-[D]iscovery.  Such collective measures, drawn from a wide variety of scientific and management disciplines, are intended only as an entry-point for further discussion, rather than an all-inclusive checklist or cookie-cutter solution to all e-[D]iscovery issues.[12]

 [9]        Indeed, more recent case law has recognized the need for quality control, including through the use of greater sampling, iterative methods, and phased productions in line with principles of proportionality.[13]  Still other case law has emphasized the need for cooperation among parties in litigation on technical subjects, especially at the margins of, or outside the range of, lawyer expertise if not basic competence.

[10]      Active or supervised “machine learning,” as referred to here in the context of e-Discovery, refers to a set of analytical tools and techniques that go by a variety of names, such as “predictive coding,” “computer-assisted review,” and “technology assisted review.”  As explained in one helpful recent monograph:

Predictive coding is the process of using a smaller set of manual reviewed and coded documents as examples to build a computer generated mathematical model that is then used to predict the coding on a larger set of documents.  It is a specialized application of a class of techniques referred to as supervised machine-learning in computer science.  Other technical terms often used to describe predictive coding include document (or text) “classification” and document (or text) “categorization.”[14]

 [11]      And as stated in The Sedona Conference Best Practices Commentary on the Use of Search and Information Retrieval Methods in E-Discovery (Post-Public Comment Version 2013):

Generally put, computer- or technology-assisted approaches are based on iterative processes where one (or more) attorneys or [Information Retrieval] experts train the software, using document exemplars, to differentiate between relevant and non-relevant documents.  In most cases, these technologies are combined with statistical and quality assurance features that assess the quality of the results.  The research . . . has demonstrated such techniques superior, in most cases, to traditional keyword based search, and, even, in some cases, to human review.

 The computer- or technology-assisted review paradigm is the joint product of human expertise (usually an attorney or IR expert working in concert with case attorneys) and technology.  The quality of the application’s output, which is an assessment or ranking of the relevance of each document in the collection, is highly dependent on the quality of the input, that is, the human training. Best practices focus on the utilization of informed, experienced, and reliable individuals training the system.  These individuals work in close consultation with the legal team handling the matter, for engineering the application. Similarly . . . the defensibility and usability of computer- or technology-assisted review tools require the application of statistically-valid approaches to selection of a “seed” or “training” set of documents, monitoring of the training process, sampling, and quantification and verification of the results.[15] 

A discussion of the mathematical algorithms that underlie predictive coding is beyond the intended scope of this Article, but the interested reader should refer to references cited at the margin to understand better what is “going on under the hood” with respect to the mathematics involved.[16]

[12]      The Da Silva Moore Precedent.  The various threads in search and retrieval law, including the need for advanced search methods applied to document review in a world of increasingly large data sets, were well known by 2012.  In February 2012, drawing on recent research and scholarship emanating out of the Text Retrieval Conference (TREC) Legal Track[17] and the 2007 public comment version of The Sedona Conference Search Commentary,[18] Judge Peck approached the Da Silva Moore case as an appropriate vehicle to provide a judicial blessing for the use of predictive coding in e-Discovery.  In doing so, however, Judge Peck’s opinion may also be viewed as setting the stage for greater use of analytics generally in the information governance practice area, beyond “mere” e-Discovery.

[13]      Plaintiffs in Da Silva Moore brought claims of gender discrimination against defendant advertising conglomerate Publicis Groupe and its United States public relations subsidiary, defendant MSL Group.[19]  Prior to the February 2012 opinion issued by Judge Peck, the parties had already agreed that defendant MSL would use predictive coding to review and produce relevant documents, but disagreed on methodology.[20]  Defendant MSL proposed starting with the manual review of a random sample of documents to create a “seed set” of documents that would be used to train the predictive coding software.[21]  Plaintiffs would participate in the creation of the “seed set” of documents by offering keywords.[22] All documents reviewed during the creation of the “seed set,” relevant or irrelevant, would be provided to plaintiffs.[23]

[14]      After creation of the seed set of documents, MSL proposed using a series of “iterative rounds” to test and stabilize the training software.[24]  The results of these iterative rounds would be provided to plaintiffs, who would be able to provide feedback to further refine the searches.[25]   Judge Peck accepted MSL’s proposal.[26]  Plaintiffs filed objections with the district judge on the grounds that Judge Peck’s approval of MSL’s protocol unlawfully disposed of MSL’s duty under Federal Rule of Civil Procedure 26(g) to certify the completeness of its document collection, and the methodology in MSL’s protocol was not sufficiently reliable to satisfy Federal Rule of Evidence 702 and Daubert.[27]

 [15]      Judge Peck found the plaintiffs’ objections to be misplaced and irrelevant.[28]  With respect to Federal Rule of Civil Procedure 26(g), Judge Peck commented that no attorney could certify the completeness of a document production as large as MSL’s. Moreover, Federal Rule of Civil Procedure 26(g) did not require the type of certification plaintiffs described.[29]  Further, Federal Rule of Evidence 702 and Daubert are applicable to expert methodology, not to methodologies used in electronic discovery.[30]  Judge Peck went on to note that the decision to allow computer-assisted review in this case was easy because the parties agreed to this method of document collection and review.[31]  While computer-assisted review may not be a perfect system, he found it to be more
efficient and effective than using manual review and keyword searches to locate responsive documents.[32]  Use of predictive coding was appropriate in this case considering:

 (1) the parties’ agreement, (2) the vast amount of ESI to be reviewed (over three million documents), (3) the superiority of computer-assisted review to the available alternatives (i.e., linear manual review or keyword searches), (4) the need for cost effectiveness and proportionality under Rule 26(b)(2)(C), and (5) the transparent process proposed by MSL.[33]

 [16]      In issuing this opinion, Judge Peck became the first judge to approve the use of computer-assisted review.[34]  He also stressed the limitations of his opinion, stating that computer-assisted review may not be appropriate in all cases, and his opinion was not intended to endorse any particular computer-assisted review method.[35]  However, Judge Peck encouraged the Bar to consider computer-assisted review as an available tool for “large-data-volume cases” where use of such methods could save significant amounts of legal fees.[36]  Judge Peck also stressed the importance of cooperation, or what he called “strategic proactive disclosure of information.”  If counsel is knowledgeable about the client’s key custodians and fully explains proposed search methods to opposing counsel and the court, those proposed search methods are more likely to be approved.  To sum up his opinion, Judge Peck noted that “[c]ounsel no longer have to worry about being the ‘first’ or ‘guinea pig’ for judicial acceptance of computer-assisted review. . . . Computer-assisted review now can be considered judicially-approved for use in appropriate cases.”[37]  In the two years since Da Silva Moore, in addition to cases in which the parties have agreed upon a predictive coding methodology,[38] courts have confronted the issue of having to rule on either the requesting or responding party’s motion to compel a judicial “blessing” of the use of predictive coding (however termed).  In Global Aerospace,[39] the responding party asked that the court approve its own use of such technique; in Kleen Products, the requesting party made an ultimately unsuccessful demand for a “do-over” in discovery, where the responding party had used keyword search methods and the plaintiffs were demanding that more advanced methods be tried.[40]  In the EOHRB case, the Court sua sponte suggested that the parties consider using predictive coding, including the same vendor.[41]  And in the In re Biomet case,[42] the court approved a predictive coding methodology over the objections of the requesting party.  These cases represent only some of the reported decisions to date, and we suspect that there will be dozens of reported cases and many more unreported ones in the near term.

[17]      As recognized in these cases (implicitly or explicitly), as well as in a growing number of commentaries,[43] predictive coding is an analytical technique holding the promise of achieving much greater efficiencies in the e-Discovery process.  Notwithstanding Da Silva Moore’s call to action, it needs to be conceded, however, that the research has not proven that active machine learning techniques will always achieve greater scores than keyword search or manual review.[44]  Additionally, we bow to the reality that in a large class of cases the use of predictive coding is currently infeasible or unwarranted, especially as a matter of cost.[45]

[18]      Nevertheless, it seems apparent that the legal profession finds itself in a new place—namely, in need of recognizing that artificial intelligence techniques are growing in strength from year to year—and thus it appears to be only a matter of time until a much greater percentage of complex cases involving a large magnitude of ESI will constitute good candidates for lawyers using predictive coding techniques, both as available currently and as improved with future technological progress.  As William Gibson once put it, “the future is here, it’s just not evenly distributed.”[46]

 B.  Information Governance and Analytics in the Era of Big Data

[19]      We are now in a post-Da Silva Moore, “Big data” era where lawyers are on constructive (if not actual) notice of a world of technology assisted review techniques available at least in the sphere of e-Discovery.  The proposition being advanced is that the greater revelation of Da Silva Moore is how similar the techniques being put forward as best practices in e-Discovery fit a larger realm of issues familiar to lawyers, many of which fall within what is increasingly being recognized as “information governance” practice.  It is here where we can break new ground in our legal practice by recommending the use of these advanced techniques to solve real-world problems of our clients.  First, however, some definitions are in order to better frame the legal issues that will follow in Section C.

[20]      Big data.  It has been noted that “Big data is a loosely defined term used to describe data sets so large and complex that they become awkward to work with using standard statistical software.”[47]  Alternatively, “Big data” is a term that “describe[s] the technologies and techniques used to capture and utilize the exponentially increasing streams of data with the goal of bringing enterprise-wide visibility and insights to make rapid critical decisions.”[48]

[21]      The fact that the data encountered within the corporate enterprise increasingly is indeed “big” means, at least according to Gartner, that it not only has volume, but velocity and complexity as well.[49]  As Bill Franks has put it, “What this means is that you aren’t just getting a lot of data when you work with big data.  It’s also coming at you fast, it’s coming at you in complex formats, and it’s coming at you from a variety of sources.”[50]  These elements all significantly contribute to the challenge of finding signals in the noise.

[22]      These definitions seem to get us closer to what makes Big data a new and interesting phenomenon in the world: it is not its volume alone, but the fact that we are able to “mine” large data sets using new and advanced techniques to uncover unexpected relationships, patterns and categories within these data sets, that makes the field potentially exciting.  Indeed, “it is tempting to understand big data solely in terms of size. But that would be misleading. Big data is also characterized by the ability to render into data many aspects of the world that have never been quantified before; call it ‘datafication.’”[51]

[23]      Analytics.  Second, we need to place “predictive coding” as one form of active machine learning in the context of the broader realm of “analytics.”  In their book, Keeping Up With the Quants: Your Guide To Understanding and Using Analytics,[52] authors Thomas Davenport and Jinho Kim provide a useful construct in categorizing the newly emergent field of “analytics”: they define analytics to mean “the extensive use of data, statistical and quantitative analysis, explanatory and predictive models, and fact-based management to drive decisions and add value,” going on to say that “[a]nalytics is all about making sense of big data, and using it for competitive advantage.”  The authors divide the world of analytics into three categories:

 (i)             descriptive analytics – gathering, organizing, tabulating and depicting data;

(ii)              predictive analytics – using data to predict future courses of action; and

(iii)             prescriptive analytics – recommendations on future courses of action.[53]

 
[24]      To the extent that “predictive coding” has been used to date to have machines “predict” relevancy in large ESI data sets, the term comfortably can be said to fall within category (ii).   But the world of analytics is a larger universe, encompassing a greater number of mathematical magic tricks,[54] and this should be kept in mind as we choose to limit our discussion here to a few examples of how predictive coding as one form of analytics may be usefully applied in non-traditional contexts.[55]

 [25]      Corporations (much ahead of the legal profession) have rushed headlong during the past half-decade to use a variety of analytics to understand the Big data they increasingly hold, to add value, and to improve the bottom line.[56]  A 2013 AIIM study indicates that corporations find analytics to be useful in a variety of settings.[57]

[26]      Information Governance.  “Information governance,” as defined in The Sedona Conference’s recently published Commentary on the subject, means:

 an organization’s coordinated, interdisciplinary approach to satisfying information legal and compliance requirements and managing information risks while optimizing information value.  As such, Information Governance encompasses and reconciles the various legal and compliance requirements and risks addressed by different information focused disciplines, such as records and information management (“RIM”), data privacy, information security, and e-[D]iscovery.[58]

 Or, as highlighted by the seminal law review article devoted to information governance written by Charles R. Ragan who quotes Barclay Blair in defining information governance as a “‘new approach’ that “builds upon and adapts disciplines like records management and retention, archiving business analytics, and IT governance to create an integrated model for harnessing and controlling enterprise information . . . [I]t is an evolutionary model that requires organizations to make real changes.”[59]

[27]      As the Sedona IG Commentary highlights, “many organizations have traditionally used siloed approaches when managing information.”[60]  The “core shortcoming” of this approach is “that those within particular silos are constrained by the culture, knowledge, and short-term goals of their business unit, administrative function, or discipline.”[61]  This leads in turn to key actors within the organization having “no knowledge of gaps and overlaps in technology or information in relation to other silos. . . .”[62]  In such situations, “[t]here is no overall governance or coordination for managing information as an asset, and there is no roadmap for the current and future use of information technology.”[63]

[28]      The Sedona IG Commentary goes on to provide eleven principles of what constitutes good IG practices, of which Principle 10 is of special relevance to our discussion here: “An organization should consider leveraging the power of new technologies in its Information Governance program.”[64]  As stated therein,

         Organizations should consider using advanced tools and technologies to perform various types of categorization and classification activities. . . such as machine learning, auto-categorization, and predictive analytics to perform multiple purposes, including (i) optimizing the governance of information for traditional RIM [records and information management]; (ii) providing more efficient and more efficacious means of accessing  information for e-discovery, compliance, and open records laws, and (iii) advancing sophisticated business intelligence across the enterprise.[65]

 With respect to the latter category, the Commentary goes on to specifically identify areas where predictive analytics may be used in compliance programs “to predict and prevent wrongful or negligent conduct that might result in data breach or loss,” as a type of “early warning system.”[66] It is precisely this latter type of conduct that we wish to primarily explore in the next section, along with a few final words on using analytics with auto-categorization for the purpose of records classification and data remediation.

C.  Applying the Lessons of E-Discovery In Using Analytics for Optimal Information Governance: Some Examples

 [29]      Advanced analytics are increasingly being used in the e-Discovery context because the legal profession has begun to realize the limitations of manual and keyword searching, while at the same time seeing how advanced techniques are at least as efficacious and far more efficient in a wide variety of substantial engagements.  But more efficient and at least as equally effective at doing what, precisely?  In e-Discovery, the primary information task involves separating relevant from non-relevant, and to a secondary degree, privileged from non-privileged information, in documents and ESI.  Indeed, lawyers are under a duty to make “reasonable”—not perfect—efforts to find all relevant documents within the scope of a given discovery request.[67]  The illusiveness of this quest in an exponentially expanding data universe is becoming increasingly apparent to many.[68]

[30]      Moreover, the degree of success in being able to either find or demand substantial amounts of relevant information is not (nor should it be) the fundamental goal or point of engaging in e-Discovery.[69]  Rather, the liberal discovery rules that at least U.S. lawyers operate within have as their underlying purpose the ferreting out of important, material facts to the case at hand.  The increasingly overwhelming nature of ESI poses clear technological obstacles to a lawyer en route to efficiently engaging in developing facts from all those relevant documents to determine what happened and why.[70]  The promise of using an advanced analytical method such as predictive coding is its ability to quickly find and rank-order the most relevant documents for answering these questions.  For once we determine how something happened and why, it is relatively straightforward to figure out the parties’ respective rights, responsibilities, and even liability.  That is precisely the point of litigation, and the purpose of the Rules that govern it.[71]  And, facts drive it all.

[31]      Given our increasing ability in litigation in finding the most relevant needles (i.e., facts) in the Big data haystack, it stands to consider whether similar methods may be successfully applied in non-litigation contexts.  Somewhat paradoxically, however, experience indicates that there are advantages to dealing with larger volumes of data when applying analytical tools and methods to solve corporate legal issues.  That is, while a vast amount of data residing in corporate networks and repositories admittedly poses complex information governance challenges, the volume of Big data also may be a boon to the investigator simply trying to figure out what happened.  This is the case because there are simply many more data points from which to derive facts.  One can liken the phenomenon to the difference in quality of a one-megapixel versus a ten-megapixel picture: the difference in the quality of the image is a function of the greater density of points of illumination.

[32]      Big data is more data, and more data means the potential for a more complete picture of what happened in a given situation of interest, assuming of course that the facts can be captured efficiently.  The problem is not one of volume, but of visibility.  In the era of Big data, the investigator with the more powerful analytical methods, who can search into vast repositories of ESI to draw out the facts that are critical to the question at hand, is king (or queen).  This is where the skillful application of advanced analytics to Big data can bring about some remarkable results.  The true strategic advant
age of advanced analytics is the speed with which an accurate answer can be ascertained.[72]

[33]      True Life Example #1.[73]  A corporate client is being sued by a former employee in a whistleblower qui tam action.[74]  Because of the False Claims Act allegations, the suit represented a significant threat to the company.  The corporation retains counsel to understand the client’s information systems as well as its key players, and to assist in the implementation of a litigation hold.  Counsel strategically targets the data most likely to shed light on the facts.  The law firm’s Fact Development Team applies advanced analytics to 675,000 documents, and within four days knows enough to defend the client’s position that the allegations are indisputably baseless.  All of this is done before the answer to the Complaint was due.

[34]      Armed with this information, counsel for the corporation approached plaintiff’s counsel and asked to meet.  Prior to the meeting, the corporation voluntarily produced 12,500 documents that laid out the parties’ position precisely.  Counsel then met with plaintiff’s counsel and walked them through the evidence, laying out all the facts.  The case ended up being settled within days for what amounted to nuisance value based on a retaliation claim—without any discovery, and at a small fraction of the cost budgeted for the litigation.

[35]      This example indicates that the real power of advanced analytics is not merely in potentially reducing the cost of vexatious litigation, but rather the strategic advantage that comes with counsel getting to an answer quickly and accurately.  This precise strategic advantage has many applications outside of litigation, each of which involves an aspect of optimizing information governance.

[36]      Only a short step away from the direct litigation realm is using advanced analytics for investigations, either in response to a regulatory inquiry or for purely internal purposes.  As we have already seen, corporate clients are often faced with circumstances where determining whether an allegation is true, and the scope of the potential problem if it is, is critically important.  Often, management must wait, unsure of their company’s exposure and how to remediate it, while traditional investigation techniques crawl along.  However, with the skillful application of advanced analytics upon the right data set, accurate answers can be determined with remarkable speed.

[37]      True Life Example #2.  A highly regulated manufacturing client decided to outsource the function of safety testing some of its products.  A director of the department whose function was being outsourced was offered a generous severance package.  Late on a Friday afternoon, the soon-to-be former director sent an email to the company’s CEO demanding four times the severance amount and threatened to go to the company’s regulator with a list of ten supposed major violations that he described in the email if he did not receive what he was asking for.  He gave the company until the following Monday to respond.

[38]      The lawyers were called in.  They analyzed the list of allegations and determined which IT systems would most likely contain data that would prove their veracity and immediately pulled the data.  Applying advanced analytics, the law firm’s Fact Development Team analyzed on the order of 275,000 documents in thirty-six hours.  By that Monday morning, counsel was able to present a report to the company’s board indisputably proving that the allegations were unfounded.

[39]      True Life Example #3.  A major company received a whistleblower letter from a reputable third party alleging that several senior personnel were involved with an elaborate kickback scheme that also involved FCPA violations.  If true, the company would have faced serious regulatory and legal issues, as well as major internal difficulties.  Because of the extremely sensitive nature of the allegations, a traditional investigation was not possible; even knowing certain personnel were under investigation could have had immense consequences.

[40]      The lawyers were tasked with determining whether there was any information within the company’s possession that shed any light on the allegations.  If there were, the company would proceed to take whatever steps were required.  The investigation was of such a secret nature that no one was authorized to involve the internal IT staff.  Fortunately, counsel knew the company and its information systems well.  Over a weekend, they were able to pull 8.5 million documents from relevant systems using the law firm’s personnel.  This turned out to be a highly complex investigation involving a number of potential subjects, where the task involved tracking the subject’s travel, meetings with suppliers, subsequent sales orders and fulfillments, rebates and promotions, all across several years.

[41]      Again, applying advanced analytics, the law firm’s Fact Development Team analyzed the 8.5 million documents in ten days.  They were able to prove that the allegations were largely baseless, and precisely where there were potential areas of concern.  Counsel also was able to make clear recommendations for areas of further investigation and for modifying compliance tracking and programs.  The company was able to act quickly and with certainty.  These real-life use cases illustrate how the power of analytics enhances the ability of lawyers to provide legal advice under conditions of “certainty” previously unobtainable, at least in the past few decades of the digital era.  “Certainty” is a somewhat foreign concept in the law—lawyers tend to be a conservative and caveating bunch, largely because certainty has historically been hard to come by, or at least prohibitively expensive.  With advanced analytics and good lawyers who know how to use these new tools, that is no longer necessarily the case.  There is so much data that if one cannot, after a reasonable effort, find evidence of a fact in the vastness of a company’s electronic information (as long as you have the right information), the fact most likely is not true.  Such has been illustrated, proving a negative is particularly useful in investigations.

[42]      Using advanced analytics (and good lawyering) for investigations is not that far removed from using it for litigation: one is still attempting to find the answer to the question of what happened and why. But there are many other questions that companies would like to ask of their data.  And indeed, both the analytics tools and the fact development techniques used in litigation and investigations can be “tuned” to solve a variety of novel issues facing our clients.

[43]      For example, analytics can be used to vet candidates for political appointments as well as candidates for senior leadership positions.  Due to the candid nature of the medium, providing access to corporate email coupled with using analytic capabilities allows for an accurate picture to be drawn before a decision is made with regard to making a candidate your next CEO or running mate.  Analytics can be used to analyze business divisions to identify good and bad leaders, how decisions are made, why a division is more successful than another, and many more similar applications.

[44]      Quite simply, a company’s data is the digital imprint of the actions and decisions of all of its managers and employees.  Having insight into those actions and decisions can be immensely valuable.  That value has lain largely fallow, hidden in plain sight because the valuable wheat could not effectively be sifted from the chaff.  With the proper application of advanced analytics, that is no longer the case.  The answers we can obtain are limited only by the creativity of management in asking the right questions.

[45]      True Life Example #4.  Advanced a
nalytics used upon the major acquisition of another company by a corporate client.  As with most acquisitions, the client undertook traditional due diligence, gathering information from the target regarding its financial performance, customers, market share, receivables, potential liabilities, and came up with a valuation, an appropriate multiplier, and a final purchase price.  Also as is typical, the acquisition agreement contained a provision such that if the disclosures made by the target were found to be off by a certain margin within thirty days of the acquisition, the purchase price would be adjusted.

[46]      The moment the acquisition closed, the corporate client then owned all of the target’s information systems.  Having some concern about the bases for some of the target’s disclosures, at the client’s request counsel proceeded to use analytics on those newly acquired systems to determine what we could about those disclosures.  Preparing a company for sale is a complicated affair, with many people involved in gathering information to present to the acquirer to satisfy due diligence.  This gathering and presentation of information is done primarily through electronic means—and leaves a trail.

[47]      Using advanced analytics, the law firm’s Fact Development Team traced the compilation of the target’s due diligence information, including all of the discussion that went along with it.  They were able to understand the source of each disclosure, the reasonableness of its basis, and any weaknesses within it.  They uncovered disagreements within the target over such things as what the right numbers were, or how much of a liability to disclose.  Using this information, counsel prepared a claim in accord with the adjustment provision seeking twenty-five percent of the purchase price totaling millions of dollars.  The claim was primarily composed using quotes from their own documents.  It is difficult to argue with yourself.

[48]      As demonstrated, using advanced analytics in the form of predictive coding and similar technologies can accomplish some notable aims.  But each of the prior examples uses data to look back to determine what has already occurred: the descriptive use of analytics.[75]  This is extremely valuable.  But for many of a law firm’s clients, it would be even more useful to be able to catch bad actors while the misconduct was occurring, or even to predict misconduct before it happens.

[49]      Based on the anecdotal experience gathered from many past investigations, the authors believe that certain kinds of misconduct follow certain patterns, and that when bad actors are acting badly, they tend to undertake the same kinds of actions, or are experiencing similar circumstances.  For example, in our experience the primary factors that pertain to a person committing fraud are personal relationship problems, financial difficulties, drug or alcohol problems, gambling, a feeling of under appreciation at work, and unreasonable pressure to achieve a work outcome without a legitimate way to accomplish it (and so they attempt illegitimate ways to do so).  These factors are often detectable in the electronic information the subject creates.  Similarly, a person who is harassing or discriminating against others also tends to undertake specific actions and use particular language in communications.  All of these indicia of misconduct are detectable using advanced analytics and skillful strategy.

[50]      Lawyers have gotten quite good at finding this information when looking back in time.  We thought, then, that it should not be too difficult to find this information while the misconduct is unfolding, or to identify warning signs that misconduct is likely to occur, and seek to provide relief of certain factors where possible or take corrective action when needed and as early as possible.  So, we put this to the test, developing Early Warning Systems (“EWS”) for some of our clients.

[51]      The idea for an EWS first occurred to one of the authors when working on a pro bono matter with the ACLU in a case against the Baltimore Police Department (“BPD”) alleging unconstitutional arrest practices in its Zero Tolerance Policing policies.[76]  As a result of the case, the BPD agreed to, among other things, implement a tracking system whereby certain data points were collected regarding police officer conduct and arrest practices that research had proven were warning signs of potential problem officers.[77]  The accumulation of certain data points with respect to an officer triggered a review of the officer’s conduct, with various remediation outcomes.[78]  We thought that a similar approach could be used for our clients.

[52]      An EWS is a tricky thing to implement, and requires careful consideration of many factors, employee privacy at the forefront.  However, with careful planning, policy development, and training, an effective EWS can be designed and implemented.  Predictive analytics applications can be trained to search for indicia of the conduct, language, or factors across information systems.  The specific systems to be targeted will vary depending on what is being sought and the systems most likely to contain it and will vary greatly from company to company.  But, when properly trained and targeted, we have found these systems to be very effective in detecting and even preventing misconduct.  We believe that this use of predictive analytics will become one of the most powerful applications of this technology in the near future.

[53]      Moving from the business intelligence aspects of information governance to the arguably more prosaic field of records and information management, the authors also count themselves as true believers in the power of analytics to optimize traditional RIM (records and information management) functionality.  A full discussion of archival and records management practices in the digital age is beyond the scope of this Article, but the interested reader will find a wealth of scholarly literature in the leading journals discussing how the traditional practice of records management is being transformed in the digital age. One of the authors has argued that predictive coding and like methods are the most promising way to open up “dark archives” in the public sector, such as digital collections of data appraised as permanent records (mostly consisting of White House email at this point), that for reasons of privacy or privilege will be otherwise inaccessible to the public for many decades to come.[79]

[54]      In the authors’ experience, email archiving using auto-categorization for recordkeeping purposes is available using existing software in the marketplace.  In such instances, email is populated in specific “buckets” in a repository depending on how it is characterized, based on either the position of the creator or recipient of the email, the subject matter, or based on some other attribute appearing as metadata.[80]  In the most advanced versions of auto-categorization software, the system “learns” as it is trained using exemplars in a seed set selected by subject matter experts (i.e., records managers or expert end users), via a protocol highly reminiscent of the methods adopted by the parties in Da Silva Moore and similar cases.  It is only a matter of time before predictive analytics is more widely used to optimize auto-classification while reducing the burden on end users to perform manual records management functions.[81]

[55]      In similar fashion, the power of predictive analytics to reliably classify content after adequate training makes such tools optimal for data remediation efforts.  The problem of legacy data in corporations is well known, and only growing over time with the inflationary expansion of the ESI universe.[82]  Using advanced analytics to classify low value data, the chaos that is the reality of most shared drives and other joint data repositories, may potentially be reduced by
orders of magnitude.  The challenge of engaging in defensible deletion is one important aspect of optimizing information governance.[83]

 

Conclusion

 [56]      As was made clear at the outset, it is the authors’ intent merely to scratch the surface of what is possible in the analytics space as applied to matters of importance for corporate information governance.  No one has a one hundred percent reliable crystal ball, but it seems evident that as computing power increases, those forms of artificial intelligence that we have referred to here as analytics will themselves only grow in importance in both our daily and professional lives.  By the end of this decade, we would be surprised if the following do not occur: pervasive use of business intelligence software; the use of more automated decision-making (also known as “operational business intelligence”); the use of alerts in the form of early warning systems including the type described above; much greater use of text mining and predictive technologies across a variety of domains.[84]

[57]      All of these developments dovetail with the expected demand on the part of corporate clients for lawyers to be familiar with state of the art practices in the information governance space, as already anticipated by the type of technology that Da Silva Moore and related cases suggest.  As best said in The Sedona Commentary on Achieving Quality in E-Discovery, “[i]n the end, cost-conscious firms, organizations, and institutions of all types that are intent on best practices . . . will demand that parties undertake new ways of thinking about how to solve e-[D]iscovery problems. . . .” [85]  The same holds true for the greater playing field of information governance.  Lawyers who have embraced analytics will have a leg up on their competition in this brave new space.

 

DownloadPDF

Cite as: Dennis R. Kiker, Defensible Data Deletion: A Practical Approach to Reducing Cost and Managing Risk Associated with Expanding Enterprise Data, 20 Rich. J.L. & Tech. 6 (2014), http://jolt.richmond.edu/v20i2/article6.pdf.

 

Dennis R. Kiker*

 

I.  Introduction

[1]        Modern businesses are hosts to steadily increasing volumes of data, creating significant cost and risk while potentially compromising the current and future performance and stability of the information systems in which the data reside.  To mitigate these costs and risks, many companies are considering initiatives to identify and eliminate information that is not needed for any business or legal purpose (a process referred to herein as “data remediation”).  There are several challenges for any such initiative, the most significant of which may be the fear that information subject to a legal preservation obligation might be destroyed.  Given the volumes of information and the practical limitations of search technology, it is simply impossible to eliminate all risk that such information might be overlooked during the identification or remediation process.  However, the law does not require that corporations eliminate “all risk.”  The law requires that corporations act reasonably and in good faith,[1] and it is entirely possible to design and execute a data remediation program that demonstrates both.   Moreover, executing a reasonable data remediation program yields more than just economic and operational benefits.  Eliminating information that has no legal or business value enables more effective and efficient identification, preservation, and production of information requested in discovery.[2]

 [2]        This Article will review the legal requirements governing data preservation in the litigation context, and will demonstrate that a company can conduct data remediation programs while complying with those legal requirements.  First, we will examine the magnitude of the information management challenge faced by companies today.  Then we will outline the legal principles associated with the preservation and disposition of information.  Finally, with that background, we will propose a framework for an effective data remediation program that demonstrates reasonableness and good faith while achieving the important business objectives of lowering cost and risk.

 

II.  The Problem: More Data Than We Want or Need

 [3]        Companies generate an enormous amount of information in the ordinary course of business.  More than a decade ago, researchers at the University of California at Berkeley School of Information Management and Systems undertook a study to estimate the amount of new information generated each year.[3]  Even ten years ago, the results were nearly beyond comprehension.  The study estimated that the worldwide production of original information as of 2002 was roughly five exabytes of data, and that the storage of new information was growing at a rate of up to 30% per year.[4]  Put in perspective, the same study estimates that five exabytes is approximately equal to all of the words ever spoken by human beings.[5]  Regardless of the precision of the study, there is little question that the volume of information, particularly electronically stored information (“ESI”) is enormous and growing at a frantic pace.  Much of that information is created by and resides in the computer and storage systems of companies.  And the timeworn adage that “storage is cheap” is simply not true when applied to large volumes of information.  Indeed, the cost of storage can be great and come from a number of different sources.

[4]        First, there is the cost of the storage media and infrastructure itself, as well as the personnel required to maintain them.  Analysts estimate the total cost to store one petabyte of information to be almost five million dollars per year.[6]  The significance of these costs is even greater when one realizes that the vast majority of the storage for which companies are currently paying is not being used for any productive purpose.  At least one survey indicates that companies could defensibly dispose of up to 70% of the electronic data currently retained.[7]

[5]        Second, there is a cost associated with keeping information that currently serves no productive business purpose.  The existence of large volumes of valueless information makes it more difficult to find information that is of use.  Numerous analysts and experts have recognized the tremendous challenge of identifying, preserving, and producing relevant information in large, unorganized data stores.[8]  As data stores increase in size, identifying particular records relevant to a specific issue becomes progressively more challenging.  One of the best things a company can do to improve its ability to preserve potentially relevant information, while also conserving corporate resources, is to eliminate information from its data stores that has no business value and is not subject to a current preservation obligation.

[6]        Eliminating information can be extremely challenging, however, due to the potential cost and complexity associated with identifying information that must be preserved to comply with existing legal obligations.  When dealing with large volumes of information, manual, item-by-item review by humans is both impractical and ineffective.  From the practical perspective, large volumes of information simply cannot be reviewed in a timely fashion with reasonable cost.  For example, consider an enterprise system containing 500 million items.  Even assuming a very aggressive review rate of 100 documents per hour, 500 million items would require five million man-hours to review.  At any hourly rate, the cost associated with such a review would be prohibitive.

[7]        Even when leveraging commonly used methods of data culling to reduce the volume required for review, such as deduplication, date culling, and key word filtering, the anticipated volume would still be unwieldy when even a 90% reduction in volume would require review of 50 million items. Moreover, studies have long demonstrated that human reviewers are often quite inconsistent with respect to identifying “relevant” information, even when assisted by key word searches.[9]

[8]        Current scholarship also shows that human reviewers do not consistently apply the concept of relevance and that the overlap, or the measure of the percentage of agreement on the relevancy of a particular document between reviewers, can be extremely low.[10]  Counter-intuitively, the result is the same even when more “senior” review attorneys set the “gold standard” for determining relevance.[11]  Recent studies comparing technology-assisted processes with traditional human review conclude that the former can and will yield better results.  Technology can improve both recall (the percentage of the total number of relevant documents in the general population that are retrieved through search) and precision (percentage of retrieved documents that are, in fact, relevant) than humans can achieve using traditional methods.[12]

[9]        There is also growing judicial acceptance of parties’ use of technology to help reduce the substantial burdens and costs associated with identifying, collecting, and reviewing ESI.  Recently, the U.S. District Court for the Southern District of New York affirmed Magistrate Judge Andrew Peck’s order approving the parties’ agreement to use “predictive coding,” a method of using specialized software to identify potentially relevant information.^[13]

[10]      Likewise, a Loudon County, Virginia Circuit Court judge recently granted a defendant’s motion for protective order allowing the use of predictive coding for document review.[14]  The defendant had a data population of 250 GB of reviewable ESI comprising as many as two million documents, which, it argued, would require 20,000 man-hours to review using traditional human review.[15]  The defendant explained that traditional methods of linear human review likely “misses on average 40% of the relevant documents, and the documents pulled by human reviewers are nearly 70% irrelevant.”[16]

[11]      Similarly, commentary included with recent revisions to Rule 502 of the Federal Rules of Evidence indicate that using computer-assisted tools may demonstrate reasonableness in the context of privilege review: “Depending on the circumstances, a party that uses advanced analytical software applications and linguistic tools in screening for privilege may be found to have taken ‘reasonable steps’ to prevent inadvertent disclosure.”[17]

[12]      Simply put, dealing with the volume of information in most business information systems is beyond what would be humanly possible without leveraging technology.  Because such systems contain hundreds of millions of records, companies effectively have three choices for searching for data subject to a preservation obligation: they can rely on the search capabilities of the application or native operating system, they can invest in and employ third-party technology to index and search the data in its native environment, or they can export all of the data to a third-party application for processing and analysis.

 

III.  The Solution: Defensible Data Remediation

[13]      Simply adding storage and retaining the ever-increasing volume of information is not a tenable option for businesses given the cost and risk involved.  However, there are risks associated with data disposition as well, specifically that information necessary to the business or required for legal or regulatory reasons will be destroyed.  Thus, the first stage of a defensible data remediation program requires an understanding of the business and legal retention requirements applicable to the data in question.  Once these are understood, it is possible to construct a remediation framework appropriate to the repository that reflects those requirements.

 A.  Retention and Preservation Requirements

 [14]      The U.S. Supreme Court has recognized that “‘[d]ocument retention policies,’ which are created in part to keep certain information from getting into the hands of others, including the Government, are common in business.”[18]  The Court noted that compliance with a valid document retention policy is not wrongful under ordinary circumstances.[19]  Document retention policies are intended to facilitate retention of information that companies need for ongoing or historical business purposes, or as mandated by some regulatory or similar legal requirement.  Before attempting remediation of a data repository, the company must first understand and document the applicable retention and preservation requirements.

[15]      It is beyond the scope of this Article to outline all of the potential business and regulatory retention requirements.[20]  Ideally, these would be reflected in the company’s record retention schedules.  However, even when a company does not have current, up-to-date retention schedules, embarking on a data remediation exercise affords the opportunity to develop or update such schedules in the context of a specific data repository.  Most data repositories contain limited types of data.  For example, an order processing system would not contain engineering documents.  Thus, a company is generally focused on a limited number of retention requirements for any given repository.  There are exceptions to this rule, such as with e-mail systems, shared-use repositories (e.g., Microsoft SharePoint), and shared network drives.  Even then, focusing on the specific repository will enable the company to likewise focus on some limited subset of its overall record retention requirements.  Once a company has identified the business and regulatory retention requirements applicable to a given data repository, information in the repository that is not subject to those requirements is eligible for deletion unless it is subject to the duty to preserve evidence.

[16]      The modern duty to preserve derives from the common law duty to preserve evidence and is not explicitly addressed in the Federal Rules of Civil Procedure.[21]  The duty does not arise until litigation is “reasonably anticipated.”[22]  Litigation is “reasonably anticipated” when a party “knows” or “should have known” that the evidence may be relevant to current or future litigation.[23] Once litigation is reasonably anticipated, a company should establish and follow a reasonable preservation plan.[24]  Although there are no specific court-sanctioned processes for complying with the preservation duty, courts generally measure the parties’ conduct in a given case against the standards of reasonableness and good faith.[25]  In this context, a “defined policy and memorialized evidence of compliance should provide strong support if the organization is called up on to prove the reasonableness of the decision-making process.”[26]

[17]      The duty to preserve is not without limits: “[e]lectronic discovery burdens should be proportional to the amount in controversy and the nature of the case” so the high cost of electronic discovery does not “overwhelm the ability to resolve disputes fairly in litigation.”[27] Moreover, courts do not equate reasonableness with “perfection.”[28] Nor does the law require a party to take “extraordinary” measures to preserve “every e-mail” even if it is technically feasible to do so.[29]  “Rather, in accordance with existing records and information management principles, it is more rational to establish a procedure by which selected items of value can be identified and maintained as necessary to meet the organization’s legal and business needs[.]”[30]

[18]      Critical tasks in a preservation plan are the identification and documentation of key custodians and other sources of potentially relevant information.[31] Custodians identified as having potentially relevant information should generally receive a written litigation hold notice.[32]  The notice should be sent by someone occupying a position of authority within the organization to increase the likelihood of compliance.[33] The Sedona Guidelines also suggest that a hold notice is most effective when it:

 1)      Identifies the persons likely to have relevant information and communicates a preservation notice to those persons;

2)      Communicates the preservation notice in a manner that ensures the recipients will receive actual, comprehensible and effective notice of the requirement to preserve information;

3)      Is in written form;

4)      Clearly defines what information is to be preserved and how the preservation is to be undertaken; and

5)      Is regularly reviewed and reissued in either its original form or an amended form when necessary.[34]

 [19]      The legal hold should also include a mechanism for confirming that recipients received and understood the notice, for following up with custodians who do not acknowledge receipt, and for escalating the issue until it is resolved.[35]  To be effective, the legal hold should be periodically reissued to remind custodians of their obligation and to apprise them of changes required by the facts and circumstances in the litigation.[36]

[20]      Experience has also shown that legal holds that are not properly managed and ultimately released are less likely to receive the appropriate level of attention by employees. Thus, the legal hold process should also include a means for determining when litigation is no longer reasonably anticipated and the hold can be released, while ensuring that information relevant to another active matter is preserved.[37]

 B.  The Remediation Framework

 [21]      Against this backdrop, it is possible to outline a framework for data remediation that is compliant with legal preservation requirements.  The following describes a high-level data remediation process that can be applied to virtually any data environment and any risk tolerance profile.  The general process is described in Figure 1 below:

Figure 1: Data Remediation Framework

1.  Assemble the Team

 [22]      A successful data remediation project depends on invested participation by at least three constituents in the organization: legal, information technology (“IT”), and records and information management (“RIM”).  In addition, the project may require additional support from experts experienced in information search and retrieval and statistical analysis.  In-house and/or outside counsel provides legal oversight and risk assessment for the project team, as well as guidance on legal preservation obligations.  IT provides the technological expertise necessary to understand the structure and capabilities of the target data repository.  RIM professionals provide guidance on business and regulatory retention obligations.  The need for information search and retrieval experts and statisticians depends on the complexity of the data remediation effort as described below.  Finally, including business users of the information may be necessary as required to fully document retention requirements applicable to a particular repository if not adequately documented in the organization’s document retention policy and schedule.

 2.  Select Target Data Repository

 [23]      Selecting the target data repository requires consideration of the costs and benefits of the data remediation exercise.  Each type of repository presents unique opportunities and challenges.  For example, e-mail systems, whether traditional or archived, are notorious for containing vast amounts of information that is not needed for any business or legal purpose.  Similarly, shared network drives tend to contain large volumes of unused and unneeded information.  Backup tapes, legacy systems, and even structured databases are other possible targets.  IT and RIM resources are invaluable in identifying a suitable target repository.  For example, IT can often run reports identifying directories and files that have not been accessed recently.

 3.  Document Retention and Preservation Obligations

[24]      As discussed above, it is critical to understand the retention and preservation obligations that are applicable to the data contained in the target repository.  Retention obligations include the business information needs as well as any regulatory requirements mandating the preservation of data.  Ideally, these are incorporated into the document retention policy and schedule for the organization.  If not, it will be important to document those requirements applicable to the target repository.

[25]      Preservation obligations are driven by existing and reasonably anticipated litigation.[38]  In some cases this may be the most challenging part of the project, particularly for highly litigious companies, because, unlike business needs and regulatory requirements, preservation obligations are constantly changing as new matters arise and circumstances evolve in existing matters.  Successful completion of the remediation project will require a detailed understanding of, and constant attention to, the preservation obligations applicable to the target repository.  As discussed below, some of the risk associated with this aspect of the project can be ameliorated through selection of the appropriate repository and culling criteria.  Nevertheless, the scope and timing of the project will be driven in large part by the preservation obligations applicable to the target repository.

4.  Inventory Target Data Repository

[26]      After selecting the target data repository, the team must inventory the information within that repository.  This does not involve creating an exhaustive list or catalog of every item within the repository.  Rather, inventorying the repository involves developing a good understanding of the types of information that are contained there, the date ranges of the information, and other criteria that will enable identifying information that must be retained and that which can be deleted.  The details of the inventory will vary by data repository.  For example, for an e-mail server, the pertinent criteria may include only date ranges and custodians, whereas for a shared network drive, the pertinent criteria may include departments and individuals with access, date ranges, and file types.

 5.  Gross Culling

 [27]      The next step is to determine the “gross culling” criteria for the data repository.  In this context, “gross culling” refers to an initial phase of data culling based on broad criteria as opposed to fine or detailed culling criteria that may be used in a later phase of the exercise.[39]  The nature of the information contained within the repository will determine the specific criteria to be used, but the objective is to locate the “low-hanging fruit,” the items within the repository that can be readily identified as not falling within any retention or preservation obligation. These are black-and-white decisions where the remediation team can definitively determine without further analysis that the items identified can be deleted.

[28]      For example, in most cases, dates are effective gross culling criteria.  Quite often, large volumes of e-mail and loose files (data retained in shared network drives or other unstructured storage) predate any existing retention or preservation obligation for such items.  Similarly, in repositories that are subject to short or no retention guidelines, the business need for the data can be evaluated in terms of the date last accessed.  In the case of shared network drives, for example, it is not uncommon to find large volumes of information that has not been accessed by any user in many years.[40]  Such information can be disposed of with very little risk.

 6.  Fine Culling

 [29]      Sometimes, the process need go no further than the gross culling stage.  Depending on the volume of data deleted and the volume and nature of the data remaining, the remediation team may determine that the cost and benefit of attempting further culling of the data are not worth the effort and risk.  In some cases, however, gross culling techniques will not identify sufficient volumes of unneeded data and more sophisticated culling strategies must be employed.

[30]      The precise culling technique and strategy will depend on the specific data repository, its native search capabilities, and the availability of other search tools.  For example, many modern e-mail archiving systems have fairly sophisticated native search capabilities that can locate with a high degree of accuracy content pertinent to selected criteria.  Other systems will require the use of third-party technology.  In either case, the fine culling process will require selection of culling criteria that will uniquely identify items not subject to a retention or preservation obligation and be susceptible to verification.  Depending on the nature of the data and the complexity of the necessary search criteria, the remediation team may need to engage an expert in information search and retrieval.

 7.  Sampling and Statistical Analysis

[31]      Regardless of the specific fine culling strategy employed, the remediation team should validate the results by sampling and analysis to ensure defensibility.  Generally, it will be advisable to engage a statistician to direct the sampling effort and perform the analysis because both can be quite complex and rife with opportunity for error.[41]  Moreover, in the event that the company’s process is ever challenged, validation by an independent expert is compelling evidence of good faith.  It is important to realize that the statistical analysis cannot demonstrate that no items subject to a preservation obligation are included in the data to be destroyed.  It can only identify the probability that this is the case, but it can do so with remarkable precision when properly performed.[42]

 8.  Iteration

[32]      Fine culling and validation should continue until the remediation team achieves results that meet its expectations regarding the volume of data identified for deletion and the probability that only data not subject to a preservation obligation are included in the result set.

 

 IV.  Conclusion

[33]      The enormity of the challenge that expanding volumes of unneeded information creates for businesses is difficult to understate.  Companies literally spend millions of dollars annually to store and maintain information that serves no useful purpose, funds that could be directed to productive uses such as hiring, research, and investment.  Facing this challenge, on the other hand, is a challenge of its own, perhaps due more to the fear of adverse consequences in litigation than any other factor.  It is possible, however, to develop a defensible data remediation process that enables a company to demonstrate good faith and reasonableness while eliminating the cost, waste, and risk of this unnecessary data.