All posts tagged Automated Workflow

Computer-Assisted Coding: Notes from U.S. Magistrate Judge Andrew Peck

Computer-Assisted Coding: Notes from U.S. Magistrate Judge Andrew Peck

On October 1, 2011, Law Technology News published an article from United States Magistrate Judge Andrew Peck titled, “Search, Forward: Time for Computer-Assisted Coding.” In this article, Judge Peck explored the use of computer-assisted coding within document review while discussing the perceived “judicial endorsement” of keyword searching. A common theme throughout his article – which echoes comments made during the Carmel Valley eDiscovery Retreat – is questioning why lawyers seem insistent on receiving a judicial blessing of this technology before using it. Indeed, this was a prevalent question and theme throughout the “What is it and how is it being leveraged?” session at the national Masters Conference that took place in Washington D.C. this week. Some attendance members noted that they were uncomfortable using this technology without agreement from opposing counsel, and may not even use it at all until there was a case certifying its use.

To address the issue of judicial endorsement, Judge Peck provided a thorough analysis regarding the problems inherent in keyword searching, citing several of the well-known opinions on this issue including United States v. O’Keefe, Equity Analytics, LLC v. Lundin, Victor Stanley, Inc. v. Creative Pipe, Inc. and his own opinion, William A. Gross Construction Associates, Inc. v. American Manufacturers Mutual Insurance Co. Judge Peck also cited the Blair and Maron study conducted in 1985, in which a database of 40,000 documents was searched by lawyers. The lawyers believed their manual search retrieved 75% of relevant documents, when only 20% were retrieved.[1]

With this analysis aside, Judge Peck then turned to the newest hot button issue – computer-assisted document review, which he defined as “tools… that use sophisticated algorithms to enable the computer to determine relevance, based on interaction with (i.e., training by) a human reviewer.” After discussing how computer-assisted review tools work, Judge Peck noted that there is no federal or state reported case to his knowledge that has ruled on the use of this technology, further stating that it “will be a long wait” for lawyers waiting for a court to conclude: “It is the opinion of this court that the use of predictive coding is a proper and acceptable means of conducting searches under the Federal Rules of Civil Procedure, and furthermore that the software provided for this purpose by [insert name of your favorite vendor] is the software of choice in this court.”

In addition, Judge Peck noted that if the use of computer-assisted review technology was presented or challenged in a case before him, he “want to know what was done and why that produced defensible results,” perhaps being less interested in the “science behind the ‘black box’…than whether it produced responsive documents with reasonably high recall and high precision.” Further, proof of quality control would be important to defending use of the technology.

Judge Peck concluded his article with: “Until there is a judicial opinion approving (or even critiquing) the use of predictive coding, counsel will just have to rely on this article as a sign of judicial approval. In my opinion, computer-assisted coding should be used in those cases where it will help ‘secure the just, speedy, and inexpensive’ (Fed. R. Civ. P. 1) determination of cases in our ediscovery world.”

This blog post is intended to merely highlight the main themes throughout this informative article. We certainly recommend it as a “must read” for those practitioners debating whether the use of computer-assisted technology is right for them. To provide further proof, here is a recent case study demonstrating the power behind the Intelligent Review Technology (IRT) offered by Kroll Ontrack in its award-winning document review platform, Ontrack® Inview™.

Embroiled in a complex patent litigation, a national law firm representing a major health care provider trusted Kroll Ontrack to ensure the document review process was efficiently conducted to meet case critical deadlines. Already faced with 334 gigabytes of data, totaling over 750,000 documents from the outset of the case, the law firm was surprised to discover an additional 325,000 documents mid-way through the review period. Moving the deadline for review was not an option, but reviewing the new batch of documents using traditional linear review methods would have necessitated a significant break from the budget and no guarantees that the deadline could be met.

Following initial filtering work and use of Ontrack® Advanceview™, the leading early data assessment solution, counsel utilized all three aspects of IRT: Automated Workflow, Intelligent Prioritization and Intelligent Categorization. Intelligent Prioritization (iP) ran from the project outset in the background, elevating documents that that were more likely to be responsive. iP significantly aided the review process, allowing the team to review and produce 50,000 documents after only two weeks.

Once the approximately 325,000 new documents were discovered and added to the review queue, counsel needed a way to review the documents while still meeting the strict deadlines, but did not have the budget available to hire additional contract attorneys. However, by leveraging Intelligent Categorization (iC) counsel met the challenge in stride. After intense sampling and analysis of reporting, counsel instructed Kroll Ontrack Document Review Service Professionals to remove documents determined to have a 90 percent confidence rating of non-responsiveness by the iC technology – instantly eliminating nearly half of the documents from the new data set. Any technology is only valuable if it is defensible, so when a rigid quality control process revealed a staggering 94 percent agreement rate between a human review “dream team” and the iC determinations across the data set, counsel was able to confidently report to the senior partners that the deadline would be met on time, within budget and without risk.

The end result? Counsel confidently removed more than 125,000 documents using iC alone, saving almost $200,000 in review costs – with $65,676 attributable to iC.

[1] David C. Blair & M.E. Maron, An Evaluation of Retrieval Effectiveness for a Full-Text Document-Retrieval System, Communications of the ACM, Vol. 28, Issue 3 (March 1985).

Two Truths and a Fib About Intelligent Categorization

Two Truths and a Fib About Intelligent Categorization

Time is money, and linear document review is almost prohibitively expensive due to the surge in electronic data volume over the past several years and the corresponding increase in resources required to review the data. Besides time and costs, having a multitude of attorneys reviewing and categorizing documents for (potentially) months on end can yield inconsistent results. Innovative technological advances have arrived on the document review scene, but concerns about overall effectiveness persist as the legal industry remains hesitant to explore new technology.

Enter Intelligent Categorization (iC). Intelligent Categorization is the third component of Intelligent Review Technology (IRT) that analyzes and learns from category decisions made by human reviewers, then identifies and elevates documents most likely to be relevant and suggests categories for documents not yet reviewed. Along with Automated Workflow and Intelligent Prioritization, the other two legs of IRT, reliance on Intelligent Categorization technology is on its way to becoming a well-established practice in 2011. Differing ideas and opinions associated with this technology have been tossed around, giving rise to certain ideas and misconceptions about what iC is, what iC is not and what iC can do for electronic discovery. Today we will dissipate the confusion and set the record straight by exploring two important truths and a common fib associated with Intelligent Categorization.

Defensible? True.

First and foremost, Intelligent Categorization is defensible. One of the early qualms about iC was that until the technology became court-tested, it was too risky to use. That simply is not the case. In fact, such fears have preceded the acceptance of all new technology, including features such as advanced searching and sampling, which are now embraced by jurists and litigants alike.[1] Case law supports the use of a systematic, well-documented and repeatable process, and Intelligent Categorization is specifically designed to increase accuracy and effectiveness while decreasing review time. Indeed, when using all three components of Intelligent Review Technology, it is possible to save 50 percent on review costs.

Intelligent Categorization also supports the notions of proportionality set forth in Rule 1 of the Federal Rules of Civil Procedure, with the goal of proceedings to be “just, speedy and inexpensive.”[2] As an integrated component of IRT, iC is fully transparent with real-time metrics and analytics available throughout the review process. In addition, experts can explain the technology to judges, opponents, clients and staff if necessary.

Further, the Sedona Conference® has endorsed the use of automated methods (although has not endorsed particular technologies to do so). The Sedona Conference Best Practices Commentary on the Use of Search and Information Retrieval Methods in Ediscovery, Practice Point 1 states:

[R]eliance solely on a manual search process for the purpose of finding documents may be infeasible or unwarranted. In such cases, the use of automated search methods should be viewed as reasonable, valuable, and even necessary.[3]

In addition, The Sedona Conference Commentary on Achieving Quality in the EDiscovery Process advises practitioners to utilize technology that “reasonably and appropriately enable a party to safely and substantially reduce the amount of ESI that must be reviewed by humans.”[4] These commentaries stress the use of technology to realize the important goal of achieving proportionality in the electronic discovery process that has unfortunately spiraled out of control in recent years.

Effective? True.

Closely linked to defensibility is effectiveness. Before investing in new technology, legal teams must be confident that the new feature will work and is worth the change. With sufficient training data, supervised learning can target documents most likely to be relevant. Using supervised learning to identify and pull responsive documents into categories early reduces the time spent organizing documents responsive to particular requests, and helps reviewers and the legal team better understand the case early on. Also, related documents can be dealt with more efficiently as a group and can even be assigned to a reviewer with expertise in a particular category.

The effectiveness of this technology may also be tested through sampling. Sampling is the key to measuring, monitoring, controlling and correcting potential errors in categorization, and is useful in any review to validate results. The technology can systematically and iteratively test the data to evaluate the accuracy of iC (in addition to Intelligent Prioritization). Without the use of sampling, some courts have concluded a party did not take reasonable steps to prevent disclosure.[5] With the flexibility to conduct as much or as little sampling as desired, iC not only reduces the time needed to complete a review, it improves the consistency of and confidence in category determinations.[6]

Independent studies are also proving that the use of Intelligent Review Technology (including Intelligent Categorization) is more effective than traditional, manual review processes. The Ediscovery Institute released a survey that showed using the technology equivalent of Intelligent Categorization resulted in reduced review costs by 45 percent or more.[7] In addition, the TREC Legal Track study from 2008 demonstrated that a “Boolean keyword search found only 24% of the total number of responsive documents in the target data set” while automated searching methods found 76 percent of the responsive documents.[8]

Devoid of Human Control? False.

Intelligent Categorization is not a process devoid of critical human insight and control. In some instances, this new technology has been pitched as a purely hands-off, eyes-off solution. In reality, Intelligent Review Technology as a whole does not replace human reviewers, nor should it. iC works by “learning” from human decisions and applying human logic when suggesting document categories. Human input is required so the technology has data sets with applied classifications from which to learn, and the system learns from both responsive and non-responsive decisions of human reviewers. As more documents are received and sorted, legal teams can rely on technology to continually improve the model while human reviewers can focus their efforts on the content and substance of the documents. In addition, because the tool was designed to increase consistency and accuracy, it affords the flexibility and scalability to give the ediscovery team more control over the review and to leverage as much or as little human input and oversight as is appropriate for the project. Thus, iC is not a substitute for skilled lawyers; rather, it enhances and compliments the work they do.

The question of whether it is reasonable to omit review of some documents altogether is an as-yet undetermined legal question. From a technical standpoint, however, IRT systems can support a range of approaches to selective review, such as extracting documents with a sufficiently low probability of responsiveness from review, guiding a review to read just the most important portions of long documents or focusing extra review on documents likely to belong to sensitive categories.

In short, Intelligent Categorization is a defensible, effective, cost-saving measure that leverages the work of talented attorneys to decrease the time required to complete document review. It is designed to meet flexibility and repeatability needs of the client, and is proving to be the key differentiator in the ability to respond to electronic discovery demands quickly and proportionately.

Note: The above post appeared in the April 2011 issue of the free, monthly e-newsletter, Case Law Update & Trends published by Kroll Ontrack. This newsletter is designed to help busy legal professionals keep pace with case law and information pertaining to electronic evidence. Subscribe and gain valuable and timely information on new ESI court decisions, as well as informative articles and tips for both the corporate and law firm audience.

[1] See, e.g., William A. Gross Constr. Assocs., Inc. v. Am. Mfrs. Mut. Ins. Co., 2009 WL 724954 (S.D.N.Y. Mar. 19, 2009).

[3] The Sedona Conference® Best Practices Commentary on the Use of Search and Information Retrieval Methods in Ediscovery. (Published August 2007).

[4] The Sedona Conference® Commentary on Achieving Quality in the Ediscovery Process, available for download at (Published May 2009).

[5] Mt. Hawley Ins. Co. v. Felman Prods. Inc., 2010 WL 1990555 (S.D.W.Va. May 18, 2010). 

[6] The Sedona Conference® Commentary on Achieving Quality in the Ediscovery Process, Principle 2, states: “In the ediscovery context, statistical sampling can serve as a check on the effectiveness of…automated tools in identifying responsive information and on the reviewers’ ability to correctly code documents.”

[7] See “Ediscovery Institute Survey on Predictive Coding,” available at

[8] For complete results from TREC Legal Track, visit

Case Law: In re Fontainebleau Las Vegas Contract Litigation

Case Law

Document Dump of Servers Leads to Privilege Waiver

In re Fontainebleau Las Vegas Contract Litig., 2011 WL 65760 (S.D. Fla. Jan. 7, 2011). In this bankruptcy litigation, the requesting party claimed the third party waived privilege by producing three servers in response to a subpoena and court orders without conducting a review for either privilege or responsiveness. Seeking to use the information but avoid any adverse consequences, the requesting party offered to “eat” the cost of searching the massive document dump of approximately 800 GB and 600,000 documents for relevant materials in exchange for the right to review and use the data free of the obligation to appraise or return any privileged documents. Reviewing the third party’s conduct, the court found that its failure to conduct any meaningful privilege review prior to production constituted voluntary disclosure and resulted in a complete waiver of applicable privileges. Noting that more than two months after production the third party had not flagged even one document as privileged, the court rejected its “belatedly and casually proffered” objections as “too little, too late.” Accordingly, the court granted the requesting party full use of these documents during pre-trial preparations of the case, but ordered it to timely advise the third party of any facially privileged information it encountered upon review.


This case demonstrates the importance of conducting a proper review and production process. The third party in this case simply “dumped” a significant amount of information onto the requesting party without engaging in a document review; instead, the third party complained that the review and production process would be unduly burdensome and delayed production numerous times. Finally, the third party produced three servers without conducting a privilege review, but did belatedly produce a privilege log for one of the servers.

If the third party had conducted a document review, it would not have been in the position to spend costs on numerous motions and would not have faced the penalty of the court declaring that privilege was waived (with the exception of the server that had an accompanying privilege log). The document review process may seem like a daunting, costly and burdensome task, but it is often a necessary step in ediscovery. Thankfully, Intelligent Review Technology exists and can save you 50 percent on review costs, while improving the quality and defensibility of document review. Through use of Automated Workflow, Intelligent Prioritization and Intelligent Categorization, counsel can avoid conducting the strictly manual process of document review that inherently leads to inaccuracies and inconsistencies. IRT also provides transparent reports and real-time metrics, giving counsel peace of mind as to the defensibility of the technology and process. If the third party in this case invested in technology to conduct the document review, their costs would arguably have been far less than what was spent on numerous motions and the cost of opposing counsel’s access to and use of privileged materials in the pre-trial preparation process.

Intelligent Review Technology Results in Proven Cost-Savings

The ediscovery landscape is proving to be more treacherous given the rise of data proliferation and the increase in sanctions for mismanaging ESI. As the courts grow increasingly intolerant of discovery failures, litigants are faced with two choices: work harder by investing more resources to ensure thorough review, or work smarter by leveraging fewer resources with cutting-edge technology to achieve superior results. Dynamic companies know the latter is always the best option, and for them, the next generation of ediscovery technology has arrived.

Intelligent Review Technology (IRT) combines the best of both worlds by delivering the discerning analytics of a human review team in an automated platform capable of increased processing speed, consistency and accuracy. Although IRT encompasses many different technologies that can work independently or conjunctively to varying degrees, workflow automation, supervised learning and statistical quality control are the cornerstone features of an effective IRT system that together allow review to be conducted faster, more efficiently and accurately than even the best human review teams equipped with current discovery technology.

IRT learns while you work, and empowers the defensibility of your arguments with transparent reports and real-time metrics. By analyzing decisions made by lawyers, the system applies human logic to identify likely responsive documents and make categorization suggestions. IRT integrates human input with smart technology, reduces costs by 50%, and improves the quality and defensibility of document review.

The technology inside, Ontrack® Inview™, has three components:

  • Workflow: Get the right documents to the right people. Workflow is an automated means to distribute and check in documents.
  • Prioritization: See the most important documents first. Prioritization evaluates reviewer decisions to identify and elevate documents that are most likely responsive, enabling reviewers to view the most relevant documents first.
  • Categorization: Harness the power of technology to learn from human decisions.  Categorization analyzes reviewer decisions and applies logic to suggest categories for the documents not yet reviewed

Kroll Ontrack Launches Final Component of Intelligent Review Technology

Innovative Categorization Features, along with Kroll Ontrack Professional Services Experts, Drive Document Review Efficiencies and Decrease Costs

Kroll Ontrack, the leading provider of information management, data recovery and legal technologies products and services, today announced the launch of Ontrack® Inview™ version 6.5, the award-winning document review tool, which now includes Categorization functionality. Completing the Intelligent Review Technology (IRT) suite within the Ontrack Inview 6.5 tool,Workflow, Prioritization and Categorization integrate expert human logic with smart technology by evaluating and learning from document review decisions made by lawyers. Committed to building innovative discovery technology, Kroll Ontrack IRT expedites document review schedules, improves the quality of review decisions and reduces overall discovery costs, all while empowering attorneys to focus on case strategy.

The newest Ontrack Inview feature, Categorization, builds upon the Workflow and Prioritization features, which were released in June and August 2010, respectively. By analyzing human decisions and then making categorization recommendations for documents not yet reviewed, Categorization addresses a common challenge posed by traditional document review – category determinations are often made inconsistently and inefficiently. In fact, studies show that a second, repeat review, even by the same staff, often leads to different category decisions. With  Categorization technology in the Ontrack Inview review tool, learning immediately begins when category decisions are made by lawyers designated as “trainers.” The system learns with each keystroke and then applies intelligent categorizations to the rest of the review set, which can then be evaluated by the review team and validated using quality control measures.

With Workflow, Prioritization and Categorization, clients can achieve greater than 50 percent costs savings on review. Kroll Ontrack IRT works in four steps:

  1. Train: Lawyers designated as “trainers” review documents and determine whether they are responsive, non-responsive, privileged or some other pre-defined category.
  2. Learn: IRT analyzes reviewer category decisions made by trainers to identify and elevate documents that are most likely relevant and suggest categories for documents not yet reviewed.
  3. Evaluate: Reviewers make categorization decisions, leveraging intelligent suggestions.
  4. Validate: IRT is fully transparent with real-time reports and metrics available to optimize the technology and experience cost savings.

“Traditional linear review is no longer tenable. The manual aspect of document review inherently produces inaccuracies and inconsistencies in categorization decisions, which hinders the implementation of a repeatable, defensible process,” said Michele Lange, director of discovery, Kroll Ontrack. “IRT in the Ontrack Inview tool is unique when compared to other tools because it automatically selects and presents sample documents for your review team versus a review team conducting manual, iterative, time-intensive searches to find key documents from which to learn. Furthermore, Kroll Ontrack IRT learns while you work and empowers the defensibility of your arguments with transparent reports and real-time metrics. Kroll Ontrack is truly leading a revolution that is going to change the face of legal discovery.”

In addition, through its Professional Services team, Kroll Ontrack is prepared to help clients leverage this powerful technology when conducting their own reviews in order to achieve consistency, speed and cost savings. Clients also have the option of using Kroll Ontrack Document Review Services, facilities and team of highly qualified document review attorneys to fully leverage the Ontrack Inview tool and its advanced capabilities.

Automated Workflow – Providing Structure to Document Review

The traditional practice of document review has a lot of moving parts. It often involves many teams of people, working under tight timeframes, searching for relevant information and striving to produce in a timely manner. As discovery practices have evolved, legal technology service providers have responded with new technologies designed to reduce data volumes, enhance search capabilities, and improve analytics and reporting. However, little time has been spent improving the process of document review.

Automated workflow has changed the document review landscape, improving this often cumbersome process. After all, administration and execution of the review process should run like clockwork. Even when all of the essentials for a document review are in place – reviewers, facilities, training and technology (including up-to-date advanced searching techniques) – the process requires significant planning, oversight and the ability to smoothly absorb hiccups along the way. Automated workflow functionality, which enables document review administrators to design, manage and monitor the document review process, implements defensible quality control mechanisms and allow reviewers to efficiently check in completed work and request more documents.

Workflow Design

Using automated workflow technology, document review administrators can create workflow components to define an automated document review process and associated rules. Rather than sketching the process on loose-leaf pages, giant whiteboard charts or even inside someone’s head, administrators can visually design the workflow directly within the review tool using an intuitive interface to view, create, copy or delete components in an on-screen display. Next, review managers can assign document sets to specific reviewer groups, define review criteria and determine how documents will flow through the review stream. To ease the setup process, each component of the workflow corresponds to a particular question to help define the state, reviewer group, document check-in criteria, document routing and routing automation. Additionally, if a review must start with limited review criteria, more complex, defined criteria can be added later. The resulting process is defensible, repeatable and efficient.

Automated Distribution

Document review administrators can also eliminate the process of manually distributing document batches to reviewers. Automated workflow features allow administrators to define and electronically distribute document sets to designated reviewer groups. During the review process, reviewers can mark documents as incomplete, flag them for a manager or check them in as complete. Documents that meet check-in criteria then automatically proceed to the next applicable stage based on the defined review stream. However, documents will fail check-in if the reviewer attempts to designate conflicting categorizations. At that point, the reviewer is notified to correct the categorization decision. This predefined process increases efficiency, productivity and accuracy. It also eliminates time otherwise wasted in manual distribution and collection and frees the administrator for other important tasks, such as quality control (QC). The result is a faster and more cost-effective review process.

Accurate Monitoring

In a manual workflow, administrators must monitor reviewers closely to minimize lag time and ensure reviewers are supplied with ongoing assignments. Status reports can be run only at the end of a workday, forcing administrators to stay late in order to track the progress of the review. Automating the process alleviates the strains associated with the often tedious spreadsheet creation. Using an intuitive interface, administrators can electronically access and manage the review, monitor progress and address problem documents as they arise. Real-time graphical status reports display high-level or detailed views of the review progress based on stages, reviewer groups or individual reviewers. Additionally, enhanced reporting capabilities enable the administrators to self-schedule project summary reports and choose to have them distributed over e-mail. With these features, the administrators can easily update team members and clients, and can more accurately project the completion date and related staffing needs.

Quality Control & Defensibility

Improved quality control is another benefit of automated workflow. Rather than running hundreds of QC searches at the end of a review, administrators can utilize automated quality control capabilities to validate the consistency of review decisions, ensure accurate categorization of documents and verify that a logical end result is reached. When a misclassification is detected, an administrator can immediately notify the reviewer to correct the issue, thereby saving time normally reserved for extensive QC in the final stages of a review.

Managing any size of a document review project can seem unwieldy, complicated and nearly endless. Taking advantage of new automated workflow capabilities to manage and execute the document review process will ease the strain on human resources and will help clients drive down costs and advance to the next stages of the litigation.

Early Case Assessment and Automated Document Review Workflow Technology Results in Increased Efficiency and Cost Savings

Through its cutting-edge technologies and noted expertise, Kroll Ontrack provides effective solutions that decrease the burden of analysis and review while saving money and time.

The financial crisis that gripped the nation two years ago has spawned litigation relating to poor investments, bad banking and a myriad of consumer disputes. A leading national bank enlisted the services of Kroll Ontrack to design an efficient solution to manage significant amounts of data involved in a series of litigations. The client turned to the proven services and technologies offered by Kroll Ontrack to reduce the volume of data for process and review, strengthen defensibility and reduce costs and risk.

Read more