All posts tagged technology assisted review

Document Review: MythBusters Edition

A couple years ago, I wrote a blog entitled, “Bust These 4 Myths on Your Next Document Review.” In this blog, I looked at four common document review myths and the realities behind the fallacies. Fast forward two years, there is no better time to revisit these myths to take the pulse of document review in 2017, considering the evolution of ediscovery technology, processes, rules and case law.

Myth #1: Document review just happens; you don’t really need a plan.

Since the adoption of the FRCP amendments we have seen courts admonish parties for:

  • Discovering new documents not in the original collection
  • Missing documents that should have been produced the first time around
  • Amassing costs for inefficient discovery methods

With document review technology at the top of its game, the misconception that document review is trivial is fading. In meeting with corporations and law firms, I hear legal teams appreciating the importance of having a review methodology. More often than not, those teams are inquiring as to how their processes can be improved.

2017 Document Review Lesson #1: Don’t procrastinate or wander aimlessly when it comes to review. Know your path from collection through production and be able to justify your methods.

Myth #2: Any attorney can conduct (or manage) a document review.

Today, document review is not the unglamorous chore of former times. With advancements in the review tools, increasingly senior attorneys are finding themselves immersed in document review more than in years past. The tools are easier to operate than ever before, and senior attorneys – typically subject matter experts on the case – are in the best position to review the most pertinent documents, especially if predictive coding is used.

At the same time, in order to fully leverage analytics and predictive coding features, the attorney will need advanced training or someone skilled in using these powerful features to guide them through. With formable technology at their fingertips and millions of documents to wrangle, today’s document reviewers are not only licensed and highly qualified attorneys; but also may have specific training and certifications in various document review platforms. Many have expertise in a different language other than English, or substantive knowledge in their practice area.

2017 Document Review Lesson #2: The days of brute force attorney review are over. Today’s document review requires subject matter experts in the case, working side-by-side with technology-minded attorneys that know how to maximize time and minimize costs.

Myth #3: All document review technology is equal.

What appears equal at face value, is not equal in action. While most major review tools function generally in a similar manner, there are enhancements unique to a particular provider and its tool set. From running searches and batching documents to using predictive coding or reviewing audio files, experienced document reviewers will recognize the fine distinctions of each provider’s platform, knowing when and how those features can be helpful. If they cannot answer a question, the reviewer should know how to get a hold of the technology provider’s technical support team to lend a hand.

2017 Document Review Lesson #3: Get into the technology weeds. Understanding the nuances of a provider’s technology is the only way to reap the benefits of a modern document review.

Myth #4: It will be obvious when you can stop review…when you run out of documents.

Predictive coding has changed how legal teams approach document review; however, even in 2017, the adoption of this is technology is marginal at best. Outmoded teams are still conducting linear reviews of every document, while progressive teams have figured out how to embrace predictive coding so that only the most vital documents are being reviewed for production. But, this does require a team that knows how to sample and interpret the metrics and reports generated by the technology.

2017 Document Review Lesson #4: The terminology related to predictive coding can cause one’s head to spin. Dust off your math skills (or leverage a specialist) – it’s the only way a savvy document review professional knows when a review is finished.

Leverage KrolLDiscovery for Document Review

Looking to modernize your document review methods?

KrolLDiscovery offers advanced document review services around the world, with fully managed review teams and up-to-date facilities in eight locations in four countries: Washington D.C., Chicago, Pittsburgh, Miami, Minnesota, London, Poland and Germany. KrolLDiscovery’s managed document review services teams provide you with specialized document review attorneys to meet your case needs. KrolLDiscovery review platforms are integrated with top-of-the line technology-assisted review and predictive coding features to search, categorize, redact and annotate documents. Our review teams utilize this technology to maximize efficiency through intelligent document prioritization and categorization, automated workflow, advanced search functionality and multilingual support.

Making Cents Out of Ediscovery Costs [Webinar]

Ediscovery does not have to be expensive. By utilizing the newest technologies, engaging in skillful preparation and obtaining knowledge of the process, you can keep ediscovery costs down. Kroll Ontrack’s latest webinar, Cost-Effective Ediscovery: How to Manage Expense and Reduce Waste provides useful tips and suggestions from three seasoned ediscovery specialists. These experts provide law firm, corporate and provider’s perspectives:

Don’t Be Late to Adopt New Technology

The myth that human review is the gold standard, as well as apprehension on how a court regards technology has led some practitioners to shy away from taking advantage of technological opportunities. However, as the webinar panelists discuss, predictive coding is an untapped resource for practitioners aiming to cut ediscovery costs.

Don’t Just Buy Ediscovery: Manage It

The webinar discusses that keeping ediscovery costs in check is not as simple as choosing the right provider. While setting a budget and considering fee schedules are important, this is not all that needs to be done to influence ediscovery costs. Panelists suggested:

  • Monitoring costs along the way; ask for regular updates
  • Communicating with corporate, law firm and provider representatives; lack of communication increases costs
  • Cooperating with opposing parties where possible; come to an agreement on things such as the number of custodians, production parameters and document review protocols

Don’t Miss the Opportunity to Get Proactive

Being prepared for ediscovery saves downstream costs. Webinar panelists highlighted the importance of ediscovery assessments and postmortem evaluations in pinpointing risks and identifying cost reduction recommendations across discovery response plans.

TAR: Building a Better Playlist

TAR

“Sometimes it seems as if our Pandora and Netflix accounts know us better than we know ourselves, and can build a better play list…”

Brett M. Anders

Brett M. Anders, Jackson Lewis

In a recent article in Today’s General Counsel, Brett M. Anders of Jackson Lewis and my Kroll Ontrack colleague Rick Anderson, seek to debunk the misconception that human lawyers alone can build a better playlist when it comes to legal document review.

Human Review Is Not the Gold Standard

Practitioners shy away from predictive coding and technology assisted review (TAR) in part because of the myth that human review is superior to that done by a computer. However, this is not the case. Humans can be inaccurate: relevant documents can be missed and accuracy suffers. This position has been verified by studies and is generally accepted by the judiciary. The use of TAR has received much support from the courts in the cases where it has been an issue. However, at this point, no court has gone so far as to mandate the use of TAR.

Rick Anderson, Kroll Ontrack

Rick Anderson, KrolLDiscovery

Using Technology Assisted Review in Litigation

As of now, several judicial opinions have surfaced regarding the use of TAR. As noted above, the courts have supported it as a cost effective method for conducting discovery. The standard for discovery responses is “reasonable and proportional to the matter,” not perfection. Therefore, the accuracy offered by the use of TAR satisfies the standard for discovery production.

Speaking of proportionality, using TAR can place a party in a better position to make an argument about proportionality when litigation costs become too high. Because TAR prioritizes which documents are most likely to be relevant, a party who makes its way through the documents with the highest relevance has a basis to argue that additional discovery would not be “proportional to the needs of the case.”

Advantages of Using TAR

As discussed by Anders and Anderson, despite the apprehension to utilize this technology in the legal community, the fact remains that TAR has many advantages.

  • TAR costs a fraction of the expense it would take to review documents manually
  • TAR is typically faster than traditional document review, while also more accurate
  • Courts have approved the use of TAR; parties no longer need to worry about being the first to use this technology in a case
  • Courts are encouraging its use, while respecting the party’s wish to keep its seed set (used to train predictive coding technology) confidential

For more information on TAR, and how to utilize its benefits in litigation, be sure to read the full article, “Building a Better Play List with Technology-Assisted Review.”

Predictive Coding Technology: A Summer Blockbuster You Won’t Want to Miss

Predictive Coding Patent

What’s better than a summer blockbuster? Few things beat the heat better than an air-conditioned theater and a bucket of buttery popcorn while watching an engaging film.

Predictive Coding Patent Blockbuster:  Technology You Don’t Want to Miss

Kroll Ontrack has come out with its own “blockbuster” this summer: a predictive coding patent. Obtaining patents, like making blockbuster movies, takes years and ours was more than four years in the making. To accompany our patent, we have another mega-hit that deserves a second screening: Kroll Ontrack’s predictive coding guide. Our newly-patented technology, paired with our ever-relevant predictive coding guide, is a superhero team that the up-to-date ediscovery practitioner cannot do without.

2016 Summer Blockbusters: Movies You Don’t Want to Miss

Of course, there are other highly anticipated blockbusters arriving this summer. To start, the smart mouth turtles-turned-mutants of our youth are back in Teenage Mutant Ninja Turtles: Out of the Shadows. In Independence Day: Resurgence the people of earth once again are in critical struggle against an alien invader (hopefully with a better spaceship design). The daffy ghost hunters have returned in revival of the ’80’s classic movie Ghostbusters, this time with an all-female cast and the same catchy theme song. Star Trek Beyond continues the story of the USS Enterprise crew in the newest installment in one of popular culture’s longest-running franchises. Finally, the list would not be complete without the newest villain-themed movie from the Batman franchise: Suicide Squad.

Live Long and TAR!

April Webinar: Got Data? Analytics to the Rescue!

Got Data Analytics to the Rescue

On April 19, 2016 join  experts Cathleen Peterson, Jim Sullivan and Kiriaki Tourikis from JP Morgan, as they discuss data analytics as both the lifeblood powering critical business operations and the kryptonite preventing the business from flexing its muscle. When investigations, litigation or compliance matters strike, organizations and their counsel that leverage analytics are more likely to win.

This session will feature hypothetical scenarios to explain the various analytics tools and how they fit into a case, data breach or investigation. At the end of the session, participants will understand how analytics can help:

  • Map the data collection and explore key points and related themes
  • Identify key players, timelines and communication patterns
  • Mine data for Personally Identifiable Information (PII)
  • Find redundant information and remove non-relevant, junk data

Predictive Coding’s Ripple Effect

There’s always a new video going viral. The best ones are often talked about over the office water-cooler, making the participants overnight sensations. Consider 5-year-old Noah Ritter of Pennsylvania. He was merely interested in the super slide at his county fair, but when a local news crew asked him how the ride was, his adorable interview immediately went viral, garnering more than 3 million views on YouTube. That single video sent a ripple effect through the internet (and likely his personal life), and his interview was featured in articles and news outlets as far away as India and earned him a spot on Good Morning America. And now, for the crème de la crème, his name even graces this fine blog.

This ripple effect of popularity got me thinking about predictive coding (no surprise there). Predictive coding certainly made a splash when the first Federal court approved its use in litigation in Da Silva Moore v. Publicis Groupe. But since then, savvy practitioners have seen predictive coding as more than a discovery tool and have devised new uses for this powerful technology. The idea of incorporating machine learning and predictive coding into everyday business tasks is not far-fetched. This ripple effect of predictive coding can be seen in many areas, but there are three specific areas where the ripple effect will likely soon be a wave.

First, early use of predictive coding can be used to narrow the scope of discovery and refine the precision of collection. In addition, using predictive coding in this way can effect settlement. Incorporating predictive coding into EDA creates value that at least one court has already recognized.

Second, predictive coding can be used as a compliance tool.  Predictive coding is a natural way to assess and detect risk patterns, and stop them from developing further. Therefore, predictive coding software could be trained to comb through all of an organization’s information to detect potential risks and enable counsel to take immediate corrective action.

Finally, predictive coding can be used to combat the trials of information governance. Predictive coding can be and should be implemented in record retention policies. Predictive coding can identify which documents should be kept – and those that can be defensibly deleted. As more and more businesses become bogged down by big data, a smarter, more automated approach to record retention will be the future of information governance.

New Pulse Benchmarks Demonstrate Trends in Source Media and Deduplication

Don’t miss a beat; the new Pulse Benchmarks are here! These metrics can help practitioners keep up with trends in the ediscovery market and better plan and execute their ediscovery projects. If you’re new to ediscovery.com Pulse, here’s a quick overview: Kroll Ontrack’s Pulse Benchmarks present aggregated and trended data from over 4,000 matters over a five-year span (2008-2012) to identify trends and key changes in the ediscovery market. Our two new Benchmarks cover interesting trends in types of source media and deduplication review volumes. To see the two new Benchmarks, please download the report.

Types of source media fluctuate as ediscovery processing becomes standardized

Understanding what data goes into a project is important to getting to the information you need. Source data for ediscovery processing can be transmitted in many formats, but data from the Pulse Benchmarks has indicated a trend away from data being transmitted on CDs/DVDs and more data being transmitted via electronic file transfer (or FTPs).

Deduplication rates rise to reduce review volumes

Cutting time and costs is what technology assisted review (TAR) is all about, and one of the important tools used in TAR is deduplication. Deduplication is the process of comparing documents based on characteristics and removing duplicate records from the data set. This process can either be leveraged on the entire data set (project level deduplication) or a subset of data (such as custodian deduplication). This Pulse Benchmark indicates that project level deduplication selections are on the rise, marking a trend toward efforts to drive down review set volumes to cut time and costs.

What do you think about the trends outlined by the new Pulse Benchmarks? Why do you think source data is move away from CDs/DVDs and toward FTPs and why do you think deduplication is being used more and more on a project level basis? Where do you expect these numbers to be in the coming years? Let us know in the comments below!

Predictive Coding Semantics: Step out of the Rain!

A few years ago, we wrote about predictive coding going mainstream; shortly thereafter we wrote a series debunking the most common myths about predictive coding; then, we even went so far as to break down predictive coding lifecycles (actually, we did that twice!), explained how to maximize training for machine learning, and taught a short lesson about the most commonly-used stats for evaluating your machine’s work. So, to make a long story short, we’re pretty big on using predictive coding as part of a search methodology, and we’re doing everything we can to demystify the process so that organizations can use this technology to increase their ediscovery efficiency and, in turn, save a significant amount of money. If you don’t believe me, check out one of our case studies illustrating the myriad benefits of employing predictive coding for search, analysis, and review.

According to Fullbright’s 9th Annual Litigation Trends Report in 2013, however, only 35% of the respondent companies indicated they were using some form of predictive coding. While that percentage is likely a significant increase from the pre-Da Silva Moore days where only a few ediscovery pioneers were brave enough to dive into their data with predictive coding, it still suggests some hesitancy on the part of practitioners.

Defining “Predictive Coding” and “Technology Assisted Review”

Whether the legal community’s hesitancy is related to cost, uncertainty, or a lack of matters they deem appropriate for predictive coding, it’s still a fairly opaque process, and it starts at one of the most basic levels: vernacular. The ediscovery community loves this technology… but it still hasn’t settled on what to call it. Whether it’s “predictive coding,” “technology assisted review,” or “computer assisted review,” there’s a smattering of terms floating around and little consensus about the preferred term.

As I stated before, Kroll Ontrack is committed to demystifying this process, and we’re drawing some lines to help clear up the semantic ambiguities related to predictive coding. It’s best to think of technology assisted review, or TAR, as an umbrella term that encompasses a variety of other advanced litigation technologies designed to help litigation teams work more efficiently and ease the burdens of standard linear review. Under this approach, predictive coding is just one prong of the larger array of complementary TAR and Early Data Assessment tools, such as near-dupe identification, email threading, visual analytics, workflow automation, and other reporting tools.

Top 5 Ediscovery Case Summaries – July 2013

Read the very latest ediscovery case law summaries

No Relevance of the Evidence? No Sanctions                                                   Cottle-Banks v. Cox Commc’ns, Inc., 2013 WL 2244333 (S.D. Cal. May 21, 2013).

Spoliation Sanctions Levied Against Damages Awarded in Rambus  Case                                                                                                                                                                                                                                                                              SK Hynix Inc. v. Rambus, Inc., 2013 WL 1915865 (N.D. Cal. May 8, 2013).

Special Master Analyzes “Overly Broad” Privilege Search Terms                                                                                                                                                   Dornoch Holdings Int’l, LLC v. Conagra Foods, Lamb Weston, Inc., 2013 WL 2384235 (D. Idaho May 1, 2013).

Sanctions Granted for Reckless Spoliation of Data                                     Pillay v. Millard Refrigerated Serv., Inc., 2013 WL 2251727 (N.D. Ill. May 22, 2013).

Minimal Court Oversight Required for Predictive Coding Protocol                                                                                                                                         Gordon v. Kaleida Health, 2013 WL 2250579 (W.D.N.Y. May 21, 2013).

 

Deploying Trainers in Technology Assisted Review (TAR) without “Spoiling the Broth”

Deploying trainers in technology-assisted review (TAR)

Leaving little room for interpretation, the court in Coquina Investments v. Rothstein, stated that the defendants’ litany of ediscovery project management pitfalls (which involved over 200 attorneys across two firms) culminated into a “case of too many cooks spoiling the broth.” While Coquina Investments involved format of production issues, the same rationale applies when deploying trainers in technology-assisted review (TAR) —too many trainers can lead to inconsistency and poor machine learning.

Taking Control of the Technology-Assisted Review Kitchen

Using TAR in litigation is strikingly similar to working in a professional kitchen. There are many parts moving on parallel tracks. Just like a pastry chef may begin working on dessert while a grill chef prepares the main dish, you may have reviewers allocated to train a recently found hard drive while a sub-team performs corrective training on a production set. And above all else, in either scenario, nothing leaves the kitchen without a taste test (quality control). But perhaps the most difficult task involves assigning appropriate roles to a diverse cast of employees during the stages of machine training.

  • Lead Attorney: The Chef de Cuisine—in charge of all things related to the kitchen. This role involves making executive decisions like when to stop review, how to provide additional training and who will train the machine.
  • Subject Matter Experts (SMEs): The Sous-Chefs—second-in-command to the Chef de Cuisine. These are attorneys that have a firm knowledge of the nature of the case and the issues involved. They are capable of making high-level decisions and have an expansive knowledge of the dispute.
  • Contract AttorneysThe Chefs de partie—line cooks responsible for certain areas of production. These are attorneys who are comfortable and trained on the issue at hand, but do not have the level of knowledge possessed by Subject Matter Experts.

Choose Your Recipe

The Chef de Cuisine works closely with the Sous-Chefs to ensure that everyone clearly understands the basics of the recipe so that when the Chef de Cuisine (the Lead Attorney) is out of the “kitchen” the quality of the output remains constant.

When it comes to dedicating a team of SMEs to train the system, the adage “less is more” carries the day. As discussed in a document produced by the TREC 2008 Legal Track, determining whether a document is responsive or not responsive is a deceptively subjective process.  Lawyers “draw lines”—often at different places—across a number of determinations like “the nature of the risk posed by production, the party requesting the information” and the willingness of the production party to face a challenge for underproduction. Because the risk of inconsistencies in deciding responsiveness is exacerbated by the introduction of more trainers, rarely will you want more than five SMEs training the system. The restaurant owner mutters, “but my project is big, there is no way that I can rely on only five reviewers.”  Generally, two to five reviewers can handle the targeted review load for even a very large project. The total amount of training documents will vary depending on if you plan to “seed” the system (and how much “seeding” you plan to do), the number of documents in your data set and your desired confidence level. Ultimately, responsiveness decisions made on this fraction of documents will be extrapolated to all remaining documents in the data set; it becomes critical that the SMEs are in sync with the goals and structure of the case.

Reduce and Stir

While the ideal structure for deploying this handful of SMEs is still up for debate, there is common consensus that there must be some process in place to arbitrate consistency when responsiveness disputes arise. I’ve seen some interesting hierarchical training structures over the years designed to handle training disputes. These are some of the most common:

Training Structures of technology-assisted review

Finally: Tasting the Broth

An effective document review and an efficient kitchen both rely upon QC measures to ensure quality and consistency of output. A well-designed plan for validating the automated technology-assisted review output is key to knowing when to stop training for quality and when the documents are ready for consumption at the next stage of the case. Where the Chef de Cuisine is responsible for ensuring that only quality dishes leave her kitchen, the Lead Attorney is also responsible for the quality of the data in her case. Only when quality control measurements reflect defensible levels of recall and precision will a Lead Attorney be in a position to move beyond first-pass review and plate the production for the requesting party—Bon Appetit!

To gain hands-on TAR experience, register now for the newest educational course offered by Kroll Ontrack, TAR Learning Labs.  The next Learning Lab is coming up in Minneapolis, MN, in early June.  Sign up soon, space is limited!

 
css.php