All posts tagged EDRM

Modern Ediscovery: Making Sweet Music in the Virtual Age

Who wants a flip phone when you can have the latest smartphone in your pocket? No one wants to be seen as behind-the-times in this technology-driven age. When it comes to ediscovery, we want to be modern, too. In fact, we must be modern; outdated methods will simply get you (and your client) in trouble.

Recently, I had the pleasure of presenting a webinar, Ediscovery Program Management: From Vinyl to Virtual, with two ediscovery gurus – one a former colleague and one a current colleague. Along with Josh Zylbershlag, ediscovery litigation services director at Paul, Weiss, Rifkind, Wharton & Harrison LLP and Tom Barce, director of consulting services at KrolLDiscovery, we explored the history of the music industry, comparing it to our own “vinyl to virtual” shift in the ediscovery industry. Are you still conducting ediscovery the same way you would buy music off the racks in the record store? Or, have you adapted to digital download or a streaming subscription?

  • Information Governance: Are you still managing huge file servers with no idea what’s on them, or are you an organization that creates, uses and governs information with an eye toward accessibility?
  • Legal Hold: Has your corporation adopted legal hold policies and guidelines? Have you taken it virtual by subscribing to technology solutions with effective legal hold management?
  • Collection: What about your collection processes? Are you sending droves of people to collect data every time there is a new case or are you leveraging modern remote collection technology and centralized discovery databases?
  • Search and Analysis: Is it time to update your review practices? Are you still in the dinosaur age, not using electronic workflow and batching, dynamic data profiling, innovative search analytics, predictive coding and sampling?

Our goal in this webinar: to make sweet music at each stage of the EDRM by discussing modern ediscovery practices.

Making Music: Ediscovery Collaboration

From a hip-hop country mix “Cruise,” to the smash hit “Thrift Shop,” 2013 was a year of musical collaborations. Nevertheless, collaborations are not new: from Sinatra’s Duets to Dolly Parton and Kenny Rogers’ “Islands in the Stream,” collaborations are a staple in pop culture. The reason behind this is while a solo artist can create beautiful music, the harmony of two contrasting voices generates a unique and compelling sound.

Great collaborations, however, are not limited to stadiums and Grammies. Since the beginning of ediscovery, collaborations, in the form of mergers, acquisitions and partnerships, have been an industry staple. It is through these partnerships that the industry moves forward. Like pop culture collaborations, two ediscovery providers working together can create beautiful music.

In ediscovery, it is not uncommon to think of ediscovery tools in a vacuum, serving a niche of the EDRM – such as a singular review program – that solves only one piece of the puzzle. These tools are unconnected, operating apart from one another, as so many musicians do for years.  For this reason, law firms and corporations often have more than one ediscovery tool at their disposal. They often run more than one software package at the same time, trying to patch together a full solution. There is no streamlined, standardized way to use different tools together. This problem has led to a push for better visibility across all tools and cases to allow consolidation of work across the ediscovery portfolio.

This is the exact reason that today, Kroll Ontrack announced it is now a Premium Hosting Partner for kCura’s Relativity platform.  kCura’s Relativity review tool will now be offered as part of Kroll Ontrack’s platform alongside the Collect, Review and Manage tools. The joint venture gives law firms and corporations review tool choice based on the needs of the project while also making it easier to consolidate their project portfolio. Regardless of the chosen review tool , transparency into all the work being done on your cases, as well as the ability to direct actions, is possible via the Manage Project Wall. With the ability to bring together ediscovery projects in a unified technology platform and manage them as a portfolio, ediscovery can transform from an art to a science. This kind of collaboration is, like some musical collaboration, pure genius.

In the above montage, from upper left, clockwise:
Colbie Callait & Jason Mraz; David Bowie & Mick Jagger; Louis Armstrong & Ella Fitzgerald; Frank Sinatra & Dean Martin; Mariah Carey & Whitney Houston; Paul McCartney & Michael Jackson; Macklemore & Ryan Lewis; Justin Timberlake & Jay-Z; Dolly Parton & Kenny Rogers

What Do Robin Thicke and the EDRM Have in Common?

The answer: blurred lines.

…Okay, now that I have your attention, I should note that post doesn’t really have anything to do with the controversial and provocative Robin Thicke song that was all over the airwaves this summer. Instead, I’m talking about two equally appealing topics (at least to those in ediscovery): Early Data Assessment (EDA) and Technology Assisted Review.

We have written a lot lately about ways to cut ediscovery costs, and both EDA and TAR (or predictive coding) are keys to reducing costs and maximizing ediscovery efficiencies. While both tools empower attorneys to build a better picture of the data and documents involved in a case, TAR solutions are often viewed as wholly separate tools. However, many practitioners now employ TAR at various stages of the EDRM workflow, and TAR often performs the analysis that is so imperative to the EDA process. Practically, TAR is a complimentary tool that enhances EDA, rather than a standalone solution.

For a more critical analysis this evolution, be sure to attend Analysis 360: Blurring the Lines Between EDA and TAR on November 20, hosted by Anthony J. Diana, partner at Mayer Brown, and Jonathan Sachs, Account Executive at Kroll Ontrack.

Hear it From the Experts: Reduce, Reuse, Recycle

Webinar or Podcast: You Choose!

In case you missed it, last week’s webinar featured a rousing discussion about how to effectively manage a portfolio of ediscovery cases. Instead of focusing on how to whittle costs in one project within the EDRM, eDJ’s Greg Buckles and John Addington from Dell discuss the massive efficiencies that can only be unlocked through a holistic, multi-matter approach to ediscovery.  Log on today and listen to the recorded webinar: “Reduce, Reuse, Recycle: Effectively Managing Ediscovery Portfolios”.

If you only have 20 minutes, then download the podcast and listen on your commute home!  John Addington from Dell talks with Michele Lange about the same topic – ediscovery portfolio management.

Don’t miss either of these great interviews!

The EDRM’s New Computer Assisted Review Reference Model: Explained

Last month I was invited by the EDRM to take part in their webinar on the Computer Assisted Review Reference Model (CARRM). I was joined by three esteemed Technology Assisted Review experts: George Socha, EDRM, Herbert Roitblat, OrcaTec and Bob Rohlf, Exterro.

We took this chance to dive into the fascinating world that is predictive coding, also known as Technology Assisted Review (TAR), Computer Assisted Review (CAR), or intelligent review.  Predictive coding is the use of computer technologies to rank or categorize a collection of documents as responsive or not based on human review of a subset of the collection.

The talk started with a discussion of how we got to predictive coding today, and why the court’s blessing to use predictive coding in certain civil litigation cases is so important. Since the first blessing by Judge Peck in February of 2012, the number of cases using predictive coding has grown substantially. Without that blessing, it is unlikely predictive coding would still be growing.

The next question we addressed was simply “Why Predictive Coding?” The other experts and I discussed the ways predictive coding saves time and resources by finding the right documents as fast as possible, sorting and grouping documents more efficiently and validating the reviewer’s work before production.

After the opening discussion, we dove into an assortment of predictive coding topics, including the variety of technologies available and their differences, how to conduct effective predictive coding, and predictive coding workflow.

We closed the forum by discussing the best practices in ediscovery and predictive coding:

  • To be efficient, you must know which questions to ask your ediscovery experts – this means doing your research.
  • You need to be proactive in your firm’s ediscovery plan: create a plan and stick to it!
  • Be sure to ensure quality controls so the results are respectable.
  • Finally, do not be afraid to ask for help when you do not understand the process. The field is very new and growing.

For those of you who missed the webinar and would like a closer look, check out the recording of the EDRM’s New Computer Assisted Review Reference Model (CARRM)—Beyond the Test Drive and be sure to check out Kroll Ontrack’s Slideshare account for the latest presentations and infographics.

Ediscovery Pricing Pains

Recently, Casey Flaherty, Corporate Counsel at Kia Motors, authored an article about how he approaches comparing costs in the  current chaotic ediscovery pricing landscape.  Flaherty boldly asserts: “Standardizing a method for comparing EDD vendor cost projections is long past due.”  He continues by suggesting that such a cost-comparison spreadsheet is a necessary tool which should be developed and maintained by a group such as the Sedona Conference, the EDRM or the EDBP.

Flaherty’s article is evidence of the gyrations that corporations and law firms go through when analyzing ediscovery pricing.  Per page.  Per gigabyte.  Per hour.  Per day.  Per month.  Flat fee.  Annual subscription.  Kudos to Flaherty for bringing his method to light and encouraging everyone else in the industry to do the same.

It makes plain business sense to structure ediscovery pricing around transparency, and Kroll Ontrack is trying to help drive the market toward that rational ediscovery pricing panacea.  Recently, Kroll Ontrack launched a new pricing model, containing a wave of pricing options including:

  1. Traditional a la carte (line-item comparisons) – traditional pricing based on different actions in the EDRM – keep the spreadsheets for those who want them
  2. Total project price – all of the bells and whistles for one bottom-line project price – for those sick of fashioning “order from chaos”
  3. Portfolio pricing – a monthly fee based on a revolving-door portfolio of matters limited only by capacity and duration – ediscovery pricing can really be as simple as X gigabytes for $Y/month

To be perfectly frank, in an industry where everything can change overnight, this problem has persisted for too long. That’s why allowing clients to price projects on their terms is a great first step in clarifying ediscovery pricing pains.

The CARRM: Beyond the Test-Drive

Whichever name you give it – predictive coding, computer assisted review (CAR), or technology-assisted review (TAR) – predictive coding is an amazing tool in the field of ediscovery. In 2012, the EDRM published a new Computer Assisted Review Reference Model (CARRM) framework to demystify this new technique.

 First and foremost, the CARRM strongly emphasizes a planning process, during which the review team should:

  • Determine the desired outcome of the predictive coding process
  • Build rules and methods specific to the case for both human reviewers and the technology
  • Educate the reviewers who are involved in the predictive coding process about those rules and methods

Next, the CARRM identifies a process of coding documents, from which the technology learns and classifies other documents in the corpus, followed by human testing and evaluation. This process is iterative, meaning these steps should continue in a cycle until appropriate retrieval metrics (such as precision, recall and f-measure) are achieved and the initial goals are met. Once the evaluation stage is complete, the predictive coding process can end and the team may move on to the next phase of review.

The benefits of predictive coding compared to human review are plentiful. Predictive coding can reduce time on administration and review, reduce the number of documents reviewed, increase the accuracy of review and even review documents quicker.

The bottom line is that while ediscovery budgets are cut and data volumes are ever-growing, predictive coding can help you save time and money by:

  • Finding the right documents quickly
  • Sorting documents efficiently
  • Confirming the reviewers’ work before production

What to know more? Register for the EDRM Webinar, “EDRM’S New Computer Assisted Review Reference Model (CARRM)-Beyond the Test Drive,” with industry experts George Socha, Herbet Roitbat, Bob Rohlf, and myself.  It should be a riveting conversation so be sure to register soon.

Technology Assisted Review Models: Let’s Move Forward

Technology-assisted Review Models: Let's Move Forward

It was exciting to see the number of significant conversations about technology-assisted review (TAR) and analysis at LegalTech New York, even though the industry still hasn’t settled on standard terminology. Nonetheless, whether they’re calling it TAR, Computer-assisted Review (CAR), Predictive Coding (PC), or any other acronym referring to this technology, people are taking notice and trying to understand how to implement this beneficial tool in their everyday litigation process.

Understanding the Basics of Technology-Assisted Review

Before you can implement any new solution it’s important to understand the basics. While the specific algorithms behind this technology can be highly technical when discussed in-depth, those details are typically less important than the process used to implement the technology. Therefore, a higher-level understanding is paramount to appropriately implementing TAR or any existing technologies on the market today.

The Electronic Discovery Reference Model (EDRM) was one such model to emphasize quality processes, and it has proven extremely successful in guiding ediscovery best practices through all stages of litigation. Given the success of this model, the EDRM search sub-team has moved forward with another visualization that aims to demystify TAR with a draft of the Computer Assisted Review Reference Model (CARRM).

I am proud to be a member of the team that worked to put this reference model together. The group was composed of people with diverse backgrounds—including both vendors and end-users—and we were all pleased to find that the overarching visualization covered the solution well, regardless of technical implementation.

The CARRM identifies eight critical steps in the TAR processComputer Assisted Review Reference Model

First and foremost, the CARRM strongly emphasizes a planning process, during which the review team should (1) determine their desired outcome in leveraging TAR; (2) build rules and methodologies for the humans and the technology specific to the case; and (3) convey those rules and methodologies to the reviewers involved in the TAR process.

Next, the CARRM identifies a process of coding documents, from which the technology learns and classifies other documents in the corpus, followed by human testing and evaluation. This process is iterative, meaning these steps should continue in a cycle until appropriate retrieval metrics (such as precision, recall and f-measure) are achieved and the initial goals are met. Once the evaluation stage is complete, the TAR process can end and the team may move on to the next phase of review.

The timing must be right to start visualizing the high level process because several other illustrative models emerged in the past year (each built without knowledge of the other models):

Predictive Coding Search diagram by Ralph Losey

While the models have differences, mainly in terminology and aesthetics, all models bear important similarities. Namely, each model illustrates the importance of:

  • thorough planning;
  • employing a cyclical process with significant interaction between the human reviewer and the technology; and
  • evaluating the results of such interplay before moving on to the next stage.

Like the ongoing debate about vernacular, the primary differences between these models are mostly a matter of semantics that will resolve themselves with time, familiarity and, eventually, increased standardization. In the meantime, pay close attention to the standards and principles advanced in each model and adhere to them when evaluating the viability of TAR for a specific case.

Most importantly, the EDRM group is looking for feedback on their model, and I highly encourage you to review their model and respond with support for the model, questions, or ideas for improvement. The sooner we can all agree on the overarching process for implementing this technology, the sooner we can focus applying this solution to everyday processes and reap the benefits of this new technology. Please send any feedback about the CARRM draft directly to

A Quick and Dirty Guide to Ediscovery Project Management (Part 2)

Ediscovery Project Management - EDRM

In a complicated litigation landscape, it’s a relief that ediscovery project management (EDPM) and the Electronic Discovery Reference Model (EDRM) fit together. An effective project manager will consider each stage of the EDRM, from identification to production, when creating the perfect ediscovery game plan for the given set of circumstances. Let’s take a look at the most important stages of the EDRM from a project management standpoint.


Project managers are key players in coordinating the effort to identify all data that may be potentially relevant to a specific matter. Ask specific questions of document custodians and walk through a comprehensive list of business and personal data sources – if a device has memory, keep it in your purview. In Coleman (Parent) Holdings v. Morgan Stanley, a 2005 Florida case, Morgan Stanley was unaware of where it stored its electronic data, and was thus sanctioned for discovery abuses. Morgan Stanley faced compensatory and punitive damages to the tune of $1.4 billion. It’s an extreme example, but nonetheless a pertinent one that shows what effective ediscovery project management should avoid.

 Collection & Preservation

This stage of the EDRM is the origin of many “gotchas” in the ediscovery process. It is certainly where most judicial sanctions stem from in the ediscovery realm. Project managers can help guide their organization and document custodians to a successful avoidance of preservation sanctions by ensuring that they know and comply with their obligations pertaining to a litigation hold. They may also play a key role in implementing and managing the organization’s overall retention plan before and during litigation.

 Early Data Assessment (EDA)

By this point in the ediscovery process, the body of knowledge that project managers have gained in coordinating the matter may qualify them as a subject matter expert and an invaluable resource to the legal analysis team. During the EDA process, ED project managers wear many hats and perform many functions. They often work with attorneys, other litigation support professionals and third party providers to separate data between critical and non-critical data groups, narrow the number of key players/custodians, test key search terms,  and identify critical case arguments.

 Review/Technology-assisted Review

Technology-assisted Review (TAR) is one of the most cost-effective, consistent and accurate methods by which to distill the mountains of data inevitably unearthed during major litigation or investigations. To connect back to the EDPM framework discussed in Part I of this post, project managers serve well to promote TAR when discussing process, budget and cost.


So, you’ve finally reached the glorious end of your ediscovery project – production! Project managers (depending on where they sit) function as a useful link between the organization, counsel and the third party ediscovery provider. Many ED project managers are responsible for actually creating and validating productions. They may also ensure that the finished product gets delivered promptly and in its entirety to the appropriate party(ies).

Leveraging Cloud-Based Delivery Capabilities for DIY Ediscovery

Leveraging Cloud-Based Delivery Capabilities for DIY Ediscovery

Today, ediscovery law and practices are in a continuous state of flux, which underscores the need for corporations and law firms to continually evolve in order to meet their ever-changing ESI obligations. IDC’s Vivian Tero recently published a whitepaper sponsored by Kroll Ontrack, “Leveraging Cloud-Based Delivery Capabilities for Do-It-Yourself (DIY) eDiscovery,” which highlights this very fact, and explains that there are several factors currently driving change in e-discovery.

Factors Contributing to the Changing Ediscovery Landscape

Most noticeably, there exists aggressive growth in the volume of data that litigants must preserve, collect, analyze, and review. Specifically, Tero points to an increase in the proportion of litigants with large matters that average at least 2.5TB per matter, as the figure below illustrates. Given this growth in data volume, organizations must have flexible e-discovery systems in place that allow them to easily scale up or down according to the size of different matters, and that allow them to strike the right balance between over-preservation and defensible legal disposition.

Average ESI Volume Collected per Matter


Further, as Tero indicates, several other factors contribute to the changing legal landscape of ediscovery. For example, an increasing number of organizations are insourcing one or more aspects of e-discovery, especially in the areas of identification, preservation, collection, processing, and first-pass review. Also, the number of e-discovery court decisions is growing quickly, as the number of decisions during early 2011 increased 82% over the number of decisions from the previous year. Introduction of new technologies into the corporate infrastructure is also bringing about the necessity for decision-makers to constantly reevaluate e-discovery laws and regulations.

Further, as Tero indicates, several other factors contribute to the changing legal landscape of ediscovery. For example, an increasing number of organizations are insourcing one or more aspects of e-discovery, especially in the areas of identification, preservation, collection, processing, and first-pass review. Also, the number of e-discovery court decisions is growing quickly, as the number of decisions during early 2011 increased 82% over the number of decisions from the previous year. Introduction of new technologies into the corporate infrastructure is also bringing about the necessity for decision-makers to constantly reevaluate e-discovery laws and regulations.

Approaches to Manage Change by Automating EDiscovery

IDC’s Tero underscores different approaches by which an organization can respond to changes in ediscovery by automating core ediscovery processes end to end. The first approach is the “single-vendor platform approach.” In this approach, an organization chooses which stages of the Electronic Discovery Reference Model (“EDRM”) to conduct internally, depending on the nature of each matter, and employs a single provider to handle the remaining stages of the EDRM with services and/or software.  This approach streamlines ediscovery activities and reduces e-discovery costs.

The second approach is the “multivendor, multiplatform hybrid approach,” currently used by nearly 65% of organizations. An organization typically supports multiple archival and content management systems, as well as multiple geographically distributed IT organizations with complex data segregation mandates. The organization may internally conduct certain stages of the EDRM, depending on the matter, but various vendors and tools complete other stages. This approach creates a multifaceted, often inefficient, ediscovery management system.  The majority of these multivendor e-discovery architectures are the outcomes of business and IT organizational developments such as mergers and acquisitions, changes in IT strategies, and siloed and reactive purchases of ad hoc technologies.

Average ESI Volume Collected per Matter

Selecting the Appropriate EDiscovery Infrastructure

As Tero notes, given the myriad of available ediscovery options, organizations should scrutinize their options carefully – looking at e-discovery services, on premise software and cloud or software-as-a-service (SaaS) platforms. Selection of the appropriate approach is influenced by many factors, including the organization’s litigation profile, the organization’s existing IT capability and long-term IT architecture strategy, as well as the business and technical attitudes of the outside provider.

At the end of the day, organizations need to engage in a strategic assessment of their needs and resources, given this constantly evolving landscape. Organizations should consider leveraging infrastructure and processes to efficiently conduct information and document management, meet preservation requirements and discovery deadlines, and ensure data security and chain-of-custody documentation. This means possibly utilizing cloud or SaaS-based DIY e-discovery technologies in addition to service providers for e-discovery obligations. Engaging in DIY e-discovery may be new to many organizations, but this option should be fully considered by organizations amid the changing market.

Download the full IDC White Paper.