Legal TREC: An In-depth Examination
of Text Retrieval

By Joe Howie
Home
Publications		 Read some of the various articles about the TREC Legal Track to understand the limitations of full text retrieval and manual review of discovery documents. TREC is the Text Retrieval Conference, an ongoing project sponsored by the National Institute of Standards and Technology and the Legal Track is a relatively new track that examines text retrieval in the context of legal discovery processing.

A few highlights from the recommended reading:

  • Full text retrieval alone finds a relatively small percent of documents found to be relevant by using a combination of retrieval methods — 22 percent in the 2007 Legal Track and 24 percent in 2008. This is about the same as the 1985 Blair and Maron study.
  • Other, newer technologies generally fared about the same as full text retrieval, although in some cases they were able to provide better results.
  • An iterative process where results are discussed with the domain expert (e.g. the attorney responsible for the production) yields the best results.

The Roitblat blog cited below also makes another interesting point: The decisions made about relevance by different reviewers varied considerably from one reviewer to another.

Considering that relevance and privilege reviews are among the largest cost factors in discovery processing, you can expect much more attention to be paid to the work of TREC and the EDiscovery Institute in the upcoming years. The Legal TREC needs volunteers, and if you’re interested in getting some hands-on experience in this important project, contact Jason Baron, director of litigation for the U.S. National Archives and Records Administration and one of the TREC coordinators (jason.baron@nara.gov).

If you have conducted internal reviews of retrieval effectiveness or of the variability of decision making by different reviewers or by the same reviewers at different times, we’d love to hear from you — the profession needs more fact-based analysis to help it arrive at best practices that can be objectively measured.

Recommended reading:

Also, an earlier ALSP Update article discussed the EDiscovery Institute’s evaluation of the effectiveness of different approaches to reviewing documents (“Cutting-Edge e-Discovery Research, an Interview with Anne Kershaw and Patrick Oot,” April, 2009) and another summarized the Blair and Maron study on the effectiveness of full text searching (“Case Law Restrictions on the Use of Full Text Searching,” March, 2009).
   

This article appeared originally in the June 2009 ALSP Update, the monthly publication of the Association of Litigation Support Professionals and is reprinted with permission. Read more about this nonprofit membership organization at www.alsponline.org.

www.HowieConsulting.com

When you have to get it write
For more information, email Joe Howie, Joe@HowieConsulting.com