INTERPRETING ACTIVE LEARNING RESULTS

Question 1

Reviewer access

As a reviewer how can you enter the review queue?

Answer 1

Relativity creates a new document list view that’s tied to the Active Learning project. This view is automatically secured to the reviewers added to the project. As an Active Learning reviewer, this view is the only place you can enter the review queue.

Question 2

Reviewer access

How would you ensure that reviewers cannot skip documents?

Answer 2

Make the project review field a required field.

Question 3

Reviewer access

Can reviewers change coding on documents they have already coded?

Answer 3

Reviewers can change the coding decision on documents they previously reviewed. These documents aren’t considered manually-selected documents. The next model build will include the most recent coding update.

Question 4

Reviewer access

Reviewers must code documents based on the so-called “four corners” rule.

What is this and why is it important for Active Learning?

Answer 4

This means that a document should be judged responsive or not responsive based solely on the extracted text of that document only. Documents that are relevant based on family members should not be coded relevant on the Active Learning review field. Although individual anomalies will not hurt the algorithm, too many in aggregate could cause the model to perform worse.

Question 5

Monitoring an Active Learning project

How many docs need to be coded before an active learning project can complete it’s first build?

Answer 5

At least five documents coded with the positive choice and five coded with the negative choice

Question 6

Monitoring an Active Learning project

How often will builds take place?

Answer 6

At maximum every 20 minutes after the previous build to include coding decisions not included in the most recent build. If reviewer activity has been idle for five minutes and there are coding decisions not included in the most recent build, the model will start a build.

Question 7

Monitoring an Active Learning project

Project Home dashboard - What is it?

Answer 7

Gives a high-level overview of the documents in your Active Learning project.

After you first create the project, the dashboard displays the Project Size and coding statistics based on the pre-coded documents in your data source

Question 8

Monitoring an Active Learning project

Project Home dashboard - What does it help you understand?

Answer 8

Shows:

• How many documents have been coded in your project.
• How many documents have been coded outside the queue (manually-selected documents).
• How many documents have been skipped.

Question 9

Monitoring an Active Learning project

Update ranks - What is it?

Answer 9

Active Learning ranks all documents from 100 to 0.
This gets updated with every model build. These values are only stored in the Document object on demand.

A project administrator can choose to update these ranks on demand.

Question 10

Monitoring an Active Learning project

Prioritized review progress - What is it?

Answer 10

The Prioritized Review Progress chart tracks the ability of the model to find relevant documents over time. It does this by monitoring the reviewers’ coding decisions on the high-ranking documents chosen by the Prioritized Review queue

Question 11

Monitoring an Active Learning project
Prioritized review progress

Should the Prioritized Review Progress relevance rate increase or decline over time?

Answer 11

As a general trend, you should expect to see the relevance rate decline over the course of the review. However, you may see a spike in the relevance rate if a large amount of new documents are added to the project, or if the definition of relevance changes during the course of the review.

Question 12

Monitoring an Active Learning project
Quality checks and checking for conflicts

Why Would you do this?

Answer 12

We recommend running ongoing quality checks over the course of the project. The Active Learning process is fairly tolerant of some inaccurate or inconsistent coding, but it’s good practice to monitor your queue for conflicts between reviewers and the Active Learning model.

Question 13

Monitoring an Active Learning project
Quality checks and checking for conflicts

How is this best achieved?

Answer 13

With Dashboards. You can create dashboards and widgets around useful fields such as

CSR -  Cat. Set::Category Rank
Categories -  Cat. Set 
 Reviewers::User
\:: Prioritized Review
\:: Coverage Review
\:: Project Validation

Question 14

Running Prioritized Review

What is it?

Answer 14

serves documents that are most likely to be coded on the positive choice (such as Relevant) with a small set of documents included for index health. The documents included for index health are selected by the system to give the model a broader range of training documents.

Question 15

Running Prioritized Review

What Documents are served in Prioritized Review?

Answer 15

A mixture:

• 10% random
• 20% scores “in the middle” (40-60%)
• 70% high ranking uncoded documents

Question 16

Running Prioritized Review

What happens if you start a Prioritized Review Queue before a model occurs?

Answer 16

the queue initially serves up random documents because no scores are available for documents until a model build has completed.

Question 17

Running Prioritized Review

After a review has started you want to edit the settings to include family. What happens?

Answer 17

You can’t edit this setting once Start Review has been clicked.

You can only enable for new projects or Prioritized Review queues that have never been started

Question 18

Running Prioritized Review

What is the maximum amount of Concurrent Reviewers recommended per project?

Answer 18

150 concurrent

Question 19

Active Learning Set Up

How many choices can the Review Field have

Answer 19

This must always be exactly two choices.

One is positive and one is negative

Question 20

Running Prioritized Review

How do you add or remove documents from the Active Learning project after review has begun?

Answer 20

• Pause Review
• Update the Saved Search that is used for the classification index
• Navigate to your classification index and Run it (Incremental)
• -Start Review

Question 21

Running Prioritized Review

Turning off index health documents - Why would you do this?

Answer 21

For large projects and projects with high richness, the Active Learning engine may stabilize and identify consistently relevant documents long before the project has finished. To speed up the later part of these projects, you can exclude index health documents from appearing in the review queue.

Question 22

Running Prioritized Review

How do you change the index health documents setting?

Answer 22

• Navigate to Project Home Tab
• Click Setting Gear in top right
• Click Edit button
• Change Include Index Health in Prioritized Review to No
• Save - Settings take effect immediately)

Question 23

Project Validation and Elusion Testing

What is it?

Answer 23

used to validate the accuracy of an Active Learning project. The goal of Project Validation is to estimate the accuracy and completeness of your relevant document set if you were to stop the project immediately and not produce any unreviewed documents below the rank cutoff.

Question 24

Project Validation and Elusion Testing

When should you run a Project Validation?

Answer 24

Relativity recommend running Project Validation when you believe the project has stabilized and the low-ranking documents have an acceptably low relevance rate. However, you can run Project Validation at any point after the first model build.

Question 25

Project Validation and Elusion Testing

How is it done?

Answer 25

Reviewers review a sample of documents and compare that to the AL Project documents to determine how likely it is that many useful documents are yet to be reviewed.

Question 26

Project Validation and Elusion Testing

What are the two validation types?

Answer 26

Elusion with Recall - samples all documents, regardless of rank or coding status, and calculates elusion rate, richness, recall, and precision. This has the following advantages:

Elusion Only - samples uncoded documents below the rank cutoff and calculates only the elusion rate

Question 27

Project Validation and Elusion Testing

How does “Elusion with Recall” compare with “Elusion Only”?

Answer 27

Elusion Recall has advantages because is takes a sample for all documents regardless of rank of coding status. Therefore gives a more accurate picture.

If the review is in late stages the it makes sense to do Elusion Recall as most documents have already been reviewed.

If early stages of review, Elusion Only may be better because it will be less docs to review and faster. But less accurate.

Question 28

Project Validation and Elusion Testing

What information will Project Validation/Elusion Tests give you?

Answer 28

Elusion Rate - Docs coded relevant in the uncoded portion of the sample
Eluded Documents - calculated by multiplying the sample elusion rate by the number of documents in the discard pile.
Richness - the percentage of relevant documents across the whole sample.
Recall - the percentage of truly positive documents which were found by the Active Learning process. A document has been “found” if it was previously coded positive, or if it is uncoded with a rank at or above the cutoff.
Precision - the percentage of found documents which are truly positive.

Question 29

Project Validation and Elusion Testing

If Project validation results are not acceptable what should you do?

Answer 29

Click Resume Project, then click again to re-open the Active Learning project. This unlocks the model and allows it to rebuild. Any documents coded since Project Validation began, including those from the Project Validation queue itself, are included in the model build.

Question 30

Active Learning - Search to return skipped documents

How do you do this?

Answer 30

To identify documents that were skipped in an active learning review queue, set search conditions to is not set AND Reviewers is set.

Question 31

Active Learning to QC previously coded data

What is this?

Answer 31

You can use Active Learning to QC human coded data.

Set up the AL project and then mass update AL the Responsiveness Choices for a few thousand docs.

Do a sample and mass code over the sample

Go to the project and update the ranks as this will publish the data over the AL fields.

Search for documents coded in the reviewer field as Responsive and Classified by the system as Not Responsive.