After the client reviews a critical mass of documents (typically 500-1000), each document receives a score for each tag, such as relevance, hotness, privilege, or issue-based (e.g., an RFP specification). The score for a given tag falls on either of two scales:
- a scale of zero through one, with the score representing the probability that the tag applies to that document, or
- a user-defined scale, commonly one through five, representing gradations of relevance to the tag.
Dagger provides a comprehensive accuracy report including graphs, tables, learning curves, and trade-offs between accuracy and additional documents to review. The client can improve the accuracy measurements by reviewing documents identified by Dagger that will best train the model using continuous active learning, random and stratified sampling, and other techniques.
Eventually, the model will meet the client’s accuracy demands or accuracy will cease to improve, and the review effort can shift from training the model to reviewing the documents associated with each tag.
Reference: Predictive Coding