AI-Powered Evidence Classification: Step-by-Step Tutorial for Legal Teams (2026)

AI is rapidly transforming legal discovery, especially in the classification and triage of digital evidence. In this hands-on tutorial, you'll learn how to build and deploy an AI-powered evidence classification workflow tailored for legal teams. We'll cover everything from data preparation to model deployment, using modern open-source tools and best practices.

For a broader overview of automating legal discovery, see our PILLAR: The 2026 Guide to Implementing AI Workflow Automation for Legal Discovery—Risks, Vendors & Best Practices. This article dives deep into the practical steps for evidence classification—one of the most impactful subdomains of AI-assisted legal work.

This tutorial is designed for legal technologists, data scientists, and IT professionals supporting legal teams, as well as forward-thinking law firms looking to streamline their discovery process.

Prerequisites

• Python 3.10+ installed on your system (download)
• Pip (comes with Python 3.4+)
• Basic familiarity with Python (reading/writing scripts, using virtual environments)
• Command-line/terminal proficiency
• Jupyter Notebook (for interactive development)
• Sample labeled evidence dataset (emails, PDFs, or text files with classification labels)
• Git (version control, optional but recommended)
• Hugging Face Transformers library (v4.40+)
• scikit-learn (v1.4+)
• FastAPI (v0.110+) for deployment (optional, for API endpoint)

1. Set Up Your Development Environment

1. Create and activate a virtual environment:

   python3 -m venv venv
   source venv/bin/activate  # On Windows: venv\Scripts\activate
           

2. Install required Python packages:

   pip install jupyter numpy pandas scikit-learn transformers torch fastapi uvicorn
           

3. Verify installations:

   python -c "import transformers; import sklearn; import pandas; import torch; print('All good!')"
           

4. Start Jupyter Notebook (optional):

   jupyter notebook
           

   Screenshot description: The Jupyter Notebook dashboard opens in your browser, showing your project folder.

2. Prepare and Explore Your Evidence Dataset

1. Organize your data: Place your evidence files (e.g., emails, PDFs, text docs) in a folder. Create a CSV file (e.g., evidence_labels.csv) with columns like:

   filename,label
   email1.txt,privileged
   email2.txt,non-privileged
   ...
           

2. Load and preview the data in Python:

   
   import pandas as pd
   
   df = pd.read_csv('evidence_labels.csv')
   print(df.head())
           

   Screenshot description: Jupyter cell output shows the first few rows of your labeled evidence.
3. Read the evidence file content into the DataFrame:

   
   def read_file_content(filename):
       with open(f'evidence/{filename}', 'r', encoding='utf-8') as f:
           return f.read()
   
   df['content'] = df['filename'].apply(read_file_content)
   print(df[['filename', 'label', 'content']].head())
           

4. Check for class balance:

   
   print(df['label'].value_counts())
           

   Tip: If your classes are highly imbalanced, consider using stratified sampling or data augmentation.

3. Choose and Fine-Tune a Pre-trained AI Model

1. Select a pre-trained transformer model: For text evidence, distilbert-base-uncased is a good starting point.

   
   from transformers import AutoTokenizer, AutoModelForSequenceClassification
   
   model_name = "distilbert-base-uncased"
   tokenizer = AutoTokenizer.from_pretrained(model_name)
           

2. Preprocess and tokenize your evidence:

   
   def tokenize_function(examples):
       return tokenizer(examples['content'], truncation=True, padding='max_length', max_length=256)
   
   from sklearn.model_selection import train_test_split
   
   train_df, test_df = train_test_split(df, test_size=0.2, stratify=df['label'], random_state=42)
   
   from datasets import Dataset
   
   train_dataset = Dataset.from_pandas(train_df)
   test_dataset = Dataset.from_pandas(test_df)
   
   train_dataset = train_dataset.map(tokenize_function, batched=True)
   test_dataset = test_dataset.map(tokenize_function, batched=True)
           

   Screenshot description: Sample tokenized data in Jupyter output, showing input IDs and attention masks.
3. Set up the model for classification:

   
   from transformers import AutoModelForSequenceClassification
   
   num_labels = df['label'].nunique()
   model = AutoModelForSequenceClassification.from_pretrained(model_name, num_labels=num_labels)
           

4. Train the model:

   
   from transformers import TrainingArguments, Trainer
   
   label2id = {label: i for i, label in enumerate(df['label'].unique())}
   id2label = {i: label for label, i in label2id.items()}
   
   def encode_labels(example):
       example['label'] = label2id[example['label']]
       return example
   
   train_dataset = train_dataset.map(encode_labels)
   test_dataset = test_dataset.map(encode_labels)
   
   training_args = TrainingArguments(
       output_dir='./results',
       num_train_epochs=3,
       per_device_train_batch_size=8,
       per_device_eval_batch_size=8,
       evaluation_strategy="epoch",
       save_strategy="epoch",
       logging_dir='./logs',
       logging_steps=10,
   )
   
   trainer = Trainer(
       model=model,
       args=training_args,
       train_dataset=train_dataset,
       eval_dataset=test_dataset,
   )
   
   trainer.train()
           

   Screenshot description: Training progress logs with loss and accuracy metrics in the notebook or terminal.
5. Save your fine-tuned model:

   
   model.save_pretrained('./evidence_classifier')
   tokenizer.save_pretrained('./evidence_classifier')
           

4. Evaluate Model Performance

1. Make predictions on the test set:

   
   import numpy as np
   
   preds = trainer.predict(test_dataset)
   pred_labels = np.argmax(preds.predictions, axis=1)
           

2. Generate a classification report:

   
   from sklearn.metrics import classification_report
   
   print(classification_report(test_dataset['label'], pred_labels, target_names=list(label2id.keys())))
           

   Screenshot description: Precision, recall, and F1-score for each evidence class.
3. Review misclassified examples for model improvement:

   
   misclassified_idxs = np.where(pred_labels != test_dataset['label'])[0]
   for idx in misclassified_idxs[:5]:
       print("Content:", test_dataset[idx]['content'])
       print("True:", id2label[test_dataset[idx]['label']], "Predicted:", id2label[pred_labels[idx]])
       print("---")
           

5. Deploy the Classifier as an API Endpoint

1. Create a FastAPI server:

   
   
   from fastapi import FastAPI
   from pydantic import BaseModel
   from transformers import AutoTokenizer, AutoModelForSequenceClassification
   import torch
   
   app = FastAPI()
   
   tokenizer = AutoTokenizer.from_pretrained('./evidence_classifier')
   model = AutoModelForSequenceClassification.from_pretrained('./evidence_classifier')
   model.eval()
   
   class EvidenceRequest(BaseModel):
       content: str
   
   @app.post("/classify")
   def classify_evidence(request: EvidenceRequest):
       inputs = tokenizer(request.content, return_tensors="pt", truncation=True, padding='max_length', max_length=256)
       with torch.no_grad():
           outputs = model(**inputs)
           pred = torch.argmax(outputs.logits, dim=1).item()
       return {"label": list(label2id.keys())[pred]}
           

2. Run the API server:

   uvicorn app:app --reload
           

   Screenshot description: Terminal output shows "Uvicorn running on http://127.0.0.1:8000".
3. Test your API with curl or Postman:

   curl -X POST "http://127.0.0.1:8000/classify" -H "Content-Type: application/json" -d '{"content": "This email contains privileged legal advice."}'
           

   Expected response:

   {"label":"privileged"}
           

6. Integrate with Legal Discovery Workflows

1. Batch classify new evidence: Write a script to send new evidence files to your API and store the results in a database or CSV.

   
   import requests
   import os
   
   api_url = "http://127.0.0.1:8000/classify"
   evidence_folder = "new_evidence"
   results = []
   
   for fname in os.listdir(evidence_folder):
       with open(os.path.join(evidence_folder, fname), 'r', encoding='utf-8') as f:
           content = f.read()
       response = requests.post(api_url, json={"content": content})
       label = response.json()['label']
       results.append({"filename": fname, "label": label})
   
   import pandas as pd
   pd.DataFrame(results).to_csv("classified_results.csv", index=False)
           

2. Connect to your eDiscovery or case management system: Use the API as a microservice within your broader legal workflow, or trigger classification as part of an automated pipeline.
   For more on automating legal workflows, see Automating Contract Review: 2026’s Best Tools for Legal Discovery Workflows and Top AI Workflow Automation Tools for Legal Teams in 2026: Review & Buyer’s Guide.

Common Issues & Troubleshooting

• Out-of-memory errors during training: Reduce per_device_train_batch_size, use shorter input lengths, or switch to a smaller model.
• Poor model accuracy: Check data labeling for errors, ensure balanced classes, or increase training epochs. Review misclassified examples for patterns.
• API server crashes on large inputs: Set a reasonable max_length for tokenization and validate input size.
• Dependency conflicts: Use a fresh virtual environment and check package versions.
• UnicodeDecodeError reading evidence files: Ensure files are UTF-8 encoded, or handle encoding errors in your read_file_content function.
• Class labels not matching between training and inference: Persist your label2id mapping and use it consistently in both training and deployment.

Next Steps

• Expand your evidence types: Adapt the pipeline for PDFs, images, or audio using OCR and multimodal models.
• Enhance compliance and auditability: Log all classifications and decisions for legal defensibility. For compliance risks, see Legal AI Workflow Automation: Key Compliance Pitfalls and How to Avoid Them in 2026.
• Incorporate human-in-the-loop review: Use your model for triage, with legal professionals reviewing edge cases or uncertain predictions.
• Monitor and retrain: Regularly update your model with new labeled data to maintain accuracy as evidence types evolve.
• Explore vendor solutions: If you need robust, production-ready solutions, review the landscape in our complete guide to AI workflow automation for legal discovery.

By following this tutorial, your legal team can dramatically accelerate the classification of digital evidence, reduce manual review time, and improve consistency. AI-powered workflows are rapidly becoming standard in forward-looking legal practices—see our AI-Powered Contract Review: Tools and Tactics for 2026 Legal Teams for further inspiration.