Skip to content

Better email labels. More useful agents

Evals for email classification

Adam Gent

Intro

Emails are a mess.

Important messages shouldn’t get lost in the noise.

But in my inbox, sometimes they do. It obviously doesn't need to be this way, imagine an inbox organized around what needs your attention, so you can focus on what matters. That clarity makes it easier to get work done and gives agents a starting point to do that work for you.

You might say, "but Adam Gmail already classifies my emails, and it does a pretty good job." I hadn't used Gmail's classifiers before, so I added them to my inbox. purchases puts all the receipts into one folder, which is great for reviewing expenses. important was applied to 48 out of the 50 most recent emails in my inbox. That's not very helpful, and I can assure you that the email with the subject line "Last chance: A quick 2-minute survey, ADAM" is definitely not important. Social hadn't been applied for the last 3 months, which felt a bit insulting. I couldn't distinguish clear boundaries from the remaining labels, Updates, Promotions, Forums so I didn't find those very helpful either. So I don't think Gmail's classification system is making it much neater.

What makes it so hard? First, the data types vary a lot. Some are written with HTML, some are plaintext with threads that go back months, each reply quoting everything below it. Inline images and other attachments can dwarf the email's text, and base64 encoding adds roughly another third to their binary size. A dozen people can be cc'd on an email, responding to different parts of a thread. Then a surprising amount of the email is actually just metadata concerning security and deliverability. So before you even do any classification, you need to pull out the small amount of data relevant to classification.

Second, if you could wave a magic wand and every email could get accurately classified in any way you wanted, is it clear what those labels would even be? Let me know if it is, and we can build that for you. The point is that coming up with a taxonomy that's helpful is part of the challenge, and this must be done before building a model to classify emails into that taxonomy. Further, even if we found a taxonomy that works for you, those labels probably wouldn't work for me. We get different types of emails, and our workflows around them differ.

But if we can get accurate classifications that are specific to each user's inbox, that unlocks a couple of really cool things.

  • The user can focus on emails that actually need their attention.
  • We can dispatch agents based on classification.
    • Scheduling: “Check my calendar and suggest times I'm free.”
    • Feature request: “Understand the user's need, check for existing solutions, and draft a proposal for the team to review.”
    • Support email: “Send a response based on our knowledge base, or escalate to upstream support.”

Approach

Find the taxonomy. Then tune the prompts.

Classification is a classic data science problem, but in the age of LLMs it looks a bit different than the methods I learned about in grad school. The idea is that instead of 'training a model' like you would a logistic regression model, you are choosing a model[1] and 'training' it by iterating on a prompt until you get the results you want.

The iteration loop looks like:

  • hand label a training set
  • build your prompt
  • model makes predictions
  • analyze disagreements and tag common reasons
  • iterate on the prompt based on those reasons, and run the whole thing back [2]

Once you’re happy with training performance, evaluate on a held-out, hand-labeled test set to check whether your prompt generalizes. See results

Deciding on a taxonomy is a significant part of this build, so I allowed it to emerge organically as I hand labeled lots of emails. For the first set, I tried to have clear boundaries between bins, instead of the amorphous Promotions, Forums, FYI.

The starting taxonomy
Sender type
HumanAutomated
Audience
BroadcastTargeted
Requires
ActionNothing
Relevance
RelevantNot relevantSpam

After hand labeling about 50 emails, the most common failure type was 'relevance is subjective'. I found Relevance difficult to classify even in my own inbox. Is a product update relevant? A newsletter from your alma mater? Google updating their TOS? An email from a recruiter? Google's answer is that user behavior can tell you whether an email is important, and I think I agree with that for relevance.

I realized that one thing we really cared about was whether a human you know actually sent you the email. Human + Targeted became Correspondence. Targeted + Automated became Notification. For Broadcast, it doesn't matter whether a human or a bot wrote it. Combining sender type and audience gave us Type, alongside the other bins in the final taxonomy below.

Final taxonomy
Click a row to see its prompt
TypeBroadcastsNotificationsCorrespondence
Requires actionYesNo
SchedulingYesNo
Cold outreachYesNo
CC’d for visibilityYesNo

These are the saved prompt templates. Owner names, email addresses, domains, and sender-history counts are substituted before the model sees them. Conditional hints appear only when their conditions are met.

Label a few emails.

Here’s a toy version of the email labeling tool I built.

EVALSDemo collection
Alex Morgan · synthetic mailbox
Label emails

Results

Where the labels agree—and where they don’t.

I hand-labeled nearly 200 emails received in one day in a single user’s inbox, then randomly split them into training and test sets. I tuned the prompts and classification cutoffs on the training set, then evaluated on the 97-email held-out test set. The results below are from that test set.

Select a classifier to explore its errors, or change a binary classifier’s cutoff using its saved probabilities.

ClassifierAccuracyPrecisionRecallCoverage

Type

Original predictions
Reference ↓ · Predicted →
AgreementDifferent predictionNo prediction

Precision / recall across cutoffs
— Precision— Recall

Type precision and recall measure Correspondence. Accuracy covers all three types; there is no binary Yes cutoff.

Among the 26 hand-labeled Correspondence emails, Cold outreach has 100% precision and recall, but 85% accuracy and 22/26 coverage. Four emails were skipped because Type classified them as Broadcast or Notification. All four were No examples, so they reduce accuracy and coverage without affecting Yes-class precision or recall. These are upstream gating errors, not failed model calls.

Methods

Turn the email into evidence.

The prompt is only half the input. Our parser turns the archived email into structured evidence: the message, its conversation, calendar details, and delivery signals. We add sender history from the archive, using only information available when the email arrived.

From email to model inputSynthetic example · actual parser output

The raw email data


    

What the model receives


    

Kept as evidence

Message text, links, recipient roles, quoted conversation, signature, footer, and selected spam/bulk signals. The calendar supplies the meeting time and Alex’s RSVP status. Sender counts come from archive history, not the email body.

Left out or not read

MIME boundaries, transfer encodings, routing and authentication headers, HTML layout, scripts, styles, hidden text, and URL queries/fragments. Replies received later are excluded. The PDF’s name, type, and size are included, but its contents are not read.

View the actual parser code (Go)

This example was generated with emailfeatures.Build and the v9 feature configuration. These are the source files used, with helpers in the same packages. The browser displays saved parser output; it does not reimplement the Go parser.

Evidence and request configuration
  • Model: gpt-6-luna, Decisions API; separate classification requests.
  • Owner: mailbox email address.
  • Target email: sender, subject, arrival time, To/CC recipients, and the owner’s recipient role.
  • Structured body: email text with quoted conversations represented separately.
  • Earlier context: up to three earlier archived messages—no later messages.
  • Sender history: counts of received emails and emails the owner sent to that sender, before this email arrived.
  • Delivery signals: Reply-To, reply relationships, unsubscribe signals, and selected spam/bulk indicators.
  • Calendar: available invitation/ICS metadata, including response status.
  • Attachments and images: metadata and availability; this does not mean PDF/document contents were read.
  • Coverage: missing content and truncation indicators, so unavailable isn’t mistaken for absent.
  • Downstream classifiers receive the relevant upstream predictions.
  • Dynamic variables: owner name/email, owner/sender domains, sender address, and historical sent/received counts are substituted before calling the model.
  • Dependencies use predictions from that same email, never hand labels.
  • No general introductory/system prompt.

Discussion and next steps

I ended up writing the prompts by hand. The LLM-written versions tended to be longer, less direct, and more confusing.

Class imbalance remains a limitation. The rarer labels have relatively few examples, but hand-labeling more than 200 emails takes more time than I can justify right now.

One of the reasons I added CC’d for visibility is that the model has a hard time recognizing who action is required from. The most common reason for misclassifying these emails was that action was required, but not from the user.

The next step is moving from Requires action to what action is required, so we can dispatch agents to start doing the work.

Want these labels in your inbox?

Sign up for Housecat and turn on the classifier to get these classifications right in your Gmail or Outlook mailbox.

Notes

References and context.

[1] OpenAI Decisions: GPT‑6 Luna in our case. The Decisions API isn’t a chat endpoint — you give it a question, a fixed list of allowed answers, and context, and it returns a choice with probabilities instead of generated text. It is purpose-built for classification.

[2] The error analysis & eval strategy is based mostly on the work of Hamel Husain.