Classifying Email

Once we have a fast cache of email data, we have a great test bed for classifying our email.

There are many ways to approach this problem: LLMs, embeddings, and classification models (e.g., Typesafe.ai’s Jev).

Here we survey these approaches and compare the pros and cons of these techniques.

History of Email Classification

We’ve been classifying email since the beginning of the protocol way back in the early ’70s. A lot of research in machine learning was rooted in email, including the now ubiquitous concept of “spam” vs. “ham”. Two key papers from modern email:

• A Plan for Spam - Paul Graham, Aug 2002
• The Learning Behind Gmail Priority Inbox - Douglas Aberdeen, Ondrej Pacovsky, Andrew Slater, 2010

Today Gmail automatically classifies:

• Important
• Categories
  • Social
  • Updates
  • Forums
  • Promotions
• Spam

So how can we classify email if we don’t use Gmail, or find Gmail lacking in its understanding of our unique communications patterns? Here we survey the main techniques.

LLMs

The obvious place to start a lot of work these days is to open Claude Desktop as an interface to Anthropic’s LLMs and ask it to help:

• List my most important emails
• Find and label cold inbound emails

When connected to Gmail tools and an artifact store, Claude will make progress towards these goals and produce a nice CSV, spreadsheet, or HTML dashboard with initial results.

Unfortunately, some major “clanker” problems appear as it:

• Starts from a fresh context
• Gets incomplete data sets due to fetching data on demand
• Turns slowly to process and classify data
• Requires additional imprecise chat to correct or steer

It’s also very expensive if you dig into the token usage and related API cost.

Coding Agents

The next obvious approach to try is to stick “Agentic Coding Harnesses” like Claude Cowork or the claude terminal UI on the task, with additional powerful capabilities:

• A sandbox virtual machine that can run code and persist data
• Tools to write and run code

In the hands of someone who can “agentic code” or “vibe code”, this prompt will return interesting results:

Fetch emails from the last 60 days, save them to disk, classify each one as “cold sales” yes / no / maybe with confidence and reasoning, then show the results in a data table UI.

Common challenges of agentic coding show up here. Without more specific prompting, agent skills, and/or code reviews, there is a lot of indeterminism in how it accomplishes the classification and everything around it, like coding language, storage schema and engine, and presentation.

Take special care to review and understand how its classifying the data – the model, prompt, data input format and data output format – to make sure it’s not cheating.

A coding toolkit like housecat-inc/scratch can lock in a few decisions:

• SQLite for storage
• claude -p for the classification pipeline
• HTML / Tailwind / HTMX for presentation

But this still requires a lot of tuning to steer both the classification pipeline and the app around it.

Embeddings

It’s important to remember that LLMs are one of many machine learning tools in our toolbox and often a very slow and expensive way to approach a lot of problems.

For classification and semantic search problems, “embeddings” can be better.

Embedding is a technique that maps complex high-dimensional data into a lower-dimensional vector space of numbers. For email, we can send all our message data to an embedding API to build a database and vector space to search and group emails by meaning.

Email triage with an embedding-based classifier by Adam Wiggins walks through building a classifier and evaluating its results and shows that embeddings do very well, both OpenAI’s text-embedding-3-small and self-hosted bge-m3.

Classification Models

Recently, Typesafe.ai’s Jev, a “System One” model purpose-built for returning structured classification data, launched to great interest, largely because of all the challenges and room for improvement with the other approaches above.

This model uses “Reinforcement Learning for Calibrated Decisions” (RLCD), an entirely different architecture from LLM “Reinforcement Learning with Human Feedback” (RLHF) and “Reinforcement Learning with Verifiable Rewards” (RLVR).

The shape of this model is very promising, returning consistent JSON responses with classification results and confidence scores significantly faster and cheaper than LLMs, and without building embeddings.

Our initial results show it is on par with or slightly better than “Codex Mini” at a fraction of the cost and time.

Further Research and Development

Like everything in ML and AI, there’s a lot more to research, build, evaluate and improve upon to “hill climb” up to accurate, fast and affordable email classification.

If you’d like to help, sign in to Housecat, get your new and improved “Important” inbox, and provide feedback on its results.