Scanning agent transcripts for secrets, without sending them anywhere

You paste a .env into a Claude Code session to debug a failing deploy. The agent reads it, fixes the config, and moves on. Twenty minutes later the task is done and you close the terminal.

The secret is still there. Not in the terminal, in the transcript. Every agent session writes a JSONL file to disk, and that file now holds your production database URL in clear text, sitting in a directory you will never think about again.

Multiply that by a few hundred sessions. That is the actual state of most machines running coding agents today.

When scanning is synonymous with leaking

The obvious fix is to scan those transcripts and mask what you find. The obvious way to scan them is to send them to something that is good at finding secrets, which in 2026 means a cloud deployed model.

Yeah that's right. To find out whether your transcripts contain credentials, you would upload all of your transcripts, including the ones that contain credentials, to a third party. The scan itself becomes the leak. We were not comfortable with this trade, so the whole PII feature had to run locally or not exist at all.

Locally means a model small enough to ship, fast enough to feel instant, and accurate enough to be worth turning on. Those three pull against each other, and most of what follows describes the places where they pull the hardest.

Precise and blind, contextual and sloppy

Our first instinct was regex, and it works better than people expect. An AWS access key has a shape. So does a JWT, an IBAN, a card number. A regex with a checksum validator behind it will find those with precision a neural network struggles to match, and it costs microseconds.

Then you hit the things that have no shape at all.

my colleague Sarah handles the billing, she's at 14 Rue de Rivoli
the staging password is the same as my dog's name plus 1999

No pattern matches those. A name is only a name in context. An address is only an address because of the words around it. And "the staging password is" is a phrase, not a format.

So we run both. The two approaches fail in opposite directions, which means the union of them is strictly better than either one. The model finds names, addresses, and password-phrased secrets that no expression will ever catch. The regex finds bare structured tokens that the model walks straight past.

A composite detector runs every ready backend and merges the results. Regex is cheap and always ready, so masking works from the first launch.

Embedding a 1.6 GB model in a menu bar app

We integrated OpenAI's privacy filter, quantized to int8 and exported to ONNX, which is run through ONNX Runtime. Weights plus tokenizer come to a little over 1.6 GB on disk.

That number is a problem for an app whose entire promise is that it sits quietly in your menu bar and does not bother your machine. A resident 1.6 GB is not quiet.

We made two decisions to handle this.

First, nobody downloads it unless they ask for it. The feature is off by default and the weights arrive only when you turn it on, so a user who never enables PII detection carries none of it. Each artifact is pinned by SHA-256 against a pinned model revision, and a hash mismatch refuses the file rather than loading it. Hashing 1.6 GB takes a few seconds, which is long enough that the UI needs a distinct verifying state or it looks frozen at 100%.

Second, the inference session has to be disposable.

timer.setEventHandler { [weak self] in
    guard let self else { return }
    self.lock.lock(); defer { self.lock.unlock() }
    if self.session != nil, Date().timeIntervalSince(self.lastUsed) >= self.idleTimeout {
        self.session = nil  // release ~1.6 GB; ORTEnv stays (cheap)
    }
}

After five minutes with no scans, the ORT session gets torn down, and the next scan spends about a second rebuilding it. That fits how the feature actually gets used. You scan a transcript, read through what it flagged, and then don't touch it again for an hour. Paying a one second rebuild every now and then beats keeping 1.6 GB pinned in memory all day for an app that lives in your menu bar.

When the labels don't add up

Once the model is running, the remaining problems are all about its output. It emits 33 classes per token in a BIOES scheme. In plain terms, for every token the model answers one question. Is this the start of a piece of PII, the middle of one, the end of one, a complete one on its own, or not part of one at all?

Take the argmax at each position independently and you get sequences that cannot mean anything. A begin-email followed directly by an outside. An inside-address with no begin before it. The labels are individually most likely and collectively nonsense, and every one of those is a span you cannot mask because you do not know where it ends.

So the decode is a constrained Viterbi pass. Illegal transitions are not penalized, they are unrepresentable.

case (.begin, .inside), (.inside, .inside):
    return from.category == to.category ? biases.insideToContinue : nil
case (.begin, .end), (.inside, .end):
    return from.category == to.category ? biases.insideToEnd : nil
default:
    return nil  // any other transition is illegal
}

Returning nil removes the edge from the lattice entirely. The best path is then the best path through legal sequences only, and the decoded output is always a set of complete, well formed spans. The decoder is pure and deterministic, which means it unit tests from synthetic logits with no model loaded, and that has caught more bugs than any other single decision in this pipeline.

What it found, not where it is

Those spans have one problem left. They live in token space, and masking happens on the original string. To mask a secret you need its exact character range in that string. Off by one and you either leak a character of the credential or mask a character of the surrounding text.

The tokenizer we use returns token ids and token surfaces. It does not return offsets back into the source. That is a real gap, and the usual answer is to build a lookup table mapping tokens to byte positions, which is fiddly and easy to get subtly wrong around multibyte characters.

There is a much better answer hiding in how byte-level BPE works. The surface form of a byte-level token is drawn from a 256 symbol alphabet where each symbol stands for exactly one source byte. So a token's length in source bytes is just the number of unicode scalars in its surface. No table, no mapping, no special cases.

var pos = 0
for surface in surfaces {
    let n = surface.unicodeScalars.count  // == source UTF-8 byte count
    ranges.append(pos..<(pos + n))
    pos += n
}

Cumulate those and you have exact byte ranges for every token, which convert to String.Index ranges by snapping to scalar boundaries. We verified it byte for byte against the reference tokenizer across ASCII, accented Latin, CJK, emoji, and code samples, because a claim like this can hold for English and quietly break on a Japanese variable name.

The problem at the seams

The model has a window. Real transcripts are much longer than it, so long inputs are processed as overlapping windows, 4096 tokens with 256 tokens of overlap on each side.

Overlapping raises a new question. If a credit card number happens to straddle the boundary between two windows, which window owns it?

The tempting answer is to only trust spans fully inside a window's core region. It is also the wrong one, because a span straddling a seam is then partly outside both cores, gets rejected twice, and renders in clear text. The bug is invisible in testing unless a secret lands exactly on a boundary.

We assign each span to exactly one window, by its starting token. The trailing overlap is still there and still gives the model context to read the entity correctly, but ownership is unambiguous and the trusted ranges partition the sequence cleanly. A span that begins in a window belongs to that window, whole, however far past the boundary it runs.

Over-mask, never under-mask

Two spans overlap. Maybe the regex and the model both flagged an email with slightly different boundaries. You have to produce one span, so you pick one and drop the other.

That is wrong, and it is wrong in this specific way, If the span you dropped extended past the one you kept, the tail you discarded renders in clear text. You have not resolved a conflict, you have created a leak out of two correct detections.

So overlaps are always unioned, never dropped. The merged span covers everything either detector flagged and carries the label of whichever was more confident.

We ended up leaning on this rule in three different places in the pipeline, and it holds because the two mistakes are nowhere near the same size. Mask two extra characters of a sentence and a careful reader might notice and shrug. Leave four characters of an API key on screen and we've failed at the reason this feature exists. Every time we had to pick between the two, that comparison made the choice for us.

Most privacy engineering comes down to noticing which of your errors are recoverable.

Where this runs

All of this ships in Actvt, a macOS menu bar app that watches your machine and the coding agents on it. The PII detection is one part of it, off by default, and everything above happens on your Mac with no network call in the path.

We wrote it this way because the alternative was asking people to upload their agent transcripts to find out whether their agent transcripts contained anything they would not want uploaded. Some problems answer themselves.

If you are running agents at any scale and have thought about this differently, we would genuinely like to hear it. The failure modes here are subtle, and we would rather find ours in a conversation than in an incident.

Oye Collective builds production AI agents inside real businesses, and Actvt gives you observability into your own. Reach us at oyecollective.com.