A Login Recorded Under the Wrong User: The Ghost Left in ThreadLocal

Let Me Start with a Confession

For a while, our system was recording a certain login under someone else’s name. B would log in, and the record would say A. Let me be clear up front: this defect is a ‘past case’ whose cause has been found and fixed. So this post isn’t a vulnerability disclosure but an autopsy report, a postmortem that records how the incident unfolded, what caused it, and how we’re keeping it from happening again. The series has SQL in its name, but this time the culprit isn’t SQL. Still, if what this series has been chasing all along is ‘data and records,’ no story belongs here more than an incident where the record itself lied.

The last post ended with this question: “When the next culprit shows up, can you present eyewitness testimony instead of circumstantial evidence?” After that case, where we never found out whose hands had produced six-year-old data, construction began on putting that witness in place in the system I help run (a regulated-industry system that tracks agricultural production history, where the record of ‘who did what’ carries unusual weight). The witness is an audit log: a record of who changed what, and when.

How we built the audit log is a story for another post. Today’s story is about something that happened mid-construction. The witness gave its first testimony before it was even finished, and that testimony pointed at us.

---

One Strange Record

While we were wiring up the audit log, one strange record caught my eye. The actor on a login record was wrong. The user who actually logged in was B, but the record said A.

The first thing I suspected was the audit log code I had just written. That’s the natural order: the newest code in a system is always the prime suspect. If the record is wrong, common sense says the recorder is broken. But the recorder was innocent. It was writing down exactly what it was given. The problem was ‘what it was given.’

The audit log doesn’t make up the actor’s name; it pulls it from the SecurityContext. The SecurityContext is the box where Spring Security keeps ‘who the user of this request is.’ So the record wasn’t wrong. The wrong person was in the box.

But nowhere in B’s login request is there any code that would put A in the box. Nobody put it there, yet the value was there. There’s only one possible answer: ‘Something that was already in there had been left behind.’ Already in there? Where, exactly?

---

We Filled It, but Never Emptied It

This system authenticates with JWT tokens. A filter validates the token carried in the request header and, if it passes, puts the user’s information into the SecurityContext. Here’s a conceptual example that carries over only the structure, not the actual code.

// Conceptual example: the structure that caused the problem
public class JwtFilter extends OncePerRequestFilter {
    @Override
    protected void doFilterInternal(HttpServletRequest request,
            HttpServletResponse response, FilterChain chain)
            throws ServletException, IOException {
        String token = resolveToken(request);
        if (token != null && validate(token)) {
            Authentication auth = buildAuthentication(token);
            SecurityContextHolder.getContext().setAuthentication(auth); // fill the box
        }
        chain.doFilter(request, response);
        // ...and then do nothing. There is no code that empties it.
    }
}

Our filter looked just like this. It never called SecurityContextHolder.clearContext() when the request ended. We filled the box but never emptied it.

“What’s wrong with not emptying it? When the request ends, the thread goes away with it.” I believed that for a long time too. The second half of that sentence being wrong is the whole story of this incident.

---

Threads Don’t Clock Out

Servlet containers like Tomcat don’t create a new thread for every request. They take one from a set of threads created in advance, the thread pool, and hand it the request. When the request is done, the thread isn’t destroyed. It goes back to the pool to be reused for the next request. It’s the standard strategy every server uses to save the cost of creating threads.

And the place where that box is kept is ThreadLocal. ThreadLocal lets each thread hold its own value for the same variable. Think of it as the ‘in-room safe’ that comes with every hotel room. This is where Spring Security’s convenience comes from: at any layer during request processing, ask “Who’s the current user?” and you get an answer. You just open the safe of the thread that’s doing the work right now. The whole API fits in three lines.

// Conceptual example: all there is to ThreadLocal
static ThreadLocal<String> safe = new ThreadLocal<>();

safe.set("ID of guest A"); // goes only into the current thread's safe
safe.get();    // opened by the same thread: the ID of A; by any other thread: null
safe.remove(); // empty the safe: this one line is the star of this whole post

Put those two facts together and a ghost is born. The safe belongs to the thread, and the thread doesn’t die when the request ends. So the contents of the safe don’t die either.

In hotel terms: the room (the thread) stays put even as the guests (the requests) change. Emptying the in-room safe (ThreadLocal) at checkout is housekeeping’s job (clearContext), but our hotel had no housekeeping. B checked into the room A had just left, and A’s ID was still sitting in the safe.

How A Ended Up on B’s Login

Now we can reconstruct the incident.

1. User A’s request is handled on thread T. The JWT filter fills T’s safe with A’s authentication. The request ends and T goes back to the pool, with the safe left as is.
2. User B sends a login request. A login request doesn’t carry a token yet, because it’s the request that goes to ‘get’ one. So the filter lets it pass without refilling the safe.
3. As luck would have it, this request is assigned to thread T. If anything during processing asks “Who’s the current user?”, the safe answers: A.
4. And that’s how B’s login gets recorded under A’s name.

The key is ‘a request for which the filter doesn’t fill the safe.’ Records created by requests like that can pick up the previous guest. This picture also explains why the ghost was spotted in login records, of all places. On an ordinary request that carries a token, the filter overwrites the safe with the real user every time, so even if a ghost was lingering, its traces get wiped first. The ghost only shows itself on requests that arrive without a token, the login itself being the classic example. On top of that, B has to happen to land on the very thread A was using, so with dozens of threads in the pool, it’s a game of chance. Most of the time it won’t reproduce, and then once in a while, someone else’s name gets stamped on a record. That’s why lingering-state bugs like this one live so long.

Tighten the conditions, and a game of chance turns into deterministic reproduction. I built a reproduction server locally and cut it down to a single thread with server.tomcat.threads.max=1: with only one thread, every request uses the same safe (for the reproduction, I simplified token validation to reading a username from a header). The first request carried the user alice, and the second, with no auth header, asked ‘Who am I right now?’ The answer: user=alice. Both responses print the same thread name, http-nio-...-exec-1, so the output even records that the same room was reused. Run the same experiment against the fixed version and the second response is user=(no authentication). The reproduction code is in the demo folder of my study notes repository.

The Fix: Writing Housekeeping into the Contract

Compared to finding the cause, the fix was almost anticlimactically short. I changed the filter to clear the context when the request ends.

// Conceptual example: the structure after the fix
try {
    // ...validate the token, fill the context...
    chain.doFilter(request, response);
} finally {
    SecurityContextHolder.clearContext(); // empty it no matter what
}

The finally is what matters. Whether or not an exception is thrown during processing, the safe has to be emptied before the thread goes back to the pool.

I’m not entirely without excuses. Spring Security’s standard filter chain includes a mechanism that cleans up the context when a request ends, so in a textbook setup you’ll rarely run into this problem. But the moment you assemble the authentication path yourself with a custom filter, the ‘responsibility to empty’ moves from the framework’s conventions to your own code. And the code that fills always gets written, because without it the feature visibly breaks. The code that empties can be missing and nothing happens right away. It just quietly turns into a ghost on top of the thread pool. Filling code and emptying code come as a set, and we were honoring only the filling half of that contract.

---

The Twist: The Witness’s First Contribution

The more I chew on it, the stranger this incident gets. This defect predates the audit log. Even before construction, the filter wasn’t clearing the context, and the ghost was probably there back then too. Nobody saw it, that’s all. In a system with no record of ‘who,’ a bug that gets ‘who’ wrong has no symptoms.

The audit log is a tool built to record what happens from now on. Yet before it was even finished, it became a tool for ‘discovering’ a defect that already existed. We put a witness in place, and the very first thing it testified about was how well we’d locked up our own house.

Honestly, incidents like this make you want to hide them, all the more once the word ‘security’ gets attached. But as this series has kept confirming (documenting the tradeoffs of a performance fix, writing down that a refactoring was deferred when you defer it), an incident that isn’t recorded never becomes the team’s knowledge, and an incident that never becomes knowledge happens again. Next time, without a witness. If a team building an audit log to record who did what and when can’t record its own incidents, that log is only half finished. This post is that record.

Recurrence Prevention Checklist

Here’s the list we left behind to keep the same mistake from happening again.

1. When you see code that fills a ThreadLocal, look for the matching code that empties it. If there isn’t any, that’s the bug. Empty it in try/finally: the thread goes back to the pool even when an exception is thrown.
2. Distinguish ‘the end of the request’ from ‘the end of the thread.’ On a thread pool, these are two different events, and state lives in the gap between them.
3. If you use a custom authentication filter, check who is responsible for cleanup. Code that fills the context directly, outside the standard filter chain, has to clean it up directly too.
4. Be suspicious of code that hands work off to another thread, too. @Async, schedulers, a separate ExecutorService: Tomcat isn’t the only place with a thread pool. ThreadLocal follows the thread, not the logical flow. If you’ve handed work off to another pool, you also have to ask who cleans out the safes in that pool.
5. Don’t write off a strange log entry as a ‘logging bug’ and delete it. Trace where the value came from before blaming the recorder: an odd record can be a signal, not noise.
6. If you suspect leftover state, test with the thread pool cut down to one. The game of chance turns into deterministic reproduction. Lingering-state bugs normally hide behind the thread count.

---

Wrapping Up: Ghosts Only Appear in Front of a Record

ThreadLocal and thread pools were tools I worked on top of every day without ever reading their contract. Since this incident, whenever I see state attached to a thread, I ask two questions: who fills it, and who empties it? If you try just one thing today, search the codebase you work on for filters that touch the SecurityContext directly and for code that calls set on a ThreadLocal, and check whether a remove() or clearContext() stands next to each one as its pair.

And the bigger conviction this incident left me with is about records themselves. In the last case, we had to give up on the culprit’s history because there was no record. This time, while putting a record in place, we caught a defect nobody could see. I thought records were something you build up for future incidents, but from the moment you put them in place, they shine a light on the present. What about your system? How many ghosts are walking around right now, invisible only because nothing records them? The construction work to put the witness in place isn’t finished yet, so this probably won’t be the last story I pick up on the construction site.