Why Your Search Tool Keeps Re-Scanning the Whole Drive (and What It Should Do Instead)

If a search tool is quietly eating your CPU and hammering your disk long after the first index finished, there is usually one design decision behind it: the tool rediscovers your files by walking every folder, over and over, on a timer.

It is worth understanding why that approach is so common before criticising it, because it is not stupidity. It is the correct-but-expensive answer.

Why full re-scanning is the default

An indexer has to answer one question repeatedly: what changed since last time? The bluntest way to answer it is to look at everything and compare. That approach has real virtues — it needs no stored state, it cannot drift out of sync, it does not depend on the operating system telling the truth, and it recovers by itself from any missed event. If you are building the first version of an indexer, it is the right thing to write.

The problem is the arithmetic. The cost of a full walk scales with how many files you own. The value of it scales with how many of them changed. On a working machine, those two numbers are wildly different: you may own hundreds of thousands of files and have touched a couple of dozen since lunch. So the overwhelming majority of every sweep is spent confirming that nothing happened.

And the cost is not only the walk itself. Every sweep re-reads directory metadata across the whole tree, which is exactly the access pattern that keeps a disk busy and a laptop warm. Users experience it as a machine that never quite settles down.

The trap of "just make the interval longer"

The obvious fix is to scan less often. It helps, but it trades one problem for another: a longer interval means a file you just saved might not be findable for a while, which undermines the entire point of having a search tool.

Worse, there is a subtler failure mode that a longer interval can hide rather than fix. If a system's idea of "idle" is measured against the same clock as its work cycle, and each cycle has a little work in it, then the idle condition may never actually be reached — the system stays permanently half-busy while looking, from the outside, like it should have gone quiet. We found precisely that in our own memory-release logic, and the fix was not a longer timer but a different signal: release when the work queue actually empties, which is an event a timer cannot miss.

That is the general shape of the lesson. When periodic work misbehaves, the answer is rarely a better interval. It is usually a better signal.

What the better signal is

Operating systems already know when files change — they have to, because everything from backup software to your file manager depends on it. Windows exposes this: a program can register interest in a folder tree and be told about creations, edits, renames and deletions as they happen.

Used well, that inverts the whole economy of the problem. Instead of asking "what changed?" by inspecting everything, the tool is told what changed, and does work proportional to the change rather than to the size of your drive. A file you save becomes searchable shortly after you save it, without anything having swept the disk to notice.

Change notifications are not a complete answer on their own, and any tool that claims otherwise is overselling. Notifications can be dropped under bursts, they behave differently on network and removable drives, and nothing arrives for changes made while your computer was off. So a periodic full walk still has a job — it is the safety net that catches what the fast path missed. The change is in its role: it goes from being the primary mechanism, run as often as possible, to being a backstop, run on a fixed and comfortably low cadence.

Keeping it under a ceiling

There is one more piece, and it matters more than it sounds. Even correct work is unwelcome if it takes the machine over.

Indexing in LocalSynapse runs under an explicit ceiling rather than opportunistically consuming whatever is available. Each cycle gets a bounded budget, and work that does not fit waits for the next one. That is deliberately slower in the best case, and the trade is worth it: the worst case is what people actually feel. A tool that indexes somewhat slower but never makes your machine unpleasant is a tool you keep installed.

The same principle governs memory. The semantic model is large, and holding it forever so that the occasional search is instant is a bad bargain on a laptop. It is now released as soon as there is nothing left to index, and reloaded when needed — with the app saying so on screen while that happens, because a pause you understand is a very different experience from a pause you do not.

What to expect from LocalSynapse now

In practice: changes you make are picked up from the filesystem's own notifications rather than waiting for the next sweep; a full walk still happens on a fixed low cadence to catch anything missed, including everything that changed while the machine was off; indexing work stays under a ceiling instead of expanding to fill the machine; and the memory used by semantic search is handed back when the queue drains rather than held indefinitely.

If you are evaluating any local search tool, this is a fair thing to ask about directly — not "is it fast?" but "what makes it decide to do work?" A tool that can only answer "a timer" will keep finding your drive, over and over, whether or not anything is there to find.