How we built Lurkx under real constraints.

Lurkx started with one narrow question: can a person enter a domain or identifier and receive useful, structured context without assembling the answer across several disconnected tools?

The eventual product searches exposure records, public security signals, infrastructure data, and configured providers from one workspace. Getting there required more subtraction than the finished interface suggests. Every new source created reliability questions. Every user-supplied target created an outbound-request risk. Every useful result carried a chance of being stale, duplicated, or wrong.

The first version was small on purpose.

The first committed version arrived in March 2026 as a Next.js interface beside a Fastify API. It used SQLite, kept a single crawl loop inside the service, and focused on domain reconnaissance: DNS, RDAP registration data, certificate-transparency history, discovered hosts, email references, and crawl snapshots.

That shape was useful because it was easy to run and easy to reason about. It also had a hard safety default. Live collection was disabled on development machines unless an operator explicitly enabled it. A developer could add targets and inspect stored data without accidentally turning a laptop or home connection into a crawler.

Starting with SQLite was also a constraint, not a bet that SQLite should run the final platform forever. It gave the first service a portable store while the data model and workflow were still changing. The cost was clear: once ingestion, teams, search, monitoring, billing, and audit history joined the same product, one embedded database was no longer the cleanest center.

We combined working systems instead of rewriting them.

By June, Lurkx was no longer one application. We had the original recon frontend and backend, a larger exposure workspace, a separate ingestion worker, payment surfaces, shared contracts, and a Rust search service. The tempting move was to rename and rewrite everything into one tidy stack. We did not do that.

We moved the applications into one npm workspace with one lockfile and one CI boundary. Existing internal package names remained where changing them would add risk without changing the user experience. The public product became Lurkx while some internal packages retained earlier names. Compatibility mattered more than cosmetic purity.

1. March
   Bounded recon service

   Next.js, Fastify, SQLite, passive domain context, and collection disabled by default.

2. June 9
   Combined workspace

   Recon, exposure search, ingestion, payments, and shared contracts moved behind one repository and release process.

3. June 10
   Public Lurkx identity

   The product name, entry flow, and interface language converged around exposure intelligence.

4. June 13
   Gated production delivery

   Builds, tests, dependency checks, source scanning, and remote activation became one deployment path.

5. July
   Guided workflows

   The interface shifted from a collection of tools toward research flows, account controls, and a clearer trust surface.

The architecture follows the evidence path.

The center of Lurkx is not the dashboard. It is the path a record takes before the dashboard can show it. Sources and providers produce uneven material. A worker pulls within explicit limits, normalizes records, adds source context, removes exact duplicates, and sends them through an authenticated ingest route. The API applies access and storage rules. Postgres holds relational state, while Meilisearch supports fast retrieval. A separate Rust service handles specialized search work.

Connector failures are normal in this kind of system. Sources time out, change formats, impose rate limits, or disappear. The worker therefore has bounded concurrency, per-source item caps, request timeouts, retries with backoff, checkpoints, health records, and a dead-letter path for failed ingestion. One broken source should become a visible degraded state, not take the whole worker down.

The browser application is a React and Vite workspace with route-level loading. Search, exposure checks, monitors, infrastructure context, cases, and administrative screens share the same shell, but access is decided by role and account state. The public landing page stays separate and static. It explains the product quickly without shipping the application bundle to every visitor.

The constraints made several product decisions for us.

Security software can appear powerful by showing a large source count and a dramatic map. The harder work is deciding what the system should refuse to do. Lurkx had five recurring constraints.

1. A reachable source is not automatically a usable source.

A June source-vetting pass checked public and Tor-hosted forum targets from the VPS. It did not use a local browser or home connection, create accounts, solve CAPTCHAs, click through challenges, or download dumps. Thirty-two targets were dead or unreachable. Twenty-two returned a challenge. Several live domains were parked, redirected, unsafe, or irrelevant.

We recorded only limited availability metadata during that pass and recommended no automatic ingestion from those results. Challenge-gated sources remained manual-review candidates. Targets marked seized, dangerous, suspected traps, or focused on fraud stayed out. The constraint was not how many URLs we could make a scraper open. It was whether a source could be collected responsibly and produce evidence worth keeping.

2. Every target field can become an SSRF primitive.

A domain-audit feature needs to make outbound network requests. That means a user can try to point it at local services, private address space, unsupported ports, or redirects that cross into a forbidden destination. We added public-target validation, redirect revalidation, scheme and port restrictions, timeouts, rate limits, and production ownership checks for deeper audits.

We also made a more revealing decision. An anonymous live domain scanner was added on June 10 and removed on June 11. The public screen became a saved sample, while the live audit moved behind authentication, CSRF protection, and production authorization. Removing a feature one day after shipping it was cheaper than defending the wrong public boundary indefinitely.

3. Exposure data is evidence, not a verdict.

Breach and OSINT records can be old, mislabeled, copied between sources, or stripped of context. A hit does not prove a current compromise. A clean search does not prove safety. That language appears in the product because the data model cannot solve uncertainty by hiding it.

We kept source references, collection lineage, timestamps, and provenance metadata with records. Ingestion supports quarantine and deduplication. The interface is meant to help a person pivot and verify, not turn one match into an accusation.

4. Product analytics could not contain the product’s sensitive inputs.

Search targets, credentials, report evidence, pasted text, recovery phrases, and form values do not belong in product analytics. Lurkx uses manually named, low-detail events for routes, actions, result counts, broad status, and role. Autocapture, session replay, heatmaps, surveys, page-leave capture, and dead-click capture are disabled.

That removes some convenient debugging data. It also avoids creating a second, less controlled copy of the material the product exists to protect. The trust note tells users what may be stored, what analytics receives, and why a result still needs independent verification.

5. One server still has boundaries.

Lurkx needed long-running workers, relational state, search indexes, scheduled collection, backups, and services that could restart independently. A static host was not enough. We deployed to a VPS, put public traffic behind Cloudflare and Caddy, kept databases and search ports on loopback, and ran the API, worker, payments API, and search service separately.

The machine uses a dedicated service account, key-only SSH, a restricted firewall, fail2ban, health checks, and scheduled backups. Collection could also run through isolated network paths and one-shot containers. None of that makes one server infinite. Concurrency caps, item limits, timeouts, and degraded-source handling remain part of product behavior because capacity is finite.

Shipping the monorepo became its own engineering problem.

Combining Node applications, a Rust service, static assets, migrations, and several stateful services made deployment more than copying a build folder. A release needed to preserve environment files, databases, caches, and writable data paths while replacing application code consistently.

The production workflow packages a committed revision, uploads that exact source archive, synchronizes it without overwriting runtime state, installs locked dependencies, builds the workspace, runs migrations, restarts services, and checks health. Early iterations exposed practical failures: build caches could disappear, service directories could lose write access, and a CI gate could fail because a scraper dependency was incomplete. Those were fixed in the deployment path rather than documented as manual exceptions.

A production push now waits on TypeScript builds and tests, dependency audits, Rust checks, Semgrep, Gitleaks, and an SBOM build. These gates do not prove the product is secure. They make a class of avoidable failures harder to ship and leave an inspectable record of what was checked.

The important architecture was the boundary around uncertain data, not the number of panels in the dashboard.

What the build taught us.

The warm black and copper interface is the visible layer, and it took several passes to settle. We tried a more conventional security-console treatment, then restored the quieter copper system because it made dense information easier to scan without turning the product into a movie prop. The landing page and app share that language, but they do different jobs.

The more durable lessons came from the parts a screenshot cannot show.

• Start with one bounded workflow before designing a platform.
• Keep working internals when a rename would only create migration risk.
• Treat source health, policy, and provenance as product state.
• Move dangerous capabilities behind stronger authorization, or remove them.
• Design analytics around the data it must never receive.
• Make finite infrastructure limits explicit in workers and queues.

Lurkx is still a changing system. A connector in source control is not a promise that the upstream source is available. A planned module is not a shipped capability. The useful measure is whether a person can move from a signal to a reviewable answer while the system keeps its uncertainty and boundaries visible.