Data Integration Platform

A data integration platform moves data between systems. That description covers loading a warehouse for analytics, keeping two operational systems in agreement, and connecting SaaS applications so a signup in one creates a record in another. Buying a tool for the wrong one of those is the usual reason these projects stall.

We sell source code you clone and run yourself, and this is a category where we are firmly not the answer. The page is mostly about telling the three apart.

The three problems

The middle column is where most pain lives and where the fewest good products exist, because two-way synchronisation between systems that both accept edits is genuinely hard. If that is your problem, be sceptical of any tool that presents it as a configuration exercise.

ETL and ELT, briefly, since the terms are unavoidable

The older pattern extracts data, transforms it, and loads the result. The current default reverses the last two: extract, load raw data into the warehouse, then transform it there using the warehouse's own compute.

The practical consequence is that the raw data survives, so when a transformation turns out to be wrong you can redo it without re-extracting from a source system that may no longer have the history. That property is worth more than the efficiency argument, and it is why the pattern became standard.

The two-way problem deserves naming properly

If both systems can change the same field, you need an answer to what happens when both do. Last write wins is a decision, not a default, and it silently loses data. The alternatives are declaring one system authoritative per field, or detecting the conflict and asking a person.

The related requirement is idempotency. Any sync will deliver the same change twice eventually, because retries exist. Operations that are not safe to repeat will double-count, and this shows up as a data quality problem months later rather than as an error at the time.

Where the effort actually goes

Not connectors, which is what tools are sold on. It goes into agreeing what a customer is.

Two systems will have different definitions, different identifiers, and different ideas about whether a deleted record still exists. Reconciling those is a business conversation rather than a technical one, and no platform resolves it. Projects that skip it produce a warehouse full of numbers nobody trusts.

When teams build integration themselves

For analytics loading, rarely: managed connectors are cheap relative to maintaining API clients against services that change without warning. Airbyte is worth knowing about as the open source option if data cannot leave your infrastructure.

For application integration, a low-code tool is usually right for internal glue, and code is right when the process is part of your product.

For operational sync, custom is common, because the conflict rules are specific to your business and no tool encodes them. That is the case where this ends up in an application rather than a platform.

What you would start from here

There is no data integration capability in this stack. No connectors, no pipelines, no scheduler, no transformation layer, no warehouse, and no sync engine.

Where this is relevant is narrow: if you are building a product that integrates with your customers' systems, then the integration lives inside your application rather than in a platform, and the Vault template provides what surrounds it. Organizations as the boundary each customer's connection belongs to, roles and invitations, a Hono backend with Prisma and Kysely on Postgres, and an admin app for looking at a customer's sync when they report a problem.

Three things worth designing early if you build that. Store credentials encrypted and per tenant, since you are holding keys to somebody else's systems. Make every operation idempotent, using an identifier from the source system, because retries are certain. And keep a per-record log of what was synced and when, since the first support question will be why a particular record did not arrive.

For loading a warehouse or gluing SaaS tools together, buy. The packs below are only relevant for integration as a product feature.