Hi everyone

I'd like to introduce apitap, an open-source data-transfer engine I've been building — Rust core, Python bindings, MIT-licensed.

The idea behind it is simple: moving a lot of data shouldn't require a lot of machine. Most ingestion pipelines cost money twice — once in wall-clock hours, once in the oversized workers those hours run on. apitap streams raw wire formats through parallel pipes with bounded memory, so the same job that needs gigabytes elsewhere finishes on the smallest container you can rent.

Where it stands today (10M rows, 15 column types, every number checksum-validated against the source; full methodology in the repo):

On a 0.5 vCPU / 256 MB container apitap still completes every route; in our runs the other tools either crawled or were OOM-killed there. (BigQuery bonus: the whole ingestion path is free — load + copy jobs only, no DML — so it works on sandbox projects without billing.)

The goal we're quietly working toward: 10M rows per minute on that 256 MB box. We're about 3× away, and honestly we may never fully get there — but every step is measured, one lever at a time, and written down with its caveats and our own mistakes: benchmarks/README.md

I'd love help. If any of this sounds fun, contributions of every size are welcome:

• New connectors — MySQL → BigQuery is next on the list, then Snowflake; the Source/Sink traits are small and documented
• Break our benchmarks — run them on your workload; if a number doesn't hold up or a type gets mangled, an issue with a repro is the most valuable gift
• Real-world testing — weird schemas, huge rows, exotic types, flaky networks
• Docs — if something confused you, that's a bug in the docs
• Platform wheels — aarch64 / macOS builds via CI (maturin-action)

Repo: GitHub - apitap/apitap-lib · GitHub
Usage guide: apitap-lib/docs/usage.md at main · apitap/apitap-lib · GitHub
Origin story: I moved 10 million rows in 9.9 seconds with pip install apitap

Full disclosure: it's built with Claude Code as pair programmer, and this project has been corrected by its own failed runs more than once. If you see something wrong, please say so loudly — that's exactly how it gets better. Thank you!

1 post - 1 participant

Read full topic