Warning

This post was published 61 days ago. The information described in this article may have changed.

Info

This post is auto-generated from RSS feed The Rust Programming Language Forum - Latest topics. Source: ApiTap – Stream APIs to PostgreSQL with SQL and Built-in Scheduler

# ApiTap: Stream APIs to PostgreSQL with SQL + Built-in Scheduler

Hi everyone! 👋

I'm excited to share **ApiTap**, a Rust-based ETL tool I've been working on. It makes API-to-database pipelines incredibly simple using SQL transformations.

## What is it?

ApiTap lets you write SQL to transform API data and load it into PostgreSQL—no Python scripts needed.

**Quick example:**

```sql
{{ sink(name="postgres_sink") }}
{{ schedule("0 */5 * * * *") }}  -- Every 5 minutes

SELECT 
    id,
    user_id,
    created_at
FROM {{ use_source("api_data") }}
WHERE status = 'active';

That's it. ApiTap handles pagination, retries, and scheduling automatically.

Key Features

• SQL-first - Write transformations in SQL, not code
• Built-in scheduler - Cron-based automation (no Airflow needed)
• Smart pagination - LimitOffset, PageNumber, PageOnly modes
• PostgreSQL 14-17 - Full support with optimized MERGE operations
• Production-ready - Profiled with flamegraph, optimized for performance

Tech Stack

Built on solid Rust foundations:

• Apache DataFusion for SQL execution
• Tokio for async runtime
• Minijinja for SQL templating
• tokio-cron-scheduler for job scheduling

Architecture

• Optimized batch processing (5000 rows per batch)
• Lock-free atomic operations
• Zero-copy streaming where possible
• Efficient async I/O with Tokio

Why I Built This

I was tired of writing boilerplate Python scripts for every API integration. Wanted something fast, simple, and production-ready. Rust seemed perfect for the job.

Links

• GitHub: https://github.com/apitap/apitap
• License: MIT
• Status: Production-ready, actively maintained

Feedback Welcome!

This is my first major Rust project. I'd love to hear:

• What features would make this more useful?
• What integrations do you need?
• Any architectural feedback?

Contributions and PRs are very welcome!

---

Built with and Rust

1 post - 1 participant

Read full topic

🏷️ Rust_feed