Pipeline Development

Incremental Load Setup

Convert full refreshes to incremental loading when processing takes too long or warehouse costs are spiraling.

Prompt Template

Copy Template

Convert [PIPELINE] to incremental loading:- Identify incremental key: [TIMESTAMP/ID_COLUMN]- Add change detection logic- Implement upsert/merge strategy- Add backfill capability

B2B SaaS

Industry Example

Copy Example

Convert integrations/hubspot_contact_sync.orch.yaml to incremental loading:- Identify incremental key: LAST_MODIFIED_TIMESTAMP from HubSpot API- Add change detection logic to only process contacts updated in last 24 hours- Implement upsert/merge strategy using contact_email as unique key- Add backfill capability for historical loads when onboarding new properties

Prompt Details