🚀 New at Matillion: Public API Endpoints for Shared Pipelines, Terraform for Streaming Pipelines, and Enhanced Maia Variables

This week, we're excited to introduce powerful new API capabilities for shared pipelines, enhanced variable support for Maia AI agents, and Terraform for Streaming runners and pipelines, making your data workflows more automated and intelligent.

🔌 New Public API endpoints for shared pipelines

We're excited to announce the availability of a new set of public API endpoints, allowing you to manage the full lifecycle of your shared pipelines programmatically.

What you can do:

• Publish versions: Publish new shared pipeline artifact versions directly from CI/CD pipelines
• List and filter: List and filter published versions by source project or specific pipeline
• Retrieve details: Retrieve detailed information about any version, including the latest by default
• Enable or disable versions: Enable or disable specific versions to support rollback and controlled rollouts
• Browse pipelines: Browse all shared pipelines in an account, including their latest enabled version and description

See the API documentation and how-to guides for further details:

• Shared pipeline artifacts API endpoints
• Understanding shared pipeline artifacts
• Publishing shared pipeline artifacts with the API

🤖 Maia AI agent update: Improved variable support

We're happy to introduce system variable and project variable support, expanding your Maia AI agent's understanding of the variables available in your project.

Why is this helpful?

When building pipelines, knowing which variables are already defined—and what system variables are available at runtime—saves time and prevents duplication. Maia AI agents can now check what exists before creating something new.

What's new:

• System variables reference: Maia AI agents now know the available system variables (${sysvar.})—environment defaults, pipeline context, component metadata, and more. Ask your Maia AI agent which variables are available for your platform and it'll guide you to the right one.
• Project variable awareness: Maia AI agents can list your project-level variables, showing their name, type, behavior, and default value—so it can reference existing configuration instead of creating redundant pipeline variables.

How do I use it?

Maia AI agents will automatically use system variables and check your project variables when building pipelines—no special prompting needed. For more control, you can use context files to define standards like which project variables to use, helping Maia AI agents stay consistent across your pipelines.

🏗️ Public Terraform provider now available for Streaming runners and pipelines

Creating multiple Streaming runners and pipelines just got significantly more scalable! The new public Terraform provider eliminates the need to manually create runners and pipelines one at a time through the UI—a process that can be extremely time-consuming when dealing with dozens of databases.

With this Terraform provider, you can now programmatically create, update, and delete Streaming runners and pipelines at scale. This is particularly valuable for organizations managing large numbers of data sources, transforming what was once a lengthy, repetitive manual process into an automated, efficient workflow.

The provider includes comprehensive examples and covers both runner and pipeline configuration. While client-side and local infrastructure requirements may vary, this solution provides a solid foundation that can be customized to meet your specific needs.

Check out the Streaming pipelines documentation to get started.

💬 We'd love to hear from you!

Let us know how these new features are improving your workflows—we're all ears! Feel free to add any comments or questions below.

Want to get involved? Join the Matillion Community to stay up to date, share feedback, and help shape our product roadmap for future innovations.

‍