📦 Environment System Variables and File Load Components for Databricks

This week, we're excited to introduce updates that will streamline your data workflows and expand your pipeline capabilities. We've enhanced system-level variable access and expanded file loading support for Databricks users. For a full list of recent changes, be sure to check our changelog updates.

📊 Environment defaults as system variables

We've added the ability to reference project environment defaults, including role, database, warehouse, and schema, as system-level variables. This eliminates the need to create separate variables when [Environment default] was unavailable, such as in SQL and Python scripts.

📦 File load components now available for Databricks

5 new file load components are now enabled for Databricks, making it easier to bring file-based data straight into your pipelines.

New file load components for Databricks:

• CSV Load
• JSON Load
• AVRO Load
• ORC Load
• Parquet Load

These components allow you to load data from files in your source location directly into Databricks tables, with automatic handling of schema inference and table creation where applicable.

Supported from agent version: 11.154.0.

💬 We'd love to hear from you!

Let us know how these new features are improving your workflows—we're all ears! Feel free to add any comments or questions below.

Want to get involved?

Join the Matillion Community to stay up to date, share feedback, and help shape our product roadmap for future innovations