Using the Medallion Skill with Maia
Traditional onboarding of new data sources into a medallion architecture often forces teams to start from scratch, requiring weeks of manual field mapping and tedious pipeline engineering to meet specific governance and quality standards. Maia addresses this challenge by applying custom, pre-taught skills to dynamically adapt to new schemas and automatically enforce proprietary business logic, reducing the time required from weeks to just minutes.
Onboarding new data sources into a Medallion architecture promises rapid time to insight, but most teams hit the same wall. Every new source means starting from scratch, manually mapping fields, and meticulously coding your unique governance standards across the bronze, silver, gold layers. Generic templates help, but they can't dynamically adapt to messy new schemas or automatically enforce your proprietary business logic.
Maia changes the operating model entirely by applying custom skills that you've taught it directly to your unique environment. What used to require weeks of tedious pipeline engineering to hit your exact compliance and quality standards now happens in minutes. Let's go.
So in a previous video, you might have seen us build this new skill called Medallion architecture. Do check it out if you haven't already watched, and you can see the skills that you have available here in your project, by simply going into the skills drop down here at the bottom and going and finding those relevant skills. And here on the right hand side, you can actually view what the contents of that skill is.
So we're going to configure a brand new Medallion architecture from one of our source databases that we have to us. So we're gonna use this Postgres database. So I'm gonna tell Maia, let's go ahead and read the source, databases, find the Postgres, and then use the Medallion skill to build a new Medallion architecture pipeline for the tables in that source. I'm also gonna tag the data landscape to give Maia a bit of understanding of which tables are relevant to this environment.
So let's just show you that data landscape file as well. Here you can see I've got a set of tables that we have, across different sources, Postgres, SQL Server, and S3, and we've got some relationships and some external source connection descriptions as well. I've got more specific callouts to things like host name, database schema, and the secret reference that I have in my secrets manager. So I'm tagging both of these markdown files for Maia to then start using the skills that we've built and also the knowledge that we've got stored in these almost config files.
So Maia's gonna go ahead and it's gonna read through the existing project structure. So when it's talking about the project structure, it's talking about everything here in this files browser. So if I pop this out for a second, you can see we've got a variety of these different files that we have orchestration and transformation pipelines, but also these YAML files, markdown files, any Python, SQL, or any other types of files like XML, CSVs that you want to feed into your Git repository.
And you can see here that we've also got those project structures. So we've got folders, naming conventions that we use for the folders like docs or backlog or dot Matillion for the config at the config level, and some in house stuff that we've built as well. But in particular, we're seeing here Maia is gonna go and create that new medallion-postgres folder for us, and it's gonna go ahead and create a folder with our bronze Postgres load orchestration pipeline as well.
Now the test of this particular skill is making sure that both the loading requirements, so incremental loading from the Postgres, and then also the naming conventions, documentation, and testing, that we went through when we built this full SQL Server pipeline matches our existing standards that we'd like to see. So this is what Maia's built out currently for this Postgres, and this is what we built the skill based off of. So you can see it's a very similar architecture design. It's even got the right coloring here in the notes as well and notations.
So as Maia is building through this Medallion architecture, it's using the skill. It's not actually using the other pipelines that we have. It's using the skill to help design the pattern that we want to load. Now Maia didn't need to go into plan mode because that skill already has enough knowledge and awareness of how we would like to build Medallion architecture patterns inside of this environment. You can see that the Medallion architecture listed here out as a skill that's available to us in this project. And you can see here that Maia is now gonna just go through and build the bronze, silver, and gold layers as well as the mark of master orchestration as required.
So in the previous video, you might have seen it's around fifteen minutes or so to go from source to target with a clean slate, no skills. But here with Maia using the skill that we've previously created, we're seeing Maia actually build a lot faster and also a lot more in line with our exact business requirements, which is exactly what we wanna see from our agentic teammates. Here you can see it's in the transformation layer. It's now building up that gold layer.
And we're hoping to now see an orchestration pipeline being built that's gonna create that flow logic that we see after we've loaded that data from Postgres, clean that up in the silver pipeline, and push this all the way through into those reporting dashboards. Fantastic. There we go. Bronze, silver to gold. Documentation on the canvas, alerting notifications as required. And Maia's bringing back some valid checks here on the left hand side with our key features as well. Let's give that a run.
Now by clicking this button here, Maia then also I can manually force the pipeline to run here as well. But by keeping the conversation live with Maia, it's then aware of what's happening here in that task history, which is super useful. So it's able to then take any failure messages that we might see, come back here, and then go and update its pipelines accordingly. I'm also hoping to see Maia bring us back a summary of the output as well from that gold layer, just to give it a sense check.
Whilst Maia's executing that pipeline, gonna manually validate in here. Let's have a look at these assets that we have. So we've got this assets table in our Postgres. So you can see the type of asset that we're that's been described as well as the criticality and the status of those assets as well. And you can see here that Maia's actually the gold layer had an issue with the compliance analytics aggregation. So it's actually gonna go through and update and fix that, as well.
You can see here where it's highlighted on the screen, the component that Maia's currently working on, and it's updating that, particular aggregation. So let's have a look and see. Looks to be fixed. Let's validate here and just check and see what the aggregation was bringing us back previously. Okay. Yeah. So Maia has fixed that now and it's updated to include the relevant assets. And now it's executing that pipeline.
Here we go. It's gonna bring us back a sample of the results as well. Fantastic. Let's commit and push our changes into production.
You might also like
Our latest resources
Data management


