Creating the Snowflake CoWork Maia Skill: Best Practices
Manually defining the semantic layer for a Snowflake intelligence agent is a highly complex and time-intensive task. It requires extensive knowledge of table structures, relationships, and data measures to ensure the agent provides reliable insights to business users. Maia addresses this by using a specialized skill to automatically analyze data pipelines and tables, creating a documented data model for human validation. It then populates a scalable metadata framework to consistently deploy and manage these agents with built-in version control and support for unstructured data.
Hi, I'm Erwan. I'm here today to talk to you about a Maia skill I've created for defining and deploying Snowflake Intelligence agents. But before we dig into that, maybe I should just start with talking about what Maia skills are. Skills are folders containing instructions, scripts and resources that Maia can discover and use to perform tasks more accurately and efficiently. I've got open here the Maia skills documentation site that kind of goes over all of that and gives a lot of the info about Maia skills.
The one thing that I want to spend a couple minutes on before we go further is just really to talk about some of these skills best practices we've got here. They're really great best practices just at the highest level about how to approach building skills. You want to write effective skill descriptions, right? So, you wanted to clearly define the trigger in which Maia would use the skill. Describe the scenario or task where the skill applies. You want to use keywords in defining the skill including terms that are likely to appear in natural language prompts from users that would have Maia to trigger that skill. Make sure you also avoid overlaps in skills. So, don't be creating a lot of skills that are similar in purpose. Try to have one single skill that does the thing.
And when you start to build the instructions of the skill, start with the goal, begin with what the skill should accomplish. Provide step by step guidance, some skills can get very complicated and when they get very complex, you can turn those in a number of steps that Maia can follow. You'll see this in play with the Snowflake Intelligence skill but using templates. A skill could be just minimal, just step by step guidance but if it gets more complex or there's a certain way you want Maia to do a thing, including templates is a very useful way of making sure that Maia does that. Specify any constraints, note any requirements, limitations or standards that might apply when utilizing the skill.
Now, that skill you're going to be testing and refining it along the way before it gets to a state that you think you'd want to share it with others. So that testing should involve testing with varied prompts. Try different phrasing, making sure the skill activates when you expect it to, make sure it doesn't activate when you don't want it to. But more importantly, you're going to be iterating and refining that skill based upon the outputs and the results that you get when you're testing. That very first time you use that skill you developed, Maia may not do exactly what you need and that feedback loop into refining that skill so that idea when you give that skill to others is that Maia will behave in a predictable way so that it typically gets the steps correct each time and every time.
Keep that skill focused. If a skill becomes too broad, you might think about maybe breaking it up into multiple specific skills. So those are some really great broad stroke type best practices when approaching building Maia skills. So now what I want to do is take a look at the actual Snowflake Intelligence skill that I built.
Now, every Maia skill is going to be located in a folder in your project in dot Matillion Maia skills and in here you can see I've got a Snowflake Intelligence folder so that's where all of the assets related to my skill are located. Now, here's another best practice that I want to share that isn't mentioned in here but I'm to share some others along the way but rule number one, use Maia. Maia is going be the one that is using the skill you create but Maia can be a really great collaborator in building the skill itself. I did exactly that. Maia was my collaborator in building this Snowflake Intelligence skill.
Now, the anatomy of a skill at a minimum is always going to include this skill markdown file. This is the instructions that Maia is going to use when triggering that skill. When you put these instructions in your skill markdown, one of the other rules that I would actually mention is the best practice is try to keep this concise. You do not want this skill markdown file to be overly large. Let's say, you know, thousands of lines of instructions. That's going to be overwhelming. You want to keep this skill concise so that Maia as an AI knows exactly what to do.
In my case with my Snowflake Intelligence skill, it's actually very robust. There's a lot of things and a lot of logic in it. But as I scan through this, you can see that this skill markdown is actually quite small. So what I've actually done is I've actually created a supplementary markdown file called this README. Now this README has all of the very in-depth things that Maia is going to do all along the way of using this skill. The skill markdown in this case now is really more of an index. You can see it right here is that the full documentation for the skill is in the README, but you must read the README before any deployment action. So the idea here is that there is different tasks in this skill and this is more of an index and a table of contents telling Maia when it gets to a certain task related to the skill, go reference what exactly to do in the README for that skill.
Snowflake Intelligence can be a very complicated thing and which is why Maia's skill really helps to make it a lot easier. One of the more time intensive tasks of building a Snowflake Intelligence agent is defining that semantic layer. Now, if you are not familiar with it, what the semantic layer defines for a Snowflake Intelligence agent is the tables that it has access to. What are the relationships between those tables? What are the dimensions and measures that are represented in the data in those tables? Verified queries around those tables, things that we know if you run this query, this is the expected outcome. We've vetted and validated this query. These are all things within that semantic layer that a Snowflake Intelligence agent relies upon to return to that business user reliable insights.
So the reason why I'm talking about that semantic layer is defining that manually can be very time intensive. It requires a lot of knowledge about the tables and the structures you're working with. Well, Maia and using Maia as a skill can really help to simplify that. So when you use the skill, one of the first things you might do is point Maia at some tables or maybe a pipeline that is managing the data in those tables and that gives the context to Maia. But if you see here in my skill markdown, this mandatory checkpoint before anything else, the first thing is that Maia when using the skill is going to prompt the user whether or not they would like for Maia to create a data model markdown file.
And what the idea here is that there is definitely a human validation step in here, but Maia what it is going to do if you give it permission in the skill when it asks is it can go analyze the pipeline that you pointed it to, the tables that are being referenced. If you've got a knowledge layer defined, that can also be additional context to help my build that data model. But what's really important about that data model is that it will be created, it is documented, the intent is for you as the human in the middle to review that data model. Make sure it aligns with your understanding of the data and the tables involved. That mutual understanding allows for Maia to build that semantic layer correctly and consistently. Okay? So data model very very important. Okay?
Now, let's talk a little bit about templates because this is definitely something that's happening here. You can see there's a templates folder in here, but let me tell you a story about how this was built. Now, I was just talking about that semantic layer and so when I originally built this Maia skill, I was looking at that semantic layer defined as a semantic model. And now, that semantic model is a YAML file. Okay? Now, subsequent to that in this skill, it supports building that YAML based semantic model, but now there's also semantic views and the skill supports that as well.
But the key is here, this is the progression of how I developed this originally. My intent was to have Maia go look at some pipelines and tables, be able to derive the structure and just build that semantic model YAML file for me. I created a template YAML file that Maia could look at. What I found in that first iteration of the skill was that Maia like was able to understand all the metadata about the tables. It had an example YAML file, but it struggled in taking the metadata that it gleaned from the context and creating a brand new context file. It got it but sometimes it would have to iterate over it or it would take some time.
And so, in that back and forth where I collaborated with Maia was that, you know what, let's build a framework. Let's make this a little bit easier for Maia. So, we built a framework and that's one of the first steps of this skill is that it's part of a framework. And that framework is some tables, a stage and a stored procedure that are used to help facilitate defining that semantic layer. Now, this pipeline is a template of how Maia would initialize that framework. In fact, Maia can inspect when it's running the skill. Does that framework exist? If not, offer to create it if it already exists, then say, do you want me to clear it out or just continue using what's out there.
But what is that framework? There's multiple tables involved. So there's like a semantic models table. So this will be one row per semantic model that you want to create, but this is extensible. You're going to have maybe multiple Snowflake Intelligence agents based upon various semantic models. So this architecture will support that scalability and that growth because you are going to have one main table that defines semantic models. This table of semantic model tables defines all of the tables within any individual semantic model. This table will define all of the dimensions within those tables that are represented in those tables. We've got a table for measures, the relationships between those tables, verified queries in those semantic models.
There is also a Cortex semantic models table. What happens is we generate the semantic model from all that metadata and that goes into this table where it is also going to be version controlled so semantic models will evolve over time and you'll have, a full version control over all of those models. So that's some of the key tables. There's also a stage because the way this is going to be built is it's a YAML based semantic model, it needs to upload it to a stage so that the Snowflake Intelligence agent can have access to it. That's where that's going to be created as well.
Now the other bit that was complicated I mentioned a little bit ago was generating that YAML file. And so, to help Maia here, but Maia helped me because what Maia ended up doing was creating a Cortex based Snowflake Cortex stored procedure that will generate the semantic model based upon the data in these metadata tables. So, that's what this procedure is and again, it's something that Maia created. Okay? And just defining all of my semantic layer information in the metadata tables actually made it quite simple to then turn those into semantic views. This could be extensible in the future if I want to integrate with maybe some other AI based business intelligence tool that also has a semantic layer but maybe needs to define in a different way. Having all of that defined in metadata tables makes us really scalable and extensible. Okay.
So let's go back to the skill because we're gonna see kind of how the evolution of this goes. There's some questions that Maia is gonna ask the user when the skill is activated. So we already talked about that first one about whether or not create that data model and validate if as documented if that's your understanding as a user. And then there are some of these other questions that will be asked all at once. Like what do you want to name the agent? It will give a suggestion. You can override that or just take the suggestion. The name of the model that's going to be created, that's also something that's going to be asked for.
The infrastructure that I was just talking about, it's going to check. Does it exist? And then based upon its existence or lack thereof, it'll ask, do you want me to initialize? I see it's not there. Do you want me to initialize that infrastructure or I see it's there. Do you want me just to use the existing? There's a deployment type because it does support both semantic views and semantic models. So definitely go with semantic views if you're doing stuff like intelligence. That's a much, it's just a better way in my opinion because now that semantic layer is defined as a actual stuff like object, right? And like I said, this fully supports it.
The other really cool feature of the support is that if one of those tables in your data model contains unstructured text, maybe it's chunked or vector document data. It will see that and ask if you want to create a Cortex search service on that unstructured text. Because stuff like intelligence supports that where you can have an agent that not only does text to SQL based upon that semantic layer, natural language saying, question, what are the metrics from XYZ and turning that in a SQL to return results. But now, you can augment that with a document search against full text data.
I have an example where I have done exactly this, created a Snowflake Intelligence agents where there are metrics that were gleaned out of Lyft and Uber quarterly financial reports and there is all stored in facts and dimensions and that's that semantic layer. But the actual raw text from those, quarterly reports are also in a table and they're exposed to Snowflake Intelligence. So you can ask questions of Snowflake Intelligence of something like, what was Lyft's Q4 twenty twenty two, EBITDA, metric and what was, what did the CEO say, about that, right? So that's kind of a broad question, but that's really the idea. It's a two part question where asking a question for some actual metric values, but augmenting that with contextual information that is represented in that full text. So, really cool features in this Snowflake Intelligence agent, right?
So, I talked about the complexity of the metadata and that's where some of these template also come into play because what you're going to find here is that I've got templates for both deployment types to a YAML model or to the view. You also actually see here this data model template, that's a good one to highlight because when it creates the data model, I've got a markdown template file that shows Maia how I want that data model documentation to be written. And this way I get data model documentation in consistent fashion. If your organization has its own standards around that, you could always swap that in. But there's some key things that are represented in here, The tables, the relationships between the tables, any text or document columns for that Cortex search they talked about. You've got some examples and about the relationships, things along those lines.
But also in these templates is about how to deploy with these different deployment types. So, there is a populate metadata pipeline here. This is really helping Maia to actually define the metadata and actually there is some key things in here that I wanted to share because there's a lot of different ways you could have done this. The first iteration of this, Maia was just trying to create SQL scripts to create all of my metadata. But what I've laid it on is this pattern that is definitely some best practices. So one is using variables. So have Maia instead of trying to generate SQL as part of that skill execution, have Maia use templates to do more of like string replacement.
So I'm using variables here for things like the name of the model and the database and schema of the tables that are going to be in that semantic layer that it is going to reference. And those are all variables because then Maia can just replace it in one place and then those variables can be used all throughout pipelines and other places. And the idea with the variable usage here is Maia only needs to replace the variable default values and everywhere these variables are used in these templates, they'll get those values.
Now, key thing is that in this higher level populate metadata pipeline, what you could see here, it's doing multiple things. It's going to clear any existing metadata for that model. The idea is you're initializing it. But you might actually be modifying existing and that'll work here as well because it's actually the models that are version control. The metadata can be cleared out and reloaded. But each step in here that you can see in here if I kind of go into it is actually a separate transformation that's going to populate the different tables in that metadata infrastructure.
So let me go into this populate tables one as an example. What we are to find is that this pattern is actually used everywhere. Instead of using SQL, what I am doing here is just using a fixed flow component. The fixed flow component is really good one because what it allows you to do is just to find data in a structure. In this case, the columns for my table metadata, these are all the columns in that metadata table. So I've defined the structure, these are static in this transformation. But what Maia will then do when using this template is it will actually fill out all of the tables this fixed flow. It's like a grid of values at that point, but it's also going to use these variables.
So it doesn't need to fill these in because the model name database and schema, these are all going to be already defined in the variables, but what you're seeing here is the all of the other information at the table level that Maia then will, that's where Maia will spend its time is defining these things in the fixed flow. So again, we're avoiding Maia needing to do anything like defining SQL and Maia could just be focused on defining values. So if we look for instance at the dimensions, right, that's going to be another one that follows the exact same model where it's this fixed flow has different columns involved because these are not dimensions, right. But at run time when Maia is using the skill, it's just simply going to populate all of these, this fixed flow with the relevant data from the data model documentation.
Another characteristic of this, so that you know, let's say all that metadata has not been populated. We've got this higher level pipeline that says how to deploy the agent and in fact before it even gets to that, there's rules in the skill that says after you've run that populate metadata, here's a validation pipeline that you can use, that Maia can use that tells it how to validate if all of the prerequisites were done. In this case, what's the metadata for that model populated? Make sure that the tables were defined, dimensions, search source tables, all of those types of things. And this will gate it before it goes to try deploy the Snowflake Intelligence agent. And these are just some high level checks to make sure it got that metadata for the semantic layer right.
Because then what it's going to do is generate that actual metadata, create the semantic view. In this case, I am using that semantic view deployment type. It'll grant some table access. If there is a search service, it'll create that. If there is none required, then it'll just go ahead and create an agent. Otherwise, it'll create that search service as well as create the agent. But you can see in here, if I were to go in a little bit more detail, these are things that are all scripted out for Maia. And Maia can just use variables where needed and replace things as needed. But everything is all scripted out so that Maia doesn't have to be generating anything at run time.
So this is what it looks like to create that Snowflake Intelligence agent with I've got a hard coded model, but we could always change that. But you can see some of the characteristics in here, but it's also, here's the tool spec, the Cortex search. Here's the tool that semantic view for the data analyst. The Cortex search service for the document search if that's what's being used. And so, after that's deployed and it's going to register the agent, create the agent usage. So this is the full deployment. When this is done, you literally have a Snowflake agent that you can start to interact with within the UI.
But before you get to that, you are going to have validation of the actual deployment, make sure that semantic view exists, check that the table count in the semantic layer aligns with what's in the metadata, things along those lines. Maia will also create deployment summary for you. So this is another markdown that will be created as an artifact. It will show everything that was created and one of my favorite things about this is that it will also create example queries that you can then just copy and paste into Snowflake Intelligence just maybe as part of validation, but it's also documentation that can be shared with end users who want to use that agent.
But it'll go as far as creating example queries just for the straight up metric type queries, text SQL type things. Maybe some other things that are just straight up document search, right? If you've got that in there. But also give you some combined queries as two part type queries, some examples that you can use there. As well as how to use the actual intelligence agent. Everything from using it via the UI to direct SQL access, Python, Snowpark. There's some instructions if you then need to update the semantic model, redeploy the agent. There's even scripts in here to clean up if needed. So things like, let's get rid of that agent, let's clean up the metadata, clean up infrastructure, all of that self contained in the skill.
But here at the end of the day, net net, this is a very involved skill. Might not be the type of skill you start with, but this is showing the art of the possible. Some of the really advanced things you can do with Maia skills to make some really time consuming processes, turn them into easily AI assisted repeatable processes. Hopefully, this is a really helpful walk through. Check out some of my other videos where you'll see this skill in action as well as some of the data that I was talking about with Lyft and Uber. Hopefully, this is helpful for everybody and looking forward to sharing some other tips and tricks with everybody in the future. Thanks.
You might also like
Our latest resources
Data management


