Build a Semantic Layer for Snowflake CoWork
Building a semantic layer for Snowflake CoWork agents is a complex task, especially when it requires providing natural language feedback from a mix of structured financial tables and unstructured PDF reports. Maia addresses this by using a specialized AI skill to automate the heavy lifting of defining table relationships and metadata, generating a robust data model, and deploying a validated intelligence agent with integrated analytical and semantic search capabilities.
Hey there. Here. Today, I wanted to have a walk through of using the Snowflake Intelligence Maia skill that I've built. Just wanted to show it being used in action. Before we go into that, maybe a little background on what I'm gonna be using for the data and tables.
I've got a pipeline open here that I had Maia I collaborated with Maia to build. It's populating some dimensions and fact tables. So it's a gold layer where I progressed some data from bronze, silver to gold, that classic medallion architecture. Gold layer is that reporting ready layer with facts and dimensions.
The actual data I'm dealing with is what I've done is I've gone out to Uber and Lyft's websites and downloaded their quarterly press release financial reports as well as some 10-K SEC filings, but this is kind of what they look like, just raw PDFs with metrics and things. And the pipelines in here extract all those metrics, do some data cleansing, but ultimately get them all shaped and ready for reporting in facts and dimensions and that's what we're gonna be working on. Okay.
So let's take a look at this Maia skill in action. So I can just simply ask Maia, please create a Snowflake Intelligence agent based on the tables managed by the, gold layer main. And there is one other table I also want to add into it. It's called financial reports vector all because that table actually contains all of the chunked and vector embedded text from the raw PDF.
Now, Snowflake Intelligence can actually create an agent that uses that unstructured text, not the embeddings but the unstructured text which is also in this table. I want that to be included as well. So let me go ahead and kick this off and we are going to see Maia will hopefully start using the Snowflake Intelligence skill which is what we see exactly right there. It's going do a little bit analysis and then it should come back offering to create a data model.
That's a really important point because if you've ever worked with Snowflake Intelligence and try to spin up an agent yourself, you'll know that there's a little bit of complexity around the semantic layer. The semantic layer is specifically the thing that defines for that Snowflake Intelligence agent. What are the tables, the relationships between those tables and all that metadata that gives that Snowflake Intelligence agent the power to provide natural language feedback from the data in those tables.
So you could see here that Maia has the context of the pipeline that is populating these tables as well as the other one that I added. It's highlighted there's six dimension tables, five fact tables plus the additional one that I asked for. Now with that semantic layer being super important, Maia is offering to create a data model for me and I'm going to allow it to do that.
This is really important because what this is going to allow once that data model is created is that I'm going to review it, make sure it aligns with my understanding of the objects and the structure. There's definitely human in the middle here but Maia is doing all the heavy lifting of being able to define what these tables are, their relationships and everything that would go into that semantic layer. So, it's going ahead and starting to build that data model.
What I'm going to do while it's doing that, I'm going to go ahead and allow it to start to inspect some of these things. While it's doing that, let me go ahead and pause, the recording. This might take a couple of moments, so we'll come back when it's got something generated and we can take a look at it together. Alright.
So Maia has just finished creating the data model documentation which is right here. Here's a little bit of the summary. But before we get too far into it, it identified six dimensions, five fact tables, one vector table with sixty five eleven chunks with seven hundred sixty eight dimension embeddings, fourteen relationships. And within that financial reports vector all table, it found a column that chunk data which is that raw chunked text from the raw PDFs that's where it's going to build a Cortex search service again.
If I review this you can kind of see what this documentation looks like. I have another video where I go into a deep dive of how I built this skill, so if you want to see a little bit more about the behind the scenes and how all this is, architected, but what we can see really here is the outcome. So here's my documentation where it's highlighted all the tables, the columns, we're gonna quick get down to the relationships right down in here.
Okay, so you see here this all looks good. There's that chunk data column in this table that we're gonna be able to use for that semantic search and there is some additional considerations, monetary values are millions, ICD type two, so all pretty robust and I will say that this is all correct.
Now as this skill proceeds, it's going to ask me some additional questions. So for instance, what do I want to name the agent that we are going to be deploying? So let's just call this the financial analyst. I could always put something else in and if I wanted to maybe I'll do that. Give me one second. So maybe we'll just call this financial intelligence. Okay.
Obviously, I could have taken one of these, suggested names, but I'm just gonna go ahead and just call it something else, just to show that that's possible. The semantic model, so that semantic layer that we were talking about, that needs to have a name as well. I will take one of the suggested ones here, the financial reports model.
So the skill and Snowflake Intelligence, that semantic layer can be represented in two ways. There can be a YAML file that's gonna be uploaded to a stage and that's a semantic model or we can use a semantic view. So the semantic view is definitely the recommended way where it's gonna create an actual Snowflake object, the semantic view. So, the skill supports both but we're gonna go ahead and use that semantic view that's definitely the recommended, method.
Which table should be included in the semantic model for structured pipelines? So, I'm gonna say all eleven tables and we'll go ahead and submit. So now it's going to go ahead and start to build everything out. This is also going to take a few moments but let's just take a look and see what feedback we get before I pause.
One thing I should also mention, I do go into more detail with this in my other video that deep dive into how I built the skill. But a really important thing to understand is the skill is based upon a infrastructure where there is a handful of tables, a stage, and a stored procedure that exist to help facilitate the deployment of this entire Snowflake Intelligence agent.
Now the way the skill works is that it's going to go out and detect to see if that infrastructure has been already initialized and it sees that it already exists. But if it saw that those tables and that infrastructure did not yet exist, it would actually offer to create it and it would allow to create it for me as part of the skill. It sees that already exist so it's gonna go ahead and just use the existing infrastructure and now it's just gonna go ahead and do its deployment.
So I'm gonna pause here and we'll come back when it's gotten to a point where we can you know walk through where it's at and where we'll go next. Okay. Got a prompt here from Maia so maybe this is a good time for me to just interject what's going on. So first of all notice here that Maia based upon the skill has identified a ten step plan and the first thing it's asking for permission around is the ability to copy some template folders and configure variables.
So as I go into in my other video that deep dive into this skill, there are a lot of template pipelines files as part of this skill. That gives Maia the whole guide as to how to roll this out in a predictable and repeatable fashion. And so right now it is looking to copy those template folders into a brand new folder within my project and so that's what it's asking for. So let me go ahead and allow it to do that.
While it's doing that, let's take a closer look at some of the other steps it's going to do. So it's working on the copying but it's gonna define the metadata for all these tables so infrastructure, that's where that semantic layer is gonna be defined by populating these metadata tables. It's gonna validate that metadata then it's gonna actually populate the metadata. Once that's done, it'll do another validation to make sure that the metadata was populated appropriately that it will deploy the agent and validate the deployment. It will also configure the Cortex search and create the full documentation.
So it's now chugging along. We can see here. So these are the things that it's working in. This is my database and schema. The name of the model that I specified, the agent name. So you can see all these details and now it's just going ahead and deploying these things. So I'm gonna pause again while it configures these things and we'll come back when it's completed its next step.
And you can see here it's actually starting to populate some of these tables, pipelines and things. But we'll do a deeper dive into what it's done once it's gotten through all of this. Alright. So Maia has gone through it to take in a few minutes but it's gone through and prepped everything up. Before I go through the kind of walking through everything it's done, let me show you over here what's actually happened behind the scenes.
What you'll see here is that I was originally in this gold layer main pipeline in this folder and the data model was created, documentation was created in that same folder. But what you see here, there was a new intelligence folder that's now been created for my financial intelligence agent. And this is where Maia has gone ahead and initialize from the templates from the skill some pipelines that will initialize the metadata.
So things like the tables, the relationships, the measures of dimensions, verified queries that comprise that semantic layer. There's some pipelines to actually deploy the agent as well as you can see in here there are pipelines that allow it to clean up the agent. So not only will it deploy the agent but it has, pipelines here if we wanted to remove the agent, reset the metadata.
So if we go back over here, Maia has gotten to the point where it's, defined all of that metadata and it defined the, you know, verified queries and it's looking to validate all of that. And so we'll allow it to validate that, but if we can also take a look, we can take a look at some of the things that is actually done here. So for instance, if I were to look at that populate tables pipeline, this is exactly what it did.
What it did was there's some variables that it needed to do some string replacement, so this is my database and schema of the name of the model. This is a fixed flow that is gonna find all define all of the tables in my semantic layers. So you see them all defined in here. Okay. The facts and dimensions. If we take a look at things like the verified queries, these are all the verified queries that are gonna be defined within that semantic model too.
So this is something where I'll just take what we've got here, but what I would actually highlight is that this is where you can extend that conversation with Maia to say, I want you know additional verified queries, maybe I've got some specifically in mind that I wanna make sure are defined, but you can see in here things like what is the total revenue for each company and here's that query that would represent how to answer that question. Right? So that's kind of, a characteristic of that semantic layer, for Snowflake Intelligence.
It's gone ahead and validated, that everything seems to look good within the pipeline configuration. Now it's asking if it's okay to go ahead and populate that metadata. So I'm gonna go ahead and allow it to do that. When it's done with that, it's gonna validate that it's actually deployed it correctly. So this is gonna take a moment. So pause while it's running that and we'll come back when it's, ready to validate.
Okay. So it's gone ahead and ran and populated that metadata. It actually did a step where it act, it validated that that metadata was created properly. So if we take a look here, there's this, validate deployment prerequisites, and that's exactly what it's doing. It's a pre canned, validation pipeline that checks that the metadata was populated, the tables are defined, just some high level checks to make sure everything looks good.
Now it's, going ahead and asking for permission to actually deploy the full Snowflake Intelligence agents. Let me go ahead and accept that. That's gonna take a moment to run but while it's running, let's just have a quick look at the actual pipeline it's gonna run. Again, is part of the template and you could see here this is a multi step process where it's going to generate the semantic model, create a semantic view from that. It'll do some, table access.
There's an if component here if, that agent needs to be defined with a search service. We are gonna be doing a search service here where that search service is against that chunked text that raw text. Okay. So for instance, you can kinda see in here if we wanna take a look at how that's done. Some pre canned SQL with Maia had filled in some of the information in here as well as reference to things like variables. Okay.
But it's gonna create that search service, grant usage on it to the Snowflake Intelligence role that we're using and this is the actual step that's actually gonna deploy the actual agent. Okay. So you can see here where it's gonna be a data analyst with the semantic view as well as the document search. So it's going ahead and deploying this in it looks like it might be iterating over it as it's found a couple issues, but it's just gonna be self reliant.
So I'm gonna pause here and just let it go through this and we'll pick it up when it's got it deployed. Alright, so Maia has just completed deploying that Snowflake Intelligence agent. It actually had a couple issues along the way that I needed to interject with and I'm gonna walk through that because you know, it's not always a perfect execution, not always one shot, but what you'll see here is, you know, with the human in the middle and a little bit of direction of my it gets it right.
Right now it's doing a little bit of validations, but let me show you exactly, what the issue was that we helped to correct. Let me get back out there. So let me see where my last things. So the main issue was that in the metadata, there were, tables and columns and, being defined and Snowflake is case sensitive. And initially when Maia defined the metadata for my semantic layer, it put everything all in upper case.
And so that was causing a couple of these agent deployment failures. So I saw that in some of the errors that we got. So for instance here I said here's the full error, looks like another metadata issue, right? Invalid identifier is financial. And so Maia was able to see, oh that is the case sensitivity issue, it went and fixed it.
But I actually had another prompt in here and just wanted to double check with my, you know, all identifiers are case sensitive. Make sure you have that correct everywhere. So it went through, cleaned everything up and reran the metadata and that allowed it to finally deploy everything. Okay? So let's create our deployment documentation because that's gonna be a really that's the final step and you're gonna see where that comes from.
While it's doing that, what we can actually do is I'm gonna pop over to Snowflake Intelligence and you can see here a lot of a few agents already existing in my Snowflake account. But what if I refresh now, I should now see that financial intelligence agent that, wasn't there a moment ago. So while that's reloading oh, here we go. So there's my financial intelligence agent, but let's go back over here.
Yep. So it's still creating my deployment summary because the reason why I'm waiting on that is it's gonna highlight in detail everything that's been deployed, but it's also gonna give me some example questions that I can ask of this Snowflake agent to validate that it's working as expected. So let's just wait for this deployment summary to be finished and then we'll pick it up with some actual validations.
Okay. So it's now finished creating my deployment summary. You can see it right in here. It talks about how it deployed both a Cortex Analyst as well as Cortex search tools to this agent. These are the five fact tables, the six dimension tables, and there is that one table for the document search. Here's some of the measures that are being highlighted, the table relationships.
So this is where you can see for instance we can ask some of these natural language questions. Let me go over here. We'll pop over to Snowflake Intelligence. I'll choose my financial intelligence agent. And actually, you can see the details here how it is both got my document search and data analyst tool. It's a financial intelligence agent for Uber and Lyft SEC filings analysis.
Right? So let me go ahead and put this in here. And I do have extended thinking on, so sometimes this takes Snowflake Intelligence a moment to answer this question. While it's doing that, let me just go back over here and take a look at some of the other example queries I can do. So this is stressing the data analysis that Cortex Analyst where it's gonna take natural language translate that into SQL to run. Right?
And so these are all things that I could absolutely copy and paste. You've also got examples here for document search where it'll just go through the actual unstructured text for more contextual search. Right? So maybe I could say that's all. I'll prep this one up. So we'll go back over here. And so what is the total revenue for each company? It was able to derive this out. Okay. And it even helps to chart it.
So this is all Snowflake Intelligence. Here's an example of a document search. Right? What do the SEC filings say about autonomous driving risks? So in this case, there's no chart but what it's gonna do is it's gonna go and you see here there's actual, links to where it's finding it in the actual SEC, in the actual raw text. Both Uber and Lyft identify several significant autonomous driving risks.
So safety and liability risks, regulatory compliance challenges, market technology risks. So this is all great. This is all being pulled out of those financial reports. But what's really powerful with Snowflake Intelligence, the ability to combine both the analyst and search, so you can ask these two part questions. So for instance, what was Uber's revenue growth last year and what do what context do the documents, source documents provide? Right?
So that's a good one because what we're gonna see here is that it's gonna pull the actual metrics for the revenue growth last year but then it'll also then reference the actual financial reports to provide some additional context behind that revenue growth. So you could see here the real power of Snowflake Intelligence defining that semantic layer. You get these trusted results back. Let me go back up here while it's working on that. Think what we should see here.
Yeah. You see here one verified query, so that's really great. It is that that's based upon the semantic layer and it's verified query. Right? So just showing that that semantic layer is working. And here is my question around what was Uber's revenue growth last year. I want context. Right? So here's the revenue growth numbers. K.
Fifty two billion for two thousand twenty five, forty three point nine seven eight billion for two thousand twenty four. And here's the contextual insights from the SEC filings. Right? The mobility segment increased by four point six billion. Delivery segment increased by three point five billion. Freight segment declined slightly. So you can see here these two part questions, could be very powerful and there's also, the reference of what the sources are for everything.
So that's Maia and the Maia skill help me to deploy my Snowflake Intelligence agent. It's offering me now I could commit and push all these changes. I've got everything out deployed in my project. If I wanted to, I could actually clean up the agent, clean up the agent metadata. All that metadata is version controlled, so Maia can actually help me to extend the data model or the semantic layer further.
Maybe this is my first iteration. Maybe a few months from now, I'm gonna wanna add some additional tables to this agent, and, the skill will fully support that as well. Thanks.
You might also like
Our latest resources
Data management


