Build a Claims Processing Pipeline with Maia and Jev AI
See what Maia can build from a short prompt. Given an S3 bucket of unstructured claims data (PDFs, JSON, text and emails), Maia designs a metadata-driven bronze, silver and gold architecture, ingests every format with Textract and S3 load components, and calls Jev AI to evaluate each claim.
The result is a star schema with claim category, liability, fraud risk, severity and a composite priority score across 440 claims. The build took around ninety minutes, including documentation and regression tests. The demo also covers CI/CD promotion to a QA environment, column-level lineage and agentic DataOps root cause analysis.
This video is part of the Building with Maia and Jev series. Read the full write-ups:
Hi. My name is Joe, principal solution architect here at Maia. This is the third video in a series of how we can leverage Jev models to provide decisions as part of a data engineering product. In this video, I'm gonna walk through both the shared pipelines that we previously built, the logic that we had Maia build to connect to the Jev models, and also give it a real life end to end data product build.
And so the other two videos cover kind of the connectivity to the Jev system one models as well as the building of the shared pipelines and some examples, and this is really focused on from a practitioner's perspective on how can we leverage these capabilities to then build a connection to Jev as part of a data product.
So here's the original prompt. We're gonna give it some, unstructured data sources in an S3 bucket, and there's all to do with different claims data. So here is that S3 bucket. We've got text, JSON, PDFs, and emails. I'm gonna share here that here's an example, claim, that has a statement, and we wanna identify whether or not this claim is valid, and we're gonna provide some rules for how we should think... How Jev needs to think about whether or not this claim is valid as well.
But a lot of this problem statement is to do with connecting to all of these different unstructured data types and then also building the interaction to the evaluation models, from TypeSafe. So come back into Maia's Mission Control where I can see all of my different data products that I'm working on and I can also see how Maia's then thought about building a plan.
So it's designed some original architecture to following sort of bronze, silver, and gold layer which is what I was after. And then it's also presented, some risks and assumptions to me as well. So I answered some questions here and then really just left Maia to carry on sort of working through and self validating and verifying. A couple of points I did need to give it some feedback. So, you know, I wanted it to center around the claim in the gold layer as opposed to around the claimant. And I wanted it to also test these pipelines end to end. So you can see most of this is autonomous.
Let's have a look at what Maia built. So here inside of Designer, I can see this readme file which provides with an overview of the architecture, both the bronze and silver layer, the different ways in which Maia has then gone and built around this different technical architectures that we have in place. So in this instance, I'm using Snowflake, an external GitHub repository, and I'm using the connectors that we have built out, to Jev.
So let's start with those connectors to Jev and then we'll walk into, and walk up into the data product. So this is a shared pipeline. This calls Jev models using parameters. And you can see here we've got some variables in with the text, the model, the questions are all parameterized. So this logic is a repeatable logic that I can then share... I have then shared across the rest of the team that are using this environment and they're able to then take this repeatable logic, and we can wrap this up into an iterator that then loops through a series of, in this case, tables which have different questions, and different requests of the underlying Jev model.
So you can then call all of the choice, null and scoring models leveraging the same component, but making the sort of dynamic entry point here using Matillion and Maia's capability of having iterators that loop through a series of variables as this operates at runtime.
And an example of this being put together, which I covered in the last one, I'll just cover very quickly here. We take a product review like love it, outstanding, great early delivery, great battery. This is what is being generated and being sent to Jev, the sort of questions around what is the overall sentiment, how satisfied is the reviewer, and do they mention a defect. And the underlying shared pipeline logic then enables us to take these values down, pass them down here into our scalar variables that then also enables us to get, some results out at the end that, we can then model and do some downstream analytics on.
So this is the kind of concept that we're looking to do except this time we're looking to not have sample data be the source, and the target but actually connect it to real unstructured data and have Maia build this end to end with Jev in the middle performing that evaluation step. And so here we've got a claims processing pipeline. This is the end to end bronze to gold layers.
So you can see here that we have our, bronze layer at the start which is ingesting all of those data sources from, in this instance, S3 with multi formats then our silver and our gold with some nice notifications if any of these fail as well. So this is the master orchestration pipeline and it's got some examples here of what it's doing in each step. But let's dive into each part of these steps one by one.
So firstly, in this ingest bronze pipeline, we're using metadata to create a registry of all of the files, inside of that S3, and storing that metadata inside of Snowflake so that we have an understanding of the files that we need to load in. And then we're passing the file format type and file pattern down into this ingestion engine. So here in this ingestion engine, if it is a PDF, then we're gonna use AWS's PDF Textract components to take that PDF data and pass it out into semi structured data that we can store as a variant data type inside of Snowflake.
If it's a JSON, then we're gonna use our S3 load component and Maia's designed this to effectively create a JSON staging layer inside of Snowflake that takes any JSON files and creates that semi structured data and the same approach for both text and email files as well using those S3 load components except each file type is gonna be slightly different as you can see.
So once we've got the... And Maia is able to build all of this logic out, I didn't need to provide it with any guidance on how to do this because it's taking advantage of the, Maia Foundation components. We don't only just support, Amazon's Textract. We also support Azure, and Snowflake as well as a variety of other vector database types and LLM models like Vertex and OpenAI that you may need to call out to any point in your journey to embed AI with inside of a pipeline logic.
Now we've connected to those data sources and loaded it into Snowflake, what we're then able to do is have a seed of questions being set. So Maia's designed some questions that we can seed inside of a table inside of Snowflake, and then it's gonna loop through that shared pipeline I showed you earlier that takes the unstructured data as the source. So the PDF information is now extracted into semi structured format and marries that up with the questions that we wanna ask around that data and we're using that same looping mechanism here to loop through those tables to get the answers back from Jev.
And so this is really the power of having Maia, an AI Data Automation platform, integrate to both your data sources as well as being able to set the questions that you want to ask your Jev and having that repeatable logic here that can be updated and used by multiple different data products and team members.
Once we've got all of that data back, Maia is then joining together the answers from Jev with the raw data to then build us out a star schema facts and dimensions model with the ultimate fact here being around claims. So working out whether or not the claim data answers the questions that we evaluated against Jev. And you can see here some of the columns that we're looking to see in our dimension and fact tables at the end.
We can actually sample this data here, you can see, the kind of journey of each of these claims, as it comes through from left to right. So we've got a... An ID for each of the claims that we've given it in a previous, step. We've got IDs that we may need to match to perhaps a central claim database of sorts that in this case I didn't have, but you can absolutely match that in. Jev could also be used there for fuzzy matching exercises if you just receive email dumps and text dumps and you need to then match up those data sources to a particular claim.
You can see here that one of the questions we evaluated against was the category of the claims. So in this instance, Jev reported it was cargo damage, the liability party, the routing decision, the fraud risk, severity, and complexity level. These are all questions that we asked, of Jev to evaluate based on this free text, source, and it provides us with the scores and the confidence intervals that it has around the categorization and of those scores as well as usefully a nice sort of token of how... Total input and output tokens were consumed as a result of processing each individual claim.
So if we bring it back up to the end to end, you can kinda see here the source data, the claim ID, the source format, and some example text, And then we can see how it then performs that evaluation in the silver layer identifying the claimant categories we just went through, and ultimately leaves us with a nice prioritization score in our gold layer based on Maia's assumptions on the fraud severity and approval risk, and we can of course manually edit that, as well.
And if we come back to the Maia conversation, we also have Maia perform some architecture diagrams just to help, you know, me as an end user understand where these different pieces were as part of the puzzle and some useful information around the actual claims themselves. So I'm gonna come back up and you can see here after Maia ran it end to end, it ran through all 440 claims. It evaluated all of them.
It's able to then categorize them for me so I get an initial read back here in the Maia team's conversation around the types of categorizations that Jev has identified as well as the sort of highest priority claims that I might need to then go ahead and inspect. So that is our end to end walk through of how Maia can build a claims processing pipeline, leveraging the new evaluation models from Jev, from TypeSafe that covers both the shared pipeline logic to then build other data products that leverage... That we wanna leverage these evaluation models with, as well as the connectivity to the TypeSafe API that is all dynamic and configurable by Maia.
And just to remind you of the prompt that we gave at the start was actually quite, sort of small. This was an end to end claims processing pipeline. We've got some unstructured data sources. Go ahead and leverage the components in the palette and I'll give you the S3 bucket information. You can, of course, level this up with spec driven development and having all of these tasks be fed from your Kanban board, your backlog of data products that you need to build and have Maia manage those autonomously, for you at scale via Mission Control and have you can see a number of different data products that I'm currently working on here all managed through this UI.
But effectively, Maia is able to take on board this new information around how to leverage an evaluation model like Jev and then produce, a claims processing pipeline that not only connects to those data sources but also gets it into a reportable state, as an output. All in all, this took around, ninety minutes or so for Maia to work through the end to end build and it documented all of the pipelines.
It also created its own test pipelines as well for each step of that journey so I can check that the bronze layer is gonna assert the right metadata correctly and I get these nice visual, pipelines as an output that I can then push down and run inside of my cloud data warehouse and walk now through my CI/CD journey of committing all of my changes here in my local branch and then pushing that out to my remote repository so that I can then run this inside of a QA environment.
And as I walk through that push here, you'll see I'm able to then promote all of these, pipelines out to my remote repository, publish this into a particular QA environment, give it a name, for the artifacts, and then go and schedule this to run on a sort of on trigger or on an hourly basis or, you know, sub five minute basis if I need it to pick up any new files that land into that S3 repository to have this fully automated end to end.
So the last thing I'll leave you with is not only do we now have the data products published, if I need to understand and inspect, the lineage of the claims table, I will be then able to come in here and if I search for claims, I should see some sort of claims, data in here. I can see that end to end lineage, of the different pipelines that created the liability confidence and the subsequent transformation pipeline that we've built out in here.
So we've got column level lineage as well as full observability of the platform as well. So I can see, pipelines that I have access to see as well as, the success pipelines of what ran and when and any subsequent root cause analysis and impact stream that any failed pipelines might have on the rest of my, sort of data products that require a fix. So if I go into this one here, I can see this pipeline failed. I've got a downstream data set that I now need to go and fix and Maia can actually take that, agentic DataOps task and go ahead and perform a nice root cause analysis for me on fixing this pipeline.
So just to show some of the maintenance side of this data product after Maia's finished building, not only do we look after the kind of connective tissue to those sources, the testing of the pipelines themselves, but also the maintenance, of these data products that you build with Maia, and in this case, with Jev as well. Thanks for watching, and any questions, please feel free to add a comment.
Featuring
You might also like
Our latest resources
Data management



