Real-Time Monitoring & Alerting
Set up proactive monitoring of pipeline health and catch failures before stakeholders do.
Prompt Template
Help me design a monitoring and alerting strategy for [PIPELINE/FOLDER]:1. Critical pipelines: - [PIPELINE_NAME_1]: SLA [REQUIREMENT], business impact [DESCRIPTION]2. What to monitor: - Pipeline failures: [SPECIFIC_FAILURE_SCENARIOS] - Performance degradation: [THRESHOLDS] - Data quality issues: [METRICS] - Business metrics: [KPIs]3. Alerting requirements: - Who to notify: [STAKEHOLDERS/TEAMS] - When to escalate: [ESCALATION_CRITERIA] - Notification channels: [EMAIL/SLACK/PAGERDUTY] - Alert frequency: [THROTTLING_RULES]Please provide:- Monitoring strategy and implementation approach- Recommended alert thresholds
Industry Example
Set up monitoring and alerting for risk-management/credit-decisioning folder:1. Critical pipelines: - credit_score_refresh.orch.yaml: SLA 2 hours, business impact: Loan applications delayed costing $50K/hour - fraud_detection_realtime.tran.yaml: SLA 5 minutes, business impact: Regulatory violation if not updated in near real-time - daily_regulatory_reporting.orch.yaml: SLA must complete by 8 AM EST, business impact: SEC filing deadline, $100K penalty per day if late2. What to monitor: - Pipeline failures: Any failure in fraud detection requires immediate page to on-call engineer - Performance degradation: Credit score refresh taking >90 minutes (normal is 45 mins) - Data quality issues: NULL credit scores >1%, duplicate customer records, scores outside valid range (300-850) - Business metrics: Loan approval rate drops >10% compared to 7-day average3. Alerting requirements: - Who to notify: Data engineering team for all failures, Risk team lead for data quality issues - When to escalate: Page on-call after 15 minutes for fraud detection - Notification channels: PagerDuty for critical, Slack for warnings, Email digest for trends - Alert frequency: Immediate for critical, throttle warnings to max 1 per hour per pipeline