MLOps / LLMOps Engineer

📋 Job Overview

Irth Solutions is seeking an MLOps/LLMOps Engineer to build a governed, multi-cloud Lakehouse on Databricks, translating this foundation into scalable, automated, secure, and observable machine learning and LLM services. The role involves working closely with Data Science, Data Engineering, Platform, Product, and domain teams to productionize ML and GenAI capabilities supporting Irth’s key industries: Damage Prevention, Asset Integrity, Land Management, and Stakeholder Engagement.

🏢 About Irth Solutions

Irth Solutions is a leading provider of cloud-based SaaS software for damage prevention, asset integrity, stakeholder engagement and land management, helping energy, utility, telecom, and infrastructure companies protect their critical network infrastructure. With nearly three decades of industry experience, Irth serves customers across North America and continues to expand its platform with new data-driven and AI-powered capabilities.

🎯 The Role

We are looking for an MLOps/LLMOps Engineer to translate this foundation into scalable, automated, secure, and observable machine learning and LLM services. You will work closely with Data Science, Data Engineering, Platform, Product, and domain teams to productionize ML and GenAI capabilities supporting Irth’s key industries: Damage Prevention, Asset Integrity, Land Management, and Stakeholder Engagement.

✅ Key Responsibilities

• Operationalize the complete ML lifecycle—including training, evaluation, packaging, deployment, and monitoring—on Databricks.
• Implement ML workflows using the Bronze → Silver → Gold medallion architecture with Delta Lake as the underlying storage layer.
• Establish implementation patterns for Unity Catalog model management, preparing model assets for catalog-based governance, lineage, discovery, and access control.
• Design, build, and maintain production-grade LLM and RAG pipelines.
• Deploy and manage model-serving and inference endpoints.
• Optimize inference workloads for performance, scalability, reliability, and cost.
• Implement batch, streaming, and online inference patterns based on business and latency requirements.
• Integrate data contracts and quality gates into ML and LLM pipelines.
• Implement automated validation for schema drift, null thresholds, duplicate records, referential integrity, data completeness, and feature-quality issues.
• Implement PII detection, classification, masking, and obfuscation before sensitive data is consumed by features or models.
• Enforce data residency requirements through policy-as-code.
• Ensure regulated or sensitive Bronze-layer data remains in the required geographic region.
• Maintain end-to-end lineage across the ML lifecycle, including Source tables/columns → Features → Models → Serving Endpoints → Applications/BI.
• Surface lineage and governance information through appropriate monitoring and governance dashboards.
• Support security, compliance, audit, and access-review requirements across the ML platform.
• Use Databricks Asset Bundles (DABs) and GitHub Actions to version, test, and promote ML/LLM assets across environments.
• Automate promotion across DEV → QA → PROD.
• Ensure production changes are deployed through controlled CI/CD processes with no direct development in PROD.
• Build automated unit, integration, regression, and data-quality test suites.
• Implement model-quality validation as part of deployment pipelines.
• Validate business KPIs and analytical outputs against the Unity Catalog semantic layer before production publication.
• Instrument ML/LLM pipelines and services to support defined SLOs, including P1 pipeline success: ≥99.5%, P1 MTTD: ≤5 minutes, and MTTR: ≤60 minutes.
• Implement proactive monitoring, alerting, and operational dashboards.
• Integrate automated Jira ticket creation for qualifying P1 production failures.
• Monitor ML systems for model performance degradation, data drift, concept drift, feature-quality degradation, and prediction distribution changes.
• Establish LLM-specific observability, including hallucination rates, response quality, latency, token consumption, API usage, inference costs, and retrieval quality.
• Define appropriate thresholds and automated alerts for model and LLM quality degradation.
• Develop and maintain production runbooks, troubleshooting procedures, and operational documentation.
• Participate in or establish appropriate on-call rotations for critical ML/LLM services.
• Develop and maintain disaster-recovery procedures aligned with tiered RTO/RPO objectives.
• Enforce mandatory cost and ownership tags across ML/LLM infrastructure, including Domain, Tenant, Environment, Cost center, Owner.
• Ensure tags are consistently propagated across jobs, clusters, warehouses, and other applicable resources.
• Support showback and chargeback reporting for ML/LLM workloads.
• Monitor compute, storage, model-serving, and LLM/API costs.
• Identify opportunities to optimize infrastructure and inference costs without compromising service quality or SLOs.
• Establish cost visibility and budget controls for production AI workloads.
• Detect and investigate abnormal cost increases or inefficient workloads.

📌 Required Qualifications

• 3–5 years of experience in MLOps, LLMOps, ML Engineering, Data Engineering, or platform-focused ML engineering.
• Hands-on experience with Databricks, including Databricks Jobs and Workflows, Delta Lake, Unity Catalog, and Databricks SQL Warehouses.
• Proven experience building and maintaining CI/CD pipelines for data and ML workloads using GitHub Actions, Databricks Asset Bundles (DABs), and Environment promotion across DEV → QA → PROD.
• Strong understanding of data contracts, schema governance, and automated data/feature validation.
• Experience implementing Great Expectations-style validation frameworks or equivalent rule-based data-quality solutions.
• Experience building observable production pipelines, including metrics, dashboards, alerting, and monitoring against defined SLOs.
• Strong security-first mindset with practical experience in RBAC/ABAC, Unity Catalog security, PII detection and obfuscation, private networking, data-access controls, and policy-as-code for data residency.
• Strong proficiency in Python and SQL.
• Working knowledge of distributed computing and job orchestration within Databricks/Spark environments.
• Ability to troubleshoot production ML/data workloads and participate in operational support and incident resolution.

⭐ Desirable Experience

• Hands-on experience with LLM/GenAI workflows, including prompt engineering, Retrieval-Augmented Generation (RAG), LLM evaluation frameworks and evaluation harnesses, AI safety and guardrails, retrieval and response-quality evaluation, latency optimization, and token and API-cost optimization.
• Experience with geospatial data and analytics, including technologies and concepts such as PostGIS, spatial joins, spatial indexing and tiling, coordinate systems and projections, and GIS-based feature engineering.
• Experience integrating Power BI with Databricks SQL Warehouses and semantic layers, including an understanding of dataset refresh SLAs, query concurrency, Row-Level Security (RLS), and Object-Level Security (OLS).
• Practical knowledge of FinOps, including resource tagging, budget management, cost monitoring, showback/chargeback, cost anomaly detection and alerting.
• Knowledge of Databricks disaster-recovery patterns, including Delta Lake Deep Clone, Delta Sharing, cross-region recovery, tiered RTO/RPO strategies, DR testing and evidence collection.
• Hands-on experience with Microsoft Azure and AWS, particularly where ML and data workloads span both environments.
• Understanding of cloud-native security patterns, including Private Link, VPC/VNet connectivity and peering, egress restrictions, KMS, AWS Secrets Manager, Azure Key Vault, data-plane isolation.
• Ability to work effectively across cloud, platform, data, ML, security, and product teams.

🎁 Benefits

Be part of a dynamic and growing company that is well-respected in its industry. Enjoy competitive compensation based on experience and qualifications, along with health insurance coverage.

🛂 Visa & Eligibility

No visa or eligibility information provided.