2-5

Quality Assurance / SDET Engineer, GenAI/AI Platform

2/7/2026

The role involves building and scaling automated test frameworks for Generative AI features across the observability platform. Responsibilities include defining test strategies, automating validation processes, and ensuring the quality and performance of AI-driven workflows.

Working Hours

40 hours/week

Company Size

1,001-5,000 employees

Language

English

Visa Sponsorship

About The Company

We’re in the midst of an information revolution, and monitoring is at the center of it. Businesses are moving from asking, “What happened?” to predicting what’s coming, solving problems before they start and using data to unlock opportunities. At LogicMonitor, we’re committed to expanding what’s possible for businesses by advancing their technology. After all, monitoring shouldn’t just help businesses see what’s in front of them—it should create new ways for them to grow. Here’s how we live our vision: Turn It On - Today, monitoring keeps the world of business turning. LogicMonitor comprehensively monitors infrastructures, from on-premises to the cloud. Within minutes, we make it possible for businesses to advance their technology, ensuring airline passengers aren’t stranded due to system outages, food stays at safe temperatures and spinning wheels don’t interrupt our favorite streaming shows. Turn the Dial - We believe monitoring should do more than visualize today’s information—it should open up new ways for businesses to grow. We go above and beyond to empower our customers, seeking to be the most extensible monitoring platform in the world. When our customers can solve problems before they start, they can move their focus from maintenance to innovation. Turn the Corner - As artificial intelligence and machine learning turn from sci-fi promises to everyday realities, our technology is enabling businesses to see what’s coming before it happens. We collaborate closely with our customers to understand their risks and anticipate their needs, providing insights that unlock their vision. Our company has a global presence with offices in Santa Barbara, Austin, London, India, Australia, and Singapore.

About the Role

About Us:   We love going to work and think you should too. Our team is dedicated to trust, customer obsession, agility, and striving to be better everyday. These values serve as the foundation of our culture, guiding our actions and driving us towards excellence. We foster a culture of performance and recognition, allowing us to transform growth as we enable our employees to do the best work of their careers. This position is located in Pune. You'll be working in a major tech center of Pune, India. Across the globe, our Centers of Energy serve as hubs where we accelerate productivity and collaboration, inspire creativity, and cultivate a culture of connection and celebration. Our teams coordinate their time in Centers of Energy to reflect how they work best. To learn more about life at LogicMonitor, check out our <a href="https://www.logicmonitor.com/careers">Careers Page</a>. What You'll Do: LogicMonitor® is the AI-first hybrid observability platform powering the next generation of digital infrastructure. LogicMonitor delivers complete visibility and actionable intelligence across on-premises, cloud, and edge environments. By anticipating issues before they strike, optimizing resources in real time, and enabling faster, smarter decisions, LogicMonitor helps IT and business leaders protect margins, accelerate innovation, and deliver exceptional digital experiences without compromise. Our customers love LogicMonitor's ability to bring cloud and traditional IT together into one view, as seen in minimal churn rates, expansion business, and exciting new customer references. In fact, LogicMonitor has received the highest Net Promoter Score of any IT Infrastructure Management provider. LogicMonitor also boasts high employee satisfaction. We have been certified as a Great Place To Work®, and named one of BuiltIn's Best Places to Work for the seventh year in a row!  <div data-hasbody="true" data-macro-name="panel"> <div data-hasbody="true" data-macro-name="panel">LogicMonitor is looking for a highly skilled QA/ SDET with 2 to 3 years of experience to build and scale automated test frameworks for Generative AI features across our observability platform.In this role, you will ensure the quality, reliability, safety, and performance of LLM-based workflows, including AI assistants, Retrieval-Augmented Generation (RAG) pipelines, AI-generated incident summaries, auto-remediation agents, tool calling, and AI-driven insights. You will work closely with engineering, product, and applied AI teams to validate AI experiences in production.</div> Here's a closer look at this key role: <h3>1. Test Strategy for GenAI Features</h3> <ul> <li>Define end-to-end test strategies for GenAI-driven product features, including:</li> <ul> <li>AI assistant chat flows (multi-turn conversations)</li> <li>AI-generated RCA summaries and incident timelines</li> <li>RAG-based responses using knowledge base, tickets, and observability signals</li> <li>Agent execution flows (tool calling, action orchestration)</li> </ul> <li>Establish quality standards for AI output across:</li> <ul> <li>Factuality, relevance, completeness, groundedness</li> <li>Hallucination risk mitigation</li> <li>User trust and explainability</li> </ul> </ul> <h3>2. Automation Framework & End-to-End Testing</h3> <ul> <li>Build scalable automation test frameworks for API and UI experiences.</li> <li>Automate validation of:</li> <ul> <li>AI endpoints (REST)</li> <li>Orchestration workflows</li> <li>Streaming behaviors</li> <li>Structured response schemas (JSON, Pydantic models, etc.)</li> </ul> <li>Develop regression test packs that run in CI/CD pipelines and validate:</li> <ul> <li>Deterministic system behavior around non-deterministic model outputs</li> </ul> </ul> <h3>3. AI Evaluation (LLM Testing)</h3> <ul> <li>Create and maintain LLM evaluation test suites, including:</li> <ul> <li>Golden datasets (prompt → expected response patterns)</li> <li>Rubric-based scoring (LLM judge + deterministic validation checks)</li> <li>Failure taxonomy (hallucinations, irrelevant retrieval, refusal bugs, etc.)</li> </ul> <li>Build automated pipelines for:</li> <ul> <li>Drift testing</li> <li>Prompt regression testing</li> <li>Retrieval quality regression testing</li> </ul> </ul> <h3>4. Reliability, Performance & Scale Testing</h3> <ul> <li>Design and implement performance and load tests for:</li> <ul> <li>High concurrency chat experiences</li> <li>Streaming response latency</li> <li>Tool execution latency</li> <li>RAG query throughput</li> </ul> <li>Ensure AI systems consistently meet SLOs and performance targets.</li> </ul> <h3>5. Safety, Security & Compliance Testing</h3> <ul> <li>Validate AI system robustness against:</li> <ul> <li>Prompt injection attacks</li> <li>System prompt leakage</li> <li>Cross-tenant data access risks</li> <li>Unsafe tool execution</li> <li>PII and sensitive data exposure</li> </ul> <li>Build guardrail validation tests for:</li> <ul> <li>Safe refusal behavior</li> <li>Policy compliance</li> <li>Approval flows for state-mutating agent actions</li> </ul> </ul> <h3>6. Observability & Debuggability for AI Testing</h3> <ul> <li>Collaborate with engineering teams to enhance AI observability:</li> <ul> <li>Tracing across agents and tool calls</li> <li>Prompt and tool execution logging</li> <li>Retrieval traceability logs</li> <li>Model output diffing</li> </ul> <li>Use monitoring and telemetry to detect regressions and report actionable issues.</li> </ul> <div>What You'll Need: <ul> <li>2+ years of experience as an SDET / QA Automation Engineer</li> <li>Strong programming skills in Python or Java</li> <li>Strong hands-on automation experience with:</li> <li>Proven ability to build test strategies for:</li> <li>Experience testing LLM-based systems, such as:</li> <li>Strong understanding of common GenAI failure patterns:</li> <li>Ability to create evaluation datasets and rubrics for AI correctness</li> </ul> Click <a href="https://www.logicmonitor.com/careers/eea-recruitment-privacy">here </a>to read our International Applicant Privacy Notice. <span data-sheets-root="1" data-sheets-value="{"1":2,"2":"Anticipated Application Close Date: xx/xx/xx"}" data-sheets-userformat="{"2":14915,"3":{"1":0},"4":{"1":2,"2":16316664},"9":0,"12":0,"14":{"1":2,"2":1907741},"15":"Slack-Lato, Slack-Fractions, appleLogo, sans-serif","16":11}">LogicMonitor is an Equal Opportunity Employer <span data-sheets-root="1" data-sheets-value="{"1":2,"2":"Anticipated Application Close Date: xx/xx/xx"}" data-sheets-userformat="{"2":14915,"3":{"1":0},"4":{"1":2,"2":16316664},"9":0,"12":0,"14":{"1":2,"2":1907741},"15":"Slack-Lato, Slack-Fractions, appleLogo, sans-serif","16":11}">At LogicMonitor, we believe that innovation thrives when every voice is heard and each individual is empowered to bring their unique perspective. We’re committed to creating a workplace where diversity is celebrated, and all employees feel inspired and supported to contribute their best. <span data-sheets-root="1" data-sheets-value="{"1":2,"2":"Anticipated Application Close Date: xx/xx/xx"}" data-sheets-userformat="{"2":14915,"3":{"1":0},"4":{"1":2,"2":16316664},"9":0,"12":0,"14":{"1":2,"2":1907741},"15":"Slack-Lato, Slack-Fractions, appleLogo, sans-serif","16":11}">For us, equal opportunity means fostering a truly inclusive culture where everyone has the chance to grow and succeed. We don’t just open doors; we invite you to step through and be part of something bigger. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, or status as a protected veteran. Notice Regarding Use of AI in Hiring We use artificial intelligence tools to assist with reviewing job applications, such as matching skills and experience to job requirements. These tools support, but do not replace, human review. All hiring decisions are made by our recruiting and hiring teams. You may opt out of AI processing at any time, and your application will still be reviewed. To opt out, please contact us at <a href="mailto:opt.out@logicmonitor.com">opt.out@logicmonitor.com</a>.  By submitting your application, you acknowledge this notice. </div> </div><div class="content-conclusion">                                                Our goal is to ensure an accessible and inclusive experience for every candidate. If you need a reasonable accommodation during the application or interview process under applicable local law, please submit a request via this <a href="https://docs.google.com/forms/d/e/1FAIpQLScvuS3dAsg2JaXD-BU6OqwFGQAWKy2So9oi27FriN9NzuAUxA/viewform?usp=sharing">Accommodation Request Form</a>. <a href="https://drive.google.com/file/d/1rZFgJPjPudLaX39IlZi-7CODysxDpQPR/view?usp=sharing" target="_blank">Know your rights: workplace discrimination is illegal. </a>Please <a href="https://drive.google.com/file/d/1GBtwrTd2u4LxWwCyP1KvVSgg9WV84I-u/view?usp=sharing">click here</a> to review LogicMonitor’s U.S. Pay Transparency Nondiscrimination Provision.</div>

Key Skills

Quality AssuranceSDETAutomation TestingPythonJavaTest StrategyLLM TestingPerformance TestingLoad TestingAI EvaluationRegression TestingObservabilityDebuggabilitySafety TestingSecurity TestingCompliance Testing