Observability Engineer
International Data Corporation Zobrazit všechny práce
- Praha
- Trvalý pracovní poměr
- Plný úvazek
- Build and maintain the observability platform that provides visibility into AI system performance, cost, and quality
- Build dashboards that track key operational metrics: response latency, LLM token usage, cost per query, and error rates across the agent pipeline
- Implement A/B testing infrastructure that enables controlled experiments on prompt strategies and pipeline changes
- Extend existing tracing infrastructure to provide deeper visibility into per-component behavior
- Build alerting systems that surface anomalies in cost, latency, or quality metrics
- Contribute to evaluation infrastructure alongside the senior evaluation engineer
- 3+ years of experience building observability, monitoring, or data visualization systems
- Experience with metrics collection, dashboarding, and alerting (e.g., Prometheus, Grafana, Datadog)
- Understanding of distributed tracing concepts
- Proficiency in Python
- Experience building or contributing to A/B testing or experimentation platforms
- Comfort working with large volumes of telemetry data
- Experience with LLM-specific observability (e.g., Langfuse, LangSmith)
- Background in cost optimization or resource monitoring for ML/AI systems
- Familiarity with statistical methods for experiment analysis
- Experience building custom dashboards or internal tools
- Background in platform or infrastructure engineering
At IDC, your work helps shape how the world understands technology and where it goes next. You collaborate with curious, high-caliber colleagues who value rigor, integrity, and shared success. As the premier global provider of trusted technology intelligence, IDC equips business and technology leaders with the evidence they need to make confident decisions. Our insights inform strategy, investment, and innovation across industries and regions.Recognized by IIAR as Analyst Firm of the Year for five consecutive years, IDC sets the standard for credibility and impact. With more than 1,000 analysts worldwide and a truly global perspective, we combine deep expertise with practical relevance. Here, your ideas matter, your voice is heard, and your contributions provide the insights leaders rely on every day. It is meaningful work, backed by a culture that supports growth, collaboration, and long-term career development with a globally respected brand.What We OfferHybrid/remote work model (about 1-2 days in the office per month).A position in a highly professional and globally respected market research and advisory firm, where initiative leading to results is rewarded.Individualized Culture: An environment where you can explore new areas outside your specialty and stay engaged with work you enjoy.Equal Opportunity EmployerIDC is committed to providing equal employment opportunities for all qualified persons. Employment eligibility verification required. We participate in E-Verify.#LI-SJ1