The Role:
You’ll help build the data foundation of our product: high‑volume web scraping systems, structured datasets and LLM‑driven processing pipelines. The role combines hands‑on engineering with architectural thinking and suits someone who enjoys turning messy web data into reliable, scalable outputs.
Key Responsibilities:
Build new structured datasets, including scraping accelerators, Form D filings and dynamic web sources.
Develop automated ETL pipelines that parse, clean and transform content using LLMs.
Define and maintain database schemas in Supabase or PostgreSQL.
Create evaluation frameworks to measure and compare LLM performance across pipeline components.
Contribute to the design of scalable data architectures using GCP services.
Improve reliability, observability and deployment workflows for scraping and data processing systems.
Requirements:
4+ years of experience building data pipelines, backend services and automated data processing systems.
Strong background in web scraping with tools like Scrapy, Playwright or similar.
Experience deploying pipelines on cloud platforms such as GCP or AWS.
Solid knowledge of ETL frameworks, workflow orchestration (Airflow) and modern data stores (BigQuery, PostgreSQL).
Comfortable working with Docker and API frameworks like FastAPI.
Clear, fluent communication in English.
NirYu is an AI platform company that builds and operates AI agents in production environments. The company develops a core platform layer that powers multi-tenant AI agents, with a focus on sophisticated backend infrastructure including LangGraph-based orchestration, agent configuration schemas, data pipeline management, and automated evaluation systems. NirYu provides enterprise customers with customizable AI agent deployments, knowledge base integration, and conversation logging capabilities. The company enables rapid customer onboarding through configuration-driven deployment models and maintains cross-client isolation across multi-tenant environments.
