Job Summary We are looking for a skilled and motivated Senior Engineer to join our forward-thinking team. The ideal candidate is passionate about building cutting-edge AI solutions, has a strong technical foundation, and excels in collaborative environments. As an Senior Engineer, you will play a key role in designing, developing, and maintaining advanced data products and AI-powered applications. You’ll work closely with cross-functional teams—including Data Scientists, Analytics Engineers, Front-End Developers, and ML Engineers—to deliver scalable and impactful solutions.
Now, if you were to come on board as our Senior Engineer, we’d ask you to do the following:
- Design and implement scalable, fault-tolerant systems capable for crawl, scrape, extract, and normalize data from diverse web sources at massive scale.
- Build robust pipelines to handle structured and semi-structured data seamlessly.
- Analyze and fine-tune crawling strategies to maximize data coverage, freshness, and quality.
- Conduct research to stay ahead of the curve in large-scale web acquisition and processing techniques.
- Take ownership of end-to-end engineering projects in data acquisition, including web crawling, data ingestion, and search.
- Continuously improve reliability, observability, and performance of data systems and tooling over time.
- Partner with cross-functional teams to prioritize crawling targets that align with key business needs.
Think you have what it takes to be our Senior Engineer? Here’s how we’ll know:
- 3+ years of industry experience working on large-scale, complex web scraping projects.
- Strong background as a backend or full-stack software engineer.
- Deep understanding of how to crawl the internet at scale.
- Strong programming skills, especially in Python and Go.
Bonus Points If You Have:
Expertise in internet-scale technologies such as DNS, networking, CDNs, WAFs, CAPTCHAs, proxies, and reverse proxies.
Experience with event-driven architectures.
Experience building observability and monitoring tools that support both engineers and non-technical stakeholders.
Exposure to large language models (LLMs) and their applications.
