Database Engineer

About the role

Ahrefs owns the data layer behind one of the largest crawlers on the web, which holds hundreds of petabytes of data stored on about 1,200 servers across data centres in Singapore, the US and the UK. The Database team owns databases and storage end to end, from source code to production, and engineers build and run the data layer themselves rather than relying on vendor support.

Key responsibilities

• Build and run ClickHouse, the MooseFS and SeaweedFS storage underneath it, and the TiDB and Elasticsearch clusters
• Build storage and query features in ClickHouse, such as replication over a shared file system and storage policies based on merge history
• Keep up with upstream ClickHouse LTS releases, keep CI green and ship regular releases
• Make MooseFS and SeaweedFS highly available and durable, including Raft-based metadata servers, follower reads, an intent log and automatic re-replication
• Find and fix performance problems in production profiles, trace data corruption and repair damaged data
• Roll out releases and storage migrations across 1,200 servers in three regions without downtime
• Run and tune TiDB, TiKV, TiCDC and Elasticsearch, including scheduling, replication lag, backups and disk balance
• Check risky designs with TLA+ and fault injection, and automate recurring tasks with watchdogs, balancers and dashboards
• Build an SRE agent that monitors database clusters, diagnoses incidents and proposes fixes

Required profile

• Strong C++, C and Rust, able to work in Go, in codebases of millions of lines
• Deep knowledge of database storage engines: merges, compaction, replication logs, query execution
• Understanding of distributed consensus (Raft or Paxos) and failure scenarios
• Knowledge of Linux storage internals: page cache, fsync, FUSE, file systems, NVMe and HDD behaviour
• Experience running databases or storage on hundreds of nodes and leading incident recovery
• Debugging with gdb, perf and sanitizers; measuring changes on production data

Required skills

• C++, C, Rust, Go
• ClickHouse, TiDB, TiKV, TiCDC, Elasticsearch, MooseFS, SeaweedFS
• Good to have: contributions to an open-source database or file system, OCaml, Puppet or a similar configuration management tool

The role is on-site in Singapore, Tokyo, Palo Alto (CA) or Cambridge (MA); candidates need the right to work in the chosen office. Ahrefs does not use agencies or third-party recruiters for this role.