Software Test Engineer (AI) - TikTok Search

About the role

TikTok Search Engineering Test Team owns the end-to-end quality of TikTok's global search experience, from query understanding, retrieval and ranking to generative search (LLM Answers and RAG) and search safety. The team is redefining search quality with Large Language Models, AI agents and intelligent automation, and is building AI-powered evaluation and testing capabilities. As a Software Test Engineer (AI) on the Search team, you will sit at the intersection of Search Engineering, Quality Engineering and Generative AI.

Key responsibilities

• Design and deliver AI-powered testing and evaluation solutions for TikTok Search, covering query understanding, retrieval, ranking, generative search, personalization and multimodal and vertical search scenarios (video, image, e-commerce, user, creator).
• Build LLM-based evaluation pipelines for relevance, answer faithfulness, groundedness, freshness, coverage, diversity and safety; maintain multilingual, multi-market query sets and golden datasets.
• Drive change impact analysis across the search stack through AI-assisted diff analysis, offline replay, side-by-side evaluation and interleaving experiments, and strengthen release quality gates.
• Partner with search engineering, algorithm, product and content operations teams to identify quality pain points and turn practical AI-powered solutions into production.
• Improve engineering productivity by building AI agents and tooling for automated test case generation, log-driven bug analysis, root-cause attribution and self-healing regression suites.

Required profile

• Bachelor's degree or above in Computer Science, Software Engineering, Information Retrieval, Data Science or a related technical discipline.
• Experience in software quality assurance or testing, preferably for search, recommendation, ranking or other large-scale data- or algorithm-driven systems.
• Strong interest in Generative AI and LLMs, with an understanding of evaluation paradigms such as LLM-as-a-Judge, RAG evaluation and human evaluation.

Required skills

• Familiarity with distributed systems and search/recommendation infrastructure (inverted indexes, vector indexes, retrieval, ranking, embeddings) and basic knowledge of iOS and/or Android.
• Hands-on experience in automation testing, performance and stress testing, algorithm evaluation or test platform/tool development.
• Preferred: proficiency in Python, Go, Java or C++; experience with search, recommendation or LLM/RAG evaluation (relevance labeling, SBS evaluation, NDCG, MRR, Recall, hallucination detection); experience with test automation frameworks, query set management tools or AI-assisted testing; experience with multilingual, cross-market or vertical search quality work.