Job Description

Job Responsibilities:

• Aggregate cross-functional requests and translate requests into actionable, high quality, datasets via synthetic data collections and on live user traffic.
• Deliver analytics, statistics, and model performance results on datasets
• Design and conduct experiments for evaluating rater quality, inter-rater reliability; write summary reports based on findings and speak on those findings in meetings.
• Develop manual and automated processes for multiple concurrent projects ensuring high-quality labeled data.
• Create and perfect LLM Quality grading queues and LLM related component quality error attribution grading queues.
• Build datasets and write guidelines in rapid fashion for new and emergent LLM data rating, creation, and annotation needs.
• Analyze system metrics such as factuality, brevity, coherence, as well as subcomponent performance, summarize findings, and communicate results.
• Dataset Building & Curation: Receive requests from engineers and research scientists, then design and execute synthetic data collections or sample from
• existing data to produce clean, labeled datasets for LLM training and evaluation.
• Annotation Queue Management: Stand up and maintain grading queues — define rubrics, write annotation guidelines, onboard raters, and iterate on queue design
• to keep pace with rapidly evolving LLM features.
• Quality Auditing & Error Attribution: Review annotated data daily, flag inconsistencies, run inter-rater reliability checks, and trace quality issues back to specific LLM subcomponents (e.g., factuality, coherence, brevity).
• Data Analysis & Reporting: Use AI to write SQL queries and Python scripts to pull metrics, compute agreement scores, and generate dashboards; summarize findings in
• weekly reports shared with cross-functional partners.
• Experiment Design: Plan and run controlled experiments to test rater calibration, guideline clarity, or model performance changes — then present results
• and recommendations in team meetings.
• Guideline Authoring: Draft, test, and refine annotation guidelines for new LLM capabilities on tight timelines, ensuring raters can consistently apply them at scale.
• Cross-Functional Collaboration: Attend syncs with engineering, research, and PM stakeholders to align on data priorities, surface quality insights, and inform roadmap decisions for smart glasses LLM features.

Skills:

• Basic Python, basic SQL with strong desire to grow required for this position. Intermediate level preferred.
• Basic knowledge of linguistics (some of Phonetics, Syntax, Semantics, Dialectology) required.
• Ability to analyze numerical rating results and draw conclusions based on data.
• Basic stats / math for data analysis.
• Written and oral communication skills necessary for communicating on project milestones and for reporting findings .
• Data Annotation
• SQL Python
• Linguistic Background
• Computational Linguistics (master’s preferred)
• Statistics & Data Science
• 4-6 year of experience

Education/Experience:

• Bachelor’s degree in Linguistics, Computational Linguistics, Speech Science, or related field OR graduate degree in linguistics OR industry similar experience required.
• Master’s degree or beyond preferred OR equivalent industry experience.
• Additional learning via Coursera, Udemy, or other online resources is great too.

Job Tags

Similar Jobs