AI Conversation Evaluation SME at Turing

AI Conversation Evaluation Specialist (SME) | Remote (Latin America, Southeastern Europe, Asia, or Africa) | Contractor, pay varies

Turing, an AI company working with leading AI labs, is hiring Subject Matter Experts to review conversations between participants and AI models, assess whether sufficient context was gathered before recommendations were made, and provide structured feedback to improve conversational quality.

What the work involves

• Review AI conversations involving real-world interpersonal and everyday decision-making scenarios.
• Identify missing context and recommend relevant follow-up questions.
• Evaluate whether additional information would have changed the AI's recommendations.
• Distinguish between critical (“load-bearing”) and non-essential information.
• Conduct qualitative reviews and document findings using structured evaluation guidelines.
• Collaborate with research teams to ensure consistent, high-quality assessments.

Required qualifications

At least 3–5 years of direct professional experience in one or more of the following areas:

• Licensed Social Work: MSW or equivalent graduate qualification plus LCSW, LICSW, LMSW, or equivalent license; experience with client assessment, case review, interviewing, or counseling.
• Counseling, Therapy, or Psychology: MA/MS with LMFT, LPC, LCPC, or equivalent license, or PhD/PsyD in Clinical, Counseling, Social, or Industrial-Organizational Psychology; client-facing experience in counseling, therapy, assessment, interviewing, or case review.
• Qualitative Research, HCI, or Communication: MA/PhD or equivalent advanced expertise in HCI, Communication, Anthropology, Sociology, or a closely related discipline; experience with qualitative research, interviews, user research, conversation analysis, thematic analysis, or qualitative coding.
• Mediation, Coaching, Student Services, or Organizational Consulting: relevant qualification, credential, or demonstrated subject-matter expertise; significant direct advisory experience gathering context, understanding needs, and providing informed recommendations.

Preferred experience

• Eliciting relevant context when clients initially provide incomplete information.
• Working across relationships, work, education, family, planning, wellbeing, and resource constraints.
• Qualitative coding or structured case review.
• Reviewing interviews, consultations, coaching sessions, or client interactions to identify missed opportunities for clarification.

Location

Applicants must be based in Latin America, Southeastern Europe, Asia, or Africa. The role is fully remote.