AI Research Peer Review Evaluator (ML/AI)

LA AI Research Peer Review Evaluator (ML/AI)
Save

Lightly is a data curation platform that helps machine learning teams select the most impactful data for training, reducing labeling costs and improving model performance.
Lightly AG
Employee count: 1-10
Switzerland only

Stay safe on Himalayas

Never send money to companies. Jobs on Himalayas will never require payment from applicants.

Lightly AG is a Zurich-based AI company and ETH/HSG spin-off, backed by Y Combinator and top-tier investors. Our machine learning and computer vision technology is trusted by global leaders in autonomous driving, medical imaging, and visual inspection.

We’re looking for researchers with strong Machine Learning / AI backgrounds to support an AI evaluation project focused on scientific peer review. You’ll evaluate reviews generated by agentic AI systems and compare them against expert human peer reviews of ML/AI research papers.

This is a remote, project-based contractor opportunity with flexible working hours.
Tasks
What you'll be doing
- Read and scan ML/AI research papers to understand their core contributions, methodology, experiments, and claims
- Review the original human peer reviews to establish an expert baseline for each paper
- Evaluate AI-generated peer reviews against that baseline using a structured scoring rubric
- Assess the technical accuracy, analytical depth, constructive value, and novelty/significance assessment of each AI review
- Identify hallucinations, unsupported claims, missed technical issues, or valuable insights surfaced by the AI reviewers
- Compare two AI-generated reviews side-by-side and determine where one provides stronger or more useful analysis
- Search and verify relevant academic literature using sources such as Google Scholar, arXiv, or Semantic Scholar, including checking whether cited prior work was available before the paper’s submission date
- Provide concise, evidence-based rationales explaining your evaluation decisions and consistently apply the project rubric

The evaluation specifically looks at whether agentic AI reviewers can provide meaningful value beyond expert human reviewers—for example, by identifying relevant prior literature that humans missed, questioning important assumptions, or resolving inconsistencies using evidence.
Requirements
You're a strong candidate if you:
- Have a Master’s, PhD, or are currently pursuing graduate study in Machine Learning, Artificial Intelligence, Computer Science, Statistics, or a closely related technical field
- Have contributed to at least one scientific/research paper , ideally as a first author, although co-authors and other substantial contributors are also welcome
- Have experience critically reading ML/AI research papers , including evaluating methodology, experimental design, results, limitations, and scientific claims
- Are familiar with major ML/AI research venues , such as NeurIPS, ICML, ICLR, ACL, CVPR, or comparable conferences and journals
- Have prior academic peer-review experience , ideally for an ML/AI conference or journal — strongly preferred
- Are comfortable conducting academic literature searches and verifying prior work, publication dates, citations, and novelty claims
- Have strong analytical and written communication skills and can distinguish meaningful technical concerns from superficial criticism
- Can provide clear, concise, evidence-based rationales for your decisions
- Can consistently apply detailed evaluation guidelines and scoring rubrics across multiple papers and reviews
- Have strong attention to detail , particularly when identifying factual inaccuracies or hallucinated technical claims

Benefits
- Fully remote and flexible — work from anywhere
- Part-time contractor role with flexible hours
- Work directly on the evaluation of cutting-edge agentic AI systems for scientific research
- Apply your ML/AI research expertise to help measure and improve the quality of AI-generated scientific peer review

Send us your CV along with a brief note about your research background and areas of expertise. Please include any relevant publications, as well as previous peer-review experience for conferences, journals, workshops, or similar academic venues.

If applicable, we'd also love to know which ML/AI research areas and conferences you’re most familiar with.
We look forward to hearing from you!

Apply now
Job expired?

Please let Lightly AG know you found this job on Himalayas. This helps us grow!

Apply now
About the job
Apply before
Nov 19, 2026
Posted on
Sep 20, 2026
Job type
Contractor
Experience level
Senior

Education
Postgraduate degree

Location requirements
Suggest an edit

Switzerland

Hiring timezones
Show UTC offsets

Switzerland +/- 0 hours

Job categories
AI Research ML AI Research Scientific Peer Review Research and Evaluation Academic Research AI Research Reviewer AI Research Evaluator AI ML Evaluation Specialist AI Research Evaluation AI Evaluation Specialist AI Evaluation Analyst AI Model Evaluation Specialist

Skills
Machine Learning AI Computer Vision Scientific Peer Review ArXiv Semantic Scholar Google Scholar Rubric Technical Assessments Computer Science Statistics Research Review Methodology Experimental Design ICML ACLs Agentic AI

Browse similar jobs
Remote Senior AI-Research Jobs Remote Contractor AI-Research Jobs Remote Senior AI-Research Jobs in Switzerland Remote Contractor Jobs in Switzerland Remote AI-Research Jobs in Switzerland

About Lightly AG

Learn more about Lightly AG and their company culture.

View company profile

Our customers face a critical challenge in the age of artificial intelligence: the available data for deep learning is growing exponentially, far outpacing human labeling capabilities and available compute power. At Lightly, we understand that more data doesn't always mean better models. In fact, redundant and low-quality data can hinder performance and skyrocket costs. That is why we built a data curation platform designed to help machine learning teams identify and work with only the most relevant samples.

We empower engineering teams to analyze and curate their datasets before model training, ensuring that every labeled sample contributes to higher accuracy. By filtering out redundancy and bias, our solution enables companies to achieve up to 20% higher model accuracy while reducing data-related costs by up to 10x. Our platform integrates seamlessly into existing workflows, supporting data types ranging from images and video to LiDAR and voice. Whether you are developing autonomous vehicles, robotics, or advanced video analytics, Lightly provides the tools to turn your raw data into a high-performance asset, accelerating your path to production-ready AI.

Tech stack

Learn about the tools and technologies that Lightly AG uses to build, market, and sell its products.

View tech stack

Python

C

TypeScript

Google Cloud Platform

Kubernetes

Terraform

React

PyTorch

Docker

Lightly AG employees can create an account to update this tech stack.

Employee benefits

Learn about the employee benefits and perks provided at Lightly AG.

View benefits
Equity benefits

Attractive cash and equity packages for employees.

Flexible working hours

Remote work options and flexible schedules are available.

Open source

Active contributions to open source projects like LightlySSL and LightlyTrain.

View Lightly AG 's employee benefits

Apply now
Job expired?

Please let Lightly AG know you found this job on Himalayas. This helps us grow!
Apply now

About the job
Apply before
Nov 19, 2026
Posted on
Sep 20, 2026
Job type
Contractor
Experience level
Senior

Education
Postgraduate degree

Location requirements
Suggest an edit

Switzerland

Hiring timezones
Show UTC offsets

Switzerland +/- 0 hours

Job categories
AI Research ML AI Research Scientific Peer Review Research and Evaluation Academic Research AI Research Reviewer AI Research Evaluator AI ML Evaluation Specialist AI Research Evaluation AI Evaluation Specialist AI Evaluation Analyst AI Model Evaluation Specialist

Skills
Machine Learning AI Computer Vision Scientific Peer Review ArXiv Semantic Scholar Google Scholar Rubric Technical Assessments Computer Science Statistics Research Review Methodology Experimental Design ICML ACLs Agentic AI

Browse similar jobs
Remote Senior AI-Research Jobs Remote Contractor AI-Research Jobs Remote Senior AI-Research Jobs in Switzerland Remote Contractor Jobs in Switzerland Remote AI-Research Jobs in Switzerland

Claim this profile LA Lightly AG

Company size

1-10 employees

Founded in

2018

Chief executive officer

Matthias Heller, Igor Susmelj

Markets
AI Model Optimization Big Data AI Machine Learning Data Curation Computer Vision Robotics Video Analytics Data Management Autonomous Vehicle

Employees live in
Switzerland

View company profile

Similar remote jobs

Here are other jobs you might want to apply for.

View all remote jobs

6 remote jobs at Lightly AG

Explore the variety of open remote roles at Lightly AG, offering flexible work options across multiple disciplines and skill levels.

View all jobs at Lightly AG

Top remote companies
Remote companies like Lightly AG

Find your next opportunity by exploring profiles of companies that are similar to Lightly AG. Compare culture, benefits, and job openings on Himalayas.

View all companies

Get matched with your dream remote job

Sign up now and join over 250,000+ remote workers who receive personalized job alerts, curated job matches, and more for free!

Sign up with Google Sign up

Save Apply now