From The Lab.

COCO vs YOLO: Which Annotation Format Is Right for Your Pipeline?

Choosing the wrong annotation format isn't just an inconvenience — it's a project delay waiting to happen. COCO JSON and YOLO TXT are the two dominant formats in computer vision, but they were built for very different use cases, and the decision you make here echoes through your entire training pipeline.

COCO JSON gives you a richly structured format with support for bounding boxes, polygons, keypoints, and segmentation masks all in a single file. It's the format of choice for complex, multi-class datasets where you need to capture fine-grained object boundaries. The trade-off is file size and parsing overhead — COCO files grow fast, and loading a large dataset at training time requires careful handling.

YOLO TXT takes the opposite approach. One text file per image, one line per object, normalised coordinates. It's lean, fast, and modern YOLO architectures are optimised to consume it directly. If you're building a real-time inference pipeline — edge deployment, robotics, live video — YOLO format removes unnecessary overhead.

The conversion gotcha nobody warns you about: COCO stores polygon coordinates as absolute pixel values, while YOLO expects normalised centre-x, centre-y, width, height. Converting between them for complex segmentation masks is non-trivial and introduces rounding errors at scale. If you know your target model architecture upfront, annotate directly in that format from day one.

What is Inter-Annotator Agreement — And Why 95% Isn't Enough

Inter-Annotator Agreement (IAA) is the metric that tells you how consistently your annotators are applying your guidelines. It's the single number that most accurately predicts downstream model performance — and the most commonly misunderstood quality metric in the industry.

Cohen's Kappa measures agreement between two annotators adjusted for chance. Fleiss' Kappa extends this to three or more. Both are more meaningful than raw percentage agreement, which is inflated by class imbalance — if 90% of your images contain no object, two annotators who always label "nothing" achieve 90% agreement without any actual agreement on the hard cases.

Here's the critical insight: a 98% IAA on a dataset where 95% of examples are clear-cut means your annotators disagree on nearly half the ambiguous edge cases. Those edge cases are exactly the scenarios your model will encounter first in production — unusual lighting, partial occlusion, out-of-distribution objects. High overall IAA can hide catastrophic disagreement precisely where it matters most.

We recommend reporting IAA separately for easy, medium, and hard examples. It takes more setup, but it's the only way to catch quality problems before they become model problems.

LiDAR Point Cloud Annotation: A Practical Guide for Robotics Teams

3D cuboid annotation in point cloud data is fundamentally different from drawing bounding boxes on images, and teams that treat it like an image task routinely ship datasets that degrade model performance. The mistakes are more expensive because they're harder to spot during QA and propagate invisibly into downstream sensor fusion steps.

The most common error: annotators tighten cuboids to the visible point cloud density rather than the physical object boundary. A vehicle in sparse LiDAR returns may have a visible cloud occupying only 60% of its actual frame. If your annotator snaps the cuboid to the visible returns, you're training your model to perceive vehicles as smaller than they are — with predictable consequences for path planning margin calculations.

Our annotation spec requires annotators to estimate the full physical extent of the object using all available cues: camera fusion, object class priors, and adjacent frame continuity. For moving objects, we annotate in bird's-eye view first, then verify in front and side projections before finalising.

RLHF From Scratch: What Your Human Annotators Actually Need to Know

Reinforcement Learning from Human Feedback sounds conceptually simple: humans rank or rate model outputs, those signals train a reward model, and that reward model guides further fine-tuning. In practice, the quality of your RLHF data is determined almost entirely by how well your annotators understand the task — and most annotator briefings vastly underestimate what that understanding requires.

The most critical concept to convey is implicit preference calibration. When an annotator rates two responses, they're not just evaluating what's written — they're encoding a value judgement about what a helpful, honest, and appropriate response should look like. Without explicit calibration sessions, annotators default to personal preferences, which vary wildly across demographic groups and produce noisy, inconsistent reward signals.

Domain expertise matters more than general writing quality judgement. An RLHF task for a medical information model requires annotators who can evaluate clinical accuracy, not just fluency. A coding assistant task requires annotators who can test whether the code actually runs. Staffing RLHF projects with general-purpose raters is one of the most common and costly mistakes we see.

For ambiguous cases — and there are always ambiguous cases — a structured disagreement protocol beats majority vote. When annotators disagree on a preference pair, surface it for senior adjudication with a written rationale requirement. These disagreements are your most valuable signal for guideline refinement.

Synthetic Data Is Only as Good as Its Human Validation Layer

The promise of synthetic data is compelling: near-infinite scale, perfect label accuracy, controllable distribution. The reality is more complicated. Generated datasets consistently ship three categories of problems that automated QA misses and that degrade real-world model performance in ways that are difficult to diagnose: distribution gaps, visual artefacts, and physics violations.

Distribution gaps occur when your synthetic data generator doesn't accurately model the statistical properties of real-world data. A synthetic pedestrian dataset generated from a 3D engine with limited character assets will underrepresent body size diversity, clothing variation, and motion patterns in ways that are invisible to automated metrics but immediately apparent to a trained human reviewer.

Visual artefacts — texture seams, incorrect shadow directions, physically impossible reflections — are reliably caught by human reviewers in seconds but missed by automated perceptual quality metrics. We've seen artefact rates above 12% in synthetic datasets that passed automated QA with scores above 95%.

The human validation layer we recommend: a stratified sample review covering at least 5% of generated images, reviewed by annotators with domain expertise in the target environment. For any artefact category appearing in more than 0.5% of reviewed samples, the full dataset should be flagged for generator-level correction before training begins.