Real-Time Object Detection in 2026: RF-DETR, YOLO26, SAM 3

This is a working map of real-time detection and segmentation as it stands in mid-2026: which models are actually worth building on, what they cost in milliseconds, and which licenses will bite you six months into a commercial project. It is written for engineers who have to ship a detector on real hardware, not for people benchmarking on a leaderboard.

Two things changed the shape of this space in the last eighteen months. Transformer detectors finally became genuinely real-time, so the "DETR is accurate but slow" heuristic is dead. And concept-level segmentation arrived in a form you can prompt with a noun phrase, which collapses a large part of the traditional label-then-train loop into a single inference call.

The accuracy versus latency frontier, in numbers

The two models most teams are choosing between are RF-DETR from Roboflow and the current Ultralytics YOLO generation, YOLO26. Both publish T4 latency at batch size 1, so the numbers are roughly comparable, with the caveat that RF-DETR quotes TensorRT FP16 and Ultralytics quotes TensorRT 10, and neither figure includes the pre and post-processing in your actual pipeline.

Model	License	COCO mAP50:95	T4 latency (bs1)	Notes
RF-DETR Nano	Apache 2.0	48.4	2.3 ms	DINOv2 backbone
RF-DETR Small	Apache 2.0	53.0	3.5 ms
RF-DETR Medium	Apache 2.0	54.7	4.4 ms
RF-DETR Large	Apache 2.0	56.5	6.8 ms	best Apache accuracy/latency point
RF-DETR 2XL	PML 1.0	60.1	17.2 ms	requires rfdetr[plus]
YOLO26n	AGPL-3.0 or commercial	40.9	1.7 ms	38.9 ms CPU ONNX
YOLO26s	AGPL-3.0 or commercial	48.6	2.5 ms	87.2 ms CPU ONNX
YOLO26m	AGPL-3.0 or commercial	53.1	4.7 ms	220 ms CPU ONNX
YOLO26x	AGPL-3.0 or commercial	57.5	11.8 ms	525.8 ms CPU ONNX
RTMDet-l	Apache 2.0	52.8	not published in same form	ships in MMDetection

Read the columns together rather than separately. RF-DETR Small hits 53.0 mAP at 3.5 ms; YOLO26m needs 4.7 ms to reach 53.1. At the small end, YOLO26n is the fastest thing on the list at 1.7 ms but gives up 7.5 mAP to RF-DETR Nano at 2.3 ms. Whether 0.6 ms matters more than 7.5 points of mAP is a question about your camera count and your false-negative tolerance, not a question about which model is better.

The COCO column also flatters models that were tuned on COCO. RF100-VL, a hundred-dataset benchmark built from real applied domains such as aerial imagery, industrial inspection, and lab imaging, is the more honest proxy for "how will this fine-tune on my weird dataset." In Roboflow's own published table, RF-DETR Medium reports 61.2 mAP50:95 and 87.4 mAP50 on RF100-VL against YOLO11m at 56.5 and 82.5. If your workload is a custom dataset of a few thousand images rather than 80 COCO classes, weight the transfer benchmark more heavily.

RF-DETR's architecture is a DETR variant on a DINOv2 backbone, which is where the domain transfer advantage plausibly comes from: the self-supervised backbone has seen a much wider visual distribution than a detector trained from scratch on COCO. The work was accepted at ICLR 2026. The rfdetr package was at 1.9.0 as of late July 2026.

pip install rfdetr

import supervision as sv
from rfdetr import RFDETRMedium
from rfdetr.assets.coco_classes import COCO_CLASSES

model = RFDETRMedium()
detections = model.predict("https://media.roboflow.com/dog.jpg", threshold=0.5)

labels = [COCO_CLASSES[class_id] for class_id in detections.class_id]
annotated = sv.BoxAnnotator().annotate(detections.metadata["source_image"], detections)
annotated = sv.LabelAnnotator().annotate(annotated, detections, labels)

Fine-tuning is a two-line affair against a COCO JSON or YOLO-format directory, which matters more than it sounds: the friction of getting a custom detector trained is usually the real cost, not the inference code.

What YOLO26 actually changed

YOLO26 landed on 14 January 2026 with Ultralytics 8.4.0, after being previewed at YOLO Vision in September 2025. The headline is that it is natively end-to-end: the default one-to-one head emits final predictions with no non-maximum suppression step. That removes a chunk of post-processing that was historically the ugliest part of exporting YOLO to a mobile or embedded runtime, where NMS either has to be baked into the graph or reimplemented in C++ on the host side.

Distribution Focal Loss is also gone, which simplifies the head and helps quantization. The training recipe picked up three things: MuSGD, a hybrid optimizer borrowing from Muon as used in LLM training, progressive loss supervision, and STAL label assignment aimed at small targets. Ultralytics claims up to 43 percent faster CPU ONNX inference, but the fine print scopes that to YOLO26n against YOLO11n on an Intel Xeon at 2.00 GHz rather than to the family as a whole.

The CPU column in the table above is the reason people still reach for YOLO. On a T4, RF-DETR wins on accuracy per millisecond almost everywhere. On a CPU-only box with no accelerator, YOLO26n at 38.9 ms per frame in ONNX is in a different category from anything transformer-based, and the NMS-free head makes that export path cleaner than it used to be.

pip install ultralytics
yolo detect predict model=yolo26n.pt source=image.jpg

The license question is not academic

Ultralytics ships under AGPL-3.0 with a paid Enterprise option, and the AGPL reading that Ultralytics itself publishes is expansive: complying means releasing the complete corresponding source of the entire derivative work, including your larger application, your modifications, your scripts and config, and where applicable your model weights. If your product is proprietary, or is a network service that users interact with, the free path is not available to you. This is not a theoretical trap. It is the single most common reason teams migrate off YOLO after a prototype succeeds.

The alternatives are genuinely permissive. RF-DETR's code and the Nano through Large checkpoints are Apache 2.0; only the XL and 2XL detection models sit behind PML 1.0 via rfdetr[plus], and the entire RF-DETR segmentation family, Nano through 2XL, is Apache 2.0. RTMDet ships inside MMDetection, which is Apache 2.0, not MIT as it is often described. SAHI is MIT. Supervision is MIT. Roboflow Inference is Apache 2.0 at the core, with a separate source-available enterprise tier.

Open-vocabulary models are a mixed bag. Grounding DINO from IDEA Research is Apache 2.0. OWL-ViT is Apache 2.0. Florence-2 is MIT. YOLO-World from Tencent AI Lab is GPL-3.0, and YOLOE is AGPL-3.0. Grounded SAM 2 is a composite repo carrying both Apache-2.0 and BSD-3-Clause components, so it needs a per-component read rather than a single answer.

Meta's SAM models use a custom SAM License rather than a standard OSI license. Commercial use is permitted, but with carve-outs: no military, warfare, nuclear, espionage, or gun and illegal weapons applications, compliance with US, UN, EU and UK trade controls, and no reverse engineering. It is not copyleft. You own the derivative works you create, but the agreement travels with any copy or derivative you distribute, so you cannot strip the terms off what you pass on. That is workable for most products and disqualifying for some. Read it before you build a pipeline around it, not after.

SAM 3 and prompting by concept

SAM 3 shipped on 19 November 2025 and is a different kind of model from its predecessors. SAM 2 segmented one object per prompt from a point, box, or mask. SAM 3 introduces promptable concept segmentation: give it a short noun phrase such as "yellow school bus," or an image exemplar, or both, and it returns masks and stable IDs for every matching instance in the image or video at once.

Architecturally it is a dual encoder-decoder transformer, a DETR-style detector plus a SAM 2-derived tracker sharing a single Perception Encoder, with a global presence head that decides whether the concept exists at all before spending capacity on localizing it. That decoupling of recognition from localization is the interesting bit, and it is why the model handles "there are no forklifts in this frame" better than a detector that has to hallucinate boxes and then threshold them away.

It is 848 million parameters across a detector and a tracker that share a single vision encoder, and Meta reports around 30 ms for a single image with more than 100 detected objects on an H200. Meta has not published a VRAM table, so size your hardware from the parameter count and your own profiling rather than from a figure quoted secondhand. On SA-Co, the accompanying benchmark, which spans roughly 270,000 unique concepts across image and video evaluation splits, the paper reports that SAM 3 doubles the accuracy of existing systems on both image and video promptable concept segmentation. The data engine behind it produced about 4 million unique concept labels, hard negatives included.

SAM 3.1 followed on 27 March 2026. The main change is object multiplexing: a shared-memory approach that tracks multiple objects jointly instead of running the tracker once per object. For videos with a medium number of objects, throughput on a single H100 went from about 16 to about 32 FPS. Video latency still scales with object count, and Meta's own framing is near real-time for roughly five concurrent objects, so "real-time" in the strict sense does not stretch to a crowded street scene.

The practical framing: SAM 3 is not a replacement for a fine-tuned detector at the edge. It is a very strong teacher, an annotation engine, and a prototyping tool. Annotation tooling is still catching up, and it is worth checking before you plan around it. CVAT ships a SAM 2 agent, added in CVAT 2.59.0 on 6 March 2026, but its changelog records no SAM 3 integration as of the 2.72.0 release in late July 2026, so a SAM 3 pre-labeling loop there is something you wire up yourself. X-AnyLabeling moved faster: it added client-side local ONNX support for SAM 3 text-grounded segmentation on 26 April 2026, which is the option to look at if you cannot send frames to a server.

Open vocabulary versus fixed classes

The honest generalization: open-vocabulary models are for discovery and labeling, fixed-class models are for production.

Grounding DINO reports 52.5 AP on COCO with no COCO training data, which is remarkable: it lands within half a point of a fine-tuned RF-DETR Small at 53.0. The gap that matters is not accuracy, it is that RF-DETR Small gets there in 3.5 ms on a T4 and Grounding DINO does not come close. The YOLO-World paper reported 35.4 AP zero-shot on LVIS at 52.0 FPS on a V100, but the repository's own current evaluation table lists YOLO-World-L at 26.8 AP on full LVIS and 33.8 on LVIS-mini, so plan against the repo numbers rather than the abstract. OWL-ViT and Florence-2 occupy similar territory with different prompting ergonomics; Florence-2 in particular is a general vision-language model where detection is one of several task prompts.

Where open vocabulary genuinely wins:

• Class sets that change weekly. Retail planograms, content moderation taxonomies, anything where "add a new class" is a product request rather than an ML project.
• Cold start. You have 10,000 unlabeled images and no annotations at all.
• Long-tail queries. Rare concepts where you will never collect enough examples to fine-tune.

Where it loses: latency, cost per frame, and reliability on domain-specific vocabulary. "Yellow school bus" works. "Class III weld porosity" does not, because the text encoder never saw it. For industrial and medical work, the text prompt is usually the wrong interface and a small labeled dataset is the right one.

Small objects: tile the image

Neither RF-DETR nor YOLO26 will find a 12-pixel person in a 4K drone frame at a 640 input size. The standard fix is slicing aided hyper inference: cut the image into overlapping tiles, run the detector on each, and merge the boxes back into image coordinates. SAHI is the reference implementation, MIT licensed, at 0.12.2 as of 20 July 2026, with backends for Ultralytics, MMDetection, HuggingFace models including RT-DETR and Grounding DINO, TorchVision, Detectron2, and RF-DETR.

pip install -U ultralytics sahi

from sahi import AutoDetectionModel
from sahi.predict import get_sliced_prediction

detection_model = AutoDetectionModel.from_pretrained(
    model_type="ultralytics",
    model_path="yolo26n.pt",
    confidence_threshold=0.3,
    device="cuda:0",
)

result = get_sliced_prediction(
    "frame.jpg",
    detection_model,
    slice_height=256,
    slice_width=256,
    overlap_height_ratio=0.2,
    overlap_width_ratio=0.2,
)

The cost is linear in tile count. A 4K frame at 256-pixel tiles with 20 percent overlap is a lot of forward passes, so slicing is a throughput decision as much as an accuracy one. Two mitigations are worth knowing. First, sv.InferenceSlicer in Supervision does the same thing model-agnostically and can run tiles in a thread pool, which helps when your detector releases the GIL during inference. Second, adaptive slicing is an active research direction: variants that derive the slice count from image resolution rather than using a fixed grid have been reported to beat fixed-grid SAHI on aerial benchmarks such as VisDrone and xView. The published margins are small and dataset-specific, so measure on your own imagery before you assume them.

If you are tracking sliced detections over time, ByteTrack pairs cleanly with a slicer callback because it only needs boxes and confidences. One migration note that will bite copy-pasted code: Supervision's bundled sv.ByteTrack is deprecated as of supervision 0.28.0 and scheduled for removal in 0.30.0, replaced by ByteTrackTracker in the separate trackers package.

Labeling: distill a big model into a small one

Autodistill formalizes the loop that most teams end up building by hand. You define an ontology mapping text prompts to class names, point a base model at a folder of unlabeled images, and get a dataset out; then you train a target model on it. Apache 2.0, and the plugin architecture means each base model carries its own license, which you still have to check.

The autodistill-sam3 plugin uses text prompts to auto-label for fine-tuning smaller detectors including RF-DETR, and it is GPU-only. Check its maturity before you depend on it: the published package is still 0.1.0 from November 2025, which is early code rather than a maintained product. The longer-standing plugins cover Grounded SAM 2, Grounding DINO, DETIC, OWL-ViT, OWLv2, and Kosmos-2 as base models, with YOLOv5, YOLOv8, YOLO-NAS, and DETR as targets.

pip install autodistill autodistill-sam3
export ROBOFLOW_API_KEY=your_key

from autodistill_sam3 import SegmentAnything3
from autodistill.detection import CaptionOntology

base_model = SegmentAnything3(
    ontology=CaptionOntology({"fruit": "fruit", "leaf": "leaf"})
)

base_model.label("./images_to_label", extension=".jpeg")

The failure mode to plan for: the student inherits the teacher's systematic errors, and auto-labeled data is confidently wrong in exactly the cases you care about. Budget for a human review pass over a stratified sample. FiftyOne is the right tool for that pass, because it lets you sort by model confidence, cluster by embedding, and find the label mistakes rather than scrolling through 10,000 images in order. Push the corrections back through CVAT and retrain.

Edge deployment: the format matters more than the model

Model choice gets a lot of attention; export format quietly delivers more speedup. Ultralytics' own Jetson Orin Nano Super benchmarks make the point at a glance, running YOLO26n under Ultralytics 8.4.33. PyTorch came in at 15.60 ms per image and ONNX at 15.76 ms, both around 0.476 to 0.479 mAP50-95. TensorRT FP32 cut that to 7.53 ms at 0.4770, FP16 to 4.57 ms at 0.4800, and INT8 to 3.80 ms at 0.4490. That is a 3.4x speedup from FP16 TensorRT with no measurable accuracy cost, and a further 17 percent from INT8 at a real 3-point mAP hit. OpenVINO on that same board went the other way at 56.23 ms, which is a useful reminder that the right export target is hardware-specific rather than universally better. Those numbers exclude pre and post-processing, which is precisely why the NMS-free YOLO26 head and RF-DETR's set-prediction output both matter on device.

Practical rules for the edge:

• On NVIDIA, always export to TensorRT. FP16 is close to free. Evaluate INT8 on your own validation set rather than trusting COCO deltas, because quantization error is dataset-dependent.
• On Intel CPUs and integrated GPUs, OpenVINO is the equivalent step, and on everything else ONNX Runtime with the right execution provider is the portable baseline.
• If you would rather not build the serving layer, Roboflow Inference wraps model loading, hardware-appropriate containers, and an HTTP API. It serves RF-DETR, YOLO variants, CLIP, SAM 2, and Florence-2, with documented install paths for Jetson, Raspberry Pi, CPU, and CUDA.

pip install inference-cli && inference server start --dev

How to choose

Start from the constraint that will not move.

If the constraint is a proprietary product and no legal appetite for AGPL, use RF-DETR. Apache 2.0 through Large, best-in-class accuracy per millisecond on GPU, and a fine-tune path that is two lines of Python. This is the default recommendation for most commercial work in 2026.

If the constraint is a CPU-only device with no accelerator, YOLO26n or YOLO26s, and budget for the Enterprise license or accept the AGPL obligations. Nothing transformer-based is competitive there yet.

If the constraint is a class list that changes faster than you can retrain, use SAM 3 or Grounding DINO directly and accept the server-class hardware bill, or use them as Autodistill teachers and retrain a small student weekly.

If the constraint is object size rather than speed, tile with SAHI before you upgrade the model. A YOLO26s on 256-pixel tiles will outperform a YOLO26x on a downscaled full frame for 15-pixel objects, and it will do it for less money.

Two things to watch over the next few months. The first is whether the Apache-licensed transformer detectors keep closing the CPU gap, because that is the last structural advantage the YOLO family holds. The second is whether concept segmentation gets small enough to run at the edge: SAM 3.1's multiplexing took a real bite out of the per-object cost, and a 100M-parameter model with SAM 3's prompting interface would make the fixed-class detector optional for a lot of applications that currently have no alternative.