Falcon Perception is TII's inference stack for two small early-fusion vision-language models — Falcon Perception (0.6B) for open-vocabulary grounding and instance segmentation, and Falcon OCR (0.3B) for document text extraction. Both are Apache-2.0, and the repository has gathered 747 stars and 72 forks since its 31 March 2026 creation. The interesting part is not the parameter count but the architectural bet behind it.

Most multimodal systems are a pipeline: a vision encoder feeds a separate text decoder, with task-specific glue bolted on for detection, segmentation or OCR. Falcon does away with the seam. A single dense autoregressive Transformer processes image patches and text tokens in a shared parameter space from the first layer, using a hybrid attention mask — image tokens attend bidirectionally to build visual context, while text and task tokens decode causally conditioned on the image. One backbone, one decoding path, and task switching by prompt rather than by module.

Segmentation is where that design pays off. For each instance the model emits a short structured sequence of task tokens in fixed order — <|coord|>, then <|size|>, then <|seg|>. The <|seg|> token functions as a mask query: its hidden state is projected and dotted against upsampled image features to produce a full-resolution binary mask in one shot, with no autoregressive mask generation. Ask it "segment the cat on the left" and you get bounding boxes plus pixel-level masks; ask Falcon OCR to extract a page and you get plain text, LaTeX for formulas or HTML for tables depending on the requested format. TII reports that at 0.3B, OCR serving on their vLLM setup runs 2–3× faster than 0.9B-class OCR VLMs, varying with sequence length and batch configuration — their own measurement, on their own harness.

The repository was updated on 13 August 2026, and TII has since published an RL post-trained revision of the perception model under the 19-08-2026 tag. It applies GRPO with a Hungarian set-matching reward that penalizes both false negatives and false positives, changing neither the architecture nor the tokenizer. On the project's own PBench benchmark, mask F1 rises from 56.8 to 59.4 on average, with the largest gain on the Dense split (72.3 → 80.5); the stated practical effects are better recall in scenes holding up to 500 instances and the removal of NMS and coordinate deduplication. Details are in Falcon Perception-HD (arXiv 2608.18881), alongside the base tech report at arXiv 2603.27365.

Deployment options are unusually complete for a research release: paged, paged-OCR and batch PyTorch inference engines for CUDA, an MLX batch engine for Apple Silicon that needs neither PyTorch nor transformers, a FastAPI inference server, a Streamlit demo app and a published vLLM Docker image. Two practical notes: PyTorch 2.5 or newer is required for FlexAttention, with a slower first call while torch.compile builds kernels, and the vLLM Docker server currently covers Falcon OCR only — the perception model runs through the PyTorch or MLX engines.