Open Source
Explore the latest AI open-source projects from GitHub and HuggingFace.
Explore the latest AI open-source projects from GitHub and HuggingFace.
**Falcon Perception** is TII's inference stack for two small early-fusion vision-language models — **Falcon Perception (0.6B)** for open-vocabulary grounding and instance segmentation, and **Falcon OCR (0.3B)** for document text extraction. Both are **Apache-2.0**, and the repository has gathered **747 stars and 72 forks** since its **31 March 2026** creation. The interesting part is not the parameter count but the architectural bet behind it. Most multimodal systems are a pipeline: a vision encoder feeds a separate text decoder, with task-specific glue bolted on for detection, segmentation or OCR. Falcon does away with the seam. A **single dense autoregressive Transformer** processes image patches and text tokens in a shared parameter space from the first layer, using a hybrid attention mask — image tokens attend bidirectionally to build visual context, while text and task tokens decode causally conditioned on the image. One backbone, one decoding path, and task switching by prompt rather than by module. Segmentation is where that design pays off. For each instance the model emits a short structured sequence of task tokens in fixed order — `<|coord|>`, then `<|size|>`, then `<|seg|>`. The `<|seg|>` token functions as a mask query: its hidden state is projected and dotted against upsampled image features to produce a **full-resolution binary mask in one shot**, with no autoregressive mask generation. Ask it *"segment the cat on the left"* and you get bounding boxes plus pixel-level masks; ask Falcon OCR to extract a page and you get plain text, **LaTeX** for formulas or **HTML** for tables depending on the requested format. TII reports that at 0.3B, OCR serving on their vLLM setup runs **2–3× faster than 0.9B-class OCR VLMs**, varying with sequence length and batch configuration — their own measurement, on their own harness. The repository was updated on **13 August 2026**, and TII has since published an **RL post-trained revision** of the perception model under the `19-08-2026` tag. It applies **GRPO** with a Hungarian set-matching reward that penalizes both false negatives and false positives, changing neither the architecture nor the tokenizer. On the project's own **PBench** benchmark, mask F1 rises from **56.8 to 59.4 on average**, with the largest gain on the Dense split (**72.3 → 80.5**); the stated practical effects are better recall in scenes holding up to 500 instances and the removal of NMS and coordinate deduplication. Details are in *Falcon Perception-HD* (arXiv 2608.18881), alongside the base tech report at arXiv 2603.27365. Deployment options are unusually complete for a research release: **paged**, **paged-OCR** and **batch** PyTorch inference engines for CUDA, an **MLX batch engine** for Apple Silicon that needs neither PyTorch nor transformers, a FastAPI inference server, a Streamlit demo app and a published vLLM Docker image. Two practical notes: **PyTorch 2.5 or newer is required** for FlexAttention, with a slower first call while `torch.compile` builds kernels, and the **vLLM Docker server currently covers Falcon OCR only** — the perception model runs through the PyTorch or MLX engines.
hacksider
Real-time AI face swap and one-click video deepfake with only a single image
harry0703
AI-powered short video generator that automates scripting, footage sourcing, subtitles, and composition — supporting 10+ LLM providers and batch production.