new

Get trending papers in your email inbox!

Subscribe

Daily Papers

byAK and the research community

Nov 16

MaPLe: Multi-modal Prompt Learning

Pre-trained vision-language (V-L) models such as CLIP have shown excellent generalization ability to downstream tasks. However, they are sensitive to the choice of input text prompts and require careful selection of prompt templates to perform well. Inspired by the Natural Language Processing (NLP) literature, recent CLIP adaptation approaches learn prompts as the textual inputs to fine-tune CLIP for downstream tasks. We note that using prompting to adapt representations in a single branch of CLIP (language or vision) is sub-optimal since it does not allow the flexibility to dynamically adjust both representation spaces on a downstream task. In this work, we propose Multi-modal Prompt Learning (MaPLe) for both vision and language branches to improve alignment between the vision and language representations. Our design promotes strong coupling between the vision-language prompts to ensure mutual synergy and discourages learning independent uni-modal solutions. Further, we learn separate prompts across different early stages to progressively model the stage-wise feature relationships to allow rich context learning. We evaluate the effectiveness of our approach on three representative tasks of generalization to novel classes, new target datasets and unseen domain shifts. Compared with the state-of-the-art method Co-CoOp, MaPLe exhibits favorable performance and achieves an absolute gain of 3.45% on novel classes and 2.72% on overall harmonic-mean, averaged over 11 diverse image recognition datasets. Our code and pre-trained models are available at https://github.com/muzairkhattak/multimodal-prompt-learning.

  • 5 authors
·
Oct 6, 2022

A Progressive Framework of Vision-language Knowledge Distillation and Alignment for Multilingual Scene

Pre-trained vision-language (V-L) models such as CLIP have shown excellent performance in many downstream cross-modal tasks. However, most of them are only applicable to the English context. Subsequent research has focused on this problem and proposed improved models, such as CN-CLIP and AltCLIP, to facilitate their applicability to Chinese and even other languages. Nevertheless, these models suffer from high latency and a large memory footprint in inference, which limits their further deployment on resource-constrained edge devices. In this work, we propose a conceptually simple yet effective multilingual CLIP Compression framework and train a lightweight multilingual vision-language model, called DC-CLIP, for both Chinese and English context. In this framework, we collect high-quality Chinese and English text-image pairs and design two training stages, including multilingual vision-language feature distillation and alignment. During the first stage, lightweight image/text student models are designed to learn robust visual/multilingual textual feature representation ability from corresponding teacher models, respectively. Subsequently, the multilingual vision-language alignment stage enables effective alignment of visual and multilingual textual features to further improve the model's multilingual performance. Comprehensive experiments in zero-shot image classification, conducted based on the ELEVATER benchmark, showcase that DC-CLIP achieves superior performance in the English context and competitive performance in the Chinese context, even with less training data, when compared to existing models of similar parameter magnitude. The evaluation demonstrates the effectiveness of our designed training mechanism.

  • 6 authors
·
Apr 17, 2024

Sentence-level Prompts Benefit Composed Image Retrieval

Composed image retrieval (CIR) is the task of retrieving specific images by using a query that involves both a reference image and a relative caption. Most existing CIR models adopt the late-fusion strategy to combine visual and language features. Besides, several approaches have also been suggested to generate a pseudo-word token from the reference image, which is further integrated into the relative caption for CIR. However, these pseudo-word-based prompting methods have limitations when target image encompasses complex changes on reference image, e.g., object removal and attribute modification. In this work, we demonstrate that learning an appropriate sentence-level prompt for the relative caption (SPRC) is sufficient for achieving effective composed image retrieval. Instead of relying on pseudo-word-based prompts, we propose to leverage pretrained V-L models, e.g., BLIP-2, to generate sentence-level prompts. By concatenating the learned sentence-level prompt with the relative caption, one can readily use existing text-based image retrieval models to enhance CIR performance. Furthermore, we introduce both image-text contrastive loss and text prompt alignment loss to enforce the learning of suitable sentence-level prompts. Experiments show that our proposed method performs favorably against the state-of-the-art CIR methods on the Fashion-IQ and CIRR datasets. The source code and pretrained model are publicly available at https://github.com/chunmeifeng/SPRC

  • 8 authors
·
Oct 9, 2023

VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks

Recently, fine-tuning language models pre-trained on large text corpora have provided huge improvements on vision-and-language (V&L) tasks as well as on pure language tasks. However, fine-tuning the entire parameter set of pre-trained models becomes impractical since the model size is growing rapidly. Hence, in this paper, we introduce adapter-based parameter-efficient transfer learning techniques to V&L models such as VL-BART and VLT5. We evaluate our methods in a unified multi-task setup on both image-text and video-text benchmarks. For the image-text tasks, we use four diverse V&L datasets: VQAv2, GQA, NLVR2 , and MSCOCO image captioning. For video-text tasks, we use TVQA, How2QA, TVC, and YC2C. With careful training and thorough experiments, we benchmark three popular adapter-based methods (Adapter, Hyperformer, Compacter) against the standard full fine-tuning and the recently proposed prompt-tuning approach. We also enhance the efficiency and performance of adapters by sharing their weights to attain knowledge across tasks. Our results demonstrate that training the adapter with the weight-sharing technique (4.18% of total parameters for image-text tasks and 3.39% for video-text tasks) can match the performance of fine-tuning the entire model. Lastly, we present a comprehensive analysis including the combination of adapter and task-specific prompts and the impact of V&L pre-training on adapters. Our code is available at: https://github.com/ylsung/VL_adapter.

  • 3 authors
·
Dec 13, 2021

AR-VLA: True Autoregressive Action Expert for Vision-Language-Action Models

We propose a standalone autoregressive (AR) Action Expert that generates actions as a continuous causal sequence while conditioning on refreshable vision-language prefixes. In contrast to existing Vision-Language-Action (VLA) models and diffusion policies that reset temporal context with each new observation and predict actions reactively, our Action Expert maintains its own history through a long-lived memory and is inherently context-aware. This structure addresses the frequency mismatch between fast control and slow reasoning, enabling efficient independent pretraining of kinematic syntax and modular integration with heavy perception backbones, naturally ensuring spatio-temporally consistent action generation across frames. To synchronize these asynchronous hybrid V-L-A modalities, we utilize a re-anchoring mechanism that mathematically accounts for perception staleness during both training and inference. Experiments on simulated and real-robot manipulation tasks demonstrate that the proposed method can effectively replace traditional chunk-based action heads for both specialist and generalist policies. AR-VLA exhibits superior history awareness and substantially smoother action trajectories while maintaining or exceeding the task success rates of state-of-the-art reactive VLAs. Overall, our work introduces a scalable, context-aware action generation schema that provides a robust structural foundation for training effective robotic policies. Code and Videos available at https://arvla.insait.ai

One Perturbation is Enough: On Generating Universal Adversarial Perturbations against Vision-Language Pre-training Models

Vision-Language Pre-training (VLP) models have exhibited unprecedented capability in many applications by taking full advantage of the multimodal alignment. However, previous studies have shown they are vulnerable to maliciously crafted adversarial samples. Despite recent success, these methods are generally instance-specific and require generating perturbations for each input sample. In this paper, we reveal that VLP models are also vulnerable to the instance-agnostic universal adversarial perturbation (UAP). Specifically, we design a novel Contrastive-training Perturbation Generator with Cross-modal conditions (C-PGC) to achieve the attack. In light that the pivotal multimodal alignment is achieved through the advanced contrastive learning technique, we devise to turn this powerful weapon against themselves, i.e., employ a malicious version of contrastive learning to train the C-PGC based on our carefully crafted positive and negative image-text pairs for essentially destroying the alignment relationship learned by VLP models. Besides, C-PGC fully utilizes the characteristics of Vision-and-Language (V+L) scenarios by incorporating both unimodal and cross-modal information as effective guidance. Extensive experiments show that C-PGC successfully forces adversarial samples to move away from their original area in the VLP model's feature space, thus essentially enhancing attacks across various victim models and V+L tasks. The GitHub repository is available at https://github.com/ffhibnese/CPGC_VLP_Universal_Attacks.

  • 8 authors
·
Jun 8, 2024

TWLV-I: Analysis and Insights from Holistic Evaluation on Video Foundation Models

In this work, we discuss evaluating video foundation models in a fair and robust manner. Unlike language or image foundation models, many video foundation models are evaluated with differing parameters (such as sampling rate, number of frames, pretraining steps, etc.), making fair and robust comparisons challenging. Therefore, we present a carefully designed evaluation framework for measuring two core capabilities of video comprehension: appearance and motion understanding. Our findings reveal that existing video foundation models, whether text-supervised like UMT or InternVideo2, or self-supervised like V-JEPA, exhibit limitations in at least one of these capabilities. As an alternative, we introduce TWLV-I, a new video foundation model that constructs robust visual representations for both motion- and appearance-based videos. Based on the average top-1 accuracy of linear probing on five action recognition benchmarks, pretrained only on publicly accessible datasets, our model shows a 4.6%p improvement compared to V-JEPA (ViT-L) and a 7.7%p improvement compared to UMT (ViT-L). Even when compared to much larger models, our model demonstrates a 7.2%p improvement compared to DFN (ViT-H), a 2.7%p improvement compared to V-JEPA~(ViT-H) and a 2.8%p improvement compared to InternVideo2 (ViT-g). We provide embedding vectors obtained by TWLV-I from videos of several commonly used video benchmarks, along with evaluation source code that can directly utilize these embeddings. The code is available on "https://github.com/twelvelabs-io/video-embeddings-evaluation-framework".

  • 21 authors
·
Aug 20, 2024 2

Leveraging Generic Foundation Models for Multimodal Surgical Data Analysis

We investigate how both the adaptation of a generic foundation model via transfer learning and the integration of complementary modalities from the operating room (OR) can support surgical data science. To this end, we use V-JEPA as the single-modality foundation of a multimodal model for minimally invasive surgery support. We analyze how the model's downstream performance can benefit (a) from finetuning on unlabeled surgical video data and (b) from providing additional time-resolved data streams from the OR in a multimodal setup. In an in-house dataset of liver surgery videos, we analyze the tasks of predicting hospital length of stay and postoperative complications. In videos of the public HeiCo dataset, we analyze the task of surgical phase recognition. As a baseline, we apply pretrained V-JEPA to all tasks. We then finetune it on unlabeled, held-out videos to investigate its change in performance after domain adaptation. Following the idea of modular decision support networks, we integrate additional data streams from the OR by training a separate encoder to form a shared representation space with V-JEPA's embeddings. Our experiments show that finetuning on domain-specific data increases model performance. On the in-house data, integrating additional time-resolved data likewise benefits the model. On the HeiCo data, accuracy of the pretrained video-only, single-modality baseline setup is on par with the top-performing submissions of the EndoVis2017 challenge, while finetuning on domain-specific data increases accuracy further. Our results thus demonstrate how surgical data science can leverage public, generic foundation models. Likewise, they indicate the potential of domain adaptation and of integrating suitable complementary data streams from the OR. To support further research, we release our code and model weights at https://github.com/DigitalSurgeryLab-Basel/ML-CDS-2025.

  • 5 authors
·
Sep 8, 2025

Efficient In-Context Learning in Vision-Language Models for Egocentric Videos

Recent advancements in text-only large language models (LLMs) have highlighted the benefit of in-context learning for adapting to new tasks with a few demonstrations. However, extending in-context learning to large vision-language models (VLMs) using a huge amount of naturalistic vision-language data has shown limited success, particularly for egocentric videos, due to high data collection costs. We propose a novel training method Efficient In-context Learning on Egocentric Videos (EILEV), which elicits in-context learning in VLMs for egocentric videos without requiring massive, naturalistic egocentric video datasets. EILEV involves architectural and training data adaptations to allow the model to process contexts interleaved with video clips and narrations, sampling of in-context examples with clusters of similar verbs and nouns, use of data with skewed marginal distributions with a long tail of infrequent verbs and nouns, as well as homonyms and synonyms. Our evaluations show that EILEV-trained models outperform larger VLMs trained on a huge amount of naturalistic data in in-context learning. Furthermore, they can generalize to not only out-of-distribution, but also novel, rare egocentric videos and texts via in-context learning, demonstrating potential for applications requiring cost-effective training, and rapid post-deployment adaptability. Our code and demo are available at https://github.com/yukw777/EILEV.

  • 4 authors
·
Nov 28, 2023

Write-Protected Discrete Bottlenecks for Language-Grounded World Models: A Structural Limitation and Sufficient Fix

How should language interface with a world model's discrete symbol system? The dominant paradigm -- end-to-end injection of LLM/VLM features into robot world models (RT-2, Octo, PaLM-E) -- implicitly assumes that language gradients can directly shape physical symbol representations. We ask whether this assumption is safe, find that it is not, and characterize the minimal architectural constraint that prevents the failure. Any language gradient entering a Gumbel-softmax-based discrete symbol bottleneck forces a structural trade-off: the vanilla estimator collapses to 2.2/64 symbols (4/5 seeds), while five anti-collapse strategies maintain diversity but fail to learn semantic labels (all <= 9.2% accuracy). No tested GumbelBottleneck variant achieves both objectives simultaneously. Within this family of discrete bottlenecks, the failure is structural rather than a matter of optimization. We characterize a sufficient set of three constraints that prevent the failure: (1) cut the gradient chain (z.detach()), preventing language signals from reaching the symbol bottleneck; (2) provide a gradient-free semantic channel -- a non-parametric Memory Table (Dict[symbol -> Counter[label]], zero parameters, zero gradients) where co-occurrence counting replaces gradient-based binding; (3) handle symbol collisions via DP-Means streaming clustering for automatic sub-cluster splitting. All three layers together achieve 97.2% grounding accuracy vs. 22.2% without Layer 3. Across two experiments spanning 74 independent runs, we demonstrate zero symbol collapse in all 32 seeds, with the blackboard achieving 79-100% semantic binding across three encoder architectures (CNN, V-JEPA 300M, CLIP ViT-L), two environments, and three texture conditions. The fix trains fewer than 2M parameters and requires no LLM fine-tuning.

  • 1 authors
·
Jul 8

Naver-News-KO: A Korean News Summarization Dataset for Open-Source Fine-Tuning of Summarization Models

We release Naver-News-KO, a Korean news summarization dataset of 27,400 (document, summary) pairs collected from Naver News over a ten-day window in July 2022 across two categories (Economy and IT/Science; 77/23 split), with train/validation/test partitions of 22,194 / 2,466 / 2,740 and a mean per-record document-to-summary character-compression ratio of 6.03x. The dataset has been publicly hosted on the Hugging Face Hub since January 2023 and, as of May 2026, receives approximately 33,000 downloads per month; community-maintained Korean summarization models fine-tuned on it include Gemma-2B-ko and Gemma2-9B variants. This technical report (i) documents the collection protocol, the column schema, and the split construction, (ii) reports corpus-level statistics (length distributions, compression ratio, and a measured 16.8% near-duplicate title-Jaccard overlap between test and train that users should be aware of), (iii) positions the resource against other open Korean summarization corpora, (iv) provides a Lead-3 extractive reference point (ROUGE-1 55.1, ROUGE-L 50.6) and two reproducible fine-tuned baselines -- KoBART (R-1 56.6, BERTScore-F1 81.5) and Gemma-2B-ko with LoRA (R-1 55.3, BERTScore-F1 78.3) -- with release-time training scripts, and (v) clarifies the licensing and intended-use scope of the resource. The goal is to provide a citable reference for downstream work that already uses this dataset, not to propose a new benchmark.

  • 1 authors
·
May 11

A region-wide, multi-year set of crop field boundary labels for Africa

African agriculture is undergoing rapid transformation. Annual maps of crop fields are key to understanding the nature of this transformation, but such maps are currently lacking and must be developed using advanced machine learning models trained on high resolution remote sensing imagery. To enable the development of such models, we delineated field boundaries in 33,746 Planet images captured between 2017 and 2023 across the continent using a custom labeling platform with built-in procedures for assessing and mitigating label error. We collected 42,403 labels, including 7,204 labels arising from tasks dedicated to assessing label quality (Class 1 labels), 32,167 from sites mapped once by a single labeller (Class 2) and 3,032 labels from sites where 3 or more labellers were tasked to map the same location (Class 4). Class 1 labels were used to calculate labeller-specific quality scores, while Class 1 and 4 sites mapped by at least 3 labellers were used to further evaluate label uncertainty using a Bayesian risk metric. Quality metrics showed that label quality was moderately high (0.75) for measures of total field extent, but low regarding the number of individual fields delineated (0.33), and the position of field edges (0.05). These values are expected when delineating small-scale fields in 3-5 m resolution imagery, which can be too coarse to reliably distinguish smaller fields, particularly in dense croplands, and therefore requires substantial labeller judgement. Nevertheless, previous work shows that such labels can train effective field mapping models. Furthermore, this large, probabilistic sample on its own provides valuable insight into regional agricultural characteristics, highlighting variations in the median field size and density. The imagery and vectorized labels along with quality information is available for download from two public repositories.

  • 30 authors
·
Dec 24, 2024

M2H-MX: Multi-Task Dense Visual Perception for Real-Time Monocular Spatial Understanding

Monocular cameras are attractive for robotic perception due to their low cost and ease of deployment, yet achieving reliable real-time spatial understanding from a single image stream remains challenging. While recent multi-task dense prediction models have improved per-pixel depth and semantic estimation, translating these advances into stable monocular mapping systems is still non-trivial. This paper presents M2H-MX, a real-time multi-task perception model for monocular spatial understanding. The model preserves multi-scale feature representations while introducing register-gated global context and controlled cross-task interaction in a lightweight decoder, enabling depth and semantic predictions to reinforce each other under strict latency constraints. Its outputs integrate directly into an unmodified monocular SLAM pipeline through a compact perception-to-mapping interface. We evaluate both dense prediction accuracy and in-the-loop system performance. On NYUDv2, M2H-MX-L achieves state-of-the-art results, improving semantic mIoU by 6.6% and reducing depth RMSE by 9.4% over representative multi-task baselines. When deployed in a real-time monocular mapping system on ScanNet, M2H-MX reduces average trajectory error by 60.7% compared to a strong monocular SLAM baseline while producing cleaner metric-semantic maps. These results demonstrate that modern multi-task dense prediction can be reliably deployed for real-time monocular spatial perception in robotic systems.

  • 3 authors
·
Mar 30

Instruct-DeBERTa: A Hybrid Approach for Aspect-based Sentiment Analysis on Textual Reviews

Aspect-based Sentiment Analysis (ABSA) is a critical task in Natural Language Processing (NLP) that focuses on extracting sentiments related to specific aspects within a text, offering deep insights into customer opinions. Traditional sentiment analysis methods, while useful for determining overall sentiment, often miss the implicit opinions about particular product or service features. This paper presents a comprehensive review of the evolution of ABSA methodologies, from lexicon-based approaches to machine learning and deep learning techniques. We emphasize the recent advancements in Transformer-based models, particularly Bidirectional Encoder Representations from Transformers (BERT) and its variants, which have set new benchmarks in ABSA tasks. We focused on finetuning Llama and Mistral models, building hybrid models using the SetFit framework, and developing our own model by exploiting the strengths of state-of-the-art (SOTA) Transformer-based models for aspect term extraction (ATE) and aspect sentiment classification (ASC). Our hybrid model Instruct - DeBERTa uses SOTA InstructABSA for aspect extraction and DeBERTa-V3-baseabsa-V1 for aspect sentiment classification. We utilize datasets from different domains to evaluate our model's performance. Our experiments indicate that the proposed hybrid model significantly improves the accuracy and reliability of sentiment analysis across all experimented domains. As per our findings, our hybrid model Instruct - DeBERTa is the best-performing model for the joint task of ATE and ASC for both SemEval restaurant 2014 and SemEval laptop 2014 datasets separately. By addressing the limitations of existing methodologies, our approach provides a robust solution for understanding detailed consumer feedback, thus offering valuable insights for businesses aiming to enhance customer satisfaction and product development.

  • 8 authors
·
Aug 22, 2024

The Alzheimer's Disease Prediction Of Longitudinal Evolution (TADPOLE) Challenge: Results after 1 Year Follow-up

We present the findings of "The Alzheimer's Disease Prediction Of Longitudinal Evolution" (TADPOLE) Challenge, which compared the performance of 92 algorithms from 33 international teams at predicting the future trajectory of 219 individuals at risk of Alzheimer's disease. Challenge participants were required to make a prediction, for each month of a 5-year future time period, of three key outcomes: clinical diagnosis, Alzheimer's Disease Assessment Scale Cognitive Subdomain (ADAS-Cog13), and total volume of the ventricles. The methods used by challenge participants included multivariate linear regression, machine learning methods such as support vector machines and deep neural networks, as well as disease progression models. No single submission was best at predicting all three outcomes. For clinical diagnosis and ventricle volume prediction, the best algorithms strongly outperform simple baselines in predictive ability. However, for ADAS-Cog13 no single submitted prediction method was significantly better than random guesswork. Two ensemble methods based on taking the mean and median over all predictions, obtained top scores on almost all tasks. Better than average performance at diagnosis prediction was generally associated with the additional inclusion of features from cerebrospinal fluid (CSF) samples and diffusion tensor imaging (DTI). On the other hand, better performance at ventricle volume prediction was associated with inclusion of summary statistics, such as the slope or maxima/minima of biomarkers. TADPOLE's unique results suggest that current prediction algorithms provide sufficient accuracy to exploit biomarkers related to clinical diagnosis and ventricle volume, for cohort refinement in clinical trials for Alzheimer's disease. However, results call into question the usage of cognitive test scores for patient selection and as a primary endpoint in clinical trials.

  • 96 authors
·
Feb 9, 2020

ReplicationBench: Can AI Agents Replicate Astrophysics Research Papers?

Frontier AI agents show increasing promise as scientific research assistants, and may eventually be useful for extended, open-ended research workflows. However, in order to use agents for novel research, we must first assess the underlying faithfulness and correctness of their work. To evaluate agents as research assistants, we introduce ReplicationBench, an evaluation framework that tests whether agents can replicate entire research papers drawn from the astrophysics literature. Astrophysics, where research relies heavily on archival data and computational study while requiring little real-world experimentation, is a particularly useful testbed for AI agents in scientific research. We split each paper into tasks which require agents to replicate the paper's core contributions, including the experimental setup, derivations, data analysis, and codebase. Each task is co-developed with the original paper authors and targets a key scientific result, enabling objective evaluation of both faithfulness (adherence to original methods) and correctness (technical accuracy of results). ReplicationBench is extremely challenging for current frontier language models: even the best-performing language models score under 20%. We analyze ReplicationBench trajectories in collaboration with domain experts and find a rich, diverse set of failure modes for agents in scientific research. ReplicationBench establishes the first benchmark of paper-scale, expert-validated astrophysics research tasks, reveals insights about agent performance generalizable to other domains of data-driven science, and provides a scalable framework for measuring AI agents' reliability in scientific research.

  • 13 authors
·
Oct 28, 2025 1

Predictive Maintenance Model Based on Anomaly Detection in Induction Motors: A Machine Learning Approach Using Real-Time IoT Data

With the support of Internet of Things (IoT) devices, it is possible to acquire data from degradation phenomena and design data-driven models to perform anomaly detection in industrial equipment. This approach not only identifies potential anomalies but can also serve as a first step toward building predictive maintenance policies. In this work, we demonstrate a novel anomaly detection system on induction motors used in pumps, compressors, fans, and other industrial machines. This work evaluates a combination of pre-processing techniques and machine learning (ML) models with a low computational cost. We use a combination of pre-processing techniques such as Fast Fourier Transform (FFT), Wavelet Transform (WT), and binning, which are well-known approaches for extracting features from raw data. We also aim to guarantee an optimal balance between multiple conflicting parameters, such as anomaly detection rate, false positive rate, and inference speed of the solution. To this end, multiobjective optimization and analysis are performed on the evaluated models. Pareto-optimal solutions are presented to select which models have the best results regarding classification metrics and computational effort. Differently from most works in this field that use publicly available datasets to validate their models, we propose an end-to-end solution combining low-cost and readily available IoT sensors. The approach is validated by acquiring a custom dataset from induction motors. Also, we fuse vibration, temperature, and noise data from these sensors as the input to the proposed ML model. Therefore, we aim to propose a methodology general enough to be applied in different industrial contexts in the future.

  • 8 authors
·
Oct 14, 2023

Planet formation in chemically diverse and evolving discs II. Chemical fingerprints in planetary atmospheres

Giant planets form in protoplanetary discs, where the coupled dynamical and chemical evolution of gas and solids determines the composition of the material they accrete. We investigate how planet formation and migration shape the primordial elemental makeup of giant-planet atmospheres. Our aim is to link atmospheric compositions to planets' formation pathways and the time-dependent chemical properties of their natal discs. We couple 1D models of viscously evolving discs - incorporating radial dust drift and volatile chemistry - with N-body simulations of planetesimals interacting with a growing and migrating giant planet. Four chemical scenarios and three representative grain sizes (0.1, 20, and 100 micron) are explored. We track the accretion of carbon, oxygen, nitrogen, and sulphur to derive atmospheric elemental ratios normalised to stellar values (* denotes stellar normalisation). We identify three atmospheric classes corresponding to distinct accretion regimes: gas-dominated, characterised by N/O* > C/O* > C/N* and unconstrained or substellar S/N* (near-stellar C/S*); planetesimal-dominated, showing N/O* < C/O* < C/N*, S/N* >= C/N*, and C/S* <= C/O*; and drift-enhanced, exhibiting N/O* < C/O* < C/N* and markedly superstellar volatile-to-refractory ratios. N/O*, C/N*, and S/N* vary systematically with migration extent, although degeneracies arise for planets forming beyond the CO and N2 snowlines; C/O* remains largely insensitive. Metallicity alone does not uniquely trace the solid-to-gas accretion balance in drift-dominated regimes. Variations in the disc's chemical state and dust size imprint distinctive volatile-ratio patterns across these classes, providing complementary constraints on disc properties. This multi-element framework establishes predictive trends to guide the interpretation of atmospheric spectra from facilities like JWST and Ariel.

  • 16 authors
·
Jun 29

Euclid Quick Data Release (Q1). Active galactic nuclei identification using diffusion-based inpainting of Euclid VIS images

Light emission from galaxies exhibit diverse brightness profiles, influenced by factors such as galaxy type, structural features and interactions with other galaxies. Elliptical galaxies feature more uniform light distributions, while spiral and irregular galaxies have complex, varied light profiles due to their structural heterogeneity and star-forming activity. In addition, galaxies with an active galactic nucleus (AGN) feature intense, concentrated emission from gas accretion around supermassive black holes, superimposed on regular galactic light, while quasi-stellar objects (QSO) are the extreme case of the AGN emission dominating the galaxy. The challenge of identifying AGN and QSO has been discussed many times in the literature, often requiring multi-wavelength observations. This paper introduces a novel approach to identify AGN and QSO from a single image. Diffusion models have been recently developed in the machine-learning literature to generate realistic-looking images of everyday objects. Utilising the spatial resolving power of the Euclid VIS images, we created a diffusion model trained on one million sources, without using any source pre-selection or labels. The model learns to reconstruct light distributions of normal galaxies, since the population is dominated by them. We condition the prediction of the central light distribution by masking the central few pixels of each source and reconstruct the light according to the diffusion model. We further use this prediction to identify sources that deviate from this profile by examining the reconstruction error of the few central pixels regenerated in each source's core. Our approach, solely using VIS imaging, features high completeness compared to traditional methods of AGN and QSO selection, including optical, near-infrared, mid-infrared, and X-rays.

  • 274 authors
·
Mar 19, 2025

REDAffectiveLM: Leveraging Affect Enriched Embedding and Transformer-based Neural Language Model for Readers' Emotion Detection

Technological advancements in web platforms allow people to express and share emotions towards textual write-ups written and shared by others. This brings about different interesting domains for analysis; emotion expressed by the writer and emotion elicited from the readers. In this paper, we propose a novel approach for Readers' Emotion Detection from short-text documents using a deep learning model called REDAffectiveLM. Within state-of-the-art NLP tasks, it is well understood that utilizing context-specific representations from transformer-based pre-trained language models helps achieve improved performance. Within this affective computing task, we explore how incorporating affective information can further enhance performance. Towards this, we leverage context-specific and affect enriched representations by using a transformer-based pre-trained language model in tandem with affect enriched Bi-LSTM+Attention. For empirical evaluation, we procure a new dataset REN-20k, besides using RENh-4k and SemEval-2007. We evaluate the performance of our REDAffectiveLM rigorously across these datasets, against a vast set of state-of-the-art baselines, where our model consistently outperforms baselines and obtains statistically significant results. Our results establish that utilizing affect enriched representation along with context-specific representation within a neural architecture can considerably enhance readers' emotion detection. Since the impact of affect enrichment specifically in readers' emotion detection isn't well explored, we conduct a detailed analysis over affect enriched Bi-LSTM+Attention using qualitative and quantitative model behavior evaluation techniques. We observe that compared to conventional semantic embedding, affect enriched embedding increases ability of the network to effectively identify and assign weightage to key terms responsible for readers' emotion detection.

  • 5 authors
·
Jan 21, 2023

Euclid preparation: I. The Euclid Wide Survey

Euclid is an ESA mission designed to constrain the properties of dark energy and gravity via weak gravitational lensing and galaxy clustering. It will carry out a wide area imaging and spectroscopy survey (EWS) in visible and near-infrared, covering roughly 15,000 square degrees of extragalactic sky on six years. The wide-field telescope and instruments are optimized for pristine PSF and reduced straylight, producing very crisp images. This paper presents the building of the Euclid reference survey: the sequence of pointings of EWS, Deep fields, Auxiliary fields for calibrations, and spacecraft movements followed by Euclid as it operates in a step-and-stare mode from its orbit around the Lagrange point L2. Each EWS pointing has four dithered frames; we simulate the dither pattern at pixel level to analyse the effective coverage. We use up-to-date models for the sky background to define the Euclid region-of-interest (RoI). The building of the reference survey is highly constrained from calibration cadences, spacecraft constraints and background levels; synergies with ground-based coverage are also considered. Via purposely-built software optimized to prioritize best sky areas, produce a compact coverage, and ensure thermal stability, we generate a schedule for the Auxiliary and Deep fields observations and schedule the RoI with EWS transit observations. The resulting reference survey RSD_2021A fulfills all constraints and is a good proxy for the final solution. Its wide survey covers 14,500 square degrees. The limiting AB magnitudes (5σ point-like source) achieved in its footprint are estimated to be 26.2 (visible) and 24.5 (near-infrared); for spectroscopy, the H_α line flux limit is 2times 10^{-16} erg cm^{-2} s^{-1} at 1600 nm; and for diffuse emission the surface brightness limits are 29.8 (visible) and 28.4 (near-infrared) mag arcsec^{-2}.

  • 241 authors
·
Aug 1, 2021

DESI 2024 VI: Cosmological Constraints from the Measurements of Baryon Acoustic Oscillations

We present cosmological results from the measurement of baryon acoustic oscillations (BAO) in galaxy, quasar and Lyman-α forest tracers from the first year of observations from the Dark Energy Spectroscopic Instrument (DESI), to be released in the DESI Data Release 1. DESI BAO provide robust measurements of the transverse comoving distance and Hubble rate, or their combination, relative to the sound horizon, in seven redshift bins from over 6 million extragalactic objects in the redshift range 0.1<z<4.2. DESI BAO data alone are consistent with the standard flat ΛCDM cosmological model with a matter density Ω_m=0.295pm 0.015. Paired with a BBN prior and the robustly measured acoustic angular scale from the CMB, DESI requires H_0=(68.52pm0.62) km/s/Mpc. In conjunction with CMB anisotropies from Planck and CMB lensing data from Planck and ACT, we find Ω_m=0.307pm 0.005 and H_0=(67.97pm0.38) km/s/Mpc. Extending the baseline model with a constant dark energy equation of state parameter w, DESI BAO alone require w=-0.99^{+0.15}_{-0.13}. In models with a time-varying dark energy equation of state parametrized by w_0 and w_a, combinations of DESI with CMB or with SN~Ia individually prefer w_0>-1 and w_a<0. This preference is 2.6σ for the DESI+CMB combination, and persists or grows when SN~Ia are added in, giving results discrepant with the ΛCDM model at the 2.5σ, 3.5σ or 3.9σ levels for the addition of Pantheon+, Union3, or DES-SN5YR datasets respectively. For the flat ΛCDM model with the sum of neutrino mass sum m_ν free, combining the DESI and CMB data yields an upper limit sum m_ν< 0.072 (0.113) eV at 95% confidence for a sum m_ν>0 (sum m_ν>0.059) eV prior. These neutrino-mass constraints are substantially relaxed in models beyond ΛCDM. [Abridged.]

  • 203 authors
·
Nov 3, 2024

Superclustering with the Atacama Cosmology Telescope and Dark Energy Survey: II. Anisotropic large-scale coherence in hot gas, galaxies, and dark matter

Statistics that capture the directional dependence of the baryon distribution in the cosmic web enable unique tests of cosmology and astrophysical feedback. We use constrained oriented stacking of thermal Sunyaev-Zel'dovich (tSZ) maps to measure the anisotropic distribution of hot gas 2.5-40 Mpc away from galaxy clusters embedded in massive filaments and superclusters. The cluster selection and orientation (at a scale of sim15 Mpc) use Dark Energy Survey (DES) Year 3 data, while expanded tSZ maps from the Atacama Cosmology Telescope Data Release 6 enable a sim3times more significant measurement of the extended gas compared to the technique's proof-of-concept. Decomposing stacks into cosine multipoles of order m, we detect a dipole (m=1) and quadrupole (m=2) at 8-10sigma, as well as evidence for m=4 signal at up to 6sigma, indicating sensitivity to late-time non-Gaussianity. We compare to the Cardinal simulations with spherical gas models pasted onto dark matter halos. The fiducial tSZ data can discriminate between two models that deplete pressure differently in low-mass halos (mimicking astrophysical feedback), preferring higher average pressure in extended structures. However, uncertainty in the amount of cosmic infrared background contamination reduces the constraining power. Additionally, we apply the technique to DES galaxy density and weak lensing to study for the first time their oriented relationships with tSZ. In the tSZ-to-lensing relation, averaged on 7.5 Mpc (transverse) scales, we observe dependence on redshift but not shape or radial distance. Thus, on large scales, the superclustering of gas pressure, galaxies, and total matter is coherent in shape and extent.

  • 76 authors
·
Sep 6, 2024

Assessing Pancreatic Ductal Adenocarcinoma Vascular Invasion: the PDACVI Benchmark

Surgical resection remains the only potentially curative treatment for pancreatic ductal adenocarcinoma (PDAC), and eligibility depends on accurate assessment of vascular invasion (VI), i.e., tumor extension into adjacent critical vessels. Despite its importance for preoperative staging and surgical planning, computational VI assessment remains underexplored. Two major challenges are the lack of public datasets and the diagnostic ambiguity at the tumor-vessel interface, which leads to substantial inter-rater variability even among expert radiologists. To address these limitations, we introduce the CURVAS-PDACVI Dataset and Challenge, an open benchmark for uncertainty-aware AI in PDAC staging based on a densely annotated dataset with five independent expert annotations per scan. We also propose a multi-metric evaluation framework that extends beyond spatial overlap to include probabilistic calibration and VI assessment. Evaluation of six state-of-the-art methods shows that strong global volumetric overlap does not necessarily translate into reliable performance at clinically critical tumor-vessel interfaces. In particular, methods optimized for binary segmentation perform competitively on average overlap metrics, but often degrade in high-complexity cases with low expert consensus, either collapsing in volume or overextending at uncertain boundaries. In contrast, methods that model inter-rater disagreement produce better calibrated probabilistic maps and show greater robustness in these ambiguous cases. The benchmark highlights the limitations of volumetric accuracy as a proxy for localized surgical utility, motivating uncertainty-aware probabilistic models for preoperative decision-making.

  • 26 authors
·
Apr 29 2

OpenH-RF: A Large-Scale Ultrasound Raw Channel Dataset Initiative

Medical ultrasound is a widely accessible, real-time imaging modality, yet progress in data-driven ultrasound research is constrained by the scarcity of large, open collections of raw channel data. Existing public datasets are predominantly based on reconstructed images, while the smaller number of open raw channel datasets is generally specialized to individual reconstruction or imaging tasks, limiting the development of general-purpose methods and foundation models that operate directly on ultrasound signals. While interpretive tasks can be performed on reconstructed images, many of the key technical challenges in ultrasound imaging, such as aberration correction and reverberation reduction, require the raw channel data itself. We introduce OpenH-RF, a community-driven open dataset initiative comprising approximately 40 TB of medical ultrasound raw channel data in 19,332 acquisitions, contributed by 38 institutions across academia and industry, spanning in-vivo, phantom, and simulated data. Each dataset is annotated with a number of supported tasks, ranging from generalized reconstruction, speed-of-sound estimation, and aberration correction to blood flow imaging, motion estimation, quantitative ultrasound, and image interpretation. OpenH-RF provides a unified, self-contained data format that captures raw channel data together with the acquisition, probe, metadata, and reconstruction information required for reproducible processing across scanners and acquisition schemes. The corpus is publicly hosted under a permissive CC BY 4.0 license and can be streamed programmatically and selectively at scale. By bringing heterogeneous ultrasound data into a common representation, backed by modern data infrastructure, OpenH-RF lowers the barrier to working with raw channel data and aims to open medical ultrasound research to new communities. See https://open-h.github.io/OpenH-RF.

  • 186 authors
·
Oct 8

OpenUniverse2024: A shared, simulated view of the sky for the next generation of cosmological surveys

The OpenUniverse2024 simulation suite is a cross-collaboration effort to produce matched simulated imaging for multiple surveys as they would observe a common simulated sky. Both the simulated data and associated tools used to produce it are intended to uniquely enable a wide range of studies to maximize the science potential of the next generation of cosmological surveys. We have produced simulated imaging for approximately 70 deg^2 of the Vera C. Rubin Observatory Legacy Survey of Space and Time (LSST) Wide-Fast-Deep survey and the Nancy Grace Roman Space Telescope High-Latitude Wide-Area Survey, as well as overlapping versions of the ELAIS-S1 Deep-Drilling Field for LSST and the High-Latitude Time-Domain Survey for Roman. OpenUniverse2024 includes i) an early version of the updated extragalactic model called Diffsky, which substantially improves the realism of optical and infrared photometry of objects, compared to previous versions of these models; ii) updated transient models that extend through the wavelength range probed by Roman and Rubin; and iii) improved survey, telescope, and instrument realism based on up-to-date survey plans and known properties of the instruments. It is built on a new and updated suite of simulation tools that improves the ease of consistently simulating multiple observatories viewing the same sky. The approximately 400 TB of synthetic survey imaging and simulated universe catalogs are publicly available, and we preview some scientific uses of the simulations.

  • 61 authors
·
Mar 4, 2025

IXPE Observation of the Low-Synchrotron Peaked Blazar S4 0954+65 During An Optical-X-ray Flare

The X-ray polarization observations made possible with the Imaging X-ray Polarimetry Explorer (IXPE) offer new ways of probing high-energy emission processes in astrophysical jets from blazars. Here we report on the first X-ray polarization observation of the blazar S4 0954+65 in a high optical and X-ray state. During our multi-wavelength campaign on the source, we detected an optical flare whose peak coincided with the peak of an X-ray flare. This optical-X-ray flare most likely took place in a feature moving along the parsec-scale jet, imaged at 43 GHz by the Very Long Baseline Array. The 43 GHz polarization angle of the moving component underwent a rotation near the time of the flare. In the optical band, prior to the IXPE observation, we measured the polarization angle to be aligned with the jet axis. In contrast, during the optical flare the optical polarization angle was perpendicular to the jet axis; after the flare, it reverted to being parallel to the jet axis. Due to the smooth behavior of the optical polarization angle during the flare, we favor shocks as the main acceleration mechanism. We also infer that the ambient magnetic field lines in the jet were parallel to the jet position angle. The average degree of optical polarization during the IXPE observation was (14.3pm4.1)%. Despite the flare, we only detected an upper limit of 14% (at 3sigma level) on the X-ray polarization degree; although a reasonable assumption on the X-ray polarization angle results in an upper limit of 8.8% (3sigma). We model the spectral energy distribution (SED) and spectral polarization distribution (SPD) of S4 0954+65 with leptonic (synchrotron self-Compton) and hadronic (proton and pair synchrotron) models. The constraints we obtain with our combined multi-wavelength polarization observations and SED modeling tentatively disfavor hadronic models for the X-ray emission in S4 0954+65.

  • 137 authors
·
Nov 25, 2024

Euclid Quick Data Release (Q1) Exploring galaxy properties with a multi-modal foundation model

Modern astronomical surveys, such as the Euclid mission, produce high-dimensional, multi-modal data sets that include imaging and spectroscopic information for millions of galaxies. These data serve as an ideal benchmark for large, pre-trained multi-modal models, which can leverage vast amounts of unlabelled data. In this work, we present the first exploration of Euclid data with AstroPT, an autoregressive multi-modal foundation model trained on approximately 300 000 optical and infrared Euclid images and spectral energy distributions (SEDs) from the first Euclid Quick Data Release. We compare self-supervised pre-training with baseline fully supervised training across several tasks: galaxy morphology classification; redshift estimation; similarity searches; and outlier detection. Our results show that: (a) AstroPT embeddings are highly informative, correlating with morphology and effectively isolating outliers; (b) including infrared data helps to isolate stars, but degrades the identification of edge-on galaxies, which are better captured by optical images; (c) simple fine-tuning of these embeddings for photometric redshift and stellar mass estimation outperforms a fully supervised approach, even when using only 1% of the training labels; and (d) incorporating SED data into AstroPT via a straightforward multi-modal token-chaining method improves photo-z predictions, and allow us to identify potentially more interesting anomalies (such as ringed or interacting galaxies) compared to a model pre-trained solely on imaging data.

  • 324 authors
·
Mar 19, 2025

Planck 2018 results. I. Overview and the cosmological legacy of Planck

The European Space Agency's Planck satellite, which was dedicated to studying the early Universe and its subsequent evolution, was launched on 14 May 2009. It scanned the microwave and submillimetre sky continuously between 12 August 2009 and 23 October 2013, producing deep, high-resolution, all-sky maps in nine frequency bands from 30 to 857GHz. This paper presents the cosmological legacy of Planck, which currently provides our strongest constraints on the parameters of the standard cosmological model and some of the tightest limits available on deviations from that model. The 6-parameter LCDM model continues to provide an excellent fit to the cosmic microwave background data at high and low redshift, describing the cosmological information in over a billion map pixels with just six parameters. With 18 peaks in the temperature and polarization angular power spectra constrained well, Planck measures five of the six parameters to better than 1% (simultaneously), with the best-determined parameter (theta_*) now known to 0.03%. We describe the multi-component sky as seen by Planck, the success of the LCDM model, and the connection to lower-redshift probes of structure formation. We also give a comprehensive summary of the major changes introduced in this 2018 release. The Planck data, alone and in combination with other probes, provide stringent constraints on our models of the early Universe and the large-scale structure within which all astrophysical objects form and evolve. We discuss some lessons learned from the Planck mission, and highlight areas ripe for further experimental advances.

  • 191 authors
·
Dec 2, 2019

Planck 2018 results. VI. Cosmological parameters

We present cosmological parameter results from the final full-mission Planck measurements of the CMB anisotropies. We find good consistency with the standard spatially-flat 6-parameter LambdaCDM cosmology having a power-law spectrum of adiabatic scalar perturbations (denoted "base LambdaCDM" in this paper), from polarization, temperature, and lensing, separately and in combination. A combined analysis gives dark matter density Omega_c h^2 = 0.120pm 0.001, baryon density Omega_b h^2 = 0.0224pm 0.0001, scalar spectral index n_s = 0.965pm 0.004, and optical depth tau = 0.054pm 0.007 (in this abstract we quote 68,% confidence regions on measured parameters and 95,% on upper limits). The angular acoustic scale is measured to 0.03,% precision, with 100theta_*=1.0411pm 0.0003. These results are only weakly dependent on the cosmological model and remain stable, with somewhat increased errors, in many commonly considered extensions. Assuming the base-LambdaCDM cosmology, the inferred late-Universe parameters are: Hubble constant H_0 = (67.4pm 0.5)km/s/Mpc; matter density parameter Omega_m = 0.315pm 0.007; and matter fluctuation amplitude sigma_8 = 0.811pm 0.006. We find no compelling evidence for extensions to the base-LambdaCDM model. Combining with BAO we constrain the effective extra relativistic degrees of freedom to be N_{rm eff} = 2.99pm 0.17, and the neutrino mass is tightly constrained to sum m_nu< 0.12eV. The CMB spectra continue to prefer higher lensing amplitudes than predicted in base -LambdaCDM at over 2,sigma, which pulls some parameters that affect the lensing amplitude away from the base-LambdaCDM model; however, this is not supported by the lensing reconstruction or (in models that also change the background geometry) BAO data. (Abridged)

  • 182 authors
·
Jul 17, 2018

DEJIMA: A Novel Large-scale Japanese Dataset for Image Captioning and Visual Question Answering

This work addresses the scarcity of high-quality, large-scale resources for Japanese Vision-and-Language (V&L) modeling. We present a scalable and reproducible pipeline that integrates large-scale web collection with rigorous filtering/deduplication, object-detection-driven evidence extraction, and Large Language Model (LLM)-based refinement under grounding constraints. Using this pipeline, we build two resources: an image-caption dataset (DEJIMA-Cap) and a VQA dataset (DEJIMA-VQA), each containing 3.88M image-text pairs, far exceeding the size of existing Japanese V&L datasets. Human evaluations demonstrate that DEJIMA achieves substantially higher Japaneseness and linguistic naturalness than datasets constructed via translation or manual annotation, while maintaining factual correctness at a level comparable to human-annotated corpora. Quantitative analyses of image feature distributions further confirm that DEJIMA broadly covers diverse visual domains characteristic of Japan, complementing its linguistic and cultural representativeness. Models trained on DEJIMA exhibit consistent improvements across multiple Japanese multimodal benchmarks, confirming that culturally grounded, large-scale resources play a key role in enhancing model performance. All data sources and modules in our pipeline are licensed for commercial use, and we publicly release the resulting dataset and metadata to encourage further research and industrial applications in Japanese V&L modeling.

  • 6 authors
·
Nov 30, 2025

CogVLA: Cognition-Aligned Vision-Language-Action Model via Instruction-Driven Routing & Sparsification

Recent Vision-Language-Action (VLA) models built on pre-trained Vision-Language Models (VLMs) require extensive post-training, resulting in high computational overhead that limits scalability and deployment.We propose CogVLA, a Cognition-Aligned Vision-Language-Action framework that leverages instruction-driven routing and sparsification to improve both efficiency and performance. CogVLA draws inspiration from human multimodal coordination and introduces a 3-stage progressive architecture. 1) Encoder-FiLM based Aggregation Routing (EFA-Routing) injects instruction information into the vision encoder to selectively aggregate and compress dual-stream visual tokens, forming a instruction-aware latent representation. 2) Building upon this compact visual encoding, LLM-FiLM based Pruning Routing (LFP-Routing) introduces action intent into the language model by pruning instruction-irrelevant visually grounded tokens, thereby achieving token-level sparsity. 3) To ensure that compressed perception inputs can still support accurate and coherent action generation, we introduce V-L-A Coupled Attention (CAtten), which combines causal vision-language attention with bidirectional action parallel decoding. Extensive experiments on the LIBERO benchmark and real-world robotic tasks demonstrate that CogVLA achieves state-of-the-art performance with success rates of 97.4% and 70.0%, respectively, while reducing training costs by 2.5-fold and decreasing inference latency by 2.8-fold compared to OpenVLA. CogVLA is open-sourced and publicly available at https://github.com/JiuTian-VL/CogVLA.

  • 5 authors
·
Aug 28, 2025 2

Scalable Vision-Language-Action Model Pretraining for Robotic Manipulation with Real-Life Human Activity Videos

This paper presents a novel approach for pretraining robotic manipulation Vision-Language-Action (VLA) models using a large corpus of unscripted real-life video recordings of human hand activities. Treating human hand as dexterous robot end-effector, we show that "in-the-wild" egocentric human videos without any annotations can be transformed into data formats fully aligned with existing robotic V-L-A training data in terms of task granularity and labels. This is achieved by the development of a fully-automated holistic human activity analysis approach for arbitrary human hand videos. This approach can generate atomic-level hand activity segments and their language descriptions, each accompanied with framewise 3D hand motion and camera motion. We process a large volume of egocentric videos and create a hand-VLA training dataset containing 1M episodes and 26M frames. This training data covers a wide range of objects and concepts, dexterous manipulation tasks, and environment variations in real life, vastly exceeding the coverage of existing robot data. We design a dexterous hand VLA model architecture and pretrain the model on this dataset. The model exhibits strong zero-shot capabilities on completely unseen real-world observations. Additionally, fine-tuning it on a small amount of real robot action data significantly improves task success rates and generalization to novel objects in real robotic experiments. We also demonstrate the appealing scaling behavior of the model's task performance with respect to pretraining data scale. We believe this work lays a solid foundation for scalable VLA pretraining, advancing robots toward truly generalizable embodied intelligence.

  • 17 authors
·
Oct 24, 2025

Mono-Hydra++: Real-Time Monocular Scene Graph Construction with Multi-Task Learning for 3D Indoor Mapping

Autonomous agile robots need more than metric geometry: they must understand objects, rooms, places, and spatial relations for search, inspection, exploration, and human robot interaction. Conventional metric maps support localization and collision avoidance, but do not provide this semantic and relational structure. 3D scene graphs address this gap by connecting geometry with object level and room level understanding. Building such representations on agile platforms remains difficult because aerial and lightweight robots operate under strict payload, power, and compute limits, making RGB-D cameras and LiDAR sensors impractical for many onboard settings. We present Mono-Hydra++, a real time monocular RGB plus IMU pipeline for indoor metric semantic mapping and hierarchical 3D scene graph construction. The system combines M2H-MX, a DINOv3 based multi-task model for depth and semantics, with a deep feature visual inertial odometry front end, sparse predicted depth constraints in the VIO derived pose graph, semantic masking for dynamic regions, and pose aware temporal alignment before volumetric fusion in the Mono-Hydra backend. On the Go-SLAM ScanNet evaluation subset, Mono-Hydra++ achieves 1.6% lower average trajectory error than the strongest RGB-D baseline in our comparison, while using only monocular RGB plus IMU input. On calibrated 7-Scenes, it improves average ATE by 29.8% over the strongest competing calibrated baseline. We further validate Mono-Hydra++ in a real ITC building deployment using RealSense RGB plus IMU and demonstrate embedded feasibility by deploying the ONNX/TensorRT FP16 M2H-MX-L perception model at 25.53 FPS on a Jetson Orin NX 16GB. These results show that Mono-Hydra++ can provide real time metric semantic mapping and scene graph construction for resource constrained robotic platforms without relying on active depth sensors.

  • 3 authors
·
May 16

A Constraint on Primordial $B$-Modes from the First Flight of the SPIDER Balloon-Borne Telescope

We present the first linear polarization measurements from the 2015 long-duration balloon flight of SPIDER, an experiment designed to map the polarization of the cosmic microwave background (CMB) on degree angular scales. Results from these measurements include maps and angular power spectra from observations of 4.8% of the sky at 95 and 150 GHz, along with the results of internal consistency tests on these data. While the polarized CMB anisotropy from primordial density perturbations is the dominant signal in this region of sky, Galactic dust emission is also detected with high significance; Galactic synchrotron emission is found to be negligible in the SPIDER bands. We employ two independent foreground-removal techniques in order to explore the sensitivity of the cosmological result to the assumptions made by each. The primary method uses a dust template derived from Planck data to subtract the Galactic dust signal. A second approach, employing a joint analysis of SPIDER and Planck data in the harmonic domain, assumes a modified-blackbody model for the spectral energy distribution of the dust with no constraint on its spatial morphology. Using a likelihood that jointly samples the template amplitude and r parameter space, we derive 95% upper limits on the primordial tensor-to-scalar ratio from Feldman-Cousins and Bayesian constructions, finding r<0.11 and r<0.19, respectively. Roughly half the uncertainty in r derives from noise associated with the template subtraction. New data at 280 GHz from SPIDER's second flight will complement the Planck polarization maps, providing powerful measurements of the polarized Galactic dust emission.

  • 71 authors
·
Mar 23, 2021

JWST Lensed Quasar Dark Matter Survey V: Measuring the minimum halo mass with strong gravitational lensing

We explore the lowest mass limit that can be placed on the halo mass function in CDM using 28 strong gravitational lenses. For this purpose, we study an extreme model in which the halo mass function and mass-concentration relation follow CDM, with a sharp cutoff at some mass scale, m_{low}. Lensing provides a unique window into this quantity as it does not depend on the presence of baryons in dark matter halos and also allows the detection of low mass halos at cosmological distances, both in the lens galaxies and along the line-of-sight. Our model incorporates the effects of tidal stripping of subhalos, leading to the presence of many subhalos below a given model cutoff scale. We place an upper limit on the low-mass cutoff of the halo mass function of m_{low}<10^{8.3} M_odot at 10:1 odds using a prior for the normalization of the subhalo mass function from the semi-analytic model {\tt galacticus} and m_{low}<10^{8.2} M_odot at 10:1 odds using a prior from N-body simulations. These limits are comparable to, or stronger than, existing constraints based on Milky Way satellite galaxies. Based on these results, we forecast more than an order of magnitude improvement with a sample of 200 quadruply imaged quasar lenses. This number represents a small subset of the thousands that are anticipated to be discovered by Rubin, Euclid, and Roman. Furthermore, with this larger sample of lenses we expect to directly constrain the normalization of the subhalo mass function, thereby eliminating a major source of uncertainty in our current measurements.

  • 22 authors
·
Apr 5

Planck 2018 results. V. CMB power spectra and likelihoods

This paper describes the 2018 Planck CMB likelihoods, following a hybrid approach similar to the 2015 one, with different approximations at low and high multipoles, and implementing several methodological and analysis refinements. With more realistic simulations, and better correction and modelling of systematics, we can now make full use of the High Frequency Instrument polarization data. The low-multipole 100x143 GHz EE cross-spectrum constrains the reionization optical-depth parameter tau to better than 15% (in combination with with the other low- and high-ell likelihoods). We also update the 2015 baseline low-ell joint TEB likelihood based on the Low Frequency Instrument data, which provides a weaker tau constraint. At high multipoles, a better model of the temperature-to-polarization leakage and corrections for the effective calibrations of the polarization channels (polarization efficiency or PE) allow us to fully use the polarization spectra, improving the constraints on the LambdaCDM parameters by 20 to 30% compared to TT-only constraints. Tests on the modelling of the polarization demonstrate good consistency, with some residual modelling uncertainties, the accuracy of the PE modelling being the main limitation. Using our various tests, simulations, and comparison between different high-ell implementations, we estimate the consistency of the results to be better than the 0.5sigma level. Minor curiosities already present before (differences between ell<800 and ell>800 parameters or the preference for more smoothing of the C_ell peaks) are shown to be driven by the TT power spectrum and are not significantly modified by the inclusion of polarization. Overall, the legacy Planck CMB likelihoods provide a robust tool for constraining the cosmological model and represent a reference for future CMB observations. (Abridged)

  • 168 authors
·
Jul 30, 2019

DESI 2024 V: Full-Shape Galaxy Clustering from Galaxies and Quasars

We present the measurements and cosmological implications of the galaxy two-point clustering using over 4.7 million unique galaxy and quasar redshifts in the range 0.1<z<2.1 divided into six redshift bins over a sim 7,500 square degree footprint, from the first year of observations with the Dark Energy Spectroscopic Instrument (DESI Data Release 1). By fitting the full power spectrum, we extend previous DESI DR1 baryon acoustic oscillation (BAO) measurements to include redshift-space distortions and signals from the matter-radiation equality scale. For the first time, this Full-Shape analysis is blinded at the catalogue-level to avoid confirmation bias and the systematic errors are accounted for at the two-point clustering level, which automatically propagates them into any cosmological parameter. When analysing the data in terms of compressed model-agnostic variables, we obtain a combined precision of 4.7\% on the amplitude of the redshift space distortion signal reaching similar precision with just one year of DESI data than with 20 years of observation from previous generation surveys. We analyse the data to directly constrain the cosmological parameters within the LambdaCDM model using perturbation theory and combine this information with the reconstructed DESI DR1 galaxy BAO. Using a Big Bang Nucleosynthesis Gaussian prior on the baryon density parameter, and a Gaussian prior on the spectral index, we constrain the matter density is Omega_m=0.296pm 0.010 and the Hubble constant H_0=(68.63 pm 0.79)[{rm km, s^{-1}Mpc^{-1}}]. Additionally, we measure the amplitude of clustering sigma_8=0.841 pm 0.034. The DESI DR1 results are in agreement with the LambdaCDM model based on general relativity with parameters consistent with those from Planck. The cosmological interpretation of these results in combination with external datasets are presented in a companion paper.

  • 198 authors
·
Nov 18, 2024

EWOCS-XI: Anisotropic expansion and shear in the cluster Westerlund 1. A 2D kinematic analysis with Gaia DR3

Westerlund 1 (Wd1) is one of the Milky Way's most massive young clusters, in which heavy extinction and field contamination have historically hampered its kinematic characterization. We aim to assess Wd1's internal kinematics, dynamical state, and formation scenario. Using a robust 5D clustering approach, we identified 559 bona fide members. We modeled their proper motions via a 2D first-order Taylor expansion to derive the velocity gradient tensor, mapping expansion, rotation, and shear. Restricting the kinematic analysis to 524 stars with proper motion uncertainties <0.5~mas,yr^{-1}, we find at least 103 gravitationally unbound members, including 14 walkaway stars. The 2D analysis reveals a statistically significant divergence (>4σ) highlighting preferred expansion along the Galactic latitude (position angle 164^circpm10.1^circ). A detected cross-gradient (partial v_b / partial l = -0.41 pm 0.08 km s^{-1} pc^{-1}) provides a clear signature of shear on the plane of the sky. Wd1 does not behave as a single cohesive population, but comprises two decoupled subsystems: a stable, bound core in dynamical equilibrium and an escaping, unbound envelope. This prominent anisotropic expansion perpendicular to the Galactic plane has a kinematic timescale (Δt_{kin} = 2.08 pm 0.44 Myr) matching the independent orbital mid-plane crossing time (Δt_{orbit} = 2.10 pm 0.05 Myr). This temporal alignment indicates the expansion onset coincided with Wd1's Galactic disk passage, triggered either by internal dynamics (e.g., early supernova feedback) or external tidal perturbations. Finally, the robust unbound population demonstrates that active cluster dissolution is underway, driven by discrete dynamical ejection events rather than long-term stellar evaporation.

  • 17 authors
·
Oct 7

Optical Signatures of Black Holes Surrounded by a Generalized Cloud of Strings

In this work, we investigate the optical signatures of black holes surrounded by a generalized cloud of strings, described by the Letelier--Alencar spacetime. We first analyze null geodesics, showing that the string parameters can either increase or decrease the photon sphere radius and the critical impact parameter relative to their Schwarzschild values. Since the spacetime is not asymptotically flat, the critical impact parameter does not directly coincide with the physical shadow radius. We use this distinction to derive constraints on the model parameters from the shadow radius bounds of Sgr A* and M87*. In the weak-field regime, we calculate the light deflection angle using both the Gauss--Bonnet method and a perturbative expansion of the null geodesic equations, obtaining consistent results that contain a contribution from the asymptotically conical geometry in addition to the local gravitational deflection. We also examine the Shapiro time delay and identify a distance dependent contribution associated with the non Minkowskian asymptotic background. Finally, we study the optical appearance of geometrically and optically thin accretion disks and construct celestial sphere images through backward ray tracing. The transfer functions and lensing images show that the generalized cloud of strings modifies the null geodesic mapping in a nonuniform manner rather than producing a simple rescaling of the Schwarzschild image. In particular, configurations with a smaller shadow may still produce an outward displacement of the main Einstein ring image, demonstrating that different optical observables probe distinct aspects of photon propagation in this geometry.

  • 4 authors
·
Oct 7

ImageFlowNet: Forecasting Multiscale Image-Level Trajectories of Disease Progression with Irregularly-Sampled Longitudinal Medical Images

Advances in medical imaging technologies have enabled the collection of longitudinal images, which involve repeated scanning of the same patients over time, to monitor disease progression. However, predictive modeling of such data remains challenging due to high dimensionality, irregular sampling, and data sparsity. To address these issues, we propose ImageFlowNet, a novel model designed to forecast disease trajectories from initial images while preserving spatial details. ImageFlowNet first learns multiscale joint representation spaces across patients and time points, then optimizes deterministic or stochastic flow fields within these spaces using a position-parameterized neural ODE/SDE framework. The model leverages a UNet architecture to create robust multiscale representations and mitigates data scarcity by combining knowledge from all patients. We provide theoretical insights that support our formulation of ODEs, and motivate our regularizations involving high-level visual features, latent space organization, and trajectory smoothness. We validate ImageFlowNet on three longitudinal medical image datasets depicting progression in geographic atrophy, multiple sclerosis, and glioblastoma, demonstrating its ability to effectively forecast disease progression and outperform existing methods. Our contributions include the development of ImageFlowNet, its theoretical underpinnings, and empirical validation on real-world datasets. The official implementation is available at https://github.com/KrishnaswamyLab/ImageFlowNet.

  • 11 authors
·
Jun 20, 2024

Measurements of the Temperature and E-mode Polarization of the Cosmic Microwave Background from the Full 500-square-degree SPTpol Dataset

Using the full four-year SPTpol 500 deg^2 dataset in both the 95 GHz and 150 GHz frequency bands, we present measurements of the temperature and E-mode polarization of the cosmic microwave background (CMB), as well as the E-mode polarization auto-power spectrum (EE) and temperature-E-mode cross-power spectrum (TE) in the angular multipole range 50<ell<8000. We find the SPTpol dataset to be self-consistent, passing several internal consistency tests based on maps, frequency bands, bandpowers, and cosmological parameters. The full SPTpol dataset is well-fit by the ΛCDM model, for which we find H_0=70.48pm2.16 km s^{-1} Mpc^{-1} and Ω_m=0.271pm0.026, when using only the SPTpol data and a Planck-based prior on the optical depth to reionization. The ΛCDM parameter constraints are consistent across the 95 GHz-only, 150 GHz-only, TE-only, and EE-only data splits. Between the ell<1000 and ell>1000 data splits, the ΛCDM parameter constraints are borderline consistent at the sim2σ level. This consistency improves when including a parameter A_L, the degree of lensing of the CMB inferred from the smearing of acoustic peaks. When marginalized over A_L, the ΛCDM parameter constraints from SPTpol are consistent with those from Planck. The power spectra presented here are the most sensitive measurements of the lensed CMB damping tail to date for roughly ell > 1700 in TE and ell > 2000 in EE.

  • 62 authors
·
Aug 1, 2025

BICEP / Keck XIII: Improved Constraints on Primordial Gravitational Waves using Planck, WMAP, and BICEP/Keck Observations through the 2018 Observing Season

We present results from an analysis of all data taken by the BICEP2, Keck Array and BICEP3 CMB polarization experiments up to and including the 2018 observing season. We add additional Keck Array observations at 220 GHz and BICEP3 observations at 95 GHz to the previous 95/150/220 GHz data set. The Q/U maps now reach depths of 2.8, 2.8 and 8.8 μ{mathrm K}_{cmb} arcmin at 95, 150 and 220 GHz respectively over an effective area of approx 600 square degrees at 95 GHz and approx 400 square degrees at 150 & 220 GHz. The 220 GHz maps now achieve a signal-to-noise on polarized dust emission exceeding that of Planck at 353 GHz. We take auto- and cross-spectra between these maps and publicly available WMAP and Planck maps at frequencies from 23 to 353 GHz and evaluate the joint likelihood of the spectra versus a multicomponent model of lensed-ΛCDM+r+dust+synchrotron+noise. The foreground model has seven parameters, and no longer requires a prior on the frequency spectral index of the dust emission taken from measurements on other regions of the sky. This model is an adequate description of the data at the current noise levels. The likelihood analysis yields the constraint r_{0.05}<0.036 at 95% confidence. Running maximum likelihood search on simulations we obtain unbiased results and find that σ(r)=0.009. These are the strongest constraints to date on primordial gravitational waves.

  • 92 authors
·
Sep 30, 2021

SN 2023ixf in the Pinwheel Galaxy M101: From Shock Breakout to the Nebular Phase

We present photometric and spectroscopic observations of SN 2023ixf covering from day one to 442 days after explosion. SN 2023ixf reached a peak V-band absolute magnitude of -18.2 pm 0.07, and light curves show that it is in the fast-decliner (IIL) subclass with a relatively short ``plateau'' phase (fewer than sim 70 days). Early-time spectra of SN 2023ixf exhibit strong, very narrow emission lines from ionized circumstellar matter (CSM), possibly indicating a Type IIn classification. But these flash/shock-ionization emission features faded after the first week and the spectrum evolved in a manner similar to that of typical Type II SNe, unlike the case of most genuine SNe~IIn in which the ejecta interact with CSM for an extended period of time and develop intermediate-width emission lines. We compare observed spectra of SN 2023ixf with various model spectra to understand the physics behind SN 2023ixf. Our nebular spectra (between 200-400 d) match best with the model spectra from a 15 rm M_{odot} progenitor which experienced enhanced mass loss a few years before explosion. A last-stage mass-loss rate of M = 0.01 rm M_{odot} yr^{-1} from the r1w6 model matches best with the early-time spectra, higher than M approx 2.4 times 10^{-3} rm M_{odot} yr^{-1} derived from the ionized H{alpha} luminosity at 1.58 d. We also use SN 2023ixf as a distance indicator and fit the light curves to derive the Hubble constant by adding SN 2023ixf to the existing sample; we obtain H_{0}=73.1^{+3.68}_{-3.50} km s^{-1} Mpc^{-1}, consistent with the results from SNe~Ia and many other independent methods.

  • 42 authors
·
Mar 18, 2025

Euclid Quick Data Release (Q1). AstroVink: A vision transformer approach to find strong gravitational lens systems

We present AstroVink, a vision transformer classifier designed for automated identification of strong lens candidates in Euclid imaging. We build upon the DINOv2 encoder, fine tuned to distinguish between lens and non-lens galaxies. Our base model, trained on simulated strong lens systems and labelled non lenses, recovers 88 of the 110 lens candidates within the top 500 ranked candidates, corresponding to an inspection efficiency of one lens per 5.7 inspected objects in our test set. After the Q1 data release, which yielded about 500 lens candidates, we retrained the model using high confidence lens candidates and new negatives, initially flagged as potential lenses by other classifiers but rejected during visual inspection. The retrained network further improves performance, achieving recovery of all 110 systems within the same ranking and reducing the inspection effort to one lens per 4.5 inspected objects, demonstrating that incorporating real examples significantly enhances model generalisation. An analysis of training subsets revealed that the inclusion of realistic negative examples played a key role in this improvement. Finally, we applied the retrained model to the Q1 original selection of 1.08M targets, followed by a new round of Space Warps citizen science inspection and expert vetting, where we identified a total of eight Grade A and 26 Grade B new lens candidates. These results demonstrate that transformer based architectures can recover strong lens candidates with high efficiency in real Euclid data, while substantially reducing the number of candidates requiring visual inspection.

  • 305 authors
·
Apr 22

Detection of the Schwarzschild precession in the orbit of the star S2 near the Galactic centre massive black hole

The star S2 orbiting the compact radio source Sgr A* is a precision probe of the gravitational field around the closest massive black hole (candidate). Over the last 2.7 decades we have monitored the star's radial velocity and motion on the sky, mainly with the SINFONI and NACO adaptive optics (AO) instruments on the ESO VLT, and since 2017, with the four-telescope interferometric beam combiner instrument GRAVITY. In this paper we report the first detection of the General Relativity (GR) Schwarzschild Precession (SP) in S2's orbit. Owing to its highly elliptical orbit (e = 0.88), S2's SP is mainly a kink between the pre-and post-pericentre directions of motion ~ +- 1 year around pericentre passage, relative to the corresponding Kepler orbit. The superb 2017-2019 astrometry of GRAVITY defines the pericentre passage and outgoing direction. The incoming direction is anchored by 118 NACO-AO measurements of S2's position in the infrared reference frame, with an additional 75 direct measurements of the S2-Sgr A* separation during bright states ('flares') of Sgr A*. Our 14-parameter model fits for the distance, central mass, the position and motion of the reference frame of the AO astrometry relative to the mass, the six parameters of the orbit, as well as a dimensionless parameter f_SP for the SP (f_SP = 0 for Newton and 1 for GR). From data up to the end of 2019 we robustly detect the SP of S2, del phi = 12' per orbital period. From posterior fitting and MCMC Bayesian analysis with different weighting schemes and bootstrapping we find f_SP = 1.10 +- 0.19. The S2 data are fully consistent with GR. Any extended mass inside S2's orbit cannot exceed ~ 0.1% of the central mass. Any compact third mass inside the central arcsecond must be less than about 1000 M_sun.

  • 56 authors
·
Apr 14, 2020

Cluster Cosmology Constraints from the 2500 deg$^2$ SPT-SZ Survey: Inclusion of Weak Gravitational Lensing Data from Magellan and the Hubble Space Telescope

We derive cosmological constraints using a galaxy cluster sample selected from the 2500~deg^2 SPT-SZ survey. The sample spans the redshift range 0.25< z<1.75 and contains 343 clusters with SZ detection significance ξ>5. The sample is supplemented with optical weak gravitational lensing measurements of 32 clusters with 0.29<z<1.13 (from Magellan and HST) and X-ray measurements of 89 clusters with 0.25<z<1.75 (from Chandra). We rely on minimal modeling assumptions: i) weak lensing provides an accurate means of measuring halo masses, ii) the mean SZ and X-ray observables are related to the true halo mass through power-law relations in mass and dimensionless Hubble parameter E(z) with a-priori unknown parameters, iii) there is (correlated, lognormal) intrinsic scatter and measurement noise relating these observables to their mean relations. We simultaneously fit for these astrophysical modeling parameters and for cosmology. Assuming a flat νΛCDM model, in which the sum of neutrino masses is a free parameter, we measure Ω_m=0.276pm0.047, σ_8=0.781pm0.037, and σ_8(Ω_m/0.3)^{0.2}=0.766pm0.025. The redshift evolution of the X-ray Y_X-mass and M_gas-mass relations are both consistent with self-similar evolution to within 1σ. The mass-slope of the Y_X-mass relation shows a 2.3σ deviation from self-similarity. Similarly, the mass-slope of the M_gas-mass relation is steeper than self-similarity at the 2.5σ level. In a νwCDM cosmology, we measure the dark energy equation of state parameter w=-1.55pm0.41 from the cluster data. We perform a measurement of the growth of structure since redshift zsim1.7 and find no evidence for tension with the prediction from General Relativity. We provide updated redshift and mass estimates for the SPT sample. (abridged)

  • 80 authors
·
May 19, 2019

ExoMiner++ on TESS with Transfer Learning from Kepler: Transit Classification and Vetting Catalog for 2-min Data

We present ExoMiner++, an enhanced deep learning model that builds on the success of ExoMiner to improve transit signal classification in 2-minute TESS data. ExoMiner++ incorporates additional diagnostic inputs, including periodogram, flux trend, difference image, unfolded flux, and spacecraft attitude control data, all of which are crucial for effectively distinguishing transit signals from more challenging sources of false positives. To further enhance performance, we leverage transfer learning from high-quality labeled data from the Kepler space telescope, mitigating the impact of TESS's noisier and more ambiguous labels. ExoMiner++ achieves high accuracy across various classification and ranking metrics, significantly narrowing the search space for follow-up investigations to confirm new planets. To serve the exoplanet community, we introduce new TESS catalogs containing ExoMiner++ classifications and confidence scores for each transit signal. Among the 147,568 unlabeled TCEs, ExoMiner++ identifies 7,330 as planet candidates, with the remainder classified as false positives. These 7,330 planet candidates correspond to 1,868 existing TESS Objects of Interest (TOIs), 69 Community TESS Objects of Interest (CTOIs), and 50 newly introduced CTOIs. 1,797 out of the 2,506 TOIs previously labeled as planet candidates in ExoFOP are classified as planet candidates by ExoMiner++. This reduction in plausible candidates combined with the excellent ranking quality of ExoMiner++ allows the follow-up efforts to be focused on the most likely candidates, increasing the overall planet yield.

  • 29 authors
·
Feb 13, 2025

Dark Energy Survey Year 3 Results: Cosmology from Cosmic Shear and Robustness to Data Calibration

This work, together with its companion paper, Secco and Samuroff et al. (2021), presents the Dark Energy Survey Year 3 cosmic shear measurements and cosmological constraints based on an analysis of over 100 million source galaxies. With the data spanning 4143 deg^2 on the sky, divided into four redshift bins, we produce the highest significance measurement of cosmic shear to date, with a signal-to-noise of 40. We conduct a blind analysis in the context of the ΛCDM model and find a 3% constraint of the clustering amplitude, S_8equiv σ_8 (Ω_{rm m}/0.3)^{0.5} = 0.759^{+0.025}_{-0.023}. A ΛCDM-Optimized analysis, which safely includes smaller scale information, yields a 2% precision measurement of S_8= 0.772^{+0.018}_{-0.017} that is consistent with the fiducial case. The two low-redshift measurements are statistically consistent with the Planck Cosmic Microwave Background result, however, both recovered S_8 values are lower than the high-redshift prediction by 2.3σ and 2.1σ (p-values of 0.02 and 0.05), respectively. The measurements are shown to be internally consistent across redshift bins, angular scales and correlation functions. The analysis is demonstrated to be robust to calibration systematics, with the S_8 posterior consistent when varying the choice of redshift calibration sample, the modeling of redshift uncertainty and methodology. Similarly, we find that the corrections included to account for the blending of galaxies shifts our best-fit S_8 by 0.5σ without incurring a substantial increase in uncertainty. We examine the limiting factors for the precision of the cosmological constraints and find observational systematics to be subdominant to the modeling of astrophysics. Specifically, we identify the uncertainties in modeling baryonic effects and intrinsic alignments as the limiting systematics.

  • 148 authors
·
May 27, 2021

DESI DR2 Results II: Measurements of Baryon Acoustic Oscillations and Cosmological Constraints

We present baryon acoustic oscillation (BAO) measurements from more than 14 million galaxies and quasars drawn from the Dark Energy Spectroscopic Instrument (DESI) Data Release 2 (DR2), based on three years of operation. For cosmology inference, these galaxy measurements are combined with DESI Lyman-α forest BAO results presented in a companion paper. The DR2 BAO results are consistent with DESI DR1 and SDSS, and their distance-redshift relationship matches those from recent compilations of supernovae (SNe) over the same redshift range. The results are well described by a flat ΛCDM model, but the parameters preferred by BAO are in mild, 2.3σ tension with those determined from the cosmic microwave background (CMB), although the DESI results are consistent with the acoustic angular scale θ_* that is well-measured by Planck. This tension is alleviated by dark energy with a time-evolving equation of state parametrized by w_0 and w_a, which provides a better fit to the data, with a favored solution in the quadrant with w_0>-1 and w_a<0. This solution is preferred over ΛCDM at 3.1σ for the combination of DESI BAO and CMB data. When also including SNe, the preference for a dynamical dark energy model over ΛCDM ranges from 2.8-4.2σ depending on which SNe sample is used. We present evidence from other data combinations which also favor the same behavior at high significance. From the combination of DESI and CMB we derive 95% upper limits on the sum of neutrino masses, finding sum m_ν<0.064 eV assuming ΛCDM and sum m_ν<0.16 eV in the w_0w_a model. Unless there is an unknown systematic error associated with one or more datasets, it is clear that ΛCDM is being challenged by the combination of DESI BAO with other measurements and that dynamical dark energy offers a possible solution.

  • 187 authors
·
Oct 8, 2025

Euclid Quick Data Release (Q1): From images to multiwavelength catalogues: the Euclid MERge Processing Function

The Euclid satellite is an ESA mission that was launched in July 2023. \Euclid is working in its regular observing mode with the target of observing an area of 14,000~deg^2 with two instruments, the Visible Camera (VIS) and the Near IR Spectrometer and Photometer (NISP) down to I_{rm E} = 24.5~mag (10, sigma) in the Euclid Wide Survey. Ground-based imaging data in the ugriz bands complement the \Euclid data to enable photo-z determination and VIS PSF modeling for week lensing analysis. Euclid investigates the distance-redshift relation and the evolution of cosmic structures by measuring shapes and redshifts of galaxies and clusters of galaxies out to zsim 2. Generating the multi-wavelength catalogues from \Euclid and ground-based data is an essential part of the \Euclid data processing system. In the framework of the \Euclid Science Ground Segment (SGS), the aim of the MER Processing Function (PF) pipeline is to detect objects in the \Euclid imaging data, measure their properties, and MERge them into a single multi-wavelength catalogue. The MER PF pipeline performs source detection on both visible (VIS) and near-infrared (NIR) images and offers four different photometric measurements: Kron total flux, aperture photometry on PSF-matched images, template fitting photometry, and S\'ersic fitting photometry. Furthermore, the MER PF pipeline measures a set of ancillary quantities, spanning from morphology to quality flags, to better characterise all detected sources. In this paper, we show how the MER PF pipeline is designed, detailing its main steps, and we show that the pipeline products meet the tight requirements that Euclid aims to achieve on photometric accuracy. We also present the other measurements (e.g. morphology) that are included in the OU-MER output catalogues and we list all output products coming out of the MER PF pipeline.

  • 348 authors
·
Mar 19, 2025

ASiT: Local-Global Audio Spectrogram vIsion Transformer for Event Classification

Transformers, which were originally developed for natural language processing, have recently generated significant interest in the computer vision and audio communities due to their flexibility in learning long-range relationships. Constrained by the data hungry nature of transformers and the limited amount of labelled data, most transformer-based models for audio tasks are finetuned from ImageNet pretrained models, despite the huge gap between the domain of natural images and audio. This has motivated the research in self-supervised pretraining of audio transformers, which reduces the dependency on large amounts of labeled data and focuses on extracting concise representations of audio spectrograms. In this paper, we propose Local-Global Audio Spectrogram vIsion Transformer, namely ASiT, a novel self-supervised learning framework that captures local and global contextual information by employing group masked model learning and self-distillation. We evaluate our pretrained models on both audio and speech classification tasks, including audio event classification, keyword spotting, and speaker identification. We further conduct comprehensive ablation studies, including evaluations of different pretraining strategies. The proposed ASiT framework significantly boosts the performance on all tasks and sets a new state-of-the-art performance in five audio and speech classification tasks, outperforming recent methods, including the approaches that use additional datasets for pretraining.

  • 5 authors
·
Nov 23, 2022

ADEM-VL: Adaptive and Embedded Fusion for Efficient Vision-Language Tuning

Recent advancements in multimodal fusion have witnessed the remarkable success of vision-language (VL) models, which excel in various multimodal applications such as image captioning and visual question answering. However, building VL models requires substantial hardware resources, where efficiency is restricted by two key factors: the extended input sequence of the language model with vision features demands more computational operations, and a large number of additional learnable parameters increase memory complexity. These challenges significantly restrict the broader applicability of such models. To bridge this gap, we propose ADEM-VL, an efficient vision-language method that tunes VL models based on pretrained large language models (LLMs) by adopting a parameter-free cross-attention mechanism for similarity measurements in multimodal fusion. This approach only requires embedding vision features into the language space, significantly reducing the number of trainable parameters and accelerating both training and inference speeds. To enhance representation learning in fusion module, we introduce an efficient multiscale feature generation scheme that requires only a single forward pass through the vision encoder. Moreover, we propose an adaptive fusion scheme that dynamically discards less relevant visual information for each text token based on its attention score. This ensures that the fusion process prioritizes the most pertinent visual features. With experiments on various tasks including visual question answering, image captioning, and instruction-following, we demonstrate that our framework outperforms existing approaches. Specifically, our method surpasses existing methods by an average accuracy of 0.77% on ScienceQA dataset, with reduced training and inference latency, demonstrating the superiority of our framework. The code is available at https://github.com/Hao840/ADEM-VL.

  • 6 authors
·
Oct 23, 2024 2

LEO-VL: Towards 3D Vision-Language Generalists via Data Scaling with Efficient Representation

Developing 3D-VL generalists capable of understanding 3D scenes and following natural language instructions to perform a wide range of tasks has been a long-standing goal in the 3D-VL community. Despite recent progress, 3D-VL models still lag behind their 2D counterparts in capability and robustness, falling short of the generalist standard. A key obstacle to developing 3D-VL generalists lies in data scalability, hindered by the lack of an efficient scene representation. We propose LEO-VL, a 3D-VL model built upon condensed feature grid (CFG), an efficient scene representation that bridges 2D perception and 3D spatial structure while significantly reducing token overhead. This efficiency unlocks large-scale training towards 3D-VL generalist, for which we curate over 700k high-quality 3D-VL data spanning four domains of real-world indoor scenes and five tasks such as captioning and dialogue. LEO-VL achieves state-of-the-art performance on a variety of 3D QA benchmarks, including SQA3D, MSQA, and Beacon3D. Ablation studies confirm the efficiency of our representation, the importance of task and scene diversity, and the validity of our data curation principle. Furthermore, we introduce SceneDPO, a novel post-training objective that enhances the robustness of 3D-VL models. We hope our findings contribute to the advancement of scalable and robust 3D-VL generalists.

  • 11 authors
·
Jun 11, 2025

Teaching Structured Vision&Language Concepts to Vision&Language Models

Vision and Language (VL) models have demonstrated remarkable zero-shot performance in a variety of tasks. However, some aspects of complex language understanding still remain a challenge. We introduce the collective notion of Structured Vision&Language Concepts (SVLC) which includes object attributes, relations, and states which are present in the text and visible in the image. Recent studies have shown that even the best VL models struggle with SVLC. A possible way of fixing this issue is by collecting dedicated datasets for teaching each SVLC type, yet this might be expensive and time-consuming. Instead, we propose a more elegant data-driven approach for enhancing VL models' understanding of SVLCs that makes more effective use of existing VL pre-training datasets and does not require any additional data. While automatic understanding of image structure still remains largely unsolved, language structure is much better modeled and understood, allowing for its effective utilization in teaching VL models. In this paper, we propose various techniques based on language structure understanding that can be used to manipulate the textual part of off-the-shelf paired VL datasets. VL models trained with the updated data exhibit a significant improvement of up to 15% in their SVLC understanding with only a mild degradation in their zero-shot capabilities both when training from scratch or fine-tuning a pre-trained model.

  • 11 authors
·
Nov 21, 2022

Going Beyond Nouns With Vision & Language Models Using Synthetic Data

Large-scale pre-trained Vision & Language (VL) models have shown remarkable performance in many applications, enabling replacing a fixed set of supported classes with zero-shot open vocabulary reasoning over (almost arbitrary) natural language prompts. However, recent works have uncovered a fundamental weakness of these models. For example, their difficulty to understand Visual Language Concepts (VLC) that go 'beyond nouns' such as the meaning of non-object words (e.g., attributes, actions, relations, states, etc.), or difficulty in performing compositional reasoning such as understanding the significance of the order of the words in a sentence. In this work, we investigate to which extent purely synthetic data could be leveraged to teach these models to overcome such shortcomings without compromising their zero-shot capabilities. We contribute Synthetic Visual Concepts (SyViC) - a million-scale synthetic dataset and data generation codebase allowing to generate additional suitable data to improve VLC understanding and compositional reasoning of VL models. Additionally, we propose a general VL finetuning strategy for effectively leveraging SyViC towards achieving these improvements. Our extensive experiments and ablations on VL-Checklist, Winoground, and ARO benchmarks demonstrate that it is possible to adapt strong pre-trained VL models with synthetic data significantly enhancing their VLC understanding (e.g. by 9.9% on ARO and 4.3% on VL-Checklist) with under 1% drop in their zero-shot accuracy.

  • 11 authors
·
Mar 30, 2023

OmniThoughtVis: A Scalable Distillation Pipeline for Deployable Multimodal Reasoning Models

Recent multimodal large language models (MLLMs) have shown strong chain-of-thought (CoT) reasoning ability on vision-language tasks, but their direct deployment in real-world systems is often limited by latency and resource constraints. In practice, smaller MLLMs are preferred for online serving, yet their reasoning performance is bottlenecked by the lack of large-scale, high-quality multimodal CoT supervision. In this paper, we present OmniThoughtVis, a scalable data curation and distillation pipeline for transferring multimodal reasoning capabilities from high-capacity teacher models to smaller, deployment-oriented MLLMs. Starting from a diverse open-source seed pool, our pipeline generates structured CoT traces and performs joint annotation of reasoning difficulty, answer quality, and semantic task tags. To maintain data quality at scale, we combine rule-based filtering, difficulty-aware selection, and tag-based diversity sampling, resulting in a curated corpus of 1.8M samples that supports controllable subset construction for downstream training. We use OmniThoughtVis to distill Qwen3-VL models from 2B to 8B parameters and evaluate them on nine multimodal reasoning benchmarks. The resulting distilled models show consistent gains across model scales, including improvements of up to +16.8 points on MathVerse and +5.6 points on MMMU-Pro for the 4B model. Notably, the distilled 4B model matches or surpasses the undistilled 8B baseline on several tasks, highlighting the practical value of scalable reasoning distillation for deployment-oriented MLLMs.

  • 5 authors
·
May 11