Search

The basic body shape of a person does not change within a single video. However, most SOTA human mesh estimation (HME) models output a slightly different body shape for each video frame, which results in inconsistent body shapes for the same person. In contrast, we leverage anthropometric measurements like tailors are already obtaining from humans for centuries. We create a model called A2B that converts such anthropometric measurements to body shape parameters of human mesh models. Moreover, we find that finetuned SOTA 3D human pose estimation (HPE) models outperform HME models regarding the precision of the estimated keypoints. We show that applying inverse kinematics (IK) to the results of such a 3D HPE model and combining the resulting body pose with the A2B body shape leads to superior and consistent human meshes for challenging datasets like ASPset or fit3D, where we can lower the MPJPE by over 30 mm compared to SOTA HME models. Further, replacing HME models estimates of the body shape parameters with A2B model results not only increases the performance of these HME models, but also leads to consistent body shapes.

In this paper we introduce a new dataset containing instance segmentation masks for ten different categories of winter sports equipment, called WSESeg (Winter Sports Equipment Segmentation). Furthermore, we carry out interactive segmentation experiments on said dataset to explore possibilities for efficient further labeling. The SAM and HQ-SAM models are conceptualized as foundation models for performing user guided segmentation. In order to measure their claimed generalization capability we evaluate them on WSESeg. Since interactive segmentation offers the benefit of creating easily exploitable ground truth data during test-time, we are going to test various online adaptation methods for the purpose of exploring potentials for improvements without having to fine-tune the models explicitly. Our experiments show that our adaptation methods drastically reduce the Failure Rate (FR) and Number of Clicks (NoC) metrics, which generally leads faster to better interactive segmentation results., Comment: 7 pages, 1 figure, 3 tables, Accepted at CBMI 2024

In panoptic scene graph generation (PSGG), models retrieve interactions between objects in an image which are grounded by panoptic segmentation masks. Previous evaluations on panoptic scene graphs have been subject to an erroneous evaluation protocol where multiple masks for the same object can lead to multiple relation distributions per mask-mask pair. This can be exploited to increase the final score. We correct this flaw and provide a fair ranking over a wide range of existing PSGG models. The observed scores for existing methods increase by up to 7.4 mR@50 for all two-stage methods, while dropping by up to 19.3 mR@50 for all one-stage methods, highlighting the importance of a correct evaluation. Contrary to recent publications, we show that existing two-stage methods are competitive to one-stage methods. Building on this, we introduce the Decoupled SceneFormer (DSFormer), a novel two-stage model that outperforms all existing scene graph models by a large margin of +11 mR@50 and +10 mNgR@50 on the corrected evaluation, thus setting a new SOTA. As a core design principle, DSFormer encodes subject and object masks directly into feature space.

Utilizing transformer architectures for semantic segmentation of high-resolution images is hindered by the attention's quadratic computational complexity in the number of tokens. A solution to this challenge involves decreasing the number of tokens through token merging, which has exhibited remarkable enhancements in inference speed, training efficiency, and memory utilization for image classification tasks. In this paper, we explore various token merging strategies within the framework of the Segformer architecture and perform experiments on multiple semantic segmentation and human pose estimation datasets. Notably, without model re-training, we, for example, achieve an inference acceleration of 61% on the Cityscapes dataset while maintaining the mIoU performance. Consequently, this paper facilitates the deployment of transformer-based architectures on resource-constrained devices and in real-time applications., Comment: 7 pages, to be published in IEEE International Conference on Multimedia Information Processing and Retrieval (MIPR) 2024

In this work, we analyze the applicability of Inverse Dynamic Game (IDG) methods based on the Minimum Principle (MP). The IDG method determines unknown cost functions in a single- or multi-agent setting from observed system trajectories by minimizing the so-called residual error, i.e. the extent to which the optimality conditions of the MP are violated with a current guess of cost functions. The main assumption of the IDG method to recover cost functions such that the resulting trajectories match the observed ones is that the given trajectories are the result of a Dynamic Game (DG) problem with known parameterized cost function structures. However, in practice, when the IDG method is used to identify the behavior of unknown agents, e.g. humans, this assumption cannot be guaranteed. Hence, we introduce the notion of the trustworthiness of the residual error and provide necessary conditions for it to define when the IDG method based on the MP is applicable to such problems. From the necessary conditions, we conclude that the MP-based IDG method cannot be used to validate DG models for unknown agents but can yield under certain conditions robust parameter identifications, e.g. to measurement noise. Finally, we illustrate these conclusions by validating a DG model for the collision avoidance behavior between two mobile robots with human operators.

Neutron interferometry has played a distinctive role in fundamental science and characterization of materials. Moir\'e neutron interferometers are candidate next-generation instruments: they offer microscopy-like magnification of the signal, enabling direct camera recording of interference patterns across the full neutron wavelength spectrum. Here we demonstrate the extension of phase-grating moir\'e interferometry to two-dimensional geometries. Our fork-dislocation phase gratings reveal phase singularities in the moir\'e pattern, and we explore orthogonal moir\'e patterns with two-dimensional phase-gratings. Our measurements of phase topologies and gravitationally induced phase shifts are in good agreement with theory. These techniques can be implemented in existing neutron instruments to advance interferometric analyses of emerging materials and precision measurements of fundamental constants.

We present a novel method for precise 3D object localization in single images from a single calibrated camera using only 2D labels. No expensive 3D labels are needed. Thus, instead of using 3D labels, our model is trained with easy-to-annotate 2D labels along with the physical knowledge of the object's motion. Given this information, the model can infer the latent third dimension, even though it has never seen this information during training. Our method is evaluated on both synthetic and real-world datasets, and we are able to achieve a mean distance error of just 6 cm in our experiments on real data. The results indicate the method's potential as a step towards learning 3D object location estimation, where collecting 3D data for training is not feasible.

Current scene graph datasets suffer from strong long-tail distributions of their predicate classes. Due to a very low number of some predicate classes in the test sets, no reliable metrics can be retrieved for the rarest classes. We construct a new panoptic scene graph dataset and a set of metrics that are designed as a benchmark for the predictive performance especially on rare predicate classes. To construct the new dataset, we propose a model-assisted annotation pipeline that efficiently finds rare predicate classes that are hidden in a large set of images like needles in a haystack. Contrary to prior scene graph datasets, Haystack contains explicit negative annotations, i.e. annotations that a given relation does not have a certain predicate class. Negative annotations are helpful especially in the field of scene graph generation and open up a whole new set of possibilities to improve current scene graph generation models. Haystack is 100% compatible with existing panoptic scene graph datasets and can easily be integrated with existing evaluation pipelines. Our dataset and code can be found here: https://lorjul.github.io/haystack/. It includes annotation files and simple to use scripts and utilities, to help with integrating our dataset in existing work.

Phase-grating moire interferometers (PGMIs) have emerged as promising candidates for the next generation of neutron interferometry, enabling the use of a polychromatic beam and manifesting interference patterns that can be directly imaged by existing neutron cameras. However, the modeling of the various PGMI configurations is limited to cumbersome numerical calculations and backward propagation models which often do not enable one to explore the setup parameters. Here we generalize the Fresnel scaling theorem to introduce a k-space model for PGMI setups illuminated by a cone beam, thus enabling an intuitive forward propagation model for a wide range of parameters. The interference manifested by a PGMI is shown to be a special case of the Talbot effect, and the optimal fringe visibility is shown to occur at the moire location of the Talbot distances. We derive analytical expressions for the contrast and the propagating intensity profiles in various conditions, and analyze the behaviour of the dark-field imaging signal when considering sample characterization. The model's predictions are compared to experimental measurements and good agreement is found between them. Lastly, we propose and experimentally verify a method to recover contrast at typically inaccessible PGMI autocorrelation lengths. The presented work provides a toolbox for analyzing and understanding existing PGMI setups and their future applications, for example extensions to two-dimensional PGMIs and characterization of samples with non-trivial structures.

Challenges drive the state-of-the-art of automated medical image analysis. The quantity of public training data that they provide can limit the performance of their solutions. Public access to the training methodology for these solutions remains absent. This study implements the Type Three (T3) challenge format, which allows for training solutions on private data and guarantees reusable training methodologies. With T3, challenge organizers train a codebase provided by the participants on sequestered training data. T3 was implemented in the STOIC2021 challenge, with the goal of predicting from a computed tomography (CT) scan whether subjects had a severe COVID-19 infection, defined as intubation or death within one month. STOIC2021 consisted of a Qualification phase, where participants developed challenge solutions using 2000 publicly available CT scans, and a Final phase, where participants submitted their training methodologies with which solutions were trained on CT scans of 9724 subjects. The organizers successfully trained six of the eight Final phase submissions. The submitted codebases for training and running inference were released publicly. The winning solution obtained an area under the receiver operating characteristic curve for discerning between severe and non-severe COVID-19 of 0.815. The Final phase solutions of all finalists improved upon their Qualification phase solutions.HSUXJM-TNZF9CHSUXJM-TNZF9C

Analyses based on the body posture are crucial for top-class athletes in many sports disciplines. If at all, coaches label only the most important keypoints, since manual annotations are very costly. This paper proposes a method to detect arbitrary keypoints on the limbs and skis of professional ski jumpers that requires a few, only partly correct segmentation masks during training. Our model is based on the Vision Transformer architecture with a special design for the input tokens to query for the desired keypoints. Since we use segmentation masks only to generate ground truth labels for the freely selectable keypoints, partly correct segmentation masks are sufficient for our training procedure. Hence, there is no need for costly hand-annotated segmentation masks. We analyze different training techniques for freely selected and standard keypoints, including pseudo labels, and show in our experiments that only a few partly correct segmentation masks are sufficient for learning to detect arbitrary keypoints on limbs and skis., Comment: accepted at CV4WS2023 (WACV 2023 Workshops)

Since COVID strongly affects the respiratory system, lung CT-scans can be used for the analysis of a patients health. We introduce a neural network for the prediction of the severity of lung damage and the detection of a COVID-infection using three-dimensional CT-data. Therefore, we adapt the recent ConvNeXt model to process three-dimensional data. Furthermore, we design and analyze different pretraining methods specifically designed to improve the models ability to handle three-dimensional CT-data. We rank 2nd in the 1st COVID19 Severity Detection Challenge and 3rd in the 2nd COVID19 Detection Challenge., Comment: 17 pages, 3 figures, informations about challenge submission

Nearly all Human Pose Estimation (HPE) datasets consist of a fixed set of keypoints. Standard HPE models trained on such datasets can only detect these keypoints. If more points are desired, they have to be manually annotated and the model needs to be retrained. Our approach leverages the Vision Transformer architecture to extend the capability of the model to detect arbitrary keypoints on the limbs of persons. We propose two different approaches to encode the desired keypoints. (1) Each keypoint is defined by its position along the line between the two enclosing keypoints from the fixed set and its relative distance between this line and the edge of the limb. (2) Keypoints are defined as coordinates on a norm pose. Both approaches are based on the TokenPose architecture, while the keypoint tokens that correspond to the fixed keypoints are replaced with our novel module. Experiments show that our approaches achieve similar results to TokenPose on the fixed keypoints and are capable of detecting arbitrary keypoints on the limbs., Comment: accepted at CVSports (CVPR 2022 Workshops)

Hyperbolic lattices underlie a new form of quantum matter with potential applications to quantum computing and simulation and which, to date, have been engineered artificially. A corresponding hyperbolic band theory has emerged, extending 2-dimensional Euclidean band theory in a natural way to higher-genus configuration spaces. Attempts to develop the hyperbolic analogue of Bloch's theorem have revealed an intrinsic role for algebro-geometric moduli spaces, notably those of stable bundles on a curve. We expand this picture to include Higgs bundles, which enjoy natural interpretations in the context of band theory. First, their spectral data encodes a crystal lattice and momentum, providing a framework for symmetric hyperbolic crystals. Second, they act as a complex analogue of crystal momentum. As an application, we elicit a new perspective on Euclidean band theory. Finally, we speculate on potential interactions of hyperbolic band theory, facilitated by Higgs bundles, with other themes in mathematics and physics., Comment: 38 pages, 13 figures, 1 table