Examining Real-World Use of Collaboration Tools through Body Tracking Sensors
Abstract
Body tracking sensors have become cheap and accessible enough for use as a component in semi-automated long-term deployment studies on how people use collaboration tools or other interactive technology in the field. In this position paper we briefly describe our experience administering a body tracking sensor setup for a small fleet of interactive ambient displays at a university, outline the benefits and drawbacks of using body tracking data to quantitatively examine how people interact with technology, and summarize the technical and methodological issues involved. We follow with a summary of our methodological framework for integrating qualitative ethnographic methods with this quantitative and sensor-based approach.
CCS Concepts
• Human-centered computing → Field studies; Empirical studies in collaborative and social computing; Empirical studies in ubiquitous and mobile computing; • Hardware → Sensor applications and deployments.
Keywords
body tracking, deployment studies, field studies, pose estimation
License
This work is licensed under a Creative Commons Attribution 4.0 International License.
Mensch und Computer 2025 – Workshopband, Gesellschaft für Informatik e.V., 31. August - 03. September 2025, Chemnitz, Germany
© 2025 Copyright held by the owner/author(s). Publication rights licensed to GI. https://doi.org/10.18420/muc2025-mci-ws13-162
1. Introduction
As part of our long-term research agenda, the HCI group at the University of the Bundeswehr Munich has been locally operating a small number of interactive ambient displays. Their central design goal is to provide serendipitous information to colleagues about things going on in different teams and institutions throughout the university, fostering awareness and encouraging future collaborations [5, 7].
In 2021 we began using body tracking sensors – technologically speaking, depth cameras – to begin to understand how people interacted with our public screens without active supervision and over longer deployment times. Our idea was that anonymous pose recordings (which show people’s body and limb orientations while mostly omitting physical features that could be used to identifiy individuals) gathered over longer periods of time would let us explore new insights into how technological collaboration tools are used in practice [4], especially in light of societal shifts towards more hybrid work and distance-based collaboration [6, 16].
This approach was, and still is, largely underexplored at a methodological level, posing significant challenges to researchers aiming to integrate large-scale quantitative body tracking data with more established qualitative ethnographic approaches [1, 15]. In this paper, we will give a short overview of our reseach so far and the open questions we have identified.
2. Handling body tracking data
While programmable sensors capable of on-device pose estimation of multiple people in their field of view have been in financial reach of researchers and enthusiasts for well over a decade now [8, 14], the first major obstacle we faced was the lack of an established format for storing and transmitting body tracking data. Existing data formats were either vendor-specific (e.g. Microsoft Kinect Studio recordings) or not suitable for stationary body tracking setups where passers-by may enter and leave the area of interest at any time (e.g. Biovision Hierarchy format). To be able to do non-trivial empirical work with body tracking data, we first had to design a format suitable for storing body tracking data as well as transferring it in bulk or in real time, and then develop software tools to read, write, and visualize (see Figure 1) data in this format. This gave us the ability to play back and scrub through individual body tracking recordings, look at interactions from different angles, and generally get an impression of the quality of the sensor data. The design process is described in more detail in Fietkau [2].
Examining larger quantities of body tracking recordings for specific hypotheses requires bespoke tooling that can analyze the specific aspects relevant to those hypotheses. Our work examining two-dimensional walking paths as clustered time series data [11] serves as a practical example of the kind of quantitative insight we have so far been able to extract from body tracking data.
Apart from the technical concerns of using and storing body tracking data, there are also significant ethical and regulatory challenges. The ostensible anonymity of body tracking data, with its absence of physically identifying features as one would see e.g. in video recordings, can fall apart when you consider ways to identify individuals from their specific movements (e.g. gait analysis [13], characteristic gestures) or interactions with external data, such as correlating people’s presence in body tracking recordings with vacation dates or lab sign-in sheets. Even though we had initially planned to publish the raw body tracking data recorded at our deployment setups, a deeper investigation of the potentials for deanonymization caused us to reverse that decision and publish only summarized statistical data.
Furthermore, we must not underestimate the normative role of body tracking models in the data gathering phase. By virtue of categorizing image areas into “humans” and “not humans,” the way the process works entails making specific assumptions about what consitutes a human body (e.g. count, orientations, and relative sizes of specific limbs) that may not be inclusive towards people with physical disabilities, whose appearance in body tracking data may be distorted or even quietly omitted altogether. We have discussed these challenges in more depth in Fietkau and Schwarzer [3].
To summarize, we were able to break new ground in the use of body tracking sensors as a complementary quantitative technique to ethnographic observations of deployment studies by creating a vendor-independent data exchange format for body tracking recordings and publishing tools to work with this kind of data. We have previously demonstrated the use of these tools for early-stage exploration as well as in-depth quantitative analysis of large amounts of body tracking data. There are ethical and legal concerns that are specific to body tracking data and that have not yet been fully explored. However, the most significant challenge we face is correlating the information from timestamped body tracking data with insights from qualitative empirical methods.
3. Integration with qualitative methods
The past few years that we have spent working with body tracking data have resulted in some fruitful research outcomes, but they have also shown us the boundaries of this quantitatively driven approach. Specifically, we frequently found ourselves unable to discern the reasons for people’s behavior from body tracking data alone. The insight we have gained into what can and cannot be drawn from such data has motivated us to continue developing a detailed methodological framework for integrating quantitative analysis of body tracking data with qualitative ethnographic methods such as diary studies [9], field observations, and interviews (see Figure 2).
Building on prior work on collaboration tools for agile software engineering teams [10], we decided to put a stronger research focus on hybrid work environments, i.e. contexts where people professionally collaborate both on-site as well as over long physical distances. Additionally, we aimed to increase our use of qualitative methods to understand the structure and purpose of collaborative processes in these environments, and to link these insights to quantitative examinations of body tracking data through a deliberate synchronization process. The preliminary methodological framework shown in Figure 2 is an effort to structure this ongoing work.
The left side of Figure 2 is rooted in prior experience researching work environments using ethnographic methods [10], where we were able to gain insight into co-located work processes through observations and interviews. The addition of diary studies is our methodological probe into the remote part of the hybrid environment, which we had previously not sufficiently addressed.
The right side of the diagram reflects the latest iteration of our approach towards analyzing body tracking data. The process is separated into data collection, preparation (plausibility checking and filtering), exploration (interactive preliminary analysis and identification of promising trends), feature extraction (honing in on specific hypotheses and transforming/mapping the data as needed), and analysis (substantiating the hypotheses).
The two tracks are linked via a synchronization process, in which we examine how the processes can inform and augment each other, e.g. by labeling segments of the quantitative data based on results from qualitative observations, or by choosing new qualitative experiments based on promising clusters in the quantitative data.
The framework is described more deeply in Schwarzer et al. [12].
4. Conclusion
Body tracking data has proven itself as an underutilized avenue for long-term field deployment studies into people’s physical behavior. While there are still significant technological and methodological challenges, it is possible to use quantitative analysis of long-term body tracking data as one component of many in mixed methods research projects into complex socio-technical environments.
We hope to continue our efforts analyzing hybrid work environments using these methods, and to keep refining the methodology in turn.
Acknowledgments
With thanks to my colleagues in Munich: Michael Koch, Laura Stojko, and Buket Garthoff; as well as our project colleagues in Hamburg: Jan Schwarzer, Susanne Draheim, and Kai von Luck. This summary is based on all of the work that we have put into our projects together over the years.