The importance of real user measurements
We independently collect and analyze mobile user experience on every major network operator around the globe, from more than 100 million Android and IOS smartphones worldwide, capturing over 3 billion measurements every day.
By analyzing on-device measurements recorded in the places where people actually live, work and travel, we report on mobile network service the way users truly experience it – with no simulations, no predictions and no idealized testing conditions. From a scientific standpoint, it’s the difference between conducting hypothetical laboratory tests and observing real events in their natural environment. Scientists have long held that the latter yields far more accurate results. We could not agree more.
Our user-centric approach provides a rich source of measurements from which network operators, equipment manufacturers, analysts and consumers can accurately understand the experience real people receive on mobile networks, at a country, regional and global level.
By sharing our insights, we address the critical need for the industry to understand mobile network experience from the source that matters most: actual users.
How we collect and analyze the data
Our measurement, aggregation and analysis process applies rigourous data science principles and follows exacting quality standards to arrive at the detailed findings we share with customers and the subset of results featured in our public reports.
- We collect data from a range of our own Smartphone apps and partner apps. The vast majority of this data is collected via automated tests that run at random points in time. Collecting measurements in this way ensures that they represent the typical experience available to a user at any given moment. The breadth of measurements we capture sets us apart from other on-device methodologies that base their findings solely on user-generated tests.
- Measurements are captured at all hours of the day, every day of the year, under conditions of normal usage, inside buildings and outdoors, in cities (urban locations), the countryside (rural locations), and everywhere in between, including places often referred to as notspots or dead zones. Our approach enables us to collect data in places other methodologies cannot.
- The majority of our measurements are captured in indoor locations, which is not surprising considering that’s where people spend the vast majority of their time. The breadth of indoor measurements we collect sets our approach apart from other methodologies (e.g., drive-testing) that collect only outdoor measurements and are limited to testing roads and therefore not representative of real-world experience.
- We run our tests on the same internet servers consumers connect to everyday, enabling us to measure the full end-to-end consumer network experience, from the user device all the way to the Content Delivery Networks (CDNs). Running our tests this way ensures our findings mirror the path of the majority of everyday internet traffic. We never run our tests on dedicated test servers, which only see part of the user-experience picture.
- Between measurement collection and reporting, we apply a rigorous post-processing system that filters the raw data test results to remove outliers and calculate the average metrics in a way that ensures they are representative of typical user experience. A detailed and robust methodology for analysis of the data is at least as — if not more — important as how the data is collected and OpenSignal’s is the most advanced and sophisticated analytical methodology that exists.
The metrics we report on
We capture a series of user measurements from our testers and report on a subset of our findings in our public reports. For more information about the metrics we publicly report on, and new metrics being developed, read this blog post: Understanding mobile network experience: What do OpenSignal’s metrics mean?
New metrics in the pipeline
We are constantly evaluating new ways to measure users’ true mobile network experience and frequently report on our findings as these metrics progress through the development stage. Stay up-to-date on our latest developmental metrics on our blog and in our newsletter.