Brief Summary
In this episode of the 8020 podcast, Rob, known as the Quantified Scientist, discusses his journey into the Quantified Self movement, his methods for testing wearables, and his insights on data accuracy. He shares his experiences with sleep tracking, heart rate monitoring, and GPS accuracy, offering recommendations for health trackers based on individual needs.
- Origin of Quantified Self Movement
- Wearable Testing Methodology
- Data Accuracy and Limitations
- Health Tracker Recommendations
How Did You Get Started in the Quantified Self Movement?
Rob, a data scientist, started by correlating his heart rate with music using a Fitbit Charge 2 to communicate science in an understandable way. This evolved into a large-scale Quantified Self project involving weekly brain MRIs, daily sleep EEGs, and microbiome measurements. The project aims to create a comprehensive dataset to study the connections between the brain, sleep, and gut, similar to the blueprint project but with a different focus.
How Did You First Get Into Science?
Rob's interest in science grew naturally from his quantitative aptitude and interest in understanding rather than memorizing. He initially studied chemistry, which led him to data science due to the quicker results and ease of communication compared to lab work. YouTube became a platform for him to share his findings and provide educational content.
How Do You Use Data Science on YouTube?
Rob utilizes his data science background on YouTube by using gold and silver standard reference devices for comparison. He collaborates with sleep labs for sleep data and uses chest straps for heart rate monitoring. He analyzes the data systematically to provide fair comparisons of different devices, balancing the depth of analysis with the breadth of devices tested.
What’s Your Process for Testing Products?
To ensure fair comparisons, Rob performs the same types of workouts in similar environments for all wearables. This includes standardized interval workouts on a home trainer, outdoor cycling sessions, and weightlifting sessions. He aims to maintain a consistent protocol while still using the workouts as a personal fitness tool.
How Do You Determine What Data Is Really Important?
Rob acknowledges the limitations of N-of-one experiments but believes they still provide valuable insights. He notes that most health reviewers are male, which may introduce bias. While general trends in device performance are likely consistent across individuals, variations can occur based on specific protocols. He emphasizes the importance of consulting multiple reviewers to get a comprehensive impression.
Software Algorithms
Software plays a crucial role in wearable performance, as demonstrated by the Oura Ring, which significantly improved its sleep stage tracking with a firmware upgrade. The latest Oura Ring algorithm is now comparable to Apple's. However, sleep trackers generally perform worse for individuals with significant sleep issues.
How Can Changes in Algorithms Impact User Habits?
Changes in algorithms can affect users who make decisions based on the data provided by wearables. While algorithm improvements are generally positive, they can lead to feelings of lost progress if metrics like deep sleep decrease. Consistency in the algorithm is important for reliable comparisons over time.
Do You See Any Value in Tracking Sleep Stages?
Tracking sleep stages can be interesting for individuals who want to understand their sleep patterns, but it's more critical to monitor awakenings, time spent awake, and time to fall asleep. Deviations from normal sleep patterns can indicate underlying issues. Scientists also find value in sleep stages for research purposes.
Wearables with Poor Tracking of Sleep Data?
While most brands can track total time spent in bed, the accuracy of awake moment tracking varies. It's best to avoid cheap Chinese brands, as they often lack the large datasets needed to train accurate sleep stage tracking models. Xiaomi and Huawei devices generally do not have good sleep tracking.
Some Examples of Golden and Silver Standards
The gold standard for sleep tracking is polysomnography (PSG), which involves multiple electrodes measuring brain waves, eye movements, and muscle activity. Silver standards are not officially defined but refer to less comprehensive methods. PSG data is manually reviewed to score sleep stages, but even trained scientists may only agree around 90% of the time.
Why Is PSG the Gold Standard?
Polysomnography (PSG) is the gold standard because sleep stages were originally defined using EEG-type measurements, providing a high level of detail. However, even with PSG, there is inherent subjectivity in scoring sleep stages. Recently, AI algorithms have been developed to automate the scoring process, aiming for more robust results.
What’s the Next Best Sleep Algorithm?
The next best sleep algorithm involves AI-driven automation, which is becoming more prevalent. These algorithms often use data from multiple scorers to improve reliability. It's important to recognize that sleep may be a continuum rather than discrete stages.
What Are People Not Seeing in How Wearables Are Tested?
Independent validation of wearables can be misleading because the research is often funded by the companies themselves. While peer review should ensure the methods are solid, there may be biases in data exclusion or experimental setup. Sponsored research may be designed to favor the company's product.
What Is Your Opinion on Affiliate Links?
Rob has specific rules for using affiliate links. He includes Amazon links for all products, regardless of their quality, due to the low commission. He only joins affiliate programs for products he uses and recommends, aiming to remain unbiased. He avoids sponsorships to maintain editorial independence.
Shervin’s Philosophy on Affiliate Links?
Shervin puts the links there even if he doesn't like the product. He tries not to look at the numbers because it changes if he makes more or less content about the product. He still tries to stay true and honest to what he would tell his friends all the videos.
How Do Different Heart Monitors Work?
Chest straps use electrocardiography (ECG) to measure the electrical signals of the heart, providing clear and sharp heart rate signals. Other trackers, such as those on the wrist or finger, use photoplethysmography (PPG), which shines light on the skin and measures the reflected light to detect heart rate. Chest straps are generally more reliable, while wrist-worn devices offer greater convenience.
When Do Wrist Sensors Work Well and When Don’t They?
Wrist sensors work best when the wrist is stable and there is less tension. Indoor cycling is generally fine, but quick changes in heart rate can be challenging. Running can be problematic due to cadence lock, where the device locks onto the running frequency instead of the heart rate.
Which Is the Best Heart Rate Tracker Among These Wrist Devices?
Based on limited testing, the Apple Watch performs best as a wrist-worn heart rate tracker. Garmin devices tend to be mediocre, especially during outdoor cycling. Huawei performs well, but access to the American market may be limited.
How Can I Ensure My Wrist Heart Rate Monitor Is Accurate?
To ensure accuracy, follow the manufacturer's specifications for tightness, but avoid making it too tight. Some people wear watches on the inside of the wrist, which may provide a better signal, but this requires further testing.
To Those Claiming a Watch’s Precision Based on Use
People who claim a watch is more accurate without any data should buy a chest strap and test it out. Skin tone is something that should be accounted for, because a lot of these watches are made for white or light-skinned people.
If I Have an Apple Watch, Do I Need to Buy a Heart Rate Strap Band?
The Apple Watch is generally good enough, but it may struggle during weightlifting. It's important to consider whether having precise heart rate data is necessary for your specific goals.
Will Having an Accurate Wearable Change My Life?
Accurate wearables can improve sleep through advice and recommendations, and tracking resting heart rate can benefit non-athletes. However, it's important not to over-rely on these metrics and to consider personal feelings and goals.
GPS Accuracy and Learnings
Rob tests GPS accuracy by cycling the same route to work five times a week and tracking consistency. This indirect method reveals significant differences between devices. Garmin is generally the best at GPS, while some devices, like the Fitbit Charge 6, have poor GPS performance.
Which Are the Best GPS Wearables?
Garmin watches are generally decent at GPS, especially when set to the best setting. While there are multiple satellite systems, the key is to connect to more satellites for better triangulation.
Which Scores Are Important?
Readiness and recovery scores can be tricky to interpret. While some studies correlate these scores with subjective metrics, it's unclear how sophisticated they are. Total sleep time may be a significant factor in these scores.
Why Do You Make All These Videos?
Rob creates these videos because he felt the content wasn't out there in the way he would want to see it. He wanted people to test it in a systematic way and actually tell him what is best for that and then actually compare it against all the others in in some kind of reasonably reliable metric.
What Are Your Future Goals?
Rob envisions a large testing center where multiple people test devices using a standardized protocol. He aims to collect data from users worldwide with reliable sleep monitors and chest straps to create a comprehensive resource for health tracking.
Research Apps
Apple has a research app that institutions can use to pull data from Apple Health. Rob is working with a startup called Tera to build a website where people can connect their wearables and share data, including Apple devices. Some companies, like Whoop, have limited data-sharing options.
Health Tracker Recommendations
Recommendations depend on individual goals and requirements. For a cheap option with reasonable tracking, the Fitbit Charge 6 is a good choice. For Apple users, the Apple Watch integrates well with their ecosystem. Whoop is solid throughout with a nice app with feedback. Aura has better sleep stage tracking but poor heart rate tracking during exercise. Garmin is good for long battery life and GPS tracking.
Rapid-Fire Questions
- $300 Choice: Choros or Garmin? Pace 3.
- Sleep with Gear: No issues.
- Origin: Dutch.
- Apple Watch: Great at heart rate and sleep, but battery life sucks and their app is not the best.
Where Can People Find You?
People can find Rob on YouTube at The Quantified Scientist.
