Activity trackers are NOT exactly the same

In a previous post, I wrote that activity trackers are "all the same." At their core, activity trackers count the number of steps you take (using an accelerometer), and most do a decent job when you compare them. I described an informal study performed by a writer at Wirecutter in which the author strapped on 4 different trackers to monitor activity over 6 days. For all trackers, the total number of steps counted were within 5-10% of each other suggesting that they produce similar step counts.

A related study in Consumer Reports "combined step count accuracy for walking for five minutes on a level treadmill at 3 mph, using an elliptical exerciser for five minutes, walking up and down several flights of stairs and picking up toys from the floor." They tested 6 trackers and gave all of them either an excellent or good score for step counting accuracy.

However what is lacking is a quantitative assessment of not only accuracy, but also precision of the trackers. Wikipedia has a nice explainer on the distinction between accuracy and precision. For example, if you measure your height with a tape measure 10 different times, then the average value of the 10 measurements compared to your real height represents the accuracy. The variation in the 10 measurements represents the precision. It is possible to be accurate but not precise if the average is correct but the individual measurements significantly differ from one another. It is possible to be precise but not accurate if the individual measurements are nearly identical, but all are wrong.

In a typical "normal" distribution of measured values (e.g. test scores), the center of the distribution is the average (mean) and the width (spread) of the distribution is summarized by the standard deviation (SD). If the average value is correct, then the measurements are accurate; if the standard deviation is small, then the measurements are precise.

A new more comprehensive study of activity trackers was recently reported in the journal JAMA. The researchers addressed the issues of accuracy and precision in a more rigorous fashion. Even more so than the Consumer Reports study, the activity was well-defined and controlled. There were 14 participants who walked on a treadmill set at 3.0 mph for 500 and 1500 steps.

The study examined 10 devices, some were worn on the waist (clip-on), others on the wrist (wristband), and the third type were smartphone apps on a smartphone in the pants pocket (device name, manufacturer, and device type):

1. Digi-Walker SW-200 pedometer (Yamax) [Waist]
2. Zip (Fitbit) [Waist] 
3. One (Fitbit) [Waist]
4. Flex (Fitbit) [Wrist] 
5. UP24 (Jawbone) [Wrist]
6. Fuelband (Nike) [Wrist]
7. Fitbit (Fitbit) [iPhone 5s smartphone]
8. Health Mate (Withings) [iPhone 5s smartphone]
9. Moves (ProtoGeo Oy) [iPhone 5s smartphone]
10. Moves (ProtoGeo Oy) [Galaxy S4 smartphone]

Two trials were performed for each device at each step count for all the subjects resulting in 27-28 total measurements (14 X 2) per device.  The results for the 500-step trials are presented in Figure 1, and the results from the 1500-step trials are presented in Figure 2 (taken from the paper). The average (dot) and standard deviation (error bars) are shown where the vertical dotted line represents the correct value. The results were quite consistent between the 500 and 1500 step trials.

Interestingly, the clip-on trackers worn on the waist did better as a group than the others. From both an accuracy and precision standpoint, the clip-on trackers were superior to the smartphone apps which performed better than the wristbands.

That being said, all the devices did reasonably well consistent with the previous reports. The one possible exception was the Nike FuelBand which counted 10-20% fewer steps. Note that the FuelBand has been discontinued by Nike.

The Fitbit One and Fitbit Zip (both worn on the waist) were the most accurate and precise; every trial was almost completely correct. The third clip-on tracker, Digi-Walker, was almost as accurate but less precise than the Fitbits. It is also a significantly cheaper device.

It makes sense that trackers worn on the waist are more accurate and precise than those worn on the wrist or put in the pocket. The waist is not subjected to the extra motion of a limb; it can capture the acceleration and deceleration of one's gait without extra movements which add noise to the signal (i.e. walking pattern). Furthermore, one's arm especially at the wrist moves more than one's leg at the thigh (where a pocket is located), and so one would expect the smartphone app to be more accurate than the wristbands. Indeed I suspect if you put your Fitbit One (which had the best scores) in your pocket or wore it on your wrist, it would do much worse.

Performing a carefully controlled experiment in the laboratory also has limitations. One does not obtain information on how the trackers would do "in the wild" monitoring the random daily activity of a person going about her business. Indeed the authors of the study commented:

"Our study is limited by being conducted with young, healthy volunteers in a controlled setting with a convenience sample of a small number of applications and devices. Results should be confirmed in other settings and with other devices."

Nevertheless one can conclude from this careful study that activity trackers are not exactly the same. The clip-on trackers worn on the waist are more accurate and precise than the trackers worn on the wrist or placed in the pocket. In particular the Fitbit One and Fitbit Zip are excellent at counting steps in a relatively controlled setting.

Figure 1. Results from 500-step trial. Mean (dot) and standard deviation (error bar) for the 10 devices are shown. The vertical dotted line indicates 500 steps. Graph from Case et al.

Figure 2. Results from 1500-step trial. Mean (dot) and standard deviation (error bar) for the 10 devices are shown. The vertical dotted line indicates 1500 steps. Graph from Case et al.