I didnt read the report, but is this measurement done on a a/b testing way? There is usually some data skew/bias when they are done on different time scale.
Confounding factors. For example, crashes can be more likely after sunset or during inclement weather so seasonality can play a role. Or maybe the drivers who had Tesla longer got used to it.
I suspect OTA update was applied simultaneously rather than staggered over the course of the year, so probably not. You can control for seasonality using some other data as a proxy, but the more adjustments you make, the more potential there is for introducing errors. Randomized test/control split is almost always better than before/after comparison.
this particular one, maybe, but there might be others. like the user base pre-ota and post ota (number-wise) could be different.
I think there is a good SaaS business with A/B testing robot/car software OTA.
I would imagine some metrics would be:
1. car speed
2. battery consumption per mile.
3. user comfort metrics (if it's possible to measure, maybe like hearthrate, body temperature etc - i can imagine if robots were driving bad, it'd be visible in heart rate or something like that).4
The list probably goes on. It would be interesting to see how much battery consumption changed after OTA, for example.
