Reposted from my G+ page once I realized it belonged here. And, my apologies for the lack of posts to this blog over the last few years. Life has been keeping me busy!
I awoke this morning with a simple half dream where a list of numbers was shown to me, representing an aggregate score. I immediately recognized upon waking that there was a problem represented in this dream and my mind ran down through several scenarios where data aggregation has been misused by accident, and abused by design. But it's not all bad.
First, what's the problem with data aggregation? You might easily see that it obscures the details. While that is the point, the obscuring of supporting details can hide valuable and meaningful information. I have a project under development presently that provides some good examples of this: time entry and reporting.
Let's say I have two employees who have both logged 40 hours this week. Just looking at that aggregate number, 40, I can't tell much about how that time was spent. Did one employee come in late every day but then stay late to make it up? Did the other work late to meet a deadline Wednesday and then knock off early on Friday? Do these things matter? They might matter, they might not, but the point is simply that their meaning is lost through the process of aggregation.
Let's look at something that impacts nearly every American, your credit score. How is this score derived? What rules and measures are used? If I rely upon your credit score as a measure of your credit worthiness, is that a good thing? The abstraction of details into an aggregate score means I don't know a great deal about you. The credit score provides a fairly anonymized way of presenting you as a number that I can infer my own value judgments upon. But is this a good thing? Are you comfortable with that? You may say yes when your score is good, and no when it is bad. Whether it is just or not, it is computed by rules set by someone, perhaps kept secret, and rightly or wrongly provided as a service to those who need to get to know you quickly to decide whether to do business with you. It's the cost of doing business if your business includes taking out loans establishing credit cards. But it has also been used to screen renters, students and customers to decide if there is some bias free way to weed out bad customers. Here, the judgment of the aggregate data has a real impact on human life, just or not.
So as a merit, aggregation provides an abridgment of tedium. In doing so, as with the case of the credit score, it buys for us expediency in commerce, but it comes at the cost of clarity which is an imperfect, impersonal representation of data lacking in deep understanding of our circumstances and recent efforts to perhaps repair damaged credit. We can't tell from a timesheet how hard someone worked and we can't tell from a budget how well it has been followed. We can't tell from a savings account balance how frugal or unlucky someone is.
Aggregation, to be sure, is a shortcut with real value, but it slips neatly into a category of summary knowledge that enables lazy thinking. It provides a seemingly scientific footing for ill founded assumptions that lack a cognisance of the detailed algorithm used to arrive at a particular aggregate number. A wholly ignorant man can speak with authority and mastery he does not possess, and nevertheless be correct in the speaking of the number itself, and yet completely wrong in what merits or demerits he attributes to or derives from said number... and horrifically, an uninformed and dispassionate audience would be none-the-wiser. And therein lies the danger of aggregation. In seeking to provide a good by publishing such information, we enable evil by the looming masses of idle minds, desperate for some measure of recognition of their imagined intelligence.
So, use great caution in both the creation, presentation and consumption of statistics and data presented to tell a narrative. As we tell our children when they go to put something in their mouth they found on the floor: "you don't know where that's been", or the implicit reason for not accepting candy from a stranger: "you don't know where it's from or what it contains".
