Big Data is the Big Thing at the moment.
It won’t be ten years from now. And not because we tackled the technical aspects of handling it, or created huge business intelligence systems capable of harvesting the treasures hidden in it. Or because it has become generally accepted and arrived at Gartner’s plateau of productivity.
It won’t be, because it is not scalable. Because the related privacy problems are insurmountable. Because its worth is directly related to the size of the harvesting company and in no way related to where it has been derived from and based on: the customers or individuals.
However, what worries me most, is that so few people voice fundamental concerns against this development. Some do. Just the other day (related to hospital data) there was a really insightful discussion on LinkedIn, unfortunately in Dutch.
Big Data is a monster, a bloated and sick by-product of technology and marketing and a distorted respect for the autonomy of big corporations. To be clear: in this article I will mainly focus on big data related to persons. I am not primarily concerned with big data derived from, say, particle generators or NASA telescopes.
Well. Not a very popular view I guess. And too highly political for my taste and (I guess) many others. Nevertheless.
Thing is, there seems to me a simple and acceptable solution for this issue.
I remember Stanley Kubrick who said that people are the owners of their own image. In the same vein: people should be the owners of their own “data”. Corporations should be prevented by law to collect data on people.
We have seen the problems related to medical data in The Netherlands, where an ambitious project, EPD (Electronic Patient Data) effectively failed because of the insurmountable issues around privacy and security.
What would be the problem if we “inverted” the problem? That is: why not store the data related to people and individuals with the people themselves? Why not effectively prohibit corporations to collect and store data about persons? What would be the issues resulting from implementing this strategy? And could those issues be solved without casualties on both sides?
I think it is possible. Even more: I think this is the only possible and scalable solution.
Let’s say you are Marie-Louise (if you are her, you know why I use your name ). You walk through a street with shops. For weeks you are thinking about buying those brown boots, but you are acutely aware of the precipitous state of your mortgage. So you have postponed buying those boots for a while.
The shops in the street are “aware” of your passing by. Being prohibited from “knowing” Marie-Louise’s purchase history, they nevertheless would be very interested in selling brown boots to her at a reduced price, if they would be able to close the sale. So how could the shop send a message to Marie-Louise that those boots, exactly the ones she is interested in (because of her recent search history on the web) are now available at a 20% discount in the shop 5 meters from her?
Simple. Marie-Louise herself knows about her interests. And her software alter ego, which she carries around with her all the time in the form of what I will call for the time being a “gem”:
This gem, with a data storage capacity of 500 TB, contains everything Marie-Louise has indicated as something she wants to “remember”, that is: her software alter ego. The shops she passes are allowed to query her gem, but not to store the results of those queries for longer than one hour. The store is informed that a potential customer is passing by, and the message is sent.
Even better: because the gem is aware of the priorities of Marie-Louise, it will inform the shop that Marie-Louise is interested in those boots, but only if the shop is willing to drop the price with 25% or more. That way Marie-Louises mortgage payments will not be affected. So Marie-Louise can safely read those messages which would otherwise be pesky irritations, because she knows the gem will protect her from those that are unwelcome. The fact that the shop has queried the gem could even be stored in the gem. I see no reason why corporations should have the same civil rights as a person.
Scalability solved. Privacy solved. Commercial interests protected.
Let’s do this.
Abonneer op de nieuwsbrief
Executive management doesn’t have a clue.
Everybody I talk to in organisations large and small agree. And I am talking managers here.
They try. They do their best. They start doubting their intelligence. They install control mechanisms. They install more control mechanisms. Heck, they might even try (and burn their fingers on) business intelligence. Maybe that will give them the edge they may think they lack.
Thing is: it’s unavoidable. It is humanly impossible to have a clue.
Let’s try to be clear on what I am talking about here. A clue about what? And is that a problem?
Abonneer op de nieuwsbrief
Society is undergoing fundamental changes.
This is not unique for our time and age, but both speed and depth of these changes are probably greater than we have seen in centuries.
Enterprises are, whether they want it or not, both party and subject in these changes.
The way businesses were organised in the past does not scale into this age, and certainly not in the “future” age. Wrestling with larger concerns, in numbers of customers (for example in the case of producers of software), in chained businesses (for example in the energy sector), or in the number of anonymous shareholders and the incomprehensible speed of the trade in those shares, these companies are looking for their new identities to survive, let alone prosper.
Enterprise architecture (hereafter shortened to EA) is both prospering and struggling. It is prospering for a select group of practitioners and enterprises who have seen the advantages that can be derived from it. We can see this reflected in the vastly growing interest in EA frameworks like TOGAF or Dragon1 or several others. It is struggling for the majority of enterprises who have maybe attempted one or more EA implementations that failed horribly by consuming resources of time and money without returning any (visible) value.
There is (still) a dire shortage of experienced and effective enterprise architects. The use of the frameworks mentioned above is mostly still in its infancy, and most enterprise architects I know working in these disciplines are struggling within the isolation of there own organizations, not communicating and interacting with their peers.
In spite of these considerations I must still defend the importance and role of EA. Well, I am an enterprise architect so that might not surprise you. But my field of expertise is sufficiently broad that I think I can claim the seniority to view the issue relatively unhindered by personal bias in this.
The reason I stress the importance of enterprise architecture this way is that I think that within the unfolding complexity we have no alternatives. EA offers the only way to condense knowledge and wisdom, and managing tools, to deal with this. We still haven’t seen enough tooling that is able to deal with a sufficient level of realism with things like stochastics, uncertainty, change, waiting lines and complexity (in its scientific definition). But these tools will hopefully mature soon. The frameworks as well still focus mainly on helping architects to get a grip on the structure of enterprises. What is needed to become more effective is tools to deal with the dynamics of enterprises, the way they change and visualising these changes with a set of robust simulations and what-if scenarios. Something that needs to originate from EA.
With dynamics I do not mean the processes, because I categorise these under the structure of enterprises. I am talking about changing those processes, about playing with them, and keeling their orientation towards demand-chained and non-deterministic processes.
With an EA empowered by those tools we can move forward and discover what enterprises will accomplish then.
Abonneer op de nieuwsbrief
Again I was struck by the tendency of many “experts” to use too much. They have a bag-of-tricks which for some reason needs to be used exhaustively. And the tools and frameworks do mention (sometimes) that you need to tailor them, but that doesn’t stick. If it’s there, it’s there for a reason, right? Let’s use it.
Frameworks are, by definition, a superset of what you need. Or did you imagine everyone needs the same things? The fact that you know all that doesn’t mean you should use all of it. Especially architects fall for this temptation. Or is it an ego-boost? Look at how big my gun is… You have learned TOGAF, so you want to apply it. Fine. But didn’t your mum tell you that temperance is virtue?
I do not recognise the expert by the size of his toolbox. I recognise him by the temperance in applying his tools. By the frugality of his actions, the close proximity of the actual need that needs to be met with his solution. And by the smallness of his steps forward.
Wouldn’t it be brilliant if the TOGAF certification or any other did not only check your knowledge of the framework, but also (and maybe even foremost) your skills in omitting parts?
Abonneer op de nieuwsbrief
It is a psychological block that keeps an artificial divide alive that never really existed in the first place.
And the block is kept in place by exactly those people that *think* they grok information technology. Myself included I fear.
It is such a nice, alluring thing, technology. Like religion in the old days it has this promise of change for the better, of a solution to so many if not all of our problems. It will save us loads of time, it will provide us with innumerable friends, it is the path to world peace.
The problem, of course, it that *they* do no grok it (isn’t it revealing that this word, grok, is from a science fiction novel by Robert Heinlein where it is introduced by someone from Mars?). They being, of course, the business.
This attitude is very interesting, psychology-wise. In the first place there is this belief in solutions. Which basically boils down to a believe in the problem.
Believing in a problem, making it so prominent and important, is not much different from investing in it, from depending on it. Your own identity, your own existence, is tied to the existence of the problem.
There is this psychological law that states that “what you fight against, you make stronger”. This law condenses the insight that often, if not always, you build up a dependency on the problem, on the enemy. You have, in fact, a vested interest in the enemy being there, and being strong, and unconsciously you help in making that true. It takes a truly brave effort to accept this insight, to “let go” of the enemy.
The business is the enemy. Not only do they not understand technology and the brave new world it brings along: they do not even understand business. We, the technologists, understand business better than they do. We have all these tools to model the business, we have invested years in understanding the business and removing vagueness and ambiguousness from it because we thought we needed that to build our technological solutions, that we came to believe we have a much more robust understanding of the business than “they” have.
How often have I not heard this said by the IT people! (you have too, don’t you deny it!)
Fact is, in a way this is true. Business doesn’t understand business. But believe me: IT understands it even less! Business is an inherently complex system, inter-tied with other systems. To make an epistemological model of, let alone a deterministic model of it which is required by most IT systems I know, is impossible. It even defies the purpose, because what is the result? (if it were successful which thank God it never is…): a deterministic business.
Well I can tell you: a deterministic business is a business already dead. The life is out of it. The life that is intertwined with its inherent complexity, its being a living, moving, willful and chaotic beast, is killed by the machine.
What IT needs to understand that it is not a thing in itself. I encountered three things that triggered me into writing this blog.
The first was an article by a good friend of mine, Daan Kalmeijer, who recently wrote a column (in Dutch) in a Dutch IT magazine (Mijn vaders tijd, article in Dutch, only for subscribers IAA) about this idea that our models should be “technology-agnostic”. This is in fact a strategy I have strongly advocated for years (called Business Centric Architecture by me — the link is to a series of Dutch articles by me on the subject — no English translation yet I fear, but many other articles on this site breathe the same philosophy). I won’t go into that aspect of his column, although I am aching to do so (some other time), but just want to mention that his gut feeling about the dissipation of technology throughout society, and thus, business, is heads-on.
The second was a challenging presentation by another friend, Eric Lopes Cardozo on the ASAS 2013 conference in his keynote speech, which was called “There is no architecture, only business”. Need I say more. His Dutch article can be found here, the slides (in English) are here.
The third was a discussion at the GIA (a Dutch professional group for Information Architects) meeting yesterday evening with the subject “Enterprise Architecture methods: an overview. During the discussion familiar recurring concerns were voiced: “what is architecture actually?”, “what can we do to be taken more seriously by “the business”", inevitably leading to an existential crisis among the participants. It struck me how recurring this pattern is on all meetings of architects (be it enterprise, solution, IT) that I have been to. Hmm, maybe I am a factor in that statistic?
We should, and can, stop re-enforcing the divide, which only exists in our minds. Technology is part of the Zeitgeist. I am certainly not saying it is irrelevant. But there is no divide, there is no impedance mismatch or incompatibility. There is only change. And change, dear readers, is a frightening thing. Especially for technologists…
Abonneer op de nieuwsbrief