This article is short and to the point and proposes an answer for one question; what is the next giant leap in analytics? I have argued and debated endlessly for the last 20 years on the necessity of data models. These models provide the structures for enforcing data integrity, and result in critical artifact byproducts. For one, a data model provides a visual representation of the data which helps the software architect design their applications. Second, the data dictionary artifact is essential for analysts and business personnel alike to better understand the business domain and its related data. Unfortunately, data modeling is expensive and time consuming, and it is frequently the process that either gets eliminated from the project all together, or rushed to the point that it provides little value. Basically, an overall lack of understanding in data management principles is the primary reason data silos exist, and why data scientists and analysts spend 80% of their time cleaning data. Just imagine the competitive edge a company would have if there was a process for automating data identification. In my opinion, this is where the next giant leap in data management will come, the automated identification and integration of disparate data.
They say a picture is worth a thousand words. Figure 1 says all I have to say on the subject. Developing this automation has been discussed in other articles, and there are more to come.This is not a simple problem to solve and will require a significant amount of engineering work. Vocabularies for each domain must be defined, and the relations between the words established. Training and testing data sets must then be developed for the machine learning algorithms that will eventually combine the words with the data. Once this occurs, analysts from many industries will spend their time doing what they should be doing, analyzing data. In fact, their time will probably be spent more on automating the analysis process that produce real-time prescriptive recommendations for leadership.
Leave a Reply
Your email is safe with us.