The world is galloping towards the heap of data and evidently, businesses are churning to analyse every known bit of it. From its known augment; Big Data today is pervasive and ineludible for businesses. Under‘Mining’ the potential future of this emerging technology in this digital transformation era, Ankur Pawa, Senior Vice President – Digital Services, Sasken Technologies adduces us to the world of Big Data. Edited Nub.
1. How is the Big Data evolving in 2018, the maturity of the market and the potential?
We can look at the evolution of Big Data from two dimensions:
a) Firstly, the data itself is changing or evolving. While the volumes of data continue to grow exponentially, new sources of data also keep emerging (estimates suggest that 90% of data ever created was created in the last two years). About 5-7 years ago, we saw a massive surge in data getting created through social network, web activity and mobile usage. Then we saw a data surge in the areas of quantified self (e.g. wearables), smart homes, etc. and now we are witnessing a massive acceleration in sensor-based data in the broader industrial and manufacturing sectors. The next wave, would potentially come from areas like autonomous cars.
b) Secondly, technology is evolving. Data warehousing has given way to technologies like Hadoop, Spark, Kafka, etc. Compute and memory have both become cheap which enables people to store more data and execute complex processing on it. While about four years back, we were seeing enterprises experiment with this stack and do proof of concepts, now they are making serious investments in data lakes based on these technologies and are putting them in production for enterprise wide usage. The benefit case of these technologies is two-fold: it lowers the cost of data storage and helps in things like ETL offload being executed a lot more economically (as compared to having data warehouses) and it also enables to run machine learning algorithms on top of all formats of data.
- Standing in 2018, has Big Data overcome the allegations of privacy and unregulated data mining? How can one secure its Big Data off or on-premise?
Data can be subject to malicious attacks, willful or inadvertent data leaks, or breaches. For malicious attacks, great technology exists but it is a continuous battle between the good and bad. There will be no end-point where we’d be able to say that now we are 100% safe and no one can enter our data vaults. The guards always have to be up and continuous innovation in security would be required to do that. In fact, this is one of the most interesting applications of Big Data where anomalies are detected based on the data collected from the various systems to flag-off a potential threat. Doug Catting, the co-creator of Hadoop has been helping Cloudera and Intel with the Apache Spot project, which is an open source, Big Data style of doing cyber security.
For data breaches, there are industry and consumer specific regulations in place which are increasingly getting stringent (e.g. EU’s General Data Protection Regulation (GDPR) which will get enforced in May of 2018). The responsibility lies on all of us as enterprises to comply with them and on us as individual consumers to be more cautious about sharing data only with trusted sources.
- How majorly has Big Data Market impacted the Analytics Market?
It has impacted in a very big way. Big Data has expanded the possibilities for the type and complexity of analytics that can be done. It helps companies run analytics on disparate sources and also enables streaming analytics on real-time data. Big data is also typically the foundational data layer in Artificial Intelligence systems and applications. Areas like fraud detection, chat bots, autonomous cars, and recommendation engines are examples of this profound impact.
- Which sectors in India are early adopters and which markets and geographies is the market set to mature evidently?
Banking, e-Commerce, and Retail sectors seem to be leading the adoption curve. We are now seeing a lot of traction in the Manufacturing/Industrial sector as they are actively pursuing the opportunities arising from machine data. North America and Europe are fairly matured markets already and will continue to lead and drive innovation in this space.
- Lastly, how is your company determining or say inking the future of Big Data in terms of technology, market adaption and revenue?
We are investing on both fronts of data – Big Data and Machine Learning/Analytics. The key to deriving value from Big Data and analytics is to keep the business problem at the center of a solution. For example, we do come across instances where data lakes are created first and then we start thinking what problems we can solve with the data we have stored. This is not the recommended approach. Instead, it is important to identify the most pressing problems in the business, then source the right data, and subsequently create an architecture that is suitable for that. This ensures that we solve the problems effectively, deliver the expected RoI and eliminate any wasteful spend in terms of effort and infrastructure. This is the philosophy that we work with and our customers see a lot of value in it.
In terms of specific areas and solutions, we have built our own reference Big Data architectures for different industries that have varying needs and requirements. We have also invested heavily in the area of AI and ML and have built several solutions. For example, we have created an Advanced Image Recognition solution which helps in drowsiness detection of a driver, fire detection in public places, etc. Another interesting example is a solution we have built in the areas of Asset Performance Management and Digital twins which helps us predict the health of machines and prescribes maintenance activity so that the remaining useful life of the machines can be increased while mitigating any risk of accidents.