by Kumar Singh, Research Director, Automation & Analytics, SAPinsider
The future of Analytics is in the cloud
If you follow the analytics space, you are cognizant of the fact that more and more vendors are moving their products to the cloud. Due to the flexibility, scalability, agility and the ease of building ecosystems that a cloud platform provides, in my opinion, it is inevitable that the future of analytics platform is in the cloud. The exponential rise in edge devices and the need for near real time data streaming also highlights the challenges that on-premise architectures will eventually run into. All the signs point to the cloud !
However, as businesses think about transitioning to the cloud, a key aspect that they need to think about is- what is the best in class aproach of building analytics solution in the cloud. There are various cloud platforms available these days and they provide the key specifications that you would expect from a leading cloud platform (examples are Amazon Web Services (AWS), Microsoft Azure, Google cloud). The key is therefore to understand how you can design the solution architecture in a way that it will deliver the value that you were expecting from transitioning to a cloud platform.
However, as organizations move from on-premises architecture to hybrid and pure cloud based architectures, they also need to understand what are some of the key components in their cloud based analytics solution architecture that they need to think about. In the subsequent section, we will discuss six key components of a best in class cloud solutions architecture. All leading cloud solutions providers share their own best practices documents and you will find these six areas as consistent theme in their documents. For this article, I will leverage AWS Analytics lens, a best in class architecture framework developed by AWS, and simplify it to walk through the architecture components.
The Six key components of a cloud based analytics solution
To simplify the components, let us follow the high level analytics process in the cloud. That process flow is not very different from any on-premise analytics solution process flow. Following are the six key components in the Analytics solution flow in the cloud:
- Data collection
- Data storage
- Data discovery
- Data processing and analytics
- User access
- Data security and governance
Now let us review these components in brief:
Data collection component: This deals with collecting data from multiple sources, with varying periodicity (like batches or near real time). Examples of sources range from website clicks, transactions and socia media streams to on-premise and cloud native data sources. The data is typically stored in a central data storage like a data lake. An AWS environment example of data collection component is Amazon Kinesis.
Data storage: Data collected is generally stored in a central data storage which then holds and maanges the data in order to make it available for downstream applications. This generally is a part of the data lake and supports data files of different formats and different structures. Elasticity and security are two additional key aspects of this layer. Amazon (S3) and Amazon Elastic Block Store are two examples of central storage in AWS ecosystems.
Data discovery : Imagine that you have built a central storage in the form of a data lake. Considering that you have data files being dumped there from multiple sources in multiple formats, it is imperative that you have an automated way of data discovery in your cloud architecture. AWS Analytics lens defines this as “the catalog and search layer” that pertains to discovering and cataloging the metadata pertaining to your data assets.
Data processing and analytics: From the perspective of analytics, this is where the real action happens. This layer enables the querying and processing of data that allows us to generate insights from the data stored and discovered. Each cloud service provider offers a gamut of products that can be leveraged in this layer. These products are often aligned to the type and format of data stored in the storage layer. Exaples for AWS environment are Amazon Athena, Amazon Redshift and Amazon Sagemaker.
User access: This is the front end of the architecture which the user will leverage to access the analytics solution. The critical aspects of this layer are ease of access, secure access and access management. Again, depending on which platform you are on, there are several options available to configure this layer. Some examples from AWS architecture are AWS Lambda and Amazon Cognito.
Data security and governance: One of the most important components/layers, this is the key towards successful adoption of cloud since data security is cited as the top apprehension executives have when citing bottlenecks towards transitioning to public cloud. This layer or component is responsible for ensuring that the data can only be accessed by users who are authorized to access the data. This layer essentially creates a secure environment for the entire architecture. AWS Identity and Access management and AWS Key Management services are widely used in AWS ecosystems.
What does this mean for SAPinsiders ?
- The future of analytics is in the cloud and SAP Analytics cloud is an example that SAP understands and sees that future. So if you are already thinking about SAP Analytics cloud but are apprehensive about cloud platforms, you should shed that apprehension
- Cloud based analytics solutions provide many advantages over on-premise solutions however, designing an optimal architecure is critical.
- Following a best in class approach suggested above will ensure a robust and resilient analytical solution
Kumar Singh, Research Director, Automation & Analytics, Supply Chain Management, SAPinsider, can be reached at email@example.com