DataStudio Deployment Guide

4. Data Ingestion

DeepIQ supports the ingestion of data from the following sources. 

• Time series: WITSML 1.4, WTIMSL 1.3, OSI PI, Aspen Tech IP21, Historians/SCADA/PLC systems that support OPC HDA/OPC UA/MQTT (Examples include Honeywell PhD, GE Proficy, Cygnet) 

• Geospatial Sources: Different types of Map formats and satellite imagery 

In addition, DeepIQ supports read/write access to the following sources: 

• Cloud Databases: AWS Databases (Redshift/Timestream), Snowflake, Delta Lake, 

• File Systems: S3 

• Streaming Sources: AWS Kinesis, Confluent Kafka, Azure Event Hub 

• On-Premise Databases: All data sources that support JDBC/ODBC, Hive, HBase, MongoDB 

5. Transformations 

DeepIQ supports 250+ inbuilt transformations for extensive scale engineering of time-series, geospatial, and structured data sources. 

6. Machine Learning 

DeepIQ provides a no-code app to build machine learning pipelines spanning supervised/unsupervised/deep learning use cases.  It also offers end-to-end machine learning features.  Special-purpose pipeline features are provided to address challenges in time series and geospatial data sources. 

DeepIQ has patent-pending algorithms that allow users to build knowledge models and use AI to optimize their performance. 

The current DeepIQ customer base spans Upstream, Downstream, Midstream, Renewable, Financial, Mining, and Agricultural verticals.  With DeepIQ, these Customers have built solutions for several high-value use cases, including: 

• Equipment Predictive Health 

• Production and Drilling Optimization 

• Real-Time Operations Monitoring Mineral Prospectivity Analysis 

• IT-OT convergence reports 

• Quality Monitoring 
• Carbon Emissions Monitoring

7. Deployment Options 

The introductory material describes all deployment options discussed in the user guide, including single-AZ, multi-AZ, and multi-region deployments.  These options provide flexibility and scalability to meet the diverse needs of different organizations. 

Managed Service: This option involves installing the application on the customer’s infrastructure.  It allows organizations to maintain control over their environment while leveraging the capabilities of DataStudio.  The managed service deployment ensures that the application is tailored to the specific requirements and configurations of the customer’s existing infrastructure. 

• SaaS Offering: The publisher hosts the applications on a dedicated infrastructure in this option.  This Software-as-a-Service (SaaS) model provides a hassle-free deployment experience as the publisher manages the infrastructure, maintenance, and updates.  Organizations can benefit from the scalability and reliability of the publisher’s infrastructure while focusing on their core business operations. 

• Single-AZ Deployment: This deployment option involves setting up the application within a single Availability Zone (AZ).  It suits organizations with lower availability requirements and can be a cost-effective solution for smaller-scale deployments. 

• Multi-AZ Deployment: This option involves deploying the application across multiple Availability Zones within a single region.  It enhances the availability and fault tolerance of the application by distributing resources across different AZs. Multi-AZ deployment is ideal for organizations that require higher availability and resilience. 

• Multi-Region Deployment: This deployment option involves setting up the application across multiple regions.  It provides the highest availability and disaster recovery capabilities by distributing resources across different geographic locations.  Multi-region deployment is suitable for organizations with global operations and stringent availability requirements. 

By offering these deployment options, DataStudio ensures that organizations can choose the most suitable configuration to meet their needs and operational requirements. 

8. Deployment Timing 

The expected time to complete the deployment is no more than 3 hours.  This timeframe includes all necessary steps to ensure smooth and efficient deployment.  The deployment process is designed to be straightforward and user-friendly, minimizing downtime and ensuring the application is up and running as quickly as possible. 

The deployment process involves several key stages: 

1. Preparation: This stage includes gathering all necessary resources and prerequisites, such as the deployment kit, IAM user credentials, and access keys.  It also involves setting up the Ubuntu VM and ensuring all required tooling is installed and configured correctly. 
2. Infrastructure Provisioning: During this stage, the necessary AWS resources are provisioned, including Databricks on AWS, EFS, EKS, IAM User, S3 Bucket, Secret Manager, VPC, Subnet, Virtual Machines, and ECR.  This stage ensures that the infrastructure is ready to support the deployment of DataStudio. 
3. Product Installation: This stage involves running the installation scripts provided by DeepIQ.  The scripts automate the installation process, configuring the application and setting up all required components.  The installation process is designed to be efficient and error-free, with built-in checks and validations to ensure successful deployment. 
4. Validation and Testing: After the installation is complete, the deployment process includes validation and testing to ensure the application functions correctly.  This stage involves running health checks and verifying the status of all components.  Any issues identified during this stage are addressed promptly to ensure smooth deployment. 

By following these stages, the deployment process is streamlined and efficient, ensuring that the application is deployed within the expected timeframe of 3 hours. 

9. Supported Regions 

All the resources needed for DataStudio can be deployed in any region, so all regions available on AWS are supported.  This flexibility ensures that organizations can choose the most suitable region based on their specific requirements, such as proximity to their user base, compliance with regional regulations, and availability of AWS services.  By supporting all AWS regions, DataStudio allows organizations to optimize their deployment for performance, cost, and redundancy. 

When selecting a region for deployment, organizations should consider factors such as: 

1. Latency: Choosing a region geographically closer to most users can reduce latency and improve the overall user experience. 
2. Compliance: Some regions may have specific regulatory requirements that organizations must comply with.  Deploying in a region meeting these requirements can help ensure local laws and regulations compliance. 
3. Service Availability: Different AWS regions may offer varying service availability and features.  Organizations should select a region that provides the AWS services and features required for deployment. 
4. Cost: AWS pricing can vary between regions.  Organizations should consider the cost implications of deploying in different regions and choose a region that aligns with their budget and cost optimization goals. 
5. Disaster Recovery: For organizations with stringent availability and disaster recovery requirements, deploying across multiple regions can provide additional redundancy and resilience.  This approach ensures that the application remains operational even during a regional outage. 

By supporting all AWS regions, DataStudio ensures that organizations can deploy their applications in the most suitable region to meet their specific needs and operational requirements. 

10. Prerequisites & Requirements 

10.1. Deployment Resources 

We use an Ubuntu VM to provision the entire infrastructure and install the product.  A script will install the necessary prerequisites and prepare the VM for product installation.  The publisher provides a deployment kit at the time of installation. 

When the deployment is complete, the following resources are set up on AWS: 

• Databricks on AWS: This service is used for big data processing and analytics.  It provides a unified analytics platform for data engineering, data science, and machine learning. 

• EFS (Elastic File System): EFS provides scalable file storage with AWS Cloud services and on-premises resources.  It is designed to be highly available and durable. 

• EKS (Elastic Kubernetes Service): EKS is a managed Kubernetes service that makes it easy to run Kubernetes on AWS without installing and operating your own Kubernetes control plane or nodes. 

• IAM User: An IAM user is created with the necessary permissions to manage the resources required for the deployment. 

• S3 Bucket: S3 is used for object storage, providing a scalable and secure location to store data. 

• Secret Manager: AWS Secrets Manager helps protect access to applications, services, and IT resources without the upfront cost and complexity of managing your hardware security module (HSM) infrastructure. 

• VPC (Virtual Private Cloud): A VPC is created to provide a logically isolated section of the AWS Cloud where you can launch AWS resources in a defined virtual network. 

• Subnet: Subnets are created within the VPC to segment the network and control traffic flow. 

• Virtual Machines: Virtual machines are provisioned to run the various components of the DataStudio application. 

• ECR (Elastic Container Registry): ECR is a fully managed Docker container registry that makes storing, managing, and deploying Docker container images easy. 

These resources are essential for the deployment and operation of DataStudio, ensuring that the application is set up in a secure, scalable, and efficient manner. 

10.1.1. Technical Prerequisites and Requirements 

Several technical prerequisites and requirements must be met to ensure a smooth and successful deployment of DataStudio.  These prerequisites cover permissions, software, skills, and environment configuration. 

Permissions and IAM Users: 

IAM users should have permission to allocate the resources required by deployment.  DeepIQ’s scripts provided as part of the AWS marketplace will initiate and manage deploying all the necessary resources. 

Consumers should be permitted to create an IAM role with authorization to provide the abovementioned resources. 

AWS CLI: 

The AWS Command Line Interface (CLI) must be installed and configured on the machine used for deployment. 

Operating System and Software: 

The deployment requires an Ubuntu 20.04 or above machine with at least 8GB of free storage. 

Skills and Specialized Knowledge: 

The user installing DataStudio needs to be familiar with AWS, specifically creating resources in AWS and adding the required permissions to the user’s IAM profile. 

The user installing DataStudio needs to have an administrator account or an account with sufficient privileges and should be familiar with setting up a service account with enough privileges to create the resources required by DataStudio. 

Environment Configuration: 

Customers must manage the domain on which the product should be hosted through Route 53 to finish the installation successfully. 

The installation script provided by the publisher adds an “A” record to the Route53 zone supplied at the installation time. 

Organizations can facilitate a smooth and efficient deployment process for DataStudio by ensuring that these technical prerequisites and requirements are met. 

10.1.2. OS and Software 

Ubuntu 20.04 or above machine with at least 8GB of free storage. 

10.2. Skills and Specialized Knowledge 

Several skills and specialized knowledge are required to ensure smooth and successful deployment of DataStudio.  These prerequisites cover familiarity with AWS, permissions, and account management. 

AWS Familiarity: 

• The user installing DataStudio needs to be familiar with AWS, specifically creating resources in AWS and adding the required permissions to the user’s IAM profile.  This includes navigating the AWS Management Console, using the AWS CLI, and managing AWS services such as EC2, S3, and IAM. 

Administrator Account: 

• The user installing DataStudio needs to have an administrator account or an account with sufficient privileges.  This account should have the necessary permissions to create and manage AWS resources, including IAM roles, policies, and access keys.  The user should be familiar with setting up a service account with enough privileges to create the resources required by DataStudio. 

Service Account Setup: 

• The user should be familiar with setting up a service account with enough privileges to create the resources required by DataStudio. This involves creating IAM roles and policies, configuring access keys, and ensuring the service account has the necessary permissions to manage the deployment process. 

Technical Expertise: 

• The user should have technical expertise in managing and configuring cloud infrastructure.  This includes understanding network configurations, security groups, VPCs, subnets, and other AWS resources.  The user should be able to troubleshoot and resolve issues related to the deployment process. 

Knowledge of Deployment Scripts: 

• The user should be familiar with running deployment scripts and configuring YAML files.  This includes understanding how to use the deployment kit provided by DeepIQ, configuring the necessary secrets and access keys, and running the installation scripts to deploy DataStudio. 

10.3. Environment Configuration 

To ensure the successful deployment of DataStudio, it is essential to configure the environment properly.  This includes managing the domain, setting up the necessary network configurations, and ensuring all required components are in place. 

Domain Management: 

• Customers must manage the domain on which the product should be hosted through Route 53 to finish the installation successfully.  This involves creating and configuring the necessary DNS records to point to the appropriate resources. 

• The installation script provided by the publisher adds an “A” record to the Route 53 zone supplied at the installation time.  This ensures the domain is correctly configured to route traffic to the DataStudio application. 

Network Configuration: 

• VPC (Virtual Private Cloud): An isolated VPC will be created without connection with existing VPCs on the customer end.  The VPC would have two subnets: Public and Private. 

• Subnets: Subnets are created within the VPC to segment the network and control traffic flow.  All resources are spawned on the private subnet, while the Application Load Balancer is made in the public subnet to accept traffic coming over the Internet. 

• Security Groups: Security groups are configured to control inbound and outbound traffic to the resources within the VPC.  This ensures that only authorized traffic is allowed to access the DataStudio application. 

• NACLs (Network Access Control Lists): NACLs provide an additional layer of security by controlling traffic at the subnet level.  This helps to protect the resources within the VPC from unauthorized access. 

• Route Tables: Route tables are configured to define the routes for traffic within the VPC.  This ensures traffic is correctly routed between the subnets and the Internet. 

Instance Metadata Service Version 1 (IMDSv1): 

• All components of the solution hosted in the customer’s account support the ability for the customer to disable Instance Metadata Service Version 1 (IMDSv1).  If your product calls to AWS APIs, you should ensure you are using the latest version of the AWS SDK.  Current SDK versions will use IMDSv2 automatically to ensure your product meets this requirement. 

11. Architecture Diagrams 

11.1. Logical and Deployment Diagram 

The following diagram illustrates a standard deployment of DeepIQ’s DataStudio on AWS.  DeepIQ DataStudio can connect directly to data sources like relational databases in the corporate customer network by leveraging AWS-Corporate Network connectivity.  Networks with Industrial Control Systems may be in secure zones where Pull requests from external sources are not allowed.  DeepIQ provides edge software for these networks that uses a Push Protocol to send data to the customer account on AWS. 

11.2. Network Diagram 

The following diagram illustrates the standard network architecture diagram for DeepIQ DataStudio installed in a customer tenant. 

12. Security 

12.1. Resources 

• DataStudio does not require root or Administrator privileges.   DeepIQ adheres to the AWS policies and procedures outlined here: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_root-user.html.     

• DeepIQ adheres to the AWS policy of least privilege for all access granted as part of the deployment, as outlined here.  https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html   

• The deployment process involves creating an IAM user with particular roles.  The roles limit access to only creating resources we require and ensure that our tooling does not use more access than it needs. 

• The DataStudio deployment process does not include any public resources 

12.2. IAM Roles and Policies 

The IAM roles and policies necessary to provision DataStudio are mentioned below. 

• Full access to EKS is needed to create and manage an EKS cluster. 

• Full access to S3 to create S3 buckets and manage related permissions. 

• Full access to routes is required to search if the domain the client has provided is present in their route; also, once the domain is found, we need to add A records for installation. 

DeepIQ adheres to the policies outlined here: https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html.  

12.3. Sensitive Information 

All the keys a customer creates should be stored in the AWS secret manager, which is used during deployment.  The following keys are used at the time of deployment. 

• IAM user secret Keys 

• Databricks Personal Access Token 

• S3 bucket keys 

DeepIQ adheres to the policies outlined here: https://docs.aws.amazon.com/secretsmanager/latest/userguide/best-practices.html.  

12.4. Data Encryption 

DataStudio will be leveraging Server-side encryption.  AWS S3 automatically encrypts data before storing it and decrypts it when retrieved.  This ensures that all data stored in S3 is protected and secure. 

For deployments involving more than a single element, it is essential to include network configuration to ensure secure communication between components.  The necessary network configurations include: 

• VPC (Virtual Private Cloud): An isolated VPC will be created without connection with existing VPCs on the customer end.  The VPC would have two subnets: Public and Private. 

• Subnets: Subnets are created within the VPC to segment the network and control traffic flow.  All resources are spawned on the private subnet, while the Application Load Balancer is made in the public subnet to accept traffic coming over the Internet. 

• Security Groups: Security groups are configured to control inbound and outbound traffic to the resources within the VPC.  This ensures that only authorized traffic is allowed to access the DataStudio application. 

• NACLs (Network Access Control Lists): NACLs provide an additional layer of security by controlling traffic at the subnet level.  This helps to protect the resources within the VPC from unauthorized access. 

• Route Tables: Route tables are configured to define the routes for traffic within the VPC.  This ensures traffic is correctly routed between the subnets and the Internet. 

DeepIQ adheres to the data encryption policies as outlined here: https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html.  

12.5. Network Configuration 

To ensure secure communication between components, it is essential to include network configuration in the deployment guide.  The necessary network configurations include: 

• VPC (Virtual Private Cloud): An isolated VPC will be created without connection with existing VPCs on the customer end.  The VPC would have two subnets: Public and Private. 

• Subnets: Subnets are created within the VPC to segment the network and control traffic flow.  All resources are spawned on the private subnet, while the Application Load Balancer is made in the public subnet to accept traffic coming over the Internet. 

• Security Groups: Security groups are configured to control inbound and outbound traffic to the resources within the VPC.  This ensures that only authorized traffic is allowed to access the DataStudio application. 

• NACLs (Network Access Control Lists): NACLs provide an additional layer of security by controlling traffic at the subnet level.  This helps to protect the resources within the VPC from unauthorized access. 

• Route Tables: Route tables are configured to define the routes for traffic within the VPC.  This ensures that traffic is correctly routed between the subnets and the Internet. 

DeepIQ follows the AWS practices outlined here:

https://docs.aws.amazon.com/AmazonS3/latest/userguide/security-best-practices.html  

https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html  

https://docs.aws.amazon.com/vpc/latest/userguide/configure-subnets.html  

https://docs.aws.amazon.com/vpc/latest/userguide/VPC_SecurityGroups.html  

https://docs.aws.amazon.com/vpc/latest/userguide/vpc-network-acls.html  

https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Route_Tables.html  

12.6. Instance Metadata Service Version 1 (IMDSv1) 

All components of the solution hosted in the customer’s account support the ability for the customer to disable Instance Metadata Service Version 1 (IMDSv1).  If your product calls to AWS APIs, you should ensure you are using the latest version of the AWS SDK.  Current SDK versions will use IMDSv2 automatically to ensure your product meets this requirement. 

DeepIQ adheres to the AWS policy:  https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-options.html.  

12.7. Customer Data 

DeepIQ does not store sensitive customer data.  However, customer data is managed and stored in various locations depending on the type of data and its usage within the DataStudio application. 

• S3 Buckets: Customer data is stored in Amazon S3 buckets, which provide scalable and secure object storage.  This includes data ingested from various sources and data generated by the application. 

• Relational Databases: Some state information is stored in relational databases deployed by the tool.  These databases are periodically backed up and stored in S3. 

• Secrets Manager: All keys created by the customer should be stored in the AWS Secrets Manager, which is used during deployment.  The following keys are used at the time of deployment:  

• IAM user secret keys 

• Databricks Personal Access Token 

• S3 bucket keys 2. 

DeepIQ adheres to the AWS policy on sensitive data if required:  https://docs.aws.amazon.com/prescriptive-guidance/latest/defining-bucket-names-data-lakes/handling-sensitive-data.html.  

13. Costs 

13.1. Billable Services 

Billable services and guidance on whether each service is mandatory/optional. 

DataStudio utilizes the following billable services 

• Databricks on AWS – billing depends on cluster usage 

• EKS – resources on EKS are used for running the DataStudio application 

• S3 – S3 is used as the primary data store for ETL 

• VPC 

• Virtual Machines 

• ECR 

• Privatelink (optional) – these are optional and can be added on by the user through DataStudio UI after the core application is installed 

13.2. Cost and Licensing models 

Customers have the flexibility to choose from various features and contract lengths available on our AWS Marketplace page  https://aws.amazon.com/marketplace/pp/prodview-pw3vfhglc3nq2  

To purchase custom installations, or for more information or assistance, please contact us at info@deepiq.com. 

*Details are in DeepIQ‘s Software License Agreement