AWSTemplateFormatVersion: 2010-09-09 Description: 'Deploy a DataClarity cluster' Metadata: AWS::CloudFormation::Interface: ParameterGroups: - Label: default: AWS Instance Parameters: - InstanceType - AssociatePublicIpAddress - KeyPair - VPC - Subnet - RAM - DiskSize - AMI - SecurityGroup - Label: default: DataClarity Application Parameters: - TLS - AuthenticationMasterPassword ParameterLabels: InstanceType: default: "The size of the Virtual Machine" SecurityGroup: default: "AWS Security Group" KeyPair: default: "AWS Key Pair Name" VPC: default: "AWS VPC Id" Subnet: default: "AWS Subnet Id" AssociatePublicIpAddress: default: "Associate Public IP Address" AuthenticationMasterPassword: default: "Password for the User Access Master Console" Parameters: InstanceType: Type: String Description: AWS EC2 instance specs AllowedValues: - 4-vCPUs-16-GB-RAM-150-GB-SSD - 4-vCPUs-32-GB-RAM-150-GB-SSD - 8-vCPUs-32-GB-RAM-300-GB-SSD - 8-vCPUs-64-GB-RAM-300-GB-SSD Default: 8-vCPUs-32-GB-RAM-300-GB-SSD AssociatePublicIpAddress: Type: String Default: 'Yes' AllowedValues: - 'Yes' - 'No' Description: Associate a public IP for the instance. Disable the public IP only if you are using an AWS VPN. If you are unsure, leave this value as 'Yes' or consult your network administrator TLS: Description: Disable or enable TLS using a self-signed certificate created on the fly Type: String AllowedValues: - Disable - Enable Default: Enable KeyPair: Type: AWS::EC2::KeyPair::KeyName Description: "(Required) Select an EC2 key pair to allow remote connections to the VM, via SSH. Username: ubuntu (has sudo privileges)" VPC: Type: AWS::EC2::VPC::Id Description: (Required) VPC to deploy the cluster into Subnet: Type: AWS::EC2::Subnet::Id Description: (Required) Subnet to deploy the cluster into. Must be in the selected VPC AuthenticationMasterPassword: Description: Enter a password for the master console, with a maximum length of 32 (leave the field blank to randomly generate a password) Type: String NoEcho: true MaxLength: 32 Default: '' SecurityGroup: Type: AWS::EC2::SecurityGroup::Id Description: (Required) Subnet to deploy the cluster into. Must be in the selected Security Group Mappings: InstanceTypes: 4-vCPUs-16-GB-RAM-150-GB-SSD: InstanceType: m5d.xlarge RAM: 16 DiskSize: 150 4-vCPUs-32-GB-RAM-150-GB-SSD: InstanceType: r5dn.xlarge RAM: 32 DiskSize: 150 8-vCPUs-32-GB-RAM-300-GB-SSD: InstanceType: m5ad.2xlarge RAM: 32 DiskSize: 300 8-vCPUs-64-GB-RAM-300-GB-SSD: InstanceType: r5ad.2xlarge RAM: 64 DiskSize: 300 RegionMap: # Ubuntu Server 22.04 LTS af-south-1: # Africa (Cape Town) AMI: ami-0bd6e451a0211224f ap-east-1: # Asia Pacific (Hong Kong) AMI: ami-051699ed177bc3d67 ap-northeast-1: # Asia Pacific (Tokyo) AMI: ami-0c597cc9c10ca9088 ap-northeast-2: # Asia Pacific (Seoul) AMI: ami-0502b8f5f0ca3ed7d ap-northeast-3: # Asia Pacific (Osaka) AMI: ami-040b504c67641f0cc ap-south-1: # Asia Pacific (Mumbai) AMI: ami-0dc886bce1f85f67e ap-south-2: # Asia Pacific (Hyderabad) AMI: ami-04d2538aa74707978 ap-southeast-1: # Asia Pacific (Singapore) AMI: ami-0d21214905506a7f4 ap-southeast-2: # Asia Pacific (Sydney) AMI: ami-09fb5e610ae14ee00 ap-southeast-3: # Asia Pacific (Jakarta) AMI: ami-0f2e6ee654b98f96b ap-southeast-4: # Asia Pacific (Melbourne) AMI: ami-0de6423a6e32d147d ca-central-1: # Canada (Central) AMI: ami-02da4d5de61d161c5 eu-central-1: # Europe (Frankfurt) AMI: ami-03f1cc6c8b9c0b899 eu-central-2: # Europe (Zurich) AMI: ami-083714d0ac0c1a80a eu-north-1: # Europe (Stockholm) AMI: ami-05baaef454dd96656 eu-south-1: # Europe (Milan) AMI: ami-0faa5fa8924e8b7de eu-south-2: # Europe (Spain) AMI: ami-0b308244dcf4a624a eu-west-1: # Europe (Ireland) AMI: ami-0786f5bc3943ad52d eu-west-2: # Europe (London) AMI: ami-0ccdcf8ea5cace030 eu-west-3: # Europe (Paris) AMI: ami-0b108d96bcb0de81c me-central-1: # Middle East (UAE) AMI: ami-050fa395f4aaae15d me-south-1: # Middle East (Bahrain) AMI: ami-0515452e9b0f40e44 sa-east-1: # South America (São Paulo) AMI: ami-0cd79c08dc1353a91 us-east-1: # US East (N. Virginia) AMI: ami-003d3d03cfe1b0468 us-east-2: # US East (Ohio) AMI: ami-05f4e4084abd205cf us-west-1: # US West (N. California) AMI: ami-0ff832bdf91944651 us-west-2: # US West (Oregon) AMI: ami-032f8589b3e7f4e5b Conditions: cCheckHTTP: !Equals [ !Ref TLS, Enable] cGeneratePassword: !Equals [!Ref AuthenticationMasterPassword, ""] PublicIpAddress: !Equals [!Ref AssociatePublicIpAddress, "Yes"] Rules: SubnetsInVPC: Assertions: - Assert: 'Fn::EachMemberIn': - 'Fn::ValueOfAll': - 'AWS::EC2::Subnet::Id' - VpcId - 'Fn::RefAll': 'AWS::EC2::VPC::Id' AssertDescription: The selected subnet must be in the VPC Resources: rSecretManager: Type: AWS::SecretsManager::Secret Condition: cGeneratePassword Properties: GenerateSecretString: SecretStringTemplate: '{"username": "dcadmin"}' GenerateStringKey: 'password' PasswordLength: 20 ExcludePunctuation: true rRandomPass: Type: AWS::SSM::Parameter Condition: cGeneratePassword Properties: Name: RandomString Type: String Value: Fn::Sub: "{{resolve:secretsmanager:${rSecretManager}::password}}" rDClarityInstance: Type: AWS::EC2::Instance Properties: ImageId: !FindInMap [RegionMap, !Ref "AWS::Region", AMI] InstanceType: !FindInMap [InstanceTypes, !Ref InstanceType, InstanceType] BlockDeviceMappings: - DeviceName: /dev/sda1 Ebs: VolumeSize: !FindInMap [InstanceTypes, !Ref InstanceType, DiskSize] DeleteOnTermination: true VolumeType: gp2 KeyName: !Ref KeyPair UserData: Fn::Base64: !Sub - | #!/bin/bash # Install Docker and Docker Compose sudo apt-get update sudo apt-get install ca-certificates curl gnupg -y install -m 0755 -d /etc/apt/keyrings curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg chmod a+r /etc/apt/keyrings/docker.gpg echo \ "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \ "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \ tee /etc/apt/sources.list.d/docker.list > /dev/null apt-get update apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin -y usermod -aG docker ubuntu systemctl enable docker.service systemctl enable containerd.service # Download and untar the DataClarity package curl -L https://github.com/DataClarityCorp/distribution/raw/main/compose/aws/dataclarity.tar.gz -o /opt/dataclarity.tar.gz tar -zxf /opt/dataclarity.tar.gz -C /opt chmod -R 777 /opt/dataclarity cd /opt/dataclarity/ # Setup a self-signed certificate if [[ ${TLS} = "Enable" ]]; then sudo apt-get install -y certbot sudo certbot certonly --standalone -d "${IP}" --register-unsafely-without-email --agree-tos --deploy-hook "cp /etc/letsencrypt/live/${IP}/privkey.pem /opt/dataclarity/assets/certs/dc/cert.key; cp /etc/letsencrypt/live/${IP}/fullchain.pem /opt/dataclarity/assets/certs/dc/cert.crt" if [[ $? -ne 0 ]]; then openssl req -x509 -nodes -days 36500 -newkey rsa:2048 -subj "/CN=${IP}" -keyout assets/certs/dc/cert.key -out assets/certs/dc/cert.crt fi sed -i 's|http://$PUBLIC_ENTRYPOINT|https://$PUBLIC_ENTRYPOINT|g' docker-compose.yml sed -i 's|http://{host}|https://{host}|g' docker-compose.yml sed -i 's|http://$PUBLIC_ENTRYPOINT|https://$PUBLIC_ENTRYPOINT|g' assets/config/notification/application.properties sed -i 's|http://$PUBLIC_ENTRYPOINT|https://$PUBLIC_ENTRYPOINT|g' assets/config/ui/config.json rm assets/config/load-balancer/nginx.conf mv assets/config/load-balancer/nginx_ssl.conf assets/config/load-balancer/nginx.conf else rm assets/config/load-balancer/nginx_ssl.conf fi # Update the entry point with the Domain, in the config files sed -i "s|\$PUBLIC_ENTRYPOINT|${IP}|g" docker-compose.yml sed -i "s|\$PUBLIC_ENTRYPOINT|${IP}|g" assets/config/notification/application.properties sed -i "s|\$PUBLIC_ENTRYPOINT|${IP}|g" assets/config/ui/config.json # Update dcadmin password, in the env file sed -i "s|\KEYCLOAK_PASSWORD=PXJ3WOymAYXYur6K|KEYCLOAK_PASSWORD=${PASSWORD}|g" .env if [[ ${RAM} = "16" ]]; then sed -i "s|DRILLBIT_MAX_PROC_MEM: 12G|DRILLBIT_MAX_PROC_MEM: 10G|g" docker-compose.yml elif [[ ${RAM} = "64" ]]; then sed -i "s|DRILLBIT_MAX_PROC_MEM: 12G|DRILLBIT_MAX_PROC_MEM: 48G|g" docker-compose.yml fi # Create the DataClarity persistent volumes docker volume create pgdata docker volume create zkdata docker volume create zklogs docker volume create drill-dfs docker volume create screenshots-storage # Create the DataClarity containers docker compose up -d - TLS: !Ref TLS RAM: !FindInMap [InstanceTypes, !Ref InstanceType, RAM] IP: !If [PublicIpAddress, $(curl http://169.254.169.254/latest/meta-data/public-hostname), $(curl http://169.254.169.254/latest/meta-data/local-ipv4)] PASSWORD: !If [cGeneratePassword, !GetAtt rRandomPass.Value, !Ref AuthenticationMasterPassword] NetworkInterfaces: - AssociatePublicIpAddress: !If [PublicIpAddress , "true", "false"] SubnetId: !Ref Subnet DeviceIndex: 0 GroupSet: - !Ref SecurityGroup Outputs: ApplicationStartNotice: Value: "Please note that it may take 5-10 minutes for the application to fully start. Please wait for the application to become operational before accessing the URL." ApplicationURL: Description: "Use these credentials to log into DataClarity. Username: admin | Password: admin. You will be required to change admin user password on the first login." Value: Fn::Join: - "" - - !If [cCheckHTTP, "https://", "http://"] - !If [PublicIpAddress, !GetAtt rDClarityInstance.PublicDnsName, !GetAtt rDClarityInstance.PrivateIp] - "/home" ApplicationMasterConsoleURL: Description: "Use this URL to access the DataClarity User Access Master Console" Value: Fn::Join: - "" - - !If [cCheckHTTP, "https://", "http://"] - !If [PublicIpAddress, !GetAtt rDClarityInstance.PublicDnsName, !GetAtt rDClarityInstance.PrivateIp] - "/auth/admin/master/console" ApplicationMasterConsoleURLCredentials: Description: "Use these credentials to log into DataClarity User Access Master Console." Value: Fn::Join: - "" - - "Username: dcadmin | Password: " - !If [cGeneratePassword, !GetAtt rRandomPass.Value, !Ref AuthenticationMasterPassword]