--- AWSTemplateFormatVersion: 2010-09-09 Description: >- This template sets up Amazon AppStream 2.0 with QGIS along side a PostGIS database on Amazon RDS. This combination allows for high performance between the QGIS GIS application and the PostGIS database while allowing end users easy access without any client-side setup. Metadata: AWS::CloudFormation::Interface: ParameterGroups: - Label: default: "VHD Builder Configuration" Parameters: - VHDBucketName - VHDBuilderEC2AmiId - Label: default: "AppStream 2.0 Configuration" Parameters: - ApplicationName - IconURL - AS2Subnet1 - AS2Subnet2 - AS2VPC - MaxConcurrentSessions - Label: default: "Streaming User Configuration" Parameters: - StreamingUserName - ValidityDuration - Label: default: "RDS PostGIS Configuration" Parameters: - DBInstanceType - DBAllocatedStorage - DBMasterUsername - DBMasterUserPassword - InstallExampleData ParameterLabels: AS2VPC: default: "Which VPC should this be deployed to?" Parameters: VHDBucketName: Description: Name of the S3 bucket to which your VHD file will be uploaded. Type: String MinLength: 3 MaxLength: 63 VHDBuilderEC2AmiId: Description: AMI ID of Windows Server 2019 based EC2 instance that builds the QGIS application and sets up PostGIS on RDS Type: AWS::SSM::Parameter::Value<AWS::EC2::Image::Id> Default: /aws/service/ami-windows-latest/Windows_Server-2019-English-Full-Base ApplicationName: Description: Name of the Application in AppStream 2.0 Default: QGIS Type: String MinLength: 3 MaxLength: 63 IconURL: Description: Publicly accessible URL of app icon image that will be shown to users in AppStream 2.0 (must be .png format) Default: https://qgis.org/en/_downloads/24e1d276c05d6aed1f63e40820ff1b6e/qgis-icon-black128.png Type: String AS2Subnet1: Description: Specify the first subnet where the AppStream 2.0 fleet will be deployed. There must be an outbound route from this subnet to S3. This subnet will also be used for the VHD Builder Instance and an RDS instance (PostgreSQL). Type: 'AWS::EC2::Subnet::Id' Default: "" AS2Subnet2: Description: Specify the second subnet where the AppStream 2.0 fleet will be deployed. There must be an outbound route from this subnet to S3. Type: 'AWS::EC2::Subnet::Id' Default: "" AS2VPC: Description: Specify the VPC where your AppStream 2.0 fleets will be deployed. This must match the VPC of the subnet you pick for the AS2Subnet1 and AS2Subnet2 parameters. Type: 'AWS::EC2::VPC::Id' Default: "" MaxConcurrentSessions: Description: The maximum number of concurrent sessions that can be run on your AppStream 2.0 fleet. Type: Number Default: 20 StreamingUserName: Description: The name of the user that be used to connect to AppStream 2.0 via streaming URL. Type: String Default: "test-user" ValidityDuration: Description: The duration (in seconds) that the generated streaming URL will be valid Type: Number Default: 7200 DBInstanceType: Description: EC2 instance type used by the RDS Postgres database Type: String Default: db.m6g.large DBAllocatedStorage: Description: RDS database storage size in GB Type: Number Default: 20 DBMasterUsername: Description: Database master username Type: String Default: postgres DBMasterUserPassword: Description: Database master password. The password must contain 8 to 128 characters and any printable ASCII character except / , ` , or @ Type: String MinLength: 8 MaxLength: 128 AllowedPattern: ^[^@`/]*$ NoEcho: true InstallExampleData: Description: If true, example dataset showing Ookla internet speeds is downloaded and put into PostGIS Type: String AllowedValues: - true - false Default: true Mappings: Variables: ReadOnlyUserNameForRDS: Value: read_only_user ReadOnlyUserPasswordForRDS: Value: please_reset_me Resources: RDSDBSecret: Type: 'AWS::SecretsManager::Secret' Properties: Name: PostGIS_db_admin_credentials Description: 'admin credentials for PostGIS database' SecretString: !Sub '{"username":"${DBMasterUsername}","password":"${DBMasterUserPassword}"}' RDSPostGIS: Type: 'AWS::RDS::DBInstance' Properties: DBInstanceClass: !Ref DBInstanceType AllocatedStorage: !Ref DBAllocatedStorage DBSubnetGroupName: !Ref RDSSubnetGroup VPCSecurityGroups: - !Ref RDSSecurityGroup Engine: postgres MasterUsername: Fn::Sub: "{{resolve:secretsmanager:${RDSDBSecret}::username}}" MasterUserPassword: Fn::Sub: "{{resolve:secretsmanager:${RDSDBSecret}::password}}" StorageEncrypted: true StorageType: gp2 PubliclyAccessible: false RDSSubnetGroup: Type: 'AWS::RDS::DBSubnetGroup' Properties: DBSubnetGroupName: RDSPostgresSubnetGroup DBSubnetGroupDescription: 'Subnet Group contains subnets that can be used by RDS. These should ideally overlap the AppStream Fleet AZs.' SubnetIds: - !Ref AS2Subnet1 - !Ref AS2Subnet2 RDSSecurityGroup: Type: 'AWS::EC2::SecurityGroup' Properties: GroupDescription: 'RDS database security group. Allows ingress from VHD EC2 instance and AppStream Fleet' VpcId: !Ref AS2VPC SecurityGroupEgress: - IpProtocol: -1 CidrIp: 127.0.0.1/32 Description: 'This effectively blocks all outbound traffic by routing it to localhost' SecurityGroupIngress: - IpProtocol: tcp FromPort: 5432 ToPort: 5432 SourceSecurityGroupId: !Ref VHDBuilderSecurityGroup Description: 'Allow Postgres access over port 5432 from the VHD Buidler EC2 instance' - IpProtocol: tcp FromPort: 5432 ToPort: 5432 SourceSecurityGroupId: !Ref FleetSecurityGroup Description: 'Allow Postgres access over port 5432 from the AppStream Fleet' VHDBuilderEC2Instance: Type: 'AWS::EC2::Instance' DependsOn: RDSPostGIS # This EC2 instance is used to install PostGIS so we wait for RDS before creating the EC2 instance CreationPolicy: ResourceSignal: Timeout: PT120M Properties: InstanceInitiatedShutdownBehavior: terminate IamInstanceProfile: !Ref VHDBuilderInstanceProfile SecurityGroupIds: - !Ref VHDBuilderSecurityGroup ImageId: !Ref VHDBuilderEC2AmiId InstanceType: t3.small SubnetId: !Ref AS2Subnet1 Monitoring: 'false' BlockDeviceMappings: - DeviceName: "/dev/sda1" Ebs: DeleteOnTermination: true Encrypted: true UserData: Fn::Base64: !Sub - | <script> REM create vhdx file and mount it mkdir C:\OSGeo4W (echo create vdisk file=C:\osgeo4w.vhdx maximum=8000 type=expandable echo select vdisk file=C:\osgeo4w.vhdx echo attach vdisk echo convert mbr echo create partition primary echo format fs=ntfs quick echo assign mount=C:\OSGeo4W ) | diskpart REM download OSGeo4W installer file powershell -Command "(New-Object Net.WebClient).DownloadFile('http://download.osgeo.org/osgeo4w/v2/osgeo4w-setup.exe','C:\osgeo4w-setup.exe')" REM install package(s) from OSGeo4W ScriptRunner.exe -appvscript C:\osgeo4w-setup.exe --advanced --autoaccept --arch x86_64 --menu-name "QGIS" --no-desktop --local-package-dir "C:\OSGeo4W-tmp" --root "C:\OSGeo4W" --packages qgis-ltr --site "http://download.osgeo.org/osgeo4w/v2" --site "http://www.norbit.de/osgeo4w/v2" --site "http://ftp.osuosl.org/pub/osgeo/download/osgeo4w/v2" --quiet-mode -appvscriptrunnerparameters -wait REM Copy necessary dlls back into vhd powershell -Command "Copy-Item -Path %WINDIR%\system32\opencl.dll -Destination C:\OSGeo4W\bin\opencl.dll -PassThru" REM connect to RDS Postgres database and add PostGIS REM https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.PostgreSQL.CommonDBATasks.PostGIS.html >C:\postgis.sql echo CREATE EXTENSION postgis; >>C:\postgis.sql echo CREATE EXTENSION fuzzystrmatch; >>C:\postgis.sql echo CREATE EXTENSION postgis_tiger_geocoder; >>C:\postgis.sql echo CREATE EXTENSION postgis_topology; >>C:\postgis.sql echo ALTER SCHEMA tiger OWNER TO rds_superuser; >>C:\postgis.sql echo ALTER SCHEMA tiger_data OWNER TO rds_superuser; >>C:\postgis.sql echo ALTER SCHEMA topology OWNER TO rds_superuser; >>C:\postgis.sql echo CREATE FUNCTION exec(text) returns text language plpgsql volatile AS $f$ BEGIN EXECUTE $1; RETURN $1; END; $f$; >>C:\postgis.sql echo SELECT exec('ALTER TABLE ' ^|^| quote_ident(s.nspname) ^|^| '.' ^|^| quote_ident(s.relname) ^|^| ' OWNER TO rds_superuser;') >>C:\postgis.sql echo FROM ( >>C:\postgis.sql echo SELECT nspname, relname >>C:\postgis.sql echo FROM pg_class c JOIN pg_namespace n ON (c.relnamespace = n.oid) >>C:\postgis.sql echo WHERE nspname in ('tiger','topology') AND >>C:\postgis.sql echo relkind IN ('r','S','v') ORDER BY relkind = 'S') >>C:\postgis.sql echo s; REM install the AWS CLI curl https://awscli.amazonaws.com/AWSCLIV2.msi --output C:\awscli.msi C:\awscli.msi /quiet REM wait for 30 seconds so AWS CLI has time to install in the background powershell -Command "Sleep 30" REM get database password from Secrets Manager and put it into PGPASSWORD env var for /F "usebackq tokens=1" %%i in (`powershell -Command "$secretsmanager = C:\PROGRA~1\Amazon\AWSCLIV2\aws.exe secretsmanager get-secret-value --secret-id PostGIS_db_admin_credentials | ConvertFrom-Json;$secretstring = $secretsmanager.SecretString | ConvertFrom-Json;$secretstring.password"`) do set PGPASSWORD=%%i C:\OSGeo4W\bin\psql.exe -h ${RDSPostGIS.Endpoint.Address} -p 5432 -U ${DBMasterUsername} -d postgres -f "C:\postgis.sql" REM Create Postgres read-only user to avoid using master user for anything other than administration tasks REM This read only user is being given a temporary password. You are encouraged to change this after you get everything set up. REM The first REVOKE line will prevent default users from creating tables. Superusers can still create tables. >C:\postgresuser.sql echo REVOKE CREATE ON SCHEMA public FROM public; >>C:\postgresuser.sql echo CREATE ROLE readonly; >>C:\postgresuser.sql echo GRANT CONNECT ON DATABASE postgres TO readonly; >>C:\postgresuser.sql echo GRANT USAGE ON SCHEMA public TO readonly; >>C:\postgresuser.sql echo GRANT SELECT ON ALL TABLES IN SCHEMA public TO readonly; >>C:\postgresuser.sql echo ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON TABLES TO readonly; >>C:\postgresuser.sql echo CREATE USER ${READONLY_USERNAME} WITH PASSWORD '${READONLY_PASSWORD}'; >>C:\postgresuser.sql echo GRANT readonly TO ${READONLY_USERNAME}; C:\OSGeo4W\bin\psql.exe -h ${RDSPostGIS.Endpoint.Address} -p 5432 -U ${DBMasterUsername} -d postgres -f "C:\postgresuser.sql" REM Only install Example Data if user sets InstallExampleData to true set INSTALL_EXAMPLE_DATA=${InstallExampleData} if %INSTALL_EXAMPLE_DATA%==true ( REM download Ookla internet speed dataset "C:\Program Files\Amazon\AWSCLIV2\aws.exe" s3 cp s3://ookla-open-data/shapefiles/performance/type=fixed/year=2021/quarter=3/2021-07-01_performance_fixed_tiles.zip C:\ookla.zip --no-sign-request REM unzip the data powershell -Command "Expand-Archive -Force C:\ookla.zip C:\ookla" REM add the downloaded Ookla dataset into RDS PostGIS C:\OSGeo4W\bin\ogr2ogr.exe -f "PostgreSQL" "PG:host=${RDSPostGIS.Endpoint.Address} user=${DBMasterUsername} dbname=postgres password=%PGPASSWORD%" "C:\ookla\gps_fixed_tiles.shp" REM download the qgs file from GitHub curl https://raw.githubusercontent.com/aws-samples/appstream-qgis-blog/main/ookla-download-speed.qgs --output C:\ookla-example-project.qgs REM find and replace the placeholder RDS hostname with the real RDS hostname and add it to the OSGeo4W package powershell -Command "(gc C:\ookla-example-project.qgs) -replace 'placeholderhostname', '${RDSPostGIS.Endpoint.Address}' | Out-File -encoding ASCII C:\OSGeo4W\ookla-example-project.qgs" ) REM detach VMDX (echo select vdisk file=C:\osgeo4w.vhdx echo detach vdisk ) | diskpart REM create a setup script for mounting VMD and running the app >C:\qgis.bat echo mkdir C:\OSGeo4W >>C:\qgis.bat echo (echo select vdisk file=C:\AppStream\AppBlocks\${ApplicationName}\osgeo4w.vhdx >>C:\qgis.bat echo echo attach vdisk >>C:\qgis.bat echo echo rescan >>C:\qgis.bat echo echo select partition 1 >>C:\qgis.bat echo echo remove all noerr >>C:\qgis.bat echo echo assign mount=C:\OSGeo4W >>C:\qgis.bat echo echo exit >>C:\qgis.bat echo ) ^| diskpart >>C:\qgis.bat echo copy /y C:\OSGeo4W\bin\opencl.dll C:\Windows\system32\ >>C:\qgis.bat echo copy /y C:\OSGeo4W\bin\libcrypto-1_1-x64.dll C:\Windows\system32\ >>C:\qgis.bat echo copy /y C:\OSGeo4W\bin\libssl-1_1-x64.dll C:\Windows\system32\ >>C:\qgis.bat echo copy /y C:\OSGeo4W\ookla-example-project.qgs "C:\Users\PhotonUser\My Files\Temporary Files" REM download QGIS icon powershell -Command "(New-Object Net.WebClient).DownloadFile('${IconURL}','C:\qgis.png')" REM upload required files to S3 powershell -Command "Write-S3Object -BucketName ${VHDBucketName} -Key 'osgeo4w.vhdx' -File C:\osgeo4w.vhdx" powershell -Command "Write-S3Object -BucketName ${VHDBucketName} -Key 'qgis.bat' -File C:\qgis.bat" powershell -Command "Write-S3Object -BucketName ${VHDBucketName} -Key 'qgis.png' -File C:\qgis.png" REM use cfn-signal to let CloudFormation know that the EC2 instance has been fully provisioned cfn-signal.exe -e %errorlevel% --stack ${AWS::StackId} --resource VHDBuilderEC2Instance --region ${AWS::Region} REM shutdown the EC2 instance, which will in turn terminate it so we don't incur unnecessary cost once the VHDX file is created shutdown /s </script> - { READONLY_USERNAME: !FindInMap [ Variables, ReadOnlyUserNameForRDS, Value ], READONLY_PASSWORD: !FindInMap [ Variables, ReadOnlyUserPasswordForRDS, Value ] } Tags: - Key: Name Value: VHD Builder for AppStream 2.0 VHDBuilderInstanceProfile: Type: AWS::IAM::InstanceProfile Properties: InstanceProfileName: vhd-builder-ec2-instance-profile Path: / Roles: - !Ref VHDBuilderEc2InstanceRole VHDBuilderEc2InstanceRole: Type: AWS::IAM::Role Properties: RoleName: vhd-builder-ec2-instance-role AssumeRolePolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Principal: Service: - ec2.amazonaws.com Action: - sts:AssumeRole Path: / ManagedPolicyArns: - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore VHDBuilderInstanceS3Policy: Type: AWS::IAM::Policy Properties: PolicyName: VHDBuilderInstanceS3Policy PolicyDocument: Version: 2012-10-17 Statement: - Sid: S3Access Effect: Allow Action: [ 's3:PutObject' ] Resource: [!GetAtt [VHDStorageS3Bucket, Arn], !Join ['', [!GetAtt [VHDStorageS3Bucket, Arn], /*]]] Roles: - !Ref VHDBuilderEc2InstanceRole VHDBuilderInstanceSecretsManagerPolicy: Type: AWS::IAM::Policy Properties: PolicyName: VHDBuilderInstanceSecretsManagerPolicy PolicyDocument: Version: 2012-10-17 Statement: - Sid: SecretsManagerAccess Effect: Allow Action: [ 'secretsmanager:GetSecretValue' ] Resource: !Ref RDSDBSecret Roles: - !Ref VHDBuilderEc2InstanceRole VHDBuilderSecurityGroup: Type: 'AWS::EC2::SecurityGroup' Properties: GroupDescription: Security group for instance which builds VHD and uploads it to S3 bucket. VpcId: !Ref AS2VPC SecurityGroupEgress: - Description: Allow all outbound traffic IpProtocol: "-1" CidrIp: 0.0.0.0/0 VHDStorageS3Bucket: Type: 'AWS::S3::Bucket' Properties: BucketName: !Ref VHDBucketName PublicAccessBlockConfiguration: BlockPublicAcls: true IgnorePublicAcls: true BlockPublicPolicy: true RestrictPublicBuckets: true VersioningConfiguration: Status: Enabled BucketEncryption: ServerSideEncryptionConfiguration: - ServerSideEncryptionByDefault: SSEAlgorithm: "AES256" VHDStorageS3BucketPolicy: Type: AWS::S3::BucketPolicy Properties: Bucket: !Ref VHDStorageS3Bucket PolicyDocument: Version: 2012-10-17 Statement: - Sid: 'AllowAppStream2.0ToRetrieveObjects' Action: - 's3:GetObject' Effect: Allow Resource: !Join - '' - - 'arn:aws:s3:::' - !Ref VHDBucketName - /* Principal: Service: - 'appstream.amazonaws.com' AppBlock: DependsOn: VHDBuilderEC2Instance Type: AWS::AppStream::AppBlock Properties: DisplayName: !Ref ApplicationName Name: !Ref ApplicationName SetupScriptDetails: ExecutablePath: !Sub 'C:\AppStream\AppBlocks\${ApplicationName}\qgis.bat' ScriptS3Location: S3Bucket: !Ref VHDBucketName S3Key: qgis.bat TimeoutInSeconds: 60 SourceS3Location: S3Bucket: !Ref VHDBucketName S3Key: osgeo4w.vhdx Application: Type: AWS::AppStream::Application Properties: AppBlockArn: !Ref AppBlock Description: !Ref ApplicationName DisplayName: !Ref ApplicationName IconS3Location: S3Bucket: !Ref VHDBucketName S3Key: qgis.png InstanceFamilies: - GENERAL_PURPOSE LaunchPath: 'C:\OSGeo4W\bin\qgis-ltr-bin.exe' LaunchParameters: '--noversioncheck' Name: !Ref ApplicationName Platforms: - WINDOWS_SERVER_2019 Fleet: Type: AWS::AppStream::Fleet Properties: Description: !Ref ApplicationName DisplayName: !Ref ApplicationName Name: !Ref ApplicationName Platform: WINDOWS_SERVER_2019 MaxConcurrentSessions: !Ref MaxConcurrentSessions FleetType: ELASTIC InstanceType: stream.standard.medium VpcConfig: SecurityGroupIds: - !Ref FleetSecurityGroup SubnetIds: - !Ref AS2Subnet1 - !Ref AS2Subnet2 CreationPolicy: StartFleet: True FleetSecurityGroup: Type: 'AWS::EC2::SecurityGroup' Properties: GroupDescription: Security group for AppStream 2.0 Fleet. VpcId: !Ref AS2VPC SecurityGroupEgress: - Description: Allow all outbound traffic IpProtocol: "-1" CidrIp: 0.0.0.0/0 ApplicationFleetAssociation: Type: AWS::AppStream::ApplicationFleetAssociation Properties: ApplicationArn: !Ref Application FleetName: !Ref Fleet Stack: Type: AWS::AppStream::Stack Properties: Description: !Ref ApplicationName DisplayName: !Ref ApplicationName Name: !Ref ApplicationName #TODO add storage connector and wire it up to existing S3 bucket #TODO add ApplicationSettings so user settings are stored between sessions StackFleetAssociation: Type: AWS::AppStream::StackFleetAssociation Properties: FleetName: !Ref Fleet StackName: !Ref Stack CreateStreamingURLLambdaFunctionRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Principal: Service: [lambda.amazonaws.com] Action: ['sts:AssumeRole'] Path: / ManagedPolicyArns: - 'arn:aws-us-gov:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole' Policies: - PolicyName: CreateStreamingURLLambdaFunctionExecutionPolicy PolicyDocument: Version: '2012-10-17' Statement: - Sid: CloudWatchLogGroupAccess Effect: Allow Action: ['logs:CreateLogGroup', 'logs:CreateLogStream', 'logs:PutLogEvents'] Resource: !Sub "arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/*" - Sid: AppStreamAccess Effect: "Allow" Action: "appstream:CreateStreamingURL" Resource: - !Sub "arn:${AWS::Partition}:appstream:${AWS::Region}:${AWS::AccountId}:fleet/${ApplicationName}" - !Sub "arn:${AWS::Partition}:appstream:${AWS::Region}:${AWS::AccountId}:stack/${ApplicationName}" CreateStreamingURLLambdaLogGroup: Type: AWS::Logs::LogGroup Properties: LogGroupName: /aws/lambda/AppStreamQGISCreateStreamingURL RetentionInDays: 7 # This ensures your logs will not be kept indefinitely CreateStreamingURLLambda: DependsOn: CreateStreamingURLLambdaLogGroup Type: AWS::Lambda::Function Properties: FunctionName: AppStreamQGISCreateStreamingURL Handler: index.lambda_handler Runtime: "python3.9" MemorySize: 128 Timeout: 30 Role: Fn::GetAtt: - CreateStreamingURLLambdaFunctionRole - Arn Code: ZipFile: | import json import cfnresponse import boto3 def lambda_handler(event, context): print('Received request:\n%s' % json.dumps(event, indent=4)) if(event['RequestType'] == 'Delete'): responseData = {} cfnresponse.send(event, context, cfnresponse.SUCCESS, responseData, "CustomResourcePhysicalID") return fleenName = event['ResourceProperties']['fleetName'] stackName = event['ResourceProperties']['stackName'] validityDuration = int(event['ResourceProperties']['validityDuration']) userName = event['ResourceProperties']['userName'] appstream_client = boto3.client('appstream') response = appstream_client.create_streaming_url( StackName=stackName, FleetName=fleenName, UserId=userName, Validity=validityDuration ) streamingURL = response['StreamingURL'] responseData = {} responseData['Data'] = streamingURL cfnresponse.send(event, context, cfnresponse.SUCCESS, responseData, "CustomResourcePhysicalID") CreateStreamingURL: DependsOn: AppBlock Type: Custom::CreateStreamingURL Properties: ServiceToken: !GetAtt CreateStreamingURLLambda.Arn stackName: !Ref ApplicationName fleetName: !Ref ApplicationName userName: !Ref StreamingUserName validityDuration: !Ref ValidityDuration Outputs: StreamingURL: Description: URL to access your AppStream fleet. Valid for 1 hour after stack creation. Value: !GetAtt CreateStreamingURL.Data ReadOnlyUserNameForRDS: Description: Username for read only user that can access sample data. Use this username to connect to Postgres from within QGIS. Value: !FindInMap [Variables, ReadOnlyUserNameForRDS, Value] ReadOnlyUserPasswordForRDS: Description: Password for read only user that can access sample data. Use this password to connect to Postgres from within QGIS. Value: !FindInMap [Variables, ReadOnlyUserPasswordForRDS, Value]