Amazon Connect Integration - Customer Hosted - Cloudformation Guide

Table of Contents

Introduction

eMite is a leading global contact center analytics solution that combines advanced analytics, data correlation, KPI management and threshold alerting into a single solution. eMite provides actionable insights from both real-time and historical data to help improve contact centre efficiency, performance, and customer experience (CX).

To know more about our software you can visit our website; https://www.emite.com/

This guide is for implementing a Customer-Hosted eMite solution with your AWS Amazon Connect instance.
This involves the deployment of several Amazon services on AWS Cloud, such as Kinesis streams and Opensearch.
For easier deployment of these components - we can make use of a CloudFormation template to create a CloudFormation stack.

No third-party tools are included in the eMite deployment.

Region Deployment

This deployment is only applicable for regions that have Amazon Connect instances available.
Here is the list of available regions:

• US East (N. Virginia) - us-east-1

• US West (Oregon - us-west-2

• Asia Pacific (Seoul) - ap-northeast-2

• Asia Pacific (Singapore) - ap-southeast-1

• Asia Pacific (Sydney) - ap-southeast-2

• Asia Pacific (Tokyo) - ap-northeast-1

• Canada (Central) - ca-central-1

• Europe (Frankfurt) - eu-central-1

• Europe (London) - eu-west-2

Prerequisites and Requirements

Time

The cloudformation template deployment will take about 2-3 hours.
Installation, configuration, and testing of eMite could take up to another 2-3hours.

Product License

For the eMite product license, please reach out to your assigned eMite sales/account manager, or contact support (support@prophecyinternational.com)
No other license is required for this deployment.

AWS Account

This guide assumes you have an active AWS Account - and that this deployment will run on the same AWS Account as your Amazon Connect instance that eMite will be integrating with.

AWS Identity and Access Management Entity

Do not use your root account to deploy the CloudFormation template.
You can use an IAM account with Full Administrator privileges for this deployment. However, you can also choose to create an IAM user or role instead.
The IAM user/role should have a policy that allows AWS CloudFormation actions and has launch/deployment permissions to all services listed in the following section.
For specific policy actions required for deployment, please refer to this section:
https://prophecyinternational.atlassian.net/wiki/spaces/AMCON/pages/1940423003/Amazon+Connect+Integration+-+Customer+Hosted+-+Cloudformation+Guide#For-Deployment

AWS Knowledge Requirement

Knowledge of the following AWS services:

• Amazon Elastic Compute Cloud (Amazon EC2)

  • Ability to create EC2 key pair

• Amazon Elastic Block Storage (Amazon EBS)

• Amazon Virtual Private Cloud (Amazon VPC)

• AWS Certificate Manager (Amazon ACM)

  • Ability to create certificates in ACM or upload certificates to ACM (if there are no existing certificate)

• AWS CloudFormation

  • Ability to deploy Cloudformation Template/Stacks

• AWS Elastic Load Balancing (Application Load Balancer)

• Amazon Kinesis Data Streams

• Amazon Connect

• Amazon Simple Notification Service (Amazon SNS)

• Amazon OpenSearch Service

Specific Skill Requirements

Customer personnel involved in deployment should have the following specific skills:

• For running the Cloudformation template (and AWS resources)

  • AWS Cloud Administration - particularly in:

    • Cloudformation Stack deployment / troubleshooting

    • Amazon Connect data streaming configuration

    • EC2 Key Pair creation/deletion

    • AWS Certificate Manager - creating or uploading signed certificates for use with an application load balancer

    • Updating rules on Network Security Groups

  • Note: Generally, individuals possessing at least the AWS Associate Certification or higher should have a sufficient skill

• For deploying eMite application after Cloudformation deployment

  • Windows Server Administration, particularly in:

    • Running/Configuring powershell scripts

    • IIS Administration

  • SQL Server Administration, particularly in:

    • Creation, Restoration of databases

  • Elasticsearch/Opensearch Administration

• For managing the eMite application post-deployment

  • AWS Cloud Administration / System Operations Engineer - particularly in:

    • Updating Cloudformation stacks for changing resource sizing

    • Viewing Metrics on Cloudwatch, and alarm creation/management

    • EC2 Operations such as increasing EBS storage and changing instance types

    • Kinesis Streams configuration

    • Opensearch domain configuration

  • Windows Server Administration - particularly for managing IIS and Windows Services

  • SQL Server Administration - particularly in running DQL queries

  • Elasticsearch/Opensearch Administration - particularly for managing cluster/shard health

Other requirements

• Internet Access

  • The Bastion Host EC2 will be deployed on public subnet and will only need incoming internet access via the Internet Gateway on the VPC

  • The eMite App EC2 will be deployed on private subnet and will only need outgoing internet access via the NAT Gateway on the VPC

• Windows Server 2019 Standard Base image

  • The deployment can use the Base Image provided by AWS, or your own custom Windows 2019 base image

  • Server Roles

    • The server will require the Web Server Role (IIS)

    • eMite team will install and configure this role as part of the eMite Installation step

• Software

  • eMite team will provide customer with the download link for the deployment package zip file

    • The package contains the eMite software installer and various powershell and db scripts for installing/setting up eMite components on the EC2

    • The package must be copied/downloaded onto the eMite App EC2 after the cloudformation deployment

    • eMite team can assist with this task as part of the eMite Installation step

  • SQL Server Express Edition installer will be provided as part of the deployment package zip file as well

    • eMite team will install this software as part of the eMite Installation step

• Ec2KeyPairNameApp

  • If you don’t have an existing EC2 Key Pair, or prefer to use a new EC2 Key Pair for the eMite App server:

    • Open the new console window and go to EC2 > Key Pairs (Left pane) > Select Create Key Pair

    • Open

      

      Provide a key pair name for the EC2 eMite Application.

• Ec2KeyPairNameBastion

  • If you don’t have an existing EC2 Key Pair, or prefer to use a new EC2 Key Pair for the Bastion Host:

    • Open the new console window and go to EC2 > Key Pairs (Left pane) > Select Create Key Pair

    • Open

      

      Provide a key pair name for the Bastion Host.

• CertificateARN

  • Open a new console window and go to Certificate Manager > Select your certificate

  • On the details tab, copy the ARN

  • Note that you can let AWS create and manage your certificate - or upload the certificate yourself

Managing AWS Service Limits

Please review your service quotas on the region where the solution will be deployed.
More information on viewing your service quotas here: https://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html
The above link on AWS Documentation also shows how you can request for a quota increase if needed.

Specific AWS Service Limits for this deployment:

Architecture

The above architecture is the standard deployment model for customer-hosted deployments.

• The setup is isolated to its own Virtual Private Cloud (VPC) and will be deployed in a Single-AZ.

• The eMite EC2 server and Opensearch Domain (Elasticsearch engine) will be deployed to a private subnet.

• For remote RDP access, a Bastion Host will be deployed in the public subnet.

• The Application Load Balancer (ALB) will be deployed in the public server which will use by the end-users to access the eMite environment.

• Internet Gateway (IGW) is used for connection from ALB to the end-users over the internet.

• NAT Gateway is used for outgoing internet access from the private subnet.

• IAM Role will be attached to the EC2 instance to access the needed AWS resources such as Kinesis Stream, Amazon Connect.

• SNS Topic would be used to send billing/usage data to eMite AWS.

AWS Resources Created

• Cloud Computing

  • EC2 with EBS

    • eMite Server - Private Subnet

      • Windows Server 2019

      • 50GB GP3 SSD Storage initially

    • Bastion Host - Public Subnet

      • Windows Server 2019

      • 30GB GP3 SSD Storage initially

• Networking

  • VPC

    • 1 new VPC will be deployed

• Subnet

  • 2 Public Subnet

    • This will be used by the application load balancer.

  • 1 Private Subnet

    • This will be used by the eMite server

• Route Tables

  • Routing tables for public subnets - routing through IGW

  • Routing tables for private subnets - routing through NAT Gateway

• Application Load Balancer - Public

  • Target Groups

    • Security Groups

  • NAT Gateway

    • This will be used by the eMite server to have access on the internet.

  • Internet Gateway

    • This will be used by in order to access the “Bastion Host”

• Opensearch Domain - Private subnet

  • Opensearch (using Elasticsearch engine) will serves as historical data storage.

  • Elasticsearch 7.9 Engine

  • 100GB SSD Storage initially

• Kinesis Streams

  • A total of 2 Kinesis streams will be deployed one for Contact Trace Record(CTR) Stream and one for Agents Events Stream

  • The two streams will pull the data from Amazon Connect and it will save the data to Opensearch.

  • Provisioned 1-shard configuration will be used initially for both streams

• SNS Topic

  • Will be used in the future by the eMite server for billing.

    • SNS Topic will publish billing information to eMite SQS

Deployment

Login to the AWS Console and go to Cloudformation Service in the same region as the Amazon Connect instance.

• Please ensure your account/role to be used for deployment has the necessary permissions detailed in the section “Prerequisites and Requirements”

VPC Stack Components

Use the VPC-Stack cloudformation template to create the VPC resources.

• Stack Name

  • Input your preferred stack name

  • E.g. vpc-emite-amazonconnect-stack

This will create a new VPC and its necessary components such as:

• Application Load Balancer

• NAT Gateway

• Internet Gateway

• Public/Private subnet.

Parameters

• NameResource

  • Your preferred identifier for your resources.

eMite Cloudformation Components

Use the eMite-Stack cloudformation template to create the VPC resources.

• Stack Name

  • Input your preferred stack name

  • E.g. emite-amazonconnect-stack

This will create the necessary components for the eMite deployment

1. Opensearch domain,

2. Kinesis streams

3. Bastion host

4. eMite App server

Parameters

• AgentsEventStreamName

  • Name of your AgentEvents kinesis stream.

• AllowedIpAccess

  • Allow IP address to access the eMite site (0.0.0.0/0 if public)

• AmazonConnectInstanceArn

  • Open a new console window and go to Amazon Connect > select your Amazon Connect instance

  • On the overview tab, copy the Instance ARN

• BastionInstanceName

  • Name of your Bastion Host eMite

• CtrStreamName

  • Name of CTR kinesis stream

• ContactLensARN

  • ARN of the S3 Bucket which contains Contact Lens data. Leave it as default if you do not have Contact Lens data.

  • Default: none

• ContactLensKMSARN

  • ARN of the KMS key that is used to encrypt the S3 Bucket which contains Contact Lens data. Leave it as default if you do not have Contact Lens data.

  • Default: none

• ConnnectReportsBucketArn

  • ARN of Amazon Connect Reports S3 Bucket. Leave it as default if you do not have Connect Reports Data.

  • Default: none

• ConnectReportsLocation

  • Location of Reports in S3 (excluding bucket name in path). Leave it as default if you do not have Connect Reports Data.

  • Default: none

• Ec2AmiId

  •  

  • Open the new console window and go to EC2 > Instances (Left pane) > Select Launch Instance

  • Search for the Microsoft Windows Server 2019 base

  • then Copy the ARN

• ESDomainName

  • Name of the Opensearch domain

• CertificateARN

  • The ARN of the provided certificate.

  • refer to the “Prerequisites and Requirements” section.

• Ec2KeyPairNameBastion

  • The name of the created Bastion hostkey pair.

  • refer to the “Prerequisites and Requirements” section.

• Ec2KeyPairNameApp

  • The name of the created EC2 key pair.

  • refer to “Prerequisites and Requirements” section.

• InstanceName

  • Name of the eMite instance

• NameResource

  • Your preferred identifier for your resources.

• VPCStack

  • Name of the VPC stack you deployed.

Configure Stack Options

• You can skip the other options here, and click Next

Review

• Tick this box under Capabilities - since the CloudFormation stack will be creating IAM roles/policies

• Click Create Stack

Check Status of Stack Creation

On the CloudFormation Console > Stacks, click the stack you created

You should see the current status on Stack Info tab, and more details under the Event tab

Wait for Status to change to CREATE_COMPLETE before proceeding to the next step below

Enable Amazon Connect Data Streaming

Now that the Kinesis Stream and Firehose have been setup, we can now enable data streaming from Amazon Connect.

• Go to Amazon Connnect > select your Amazon Connect instance

• Go to Data Streaming

  • Tick Enable data streaming

  • Contact Trace Records

    • Select Kinesis Stream

    • Select connect-ctr

  • Agent Events

    • Select Kinesis Stream

    • Select connect-agentevents

Complete Registration / Send data to eMite team

Data to be sent to eMite team can be found on the Outputs tab of the created stack:
Please send these details to eMite Cloud Operations Team (cloudops@prophecyinternational.com)

• Opensearch Domain Endpoint URL:

  • Copy the Value for ESDomainEndpointURL

• AgentEvents Kinesis Stream ARN

  • Copy the Value returned for AgentEventsKinesisARN

• CTR Kinesis Stream ARN

  • Copy the Value returned for CTRKinesisARN

• Bastion Host Public IP

  • Copy the Value returned by BastionHostPubIP

• eMite Server Private IP

  • Copy the Value returned by AppEmiteIP

eMite Cloud Operations team will use these details to do the installation/configuration of the eMite App.

Troubleshooting common issues with Cloudformation Stack Deployment

• “Create stack” or a “Create Resource” operation fails in CloudFormation.

  • Please check that your AWS user/role has the appropriate permissions to “Create Stack”.

    • The necessary permissions are detailed in the section “Prerequisites and Requirements”

    • Contact your AWS account admin for permissions, or AWS Support if you continue to have issues

• Contact eMite Cloud Operations Team (cloudops@prophecyinternational.com) if other issues arise during cloudformation deployment

  • eMite team can guide you during the cloudformation deployment over a zoom / Teams meeting session

eMite Installation

Once the Cloud Operations team receives the output data we will then schedule a remote session for eMite installation.
Once the eMite installation is complete the eMite dashboard should start to show data from Amazon Connect.
Additional Network Checklist/QA will be done by eMite before final handoff to the customer.

Expected Output using one of the out-of-the-box dashboards:

Monitoring

AWS Resources

We advise the customer to use Amazon Cloudwatch, or your preferred monitoring software to monitor the utilization of the EC2 instances, Opensearch and Kinesis Streams
The recommended metrics to monitor are the following:

• EC2 instance

  • CPU Utilization

    • Recommended Threshold: 80%

    • If approaching threshold, consider increasing EC2 instance type to add more CPU

  • These other recommended metrics may require custom AWS metrics

    • EBS Storage Utilization

      • Recommended Threshold: 80%

      • If approaching threshold, consider increasing EBS storage size to add more storage

    • Memory Utilization

      • Recommended Threshold: 80%

      • If approaching threshold, consider increasing EC2 instance type to add more Memory

• Opensearch Domain

  • Cluster Status

    • Must always be Green

      • A Red cluster state indicates there’s an issue with the Elasticsearch indices

      • This is typically caused by a corrupted index

    • Please file a support ticket with eMite support team, detailed in the next section “Contacting eMite Support”

  • Cluster Write Status

    • Must always be Green

      • A Red cluster state indicates there’s an issue with writing data

      • This is typically caused by storage getting full, or corrupted index

    • Please increase the storage allocated to the cluster

  • Snapshot Failure Status

    • Must always be Green

      • A Red cluster state indicates there’s an issue with backups

      • This is typically caused by storage getting full, or corrupted index

    • Please increase the storage allocated to the cluster

  • Total Free Space

    • Recommended threshold: 20%

    • If approaching this threshold, consider increasing storage allocation on the cluster

  • Search Latency

    • Recommended threshold: 10,000ms

      • Increased search latency means that the data within the eMite datablocks take longer to get loaded

      • If approaching this threshold, consider increasing the Data Node instance type for more CPU

  • Data Node CPU Utilization

    • Recommended Threshold: 80%

    • If approaching this threshold, consider increasing the Data Node instance type for more CPU

  • Data Node JVM Memory Pressure

    • It is normal for the utilization to reach 80% and then go back down in a sawtooth-like pattern

    • Critical Threshold: above 80%

      • If the utilization ever goes above 80%, consider increasing the Data Node instance type for more memory

  • JVM Thread Pool Search Queue

    • Recommended threshold: No Queue Length above maximum for 10 minutes consecutively

    • If this max threshold is met frequently, it would mean that Opensearch Domain does not have enough Search Threads allocated to handle incoming search requests - hence the queue becomes longer

      • Longer Queue = Delay in data being loaded onto the eMite datablocks

• Kinesis

  • ReadProvisionedThroughputExceeded and WriteProvisionedThroughputExceeded

    • If any of these provisioned throughput are exceeded, it means that the Kinesis Stream shards GET / PUT limits are being hit - as each shard in the stream can only handle a fixed amount of Get / Put requests per second

      • If so, consider switching to On-Demand stream mode, or adding more provisioned shards

eMite Application Health

We advise the customer to use Amazon Cloudwatch, or your preferred monitoring software to monitor the following for eMite application health:

• eMite Dashboard

  • Monitor the eMite URL (https://<emite access domain>/emite/dashboard)

  • You can use AWS Canary for this

  • If the site is inaccessible, it may mean the EC2 instance is down, or one of the below critical services is not running

• eMite Critical Services

  • These services must always be running

  • Monitoring these using AWS Cloudwatch may require custom metrics

    • World Wide Web Service

    • eMite Scheduler Service

    • SQL Server

  • If any of these processes are stopped, please restart the services

Basic Troubleshooting

All of the required testing/troubleshooting for this deployment will be done by eMite as part of the deployment activity.
eMite team will handover the environment to the customer only after testing has been completed successfully.
eMite also has a support team that can help resolve issues after deployment.

Site Access Troubleshooting

• Issue

  • Dashboards/site is not accessible

• Check that critical services are running

  • World Wide Web Publishing Service

    •  

• Check the Application Load Balancer Security Group

  • Ensure that inbound rule TCP 443 port allows traffic from the end-user IP’s / IP Ranges, or it allows open traffic (0.0.0.0/0)

• If service is running, and the Security Group inbound rules are correctly defined, but the issue still persists

  • Please file a support ticket with eMite support team, detailed in the next section “Contacting eMite Support”

General Adapter Troubleshooting

• Issue

  • Realtime AgentEvent data is not being displayed on the dashboard

  • No new CTR or AgentEvents adapter data is coming in for the past few hours

• Check that critical services are running

  • World Wide Web Publishing Service

  • eMite Services

    • eMite Scheduler Service

    • eMite Metric Service

    • eMite License Manager

• If all services are running, please try restarting the adapters:

  • Restart adapter through eMite schema page.

    1. Go to the schema page https://au-cloud.emite.com/emite/"customername"/schema

    2. Login as admin account

    3. Then Select > Adapters >

       •  

    4. To restart an adapter you just need to ‘Right Click then select Cancel'

       •  

       • This notification will show as an indication that the adapter has been stopped.

    5. Then ‘Right Click then select Run’

       •  

  • Alternative: Restart through Windows Services

    1. Open Services > Look for “eMite Scheduler Service (customer name)

       •  

    2. In order to reboot the adapters select the STOP then START the service.

       •  

• If issue still persists even after restarting adapters,

  • Please file a support ticket with eMite support team, detailed in the next section “Contacting eMite Support”

  • In the support ticket, include any instances of the “ERROR|” keyword on the adapter logs

    • To access the logs, go to C:\Program Files (x86)\eMite\”customername”\logs\Adapters

For all other issues (i.e. data mismatch), you can directly file a support ticket with eMite support team, detailed in the next section “Contacting eMite Support”

Contacting eMite Support

This section details the process for filing a support ticket with the support team
Referenced from: eMite Support Handbook - eMite - Confluence (atlassian.net)

As an example, to report an issue:

1. Log into the eMite Support Portal. https://jsd.prophecyinternational.com/servicedesk/customer/portal/2

• Select Report an Issue.

• Fill out all mandatory fields.

1. Click Create.

2. Track and manage your request going forward using the tools available within the portal:

Support Request Priority

Referenced from:
Please refer to this for the latest information

Service-Level Agreement (SLA) defines the level of service expected by a customer from our support.
eMite will make the eMite Service available 24 hours a day, 7 days a week, and use commercially reasonable best efforts to provide 100% uptime, except for the following “Uptime Exclusions”:
(i) occasional planned downtime at non-peak hours (for which we will provide advance notice); or (ii) any unavailability caused by circumstances beyond our reasonable control, including failure or delay of customer Internet connection, misconfiguration by customer or any third party acting on customer’s behalf, issues on customer network, or telecommunications services contracted directly by customer.

Note: There are currently no support tiers with corresponding SLA’s Target Response Times.
Special/Specific SLA’s and Target Response Times will be discussed as part of contract/agreement with the customer as required.

Support Request Priority is automatically calculated using customer provided inputs (Impact and Urgency) based on the Priority Matrix (shown below). Other considerations may also affect the assigned priority - including size, scope, complexity, and resources required for the resolution to be carried out.

Priority Matrix

eMite support requests have 5 levels of priority, as defined below:

• P1 - Highest: Critical issue that needs to be fixed immediately

• P2 - High: Issue needs to be fixed with high urgency

• P3 - Medium: There is an issue, but service is still running

• P4 - Low: Minor issue or easily worked around

• P5 - Lowest: Trivial issue with little or no impact on progress

Priority is automatically calculated based on the following priority matrix:

Impact

Impact is a measure of the extent of the incident and of the potential damage caused by the incident before it can be resolved. Customers are asked to communicate the impact of a support request when submitting it via the Support Portal.

Urgency

Urgency is how quickly a resolution of the incident is required. Customers are asked to communicate the urgency of a support request when submitting it via the Support Portal.

Target Response Times

eMite Software Upgrades

Please note that eMite does not automatically upgrade the application or adapters.
The upgrade process would be initiated by the eMite team, and will involve reaching out to the customer first for a discussion on the new version, as well as coordinating a schedule to perform the upgrade.

As with the initial eMite installation, Cloud Operations team will upgrade the eMite application and/or adapters over a remote session to the eMite APP server.
Once the eMite upgrade is complete, the eMite dashboard should start to show data from Amazon Connect again.
Additional Network Checklist/QA will be done by eMite before handing off to the customer.

eMite License Renewal

For expired/expiring eMite product license, please reach out to your assigned eMite sales/account manager, or contact support (support@prophecyinternational.com).
Once your license is renewed, instructions would then be sent to you on how to apply the new updated license to eMite.

Costs

Here is a sample calculation of AWS resources costs with the instances listed for the standard “< 1000 agents” tier listed in the Sizing section of this guide.
The service in the table are the minimum requirement for this deployment.
Notes:

• SNS usage is for eMite billing notifications only, and would not exceed free tier limits

• The EC2 bastion host instance can be stopped if not in use to save monthly costs.

• Data transfer are not included as these vary depending on the concurrent dashboard users, and the level of activity of the Amazon Connect instance

• Pricing may vary slightly depending on region of deployment

  • For the below sample computation, we’ll be using us-east-1

  • Please consult AWS Pricing for the latest information

With regards to the the eMite license costs for your deployment

• Costs will depend on the total connected-minutes per month of your Amazon Connect instance.

• For an estimate or exact pricing, please contact your assigned Account Manager, or eMite sales team (sales@prophecyinternational.com)

The prices may vary depending on the following:

• Number of Agents

  • Please see general sizing guide in the next section, as this would affect the instance type of the EC2’s and Elasticsearch, as well as the recommended storage size

Sizing

The following table outlines recommendations for Amazon EC2, Opensearch Domain instance type and EBS Storage.
If your Amazon Connect instance has more than 7000 Agents, please contact eMite Cloud Operations Team for additional sizing guidance.
Please refer to the Costs section above for a pricing example.

Backups

Opensearch Domain

AWS automatically creates a special repository for the Opensearch Domain.
This repository contains automated hourly snapshots, that can be used to recover index data at that point in time.
These snapshots are retained for up to 2 weeks.

EC2 - Creating Backup with AWS Data Lifecycle Manager

AWS Data Lifecycle manager service is an automatic backup for the EBS to lessen the burden of System Admin on managing their backup systems.
The main use of this service is to automate the creation of backups, retention, and deletion of EBS backups.

Procedure on how to setup a Data Lifecycle Manager (DLM)Policy:

• Search “Lifecycle Manager” > Select Create Lifecycle Policy >

• Select Policy Type > EBS-Backed AMI Policy

• This option provides a quicker way to restore or replace a failed EC2 instance.

• Once done, fill up the needed details. It will create an IAM role if the role does not exist.

  • You will need to specify the Target Resource - in this case it will only trigger on instances that have “emite-instance” as their name based on the tags.

• Then Select Next if done with the settings.

  • In this part, you provide the schedule of when the backup triggers and how many AMI will be retained.

    • We recommend at least a daily backup, with 3 AMI’s retention

  • You also have an option on how many AMI will be deprecated must be less than the AMI you need to retain. (Only if enabled)

• Then select Review Policy so that you can review your final backup setup.

• If you are satisfied with this backup setup you can now select the Create Policy.

Recovery

Opensearch Domain

To restore the Opensearch Domain via an automated snapshot:

• RDP to the eMIte server using an account with local administrator privileges

• Stop the eMite services on the EC2 instance

• Open a powershell session as administrator

• Enable TLS 1.2

  • [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12

• Get the list of available snapshots

  • You can get the Opensearch Domain Endpoint from the Opensearch Console on AWS

  • (Invoke-webrequest <Opensearch Domain Endpoint>/_snapshot/cs-automated-enc/*).Content | ConvertFrom-Json|ConvertTo-Json

  • This should return a list of snapshots

    • We typically restore from the latest available snapshot

    • This is the last entry that appears in the long list output

    • Example:

• Delete all indices on elasticsearch

  • (Invoke-webrequest <Opensearch Domain Endpoint>/* -Method DELETE).Content | ConvertFrom-Json | ConvertTo-Json

  • Wait for the deletion to finish

• Restore snapshot

  • Copy the name of the snapshot to restore to

    • In the example above, this would be 2022-08-26t06-17-59.466c2be5-50c8-4b43-86ed-2386f79d5c8c

  • Run the following to start the restoration process

  • This should return a response of “Acknowledged: true” - meaning the restore operation has begun

• You can view the recovery progress using this command

  • It will show the recovery status of each shard

  • Recovery will be complete when all shards are 100% restored.

  • For example:

    •  

• After recovery, the Amazon Connect data should be visible on the eMite site/dashboards after a browser refresh

EC2 - Recovery from AMI

Once you are done creating a backup for the eMite instance. You can now restore it in case of failure.

• Procedure on how to restore from AMI:

• Go to EC2 Service > Select “AMI” under images > highlight the image you want to restore > then “select launch instance from this image”

• The following steps will be the same as how to launch an instance.

  • Please ensure to Select the same security group from the failed instance.

  • Once the instance Is deployed. You just need to add it to the Application Load Balancer (ALB)

  • Go to EC2 Service > Select “Target Groups” under images > Select the “emite-tg” >

• You can now register the new EC2 instance that you’ve restored.

• After registering, you should be able to view the eMite site again after a few minutes.

Answers to common Security Concerns for Customer-Hosted Deployment

IAM Role and Policies

For Deployment

• eMite only provides the cloudformation template to the customer.

  • Customers AWS Cloud Administrators typically deploys this template - but we can also provide some guidance if requested.

• The region of deployment must be in the same region where the Amazon Connect is deployed.

• Root privilege is not required to deploy the cloudformation template.

The AWS user or role that would be used for deployment must have the minimum following permissions in policy attached to said user or role.
This user or role would be typically used by an AWS Cloud Administrator.

• Cloudformation - deployment / potential rollback of cloudformation stack

• VPC - creation of vpc components, including subnets, internet and nat gateways, routing tables

• EC2 and EBS - creation of EC2 and EBS instances, as well as security groups

• ELB - creation of application load balancer, and target group, rule

• Opensearch - creation of opensearch domain

• SNS - creation of SNS topic and subscription

• Kinesis Data Streams - creation of Kinesis Data Stream

• IAM - Creation of IAM Role/Instance Profile (for role attachment to EC2) and the Policies required by the role

For Management after deployment

The AWS user or role that would be used for managing the resources after deployment must have the minimum following permissions in policy attached to said user or role.
This user or role would be typically used by an AWS Cloud Administrator, or a Systems Operations Engineer.

• CloudWatch - for viewing metrics recommended for monitoring

• Cloudformation - for viewing and updating stacks (e.g. if need to increase EC2 instance resources)

• VPC - view created vpc, subnets, nat and internet gateways

• EC2 and EBS - view and modify EC2’s and EBS’s created

• ELB - View created application load balancer

• Opensearch - View the opensearch domain, and includes permissions for upgrade, in case a version upgrade may need to be performed for future eMite application versions

• SNS - view the SNS topic and subscription created

• Kinesis Data Streams - view the kinesis stream

• IAM - for viewing the created role/policies

Details on Role and Policies created by Cloudformation for eMite App use

• The deployment only creates 1 IAM role with policies attached to it.

  • eMite uses the IAM role to access the needed AWS resources.

• During Cloudformation deployment the following policies are created and attached to the IAM Role

  • IamPolicyForAmazonConnectAccess

    • This policy basically allows eMite to read data from the specific AmazonConnect instance

      • AmazonConnectInstanceArn is the ARN of the Amazon Connect Instance that eMite will be integrating with

      • The DenyFederationTokens section denies the GetFederationTokens action, as it is not needed

  • IamPolicyKinesisAccess

    • This policy basically allows eMite to read data from the specific AgentEvents and CTR Kinesis streams

      • AgentsEventStreamName - name of the AgentEvents Kinesis Stream

      • CtrStreamName - name of the CTR Kinesis Stream

• IamPolicyForSns

  • This policy basically allows eMite to publish data to an SNS topic for billing

Encryption Details

• Data Encryption for Opensearch

  • Customer data from Amazon Connect and Kinesis streams is stored in the Opensearch domain.

  • The cloudformation deployment configures the Opensearch storage to be encrypted with AWS-managed keys.

    • AWS KMS automatically rotates AWS managed keys every year (approximately 365 days). Customer cannot enable or disable key rotation for AWS managed keys.

  • More information:

    • OpenSearch Service domains offer encryption of data at rest, a security feature that helps prevent unauthorized access to your data.

• Data Encryption for Kinesis Streams

  • The cloudformation deployment configures the Kinesis Streams to be encrypted with AWS-managed keys.

    • AWS KMS automatically rotates AWS managed keys every year (approximately 365 days). Customer cannot enable or disable key rotation for AWS managed keys.

  • More information:

    • Server-side encryption is a feature in Amazon Kinesis Data Streams that automatically encrypts data before it's at rest.

    • Data is encrypted before it's written to the Kinesis stream storage layer, and decrypted after it’s retrieved from storage. As a result, your data is encrypted at rest within the Kinesis Data Streams service. This allows you to meet strict regulatory requirements and enhance the security of your data.

• Data Encryption for EBS

  • The cloudformation deployment configures the EBS attached to the eMite application server and bastion host to be encrypted with AWS-managed keys.

    • AWS KMS automatically rotates AWS managed keys every year (approximately 365 days). Customer cannot enable or disable key rotation for AWS managed keys.

  • More information:

    • Use Amazon EBS encryption as a straight-forward encryption solution for your EBS resources associated with your EC2 instances.

    • With Amazon EBS encryption, you aren't required to build, maintain, and secure your own key management infrastructure. Amazon EBS encryption uses AWS KMS keys when creating encrypted volumes and snapshots.

• Other Encryption and Keys Handling Concerns:

  • The EC2 KeyPair will be handled by the customer, it is used to deploy the EC2 instances.

    • KeyPairs list can be viewed on the EC2 Console, in the region where the EC2 instances were deployed by cloudformation.

  • Customer end-user credentials for eMite accounts are stored in the eMite server’s local database, and are hashed/encrypted.

  • For the eMite dashboard access, the customer must provide an ACM certificate in order to enable the encryption-in-transit via HTTPS.

OS Security Patching

• It is the customer’s system team responsibility to maintain the Windows OS security patches to the EC2 Instances.

  • We advise patching the system at least on a regular basis, depending on the customer’s internal security policies.

• OS Patches would not impact the eMite application on the eMite App server EC2.

  • Although please note that Windows OS patches typically require a restart - so we recommend patching during off-peak hours or outside the end-user hours.

Public Accessibility

• By default, the eMite dashboard is accessible publicly over HTTPS

• However, the customer can limit access to the eMite site by modifying the Application Load Balancer Security Group configuration

  • Inbound rule for HTTPS can be modified to only allow traffic from select IP’s or IP ranges

• Customers may opt to use internal load balancer or not to use load balancer at all

  • However this will require further discussion with eMite, as it will require a custom modification of the cloudformation template and potential changes to the architecture.

Disabling IMDSv1

• The eMite application will need access to the EC2 metadata to get security tokens for accessing the AWS resources.

  • The eMite application can work with either IMDSv1 or IMDSv2, so we do recommend disabling IMDSv1 on the EC2

• To disable IMDSv1 on the EC2 instances:

  • Login to the AWS Console

    • We recommend to use an AWS Administrator account, or a user with modification permissions to the EC2’s

  • Go to the region where the EC2 instances were deployed by cloudformation

  • Get the Instance ID’s of the eMite App Server, and the Bastion Host

  • Open a CloudShell session

  • Execute the following

• To further enforce IMDSv2 only - we recommend that you add the following policy for AWS Administrators / System Operations Engineer accounts to prevent running EC2 instances that don’t have IMDSv1 disabled:

• More information here:

  • https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ExamplePolicies_EC2.html#iam-example-instance-metadata