Getting started

Organization access

The Dormeto manager reads your load balancers and installs its routers through two roles it assumes in each account. It lists your accounts through a delegation policy of your management account. Dormeto never creates them itself. You create them, with Terraform, the AWS CLI or a StackSet of yours. The manager checks them at each discovery run.

17 min readManager 0.21.0

Management accountdelegation policy

Member accountDmtDiscoveryRoleDmtCommandRole

Member accountDmtDiscoveryRoleDmtCommandRole

checked at every run

What it needs

The roles trust only the manager's roles, by name in the manager's account. These are its discovery role (dmt-<env>-manager-discovery) and its operations role (dmt-<env>-manager-operations). The discovery role is trusted only from within your organization when you give the module organization_id. They can be created before the manager, and keep working when it is reinstalled. Several managers (production and staging, say) share an account's roles: list every environment in envs. The manager's own account gets its roles from the manager's module. Other names than DmtDiscoveryRole and DmtCommandRole are set in the manager's Terraform (discovery_role_name, command_role_name) and in the access module. The manager never assumes another role, an administrator role included, unless you name it there. How the manager uses the command role, and how to bring your own: How the manager installs routers.

With Terraform

Each release publishes two modules beside the manager's. Replace 111122223333 with the manager's account, prod with its environment (or list several) and o-exampleorgid with your organization.

The management account

module "dormeto_organization" {
  source             = "https://releases.dormeto.com/manager/0.21.0/dormeto-organization-0.21.0.zip"
  manager_account_id = "111122223333"
  envs               = ["prod"]
  # additional_statements = [ … the policy's other statements … ]
}

Each member account

One provider and one module instance per account, with administrator rights there (for example through a role you already use to administer it). Two accounts, shop and data:

provider "aws" {
  alias  = "shop"
  region = "us-east-1"
  assume_role {
    role_arn = "arn:aws:iam::222233334444:role/YourAdministrationRole"
  }
}

provider "aws" {
  alias  = "data"
  region = "us-east-1"
  assume_role {
    role_arn = "arn:aws:iam::555566667777:role/YourAdministrationRole"
  }
}

module "dormeto_access_shop" {
  source             = "https://releases.dormeto.com/manager/0.21.0/dormeto-access-0.21.0.zip"
  providers          = { aws = aws.shop }
  manager_account_id = "111122223333"
  envs               = ["prod"]
  organization_id    = "o-exampleorgid"
}

module "dormeto_access_data" {
  source              = "https://releases.dormeto.com/manager/0.21.0/dormeto-access-0.21.0.zip"
  providers           = { aws = aws.data }
  manager_account_id  = "111122223333"
  envs                = ["prod"]
  organization_id     = "o-exampleorgid"
  create_command_role = false # discovered, never napped
}

For an account where workloads are discovered but never napped, set create_command_role = false, as for data above. Its discovery role then says so. The manager counts it as discovery only rather than missing the command role.

With the AWS CLI

Each release also publishes the same as a script. Set ACCOUNT, MANAGER_ACCOUNT, ENVS (the manager's environments, separated by spaces) and ORGANIZATION at its top or in the environment. Then run it with administrator credentials of the account. It checks them, then creates the roles, or updates them when run again. Role names may carry a path (DISCOVERY_ROLE, COMMAND_ROLE). An empty COMMAND_ROLE sets up an account discovered but never napped.

curl -O https://releases.dormeto.com/manager/0.21.0/dormeto-access-commands-0.21.0.sh
curl -O https://releases.dormeto.com/manager/0.21.0/SHA256SUMS
shasum -a 256 -c --ignore-missing SHA256SUMS
ACCOUNT=222233334444 MANAGER_ACCOUNT=111122223333 ENVS=prod ORGANIZATION=o-exampleorgid bash dormeto-access-commands-0.21.0.sh

The delegation policy, in the management account. Replace the manager's account and environment in the policy's principal (arn:aws:iam::MANAGER_ACCOUNT:role/dmt-ENV-manager-discovery, one per environment). Then merge it with the organization's current policy, if any. The organization has one policy: putting it replaces the other statements. With jq:

curl -O https://releases.dormeto.com/manager/0.21.0/dormeto-organization-policy.json
aws organizations describe-resource-policy --query ResourcePolicy.Content --output text > current.json \
  || echo '{"Statement":[]}' > current.json
jq -s '{Version: "2012-10-17", Statement: ([.[0].Statement[] | select(.Sid != "DormetoReadOrganization")] + .[1].Statement)}' \
  current.json dormeto-organization-policy.json > merged.json
aws organizations put-resource-policy --content file://merged.json

How the manager installs routers

The manager holds no credentials for your accounts. To install, upgrade or remove a router, it assumes one role in the router's account: the command role (DmtCommandRole by default). It acts through that role by AWS API calls. Each call is in that account's CloudTrail, made by assumed-role/DmtCommandRole/dmt-<env>-operations.

From your request to the router

1. You ask, in the dashboard or with the CLI: install, upgrade or uninstall a router. Or onboard a workload in an account and region without a router. The onboarding installs it first. With event-driven refresh, a discovery run also connects a new account the same way: a forwarder role, and a rule per region.
2. The manager's API records the operation and starts the Step Functions workflow dmt-<env>-manager-operations, in the manager's account.
3. Each step runs in the Lambda function dmt-<env>-manager-operations, as the role of the same name. That role may assume one role name only, the command role's, and only in the accounts of your organization (aws:ResourceOrgID).
4. The command role: the step calls sts:AssumeRole on arn:aws:iam::<account>:role/DmtCommandRole, session dmt-<env>-operations, for one hour. The role's trust policy accepts that role of the manager's account only, by name.
5. The router's resources: with that session, the step creates the router from its blueprint, in its own namespace. First come the roles dmtr-<env>-<region>-controller, -waker, -scheduler and -onboarder under /dormeto/router/. Each carries the permissions boundary DmtRouterBoundary. Then come the functions, table, queues, log groups and schedule dmtr-<env>-*. Then the routers' access entries in the region's EKS clusters. For private clusters, their proxy functions in the clusters' VPCs.
6. The router then runs as its own roles, in your account. Status, wakes, naps and pauses do not use the command role. The manager's API role (dmt-<env>-manager-api) calls the router's functions, whose resource policies allow it.

At each discovery run, the access check assumes the command role in each account for 15 minutes. It only reports whether it could. The check is dmt-<env>-manager-access-check, under the operations role. Discovery lists the accounts where it could not, and onboarding there is refused with the reason.

The default role

DmtCommandRole, at the root path, in every account where workloads nap: one name for the whole organization. The access module and the AWS CLI script create it, with its policy and the DmtRouterBoundary policy (above). The manager's own account gets them from the manager's module.

Another name or path

Name it as your rules want, a path included, the same in every account. Give that name to the manager and to what creates the role:

module "dormeto" {
  source            = "https://releases.dormeto.com/manager/0.21.0/terraform-aws-dormeto-0.21.0.zip//modules/manager"
  # … your other inputs
  command_role_name = "platform/DormetoRouters"
}

module "dormeto_access_shop" {
  source            = "https://releases.dormeto.com/manager/0.21.0/dormeto-access-0.21.0.zip"
  # … as above
  command_role_name = "platform/DormetoRouters"
}

With the AWS CLI script, COMMAND_ROLE=platform/DormetoRouters. The manager's operations role may then assume arn:aws:iam::*:role/platform/DormetoRouters only. Create the role in every account first, then apply the manager with its name. Until then, Discovery lists the accounts where it cannot assume it.

Your own role

You may create the command role with your own tooling. The manager checks only that it can assume it. Each step then needs these:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": "sts:AssumeRole",
    "Principal": { "AWS": "arn:aws:iam::111122223333:root" },
    "Condition": {
      "ArnLike": { "aws:PrincipalArn": ["arn:aws:iam::111122223333:role/dmt-prod-manager-operations"] }
    }
  }]
}

• You may add tags, a description and a condition on aws:PrincipalOrgID. You may also add a permissions boundary of your own on the command role itself, if your rules want one on every role. It must allow every action of the policy.
• Keep the boundary's name, the namespace and every statement. A step that lacks a permission fails, with AWS's error, in the Operations view.
• A service control policy that requires its own boundary on every new role must accept DmtRouterBoundary on the roles under /dormeto/router/. A role carries one boundary.

The two samples below create the command role, the boundary and the discovery role, as the access module does. The discovery role keeps its name, DmtDiscoveryRole: the command policy names it for the clusters' access entries.

Your own role with Terraform

The release's policies, unchanged, in a module of yours. Download the archive, check it and unzip it beside your code:

curl -O https://releases.dormeto.com/manager/0.21.0/dormeto-access-0.21.0.zip
curl -O https://releases.dormeto.com/manager/0.21.0/SHA256SUMS
shasum -a 256 -c --ignore-missing SHA256SUMS
unzip dormeto-access-0.21.0.zip -d dormeto-access

dormeto-roles/main.tf:

terraform {
  required_providers {
    aws = { source = "hashicorp/aws" }
  }
}

variable "manager_account_id" { type = string }
variable "envs" { type = list(string) }
variable "organization_id" { type = string }

data "aws_caller_identity" "current" {}
data "aws_partition" "current" {}

locals {
  files   = "${path.module}/../dormeto-access" # dormeto-access-0.21.0.zip, unzipped
  vars    = { account = data.aws_caller_identity.current.account_id, partition = data.aws_partition.current.partition }
  manager = "arn:${local.vars.partition}:iam::${var.manager_account_id}"
  # Each role trusts one role of the manager, by name in its account, one per environment.
  trust = { for role in ["operations", "discovery"] : role => jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Action    = "sts:AssumeRole"
      Principal = { AWS = "${local.manager}:root" }
      Condition = {
        ArnLike      = { "aws:PrincipalArn" = [for env in var.envs : "${local.manager}:role/dmt-${env}-manager-${role}"] }
        StringEquals = { "aws:PrincipalOrgID" = var.organization_id }
      }
    }]
  }) }
}

# Your name and path: give the same to the manager (command_role_name).
resource "aws_iam_role" "command" {
  name                 = "DormetoRouters"
  path                 = "/platform/"
  max_session_duration = 3600
  assume_role_policy   = local.trust["operations"]
  # permissions_boundary = "arn:aws:iam::222233334444:policy/YourBoundary"
}

resource "aws_iam_role_policy" "command" {
  name   = "command"
  role   = aws_iam_role.command.id
  policy = templatefile("${local.files}/command-role/command-policy.json", local.vars)
}

# This name, at the root: the manager gives it to every role it creates for a router.
resource "aws_iam_policy" "boundary" {
  name        = "DmtRouterBoundary"
  description = "Permissions boundary of every role Dormeto creates for a router."
  policy      = templatefile("${local.files}/router-boundary.json", local.vars)
}

resource "aws_iam_role" "discovery" {
  name                 = "DmtDiscoveryRole"
  max_session_duration = 3600
  assume_role_policy   = local.trust["discovery"]
  tags                 = { "dormeto:component" = "discovery" } # it reads its own role's tags by this one
}

resource "aws_iam_role_policy" "discovery" {
  name   = "discovery"
  role   = aws_iam_role.discovery.id
  policy = file("${local.files}/discovery-policy.json")
}

One instance per account, with a provider per account as above, then command_role_name = "platform/DormetoRouters" in the manager's module:

module "dormeto_roles_shop" {
  source             = "./dormeto-roles"
  providers          = { aws = aws.shop }
  manager_account_id = "111122223333"
  envs               = ["prod"]
  organization_id    = "o-exampleorgid"
}

At a new release, unzip its archive in place of the previous one and plan. Its Releases entry says whether the roles change.

Your own role with a CloudFormation StackSet

If your organization already deploys its roles with StackSets, the same roles fit in one stack. It reaches every new account of an organizational unit. Dormeto never creates a stack itself. You build the template from the release's policies, so it follows each release. With jq, where you unzipped the archive:

MANAGER_ACCOUNT=111122223333 ENVS="prod" ORGANIZATION=o-exampleorgid COMMAND_ROLE=platform/DormetoRouters
jq -n --arg manager "$MANAGER_ACCOUNT" --arg envs "$ENVS" --arg org "$ORGANIZATION" --arg command "$COMMAND_ROLE" \
  --slurpfile commandPolicy dormeto-access/command-role/command-policy.json \
  --slurpfile boundary dormeto-access/router-boundary.json \
  --slurpfile discovery dormeto-access/discovery-policy.json '
  def name($full): ($full | split("/")) as $p
    | {RoleName: $p[-1], Path: (if ($p | length) > 1 then "/\($p[:-1] | join("/"))/" else "/" end)};
  def trust($role): {Version: "2012-10-17", Statement: [{Effect: "Allow", Action: "sts:AssumeRole",
    Principal: {AWS: "arn:${partition}:iam::\($manager):root"},
    Condition: {ArnLike: {"aws:PrincipalArn": [$envs | splits(" +") | select(. != "")
        | "arn:${partition}:iam::\($manager):role/dmt-\(.)-manager-\($role)"]},
      StringEquals: {"aws:PrincipalOrgID": $org}}}]};
  {AWSTemplateFormatVersion: "2010-09-09", Description: "Dormeto organization roles", Resources: {
    CommandRole: {Type: "AWS::IAM::Role", Properties: (name($command) + {MaxSessionDuration: 3600,
      AssumeRolePolicyDocument: trust("operations"), Policies: [{PolicyName: "command", PolicyDocument: $commandPolicy[0]}]})},
    RouterBoundary: {Type: "AWS::IAM::ManagedPolicy", Properties: {ManagedPolicyName: "DmtRouterBoundary",
      Description: "Permissions boundary of every role Dormeto creates for a router.", PolicyDocument: $boundary[0]}},
    DiscoveryRole: {Type: "AWS::IAM::Role", Properties: {RoleName: "DmtDiscoveryRole", MaxSessionDuration: 3600,
      AssumeRolePolicyDocument: trust("discovery"), Policies: [{PolicyName: "discovery", PolicyDocument: $discovery[0]}],
      Tags: [{Key: "dormeto:component", Value: "discovery"}]}}}}
  | walk(if type == "string" and test("\\$\\{(account|partition)\\}")
      then {"Fn::Sub": (gsub("\\$\\{account\\}"; "${AWS::AccountId}") | gsub("\\$\\{partition\\}"; "${AWS::Partition}"))}
      else . end)' > dormeto-roles.json

Then, from the management account or a delegated administrator for StackSets (add --call-as DELEGATED_ADMIN), with trusted access for StackSets on (aws cloudformation activate-organizations-access):

aws cloudformation create-stack-set --stack-set-name dormeto-roles \
  --template-body file://dormeto-roles.json --capabilities CAPABILITY_NAMED_IAM \
  --permission-model SERVICE_MANAGED --auto-deployment Enabled=true,RetainStacksOnAccountRemoval=false
aws cloudformation create-stack-instances --stack-set-name dormeto-roles --regions us-east-1 \
  --deployment-targets OrganizationalUnitIds=ou-exmp-12345678,Accounts=111122223333,AccountFilterType=DIFFERENCE

• One region: IAM is global. A second region would create the same roles again, and fail.
• Without the manager's account (AccountFilterType=DIFFERENCE): the manager's module creates its roles. A StackSet never reaches the management account either, which needs only the delegation policy.
• A new account in the organizational unit gets the roles, and the next discovery run finds it.
• A new release: build the template again from its archive, then aws cloudformation update-stack-set --stack-set-name dormeto-roles --template-body file://dormeto-roles.json --capabilities CAPABILITY_NAMED_IAM.
• Remove: uninstall the routers first, then delete-stack-instances with the same targets and --no-retain-stacks, and delete-stack-set.

Kubernetes (EKS)

Dormeto naps the Deployments and StatefulSets of an EKS cluster behind an ALB of the AWS Load Balancer Controller. The routers call the cluster's API with their own roles, through EKS access entries: nothing runs in the cluster. The cluster's API endpoint must be public and open to every address (0.0.0.0/0), or its private access must be on. Then the routers reach it through a small proxy function in its VPC (Private clusters).

• Access entries: the routers' roles (dmtr-<env>-<region>-controller, -waker, -onboarder, under /dormeto/router/) in the Kubernetes group dormeto:router, and DmtDiscoveryRole with the read-only AmazonEKSViewPolicy (no Secrets). The manager makes them through DmtCommandRole when it installs or upgrades a router. It does so in every cluster of the router's account and region whose authentication mode is API or API_AND_CONFIG_MAP. Removing the router deletes its roles' entries.
• The rights of dormeto:router, published with each release. A ClusterRole holds exactly what the routers check: read Deployments, StatefulSets, ReplicaSets, Pods, Services, Events, Ingresses, HorizontalPodAutoscalers and KEDA ScaledObjects. Read Nodes. Scale Deployments and StatefulSets. Annotate ScaledObjects. Create, change and delete Ingresses. Its binding comes with it, and an admission policy. That policy refuses the group's creation and deletion of an Ingress without the dormeto.com/managed label. It also refuses the group's changes adding or removing that label. The policy needs Kubernetes 1.30 or later. Before, the group may create and delete any Ingress. Apply it once per cluster, with an administrator's access:

curl -O https://releases.dormeto.com/manager/0.21.0/dormeto-kubernetes-0.21.0.yaml
curl -O https://releases.dormeto.com/manager/0.21.0/SHA256SUMS
shasum -a 256 -c --ignore-missing SHA256SUMS
kubectl apply -f dormeto-kubernetes-0.21.0.yaml

To write the access entries in your own Terraform instead (account 222233334444, environment prod, region us-east-1):

resource "aws_eks_access_entry" "dormeto_router" {
  for_each          = toset(["controller", "waker", "onboarder"])
  cluster_name      = "my-cluster"
  principal_arn     = "arn:aws:iam::222233334444:role/dormeto/router/dmtr-prod-us-east-1-${each.key}"
  kubernetes_groups = ["dormeto:router"]
}

resource "aws_eks_access_entry" "dormeto_discovery" {
  cluster_name  = "my-cluster"
  principal_arn = "arn:aws:iam::222233334444:role/DmtDiscoveryRole"
}

resource "aws_eks_access_policy_association" "dormeto_discovery" {
  cluster_name  = "my-cluster"
  principal_arn = aws_eks_access_entry.dormeto_discovery.principal_arn
  policy_arn    = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSViewPolicy"
  access_scope {
    type = "cluster"
  }
}

A cluster on the aws-auth ConfigMap only (authentication mode CONFIG_MAP) has no access entries. Switch it to API_AND_CONFIG_MAP (your call: the switch cannot be undone, and aws-auth keeps working). Then upgrade the router or onboard a workload, and the manager makes the entries. Or map the roles in aws-auth yourself, which matches a role by its ARN without its path. Map the routers' roles in dormeto:router. Map DmtDiscoveryRole in a group bound to the read-only ClusterRole view (kubectl create clusterrolebinding dormeto-discovery --clusterrole=view --group=dormeto:discovery):

mapRoles: |
  - rolearn: arn:aws:iam::222233334444:role/dmtr-prod-us-east-1-controller
    username: dormeto-controller
    groups: [dormeto:router]
  - rolearn: arn:aws:iam::222233334444:role/dmtr-prod-us-east-1-waker
    username: dormeto-waker
    groups: [dormeto:router]
  - rolearn: arn:aws:iam::222233334444:role/dmtr-prod-us-east-1-onboarder
    username: dormeto-onboarder
    groups: [dormeto:router]
  - rolearn: arn:aws:iam::222233334444:role/DmtDiscoveryRole
    username: dormeto-discovery
    groups: [dormeto:discovery]

Check: the router asks the cluster what it may do when it is installed or upgraded and before each onboarding. Its view in Routers lists the clusters of its account and region, whether it reaches them and the rights it lacks. The onboarding dialog of a Kubernetes workload shows the same. The onboarding stops while a right is missing: apply the manifest, then onboard again.

The check also says how each cluster is reached: directly, or through its proxy (its name and version).

Private clusters

A cluster with private access on (its public endpoint off, or limited to some addresses) is reached through a proxy function in its VPC. A cluster whose endpoint is public to every address is called directly, as before, even with private access also on. A cluster with private access off and a public endpoint limited to some addresses is not onboardable. Discovery says so: turn on private access.

• The proxy: one small Lambda function per cluster, <router>-kube-<cluster>, attached to the cluster's subnets. It has a security group of its own: TCP 443 out within the VPC. One rule on the cluster security group admits TCP 443 from it. The router and Discovery invoke it with the Lambda API and send their own EKS tokens. The proxy has no Kubernetes identity and no AWS permission beyond its network interfaces, and talks to its cluster's endpoint only. The access entries above are still needed, since the proxy carries the caller's identity.
• What it costs: Lambda invocations only, a few cents a month. No NAT gateway, no VPC endpoint, and nothing runs in the cluster.
• Who makes it: the manager creates, updates and removes the proxies of the routers it installs. It does so at install, upgrade, onboarding and cluster access. Uninstalling the router removes them. They need router 0.33.0.
• With Terraform, a router you wrote gets its proxies from the router repository's module kubernetes-proxy, one per cluster, beside eks-access. Its inputs are router_name and lambda_source_dir (as given to the router module), cluster_name and router_version, and optionally subnet_ids and security_group_ids. With security_group_ids given, the module adds no rule. You provide groups that reach the VPC on TCP 443, admitted by the cluster security group on TCP 443. subnet_ids changes nothing of this. Destroy it before the cluster's VPC.
• The new permissions, to apply again in each member account before upgrading its routers. DmtRouterBoundary gains ec2:CreateNetworkInterface, ec2:DescribeNetworkInterfaces, ec2:DescribeSubnets, ec2:DeleteNetworkInterface, ec2:AssignPrivateIpAddresses and ec2:UnassignPrivateIpAddresses (for the proxy's role). DmtCommandRole gains ec2:DescribeSecurityGroups, ec2:DescribeSubnets, ec2:DescribeVpcs and ec2:DescribeNetworkInterfaces. It gains ec2:CreateSecurityGroup and ec2:CreateTags, with the tag dormeto:role = kubernetes-proxy at creation, in the cluster's VPC whichever account owns it. A VPC shared through RAM works. It gains lambda:ListTags on its dmtr-* functions, to restore a proxy's tags. It gains egress rules and deletion on the groups with that tag. It gains ingress rules on the EKS cluster security groups (tagged aws:eks:cluster-name). DmtDiscoveryRole may invoke (lambda:InvokeFunction) the functions named *-kube-* tagged dormeto:role = kubernetes-proxy.

Remove

Uninstall the routers from the manager first. That deletes their access entries in the EKS clusters. It also deletes the proxy functions of private clusters, with their TCP 443 rule on the cluster's security group. In each EKS cluster, run kubectl delete -f dormeto-kubernetes-0.21.0.yaml and aws eks delete-access-entry --cluster-name my-cluster --principal-arn arn:aws:iam::222233334444:role/DmtDiscoveryRole. Or run terraform destroy on your access entries, and on your kubernetes-proxy modules before the clusters' VPCs. In each member account, run terraform destroy on the access module. With the AWS CLI, delete the inline policies discovery and command, the roles and the DmtRouterBoundary policy. In the management account, remove Dormeto's statement only. With the module, run terraform destroy when the policy holds nothing else. With additional_statements, remove the module from your state with terraform state rm instead, then run the commands below. With the AWS CLI:

aws organizations describe-resource-policy --query ResourcePolicy.Content --output text \
  | jq '.Statement |= map(select(.Sid != "DormetoReadOrganization"))' > rest.json
jq -e '.Statement | length > 0' rest.json > /dev/null \
  && aws organizations put-resource-policy --content file://rest.json \
  || aws organizations delete-resource-policy

Check

In the manager, Discovery shows Organization access: the accounts ready, those missing a role and why. Run now checks them again. Where the command role is missing, the dashboard says so before you onboard. Where the delegation policy is missing, the Perimeter page says organization mode is unavailable.