on automation | aws | cloudwatch | systemsmanager | powershell

Part 2: Making Sure Systems Manager Actually Works and Logs Are Sent to CloudWatch

Introduction: Why This Post Exists

So you followed Part 1, created your IAM roles, spun up an EC2 instance, installed the CisDsc module, and even uploaded your first MOF file to S3. You're ready to deploy those CIS benchmarks at scale, right?

Not so fast.

If you jumped straight to Part 3 and tried to run those Systems Manager commands, you might have been greeted with... nothing. No errors, no success messages, just commands that seem to disappear into the AWS void. Or worse, errors like "InvalidInstanceId" even though you can clearly see your instance in EC2.

Here's the thing: having SSM Agent installed and having Systems Manager actually able to manage your instance are two different things. It's like having a phone with no signal bars - all the hardware is there, but you can't make calls.

This post fills the gap between Parts 1 and 2. We'll make sure Systems Manager can actually talk to your instances before we try to push 300+ security settings to them. Trust me, spending 10 minutes on this now will save you hours of troubleshooting later.

Understanding Systems Manager Requirements

What Systems Manager Needs to Work

Systems Manager isn't magic (though it feels like it when it works). It needs several things to be in place:

  1. SSM Agent installed and running - Pre-installed on Windows AMIs from November 2016 onward
  2. IAM permissions - The instance needs permission to talk to Systems Manager
  3. Network connectivity - The instance needs to reach AWS endpoints
  4. Time to register - New instances take a few minutes to appear

Let's check each of these systematically.

The Network Piece Most People Miss

Here's what catches most people: Systems Manager doesn't work through public IPs like you might expect. The SSM Agent on your instance needs to make outbound HTTPS connections to several AWS endpoints:

  • ssm.{region}.amazonaws.com - Core Systems Manager API
  • ssmmessages.{region}.amazonaws.com - For Session Manager and interactive commands
  • ec2messages.{region}.amazonaws.com - For various EC2 operations
  • s3.{region}.amazonaws.com - To download your DSC configurations

If your instance is in a private subnet without internet access, you'll need VPC endpoints or a NAT gateway. But let's start with the basics.

Step-by-Step Verification

Important: Enable Systems Manager's New Experience

When you first navigate to Systems Manager, you'll see a yellow banner asking you to "Enable the new experience." Let's do that now:

  1. Click the "Enable the new experience" button in the yellow banner
  2. You'll see a confirmation about the new integrated experience
  3. Confirm to proceed

AWS is migrating everyone to this new interface, so we might as well start with it. The new experience provides:

  • Better organization of features under "Node Tools"
  • Improved navigation between related services
  • A more modern interface that AWS will continue to enhance

Throughout this tutorial, I'll use the new experience navigation paths.

Step 1: Check Your Instance Status

Now let's see if Systems Manager knows your instance exists. In the new experience:

  1. Navigate to AWS Systems Manager > Node Tools> Fleet Manager
  2. You'll first see the Fleet Manager landing page with "Streamline your node management"
  3. Click the "Get started" button (or if you see a list already, skip to step 4)
  4. You'll see a blue banner about the "new AWS Systems Manager unified console" - you can click the X to dismiss it or click "Learn more" if curious
  5. Look for your instance in the list

What you want to see:

  • Your instance listed with its instance ID
  • Ping status: Online (green dot with "Online" text)
  • Node state: Running (green circle with "Running" text)
  • Platform type: Windows
  • Agent version: Should show a version number (like 3.3.2299.0)

What you might see instead:

  • An empty list with "No managed nodes found"
  • Your instance not listed at all
  • Ping status: Connection Lost (red)
  • Missing agent version

If your instance isn't there or shows as offline, don't panic. Let's troubleshoot.

Note: The interface shows "Managed Nodes (1)" at the top - this number indicates how many instances Systems Manager can see. If it shows (0), your instance isn't registered yet.

Step 2: Verify from the Command Line

If you're not a fan of the GUI you can also check via AWS CLI:

# List all managed instances
aws ssm describe-instance-information \
    --query 'InstanceInformationList[*].[InstanceId,PingStatus,PlatformType,AgentVersion]' \
    --output table

# Check specific instance
aws ssm describe-instance-information \
    --filters "Key=InstanceIds,Values=i-YOUR-INSTANCE-ID" \
    --query 'InstanceInformationList[0]'

If your instance doesn't appear, we need to dig deeper.

Step 3: Check SSM Agent on the Instance

RDP into your Windows instance and open PowerShell as Administrator:

# Check if SSM Agent is installed and running
Get-Service AmazonSSMAgent

# Expected output:
# Status   Name               DisplayName
# ------   ----               -----------
# Running  AmazonSSMAgent     Amazon SSM Agent

# If it's not running:
Start-Service AmazonSSMAgent

# Check the version
& "C:\Program Files\Amazon\SSM\amazon-ssm-agent.exe" -version

If the service isn't there at all, you'll need to install it:

# Download and install latest SSM Agent
$progressPreference = 'SilentlyContinue'
Invoke-WebRequest `
    https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/windows_amd64/AmazonSSMAgentSetup.exe `
    -OutFile $env:TEMP\SSMAgent_latest.exe

Start-Process -FilePath "$env:TEMP\SSMAgent_latest.exe" -ArgumentList "/S" -Wait -NoNewWindow

# Start the service
Start-Service AmazonSSMAgent

Step 4: Verify Network Connectivity

This is where most issues hide. From your instance, test connectivity to Systems Manager endpoints:

# Function to test all required endpoints
function Test-SSMConnectivity {
    param(
        [string]$Region = 'us-east-1'  # Change to your region
    )
    
    $endpoints = @(
        "ssm.$Region.amazonaws.com",
        "ssmmessages.$Region.amazonaws.com",
        "ec2messages.$Region.amazonaws.com",
        "s3.$Region.amazonaws.com"
    )
    
    $results = @()
    
    foreach ($endpoint in $endpoints) {
        Write-Host "Testing $endpoint..." -NoNewline
        $test = Test-NetConnection -ComputerName $endpoint -Port 443 -InformationLevel Quiet
        
        $results += [PSCustomObject]@{
            Endpoint = $endpoint
            Reachable = $test
            Status = if ($test) { "✓ OK" } else { "✗ FAILED" }
        }
        
        Write-Host $(if ($test) { " OK" } else { " FAILED" }) -ForegroundColor $(if ($test) { "Green" } else { "Red" })
    }
    
    return $results
}

# Run the test
$connectivityTest = Test-SSMConnectivity -Region 'us-east-1'  # Use your region
$connectivityTest | Format-Table -AutoSize

# If any fail, check your security groups and NACLs

Step 5: Verify IAM Role

Make sure your instance actually has the IAM role attached:

# From the instance, check if we can access instance metadata
$token = Invoke-RestMethod -Headers @{"X-aws-ec2-metadata-token-ttl-seconds" = "21600"} `
    -Method PUT -Uri http://169.254.169.254/latest/api/token

$role = Invoke-RestMethod -Headers @{"X-aws-ec2-metadata-token" = $token} `
    -Method GET -Uri http://169.254.169.254/latest/meta-data/iam/security-credentials/

if ($role) {
    Write-Host "IAM Role attached: $role" -ForegroundColor Green
    
    # Get temporary credentials to verify they work
    $creds = Invoke-RestMethod -Headers @{"X-aws-ec2-metadata-token" = $token} `
        -Method GET -Uri "http://169.254.169.254/latest/meta-data/iam/security-credentials/$role"
    
    Write-Host "Credentials expire at: $($creds.Expiration)"
} else {
    Write-Host "No IAM role attached!" -ForegroundColor Red
}

Common Issues and Solutions

Issue 1: Instance Not Appearing in Systems Manager

Symptoms:

  • Instance running fine in EC2
  • SSM Agent running on instance
  • No errors, just... nothing in Systems Manager

Solutions:

  1. Wait a bit longer - Seriously, it can take 5-10 minutes for a new instance to appear
  2. Restart SSM Agent to force registration:
Restart-Service AmazonSSMAgent
# Wait 2-3 minutes and check again
  1. Check Logs for SSM Agent errors:
# SSM Agent logs location
Get-Content "C:\ProgramData\Amazon\SSM\Logs\amazon-ssm-agent.log" -Tail 50
Get-Content "C:\ProgramData\Amazon\SSM\Logs\errors.log" -Tail 50
  1. Force registration with a specific activation (advanced):
# Create an activation (from your local machine)
aws ssm create-activation \
    --default-instance-name "MyWindowsServer" \
    --description "Manual activation for troubleshooting" \
    --iam-role "EC2-SSM-Role" \
    --registration-limit 1

What happens when you run this: The command returns an Activation Code and Activation ID that you'll use on your Windows server to register it with SSM:

{
    "ActivationId": "1234abcd-12ab-34cd-56ef-1234567890ab",
    "ActivationCode": "ABCDEFGHIJKLMNOP"
}

You'd then use these values on your Windows server with a command like:


amazon-ssm-agent -register -code "ABCDEFGHIJKLMNOP" -id "1234abcd-12ab-34cd-56ef-1234567890ab" -region "us-east-1"

This is commonly used for hybrid environments where you want to manage non-AWS instances through Systems Manager.

Issue 2: Security Group Blocking Outbound HTTPS

Symptoms:

  • Network connectivity tests fail
  • SSM Agent logs show connection timeouts

Solution:

Your security group needs to allow outbound HTTPS. Here's a minimal security group configuration:

# Get your security group ID
INSTANCE_ID="i-YOUR-INSTANCE-ID"
SG_ID=$(aws ec2 describe-instances --instance-ids $INSTANCE_ID \
    --query 'Reservations[0].Instances[0].SecurityGroups[0].GroupId' --output text)

# Add outbound HTTPS rule if missing
aws ec2 authorize-security-group-egress \
    --group-id $SG_ID \
    --protocol tcp \
    --port 443 \
    --cidr 0.0.0.0/0

Or add a more restrictive rule for just AWS services:

# From the instance, get the region
$region = Invoke-RestMethod -Headers @{"X-aws-ec2-metadata-token" = $token} `
    -Method GET -Uri http://169.254.169.254/latest/meta-data/placement/region

# The CIDR blocks for AWS services in each region are published
# For production, consider using VPC endpoints instead

Issue 3: Instance in Private Subnet

Symptoms:

  • Instance has no public IP
  • Can't reach AWS endpoints
  • Network tests fail

Solutions:

  1. Add a NAT Gateway (costs money but easiest)
  2. Create VPC Endpoints (better for production):
# Create VPC endpoints for Systems Manager
VPC_ID="vpc-YOUR-VPC-ID"
SUBNET_ID="subnet-YOUR-PRIVATE-SUBNET"
REGION="us-east-1"

# Create endpoints for each service
for service in ssm ssmmessages ec2messages; do
    aws ec2 create-vpc-endpoint \
        --vpc-id $VPC_ID \
        --service-name com.amazonaws.$REGION.$service \
        --route-table-ids rtb-YOUR-ROUTE-TABLE \
        --subnet-ids $SUBNET_ID \
        --security-group-ids $SG_ID
done

# Don't forget S3 endpoint (gateway type)
aws ec2 create-vpc-endpoint \
    --vpc-id $VPC_ID \
    --service-name com.amazonaws.$REGION.s3 \
    --route-table-ids rtb-YOUR-ROUTE-TABLE

Issue 4: Time Sync Problems

Symptoms:

  • Weird authentication errors
  • SSM Agent logs show signature errors

Solution:

Windows instances need accurate time for AWS API signatures:

# Check time sync
w32tm /query /status

# Force sync
w32tm /resync /force

# Verify NTP configuration
Get-ItemProperty "HKLM:\SYSTEM\CurrentControlSet\Services\W32Time\Parameters"

Testing Systems Manager Works

Once your instance appears in Systems Manager, let's verify it actually works before moving to complex DSC deployments.

Basic Command Test

# Send a simple command

INSTANCE_ID="i-YOUR-INSTANCE-ID"

COMMAND_ID=$(aws ssm send-command \
    --document-name "AWS-RunPowerShellScript" \
    --targets "Key=instanceids,Values=$INSTANCE_ID" \
    --parameters 'commands=["Write-Host \"Hello from Systems Manager!\"","Get-Date"]' \
    --query 'Command.CommandId' \
    --output text)

echo "Command ID: $COMMAND_ID"

# Wait a few seconds, then check result
sleep 5

aws ssm get-command-invocation \
    --command-id $COMMAND_ID \
    --instance-id $INSTANCE_ID \
    --query '[Status,StandardOutputContent]' \
    --output text

Test S3 Access

Since we'll be downloading DSC configurations from S3:

# Test S3 access through Systems Manager

INSTANCE_ID="i-YOUR-INSTANCE-ID"

aws ssm send-command \
    --document-name "AWS-RunPowerShellScript" \
    --targets "Key=instanceids,Values=$INSTANCE_ID" \
    --parameters 'commands=[
        "Get-Module -ListAvailable AWS.Tools.S3",
        "Get-S3Bucket | Select-Object -First 5"
    ]'
    
echo "Command ID: $COMMAND_ID"

# Wait a few seconds, then check result
sleep 5

# Check if the command status
aws ssm get-command-invocation \
    --command-id $COMMAND_ID \
    --instance-id $INSTANCE_ID \
    --query '[Status,StandardOutputContent]' \
    --output text

Create a Test Association

Let's create a simple association that runs every 30 minutes to keep the connection warm:

# Create a heartbeat association
aws ssm create-association \
    --name "AWS-RunPowerShellScript" \
    --targets "Key=instanceids,Values=i-YOUR-INSTANCE-ID" \
    --parameters 'commands=["Write-Host \"Heartbeat: $(Get-Date)\""]' \
    --schedule-expression "rate(30 minutes)" \
    --output-location '{
        "S3Location": {
            "OutputS3BucketName": "your-s3-bucket",
            "OutputS3KeyPrefix": "ssm-heartbeat/"
        }
    }'

You can view associations in the console under Systems Manager > Node Tools > State Manager.

Monitoring Setup

Set up CloudWatch alarms so you know if Systems Manager loses connection:

# Create SNS topic for alerts
TOPIC_ARN=$(aws sns create-topic --name SSM-Connection-Alerts --query 'TopicArn' --output text)

# Subscribe your email
aws sns subscribe \
    --topic-arn $TOPIC_ARN \
    --protocol email \
    --notification-endpoint your-email@example.com

# Create alarm for lost connections
aws cloudwatch put-metric-alarm \
    --alarm-name "SSM-Instance-Connection-Lost" \
    --alarm-description "Alert when instance loses SSM connection" \
    --metric-name "PingStatus" \
    --namespace "AWS/SSM" \
    --statistic "Maximum" \
    --period 300 \
    --evaluation-periods 2 \
    --threshold 1 \
    --comparison-operator LessThanThreshold \
    --dimensions Name=InstanceId,Value=i-YOUR-INSTANCE-ID \
    --alarm-actions $TOPIC_ARN

Automation Script

Here's a PowerShell script that checks from the server to troubleshoot for most of what we've covered at once:

<#
.SYNOPSIS
    Tests AWS Systems Manager connectivity and setup on Windows instances.

.DESCRIPTION
    This script performs a comprehensive verification of AWS Systems Manager (SSM)
    prerequisites and connectivity on Windows EC2 instances, helping to diagnose
    common SSM connection issues.

.NOTES
    File Name     : Test-SystemsManagerSetup.ps1
    Author        : Jeffrey Stuhr
    Blog Reference: This is a companion script for the blog post available at:
                   https://www.techbyjeff.net/part-1-5-making-sure-systems-manager-actually-works-and-logs-are-sent-to-cloudwatch/

.LINK
    https://www.techbyjeff.net/part-1-5-making-sure-systems-manager-actually-works-and-logs-are-sent-to-cloudwatch/

.EXAMPLE
    .\Test-SystemsManagerSetup.ps1

    Runs the script with auto-detected instance ID and region.

.EXAMPLE
    .\Test-SystemsManagerSetup.ps1 -InstanceId "i-0123456789abcdef0" -Region "us-east-1"

    Runs the script with specified instance ID and region.
#>

function Test-SystemsManagerSetup {
    param(
        [string]$InstanceId = (Invoke-RestMethod -Headers @{"X-aws-ec2-metadata-token" = `
            (Invoke-RestMethod -Headers @{"X-aws-ec2-metadata-token-ttl-seconds" = "21600"} `
            -Method PUT -Uri http://169.254.169.254/latest/api/token)} `
            -Method GET -Uri http://169.254.169.254/latest/meta-data/instance-id),
        [string]$Region = (Invoke-RestMethod -Headers @{"X-aws-ec2-metadata-token" = `
            (Invoke-RestMethod -Headers @{"X-aws-ec2-metadata-token-ttl-seconds" = "21600"} `
            -Method PUT -Uri http://169.254.169.254/latest/api/token)} `
            -Method GET -Uri http://169.254.169.254/latest/meta-data/placement/region)
    )

    Write-Host "=== Systems Manager Setup Verification ===" -ForegroundColor Cyan
    Write-Host "Instance ID: $InstanceId"
    Write-Host "Region: $Region"
    Write-Host ""

    $results = @{
        InstanceId = $InstanceId
        Region = $Region
        Checks = @{}
    }

    # Check 1: SSM Agent Service
    Write-Host "[1/6] Checking SSM Agent Service..." -NoNewline
    $ssmService = Get-Service AmazonSSMAgent -ErrorAction SilentlyContinue
    if ($ssmService -and $ssmService.Status -eq 'Running') {
        Write-Host " PASS" -ForegroundColor Green
        $results.Checks.SSMAgent = "PASS"
    } else {
        Write-Host " FAIL" -ForegroundColor Red
        $results.Checks.SSMAgent = "FAIL: Service not running"
    }

    # Check 2: IAM Role
    Write-Host "[2/6] Checking IAM Role Assigned..." -NoNewline
    try {
        $token = Invoke-RestMethod -Headers @{"X-aws-ec2-metadata-token-ttl-seconds" = "21600"} `
            -Method PUT -Uri http://169.254.169.254/latest/api/token
        $role = Invoke-RestMethod -Headers @{"X-aws-ec2-metadata-token" = $token} `
            -Method GET -Uri http://169.254.169.254/latest/meta-data/iam/security-credentials/

        if ($role) {
            Write-Host " PASS (Role: $role)" -ForegroundColor Green

            # Now check if the role has SSM permissions by testing credentials
            Write-Host "      Verifying SSM permissions..." -NoNewline
            try {
                # Get the credentials from the instance metadata
                $credentials = Invoke-RestMethod -Headers @{"X-aws-ec2-metadata-token" = $token} `
                    -Method GET -Uri "http://169.254.169.254/latest/meta-data/iam/security-credentials/$role"

                # Check if credentials look valid (they should have AccessKeyId, SecretAccessKey, and Token)
                if ($credentials.AccessKeyId -and $credentials.SecretAccessKey -and $credentials.Token) {
                    # Try a simple unsigned request to check network connectivity to AWS endpoints
                    try {
                        $testEndpoint = "https://sts.$Region.amazonaws.com"
                        $connectTest = Invoke-WebRequest -Uri $testEndpoint -Method HEAD -TimeoutSec 5 -ErrorAction Stop
                        Write-Host " PASS (AWS credentials available, endpoints reachable)" -ForegroundColor Green
                        $results.Checks.IAMRole = "PASS: $role (credentials present and AWS endpoints accessible)"
                    } catch {
                        Write-Host " WARNING (Credentials present but endpoint test failed)" -ForegroundColor Yellow
                        $results.Checks.IAMRole = "WARNING: $role (credentials present but AWS endpoint connectivity failed)"
                    }
                } else {
                    Write-Host " FAIL (Invalid credentials)" -ForegroundColor Red
                    $results.Checks.IAMRole = "FAIL: $role has invalid or incomplete credentials"
                }
            } catch {
                Write-Host " WARNING (Cannot retrieve credentials)" -ForegroundColor Yellow
                $results.Checks.IAMRole = "WARNING: $role attached but cannot retrieve credentials - $($_.Exception.Message)"
            }
        } else {
            Write-Host " FAIL" -ForegroundColor Red
            $results.Checks.IAMRole = "FAIL: No role attached"
        }
    } catch {
        Write-Host " FAIL" -ForegroundColor Red
        $results.Checks.IAMRole = "FAIL: Cannot access metadata"
    }

    # Check 3: Network Connectivity
    Write-Host "[3/6] Checking Network Connectivity..."
    $endpoints = @(
        "ssm.$Region.amazonaws.com",
        "ssmmessages.$Region.amazonaws.com",
        "ec2messages.$Region.amazonaws.com",
        "s3.$Region.amazonaws.com"
    )

    $networkPass = $true
    foreach ($endpoint in $endpoints) {
        Write-Host "      Testing $endpoint..." -NoNewline
        $test = Test-NetConnection -ComputerName $endpoint -Port 443 -InformationLevel Quiet -WarningAction SilentlyContinue
        if ($test) {
            Write-Host " PASS" -ForegroundColor Green
        } else {
            Write-Host " FAIL" -ForegroundColor Red
            $networkPass = $false
        }
    }
    $results.Checks.Network = if ($networkPass) { "PASS" } else { "FAIL: Some endpoints unreachable" }

    # Check 4: Time Sync
    Write-Host "[4/6] Checking Time Sync..." -NoNewline
    try {
        # Get detailed time status
        $w32tmStatus = w32tm /query /status /verbose 2>$null

        if ($w32tmStatus) {
            # Check if time service is running and synchronized - be more flexible with the state check
            $serviceRunning = $w32tmStatus | Select-String "State:"
            $lastSync = $w32tmStatus | Select-String "Last Successful Sync Time:"

            # Check for any indication of synchronization
            $syncIndicators = $w32tmStatus | Select-String "(Synchronized|NtpClient|time.windows.com|pool.ntp.org)"

            if ($lastSync) {
                # Extract the last sync time and check if it's recent (within last 24 hours)
                $syncTimeMatch = $lastSync -match "(\d{1,2}/\d{1,2}/\d{4} \d{1,2}:\d{2}:\d{2} [AP]M)"

                if ($syncTimeMatch) {
                    try {
                        $syncTime = [DateTime]::Parse($matches[1])
                        $timeDiff = (Get-Date) - $syncTime

                        if ($timeDiff.TotalHours -le 24) {
                            Write-Host " PASS (Last sync: $($timeDiff.Hours)h $($timeDiff.Minutes)m ago)" -ForegroundColor Green
                            $results.Checks.TimeSync = "PASS: Recent sync within 24 hours"
                        } else {
                            Write-Host " WARNING (Last sync: $([int]$timeDiff.TotalDays) days ago)" -ForegroundColor Yellow
                            $results.Checks.TimeSync = "WARNING: Last sync was $([int]$timeDiff.TotalDays) days ago"
                        }
                    } catch {
                        Write-Host " PASS (Sync detected but time parsing failed)" -ForegroundColor Green
                        $results.Checks.TimeSync = "PASS: Time sync service active"
                    }
                } else {
                    Write-Host " PASS (Time service has sync history)" -ForegroundColor Green
                    $results.Checks.TimeSync = "PASS: Time sync service active"
                }
            } elseif ($syncIndicators) {
                # No explicit sync time but shows sync-related activity
                Write-Host " PASS (Time sync service active)" -ForegroundColor Green
                $results.Checks.TimeSync = "PASS: Time sync indicators found"
            } else {
                Write-Host " WARNING (Time service may not be synchronized)" -ForegroundColor Yellow
                $results.Checks.TimeSync = "WARNING: Time service not properly synchronized"
            }
        } else {
            Write-Host " WARNING (Cannot query time service)" -ForegroundColor Yellow
            $results.Checks.TimeSync = "WARNING: Cannot query time service status"
        }
    } catch {
        Write-Host " WARNING (Time sync check failed)" -ForegroundColor Yellow
        $results.Checks.TimeSync = "WARNING: Time sync verification failed - $($_.Exception.Message)"
    }

    # Check 5: AWS CLI/PowerShell
    Write-Host "[5/6] Checking AWS PowerShell Module..." -NoNewline
    if (Get-Module -ListAvailable -Name AWS.Tools.* | Where-Object {$_.Name -eq 'AWS.Tools.S3'}) {
        Write-Host " PASS" -ForegroundColor Green
        $results.Checks.AWSModule = "PASS"
    } else {
        Write-Host " WARNING (Optional)" -ForegroundColor Yellow
        $results.Checks.AWSModule = "WARNING: AWS.Tools not installed"
    }

# Check 6: SSM Registration Status (log-based verification)
    Write-Host "[6/6] Checking SSM Registration Logs..." -NoNewline
    try {
        $ssmLogPath = "C:\ProgramData\Amazon\SSM\Logs\amazon-ssm-agent.log"

        if (Test-Path $ssmLogPath) {
            # Look for successful registration indicators in more recent logs (last 200 lines to catch older registration)
            $recentLogs = Get-Content $ssmLogPath -Tail 200 | Where-Object {
                $_ -match "(successfully registered|ping reply|health ping succeeded|registration completed|managed instance|fingerprint matched)"
            }

            # Also look for ongoing activity indicators (these show SSM is actively working)
            $activityLogs = Get-Content $ssmLogPath -Tail 100 | Where-Object {
                $_ -match "(received message|command execution|document execution|polling|heartbeat)" -and
                $_ -notmatch "error|failed"
            }

            # Look for recent errors that would indicate problems
            $recentErrors = Get-Content $ssmLogPath -Tail 100 | Where-Object {
                $_ -match "(error|failed|timeout)" -and
                $_ -match "(registration|ssm|connection)" -and
                $_ -notmatch "retrying|retry"
            }

            # Enhanced logic: Consider both registration events AND ongoing activity
            if ($recentLogs.Count -gt 0 -and $recentErrors.Count -eq 0) {
                Write-Host " PASS (Logs show successful registration)" -ForegroundColor Green
                $results.Checks.SSMRegistration = "PASS: Registration verified in logs"
            } elseif ($activityLogs.Count -gt 0 -and $recentErrors.Count -eq 0) {
                Write-Host " PASS (Active SSM communication detected)" -ForegroundColor Green
                $results.Checks.SSMRegistration = "PASS: Active SSM communication indicates successful registration"
            } elseif (($recentLogs.Count -gt 0 -or $activityLogs.Count -gt 0) -and $recentErrors.Count -le 2) {
                Write-Host " WARNING (Some errors but registration appears active)" -ForegroundColor Yellow
                $results.Checks.SSMRegistration = "WARNING: Minor errors detected but registration appears active"
            } elseif ($recentErrors.Count -gt 2) {
                Write-Host " FAIL (Multiple recent errors)" -ForegroundColor Red
                $results.Checks.SSMRegistration = "FAIL: Multiple recent errors in agent logs"
            } else {
                # Final fallback: if no clear indicators, check if agent is running and other checks passed
                $agentRunning = (Get-Service AmazonSSMAgent -ErrorAction SilentlyContinue).Status -eq 'Running'
                $hasRole = $results.Checks.IAMRole -like "PASS*"
                $hasNetwork = $results.Checks.Network -like "PASS*"

                if ($agentRunning -and $hasRole -and $hasNetwork) {
                    Write-Host " WARNING (Likely registered but cannot verify from logs)" -ForegroundColor Yellow
                    $results.Checks.SSMRegistration = "WARNING: Prerequisites met but no clear log indicators (may be registered earlier)"
                } else {
                    Write-Host " WARNING (Cannot verify registration)" -ForegroundColor Yellow
                    $results.Checks.SSMRegistration = "WARNING: No clear registration indicators in recent logs"
                }
            }
        } else {
            Write-Host " WARNING (Log file not found)" -ForegroundColor Yellow
            $results.Checks.SSMRegistration = "WARNING: SSM agent log file not accessible"
        }

    } catch {
        Write-Host " WARNING (Cannot read logs)" -ForegroundColor Yellow
        $results.Checks.SSMRegistration = "WARNING: Cannot read SSM agent logs - $($_.Exception.Message)"
    }

    # Summary
    Write-Host ""
    Write-Host "=== Summary ===" -ForegroundColor Cyan
    $passCount = ($results.Checks.Values | Where-Object {$_ -like "PASS*"}).Count
    $totalCount = $results.Checks.Count

    if ($passCount -eq $totalCount) {
        Write-Host "All checks passed! Your instance is ready for Systems Manager." -ForegroundColor Green
    } elseif ($passCount -ge 4) {
        Write-Host "Most checks passed. Review warnings above." -ForegroundColor Yellow
    } else {
        Write-Host "Multiple checks failed. Please review and fix issues above." -ForegroundColor Red
    }

    return $results
}


# Run the test
$testResults = Test-SystemsManagerSetup

# Save results
$testResults | ConvertTo-Json -Depth 10 | Out-File "SSM-Setup-Test-$(Get-Date -Format 'yyyyMMdd-HHmmss').json"

Before we wrap up, let's set up CloudWatch Logs (since I'm sure you're wondering what it is when I mentioned above). This isn't required for Systems Manager to work, but you'll want it for:

  • Centralized logging across all instances
  • Troubleshooting DSC deployments
  • Creating alerts on errors
  • Following along with monitoring examples in Part 3

Step 1: Update IAM Role

First, let's add CloudWatch permissions to the role we created in Part 1:

  1. Navigate to IAM > Roles
  2. Find and click on your EC2-SSM-Role
  3. Click Add permissions > Attach policies
  4. Search for and select CloudWatchAgentServerPolicy
  5. Click Add permissions

Or via CLI:

aws iam attach-role-policy \
    --role-name EC2-SSM-Role \
    --policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy

Step 2: Install CloudWatch Agent

Good news - we can use Systems Manager to install the CloudWatch agent! No need to RDP into each instance.

# Install CloudWatch agent via Systems Manager
aws ssm send-command \
    --document-name "AWS-ConfigureAWSPackage" \
    --targets "Key=instanceids,Values=i-YOUR-INSTANCE-ID" \
    --parameters '{
        "action": ["Install"],
        "name": ["AmazonCloudWatchAgent"],
        "version": ["latest"]
    }' \
    --comment "Install CloudWatch Agent"

# Check installation status
aws ssm list-command-invocations \
    --instance-id "i-YOUR-INSTANCE-ID" \
    --details \
    --query 'CommandInvocations[0].Status'

Step 3: Configure CloudWatch Agent

Now we need to tell the agent what to collect. Create this configuration:

{
    "agent": {
        "metrics_collection_interval": 60
    },
    "logs": {
        "logs_collected": {
            "windows_events": {
                "collect_list": [
                    {
                        "event_name": "Microsoft-Windows-Desired State Configuration/Operational",
                        "event_levels": ["ERROR", "WARNING"],
                        "log_group_name": "/aws/systemsmanager/dsc",
                        "log_stream_name": "{instance_id}"
                    },
                    {
                        "event_name": "System",
                        "event_levels": ["ERROR", "WARNING"],
                        "log_group_name": "/aws/systemsmanager/system",
                        "log_stream_name": "{instance_id}"
                    }
                ]
            },
            "files": {
                "collect_list": [
                    {
                        "file_path": "C:\\ProgramData\\Amazon\\SSM\\Logs\\amazon-ssm-agent.log",
                        "log_group_name": "/aws/systemsmanager/ssm-agent",
                        "log_stream_name": "{instance_id}"
                    },
                    {
                        "file_path": "C:\\ProgramData\\Amazon\\SSM\\Logs\\errors.log",
                        "log_group_name": "/aws/systemsmanager/ssm-errors",
                        "log_stream_name": "{instance_id}"
                    },
                    {
                        "file_path": "C:\\Logs\\DSC\\*.log",
                        "log_group_name": "/aws/systemsmanager/dsc-files",
                        "log_stream_name": "{instance_id}"
                    }
                ]
            }
        }
    }
}

Save this as cloudwatch-config.json and store it in Parameter Store:

# Store configuration in Parameter Store
aws ssm put-parameter \
    --name "AmazonCloudWatch-windows-dsc" \
    --type "String" \
    --value file://cloudwatch-config.json \
    --description "CloudWatch config for DSC monitoring"

# Apply configuration to instances
aws ssm send-command \
    --document-name "AmazonCloudWatch-ManageAgent" \
    --targets "Key=instanceids,Values=i-YOUR-INSTANCE-ID" \
    --parameters '{
        "action": ["configure"],
        "mode": ["ec2"],
        "optionalConfigurationSource": ["ssm"],
        "optionalConfigurationLocation": ["AmazonCloudWatch-windows-dsc"],
        "optionalRestart": ["yes"]
    }'

Step 4: Verify Logs are Flowing

After a few minutes, check if logs are appearing:

# List log groups - you should see the ones we created
aws logs describe-log-groups --log-group-name-prefix "/aws/systemsmanager"

# Check for recent log streams
aws logs describe-log-streams \
    --log-group-name "/aws/systemsmanager/ssm-agent" \
    --order-by LastEventTime \
    --descending \
    --limit 5

Or check in the console:

  1. Navigate to CloudWatch > Log > Log groups
  2. Look for log groups starting with /aws/systemsmanager/
  3. Click into a log group and find your instance ID
  4. You should see recent log entries

Troubleshooting CloudWatch Logs

If logs aren't appearing:

# From the instance, check if CloudWatch agent is running
Get-Service AmazonCloudWatchAgent

# Check agent configuration
& "C:\Program Files\Amazon\AmazonCloudWatchAgent\amazon-cloudwatch-agent-ctl.ps1" `
    -m ec2 -a status

# View agent logs
Get-Content "C:\ProgramData\Amazon\AmazonCloudWatchAgent\Logs\amazon-cloudwatch-agent.log" -Tail 50

Common issues:

  • No logs appearing: Check IAM permissions and that the instance can reach CloudWatch endpoints
  • Agent not starting: Verify the configuration JSON is valid
  • Missing DSC logs: DSC events only appear after you start applying configurations

Cost Considerations

CloudWatch Logs pricing (as of 2024):

  • Ingestion: $0.50 per GB
  • Storage: $0.03 per GB per month
  • Free tier: 5GB ingestion, 5GB storage per month

For typical DSC deployments:

  • Each instance generates ~10-50MB of logs per month
  • 100 instances ≈ 1-5GB/month ≈ $0.50-$2.50/month

You can reduce costs by:

  • Adjusting log retention periods
  • Filtering to only ERROR and WARNING events
  • Using log sampling for high-volume logs

Quick Reference Checklist

Before moving to Part 3, ensure:

✅ Instance appears in Systems Manager Fleet Manager with "Online" status
✅ SSM Agent is running (version 3.0+ recommended)
✅ Network connectivity verified to all four endpoints
✅ IAM role attached with AmazonSSMManagedInstanceCore policy
✅ Test command executed successfully via Systems Manager
✅ S3 bucket accessible from the instance

What's Next?

If all checks pass, you're ready for Part 3! Your instances can now:

  • Receive commands from Systems Manager
  • Download configurations from S3
  • Report compliance status back
  • Scale to hundreds or thousands of instances

If you're still having issues, common next steps:

  1. Check CloudWatch Logs for SSM Agent errors
  2. Enable VPC Flow Logs to see if traffic is being blocked
  3. Try with a fresh instance in a public subnet first
  4. Post in the AWS forums with your specific error messages

Remember: Systems Manager is the foundation for everything we're building. It's worth getting this right before moving on to the fun stuff with DSC and CIS benchmarks.

Majority of code listed in this article can also be viewed at the companion GitHub.

One Last Thing

Here's a pro tip: Systems Manager can be flaky with instances that have been stopped/started frequently or have had their network settings changed. If you've been troubleshooting for a while and nothing works, sometimes the fastest solution is to:

  1. Terminate the instance (after backing up any work)
  2. Launch a fresh one with the IAM role attached from the start
  3. Run the verification script immediately

It's not elegant, but it works. And once you have Systems Manager working, it tends to stay working.

Ready for the real deployment action? See you in Part 3 where we will:

  • Deploying configurations through Systems Manager
  • Learn how to scale those deployments
  • Monitor compliance status in real-time

Introduction: Why This Post Exists

So you followed Part 1, created your IAM roles, spun up an EC2 instance, installed the CisDsc module, and even uploaded your first MOF file to S3. You're ready to deploy those CIS benchmarks at scale, right?

Not so fast.

If you jumped straight to Part 3 and tried to run those Systems Manager commands, you might have been greeted with... nothing. No errors, no success messages, just commands that seem to disappear into the AWS void. Or worse, errors like "InvalidInstanceId" even though you can clearly see your instance in EC2.

Here's the thing: having SSM Agent installed and having Systems Manager actually able to manage your instance are two different things. It's like having a phone with no signal bars - all the hardware is there, but you can't make calls.

This post fills the gap between Parts 1 and 3. We'll make sure Systems Manager can actually talk to your instances before we try to push 300+ security settings to them. Trust me, spending 10 minutes on this now will save you hours of troubleshooting later.

Understanding Systems Manager Requirements

What Systems Manager Needs to Work

Systems Manager isn't magic (though it feels like it when it works). It needs several things to be in place:

  1. SSM Agent installed and running - Pre-installed on Windows AMIs from November 2016 onward
  2. IAM permissions - The instance needs permission to talk to Systems Manager
  3. Network connectivity - The instance needs to reach AWS endpoints
  4. Time to register - New instances take a few minutes to appear

Let's check each of these systematically.

The Network Piece Most People Miss

Here's what catches most people: Systems Manager doesn't work through public IPs like you might expect. The SSM Agent on your instance needs to make outbound HTTPS connections to several AWS endpoints:

  • ssm.{region}.amazonaws.com - Core Systems Manager API
  • ssmmessages.{region}.amazonaws.com - For Session Manager and interactive commands
  • ec2messages.{region}.amazonaws.com - For various EC2 operations
  • s3.{region}.amazonaws.com - To download your DSC configurations

If your instance is in a private subnet without internet access, you'll need VPC endpoints or a NAT gateway. But let's start with the basics.

Step-by-Step Verification

Important: Enable Systems Manager's New Experience

When you first navigate to Systems Manager, you'll see a yellow banner asking you to "Enable the new experience." Let's do that now:

  1. Click the "Enable the new experience" button in the yellow banner
  2. You'll see a confirmation about the new integrated experience
  3. Confirm to proceed

AWS is migrating everyone to this new interface, so we might as well start with it. The new experience provides:

  • Better organization of features under "Node Tools"
  • Improved navigation between related services
  • A more modern interface that AWS will continue to enhance

Throughout this tutorial, I'll use the new experience navigation paths.

Step 1: Check Your Instance Status

Now let's see if Systems Manager knows your instance exists. In the new experience:

  1. Navigate to AWS Systems Manager > Node Tools> Fleet Manager
  2. You'll first see the Fleet Manager landing page with "Streamline your node management"
  3. Click the "Get started" button (or if you see a list already, skip to step 4)
  4. You'll see a blue banner about the "new AWS Systems Manager unified console" - you can click the X to dismiss it or click "Learn more" if curious
  5. Look for your instance in the list

What you want to see:

  • Your instance listed with its instance ID
  • Ping status: Online (green dot with "Online" text)
  • Node state: Running (green circle with "Running" text)
  • Platform type: Windows
  • Agent version: Should show a version number (like 3.3.2299.0)

What you might see instead:

  • An empty list with "No managed nodes found"
  • Your instance not listed at all
  • Ping status: Connection Lost (red)
  • Missing agent version

If your instance isn't there or shows as offline, don't panic. Let's troubleshoot.

Note: The interface shows "Managed Nodes (1)" at the top - this number indicates how many instances Systems Manager can see. If it shows (0), your instance isn't registered yet.

Step 2: Verify from the Command Line

If you're not a fan of the GUI you can also check via AWS CLI:

# List all managed instances
aws ssm describe-instance-information \
    --query 'InstanceInformationList[*].[InstanceId,PingStatus,PlatformType,AgentVersion]' \
    --output table

# Check specific instance
aws ssm describe-instance-information \
    --filters "Key=InstanceIds,Values=i-YOUR-INSTANCE-ID" \
    --query 'InstanceInformationList[0]'

If your instance doesn't appear, we need to dig deeper.

Step 3: Check SSM Agent on the Instance

RDP into your Windows instance and open PowerShell as Administrator:

# Check if SSM Agent is installed and running
Get-Service AmazonSSMAgent

# Expected output:
# Status   Name               DisplayName
# ------   ----               -----------
# Running  AmazonSSMAgent     Amazon SSM Agent

# If it's not running:
Start-Service AmazonSSMAgent

# Check the version
& "C:\Program Files\Amazon\SSM\amazon-ssm-agent.exe" -version

If the service isn't there at all, you'll need to install it:

# Download and install latest SSM Agent
$progressPreference = 'SilentlyContinue'
Invoke-WebRequest `
    https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/windows_amd64/AmazonSSMAgentSetup.exe `
    -OutFile $env:TEMP\SSMAgent_latest.exe

Start-Process -FilePath "$env:TEMP\SSMAgent_latest.exe" -ArgumentList "/S" -Wait -NoNewWindow

# Start the service
Start-Service AmazonSSMAgent

Step 4: Verify Network Connectivity

This is where most issues hide. From your instance, test connectivity to Systems Manager endpoints:

# Function to test all required endpoints
function Test-SSMConnectivity {
    param(
        [string]$Region = 'us-east-1'  # Change to your region
    )
    
    $endpoints = @(
        "ssm.$Region.amazonaws.com",
        "ssmmessages.$Region.amazonaws.com",
        "ec2messages.$Region.amazonaws.com",
        "s3.$Region.amazonaws.com"
    )
    
    $results = @()
    
    foreach ($endpoint in $endpoints) {
        Write-Host "Testing $endpoint..." -NoNewline
        $test = Test-NetConnection -ComputerName $endpoint -Port 443 -InformationLevel Quiet
        
        $results += [PSCustomObject]@{
            Endpoint = $endpoint
            Reachable = $test
            Status = if ($test) { "✓ OK" } else { "✗ FAILED" }
        }
        
        Write-Host $(if ($test) { " OK" } else { " FAILED" }) -ForegroundColor $(if ($test) { "Green" } else { "Red" })
    }
    
    return $results
}

# Run the test
$connectivityTest = Test-SSMConnectivity -Region 'us-east-1'  # Use your region
$connectivityTest | Format-Table -AutoSize

# If any fail, check your security groups and NACLs

Step 5: Verify IAM Role

Make sure your instance actually has the IAM role attached:

# From the instance, check if we can access instance metadata
$token = Invoke-RestMethod -Headers @{"X-aws-ec2-metadata-token-ttl-seconds" = "21600"} `
    -Method PUT -Uri http://169.254.169.254/latest/api/token

$role = Invoke-RestMethod -Headers @{"X-aws-ec2-metadata-token" = $token} `
    -Method GET -Uri http://169.254.169.254/latest/meta-data/iam/security-credentials/

if ($role) {
    Write-Host "IAM Role attached: $role" -ForegroundColor Green
    
    # Get temporary credentials to verify they work
    $creds = Invoke-RestMethod -Headers @{"X-aws-ec2-metadata-token" = $token} `
        -Method GET -Uri "http://169.254.169.254/latest/meta-data/iam/security-credentials/$role"
    
    Write-Host "Credentials expire at: $($creds.Expiration)"
} else {
    Write-Host "No IAM role attached!" -ForegroundColor Red
}

Common Issues and Solutions

Issue 1: Instance Not Appearing in Systems Manager

Symptoms:

  • Instance running fine in EC2
  • SSM Agent running on instance
  • No errors, just... nothing in Systems Manager

Solutions:

  1. Wait a bit longer - Seriously, it can take 5-10 minutes for a new instance to appear
  2. Restart SSM Agent to force registration:
Restart-Service AmazonSSMAgent
# Wait 2-3 minutes and check again
  1. Check Logs for SSM Agent errors:
# SSM Agent logs location
Get-Content "C:\ProgramData\Amazon\SSM\Logs\amazon-ssm-agent.log" -Tail 50
Get-Content "C:\ProgramData\Amazon\SSM\Logs\errors.log" -Tail 50
  1. Force registration with a specific activation (advanced):
# Create an activation (from your local machine)
aws ssm create-activation \
    --default-instance-name "MyWindowsServer" \
    --description "Manual activation for troubleshooting" \
    --iam-role "EC2-SSM-Role" \
    --registration-limit 1

What happens when you run this: The command returns an Activation Code and Activation ID that you'll use on your Windows server to register it with SSM:

{
    "ActivationId": "1234abcd-12ab-34cd-56ef-1234567890ab",
    "ActivationCode": "ABCDEFGHIJKLMNOP"
}

You'd then use these values on your Windows server with a command like:


amazon-ssm-agent -register -code "ABCDEFGHIJKLMNOP" -id "1234abcd-12ab-34cd-56ef-1234567890ab" -region "us-east-1"

This is commonly used for hybrid environments where you want to manage non-AWS instances through Systems Manager.

Issue 2: Security Group Blocking Outbound HTTPS

Symptoms:

  • Network connectivity tests fail
  • SSM Agent logs show connection timeouts

Solution:

Your security group needs to allow outbound HTTPS. Here's a minimal security group configuration:

# Get your security group ID
INSTANCE_ID="i-YOUR-INSTANCE-ID"
SG_ID=$(aws ec2 describe-instances --instance-ids $INSTANCE_ID \
    --query 'Reservations[0].Instances[0].SecurityGroups[0].GroupId' --output text)

# Add outbound HTTPS rule if missing
aws ec2 authorize-security-group-egress \
    --group-id $SG_ID \
    --protocol tcp \
    --port 443 \
    --cidr 0.0.0.0/0

Or add a more restrictive rule for just AWS services:

# From the instance, get the region
$region = Invoke-RestMethod -Headers @{"X-aws-ec2-metadata-token" = $token} `
    -Method GET -Uri http://169.254.169.254/latest/meta-data/placement/region

# The CIDR blocks for AWS services in each region are published
# For production, consider using VPC endpoints instead

Issue 3: Instance in Private Subnet

Symptoms:

  • Instance has no public IP
  • Can't reach AWS endpoints
  • Network tests fail

Solutions:

  1. Add a NAT Gateway (costs money but easiest)
  2. Create VPC Endpoints (better for production):
# Create VPC endpoints for Systems Manager
VPC_ID="vpc-YOUR-VPC-ID"
SUBNET_ID="subnet-YOUR-PRIVATE-SUBNET"
REGION="us-east-1"

# Create endpoints for each service
for service in ssm ssmmessages ec2messages; do
    aws ec2 create-vpc-endpoint \
        --vpc-id $VPC_ID \
        --service-name com.amazonaws.$REGION.$service \
        --route-table-ids rtb-YOUR-ROUTE-TABLE \
        --subnet-ids $SUBNET_ID \
        --security-group-ids $SG_ID
done

# Don't forget S3 endpoint (gateway type)
aws ec2 create-vpc-endpoint \
    --vpc-id $VPC_ID \
    --service-name com.amazonaws.$REGION.s3 \
    --route-table-ids rtb-YOUR-ROUTE-TABLE

Issue 4: Time Sync Problems

Symptoms:

  • Weird authentication errors
  • SSM Agent logs show signature errors

Solution:

Windows instances need accurate time for AWS API signatures:

# Check time sync
w32tm /query /status

# Force sync
w32tm /resync /force

# Verify NTP configuration
Get-ItemProperty "HKLM:\SYSTEM\CurrentControlSet\Services\W32Time\Parameters"

Testing Systems Manager Works

Once your instance appears in Systems Manager, let's verify it actually works before moving to complex DSC deployments.

Basic Command Test

# Send a simple command

INSTANCE_ID="i-YOUR-INSTANCE-ID"

COMMAND_ID=$(aws ssm send-command \
    --document-name "AWS-RunPowerShellScript" \
    --targets "Key=instanceids,Values=$INSTANCE_ID" \
    --parameters 'commands=["Write-Host \"Hello from Systems Manager!\"","Get-Date"]' \
    --query 'Command.CommandId' \
    --output text)

echo "Command ID: $COMMAND_ID"

# Wait a few seconds, then check result
sleep 5

aws ssm get-command-invocation \
    --command-id $COMMAND_ID \
    --instance-id $INSTANCE_ID \
    --query '[Status,StandardOutputContent]' \
    --output text

Test S3 Access

Since we'll be downloading DSC configurations from S3:

# Test S3 access through Systems Manager

INSTANCE_ID="i-YOUR-INSTANCE-ID"

aws ssm send-command \
    --document-name "AWS-RunPowerShellScript" \
    --targets "Key=instanceids,Values=$INSTANCE_ID" \
    --parameters 'commands=[
        "Get-Module -ListAvailable AWS.Tools.S3",
        "Get-S3Bucket | Select-Object -First 5"
    ]'
    
echo "Command ID: $COMMAND_ID"

# Wait a few seconds, then check result
sleep 5

# Check if the command status
aws ssm get-command-invocation \
    --command-id $COMMAND_ID \
    --instance-id $INSTANCE_ID \
    --query '[Status,StandardOutputContent]' \
    --output text

Create a Test Association

Let's create a simple association that runs every 30 minutes to keep the connection warm:

# Create a heartbeat association
aws ssm create-association \
    --name "AWS-RunPowerShellScript" \
    --targets "Key=instanceids,Values=i-YOUR-INSTANCE-ID" \
    --parameters 'commands=["Write-Host \"Heartbeat: $(Get-Date)\""]' \
    --schedule-expression "rate(30 minutes)" \
    --output-location '{
        "S3Location": {
            "OutputS3BucketName": "your-s3-bucket",
            "OutputS3KeyPrefix": "ssm-heartbeat/"
        }
    }'

You can view associations in the console under Systems Manager > Node Tools > State Manager.

Monitoring Setup

Set up CloudWatch alarms so you know if Systems Manager loses connection:

# Create SNS topic for alerts
TOPIC_ARN=$(aws sns create-topic --name SSM-Connection-Alerts --query 'TopicArn' --output text)

# Subscribe your email
aws sns subscribe \
    --topic-arn $TOPIC_ARN \
    --protocol email \
    --notification-endpoint your-email@example.com

# Create alarm for lost connections
aws cloudwatch put-metric-alarm \
    --alarm-name "SSM-Instance-Connection-Lost" \
    --alarm-description "Alert when instance loses SSM connection" \
    --metric-name "PingStatus" \
    --namespace "AWS/SSM" \
    --statistic "Maximum" \
    --period 300 \
    --evaluation-periods 2 \
    --threshold 1 \
    --comparison-operator LessThanThreshold \
    --dimensions Name=InstanceId,Value=i-YOUR-INSTANCE-ID \
    --alarm-actions $TOPIC_ARN

Automation Script

Here's a PowerShell script that checks from the server to troubleshoot for most of what we've covered at once:

<#
.SYNOPSIS
    Tests AWS Systems Manager connectivity and setup on Windows instances.

.DESCRIPTION
    This script performs a comprehensive verification of AWS Systems Manager (SSM)
    prerequisites and connectivity on Windows EC2 instances, helping to diagnose
    common SSM connection issues.

.NOTES
    File Name     : Test-SystemsManagerSetup.ps1
    Author        : Jeffrey Stuhr
    Blog Reference: This is a companion script for the blog post available at:
                   https://www.techbyjeff.net/part-1-5-making-sure-systems-manager-actually-works-and-logs-are-sent-to-cloudwatch/

.LINK
    https://www.techbyjeff.net/part-1-5-making-sure-systems-manager-actually-works-and-logs-are-sent-to-cloudwatch/

.EXAMPLE
    .\Test-SystemsManagerSetup.ps1

    Runs the script with auto-detected instance ID and region.

.EXAMPLE
    .\Test-SystemsManagerSetup.ps1 -InstanceId "i-0123456789abcdef0" -Region "us-east-1"

    Runs the script with specified instance ID and region.
#>

function Test-SystemsManagerSetup {
    param(
        [string]$InstanceId = (Invoke-RestMethod -Headers @{"X-aws-ec2-metadata-token" = `
            (Invoke-RestMethod -Headers @{"X-aws-ec2-metadata-token-ttl-seconds" = "21600"} `
            -Method PUT -Uri http://169.254.169.254/latest/api/token)} `
            -Method GET -Uri http://169.254.169.254/latest/meta-data/instance-id),
        [string]$Region = (Invoke-RestMethod -Headers @{"X-aws-ec2-metadata-token" = `
            (Invoke-RestMethod -Headers @{"X-aws-ec2-metadata-token-ttl-seconds" = "21600"} `
            -Method PUT -Uri http://169.254.169.254/latest/api/token)} `
            -Method GET -Uri http://169.254.169.254/latest/meta-data/placement/region)
    )

    Write-Host "=== Systems Manager Setup Verification ===" -ForegroundColor Cyan
    Write-Host "Instance ID: $InstanceId"
    Write-Host "Region: $Region"
    Write-Host ""

    $results = @{
        InstanceId = $InstanceId
        Region = $Region
        Checks = @{}
    }

    # Check 1: SSM Agent Service
    Write-Host "[1/6] Checking SSM Agent Service..." -NoNewline
    $ssmService = Get-Service AmazonSSMAgent -ErrorAction SilentlyContinue
    if ($ssmService -and $ssmService.Status -eq 'Running') {
        Write-Host " PASS" -ForegroundColor Green
        $results.Checks.SSMAgent = "PASS"
    } else {
        Write-Host " FAIL" -ForegroundColor Red
        $results.Checks.SSMAgent = "FAIL: Service not running"
    }

    # Check 2: IAM Role
    Write-Host "[2/6] Checking IAM Role Assigned..." -NoNewline
    try {
        $token = Invoke-RestMethod -Headers @{"X-aws-ec2-metadata-token-ttl-seconds" = "21600"} `
            -Method PUT -Uri http://169.254.169.254/latest/api/token
        $role = Invoke-RestMethod -Headers @{"X-aws-ec2-metadata-token" = $token} `
            -Method GET -Uri http://169.254.169.254/latest/meta-data/iam/security-credentials/

        if ($role) {
            Write-Host " PASS (Role: $role)" -ForegroundColor Green

            # Now check if the role has SSM permissions by testing credentials
            Write-Host "      Verifying SSM permissions..." -NoNewline
            try {
                # Get the credentials from the instance metadata
                $credentials = Invoke-RestMethod -Headers @{"X-aws-ec2-metadata-token" = $token} `
                    -Method GET -Uri "http://169.254.169.254/latest/meta-data/iam/security-credentials/$role"

                # Check if credentials look valid (they should have AccessKeyId, SecretAccessKey, and Token)
                if ($credentials.AccessKeyId -and $credentials.SecretAccessKey -and $credentials.Token) {
                    # Try a simple unsigned request to check network connectivity to AWS endpoints
                    try {
                        $testEndpoint = "https://sts.$Region.amazonaws.com"
                        $connectTest = Invoke-WebRequest -Uri $testEndpoint -Method HEAD -TimeoutSec 5 -ErrorAction Stop
                        Write-Host " PASS (AWS credentials available, endpoints reachable)" -ForegroundColor Green
                        $results.Checks.IAMRole = "PASS: $role (credentials present and AWS endpoints accessible)"
                    } catch {
                        Write-Host " WARNING (Credentials present but endpoint test failed)" -ForegroundColor Yellow
                        $results.Checks.IAMRole = "WARNING: $role (credentials present but AWS endpoint connectivity failed)"
                    }
                } else {
                    Write-Host " FAIL (Invalid credentials)" -ForegroundColor Red
                    $results.Checks.IAMRole = "FAIL: $role has invalid or incomplete credentials"
                }
            } catch {
                Write-Host " WARNING (Cannot retrieve credentials)" -ForegroundColor Yellow
                $results.Checks.IAMRole = "WARNING: $role attached but cannot retrieve credentials - $($_.Exception.Message)"
            }
        } else {
            Write-Host " FAIL" -ForegroundColor Red
            $results.Checks.IAMRole = "FAIL: No role attached"
        }
    } catch {
        Write-Host " FAIL" -ForegroundColor Red
        $results.Checks.IAMRole = "FAIL: Cannot access metadata"
    }

    # Check 3: Network Connectivity
    Write-Host "[3/6] Checking Network Connectivity..."
    $endpoints = @(
        "ssm.$Region.amazonaws.com",
        "ssmmessages.$Region.amazonaws.com",
        "ec2messages.$Region.amazonaws.com",
        "s3.$Region.amazonaws.com"
    )

    $networkPass = $true
    foreach ($endpoint in $endpoints) {
        Write-Host "      Testing $endpoint..." -NoNewline
        $test = Test-NetConnection -ComputerName $endpoint -Port 443 -InformationLevel Quiet -WarningAction SilentlyContinue
        if ($test) {
            Write-Host " PASS" -ForegroundColor Green
        } else {
            Write-Host " FAIL" -ForegroundColor Red
            $networkPass = $false
        }
    }
    $results.Checks.Network = if ($networkPass) { "PASS" } else { "FAIL: Some endpoints unreachable" }

    # Check 4: Time Sync
    Write-Host "[4/6] Checking Time Sync..." -NoNewline
    try {
        # Get detailed time status
        $w32tmStatus = w32tm /query /status /verbose 2>$null

        if ($w32tmStatus) {
            # Check if time service is running and synchronized - be more flexible with the state check
            $serviceRunning = $w32tmStatus | Select-String "State:"
            $lastSync = $w32tmStatus | Select-String "Last Successful Sync Time:"

            # Check for any indication of synchronization
            $syncIndicators = $w32tmStatus | Select-String "(Synchronized|NtpClient|time.windows.com|pool.ntp.org)"

            if ($lastSync) {
                # Extract the last sync time and check if it's recent (within last 24 hours)
                $syncTimeMatch = $lastSync -match "(\d{1,2}/\d{1,2}/\d{4} \d{1,2}:\d{2}:\d{2} [AP]M)"

                if ($syncTimeMatch) {
                    try {
                        $syncTime = [DateTime]::Parse($matches[1])
                        $timeDiff = (Get-Date) - $syncTime

                        if ($timeDiff.TotalHours -le 24) {
                            Write-Host " PASS (Last sync: $($timeDiff.Hours)h $($timeDiff.Minutes)m ago)" -ForegroundColor Green
                            $results.Checks.TimeSync = "PASS: Recent sync within 24 hours"
                        } else {
                            Write-Host " WARNING (Last sync: $([int]$timeDiff.TotalDays) days ago)" -ForegroundColor Yellow
                            $results.Checks.TimeSync = "WARNING: Last sync was $([int]$timeDiff.TotalDays) days ago"
                        }
                    } catch {
                        Write-Host " PASS (Sync detected but time parsing failed)" -ForegroundColor Green
                        $results.Checks.TimeSync = "PASS: Time sync service active"
                    }
                } else {
                    Write-Host " PASS (Time service has sync history)" -ForegroundColor Green
                    $results.Checks.TimeSync = "PASS: Time sync service active"
                }
            } elseif ($syncIndicators) {
                # No explicit sync time but shows sync-related activity
                Write-Host " PASS (Time sync service active)" -ForegroundColor Green
                $results.Checks.TimeSync = "PASS: Time sync indicators found"
            } else {
                Write-Host " WARNING (Time service may not be synchronized)" -ForegroundColor Yellow
                $results.Checks.TimeSync = "WARNING: Time service not properly synchronized"
            }
        } else {
            Write-Host " WARNING (Cannot query time service)" -ForegroundColor Yellow
            $results.Checks.TimeSync = "WARNING: Cannot query time service status"
        }
    } catch {
        Write-Host " WARNING (Time sync check failed)" -ForegroundColor Yellow
        $results.Checks.TimeSync = "WARNING: Time sync verification failed - $($_.Exception.Message)"
    }

    # Check 5: AWS CLI/PowerShell
    Write-Host "[5/6] Checking AWS PowerShell Module..." -NoNewline
    if (Get-Module -ListAvailable -Name AWS.Tools.* | Where-Object {$_.Name -eq 'AWS.Tools.S3'}) {
        Write-Host " PASS" -ForegroundColor Green
        $results.Checks.AWSModule = "PASS"
    } else {
        Write-Host " WARNING (Optional)" -ForegroundColor Yellow
        $results.Checks.AWSModule = "WARNING: AWS.Tools not installed"
    }

# Check 6: SSM Registration Status (log-based verification)
    Write-Host "[6/6] Checking SSM Registration Logs..." -NoNewline
    try {
        $ssmLogPath = "C:\ProgramData\Amazon\SSM\Logs\amazon-ssm-agent.log"

        if (Test-Path $ssmLogPath) {
            # Look for successful registration indicators in more recent logs (last 200 lines to catch older registration)
            $recentLogs = Get-Content $ssmLogPath -Tail 200 | Where-Object {
                $_ -match "(successfully registered|ping reply|health ping succeeded|registration completed|managed instance|fingerprint matched)"
            }

            # Also look for ongoing activity indicators (these show SSM is actively working)
            $activityLogs = Get-Content $ssmLogPath -Tail 100 | Where-Object {
                $_ -match "(received message|command execution|document execution|polling|heartbeat)" -and
                $_ -notmatch "error|failed"
            }

            # Look for recent errors that would indicate problems
            $recentErrors = Get-Content $ssmLogPath -Tail 100 | Where-Object {
                $_ -match "(error|failed|timeout)" -and
                $_ -match "(registration|ssm|connection)" -and
                $_ -notmatch "retrying|retry"
            }

            # Enhanced logic: Consider both registration events AND ongoing activity
            if ($recentLogs.Count -gt 0 -and $recentErrors.Count -eq 0) {
                Write-Host " PASS (Logs show successful registration)" -ForegroundColor Green
                $results.Checks.SSMRegistration = "PASS: Registration verified in logs"
            } elseif ($activityLogs.Count -gt 0 -and $recentErrors.Count -eq 0) {
                Write-Host " PASS (Active SSM communication detected)" -ForegroundColor Green
                $results.Checks.SSMRegistration = "PASS: Active SSM communication indicates successful registration"
            } elseif (($recentLogs.Count -gt 0 -or $activityLogs.Count -gt 0) -and $recentErrors.Count -le 2) {
                Write-Host " WARNING (Some errors but registration appears active)" -ForegroundColor Yellow
                $results.Checks.SSMRegistration = "WARNING: Minor errors detected but registration appears active"
            } elseif ($recentErrors.Count -gt 2) {
                Write-Host " FAIL (Multiple recent errors)" -ForegroundColor Red
                $results.Checks.SSMRegistration = "FAIL: Multiple recent errors in agent logs"
            } else {
                # Final fallback: if no clear indicators, check if agent is running and other checks passed
                $agentRunning = (Get-Service AmazonSSMAgent -ErrorAction SilentlyContinue).Status -eq 'Running'
                $hasRole = $results.Checks.IAMRole -like "PASS*"
                $hasNetwork = $results.Checks.Network -like "PASS*"

                if ($agentRunning -and $hasRole -and $hasNetwork) {
                    Write-Host " WARNING (Likely registered but cannot verify from logs)" -ForegroundColor Yellow
                    $results.Checks.SSMRegistration = "WARNING: Prerequisites met but no clear log indicators (may be registered earlier)"
                } else {
                    Write-Host " WARNING (Cannot verify registration)" -ForegroundColor Yellow
                    $results.Checks.SSMRegistration = "WARNING: No clear registration indicators in recent logs"
                }
            }
        } else {
            Write-Host " WARNING (Log file not found)" -ForegroundColor Yellow
            $results.Checks.SSMRegistration = "WARNING: SSM agent log file not accessible"
        }

    } catch {
        Write-Host " WARNING (Cannot read logs)" -ForegroundColor Yellow
        $results.Checks.SSMRegistration = "WARNING: Cannot read SSM agent logs - $($_.Exception.Message)"
    }

    # Summary
    Write-Host ""
    Write-Host "=== Summary ===" -ForegroundColor Cyan
    $passCount = ($results.Checks.Values | Where-Object {$_ -like "PASS*"}).Count
    $totalCount = $results.Checks.Count

    if ($passCount -eq $totalCount) {
        Write-Host "All checks passed! Your instance is ready for Systems Manager." -ForegroundColor Green
    } elseif ($passCount -ge 4) {
        Write-Host "Most checks passed. Review warnings above." -ForegroundColor Yellow
    } else {
        Write-Host "Multiple checks failed. Please review and fix issues above." -ForegroundColor Red
    }

    return $results
}


# Run the test
$testResults = Test-SystemsManagerSetup

# Save results
$testResults | ConvertTo-Json -Depth 10 | Out-File "SSM-Setup-Test-$(Get-Date -Format 'yyyyMMdd-HHmmss').json"

Before we wrap up, let's set up CloudWatch Logs (since I'm sure you're wondering what it is when I mentioned above). This isn't required for Systems Manager to work, but you'll want it for:

  • Centralized logging across all instances
  • Troubleshooting DSC deployments
  • Creating alerts on errors
  • Following along with monitoring examples in Part 3

Step 1: Update IAM Role

First, let's add CloudWatch permissions to the role we created in Part 1:

  1. Navigate to IAM > Roles
  2. Find and click on your EC2-SSM-Role
  3. Click Add permissions > Attach policies
  4. Search for and select CloudWatchAgentServerPolicy
  5. Click Add permissions

Or via CLI:

aws iam attach-role-policy \
    --role-name EC2-SSM-Role \
    --policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy

Step 2: Install CloudWatch Agent

Good news - we can use Systems Manager to install the CloudWatch agent! No need to RDP into each instance.

# Install CloudWatch agent via Systems Manager
aws ssm send-command \
    --document-name "AWS-ConfigureAWSPackage" \
    --targets "Key=instanceids,Values=i-YOUR-INSTANCE-ID" \
    --parameters '{
        "action": ["Install"],
        "name": ["AmazonCloudWatchAgent"],
        "version": ["latest"]
    }' \
    --comment "Install CloudWatch Agent"

# Check installation status
aws ssm list-command-invocations \
    --instance-id "i-YOUR-INSTANCE-ID" \
    --details \
    --query 'CommandInvocations[0].Status'

Step 3: Configure CloudWatch Agent

Now we need to tell the agent what to collect. Create this configuration:

{
    "agent": {
        "metrics_collection_interval": 60
    },
    "logs": {
        "logs_collected": {
            "windows_events": {
                "collect_list": [
                    {
                        "event_name": "Microsoft-Windows-Desired State Configuration/Operational",
                        "event_levels": ["ERROR", "WARNING", "INFORMATION"],
                        "log_group_name": "/aws/systemsmanager/dsc",
                        "log_stream_name": "{instance_id}"
                    },
                    {
                        "event_name": "System",
                        "event_levels": ["ERROR", "WARNING"],
                        "log_group_name": "/aws/systemsmanager/system",
                        "log_stream_name": "{instance_id}"
                    }
                ]
            },
            "files": {
                "collect_list": [
                    {
                        "file_path": "C:\\ProgramData\\Amazon\\SSM\\Logs\\amazon-ssm-agent.log",
                        "log_group_name": "/aws/systemsmanager/ssm-agent",
                        "log_stream_name": "{instance_id}"
                    },
                    {
                        "file_path": "C:\\ProgramData\\Amazon\\SSM\\Logs\\errors.log",
                        "log_group_name": "/aws/systemsmanager/ssm-errors",
                        "log_stream_name": "{instance_id}"
                    }
                ]
            }
        }
    }
}

Save this as cloudwatch-config.json and store it in Parameter Store:

# Store configuration in Parameter Store
aws ssm put-parameter \
    --name "AmazonCloudWatch-windows-dsc" \
    --type "String" \
    --value file://cloudwatch-config.json \
    --description "CloudWatch config for DSC monitoring"

# Apply configuration to instances
aws ssm send-command \
    --document-name "AmazonCloudWatch-ManageAgent" \
    --targets "Key=instanceids,Values=i-YOUR-INSTANCE-ID" \
    --parameters '{
        "action": ["configure"],
        "mode": ["ec2"],
        "optionalConfigurationSource": ["ssm"],
        "optionalConfigurationLocation": ["AmazonCloudWatch-windows-dsc"],
        "optionalRestart": ["yes"]
    }'

Step 4: Verify Logs are Flowing

After a few minutes, check if logs are appearing:

# List log groups - you should see the ones we created
aws logs describe-log-groups --log-group-name-prefix "/aws/systemsmanager"

# Check for recent log streams
aws logs describe-log-streams \
    --log-group-name "/aws/systemsmanager/ssm-agent" \
    --order-by LastEventTime \
    --descending \
    --limit 5

Or check in the console:

  1. Navigate to CloudWatch > Log > Log groups
  2. Look for log groups starting with /aws/systemsmanager/
  3. Click into a log group and find your instance ID
  4. You should see recent log entries

Troubleshooting CloudWatch Logs

If logs aren't appearing:

# From the instance, check if CloudWatch agent is running
Get-Service AmazonCloudWatchAgent

# Check agent configuration
& "C:\Program Files\Amazon\AmazonCloudWatchAgent\amazon-cloudwatch-agent-ctl.ps1" `
    -m ec2 -a status

# View agent logs
Get-Content "C:\ProgramData\Amazon\AmazonCloudWatchAgent\Logs\amazon-cloudwatch-agent.log" -Tail 50

Common issues:

  • No logs appearing: Check IAM permissions and that the instance can reach CloudWatch endpoints
  • Agent not starting: Verify the configuration JSON is valid
  • Missing DSC logs: DSC events only appear after you start applying configurations

Cost Considerations

CloudWatch Logs pricing (as of 2024):

  • Ingestion: $0.50 per GB
  • Storage: $0.03 per GB per month
  • Free tier: 5GB ingestion, 5GB storage per month

For typical DSC deployments:

  • Each instance generates ~10-50MB of logs per month
  • 100 instances ≈ 1-5GB/month ≈ $0.50-$2.50/month

You can reduce costs by:

  • Adjusting log retention periods
  • Filtering to only ERROR and WARNING events
  • Using log sampling for high-volume logs

Quick Reference Checklist

Before moving to Part 3, ensure:

✅ Instance appears in Systems Manager Fleet Manager with "Online" status
✅ SSM Agent is running (version 3.0+ recommended)
✅ Network connectivity verified to all four endpoints
✅ IAM role attached with AmazonSSMManagedInstanceCore policy
✅ Test command executed successfully via Systems Manager
✅ S3 bucket accessible from the instance

What's Next?

If all checks pass, you're ready for Part 3! Your instances can now:

  • Receive commands from Systems Manager
  • Download configurations from S3
  • Report compliance status back
  • Scale to hundreds or thousands of instances

If you're still having issues, common next steps:

  1. Check CloudWatch Logs for SSM Agent errors
  2. Enable VPC Flow Logs to see if traffic is being blocked
  3. Try with a fresh instance in a public subnet first
  4. Post in the AWS forums with your specific error messages

Remember: Systems Manager is the foundation for everything we're building. It's worth getting this right before moving on to the fun stuff with DSC and CIS benchmarks.

Majority of code listed in this article can also be viewed at the companion GitHub.

One Last Thing

Here's a pro tip: Systems Manager can be flaky with instances that have been stopped/started frequently or have had their network settings changed. If you've been troubleshooting for a while and nothing works, sometimes the fastest solution is to:

  1. Terminate the instance (after backing up any work)
  2. Launch a fresh one with the IAM role attached from the start
  3. Run the verification script immediately

It's not elegant, but it works. And once you have Systems Manager working, it tends to stay working.

Ready for the real deployment action? See you in Part 3!