AI-DAST

Overview

Detect and analyze runtime vulnerabilities in AI models and agents through dynamic security testing. Identify AI-specific security risks including prompt injection attacks, data leakage, model manipulation, and insecure AI configurations. Gain actionable insights into your AI security posture by testing foundation models, custom models, and imported models deployed in AWS Bedrock. Enable comprehensive security testing and monitoring to strengthen the security of your AI applications and protect sensitive data.

Configurations

Configuration
Description

Blueprint Account Name

A human-readable name for your AI-DAST configuration that will be used to identify this configuration across the application.

IAM Role ARN

The ARN of the IAM role created by the CloudFormation template that Kaleidoscope will assume for AI model access and security testing. Details in the Permissions section.

External ID

The external ID required to assume the IAM role. This is automatically generated by the CloudFormation template and stored in AWS Secrets Manager. Retrieve it from the Secrets Manager console using the secret name provided in the stack outputs. Details in the Permissions section.

Resource ARNs

List of AWS Bedrock model ARNs and agent ARNs to scan for security vulnerabilities. See Resource ARN Formats for supported formats.

Data Crawl Frequency

The frequency at which Kaleidoscope will perform security testing on the specified AI models and agents.

Resource ARN Formats

The AI-DAST blueprint supports the following AWS Bedrock resource ARN formats for models and agents:

Foundation Models

arn:aws:bedrock:REGION::foundation-model/MODEL_ID

Example: arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-3-sonnet-20240229-v1:0

Foundation models are pre-trained models provided by AWS and model providers, accessible across all AWS accounts in a region.

Imported Models

arn:aws:bedrock:REGION:ACCOUNT_ID:imported-model/MODEL_ID

Example: arn:aws:bedrock:us-east-1:123456789012:imported-model/my-imported-model

Imported models are models you've imported into your AWS account from external sources.

Custom Model Deployments

arn:aws:bedrock:REGION:ACCOUNT_ID:custom-model-deployment/MODEL_ID

Example: arn:aws:bedrock:us-east-1:123456789012:custom-model-deployment/my-deployment

Custom model deployments represent specific deployments of your custom models with dedicated resources.

Inference Profiles

arn:aws:bedrock:REGION:ACCOUNT_ID:inference-profile/MODEL_ID

Example: arn:aws:bedrock:us-east-1:123456789012:inference-profile/my-inference-profile

Inference profiles allow you to configure and manage inference settings for models.

Bedrock Agents

arn:aws:bedrock:REGION:ACCOUNT_ID:agent/AGENT_ID

Example: arn:aws:bedrock:us-east-1:123456789012:agent/ABCDEFGHIJ

Bedrock agents are autonomous AI agents that can orchestrate tasks, call APIs, and interact with knowledge bases.

Notes on Resource ARNs

  • REGION: The AWS region where the resource is available (e.g., us-east-1, us-west-2)

  • ACCOUNT_ID: Your 12-digit AWS account ID (not required for foundation models)

  • MODEL_ID: The unique identifier for the model

  • AGENT_ID: The unique identifier for the agent (10-character alphanumeric string)

You can list available resources in your account using the AWS CLI:

# List foundation models
aws bedrock list-foundation-models --region us-east-1

# List custom models
aws bedrock list-custom-models --region us-east-1

# List imported models
aws bedrock list-imported-models --region us-east-1

# List agents
aws bedrock-agent list-agents --region us-east-1

Permissions

The AI-DAST blueprint requires specific AWS Bedrock permissions to perform dynamic security testing on AI models and agents.

Overview

The AI-DAST blueprint uses a CloudFormation template to provision the following resources:

  1. AI Security Testing Infrastructure:

    • IAM role with Bedrock model and agent invocation permissions

    • Auto-generated External ID stored in AWS Secrets Manager

    • Cross-account assume role permissions with external ID for enhanced security

IAM Role Permissions

The IAM role created by the CloudFormation template includes:

Bedrock Invoke Policy

Grants permissions to:

  • Invoke AI models for security testing

  • Execute streaming model invocations for testing

  • Invoke Bedrock agents for security testing

  • Perform dynamic analysis of model and agent responses and behavior

Required permissions: bedrock:InvokeModel, bedrock:InvokeModelWithResponseStream, bedrock:InvokeAgent

Security Considerations

  • Principle of Least Privilege: The policy grants only the permissions required for AI security testing

  • Invocation Only: The role can invoke models and agents but cannot modify or delete them

  • Cross-Account Security: External ID validation ensures only authorized Kaleidoscope accounts can assume the role

  • Audit Trail: All model and agent invocations are logged for security audit and compliance purposes

CloudFormation Template

To set up the AI-DAST blueprint, deploy the provided CloudFormation template in your AWS account.

Note: This CloudFormation template is intended for initial stack creation only. If you need to update the stack or modify its resources after deployment, you will need to update the stack manually through the AWS CloudFormation console or AWS CLI.

What the Template Creates

The CloudFormation template provisions:

  1. IAM Role: Cross-account role with Bedrock model and agent invocation permissions

  2. External ID: Auto-generated external ID stored in AWS Secrets Manager

  3. Trust Relationship: Allows Kaleidoscope to assume the role securely

Template Parameters

Parameter
Description
Default

ResourcePrefix

Prefix for all created resources

kscope

TrustedAccountId

Kaleidoscope AWS Account ID that can assume the AI DAST role

(Required)

Template Outputs

Output
Description

AIDASTRoleArn

The ARN of the created IAM role (use for IAM Role ARN configuration)

ExternalIdSecretName

Name of the Secrets Manager secret containing the external ID

Deployment Steps

Before deploying the CloudFormation template, you will need to get the Trusted Account ID. Copy the AWS account ID from the Kaleidoscope blueprint configuration page. This is the account that will be authorized to assume the IAM role for AI security testing.

Quick Deploy

Deploy the AI-DAST CloudFormation template to your AWS account:

Launch Stack

This will open the AWS CloudFormation console with the template pre-loaded and default values:

  • ResourcePrefix: kscope (you can customize this)

  • TrustedAccountId: Enter your Kaleidoscope AWS account ID (from the blueprint configuration page)

  1. Retrieve Configuration Values:

    • After successful deployment, go to the stack Outputs tab

    • Copy the AIDASTRoleArn value

    • Go to AWS Secrets Manager console

    • Find the secret with the name from ExternalIdSecretName output

    • Retrieve the externalId value from the secret

  2. Configure AI-DAST Blueprint in Kaleidoscope:

    • Navigate to the App Hub in Kaleidoscope

    • Select the AI-DAST blueprint

    • Enter the AIDASTRoleArn as the IAM Role ARN

    • Enter the externalId as the External ID

    • Specify the Bedrock model ARNs and agent ARNs you want to test

    • Configure the data crawl frequency

    • Save the configuration

  3. Start Security Testing:

    • Initiate the first crawl to begin AI security testing on models and agents

    • Review findings in the AI-DAST dashboard

Updating the Stack

If you need to update the CloudFormation stack after the initial deployment (e.g., to update permissions or modify configurations), follow these steps:

Note: The Quick Deploy button is only for initial stack creation. For updates, you must manually update the stack through the AWS CloudFormation console.

  1. Navigate to CloudFormation Console:

    • Sign in to the AWS Management Console

    • Go to the CloudFormation service

    • Select the stack you want to update (e.g., kaleidoscope-ai-dast-blueprint)

  2. Initiate Stack Update:

    • Click Update button at the top

    • Select Replace current template

    • Choose Amazon S3 URL and enter: https://kaleidoscope-blueprint-configurator.s3.amazonaws.com/aws-ai-dast/aws-ai-dast-latest.yml

    • Click Next

  3. Review Parameters:

    • Review and modify stack parameters if needed

    • Click Next

  4. Configure Stack Options:

    • Review stack options (tags, permissions, etc.)

    • Click Next

  5. Create Change Set:

    • Review the changes that will be made

    • Check the box to acknowledge that AWS CloudFormation might create IAM resources

    • Click Create change set

    • Wait for the change set to be created

  6. Execute Change Set:

    • Review the change set details to ensure the changes are correct

    • Click Execute change set

    • Confirm the execution

    • Wait for the stack update to complete

  7. Verify Update:

    • Check the stack status shows UPDATE_COMPLETE

    • Review the Events tab to ensure no errors occurred

    • Verify the Outputs tab for any updated values

PreviousWizNextAstrila

Last updated

Was this helpful?