PuppyIP Resource Center
AI Security Updates 9 minutes Published 2026-11-14

AWS Continuum CI/CD pentest gate: PASS, defaults and baselines

This integration runs a dynamic penetration test after a preproduction deployment and uses the result to decide whether promotion can continue. A green pipeline can mean the change was tested with no finding at the threshold, the change was scoped out, or an infrastructure warning passed under fail-open; read the scope decision and baseline before treating PASS as evidence.

AWS Continuum CI/CD Penetration testing Post-deployment gates Fail-open Commit baselines

Service eligibility and regional restrictions

PuppyIP serves only compliant overseas businesses and their authorized personnel. Proxy services are not available in mainland China. The service may only be used for lawful business activities outside mainland China. Use of this service within mainland China is prohibited.

Hosting a proxy IP or server overseas does not change these restrictions. The service must not be provided to end users in mainland China through relaying, forwarding, sharing or resale. Before use, read the Terms of Service.

Key Takeaways

  • This is a post-deployment penetration-test gate for deployed commit ranges, not a pull-request static check. Use it only for an application you own or are authorized to test in preproduction.
  • GitHub Actions, GitLab CI/CD, Bitbucket Pipelines and Azure DevOps share the defaults: HIGH, 60 minutes, block on scope conflict, fail-on-infrastructure-error false and dry-run false.
  • PASS can follow a completed test with no finding at the threshold, a scoped-out change, or a warning that was allowed through. A scope conflict means the change was not tested.
  • Permissive outcomes can advance a baseline. Tightening settings later does not automatically retest cleared commits; set an explicit base for the range you need to reassess.

This is a post-deployment gate, not a pull-request scan

On October 5, 2026, AWS announced the public-preview CI/CD integration for AWS Continuum for Penetration Testing, formerly AWS Security Agent. The current user guide still marks the integration as a preview that is subject to change.

Place the gate after deploying an application you own or are authorized to test in preproduction. It runs a dynamic test against the deployed change and uses the result to block or allow promotion to the next stage. The penetration test runs in your AWS account; the pipeline starts the job, waits and reports its result.

The integration tests a deployed commit range, which can span multiple pull requests. It is not a static code review of an individual pull request before merge. AWS currently lists GitHub Actions, GitLab CI/CD, Bitbucket Pipelines and Azure DevOps Pipelines.

Prepare the test, repository and Region first

The pipeline step starts an existing configured penetration test; it does not create a target or test for you. Verify the target domain, configure the test in an Agent Space with its target, credentials and network scope, enable CI/CD mode, and connect the repository as a source-control integration.

The target must be an application you own or are authorized to test; this workflow is scoped to preproduction. AWS Security Agent is available in a subset of AWS Regions, so select a pipeline Region that matches the Agent Space and penetration test.

Use your CI/CD provider’s OIDC identity to obtain temporary AWS STS credentials rather than storing long-lived AWS keys as pipeline variables. Restrict the IAM trust condition to the real repository, project or service connection. Add a branch or environment condition only where that provider exposes it: Bitbucket has no branch claim, and Azure DevOps identifies a service connection in its subject.

Read the scope decision before interpreting findings

In scope means the change touches the configured attack surface, so AWS evaluates findings against the severity threshold. The default is HIGH: HIGH or CRITICAL findings block promotion, while lower severities are reported without blocking.

Scoped out means the change is within the test’s configured scope but does not modify the tested attack surface. No test is needed and the pipeline passes; that PASS does not mean the change was scanned.

Scope conflict means the change is outside the configured penetration-test scope and was not tested. The default is to block the pipeline, though you can change it to warn and pass. A green status after that warning still does not prove coverage.

The fail-open default can turn some infrastructure errors into PASS

All four integrations default to a 60-minute timeout, blocking scope conflicts, fail-on-infrastructure-error false, and dry-run false, which starts a real penetration-test job. If an unexpected infrastructure error prevents a job from starting or completing, the step can pass with a warning.

Fail-open does not pass every failure: a finding at the threshold, timeout, blocked scope conflict, HTTP 401 or 403 from the AWS Security Agent control plane, and configuration errors the integration recognizes still fail. If OIDC credential exchange fails before the gate calls AWS Security Agent, the CI/CD provider’s credential step determines the pipeline result.

For a promotion that requires security sign-off, do not weaken an existing gate just to get a green result. First run a dry-run to validate AWS identity, Region and repository integration, then set fail-on-infrastructure-error to true. NONE only prevents findings from failing the step; testing and reporting still run, but it is not a substitute for security sign-off.

How PASS moves the baseline determines what gets tested next

The baseline records the last commit cleared by the gate; the next run normally tests only changes after it. If no usable record exists, the integration uses a trigger-specific fallback rather than switching to a full-scope test. A normal pass, including no finding at the threshold or a scoped-out change, advances the baseline.

A passing NONE run advances the baseline on all four providers, even though findings are still reported. A passing scope-conflict warning or fail-open infrastructure warning advances it on GitHub Actions, GitLab CI/CD and Azure DevOps; Bitbucket advances only when AWS returned an actual verdict for that range, so these two warning-only outcomes do not advance it there.

A failed run, dry-run or any run with an explicitly pinned head does not advance the baseline. Raising the threshold or changing fail-open to fail-closed later does not revisit commits already recorded as cleared. To retest a known range, set the base explicitly to the commit before that range and check its relationship to the head.

Validate a small scope before using the gate for promotion

On a controlled preproduction deployment, check the Scoping view for the base, head and scope decision. Confirm that findings at HIGH or above, scope conflicts and infrastructure errors produce the expected pipeline results; retain the original job and commit range when a warning or unexpected status needs investigation.

Pin the GitHub Action, GitLab component or container integration to an immutable version so upstream changes do not silently change gate behavior. GitLab pipeline variables can override defaults committed in the job; restrict who can override them or the committed configuration is not an enforceable gate.

A validated run confirms only that configuration and commit range. It does not establish that the application has no vulnerabilities or that later deployments were fully covered. Keep the scope decision, commit baseline, error handling and owner together for future retests and reviews.

Sources

Frequently Asked Questions

Does cancelling the pipeline stop the remote penetration-test job?

Not necessarily. The job runs in AWS and may continue after the pipeline stops waiting. GitHub and Azure integrations attempt to stop it during cancellation; GitLab relies on after_script being allowed to run, while a Bitbucket cancellation can leave the job running. Check the original job’s terminal status and stop it in the AWS Security Agent console if it is still active before deciding to rerun.

Does dry-run start a real penetration test?

No. It checks AWS identity, Region and repository integration, then exits before starting the job. Validation failures always fail the step, and dry-run does not advance the baseline.

How long should OIDC credentials remain valid?

Credentials do not refresh while the step runs, so they need to cover the full test and cleanup. Set the role’s maximum session duration to at least (gate timeout plus 10 minutes) times 60 seconds; the default 60-minute timeout therefore needs at least 4,200 seconds. Request that duration from providers that let you specify it.

Are all related AWS charges waived during the preview?

No. AWS says it does not charge for continuous penetration-test runs during the preview, but other AWS resources, such as hosting the application under test, can still incur charges. Other types of penetration tests remain metered as usual, and preview terms can change.