Skip to main content

Cloud Permissions Firewall Troubleshooting

One-Click Least Privilege. Zero Disruption.



© 2025 Sonrai Security. All rights reserved.

Overview​

As a Sonrai Cloud Permission Firewall (CPF) Administrator, you need to know how to avoid access lockouts and be able to safely restore access if CPF ever experiences degraded performance or becomes unavailable. Don't worry — we’ve got this covered.

This page gives detailed instructions about how to troubleshoot issues, or even temporarily switch off CPF through AWS if needed. Start with the least‑impactful steps and escalate when required.

tip

Before taking actions described on this page, always contact Sonrai Customer Success to let us know about your issues. We're here to help!

Scenario 1:
ChatOps notifications (Slack/Teams) are missing, but the Sonrai CPF App is still available...

Try these ChatOps Workarounds.

Scenario 2:
The Sonrai Web App is responsive, but you must temporarily remove all CPF controls...

Remove Controls using Sonrai CPF.

Scenario 3:
The Sonrai Web App is not responding, and you need to remove all CPF controls (outage)...

Perform an Emergency Stop from AWS.


After any scenario, be sure to complete the Recovery & Re‑Sync process to restore your Cloud Permissions Firewall deployment.

ChatOps Workarounds​

ChatOps workflows make Permission on Demand, JIT Access, and Zombie Wake-Up workflows fast and easy - but what should you do when ChatOps notifications aren't being received?

Permissions on Demand (PoD) Request Missing in Chat​

Workaround steps:

  • Manually approve requests in your CPF web application: In the Requests view, check for the PoD request. If it exists, have an approver in the relevant scope approve directly; if not, continue with the next step.
  • Manually add an (admin) service exemption: In the Services view, choose the OU or Account in your Org Selector dropdown. Select the Service in the table for the blocked permission, and add the blocked identity (IAM User/Role or SSO user) into the table manually. After ~30 seconds to propagate, the identity should receive the required sensitive permissions for that service.

If this doesn't grant access, try temporarily Removing Controls and report your issue to Sonrai Customer Success.


JIT Session Doesn't Trigger Justification/Approval in Chat​

Workaround steps:

  • Ask the requester to invoke the chat command directly, using the /sonrai jit request command in Slack, or jit in Teams.
  • If an approval message still isn’t received via chat, have an approver check the Requests view in the Sonrai Web App and approve there.

If this doesn't grant access, try temporarily Removing Controls and report your issue to Sonrai Customer Success.


Attempted Zombie Wake-Up Without Chat Alert​

note

Only audited events in AWS will trigger a Zombie Permissions on Demand approval.

Workaround steps:

  • In the Services view, open your Quarantine List using the ☰ (three‑bar) menu located at the top‑right of your screen and search for the Zombie identity. Click X to unquarantine it. Expect usability to return within ~30 seconds.

If this doesn't grant access, try temporarily Removing Controls and report your issue to Sonrai Customer Success.

Report Outstanding ChatOps Issues to Sonrai Customer Success​

If the ChatOps issue persists, open a Support ticket noting:

ChatOps notifications are not working consistent with EventBridge trigger issues.

To ensure a rapid and accurate response to your support ticket, also include:

  • timestamps for requests
  • any Chat app error output
  • a list of the affected identities, account IDs, and services

We will investigate and determine what happened, and then work with you to resolve the issue.

Remove Controls using Sonrai CPF​

You can use the CPF application to remove all applied policy changes from your cloud environment.

warning

Temporarily removing controls reduces protections and guardrails in your system; keep the window as short as possible, and complete the Recovery and Re-Sync Controls process as soon as possible to restore CPF controls.

Removed controls are preserved as Pending Changes, so that your configuration can safely be re-deployed after the issue has been resolved. Do not delete the resulting Pending Changes after resetting CPF; you’ll use these changes to restore the configuration.

Prerequisites

  • Ensure that you are logged in to AWS using your management account. (Making changes to the Cloud Formation template for CPF relies on being logged in with an administrator role.)

  1. In the Cloud Permissions Firewall app, ensure that you have the correct organization selected.
  1. To remove the protections you have deployed to one of your organizations, open the menu while viewing Firewall Services for that organization, and click the Remove Permissions Firewall from the Org <your organization> option.

  1. Confirm that you want to proceed by typing undo in the text field and clicking .

    This triggers the removal of all elements from Cloud Permissions Firewall in your cloud, reverting it to its original pre-firewall state.

  1. If CPF controls have been deployed previously, then you will be prompted to remove the SonraiCloudPermissionFirewall stack in AWS Cloud Formation.

    Click the provided link to be taken to your CPF Cloud Formation Stack directly, or locate your stack manually in Cloud Formation.

  1. Delete your CPF Stack in AWS Cloud Formation.
  1. Cloud Permissions Firewall has returned your environment to its original state. Click when the process is finished.

Any control and policy changes that were rolled back are now available as Pending Changes that you can then review, remove, or redeploy as desired.

Emergency Stop from AWS​

In rare cases, the Sonrai CPF application may be unavailable or unresponsive - preventing you from removing controls using the web interface. In this scenario, an emergency stop script is available to quickly disable Sonrai policies.

warning

Temporarily removing controls reduces protections and guardrails in your system; keep the window as short as possible, and complete the Recovery and Re-Sync Controls process as soon as possible to restore CPF controls.

Prerequisites

  • Access to the AWS Management Account (or delegated admin with equivalent rights).
  • Ability to download from the Sonrai S3 bucket in your Management Account.

Locate Your Emergency Stop Script​

  1. Sign in to the AWS Management Account (or delegated admin).
  2. Locate the Sonrai S3 bucket with a naming pattern similar to:
s3://sonrai-crc<code><OrgId>-<YourManagementAccountId>

s3://sonrai-crc13294495-905418253619/org/crc13294495/cfTemplates/ 1750442020-783f9740-823b-4f17-a4ae-1ccaff3552d3/SonraiCPF.zip

  1. Inside the Sonrai S3 bucket, navigate to: org/crc<OrgId>/cfTemplates/
  2. From the most recent folder (excluding "current"), download the latest SonraiCPF.zip archive.
  3. Extract the CPFEmergencyStop.py python file from the archive.

Execute An Emergency Stop​

With your Org/Management Account environment (AWS CloudShell can be used if you do not have the AWS CLI set up), run:

python CPFEmergencyStop.py --action stop

You will be prompted to confirm that you want to remove all CPF controls, after which the script detaches all Sonrai‑named CMPs, SCPs, and RCPs.


note

Important note on drift: When controls are detached, configuration drift may occur. A resync using "Resubmit Existing Controls" via the Sonrai Web App is recommended after services are restored, as described in Recovery & Re‑Sync Controls.

Recovery & Re‑Sync Controls​

After taking any emergency action where CPF controls are temporarily disabled, it's important to re-sync and restore your controls from Pending Changes to ensure your cloud environment is protected.

  1. Redeploy CPF controls from Pending Changes:
    • In your Sonrai web application, open the Pending Changes view
    • Click to re-sync existing controls, or if you removed controls using the CPF application.
    • Monitor for successful apply across accounts
  2. Verify access and guardrails:
    • Confirm that identities regain Just‑in‑Time and Permission on Demand access
    • Spot‑check in target accounts/services
  3. Close the loop with Sonrai Support: Reach out and provide an incident summary and timing, helping us to prevent problems from reoccurring.

Break Glass Readiness (Preventative Measure)​

Once CPF protections are back in place, you might want to set up a specific set of identity exemptions that can be used in the event of emergency. Two options are provided below; reach out to Sonrai Customer Success to discuss other implementations for your specific needs.

Option 1: Create a scoped exemption at the top of your Org

Some customers choose to exempt the OrganizationalAccessRole. Example pattern in Exempt at Scope menu:

*e/OrganizationalAccessRole

Option 2: Exempt an SSO User

Another option is to exempt an SSO user that maps to an Administrative Permission Set in all accounts. You do this the same as the above example by creating a scoped exemption at the top of the Org and choosing SSO User.


Learn more about how and why you might want to exempt identities from CPF protections.

Last updated on