Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/cloud-armor-edge-refusal.changed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Requests to api.policyengine.org without an Authorization header, outside the Public API's public routes, are refused at the load balancer with 403 instead of reaching the Public API, which answers them with 401; the Cloud Armor runbook records the rules.
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
creationTimestamp: '2026-07-21T07:16:39.177-07:00'
description: Rate limiting for api.policyengine.org (lb-api). See docs/migration/lb-cloud-armor-runbook.md
fingerprint: w1MANeduEPA=
id: '200433714284642728'
kind: compute#securityPolicy
labelFingerprint: 42WmSpB8rSM=
name: pol-api-lb
rules:
- action: throttle
description: ''
kind: compute#securityPolicyRule
match:
expr:
expression: request.path.matches('^/[a-z]{2}/metadata$')
preview: false
priority: 1000
rateLimitOptions:
conformAction: allow
enforceOnKey: IP
exceedAction: deny(429)
rateLimitThreshold:
count: 30
intervalSec: 60
- action: throttle
description: ''
kind: compute#securityPolicyRule
match:
expr:
expression: request.path.matches('^/[a-z]{2}/calculate(?:-full)?$')
preview: false
priority: 1100
rateLimitOptions:
conformAction: allow
enforceOnKey: IP
exceedAction: deny(429)
rateLimitThreshold:
count: 75
intervalSec: 60
- action: allow
description: default rule
kind: compute#securityPolicyRule
match:
config:
srcIpRanges:
- '*'
versionedExpr: SRC_IPS_V1
preview: false
priority: 2147483647
- action: allow
description: Public route GET/HEAD /health served without a credential (API lockdown
5.1 · 4)
kind: compute#securityPolicyRule
match:
expr:
expression: request.path == '/health' && (request.method == 'GET' || request.method
== 'HEAD')
preview: false
priority: 770
- action: allow
description: Public route GET /v2/openapi.json served without a credential (API
lockdown 5.1 · 4)
kind: compute#securityPolicyRule
match:
expr:
expression: request.path == '/v2/openapi.json' && request.method == 'GET'
preview: false
priority: 771
- action: allow
description: CORS preflights served without a credential (API lockdown 5.1 · 4)
kind: compute#securityPolicyRule
match:
expr:
expression: request.method == 'OPTIONS' && has(request.headers['origin']) &&
has(request.headers['access-control-request-method'])
preview: false
priority: 772
- action: deny(403)
description: Refuse requests without Authorization outside the public routes; API
v1 refuses them too (API lockdown 5.1 · 4)
kind: compute#securityPolicyRule
match:
expr:
expression: "!has(request.headers['authorization']) || request.headers['authorization'].matches('^[[:blank:]]*$')"
preview: true
priority: 800
- action: allow
description: Requests carrying a bearer token skip the per-address /metadata and
/calculate limits; API v1 refuses any that fails validation (API lockdown 2.2
· 5). See docs/migration/lb-cloud-armor-runbook.md
kind: compute#securityPolicyRule
match:
expr:
expression: request.path.matches('^/[a-z]{2}/(?:metadata|calculate(?:-full)?)$')
&& has(request.headers['authorization']) && request.headers['authorization'].matches('^[Bb]earer
[A-Za-z0-9._~+/-]+=*$')
preview: false
priority: 900
selfLink: https://www.googleapis.com/compute/v1/projects/policyengine-api/global/securityPolicies/pol-api-lb
type: CLOUD_ARMOR
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
creationTimestamp: '2026-07-21T07:16:39.177-07:00'
description: Rate limiting for api.policyengine.org (lb-api). See docs/migration/lb-cloud-armor-runbook.md
fingerprint: JyTEO8FdBD8=
id: '200433714284642728'
kind: compute#securityPolicy
labelFingerprint: 42WmSpB8rSM=
name: pol-api-lb
rules:
- action: throttle
description: ''
kind: compute#securityPolicyRule
match:
expr:
expression: request.path.matches('^/[a-z]{2}/metadata$')
preview: false
priority: 1000
rateLimitOptions:
conformAction: allow
enforceOnKey: IP
exceedAction: deny(429)
rateLimitThreshold:
count: 30
intervalSec: 60
- action: throttle
description: ''
kind: compute#securityPolicyRule
match:
expr:
expression: request.path.matches('^/[a-z]{2}/calculate(?:-full)?$')
preview: false
priority: 1100
rateLimitOptions:
conformAction: allow
enforceOnKey: IP
exceedAction: deny(429)
rateLimitThreshold:
count: 75
intervalSec: 60
- action: allow
description: default rule
kind: compute#securityPolicyRule
match:
config:
srcIpRanges:
- '*'
versionedExpr: SRC_IPS_V1
preview: false
priority: 2147483647
- action: allow
description: Public route GET/HEAD /health served without a credential (API lockdown
5.1 · 4)
kind: compute#securityPolicyRule
match:
expr:
expression: request.path == '/health' && (request.method == 'GET' || request.method
== 'HEAD')
preview: false
priority: 770
- action: allow
description: Public route GET /v2/openapi.json served without a credential (API
lockdown 5.1 · 4)
kind: compute#securityPolicyRule
match:
expr:
expression: request.path == '/v2/openapi.json' && request.method == 'GET'
preview: false
priority: 771
- action: allow
description: CORS preflights served without a credential (API lockdown 5.1 · 4)
kind: compute#securityPolicyRule
match:
expr:
expression: request.method == 'OPTIONS' && has(request.headers['origin']) &&
has(request.headers['access-control-request-method'])
preview: false
priority: 772
- action: deny(403)
description: Refuse requests without Authorization outside the public routes; API
v1 refuses them too (API lockdown 5.1 · 4)
kind: compute#securityPolicyRule
match:
expr:
expression: "!has(request.headers['authorization']) || request.headers['authorization'].matches('^[[:blank:]]*$')"
preview: false
priority: 800
- action: allow
description: Requests carrying a bearer token skip the per-address /metadata and
/calculate limits; API v1 refuses any that fails validation (API lockdown 2.2
· 5). See docs/migration/lb-cloud-armor-runbook.md
kind: compute#securityPolicyRule
match:
expr:
expression: request.path.matches('^/[a-z]{2}/(?:metadata|calculate(?:-full)?)$')
&& has(request.headers['authorization']) && request.headers['authorization'].matches('^[Bb]earer
[A-Za-z0-9._~+/-]+=*$')
preview: false
priority: 900
selfLink: https://www.googleapis.com/compute/v1/projects/policyengine-api/global/securityPolicies/pol-api-lb
type: CLOUD_ARMOR
97 changes: 80 additions & 17 deletions docs/migration/lb-cloud-armor-runbook.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,14 +11,17 @@ record of what is currently deployed is the newest exported snapshot in
| | |
|---|---|
| Policy | `pol-api-lb`, attached to the Cloud Run backend service of the public API load balancer |
| Rules | Per-IP throttles on the metadata and calculate path families; thresholds in the snapshot |
| Mode | metadata rule **ENFORCED** 2026-07-21 (worst legit observed 5/min vs 30 threshold — 6× headroom); calculate rule **stays in preview at 75/60s** — observed legit/partner clients run 39–88/min there, so enforcing would 429 real use (incl. the partner API fallback) |

Note on preview validation: Cloud Armor **throttle** rules in preview only ever
log `CONFORM` (never `EXCEEDED`), so the `previewSecurityPolicy` outcome field
cannot be used to count would-be-throttled requests. Judge a throttle rule's
enforcement readiness from **raw per-IP request-rate analysis of LB logs**, not
from preview outcome counts.
| Rules | Per-IP throttles on the metadata and calculate path families (1000, 1100; thresholds in the snapshot); a bearer-token exemption from them (900); a 403 for requests without `Authorization` outside the Public API's public routes (770–800) |
| Mode | metadata rule **ENFORCED** 2026-07-21 (worst legit observed 5/min vs 30 threshold — 6× headroom); calculate rule **ENFORCED** 2026-11-09 15:01 UTC (it had stayed in preview because observed legit/partner clients ran 39–88/min there, including a partner API fallback that was removed in July 2026); rules 900 and 770–800 enforced |

Note on preview validation: Cloud Armor **throttle** rules in preview do log
would-be throttling. `jsonPayload.previewSecurityPolicy.outcome` is
`RATE_LIMIT_THRESHOLD_EXCEED` for a request over the threshold: in the 30 days
before rule 1100 was enforced, its preview logged that outcome for 29,037 of
162,011 matching requests. (This note once said preview logs only `CONFORM`.)
Count those outcomes, and still judge a throttle rule's enforcement readiness
from **raw per-IP request-rate analysis of LB logs**, which shows who would be
throttled and by how much.

Origin: overnight bot waves saturated backends / churned autoscaling
(2026-07-13 → 2026-07-21 incidents; details in the cutover execution plan).
Expand Down Expand Up @@ -47,7 +50,8 @@ Origin: overnight bot waves saturated backends / churned autoscaling
## Known caveat

Cloud Armor only sees traffic that crosses the load balancer. Serverless
default URLs (`*.run.app`) bypass it entirely. This is
default URLs (`*.run.app`) bypass it entirely, including rule 800's refusal of
requests without a credential; the Public API refuses those itself. This is
acceptable while ingress must stay open for CI smoke tests; revisit when
ingress is locked down to the LB.

Expand All @@ -68,15 +72,16 @@ client. Those routes apply their own per-browser limits on Vercel.
made-up token is not served. Apply rule 900 only while that check is
deployed.
- **Why the exact pattern:** the Public API treats a blank `Authorization`
header as no credential and serves it, so a rule that only checked the
header's presence would let blank headers skip the throttle and still be
served. Two headers reach Cloud Armor joined by a comma, which the pattern
header as no credential, so a rule that only checked the header's presence
would let blank headers skip the throttle (and, before 2026-11-17, be
served). Two headers reach Cloud Armor joined by a comma, which the pattern
rejects.
- **What it gives up:** until the Public API refuses requests without a
server credential, a signed-in user's own valid token in `Authorization`
skips the throttle and is served; requests with made-up tokens are not
capped at the load balancer, though each costs the Public API one
signature check.
- **What it gives up:** requests with made-up tokens are not capped at the
load balancer, though each costs the Public API one signature check. Before
the Public API refused requests without a server credential (2026-11-17), a
signed-in user's own valid token in `Authorization` also skipped the
throttle and was served; since then the Public API refuses it outside its
public routes.

Check it with requests of your own: 40 requests a minute to
`/us/metadata` with `Authorization: Bearer made-up` should all get 401 and
Expand All @@ -85,6 +90,64 @@ none 429, and the load balancer log should show
`gcloud compute security-policies rules update 900 --preview` (back to
log-only) or `rules delete 900`, then export a snapshot.

## Requests without a credential (rules 770–800)

Since 2026-11-17 the Public API runs with
`PUBLIC_API_SERVER_CREDENTIAL_REQUIRED=1`: it refuses every request without a
server credential with 401 before routing, except its public routes. Since the
same day, rule 800 refuses at the load balancer, with 403, the requests among
those that carry no `Authorization` header or a blank one (only spaces and
tabs). Rules 770–772 allow the public routes first:

| Priority | Action | Matches |
|---|---|---|
| 770 | allow | `GET` and `HEAD /health` |
| 771 | allow | `GET /v2/openapi.json` |
| 772 | allow | CORS preflights: `OPTIONS` with `Origin` and `Access-Control-Request-Method` |
| 800 | `deny(403)` | `!has(request.headers['authorization']) \|\| request.headers['authorization'].matches('^[[:blank:]]*$')` |

Rule 800 refuses only requests the Public API would refuse anyway, so they
no longer reach Cloud Run.

- **What callers see:** Cloud Armor can only deny with 403, 404 or 502, and
its 403 is Google's HTML page. Through `api.policyengine.org`, a request
without a credential gets that page, not the Public API's 401 with
`WWW-Authenticate: Bearer` and a JSON body. Requests sent to the `*.run.app`
URLs skip the load balancer and get the Public API's 401.
- **Tokens:** Cloud Armor can't validate a token, so any non-blank
`Authorization` header passes rule 800 and the Public API checks it. A
made-up token still gets the Public API's 401.
- **Keep the allow rules matched to the Public API.** Rules 770–772 copy
`PUBLIC_ROUTES` and `is_public_request` in
`policyengine_api/request_caller.py`. A route made public there must be
allowed here before rule 800, or the load balancer refuses it.
- **Apply rule 800 only while enforcement is on.** With
`PUBLIC_API_SERVER_CREDENTIAL_REQUIRED` off, the Public API serves requests
without a credential, and rule 800 would refuse them.
- **Preflights and the throttles:** rule 772 matches before rules 1000 and
1100, so CORS preflights to the metadata and calculate paths no longer count
toward those per-address limits. The Public API's CORS middleware answers a
preflight without running a route.
- **The blank pattern:** Cloud Armor refused `'^[ \t]*$'` with "Invalid
Argument"; the POSIX class `'^[[:blank:]]*$'` (space or tab) is accepted.

Rule 800 ran in preview for 2 hours before it was enforced, shorter than the
24 hours under "Preview-to-enforcement checks", by decision. It logged 264
would-be refusals, and the Public API had answered every one of them with 401.
A deny rule in preview logs `DENY` in
`jsonPayload.previewSecurityPolicy.outcome`; group those entries by
`httpRequest.status` to see whether the Public API served any of them.

Check it with requests of your own, without a credential, to
`api.policyengine.org`: `/`, `/us/metadata`, `POST /us/calculate` and `/v2/me`
get 403 with `text/html`; `GET` and `HEAD /health`, `/v2/openapi.json` and a
CORS preflight get 200; `Authorization: Bearer made-up` gets the Public API's
401. The load balancer log shows `jsonPayload.enforcedSecurityPolicy.priority=800`
with outcome `DENY` for the refused requests. Roll it back with
`gcloud compute security-policies rules update 800 --preview` (back to
log-only, within minutes) or `rules delete 800`, then export a snapshot. Rules
770–772 can stay; deleting 772 puts preflights back under the throttles.

## Procedures

Set once per shell:
Expand Down
Loading