Caches anonymous responses for 120 s (PAGE_CACHE_TTL_LIST), bypasses
cache for authenticated users so CSRF tokens and edit controls stay fresh.
Migrates PesquisarSessaoPlenariaView from cache_page (all-users) to
AnonCachePageMixin for consistency.
Views covered: PesquisarParlamentarView, PesquisarColigacaoView,
PesquisarPartidoView, PesquisarStatusTramitacaoView,
MateriaLegislativaPesquisaView, PesquisarAssuntoNormaView,
NormaPesquisaView, PesquisarSessaoPlenariaView.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Coordinated DDoS attack used repeated page= params
(?page=3&page=2&page=3&...) as a scraping fingerprint, likely
harvested from SAPL's own paginated search links which had a
long-standing bug producing those polluted URLs.
Reject duplicate page= at the middleware layer:
RateLimitMiddleware.__call__ returns 400 (param_pollution) if
request.GET.getlist('page') has more than one value — before any
Redis or DB work runs, covering all paths universally.
PesquisarSessaoPlenariaView.get has the same check as a backstop.
Fix the root cause — page= leaking into filter_url on 9 search views:
All affected views built filter_url from the raw QUERY_STRING and
guarded with startswith("&page"), which only strips page= when it
is the first param. With ?filter=X&page=2 the page= leaked through
and paginacao.html produced ?page=N&filter=X&page=2 on every link.
Replaced with qr = request.GET.copy(); qr.pop('page', None).
Views fixed: PesquisarStatusTramitacaoView, PesquisarAssuntoNormaView,
PesquisarAuditLogView, PesquisarParlamentarView, PesquisarColigacaoView,
PesquisarPartidoView, ProtocoloPesquisaView,
PesquisarDocumentoAdministrativoView, PesquisarSessaoPlenariaView.
Cache anonymous GET on PesquisarSessaoPlenariaView (2 min TTL) to
reduce ORM load from repeated identical queries.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
A permanent operator-curated deny decision (rl:ip_prefix:blocked) is not a
transient rate limit, so it should surface as 403 Forbidden with no
Retry-After rather than 429. Also brings RATE-LIMITER-PLAN.md up to date
with the IP-prefix blocklist feature and the same-origin bypass fix,
including redis-cli commands to populate/remove/list the deny-list set.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
_handle_api checked _is_same_origin() before any of the IP-prefix, global
(rl:ip:<ip>:blocked), and API-specific (rl:api:ip:<ip>:blocked) block keys —
short-circuiting straight to get_response() on a match. Since Origin/Referer
are entirely client-controlled and trivially spoofable, any caller could
defeat every /api/ block and counter (including an operator-set global block)
by simply sending Origin: https://<sapl-host>.
Reorder the checks so IP-prefix/global/API block lookups always run first;
the same-origin bypass now only exempts legitimate same-origin polling from
quota and per-minute rate-limit accounting, never from an active block.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Operators can now SADD/SREM dotted-decimal prefixes (e.g. "103.124.225")
into rl:ip_prefix:blocked to block entire ranges of abusive traffic
network-wide, mirroring the existing runtime UA deny-list pattern
(periodic SMEMBERS refresh into an in-process cache). Checked first in
both _handle_api and _evaluate, ahead of all auth-aware exemptions, so
it applies universally to anonymous and authenticated traffic alike.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- RelatorioMateriasTramitacao: strip UI-only params before deciding whether
to run a query, short-circuit RelatorioMixin.get() on permission checks,
remove redundant per-view @ratelimit decorators (RateLimitMiddleware
already covers them with stricter checks), and cache get_report_urls_map()
with lru_cache
- RelatorioMateriasTramitacaoView: rewrite the materia_materiaemtramitacao
view (migration 0088) to use DISTINCT ON instead of a correlated subquery,
add a composite index on materia_tramitacao(materia_id, id DESC), and drop
the now-redundant .distinct() from the filterset queryset
- customize_link_materia (MateriaOrdemDiaCrud / ExpedienteMateriaCrud):
eliminate per-row N+1 queries via select_related/prefetch_related with
to_attr caches, resolve sessao_plenaria once per page, and use a flat
paginate_by=100 instead of the count()-dependent 50/None toggle
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add RATE_LIMITER_INDEX_SHARDS setting (default 3): each blocked-IP write
routes to rl:index:blocked_ips:{shard} via md5(ip) % N, distributing
write contention across N keys.
- _BLOCK_LUA now runs ZREMRANGEBYSCORE before ZADD, pruning expired entries
from the target shard inline. Each shard stays bounded to active-only
members; no separate maintenance job needed.
- _index_shard(ip, index_base) computes the sharded key; all four _set_block
call sites updated.
- Fix 5 pre-existing test failures: suspicious-headers tests needed
HTTP_USER_AGENT removed; auth_user_rate block assertion corrected (no
persistent block key by design); ip_rate / ua_rotation tests now mock
_set_block directly instead of checking mock_cache.set.
- Update RATE-LIMITER-PLAN.md: key schema table, Redis CLI examples, and
ZSET index description reflect sharded keys and inline pruning.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
RATE_LIMIT_WHITELIST_IPS setting, self.whitelist bypass checks in
RateLimitMiddleware, and the corresponding test are removed. The
bypass_paths mechanism covers the legitimate high-frequency paths
(/painel, /sessao, /voto-individual); no IP-level exemption is needed.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
RL_API_IP_REQUESTS and RL_API_IP_BLOCKED now include {ns} so a block
in one k8s pod namespace does not leak into other tenants sharing the
same Redis instance. RL_INDEX_API_BLOCKED_IPS remains namespace-free
(operational index; members carry the namespace in their key string).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
At 35 req/min the old 500/day cap fired in ~14 min, making it
redundant with the per-minute block. The new values target slow-drip
scrapers (10–20 req/min sustained all day) while leaving legitimate
integrations (< 500/day) well within budget.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Forces script/integration owners toward sane polling intervals.
35/min is still well above any legitimate use case (a live session
panel at 10 s intervals needs only 6/min). Threshold remains
env-configurable (API_RATE_LIMIT_THRESHOLD) for future adjustment.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Authenticating must not bypass /api/ rate controls. The per-minute
threshold (60/min) and daily/weekly quota (500/day · 3 500/week) now
apply to all callers keyed by IP — auth status is not checked.
Auth users no longer fall through to _evaluate for /api/ requests;
_evaluate (240/min per-user) still governs all non-/api/ paths.
QUOTA_USER_DAILY/WEEKLY key templates removed (no longer written).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Auth users now share the same 500/day · 3 500/week cap as anonymous
callers. Auth quota is keyed by user pk (NAT-safe); anon quota by IP.
Both limits come from a single pair of settings (API_QUOTA_DAILY /
API_QUOTA_WEEKLY), replacing the anon-only API_QUOTA_ANON_* names.
QUOTA_USER_DAILY/WEEKLY Redis key templates are restored accordingly.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Auth users hitting /api/ are already governed by _evaluate's per-user
240 req/min rate limit (NAT-safe, scoped by user pk). The per-day/week
envelope (5 000/day · 35 000/week) fired too early for legitimate
integrations and added needless complexity.
Anonymous callers retain their quota (500/day · 3 500/week) since
they have no persistent per-user rate control.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Delete ApiEmergencySameSiteOnlyMiddleware (api_emergency_block.py);
replace with permanent _handle_api logic inside RateLimitMiddleware.
- _handle_api decision chain: OPTIONS pass → same-origin pass →
global IP block check → API-specific block check → quota → API
per-minute counter → anon pass / auth _evaluate.
- New Redis keys: rl:api:ip:<ip>:reqs, rl:api:ip:<ip>:blocked,
rl:index:api_blocked_ips. Global rl:ip:<ip>:blocked is never
written because of /api/ abuse — prevents NAT lockout.
- _is_same_origin: strips port, lowercases, checks Origin first then
Referer (sequential, not OR — wrong Origin blocks even if Referer matches).
- Five new settings: API_RATE_LIMIT_{ENABLED,THRESHOLD,WINDOW_SECONDS,
BLOCK_SECONDS,SAME_ORIGIN_BYPASS} with safe defaults.
- 16 new tests; _make_middleware extended with explicit setting values.
- RATE-LIMITER-PLAN.md updated with new key schema rows and _handle_api section.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Anonymous API requests now pass through after the quota check without
incrementing rl:ip:{ip}:reqs or writing a block key. A misbehaving
script or JS snippet behind a NAT IP can no longer lock out the org's
page requests by hammering /api/.
Enforcement for anonymous /api/:
- nginx sapl_api zone (60r/m, burst=120) — burst gate
- API quota (500/day, 3500/week) — daily cap
Authenticated /api/ still falls through to _evaluate_authenticated
(per-user counter keyed by uid, NAT-safe).
Interim measure until APP_ACCESS_KEYs per tenant org are introduced.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
SAPL pages fire 12-45 parallel requests; the old 30r/m nginx zone and
35/m Django threshold blocked normal navigation. Key changes:
nginx (nginx.conf / sapl.conf / start.sh):
- Split sapl_general (30r/m) into four dedicated zones:
sapl_general 90r/m burst=180 (HTML pages)
sapl_media 180r/m burst=180 (/media/ — own bucket, no longer drains general)
sapl_api 60r/m burst=120 (/api/ — quota layer is the real constraint)
sapl_heavy 10r/m burst=20 (/relatorios/ — unchanged, nodelay kept)
- /media/ and /api/ location blocks now reference their own zones
Django (settings.py):
- RATE_LIMITER_RATE: 35/m → 120/m
- RATE_LIMITER_RATE_AUTHENTICATED: 120/m → 240/m
- RATE_LIMIT_404_THRESHOLD: 10 → 20
- API_QUOTA_ANON_DAILY: 50 → 500 / weekly 350 → 3500
- API_QUOTA_AUTH_DAILY: 1000 → 5000 / weekly 7000 → 35000
Middleware (ratelimit.py):
- Authenticated users no longer receive a persistent 300s block key on
rate breach — they get 429 for the over-limit request and the window
resets naturally after 60s. A 5-minute lockout is wrong for a logged-in
user who clicked too fast.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Multiple councilmembers behind a shared NAT IP were getting 429s during
live votes because nginx burst exhaustion fired before Django's per-user
rate counter could run. Exempt /voto-individual/ and /sessao/<pk>/ from
nginx limit_req; mirror in RATE_LIMIT_BYPASS_PATHS as defense-in-depth.
Add docs/rate-limiter-incidents.md with root-cause analysis, architecture
diagrams, tradeoff discussion, and pending investigations for the
PatoBranco-PR 2026-05-06 incident.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Files under sapl/private/documentoadministrativo/ are public when the
AppConfig.documentos_administrativos setting is DOC_ADM_OSTENSIVO. The
previous gate blocked all sapl/private/ paths unconditionally, forcing
anonymous users to log in even for ostensivo documents.
_is_public_docadm() checks the cached AppConfig setting to exempt
ostensivo documents while keeping proposicao and restritivo documents
behind the auth redirect. Also fixes wrong import (sapl.base.apps.AppConfig
is Django's app-config class; the SAPL model is in sapl.base.models).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Each block-key write now also ZADDs the full key name into a permanent ZSET
(score = expiry unix timestamp) via a single Lua round-trip (_BLOCK_LUA).
Replaces four _rl_cache.set() calls with _set_block() which degrades to a
plain cache.set when Redis is unavailable. Indexes enable O(log N) enumeration
of active blocks (ZRANGEBYSCORE) without a SCAN; prunable with ZREMRANGEBYSCORE.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- @never_cache on get_dados_painel: ConditionalGetMiddleware was computing
an ETag from the unchanged JsonResponse body and returning 304 with no
body; the display always needs a live response.
- Guard logo src update: jQuery .attr("src", ...) fired a browser HTTP GET
on every 500 ms poll even when the URL hadn't changed — 120 media
requests/min per user, hitting the auth_threshold and triggering
user_blocked for painel operators.
- Fix poll scheduling: setTimeout was evaluated at $.ajax options
construction time, scheduling the next poll 500 ms after the request
started rather than after it finished. Slow responses (>500 ms) stacked
concurrent in-flight requests, creating a self-amplifying load loop and
a DOM race condition where older responses could overwrite newer ones.
Moved to a proper complete callback so at most one request is in-flight
at any time.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- nginx: exempt /painel/<pk>/dados from rate limiting (polling endpoint,
will become WebSocket); dedicated location block with no limit_req
- ratelimit.py: bypass RATE_LIMIT_BYPASS_PATHS paths before _evaluate;
add layer=django to block log; increment daily Redis metrics counter
rl:metrics:{ns}:{date}:blocked:{reason} (TTL 8 days) on every block
- ratelimit.py: add quiltbot and AwarioBot to BOT_UA_FRAGMENTS
- ratelimit.py: fix _is_suspicious_headers to require missing UA before blocking
- settings: add RATE_LIMIT_BYPASS_PATHS with /painel/<pk>/dados pattern
- plan: extend UA blocklist SADD seed command with missing bot tokens
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- redis-configmap: move inline comment to its own line (Redis fatal parse error)
- settings: add CACHE_MIDDLEWARE_KEY_PREFIX='p' to remove double-dot in cache_page keys
- settings: monkey-patch _i18n_cache_key_suffix to strip pt-br/timezone suffix from keys
- ratelimit.py, settings: update example namespace from patobranco-pr to sapl31demo-df
- robots.txt: add AwarioSmartBot block
- plan: add rl:ip:*:blocked scan commands with TTL/value output
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Renames /private/media/ to /internal/media/ in nginx and serve_media().
Adds Content-Type and Content-Disposition to the X-Accel-Redirect response.
Replaces manual file reads in proposicao_texto and doc_texto_integral with
redirects to the media URL, removing the unused get_mime_type_from_file_extension helper.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- _norma_last_modified: use ultima_edicao instead of data_ultima_atualizacao
- serve_media: gate on sapl/private/ instead of documentos_privados/
- plan: update norma freshness field reference
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- ConditionalGetMiddleware added to MIDDLEWARE (ETag/304 for all views)
- @condition(etag_func, last_modified_func) on MateriaLegislativa and
NormaJuridica detail views — skips view execution on cache hit via
data_ultima_atualizacao (auto_now=True) as freshness signal
- nginx /static/: expires 90m + Cache-Control public, max-age=5400
- nginx: removed upload-endpoint special-casing (location ~* ^/(protocoloadm/criar-protocolo|...))
- plan/RATE-LIMITER-PLAN.md updated to reflect all Phase 7 changes
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Remove docker/geoip/.gitignore so the .mmdb is versioned alongside the
codebase. The Dockerfile now exits non-zero if the file is missing,
making the build self-validating.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
docker/Dockerfile:
- GeoIP offline build with MaxMind secret; optional build args for
graphviz, poppler, psql client; envsubst for nginx burst vars
docker/docker-compose.yaml:
- saplredis service (redis:7-alpine, allkeys-lru, 512 MB)
- REDIS_URL + CACHE_BACKEND wired into sapl service
docker/startup_scripts/start.sh:
- configure_redis_cache(): builds CACHES dict, sets REDIS_CACHE waffle
switch, falls back to file cache gracefully
- POD_NAMESPACE resolution (k8s Downward API → hostname fallback)
- DATABASE_URL exported before migrate
docker/k8s/redis/ (moved from docker/k8s/):
- redis-configmap.yaml, redis-deployment.yaml, redis-service.yaml
- ClusterIP service on port 6379, sapl-redis namespace
docker/k8s/sapl-k8s.yaml:
- REDIS_URL env var injected; app.kubernetes.io/name=sapl label for
fleet-wide discovery
sapl/middleware/test_ratelimiter.py:
- Unit tests for RateLimitMiddleware with mocked Redis
scripts/test_ratelimiter.py:
- CLI smoke-test: fires N requests and reports first 429
Removed: rate-limiter-v2.md (content migrated to plan/RATE_LIMITER_PLAN.md),
scripts/test_ratelimiter.sh (replaced by .py),
docker/k8s/README.md (merged into plan/RATE_LIMITER_PLAN.md),
docker/scripts/redis_populate_test_data.py (renamed to redis_inject_test_data.py)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
nginx:
- /media/ proxied through Gunicorn (sapl_general rate limit) instead of
direct alias — Django middleware now runs on every media request
- /_accel/media/ internal location serves file bytes via X-Accel-Redirect
sapl/base/media.py (new):
- serve_media() gate: path traversal guard, auth redirect for
documentos_privados/, per-path Redis counter, content-type metadata
cache, X-Accel-Redirect response; falls back to Django serve() in DEBUG
sapl/middleware/ratelimit.py:
- RL_PATH_REQUESTS, RL_UA_BLOCKLIST, FILE_META_KEY constants
- _incr_with_ttl() extracted to module level (reused by media.py)
- Runtime UA deny list: _refresh_ua_blocklist() fetches rl:bot:ua:blocked
SET from Redis (SMEMBERS, cached per worker, TTL=RATE_LIMITER_UA_BLOCKLIST_REFRESH);
_is_redis_blocked_ua() tokenises UA and checks sha256 of each token
sapl/settings.py:
- RATE_LIMITER_UA_BLOCKLIST_REFRESH, MEDIA_PATH_COUNTER_TTL,
MEDIA_FILE_CACHE_TTL added (all env-tunable via config())
plan/RATE_LIMITER_PLAN.md:
- Key schema table updated; media file serving section added;
decision flow documented; UA deny list seed section expanded
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
django-ratelimit (the @ratelimit view decorator) writes its counters
through Django's cache framework. Django's default key function produces
'{KEY_PREFIX}:{VERSION}:{key}', so with KEY_PREFIX='' and VERSION=1
(defaults) the decorator keys appear in Redis as ':1:rl:{hash}' — an
ugly leading colon that makes them look distinct from the clean 'rl:*'
keys written by RateLimitMiddleware via get_redis_connection().
Add make_ratelimit_cache_key() to sapl/middleware/ratelimit.py (a simple
pass-through) and wire it into the 'ratelimit' cache config via
KEY_FUNCTION. Both key families now share the same 'rl:*' namespace:
decorator keys → rl:{hash}
middleware keys → rl:ip:{ip}:reqs
rl:{ns}:user:{uid}:reqs
rl:{ns}:ip:{ip}:w:{bucket}
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
MateriaLegislativaCrud.DetailView and SessaoCrud.DetailView are the two
highest-traffic public views not yet covered by anonymous page caching.
Both are read-only for anonymous visitors, making them safe cache targets.
- MateriaLegislativaCrud.DetailView: 300s TTL (PAGE_CACHE_TTL_DETAIL)
- SessaoCrud.DetailView: 120s TTL (PAGE_CACHE_TTL_LIST — sessions update
more frequently during active legislative sittings)
NormaCrud.DetailView intentionally left uncached: it writes NormaEstatisticas
on every access, and caching would suppress per-visit statistics for anonymous
users.
Also includes the RATELIMIT_DRY_RUN=False docker-compose.yaml change
from the previous session (rate limiting now enforced in docker-compose).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
waffle_switch REDIS_CACHE off --create always wrote 'off' even when the
operator had previously enabled it, making docker compose restart always
reset the switch and leaving Redis cache permanently disabled.
Replace with a Django shell get_or_create call that only inserts the row
with active=False on first boot; subsequent restarts leave the existing
value untouched.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
wait_for_pg() set DATABASE_URL via := default syntax but never exported it,
so Python child processes (manage.py migrate, waffle_switch) could not read
it from the environment. The .env file does not exist yet at that point —
write_env_file runs later — so decouple raised UndefinedValueError.
Add 'export DATABASE_URL' immediately after the default is resolved in
wait_for_pg(), which is the earliest point in the startup sequence where
the value is known and already used by configure_pg_timezone right after.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
