ngx_http_lua_module depends on the nginx development kit (NDK) for the
ndk_set_var_value symbol. Must install libnginx-mod-http-ndk and load
ndk_http_module.so before ngx_http_lua_module.so.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
libnginx-mod-http-lua is a dynamic module in Debian nginx; it must be
explicitly loaded with load_module or Lua directives are unrecognized.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
lua-resty-redis is an OpenResty library not packaged in Debian repos.
Download resty_redis.lua from upstream and install it to /usr/lib/lua/resty/redis.lua
at image build time. Update lua_package_path to match.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
OpenResty has no arm64 packages for Debian Bookworm; the official repo
only publishes amd64. Switch to Debian's own packages which support both
architectures and avoid any external repo setup:
nginx libnginx-mod-http-geoip2 libnginx-mod-http-lua lua-resty-redis
GeoIP2 C module returns (same nginx version → compatible). ASN/UA blocking
goes back to nginx if() blocks. blocklist.lua handles only the Redis checks
(prefix shared dict + pipelined GET for global/API block keys).
lua_package_path set to /usr/share/lua/5.1/ where Debian installs resty.*.
All paths revert to /etc/nginx/; start.sh reverts to /usr/sbin/nginx.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
apt 2.6 (Bookworm) accepts ASCII-armored keys via signed-by directly
when the file has a .asc extension. This removes the gpg pipeline whose
silent failure (no pipefail in /bin/sh) was producing an empty keyring.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The echo "..." \ <newline> > file pattern has the backslash consumed
by the Dockerfile parser, leaving a bare redirect that /bin/sh may not
handle reliably. Switch to pipe through tee for both writes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace nginx + libnginx-mod-http-geoip2 with OpenResty so that blocked
IPs are rejected before reaching Gunicorn, saving worker CPU on DDoS.
nginx layer (read-only, DB 1):
- blocklist.lua: UA check (nginx map var), ASN check (lua-resty-maxminddb,
replaces geoip2 C module), IP-prefix check (shared dict refreshed every
60s, 4-candidate O(1) lookup), pipelined GET for global IP block and
per-tenant API block. Parses REDIS_URL. Fail-open on Redis error.
- lua_shared_dict ip_prefix_blocked 1m: in-process prefix cache.
- init_by_lua_block: opens MaxMind ASN DB once in master process.
- init_worker_by_lua_block: refreshes prefix SET from Redis every 60s.
Django (ratelimit.py):
- _refresh_ip_prefix_blocklist: normalises entries to trailing-dot form
on load so per-request checks are O(1) set membership, not iteration.
- _is_ip_prefix_blocked: 4-candidate check (p1., p1.p2., p1.p2.p3., ip)
against the local set; same 60s refresh cadence as before.
Capacity (1,200 tenants, single Redis):
- Django pool: max_connections 6 → 3 (7,200 peak connections).
- nginx keepalive pool: 1 connection/worker (4,800 peak connections).
- Total: ~12,200 connections — 39% headroom under maxclients 20,000.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Coordinated DDoS attack used repeated page= params
(?page=3&page=2&page=3&...) as a scraping fingerprint, likely
harvested from SAPL's own paginated search links which had a
long-standing bug producing those polluted URLs.
Reject duplicate page= at the middleware layer:
RateLimitMiddleware.__call__ returns 400 (param_pollution) if
request.GET.getlist('page') has more than one value — before any
Redis or DB work runs, covering all paths universally.
PesquisarSessaoPlenariaView.get has the same check as a backstop.
Fix the root cause — page= leaking into filter_url on 9 search views:
All affected views built filter_url from the raw QUERY_STRING and
guarded with startswith("&page"), which only strips page= when it
is the first param. With ?filter=X&page=2 the page= leaked through
and paginacao.html produced ?page=N&filter=X&page=2 on every link.
Replaced with qr = request.GET.copy(); qr.pop('page', None).
Views fixed: PesquisarStatusTramitacaoView, PesquisarAssuntoNormaView,
PesquisarAuditLogView, PesquisarParlamentarView, PesquisarColigacaoView,
PesquisarPartidoView, ProtocoloPesquisaView,
PesquisarDocumentoAdministrativoView, PesquisarSessaoPlenariaView.
Cache anonymous GET on PesquisarSessaoPlenariaView (2 min TTL) to
reduce ORM load from repeated identical queries.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
SAPL pages fire 12-45 parallel requests; the old 30r/m nginx zone and
35/m Django threshold blocked normal navigation. Key changes:
nginx (nginx.conf / sapl.conf / start.sh):
- Split sapl_general (30r/m) into four dedicated zones:
sapl_general 90r/m burst=180 (HTML pages)
sapl_media 180r/m burst=180 (/media/ — own bucket, no longer drains general)
sapl_api 60r/m burst=120 (/api/ — quota layer is the real constraint)
sapl_heavy 10r/m burst=20 (/relatorios/ — unchanged, nodelay kept)
- /media/ and /api/ location blocks now reference their own zones
Django (settings.py):
- RATE_LIMITER_RATE: 35/m → 120/m
- RATE_LIMITER_RATE_AUTHENTICATED: 120/m → 240/m
- RATE_LIMIT_404_THRESHOLD: 10 → 20
- API_QUOTA_ANON_DAILY: 50 → 500 / weekly 350 → 3500
- API_QUOTA_AUTH_DAILY: 1000 → 5000 / weekly 7000 → 35000
Middleware (ratelimit.py):
- Authenticated users no longer receive a persistent 300s block key on
rate breach — they get 429 for the over-limit request and the window
resets naturally after 60s. A 5-minute lockout is wrong for a logged-in
user who clicked too fast.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Multiple councilmembers behind a shared NAT IP were getting 429s during
live votes because nginx burst exhaustion fired before Django's per-user
rate counter could run. Exempt /voto-individual/ and /sessao/<pk>/ from
nginx limit_req; mirror in RATE_LIMIT_BYPASS_PATHS as defense-in-depth.
Add docs/rate-limiter-incidents.md with root-cause analysis, architecture
diagrams, tradeoff discussion, and pending investigations for the
PatoBranco-PR 2026-05-06 incident.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- nginx: exempt /painel/<pk>/dados from rate limiting (polling endpoint,
will become WebSocket); dedicated location block with no limit_req
- ratelimit.py: bypass RATE_LIMIT_BYPASS_PATHS paths before _evaluate;
add layer=django to block log; increment daily Redis metrics counter
rl:metrics:{ns}:{date}:blocked:{reason} (TTL 8 days) on every block
- ratelimit.py: add quiltbot and AwarioBot to BOT_UA_FRAGMENTS
- ratelimit.py: fix _is_suspicious_headers to require missing UA before blocking
- settings: add RATE_LIMIT_BYPASS_PATHS with /painel/<pk>/dados pattern
- plan: extend UA blocklist SADD seed command with missing bot tokens
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- redis-configmap: move inline comment to its own line (Redis fatal parse error)
- settings: add CACHE_MIDDLEWARE_KEY_PREFIX='p' to remove double-dot in cache_page keys
- settings: monkey-patch _i18n_cache_key_suffix to strip pt-br/timezone suffix from keys
- ratelimit.py, settings: update example namespace from patobranco-pr to sapl31demo-df
- robots.txt: add AwarioSmartBot block
- plan: add rl:ip:*:blocked scan commands with TTL/value output
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Renames /private/media/ to /internal/media/ in nginx and serve_media().
Adds Content-Type and Content-Disposition to the X-Accel-Redirect response.
Replaces manual file reads in proposicao_texto and doc_texto_integral with
redirects to the media URL, removing the unused get_mime_type_from_file_extension helper.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- ConditionalGetMiddleware added to MIDDLEWARE (ETag/304 for all views)
- @condition(etag_func, last_modified_func) on MateriaLegislativa and
NormaJuridica detail views — skips view execution on cache hit via
data_ultima_atualizacao (auto_now=True) as freshness signal
- nginx /static/: expires 90m + Cache-Control public, max-age=5400
- nginx: removed upload-endpoint special-casing (location ~* ^/(protocoloadm/criar-protocolo|...))
- plan/RATE-LIMITER-PLAN.md updated to reflect all Phase 7 changes
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Remove docker/geoip/.gitignore so the .mmdb is versioned alongside the
codebase. The Dockerfile now exits non-zero if the file is missing,
making the build self-validating.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
docker/Dockerfile:
- GeoIP offline build with MaxMind secret; optional build args for
graphviz, poppler, psql client; envsubst for nginx burst vars
docker/docker-compose.yaml:
- saplredis service (redis:7-alpine, allkeys-lru, 512 MB)
- REDIS_URL + CACHE_BACKEND wired into sapl service
docker/startup_scripts/start.sh:
- configure_redis_cache(): builds CACHES dict, sets REDIS_CACHE waffle
switch, falls back to file cache gracefully
- POD_NAMESPACE resolution (k8s Downward API → hostname fallback)
- DATABASE_URL exported before migrate
docker/k8s/redis/ (moved from docker/k8s/):
- redis-configmap.yaml, redis-deployment.yaml, redis-service.yaml
- ClusterIP service on port 6379, sapl-redis namespace
docker/k8s/sapl-k8s.yaml:
- REDIS_URL env var injected; app.kubernetes.io/name=sapl label for
fleet-wide discovery
sapl/middleware/test_ratelimiter.py:
- Unit tests for RateLimitMiddleware with mocked Redis
scripts/test_ratelimiter.py:
- CLI smoke-test: fires N requests and reports first 429
Removed: rate-limiter-v2.md (content migrated to plan/RATE_LIMITER_PLAN.md),
scripts/test_ratelimiter.sh (replaced by .py),
docker/k8s/README.md (merged into plan/RATE_LIMITER_PLAN.md),
docker/scripts/redis_populate_test_data.py (renamed to redis_inject_test_data.py)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
nginx:
- /media/ proxied through Gunicorn (sapl_general rate limit) instead of
direct alias — Django middleware now runs on every media request
- /_accel/media/ internal location serves file bytes via X-Accel-Redirect
sapl/base/media.py (new):
- serve_media() gate: path traversal guard, auth redirect for
documentos_privados/, per-path Redis counter, content-type metadata
cache, X-Accel-Redirect response; falls back to Django serve() in DEBUG
sapl/middleware/ratelimit.py:
- RL_PATH_REQUESTS, RL_UA_BLOCKLIST, FILE_META_KEY constants
- _incr_with_ttl() extracted to module level (reused by media.py)
- Runtime UA deny list: _refresh_ua_blocklist() fetches rl:bot:ua:blocked
SET from Redis (SMEMBERS, cached per worker, TTL=RATE_LIMITER_UA_BLOCKLIST_REFRESH);
_is_redis_blocked_ua() tokenises UA and checks sha256 of each token
sapl/settings.py:
- RATE_LIMITER_UA_BLOCKLIST_REFRESH, MEDIA_PATH_COUNTER_TTL,
MEDIA_FILE_CACHE_TTL added (all env-tunable via config())
plan/RATE_LIMITER_PLAN.md:
- Key schema table updated; media file serving section added;
decision flow documented; UA deny list seed section expanded
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
MateriaLegislativaCrud.DetailView and SessaoCrud.DetailView are the two
highest-traffic public views not yet covered by anonymous page caching.
Both are read-only for anonymous visitors, making them safe cache targets.
- MateriaLegislativaCrud.DetailView: 300s TTL (PAGE_CACHE_TTL_DETAIL)
- SessaoCrud.DetailView: 120s TTL (PAGE_CACHE_TTL_LIST — sessions update
more frequently during active legislative sittings)
NormaCrud.DetailView intentionally left uncached: it writes NormaEstatisticas
on every access, and caching would suppress per-visit statistics for anonymous
users.
Also includes the RATELIMIT_DRY_RUN=False docker-compose.yaml change
from the previous session (rate limiting now enforced in docker-compose).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
waffle_switch REDIS_CACHE off --create always wrote 'off' even when the
operator had previously enabled it, making docker compose restart always
reset the switch and leaving Redis cache permanently disabled.
Replace with a Django shell get_or_create call that only inserts the row
with active=False on first boot; subsequent restarts leave the existing
value untouched.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
wait_for_pg() set DATABASE_URL via := default syntax but never exported it,
so Python child processes (manage.py migrate, waffle_switch) could not read
it from the environment. The .env file does not exist yet at that point —
write_env_file runs later — so decouple raised UndefinedValueError.
Add 'export DATABASE_URL' immediately after the default is resolved in
wait_for_pg(), which is the earliest point in the startup sequence where
the value is known and already used by configure_pg_timezone right after.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
GeoIP (docker/Dockerfile):
Remove at-build-time MaxMind download (required BuildKit secrets, caused
cache-miss issues). Replace with COPY from docker/geoip/GeoLite2-ASN.mmdb
(git-ignored binary). If absent, build succeeds with ASN blocking disabled.
Add docker/geoip/update_geoip.sh — run before each build to refresh the
database from MaxMind using MAXMIND_LICENSE_KEY from env or .env file.
Redis inspection / synthetic test data:
Add docker/scripts/redis_populate_test_data.py — injects synthetic rl:*
entries into Redis DB1 to validate key schema and blocking thresholds
without waiting for real traffic. Supports DRY_RUN and CLEAR modes.
Add §4.5 (Redis CLI quick-reference + RedisInsight guide) to rate-limiter-v2.md.
Auth-aware @ratelimit decorators (smart_rate / smart_key):
All 51 @ratelimit decorators across 9 files used rate=RATE_LIMITER_RATE
(35/m) regardless of authentication, silently over-throttling logged-in
users compared to what RateLimitMiddleware allows (120/m).
Add smart_key() and smart_rate() to sapl/middleware/ratelimit.py:
- smart_key: user pk for authenticated requests, masked IP for anon
- smart_rate: RATE_LIMITER_RATE_AUTHENTICATED (120/m) for auth,
RATE_LIMITER_RATE (35/m) for anon — mirrors middleware thresholds
Update all 51 decorators across crud/base.py + 8 view files.
Remove now-unused RATE_LIMITER_RATE imports from those files.
Cache KEY_PREFIX (settings.py):
Change KEY_PREFIX from POD_NAMESPACE ("sapl") to f"cache:{POD_NAMESPACE}"
so DB0 cache keys are unambiguously prefixed cache:{ns}:* — distinct from
any future static or file cache key patterns.
Update key schema table and code examples in rate-limiter-v2.md to match.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add AnonCachePageMixin (sapl/middleware/page_cache.py) that stores full
view responses in the default Redis cache for anonymous (unauthenticated)
GET requests only. Authenticated users always bypass the cache so CSRF
tokens and user-specific UI controls are never served stale.
Applied to:
- ParlamentarCrud.ListView / DetailView — TTL 600 s (changes each term)
- AudienciaCrud.ListView — TTL 120 s (hearings added infrequently)
- ComissaoCrud.ListView — TTL 300 s (committees change rarely)
Also:
- Add PAGE_CACHE_TTL_LIST/DETAIL/STABLE settings (env-configurable)
- Add bingbot + SERankingBacklinksBot to nginx UA blocklist (were already
in BOT_UA_FRAGMENTS / robots.txt; nginx map was the only gap)
- Remove unused ratelimit/method_decorator/RATE_LIMITER_RATE imports from
audiencia/views.py that crept in during Phase 2
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ratelimit.py: fix module docstring to reflect current _NAMESPACE resolution
(settings.POD_NAMESPACE, not K8s SA files read inside the middleware).
docker-compose.yaml:
- Add saplredis service (redis:7-alpine, no persistence, 512 MB maxmemory,
allkeys-lru, 4 databases, same policy as k8s ConfigMap).
- Add REDIS_URL=redis://saplredis:6379 and CACHE_BACKEND=redis to the
sapl service so local docker-compose runs use Redis out of the box.
- sapl depends_on now includes saplredis.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1. user_id: str(request.user.pk) — pk is int, lower()/strip() were no-ops
2. Redis key constants: RL_IP_REQUESTS, RL_IP_BLOCKED, RL_USER_REQUESTS,
RL_USER_BLOCKED, RL_NS_WINDOW — no more inline f-string literals
3. Tenant namespace: _NAMESPACE resolved once at module load from
POD_NAMESPACE env var (K8s Downward API) → service-account namespace
file → 'global' fallback. No per-request getattr(request, 'tenant').
4. KEY_PREFIX in CACHES['default'] set to POD_NAMESPACE (e.g. patobranco-pr)
so each tenant's cache keys are isolated in shared Redis.
5. Logger extra: replaced getattr(request, 'tenant', 'unknown') with
_NAMESPACE (the actual resolved constant).
settings.py: add POD_NAMESPACE = config('POD_NAMESPACE', default='sapl');
use it as KEY_PREFIX.
start.sh: add resolve_pod_namespace() (Downward API → SA file → fallback);
call it before resolve_redis_url(); write POD_NAMESPACE into .env.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Edward
joaohortsenado
LeandroJatai