- Add RATE_LIMITER_INDEX_SHARDS setting (default 3): each blocked-IP write
routes to rl:index:blocked_ips:{shard} via md5(ip) % N, distributing
write contention across N keys.
- _BLOCK_LUA now runs ZREMRANGEBYSCORE before ZADD, pruning expired entries
from the target shard inline. Each shard stays bounded to active-only
members; no separate maintenance job needed.
- _index_shard(ip, index_base) computes the sharded key; all four _set_block
call sites updated.
- Fix 5 pre-existing test failures: suspicious-headers tests needed
HTTP_USER_AGENT removed; auth_user_rate block assertion corrected (no
persistent block key by design); ip_rate / ua_rotation tests now mock
_set_block directly instead of checking mock_cache.set.
- Update RATE-LIMITER-PLAN.md: key schema table, Redis CLI examples, and
ZSET index description reflect sharded keys and inline pruning.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Delete ApiEmergencySameSiteOnlyMiddleware (api_emergency_block.py);
replace with permanent _handle_api logic inside RateLimitMiddleware.
- _handle_api decision chain: OPTIONS pass → same-origin pass →
global IP block check → API-specific block check → quota → API
per-minute counter → anon pass / auth _evaluate.
- New Redis keys: rl:api:ip:<ip>:reqs, rl:api:ip:<ip>:blocked,
rl:index:api_blocked_ips. Global rl:ip:<ip>:blocked is never
written because of /api/ abuse — prevents NAT lockout.
- _is_same_origin: strips port, lowercases, checks Origin first then
Referer (sequential, not OR — wrong Origin blocks even if Referer matches).
- Five new settings: API_RATE_LIMIT_{ENABLED,THRESHOLD,WINDOW_SECONDS,
BLOCK_SECONDS,SAME_ORIGIN_BYPASS} with safe defaults.
- 16 new tests; _make_middleware extended with explicit setting values.
- RATE-LIMITER-PLAN.md updated with new key schema rows and _handle_api section.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Each block-key write now also ZADDs the full key name into a permanent ZSET
(score = expiry unix timestamp) via a single Lua round-trip (_BLOCK_LUA).
Replaces four _rl_cache.set() calls with _set_block() which degrades to a
plain cache.set when Redis is unavailable. Indexes enable O(log N) enumeration
of active blocks (ZRANGEBYSCORE) without a SCAN; prunable with ZREMRANGEBYSCORE.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- nginx: exempt /painel/<pk>/dados from rate limiting (polling endpoint,
will become WebSocket); dedicated location block with no limit_req
- ratelimit.py: bypass RATE_LIMIT_BYPASS_PATHS paths before _evaluate;
add layer=django to block log; increment daily Redis metrics counter
rl:metrics:{ns}:{date}:blocked:{reason} (TTL 8 days) on every block
- ratelimit.py: add quiltbot and AwarioBot to BOT_UA_FRAGMENTS
- ratelimit.py: fix _is_suspicious_headers to require missing UA before blocking
- settings: add RATE_LIMIT_BYPASS_PATHS with /painel/<pk>/dados pattern
- plan: extend UA blocklist SADD seed command with missing bot tokens
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- redis-configmap: move inline comment to its own line (Redis fatal parse error)
- settings: add CACHE_MIDDLEWARE_KEY_PREFIX='p' to remove double-dot in cache_page keys
- settings: monkey-patch _i18n_cache_key_suffix to strip pt-br/timezone suffix from keys
- ratelimit.py, settings: update example namespace from patobranco-pr to sapl31demo-df
- robots.txt: add AwarioSmartBot block
- plan: add rl:ip:*:blocked scan commands with TTL/value output
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Renames /private/media/ to /internal/media/ in nginx and serve_media().
Adds Content-Type and Content-Disposition to the X-Accel-Redirect response.
Replaces manual file reads in proposicao_texto and doc_texto_integral with
redirects to the media URL, removing the unused get_mime_type_from_file_extension helper.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- _norma_last_modified: use ultima_edicao instead of data_ultima_atualizacao
- serve_media: gate on sapl/private/ instead of documentos_privados/
- plan: update norma freshness field reference
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- ConditionalGetMiddleware added to MIDDLEWARE (ETag/304 for all views)
- @condition(etag_func, last_modified_func) on MateriaLegislativa and
NormaJuridica detail views — skips view execution on cache hit via
data_ultima_atualizacao (auto_now=True) as freshness signal
- nginx /static/: expires 90m + Cache-Control public, max-age=5400
- nginx: removed upload-endpoint special-casing (location ~* ^/(protocoloadm/criar-protocolo|...))
- plan/RATE-LIMITER-PLAN.md updated to reflect all Phase 7 changes
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
nginx:
- /media/ proxied through Gunicorn (sapl_general rate limit) instead of
direct alias — Django middleware now runs on every media request
- /_accel/media/ internal location serves file bytes via X-Accel-Redirect
sapl/base/media.py (new):
- serve_media() gate: path traversal guard, auth redirect for
documentos_privados/, per-path Redis counter, content-type metadata
cache, X-Accel-Redirect response; falls back to Django serve() in DEBUG
sapl/middleware/ratelimit.py:
- RL_PATH_REQUESTS, RL_UA_BLOCKLIST, FILE_META_KEY constants
- _incr_with_ttl() extracted to module level (reused by media.py)
- Runtime UA deny list: _refresh_ua_blocklist() fetches rl:bot:ua:blocked
SET from Redis (SMEMBERS, cached per worker, TTL=RATE_LIMITER_UA_BLOCKLIST_REFRESH);
_is_redis_blocked_ua() tokenises UA and checks sha256 of each token
sapl/settings.py:
- RATE_LIMITER_UA_BLOCKLIST_REFRESH, MEDIA_PATH_COUNTER_TTL,
MEDIA_FILE_CACHE_TTL added (all env-tunable via config())
plan/RATE_LIMITER_PLAN.md:
- Key schema table updated; media file serving section added;
decision flow documented; UA deny list seed section expanded
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Edward Ribeiro