Files under sapl/private/documentoadministrativo/ are public when the
AppConfig.documentos_administrativos setting is DOC_ADM_OSTENSIVO. The
previous gate blocked all sapl/private/ paths unconditionally, forcing
anonymous users to log in even for ostensivo documents.
_is_public_docadm() checks the cached AppConfig setting to exempt
ostensivo documents while keeping proposicao and restritivo documents
behind the auth redirect. Also fixes wrong import (sapl.base.apps.AppConfig
is Django's app-config class; the SAPL model is in sapl.base.models).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Each block-key write now also ZADDs the full key name into a permanent ZSET
(score = expiry unix timestamp) via a single Lua round-trip (_BLOCK_LUA).
Replaces four _rl_cache.set() calls with _set_block() which degrades to a
plain cache.set when Redis is unavailable. Indexes enable O(log N) enumeration
of active blocks (ZRANGEBYSCORE) without a SCAN; prunable with ZREMRANGEBYSCORE.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- @never_cache on get_dados_painel: ConditionalGetMiddleware was computing
an ETag from the unchanged JsonResponse body and returning 304 with no
body; the display always needs a live response.
- Guard logo src update: jQuery .attr("src", ...) fired a browser HTTP GET
on every 500 ms poll even when the URL hadn't changed — 120 media
requests/min per user, hitting the auth_threshold and triggering
user_blocked for painel operators.
- Fix poll scheduling: setTimeout was evaluated at $.ajax options
construction time, scheduling the next poll 500 ms after the request
started rather than after it finished. Slow responses (>500 ms) stacked
concurrent in-flight requests, creating a self-amplifying load loop and
a DOM race condition where older responses could overwrite newer ones.
Moved to a proper complete callback so at most one request is in-flight
at any time.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- nginx: exempt /painel/<pk>/dados from rate limiting (polling endpoint,
will become WebSocket); dedicated location block with no limit_req
- ratelimit.py: bypass RATE_LIMIT_BYPASS_PATHS paths before _evaluate;
add layer=django to block log; increment daily Redis metrics counter
rl:metrics:{ns}:{date}:blocked:{reason} (TTL 8 days) on every block
- ratelimit.py: add quiltbot and AwarioBot to BOT_UA_FRAGMENTS
- ratelimit.py: fix _is_suspicious_headers to require missing UA before blocking
- settings: add RATE_LIMIT_BYPASS_PATHS with /painel/<pk>/dados pattern
- plan: extend UA blocklist SADD seed command with missing bot tokens
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- redis-configmap: move inline comment to its own line (Redis fatal parse error)
- settings: add CACHE_MIDDLEWARE_KEY_PREFIX='p' to remove double-dot in cache_page keys
- settings: monkey-patch _i18n_cache_key_suffix to strip pt-br/timezone suffix from keys
- ratelimit.py, settings: update example namespace from patobranco-pr to sapl31demo-df
- robots.txt: add AwarioSmartBot block
- plan: add rl:ip:*:blocked scan commands with TTL/value output
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Renames /private/media/ to /internal/media/ in nginx and serve_media().
Adds Content-Type and Content-Disposition to the X-Accel-Redirect response.
Replaces manual file reads in proposicao_texto and doc_texto_integral with
redirects to the media URL, removing the unused get_mime_type_from_file_extension helper.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- _norma_last_modified: use ultima_edicao instead of data_ultima_atualizacao
- serve_media: gate on sapl/private/ instead of documentos_privados/
- plan: update norma freshness field reference
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- ConditionalGetMiddleware added to MIDDLEWARE (ETag/304 for all views)
- @condition(etag_func, last_modified_func) on MateriaLegislativa and
NormaJuridica detail views — skips view execution on cache hit via
data_ultima_atualizacao (auto_now=True) as freshness signal
- nginx /static/: expires 90m + Cache-Control public, max-age=5400
- nginx: removed upload-endpoint special-casing (location ~* ^/(protocoloadm/criar-protocolo|...))
- plan/RATE-LIMITER-PLAN.md updated to reflect all Phase 7 changes
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Remove docker/geoip/.gitignore so the .mmdb is versioned alongside the
codebase. The Dockerfile now exits non-zero if the file is missing,
making the build self-validating.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
docker/Dockerfile:
- GeoIP offline build with MaxMind secret; optional build args for
graphviz, poppler, psql client; envsubst for nginx burst vars
docker/docker-compose.yaml:
- saplredis service (redis:7-alpine, allkeys-lru, 512 MB)
- REDIS_URL + CACHE_BACKEND wired into sapl service
docker/startup_scripts/start.sh:
- configure_redis_cache(): builds CACHES dict, sets REDIS_CACHE waffle
switch, falls back to file cache gracefully
- POD_NAMESPACE resolution (k8s Downward API → hostname fallback)
- DATABASE_URL exported before migrate
docker/k8s/redis/ (moved from docker/k8s/):
- redis-configmap.yaml, redis-deployment.yaml, redis-service.yaml
- ClusterIP service on port 6379, sapl-redis namespace
docker/k8s/sapl-k8s.yaml:
- REDIS_URL env var injected; app.kubernetes.io/name=sapl label for
fleet-wide discovery
sapl/middleware/test_ratelimiter.py:
- Unit tests for RateLimitMiddleware with mocked Redis
scripts/test_ratelimiter.py:
- CLI smoke-test: fires N requests and reports first 429
Removed: rate-limiter-v2.md (content migrated to plan/RATE_LIMITER_PLAN.md),
scripts/test_ratelimiter.sh (replaced by .py),
docker/k8s/README.md (merged into plan/RATE_LIMITER_PLAN.md),
docker/scripts/redis_populate_test_data.py (renamed to redis_inject_test_data.py)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
nginx:
- /media/ proxied through Gunicorn (sapl_general rate limit) instead of
direct alias — Django middleware now runs on every media request
- /_accel/media/ internal location serves file bytes via X-Accel-Redirect
sapl/base/media.py (new):
- serve_media() gate: path traversal guard, auth redirect for
documentos_privados/, per-path Redis counter, content-type metadata
cache, X-Accel-Redirect response; falls back to Django serve() in DEBUG
sapl/middleware/ratelimit.py:
- RL_PATH_REQUESTS, RL_UA_BLOCKLIST, FILE_META_KEY constants
- _incr_with_ttl() extracted to module level (reused by media.py)
- Runtime UA deny list: _refresh_ua_blocklist() fetches rl:bot:ua:blocked
SET from Redis (SMEMBERS, cached per worker, TTL=RATE_LIMITER_UA_BLOCKLIST_REFRESH);
_is_redis_blocked_ua() tokenises UA and checks sha256 of each token
sapl/settings.py:
- RATE_LIMITER_UA_BLOCKLIST_REFRESH, MEDIA_PATH_COUNTER_TTL,
MEDIA_FILE_CACHE_TTL added (all env-tunable via config())
plan/RATE_LIMITER_PLAN.md:
- Key schema table updated; media file serving section added;
decision flow documented; UA deny list seed section expanded
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
django-ratelimit (the @ratelimit view decorator) writes its counters
through Django's cache framework. Django's default key function produces
'{KEY_PREFIX}:{VERSION}:{key}', so with KEY_PREFIX='' and VERSION=1
(defaults) the decorator keys appear in Redis as ':1:rl:{hash}' — an
ugly leading colon that makes them look distinct from the clean 'rl:*'
keys written by RateLimitMiddleware via get_redis_connection().
Add make_ratelimit_cache_key() to sapl/middleware/ratelimit.py (a simple
pass-through) and wire it into the 'ratelimit' cache config via
KEY_FUNCTION. Both key families now share the same 'rl:*' namespace:
decorator keys → rl:{hash}
middleware keys → rl:ip:{ip}:reqs
rl:{ns}:user:{uid}:reqs
rl:{ns}:ip:{ip}:w:{bucket}
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
MateriaLegislativaCrud.DetailView and SessaoCrud.DetailView are the two
highest-traffic public views not yet covered by anonymous page caching.
Both are read-only for anonymous visitors, making them safe cache targets.
- MateriaLegislativaCrud.DetailView: 300s TTL (PAGE_CACHE_TTL_DETAIL)
- SessaoCrud.DetailView: 120s TTL (PAGE_CACHE_TTL_LIST — sessions update
more frequently during active legislative sittings)
NormaCrud.DetailView intentionally left uncached: it writes NormaEstatisticas
on every access, and caching would suppress per-visit statistics for anonymous
users.
Also includes the RATELIMIT_DRY_RUN=False docker-compose.yaml change
from the previous session (rate limiting now enforced in docker-compose).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
waffle_switch REDIS_CACHE off --create always wrote 'off' even when the
operator had previously enabled it, making docker compose restart always
reset the switch and leaving Redis cache permanently disabled.
Replace with a Django shell get_or_create call that only inserts the row
with active=False on first boot; subsequent restarts leave the existing
value untouched.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
wait_for_pg() set DATABASE_URL via := default syntax but never exported it,
so Python child processes (manage.py migrate, waffle_switch) could not read
it from the environment. The .env file does not exist yet at that point —
write_env_file runs later — so decouple raised UndefinedValueError.
Add 'export DATABASE_URL' immediately after the default is resolved in
wait_for_pg(), which is the earliest point in the startup sequence where
the value is known and already used by configure_pg_timezone right after.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
GeoIP (docker/Dockerfile):
Remove at-build-time MaxMind download (required BuildKit secrets, caused
cache-miss issues). Replace with COPY from docker/geoip/GeoLite2-ASN.mmdb
(git-ignored binary). If absent, build succeeds with ASN blocking disabled.
Add docker/geoip/update_geoip.sh — run before each build to refresh the
database from MaxMind using MAXMIND_LICENSE_KEY from env or .env file.
Redis inspection / synthetic test data:
Add docker/scripts/redis_populate_test_data.py — injects synthetic rl:*
entries into Redis DB1 to validate key schema and blocking thresholds
without waiting for real traffic. Supports DRY_RUN and CLEAR modes.
Add §4.5 (Redis CLI quick-reference + RedisInsight guide) to rate-limiter-v2.md.
Auth-aware @ratelimit decorators (smart_rate / smart_key):
All 51 @ratelimit decorators across 9 files used rate=RATE_LIMITER_RATE
(35/m) regardless of authentication, silently over-throttling logged-in
users compared to what RateLimitMiddleware allows (120/m).
Add smart_key() and smart_rate() to sapl/middleware/ratelimit.py:
- smart_key: user pk for authenticated requests, masked IP for anon
- smart_rate: RATE_LIMITER_RATE_AUTHENTICATED (120/m) for auth,
RATE_LIMITER_RATE (35/m) for anon — mirrors middleware thresholds
Update all 51 decorators across crud/base.py + 8 view files.
Remove now-unused RATE_LIMITER_RATE imports from those files.
Cache KEY_PREFIX (settings.py):
Change KEY_PREFIX from POD_NAMESPACE ("sapl") to f"cache:{POD_NAMESPACE}"
so DB0 cache keys are unambiguously prefixed cache:{ns}:* — distinct from
any future static or file cache key patterns.
Update key schema table and code examples in rate-limiter-v2.md to match.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Removed ratelimit_ip from the sapl.utils import line (sapl.utils no longer
exports it) and added the canonical import from sapl.middleware.ratelimit.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Import fixes (all three imported get_client_ip/ratelimit_ip from sapl.utils
which no longer exports them — causing the ImportError at startup):
- sapl/materia/forms.py: move get_client_ip to sapl.middleware.ratelimit
- sapl/materia/views.py: move get_client_ip + ratelimit_ip; keep
RATE_LIMITER_RATE in sapl.settings (used by @ratelimit decorators)
- sapl/base/views.py: same pattern as materia/views.py
Docs:
- rate-limiter-v2.md: remove Phase 5 section (§8); renumber Open Questions
to §8; update Table of Contents
- work_queues.md (new): Async PDF via Celery + Django Channels WebSocket
voting panel, with full context, Redis B topology rationale, k8s manifest
list, and open questions. Planned start: after rate-limiter-2026 is stable.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add AnonCachePageMixin (sapl/middleware/page_cache.py) that stores full
view responses in the default Redis cache for anonymous (unauthenticated)
GET requests only. Authenticated users always bypass the cache so CSRF
tokens and user-specific UI controls are never served stale.
Applied to:
- ParlamentarCrud.ListView / DetailView — TTL 600 s (changes each term)
- AudienciaCrud.ListView — TTL 120 s (hearings added infrequently)
- ComissaoCrud.ListView — TTL 300 s (committees change rarely)
Also:
- Add PAGE_CACHE_TTL_LIST/DETAIL/STABLE settings (env-configurable)
- Add bingbot + SERankingBacklinksBot to nginx UA blocklist (were already
in BOT_UA_FRAGMENTS / robots.txt; nginx map was the only gap)
- Remove unused ratelimit/method_decorator/RATE_LIMITER_RATE imports from
audiencia/views.py that crept in during Phase 2
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ratelimit.py: fix module docstring to reflect current _NAMESPACE resolution
(settings.POD_NAMESPACE, not K8s SA files read inside the middleware).
docker-compose.yaml:
- Add saplredis service (redis:7-alpine, no persistence, 512 MB maxmemory,
allkeys-lru, 4 databases, same policy as k8s ConfigMap).
- Add REDIS_URL=redis://saplredis:6379 and CACHE_BACKEND=redis to the
sapl service so local docker-compose runs use Redis out of the box.
- sapl depends_on now includes saplredis.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- sapl/audiencia/views.py: relative import 'from ..utils import ratelimit_ip'
missed in previous migration; updated to
'from sapl.middleware.ratelimit import ratelimit_ip'.
- settings.py: POD_NAMESPACE now defaults to `host` (socket.gethostname()
result already computed at line 29) so bare-metal, VM and docker-compose
deployments get a meaningful, unique KEY_PREFIX without any extra config.
K8s deployments override it via POD_NAMESPACE env var (Downward API /
start.sh service-account detection).
- ratelimit.py: _NAMESPACE = settings.POD_NAMESPACE — single source of
truth; removes duplicate K8s SA-file reading that settings.py/start.sh
already handle. Drops now-unused `import os`.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1. user_id: str(request.user.pk) — pk is int, lower()/strip() were no-ops
2. Redis key constants: RL_IP_REQUESTS, RL_IP_BLOCKED, RL_USER_REQUESTS,
RL_USER_BLOCKED, RL_NS_WINDOW — no more inline f-string literals
3. Tenant namespace: _NAMESPACE resolved once at module load from
POD_NAMESPACE env var (K8s Downward API) → service-account namespace
file → 'global' fallback. No per-request getattr(request, 'tenant').
4. KEY_PREFIX in CACHES['default'] set to POD_NAMESPACE (e.g. patobranco-pr)
so each tenant's cache keys are isolated in shared Redis.
5. Logger extra: replaced getattr(request, 'tenant', 'unknown') with
_NAMESPACE (the actual resolved constant).
settings.py: add POD_NAMESPACE = config('POD_NAMESPACE', default='sapl');
use it as KEY_PREFIX.
start.sh: add resolve_pod_namespace() (Downward API → SA file → fallback);
call it before resolve_redis_url(); write POD_NAMESPACE into .env.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Remove get_client_ip and ratelimit_ip from all sapl.utils import lines
and replace with direct imports from sapl.middleware.ratelimit, which
is now the canonical location for both functions.
Affected files:
sapl/comissoes/views.py
sapl/protocoloadm/views.py
sapl/crud/base.py
sapl/norma/views.py
sapl/sessao/views.py
sapl/painel/views.py
sapl/parlamentares/views.py
sapl/utils.py (re-export shim removed)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Move get_client_ip() and ratelimit_ip() from utils.py to
sapl/middleware/ratelimit.py (canonical location).
utils.py re-exports both via a single import line so all existing
callers (comissoes, crud, norma, sessao, painel, parlamentares,
protocoloadm) keep working with zero changes.
- get_client_ip() is now used inside RateLimitMiddleware instead of
the weaker _get_ip(): gains ip_mask() for IPv6 /64 collapsing and
HTTP_X_REAL_IP fallback.
- Replace getattr(settings, 'X', default) with settings.X throughout
__init__: settings.py always defines these vars, defaults were
duplicated and would silently drift. django.conf.settings proxy also
honours @override_settings in tests, unlike direct module imports.
- Replace getattr(..., []) or [] with set(settings.RATE_LIMIT_WHITELIST_IPS):
the cast in settings.py always returns a list, the double guard was
redundant.
- Remove unused _get_ip() and 'from sapl.settings import RATE_LIMITER_RATE'.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
sapl/middleware/ratelimit.py:
- Decision chain: known UA → IP blocked → authenticated → anonymous
- Authenticated: 120 req/min per user (rl:{ns}:user:{id}:reqs, DB1)
- Anonymous: 35 req/min per IP (rl:ip:{ip}:reqs) + per-ns/IP/window
counter to catch UA rotators (rl:ns:{ns}:ip:{ip}:w:{bucket})
- Blocking keys expire after 300 s (BLOCK_TTL)
- Thresholds driven by RATE_LIMITER_RATE / RATE_LIMITER_RATE_AUTHENTICATED
- RATE_LIMIT_WHITELIST_IPS for legislative-house IP ranges (future)
- Atomic INCR+EXPIRE via Redis Lua script; falls back to non-atomic
cache get/set when Redis unavailable (dry-run / file-cache safe)
- RATELIMIT_DRY_RUN=True by default — logs only, no 429s returned
- OAI-SearchBot added to BOT_UA_FRAGMENTS
- Suspicious-header check: missing Accept-Language + Accept (2/2)
- Whitelist check short-circuits all other checks
settings.py:
- RateLimitMiddleware inserted after AuthenticationMiddleware so
request.user is available for authenticated-vs-anonymous branching
- RATELIMIT_DRY_RUN (default True)
- RATE_LIMITER_RATE_AUTHENTICATED (default '120/m')
- RATE_LIMITER_RATE_BOT (default '5/m')
- RATE_LIMIT_WHITELIST_IPS (default empty)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Corrige polling excessivo em /voto-individual/
- Adiciona @login_required em votante_view para que usuários
anônimos sejam redirecionados ao login em vez de receberem o
template com auto-reload.
- Corrige bug do relógio: setInterval recursivo dentro de
startTime() acumulava timers exponencialmente. Trocado por
setTimeout, mantendo apenas um timer pendente.
- Move o script de reload/relógio para dentro do bloco
{% if not error_message %} para que a variante de erro não
agende reloads de 30s.
Reduz drasticamente o volume de requisições que disparavam o
rate limiter (RATE_LIMITER_RATE=35/m) quando vários parlamentares
mantinham a tela aberta atrás do mesmo IP NAT.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Bloqueia /voto-individual/ para usuários sem Votante/can_vote
Antes, usuários autenticados sem registro de Votante ou sem a
permissão parlamentares.can_vote recebiam HTTP 200 com um template
de erro (e, antes do commit anterior, ainda traziam o JS de auto-
reload). Agora a view exige as duas condições no início e retorna
HTTP 403 caso falhe:
- @permission_required('parlamentares.can_vote', raise_exception=True)
garante a permissão (com raise_exception para 403 ao invés do
redirect padrão pra LOGIN_URL).
- Checagem inline de Votante.objects.filter(user=...).exists() pega
o caso de superusers (que passam pelo bypass automático de
permissões do Django mas não têm cadastro de Votante) e de
qualquer custom group que conceda can_vote sem o cadastro.
Mantém HTTP 200 com mensagem amigável para falhas de estado que
um votante legítimo pode encontrar (nenhuma matéria aberta,
parlamentar não presente na sessão).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: joao <joao@mezzoplanejamento.com.br>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refact: cria método get_proximo_numero
* feat: impl numeração automática em cadastros via API
* Update sapl/materia/models.py
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update sapl/api/serializers.py
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update sapl/api/views_materia.py
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update sapl/materia/models.py
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update sapl/materia/models.py
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Add transaction atomic no endpoint create
* add validação de tipo se tipo não é objeto do model TipoMateriaLegislativa
* refact: aplica solicitações de reviewer e cria testes
* fix: altera numero_preferido para numero_candidato
* fix: remove espaços entre classes
* fix: corrige uso de transaction e ausencia dele
* fix: corrige testes devido mudança de norme de variável
* fix: altera seleção para select_for_update
* fix: mudança de nome remanescente para numero_candidato
* fix: retorna decorator na view function recuperar_materia
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* feat: impl HEADER LastModifiedDecorator na api
* fix: ajuste no frontend para evitar cache automático devido a LastModified sem tratamento adequado
* rebuild frontend
* fix: corrige last_modified_func para retorna sob retrieve
* refact: impl LastModified diretamente nos models que possuem campos específicos
* Seleciona tipo de votação para múltiplas matérias ao incluir na Ordem do Dia ou Expediente
* Ajustes solicitados na seleção de múltiplas Matérias para Ordem/Expediente
* Update adicionar_varias_materias_expediente.html
* Fix read-only mount on k8s
* Fix recibo proposição e adiciona rate limiter em matéria e norma
* Update forms.py
Alteração do nome do campo todos, conforme solicitação.
* Update adicionar_varias_materias_expediente.html
Ajuste no template por alteração do nome do campo "todos".
* Fix recibo proposição e adiciona rate limiter em matéria e norma
---------
Co-authored-by: root <root@info38.camaranh>
Co-authored-by: Edward Oliveira <edwardr@senado.gov.br>
Co-authored-by: Edward <9326037+edwardoliveira@users.noreply.github.com>
joaohortsenado
LeandroJatai
cristian-longhi