Remove docker/geoip/.gitignore so the .mmdb is versioned alongside the
codebase. The Dockerfile now exits non-zero if the file is missing,
making the build self-validating.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
docker/Dockerfile:
- GeoIP offline build with MaxMind secret; optional build args for
graphviz, poppler, psql client; envsubst for nginx burst vars
docker/docker-compose.yaml:
- saplredis service (redis:7-alpine, allkeys-lru, 512 MB)
- REDIS_URL + CACHE_BACKEND wired into sapl service
docker/startup_scripts/start.sh:
- configure_redis_cache(): builds CACHES dict, sets REDIS_CACHE waffle
switch, falls back to file cache gracefully
- POD_NAMESPACE resolution (k8s Downward API → hostname fallback)
- DATABASE_URL exported before migrate
docker/k8s/redis/ (moved from docker/k8s/):
- redis-configmap.yaml, redis-deployment.yaml, redis-service.yaml
- ClusterIP service on port 6379, sapl-redis namespace
docker/k8s/sapl-k8s.yaml:
- REDIS_URL env var injected; app.kubernetes.io/name=sapl label for
fleet-wide discovery
sapl/middleware/test_ratelimiter.py:
- Unit tests for RateLimitMiddleware with mocked Redis
scripts/test_ratelimiter.py:
- CLI smoke-test: fires N requests and reports first 429
Removed: rate-limiter-v2.md (content migrated to plan/RATE_LIMITER_PLAN.md),
scripts/test_ratelimiter.sh (replaced by .py),
docker/k8s/README.md (merged into plan/RATE_LIMITER_PLAN.md),
docker/scripts/redis_populate_test_data.py (renamed to redis_inject_test_data.py)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
nginx:
- /media/ proxied through Gunicorn (sapl_general rate limit) instead of
direct alias — Django middleware now runs on every media request
- /_accel/media/ internal location serves file bytes via X-Accel-Redirect
sapl/base/media.py (new):
- serve_media() gate: path traversal guard, auth redirect for
documentos_privados/, per-path Redis counter, content-type metadata
cache, X-Accel-Redirect response; falls back to Django serve() in DEBUG
sapl/middleware/ratelimit.py:
- RL_PATH_REQUESTS, RL_UA_BLOCKLIST, FILE_META_KEY constants
- _incr_with_ttl() extracted to module level (reused by media.py)
- Runtime UA deny list: _refresh_ua_blocklist() fetches rl:bot:ua:blocked
SET from Redis (SMEMBERS, cached per worker, TTL=RATE_LIMITER_UA_BLOCKLIST_REFRESH);
_is_redis_blocked_ua() tokenises UA and checks sha256 of each token
sapl/settings.py:
- RATE_LIMITER_UA_BLOCKLIST_REFRESH, MEDIA_PATH_COUNTER_TTL,
MEDIA_FILE_CACHE_TTL added (all env-tunable via config())
plan/RATE_LIMITER_PLAN.md:
- Key schema table updated; media file serving section added;
decision flow documented; UA deny list seed section expanded
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
MateriaLegislativaCrud.DetailView and SessaoCrud.DetailView are the two
highest-traffic public views not yet covered by anonymous page caching.
Both are read-only for anonymous visitors, making them safe cache targets.
- MateriaLegislativaCrud.DetailView: 300s TTL (PAGE_CACHE_TTL_DETAIL)
- SessaoCrud.DetailView: 120s TTL (PAGE_CACHE_TTL_LIST — sessions update
more frequently during active legislative sittings)
NormaCrud.DetailView intentionally left uncached: it writes NormaEstatisticas
on every access, and caching would suppress per-visit statistics for anonymous
users.
Also includes the RATELIMIT_DRY_RUN=False docker-compose.yaml change
from the previous session (rate limiting now enforced in docker-compose).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
waffle_switch REDIS_CACHE off --create always wrote 'off' even when the
operator had previously enabled it, making docker compose restart always
reset the switch and leaving Redis cache permanently disabled.
Replace with a Django shell get_or_create call that only inserts the row
with active=False on first boot; subsequent restarts leave the existing
value untouched.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
wait_for_pg() set DATABASE_URL via := default syntax but never exported it,
so Python child processes (manage.py migrate, waffle_switch) could not read
it from the environment. The .env file does not exist yet at that point —
write_env_file runs later — so decouple raised UndefinedValueError.
Add 'export DATABASE_URL' immediately after the default is resolved in
wait_for_pg(), which is the earliest point in the startup sequence where
the value is known and already used by configure_pg_timezone right after.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
GeoIP (docker/Dockerfile):
Remove at-build-time MaxMind download (required BuildKit secrets, caused
cache-miss issues). Replace with COPY from docker/geoip/GeoLite2-ASN.mmdb
(git-ignored binary). If absent, build succeeds with ASN blocking disabled.
Add docker/geoip/update_geoip.sh — run before each build to refresh the
database from MaxMind using MAXMIND_LICENSE_KEY from env or .env file.
Redis inspection / synthetic test data:
Add docker/scripts/redis_populate_test_data.py — injects synthetic rl:*
entries into Redis DB1 to validate key schema and blocking thresholds
without waiting for real traffic. Supports DRY_RUN and CLEAR modes.
Add §4.5 (Redis CLI quick-reference + RedisInsight guide) to rate-limiter-v2.md.
Auth-aware @ratelimit decorators (smart_rate / smart_key):
All 51 @ratelimit decorators across 9 files used rate=RATE_LIMITER_RATE
(35/m) regardless of authentication, silently over-throttling logged-in
users compared to what RateLimitMiddleware allows (120/m).
Add smart_key() and smart_rate() to sapl/middleware/ratelimit.py:
- smart_key: user pk for authenticated requests, masked IP for anon
- smart_rate: RATE_LIMITER_RATE_AUTHENTICATED (120/m) for auth,
RATE_LIMITER_RATE (35/m) for anon — mirrors middleware thresholds
Update all 51 decorators across crud/base.py + 8 view files.
Remove now-unused RATE_LIMITER_RATE imports from those files.
Cache KEY_PREFIX (settings.py):
Change KEY_PREFIX from POD_NAMESPACE ("sapl") to f"cache:{POD_NAMESPACE}"
so DB0 cache keys are unambiguously prefixed cache:{ns}:* — distinct from
any future static or file cache key patterns.
Update key schema table and code examples in rate-limiter-v2.md to match.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add AnonCachePageMixin (sapl/middleware/page_cache.py) that stores full
view responses in the default Redis cache for anonymous (unauthenticated)
GET requests only. Authenticated users always bypass the cache so CSRF
tokens and user-specific UI controls are never served stale.
Applied to:
- ParlamentarCrud.ListView / DetailView — TTL 600 s (changes each term)
- AudienciaCrud.ListView — TTL 120 s (hearings added infrequently)
- ComissaoCrud.ListView — TTL 300 s (committees change rarely)
Also:
- Add PAGE_CACHE_TTL_LIST/DETAIL/STABLE settings (env-configurable)
- Add bingbot + SERankingBacklinksBot to nginx UA blocklist (were already
in BOT_UA_FRAGMENTS / robots.txt; nginx map was the only gap)
- Remove unused ratelimit/method_decorator/RATE_LIMITER_RATE imports from
audiencia/views.py that crept in during Phase 2
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ratelimit.py: fix module docstring to reflect current _NAMESPACE resolution
(settings.POD_NAMESPACE, not K8s SA files read inside the middleware).
docker-compose.yaml:
- Add saplredis service (redis:7-alpine, no persistence, 512 MB maxmemory,
allkeys-lru, 4 databases, same policy as k8s ConfigMap).
- Add REDIS_URL=redis://saplredis:6379 and CACHE_BACKEND=redis to the
sapl service so local docker-compose runs use Redis out of the box.
- sapl depends_on now includes saplredis.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1. user_id: str(request.user.pk) — pk is int, lower()/strip() were no-ops
2. Redis key constants: RL_IP_REQUESTS, RL_IP_BLOCKED, RL_USER_REQUESTS,
RL_USER_BLOCKED, RL_NS_WINDOW — no more inline f-string literals
3. Tenant namespace: _NAMESPACE resolved once at module load from
POD_NAMESPACE env var (K8s Downward API) → service-account namespace
file → 'global' fallback. No per-request getattr(request, 'tenant').
4. KEY_PREFIX in CACHES['default'] set to POD_NAMESPACE (e.g. patobranco-pr)
so each tenant's cache keys are isolated in shared Redis.
5. Logger extra: replaced getattr(request, 'tenant', 'unknown') with
_NAMESPACE (the actual resolved constant).
settings.py: add POD_NAMESPACE = config('POD_NAMESPACE', default='sapl');
use it as KEY_PREFIX.
start.sh: add resolve_pod_namespace() (Downward API → SA file → fallback);
call it before resolve_redis_url(); write POD_NAMESPACE into .env.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Edward
joaohortsenado
LeandroJatai