mirror of https://github.com/interlegis/sapl.git
Browse Source
nginx: - /media/ proxied through Gunicorn (sapl_general rate limit) instead of direct alias — Django middleware now runs on every media request - /_accel/media/ internal location serves file bytes via X-Accel-Redirect sapl/base/media.py (new): - serve_media() gate: path traversal guard, auth redirect for documentos_privados/, per-path Redis counter, content-type metadata cache, X-Accel-Redirect response; falls back to Django serve() in DEBUG sapl/middleware/ratelimit.py: - RL_PATH_REQUESTS, RL_UA_BLOCKLIST, FILE_META_KEY constants - _incr_with_ttl() extracted to module level (reused by media.py) - Runtime UA deny list: _refresh_ua_blocklist() fetches rl:bot:ua:blocked SET from Redis (SMEMBERS, cached per worker, TTL=RATE_LIMITER_UA_BLOCKLIST_REFRESH); _is_redis_blocked_ua() tokenises UA and checks sha256 of each token sapl/settings.py: - RATE_LIMITER_UA_BLOCKLIST_REFRESH, MEDIA_PATH_COUNTER_TTL, MEDIA_FILE_CACHE_TTL added (all env-tunable via config()) plan/RATE_LIMITER_PLAN.md: - Key schema table updated; media file serving section added; decision flow documented; UA deny list seed section expanded Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>rate-limiter-2026
9 changed files with 2303 additions and 91 deletions
@ -0,0 +1,145 @@ |
|||
# CLAUDE.md |
|||
|
|||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. |
|||
|
|||
## Project Overview |
|||
|
|||
SAPL (Sistema de Apoio ao Processo Legislativo) is a Django-based legislative management system used by Brazilian municipal and state legislative houses. It manages bills, parliamentary sessions, committees, norms, protocols, and related legislative workflows. |
|||
|
|||
## Commands |
|||
|
|||
### Development |
|||
|
|||
```bash |
|||
# Run dev server |
|||
python manage.py runserver |
|||
|
|||
# Docker (dev, without bundled DB) |
|||
docker-compose -f docker/docker-compose-dev.yml up |
|||
|
|||
# Docker (dev, with PostgreSQL container) |
|||
docker-compose -f docker/docker-compose-dev-db.yml up |
|||
``` |
|||
|
|||
### Database Setup (local PostgreSQL) |
|||
|
|||
```bash |
|||
sudo -u postgres psql -c "CREATE ROLE sapl LOGIN ENCRYPTED PASSWORD 'sapl' NOSUPERUSER INHERIT CREATEDB NOCREATEROLE NOREPLICATION;" |
|||
sudo -u postgres psql -c "CREATE DATABASE sapl WITH OWNER=sapl ENCODING='UTF8' LC_COLLATE='pt_BR.UTF-8' LC_CTYPE='pt_BR.UTF-8' CONNECTION LIMIT=-1 TEMPLATE template0;" |
|||
python manage.py migrate |
|||
``` |
|||
|
|||
### Testing |
|||
|
|||
```bash |
|||
# All tests (reuses DB by default for speed) |
|||
pytest |
|||
|
|||
# Single test file or test function |
|||
pytest sapl/materia/tests/test_materia.py |
|||
pytest sapl/materia/tests/test_materia.py::test_function_name |
|||
|
|||
# Force DB recreation |
|||
pytest --create-db |
|||
|
|||
# With coverage |
|||
pytest --cov=sapl |
|||
``` |
|||
|
|||
Tests require `DJANGO_SETTINGS_MODULE=sapl.settings` (set in `pytest.ini`). All tests must be marked with `@pytest.mark.django_db`. The `conftest.py` root fixture provides an `app` fixture (WebTest `DjangoTestApp`). |
|||
|
|||
### Linting / Formatting |
|||
|
|||
```bash |
|||
flake8 . |
|||
isort . |
|||
autopep8 --in-place <file.py> |
|||
``` |
|||
|
|||
### Restore Database from Backup |
|||
|
|||
```bash |
|||
./scripts/restore_db.sh -f /path/to/dump |
|||
./scripts/restore_db.sh -f /path/to/dump -p 5433 # Docker port |
|||
``` |
|||
|
|||
## Architecture |
|||
|
|||
### Django Apps |
|||
|
|||
Apps are under `sapl/` and follow domain boundaries: |
|||
|
|||
| App | Domain | |
|||
|-----|--------| |
|||
| `base` | `CasaLegislativa` (legislative house config), `AppConfig`, `Autor` (authorship) | |
|||
| `parliamentary` | `Parlamentar`, `Legislatura`, `SessaoLegislativa`, `Coligacao` | |
|||
| `materia` | Bills (`MateriaLegislativa`), types, tracking, annexes | |
|||
| `norma` | Laws/norms (`NormaJuridica`) and hierarchies | |
|||
| `sessao` | Plenary sessions, agenda, attendance, voting | |
|||
| `comissoes` | Committees (`Comissao`) and meetings (`Reuniao`) | |
|||
| `protocoloadm` | Administrative protocols and document intake | |
|||
| `compilacao` | Structured/articulated texts (LexML-like tree structure) | |
|||
| `lexml` | LexML XML standard integration | |
|||
| `audiencia` | Public hearings | |
|||
| `painel` | Real-time session display panel | |
|||
| `relatorios` | PDF report generation | |
|||
| `api` | REST API entry point (auto-generated ViewSets) | |
|||
| `crud` | Generic CRUD base views | |
|||
| `rules` | Business rules and permission definitions | |
|||
|
|||
### REST API |
|||
|
|||
The API uses a custom `drfautoapi` package (`drfautoapi/drfautoapi.py`) that auto-generates DRF ViewSets, Serializers, and FilterSets from Django models. Authentication is Token + Session. Permissions use a custom `SaplModelPermissions` class that maps HTTP methods to Django model permissions. |
|||
|
|||
OpenAPI 3.0 docs are generated by drf-spectacular. |
|||
|
|||
### Caching |
|||
|
|||
- **Default:** File-based (`/var/tmp/django_cache`) |
|||
- **Production:** Redis via django-redis; configured at startup by `configure_redis_cache()` in `sapl/settings.py` |
|||
- **Cache key prefix:** `cache:{POD_NAMESPACE}:` (namespace-isolated for multi-tenant k8s) |
|||
- **Rate limiter state** is shared via Redis keys |
|||
|
|||
### Feature Flags |
|||
|
|||
django-waffle is used for feature flags. Switches (global on/off) can be toggled via: |
|||
|
|||
```bash |
|||
python manage.py waffle_switch <switch_name> on|off |
|||
``` |
|||
|
|||
### Key Environment Variables |
|||
|
|||
| Variable | Purpose | |
|||
|----------|---------| |
|||
| `DATABASE_URL` | PostgreSQL connection string | |
|||
| `SECRET_KEY` | Django secret key | |
|||
| `DEBUG` | Debug mode | |
|||
| `REDIS_URL` | Redis host:port | |
|||
| `CACHE_BACKEND` | `file` or `redis` | |
|||
| `POD_NAMESPACE` | K8s namespace (used in cache key prefix) | |
|||
| `USE_SOLR` | Enable Haystack/Solr full-text search | |
|||
| `SOLR_URL` / `SOLR_COLLECTION` | Solr connection | |
|||
|
|||
### Docker Build |
|||
|
|||
The production build requires a MaxMind GeoLite2-ASN license key (for nginx ASN-based bot blocking): |
|||
|
|||
```bash |
|||
docker build --secret id=maxmind_key,src=.env -f docker/Dockerfile -t sapl:local . |
|||
``` |
|||
|
|||
Optional build args: `WITH_NGINX`, `WITH_GRAPHVIZ`, `WITH_POPPLER`, `WITH_PSQL_CLIENT`. |
|||
|
|||
### Key File Locations |
|||
|
|||
| File | Purpose | |
|||
|------|---------| |
|||
| `sapl/settings.py` | All Django settings, including cache/rate-limit setup | |
|||
| `pytest.ini` | Test configuration (DJANGO_SETTINGS_MODULE, addopts) | |
|||
| `conftest.py` | Root pytest fixtures | |
|||
| `drfautoapi/drfautoapi.py` | Auto-API generation logic | |
|||
| `docker/startup_scripts/start.sh` | Container entrypoint (migrations, waffle, gunicorn) | |
|||
| `requirements/requirements.txt` | Production deps | |
|||
| `requirements/test-requirements.txt` | Test deps | |
|||
| `requirements/dev-requirements.txt` | Dev/lint deps | |
|||
@ -0,0 +1,169 @@ |
|||
#!/usr/bin/env python3 |
|||
""" |
|||
redis_inject_test_data.py — inject synthetic rate-limiter entries into Redis. |
|||
|
|||
Purpose: validate that RateLimitMiddleware reads the expected key schema, |
|||
that Redis CLI / RedisInsight shows the right structure, and that blocking |
|||
logic fires correctly without waiting for real traffic. |
|||
|
|||
Usage: |
|||
# Against docker-compose Redis (default) |
|||
python3 docker/scripts/redis_inject_test_data.py |
|||
|
|||
# Against a different host/port |
|||
REDIS_URL=redis://localhost:6379 python3 docker/scripts/redis_inject_test_data.py |
|||
|
|||
# Clear all synthetic keys written by a previous run |
|||
CLEAR=1 python3 docker/scripts/redis_inject_test_data.py |
|||
|
|||
Key schema (DB 1 — rate limiter): |
|||
rl:ip:{ip}:reqs INCR counter — anonymous request count (TTL 60s) |
|||
rl:ip:{ip}:blocked string "1" — IP hard-blocked (TTL 300s) |
|||
rl:{ns}:user:{uid}:reqs INCR counter — auth user request count (TTL 60s) |
|||
rl:{ns}:user:{uid}:blocked string "1" — user hard-blocked (TTL 300s) |
|||
rl:{ns}:ip:{ip}:w:{bucket} INCR — namespace/IP sliding window (TTL 120s) |
|||
""" |
|||
|
|||
import os |
|||
import sys |
|||
import time |
|||
from decouple import config |
|||
|
|||
# ── dependency check ────────────────────────────────────────────────────── |
|||
try: |
|||
import redis |
|||
except ImportError: |
|||
print("ERROR: redis-py not installed. Run: pip install redis", file=sys.stderr) |
|||
sys.exit(1) |
|||
|
|||
# ── config ──────────────────────────────────────────────────────────────── |
|||
REDIS_URL = config("REDIS_URL", default="redis://localhost:6379") |
|||
RATELIMIT_DB = 1 # DB1 is the rate-limiter database |
|||
CLEAR = config("CLEAR", default="0").lower() in ("1", "true", "yes") |
|||
|
|||
# Synthetic values — tweak to exercise different code paths |
|||
NAMESPACE = "sapl" # POD_NAMESPACE value (hostname or k8s namespace) |
|||
ANON_WINDOW = 60 # seconds — must match settings.RATE_LIMITER_RATE period |
|||
AUTH_WINDOW = 60 |
|||
BLOCK_TTL = 300 |
|||
|
|||
TEST_IPS = [ |
|||
"203.0.113.1", # below threshold (20 reqs) |
|||
"203.0.113.2", # AT threshold (35 reqs — should trigger block) |
|||
"203.0.113.3", # already blocked |
|||
"203.0.113.4", # namespace/window counter near threshold |
|||
] |
|||
|
|||
TEST_USERS = [ |
|||
{"uid": "42", "reqs": 50, "blocked": False}, # normal auth user |
|||
{"uid": "99", "reqs": 120, "blocked": False}, # AT auth threshold |
|||
{"uid": "7", "reqs": 10, "blocked": True}, # pre-blocked user |
|||
] |
|||
|
|||
# ── helpers ─────────────────────────────────────────────────────────────── |
|||
|
|||
def key_ip_reqs(ip): |
|||
return f"rl:ip:{ip}:reqs" |
|||
|
|||
def key_ip_blocked(ip): |
|||
return f"rl:ip:{ip}:blocked" |
|||
|
|||
def key_user_reqs(ns, uid): |
|||
return f"rl:{ns}:user:{uid}:reqs" |
|||
|
|||
def key_user_blocked(ns, uid): |
|||
return f"rl:{ns}:user:{uid}:blocked" |
|||
|
|||
def key_ns_window(ns, ip, bucket): |
|||
return f"rl:{ns}:ip:{ip}:w:{bucket}" |
|||
|
|||
|
|||
def write(r, key, value, ttl, label): |
|||
if isinstance(value, int): |
|||
pipe = r.pipeline() |
|||
pipe.set(key, value, ex=ttl) |
|||
pipe.execute() |
|||
else: |
|||
r.set(key, value, ex=ttl) |
|||
print(f" SET {key!r} = {value!r} EX {ttl}s ({label})") |
|||
|
|||
|
|||
def delete_pattern(r, pattern): |
|||
keys = r.keys(pattern) |
|||
if keys: |
|||
r.delete(*keys) |
|||
print(f" DEL {len(keys)} keys matching {pattern!r}") |
|||
else: |
|||
print(f" (no keys matching {pattern!r})") |
|||
|
|||
|
|||
# ── main ────────────────────────────────────────────────────────────────── |
|||
|
|||
def main(): |
|||
r = redis.from_url(REDIS_URL, db=RATELIMIT_DB, decode_responses=True) |
|||
try: |
|||
r.ping() |
|||
except redis.ConnectionError as exc: |
|||
print(f"ERROR: cannot connect to Redis at {REDIS_URL}: {exc}", file=sys.stderr) |
|||
sys.exit(1) |
|||
|
|||
print(f"Redis: {REDIS_URL} DB={RATELIMIT_DB} clear={CLEAR}") |
|||
print() |
|||
|
|||
# ── clear mode ──────────────────────────────────────────────────────── |
|||
if CLEAR: |
|||
print("=== Clearing synthetic test keys ===") |
|||
for ip in TEST_IPS: |
|||
delete_pattern(r, f"rl:ip:{ip}:*") |
|||
delete_pattern(r, f"rl:{NAMESPACE}:ip:{ip}:*") |
|||
for u in TEST_USERS: |
|||
delete_pattern(r, f"rl:{NAMESPACE}:user:{u['uid']}:*") |
|||
print("Done.") |
|||
return |
|||
|
|||
# ── anonymous IP counters ───────────────────────────────────────────── |
|||
print("=== Anonymous IP request counters (DB1) ===") |
|||
write(r, key_ip_reqs(TEST_IPS[0]), 20, ANON_WINDOW, "below threshold") |
|||
write(r, key_ip_reqs(TEST_IPS[1]), 35, ANON_WINDOW, "AT threshold → middleware will block on next req") |
|||
write(r, key_ip_reqs(TEST_IPS[3]), 30, ANON_WINDOW, "below threshold") |
|||
print() |
|||
|
|||
# ── blocked IPs ─────────────────────────────────────────────────────── |
|||
print("=== Blocked IPs (DB1) ===") |
|||
write(r, key_ip_blocked(TEST_IPS[2]), "1", BLOCK_TTL, "hard-blocked") |
|||
print() |
|||
|
|||
# ── namespace/IP sliding window ─────────────────────────────────────── |
|||
print("=== Namespace/IP sliding window (DB1) ===") |
|||
bucket = int(time.time() // ANON_WINDOW) |
|||
write(r, key_ns_window(NAMESPACE, TEST_IPS[3], bucket), 34, ANON_WINDOW * 2, |
|||
"near window threshold (next req triggers ua_rotation block)") |
|||
print() |
|||
|
|||
# ── authenticated user counters ─────────────────────────────────────── |
|||
print("=== Authenticated user request counters (DB1) ===") |
|||
for u in TEST_USERS: |
|||
if not u["blocked"]: |
|||
write(r, key_user_reqs(NAMESPACE, u["uid"]), u["reqs"], AUTH_WINDOW, |
|||
f"uid={u['uid']} reqs={u['reqs']}") |
|||
print() |
|||
|
|||
# ── blocked users ───────────────────────────────────────────────────── |
|||
print("=== Blocked users (DB1) ===") |
|||
for u in TEST_USERS: |
|||
if u["blocked"]: |
|||
write(r, key_user_blocked(NAMESPACE, u["uid"]), "1", BLOCK_TTL, |
|||
f"uid={u['uid']} hard-blocked") |
|||
print() |
|||
|
|||
# ── summary ─────────────────────────────────────────────────────────── |
|||
all_keys = r.keys("rl:*") |
|||
print(f"=== DB{RATELIMIT_DB} now contains {len(all_keys)} rl:* keys ===") |
|||
for k in sorted(all_keys): |
|||
ttl = r.ttl(k) |
|||
val = r.get(k) |
|||
print(f" {k!r:55s} val={val!r:5} ttl={ttl}s") |
|||
|
|||
|
|||
if __name__ == "__main__": |
|||
main() |
|||
@ -0,0 +1,474 @@ |
|||
# SAPL — Kubernetes Redis |
|||
|
|||
Manifests for the shared Redis instance used by all SAPL pods for |
|||
cross-pod rate limiting (DB 1) and view/static-file caching (DB 0). |
|||
|
|||
--- |
|||
|
|||
## Directory layout |
|||
|
|||
``` |
|||
docker/k8s/ |
|||
└── redis/ |
|||
├── redis-configmap.yaml # redis.conf — no persistence, allkeys-lru, 5 GB ceiling |
|||
├── redis-deployment.yaml # Deployment (1 replica, redis:7-alpine) |
|||
└── redis-service.yaml # ClusterIP service on port 6379 |
|||
``` |
|||
|
|||
--- |
|||
|
|||
## Prerequisites |
|||
|
|||
- `kubectl` configured to talk to the target cluster. |
|||
- A `sapl-redis` namespace (created below if it doesn't exist). |
|||
|
|||
--- |
|||
|
|||
## Deploy |
|||
|
|||
```bash |
|||
# 1. Create the namespace (idempotent) |
|||
rancher kubectl create namespace sapl-redis --dry-run=client -o yaml | rancher kubectl apply -f - |
|||
|
|||
# 2. Apply all three manifests |
|||
rancher kubectl apply -f docker/k8s/redis/redis-configmap.yaml |
|||
rancher kubectl apply -f docker/k8s/redis/redis-deployment.yaml |
|||
rancher kubectl apply -f docker/k8s/redis/redis-service.yaml |
|||
|
|||
# 3. Verify the pod is Running |
|||
rancher kubectl -n sapl-redis get pods -l app=sapl-redis |
|||
``` |
|||
|
|||
Expected output: |
|||
``` |
|||
NAME READY STATUS RESTARTS AGE |
|||
sapl-redis-6d9f8b7c4d-xk2lm 1/1 Running 0 30s |
|||
``` |
|||
|
|||
--- |
|||
|
|||
## Verify the rate limiter |
|||
|
|||
`scripts/test_ratelimiter.py` fires repeated GET requests at a SAPL URL and reports |
|||
when the first 429 is returned. |
|||
|
|||
### Usage |
|||
|
|||
``` |
|||
python scripts/test_ratelimiter.py <URL> [-n NUM] [-d DELAY] [-t TIMEOUT] |
|||
``` |
|||
|
|||
| Flag | Default | Meaning | |
|||
|------|---------|---------| |
|||
| `url` | *(required)* | Full URL including scheme, e.g. `http://localhost` | |
|||
| `-n`, `--num-requests` | `50` | Maximum requests to send | |
|||
| `-d`, `--delay` | `0.1` | Seconds between requests | |
|||
| `-t`, `--timeout` | `10` | Per-request timeout in seconds | |
|||
|
|||
The script stops and prints a summary as soon as a 429 is received. |
|||
|
|||
### Examples |
|||
|
|||
```bash |
|||
# Hit the anonymous threshold (35 req/min) — fire 40 requests with minimal delay |
|||
python scripts/test_ratelimiter.py http://localhost -n 40 -d 0.05 |
|||
|
|||
# Slower fire — check that legitimate traffic is not rate-limited |
|||
python scripts/test_ratelimiter.py http://localhost -n 20 -d 2 |
|||
|
|||
# Test against a staging pod via port-forward |
|||
rancher kubectl port-forward -n <NAMESPACE> deploy/sapl 8080:80 & |
|||
python scripts/test_ratelimiter.py http://localhost:8080 -n 40 -d 0.05 |
|||
``` |
|||
|
|||
### Reading the output |
|||
|
|||
``` |
|||
Request 1: Status 200 | Time: 0.045s |
|||
... |
|||
Request 36: Status 429 | Time: 0.038s |
|||
-> Rate limited on request 36 |
|||
|
|||
Summary: |
|||
Total requests attempted: 36 |
|||
Successful (200): 35 |
|||
Rate limited (429): 1 |
|||
First 429 occurred at request: 36 |
|||
``` |
|||
|
|||
A first-429 near the configured anonymous threshold (35 req/min) confirms the |
|||
middleware is wired correctly. A first-429 much earlier points to nginx `limit_req` |
|||
firing before Django sees the request. |
|||
|
|||
--- |
|||
|
|||
## Inject REDIS_URL into SAPL instances |
|||
|
|||
`REDIS_URL` points at the shared instance: |
|||
|
|||
``` |
|||
redis://redis.sapl-redis.svc.cluster.local:6379 |
|||
^^^^^ ^^^^^^^^^^ |
|||
svc namespace |
|||
``` |
|||
|
|||
`start.sh` picks it up on every pod startup and sets the `REDIS_CACHE` waffle switch |
|||
automatically — no further intervention needed. |
|||
|
|||
### Fleet-wide rollout |
|||
|
|||
Uses the `app.kubernetes.io/name=sapl` pod label to discover every SAPL namespace |
|||
automatically — onboarding a new municipality requires no script changes. |
|||
|
|||
```bash |
|||
for ns in $(rancher kubectl get pods -A -l app.kubernetes.io/name=sapl \ |
|||
-o jsonpath='{.items[*].metadata.namespace}' | tr ' ' '\n' | sort -u); do |
|||
rancher kubectl set env deployment/sapl \ |
|||
REDIS_URL=redis://redis.sapl-redis.svc.cluster.local:6379 \ |
|||
-n $ns |
|||
done |
|||
``` |
|||
|
|||
### Roll back |
|||
|
|||
```bash |
|||
for ns in $(rancher kubectl get pods -A -l app.kubernetes.io/name=sapl \ |
|||
-o jsonpath='{.items[*].metadata.namespace}' | tr ' ' '\n' | sort -u); do |
|||
rancher kubectl set env deployment/sapl REDIS_URL- -n $ns |
|||
done |
|||
``` |
|||
|
|||
`kubectl set env deployment/sapl REDIS_URL-` (trailing `-`) removes the variable. |
|||
`start.sh` then falls back to file-based cache automatically. |
|||
|
|||
--- |
|||
|
|||
## Monitor |
|||
|
|||
### Pod and events |
|||
|
|||
```bash |
|||
# Pod status |
|||
rancher kubectl -n sapl-redis get pods -l app=sapl-redis -o wide |
|||
|
|||
# Deployment events (useful right after apply) |
|||
rancher kubectl -n sapl-redis describe deployment sapl-redis |
|||
|
|||
# Pod events (OOMKill, restarts, etc.) |
|||
rancher kubectl -n sapl-redis describe pod -l app=sapl-redis |
|||
``` |
|||
|
|||
### Logs |
|||
|
|||
```bash |
|||
# Tail live logs |
|||
rancher kubectl -n sapl-redis logs -f deploy/sapl-redis |
|||
|
|||
# Last 100 lines |
|||
rancher kubectl -n sapl-redis logs deploy/sapl-redis --tail=100 |
|||
``` |
|||
|
|||
### Redis INFO |
|||
|
|||
```bash |
|||
# Memory usage |
|||
rancher kubectl exec -n sapl-redis deploy/sapl-redis -- \ |
|||
redis-cli info memory \ |
|||
| grep -E 'used_memory_human|maxmemory_human|mem_fragmentation_ratio' |
|||
|
|||
# Connection pressure |
|||
rancher kubectl exec -n sapl-redis deploy/sapl-redis -- \ |
|||
redis-cli info stats \ |
|||
| grep -E 'rejected_connections|instantaneous_ops_per_sec' |
|||
|
|||
# Key distribution per DB |
|||
rancher kubectl exec -n sapl-redis deploy/sapl-redis -- redis-cli info keyspace |
|||
|
|||
# Recent slow queries |
|||
rancher kubectl exec -n sapl-redis deploy/sapl-redis -- redis-cli slowlog get 10 |
|||
|
|||
# Live command sampling (1-second window) |
|||
rancher kubectl exec -n sapl-redis deploy/sapl-redis -- redis-cli --latency-history -i 1 |
|||
``` |
|||
|
|||
### Rate-limiter keys (DB 1) |
|||
|
|||
```bash |
|||
rancher kubectl exec -n sapl-redis deploy/sapl-redis -- \ |
|||
redis-cli -n 1 dbsize |
|||
|
|||
rancher kubectl exec -n sapl-redis deploy/sapl-redis -- \ |
|||
redis-cli -n 1 --scan --pattern 'rl:ip:*' | head -20 |
|||
``` |
|||
|
|||
--- |
|||
|
|||
## Seed the UA deny list (once after first deploy) |
|||
|
|||
`rl:bot:ua:blocked` is a permanent Redis SET in DB 1. Each member is the |
|||
SHA-256 of a **UA token** — the identifying fragment extracted after splitting |
|||
on `/`, spaces, `;`, `(`, `)`, e.g.: |
|||
|
|||
``` |
|||
UA string: "GPTBot/1.1 (+https://openai.com/gptbot)" |
|||
Tokens: GPTBot 1.1 +https: ... |
|||
Hash stored: sha256("GPTBot") |
|||
``` |
|||
|
|||
The middleware (`_is_redis_blocked_ua`) tokenises the incoming UA the same |
|||
way and checks each token hash against the cached set. The SET is fetched |
|||
from Redis at most once per `RATE_LIMITER_UA_BLOCKLIST_REFRESH` seconds (default 60) |
|||
per worker process. |
|||
|
|||
The bots in `BOT_UA_FRAGMENTS` (Python list, always active) and this Redis |
|||
SET are **independent** — the Python list provides the baseline and the Redis |
|||
SET allows adding new offenders at runtime **without a code deploy**. |
|||
|
|||
```bash |
|||
rancher kubectl exec -n sapl-redis deploy/sapl-redis -- redis-cli -n 1 \ |
|||
SADD rl:bot:ua:blocked \ |
|||
"$(echo -n 'GPTBot' | sha256sum | cut -d' ' -f1)" \ |
|||
"$(echo -n 'ClaudeBot' | sha256sum | cut -d' ' -f1)" \ |
|||
"$(echo -n 'PerplexityBot' | sha256sum | cut -d' ' -f1)" \ |
|||
"$(echo -n 'Bytespider' | sha256sum | cut -d' ' -f1)" \ |
|||
"$(echo -n 'AhrefsBot' | sha256sum | cut -d' ' -f1)" \ |
|||
"$(echo -n 'meta-externalagent' | sha256sum | cut -d' ' -f1)" |
|||
|
|||
# Add a new offender at runtime (picked up within RATE_LIMITER_UA_BLOCKLIST_REFRESH seconds) |
|||
rancher kubectl exec -n sapl-redis deploy/sapl-redis -- redis-cli -n 1 \ |
|||
SADD rl:bot:ua:blocked "$(echo -n 'NewBot' | sha256sum | cut -d' ' -f1)" |
|||
``` |
|||
|
|||
--- |
|||
|
|||
## Local standalone Redis (development / testing) |
|||
|
|||
No Kubernetes? Run Redis directly with Docker: |
|||
|
|||
```bash |
|||
sudo docker run --rm -p 6379:6379 redis:7-alpine \ |
|||
redis-server --save "" --appendonly no |
|||
``` |
|||
|
|||
Then point Django at it by exporting the env var before starting the dev server: |
|||
|
|||
```bash |
|||
export REDIS_URL="redis://localhost:6379" |
|||
export CACHE_BACKEND="redis" |
|||
python manage.py runserver |
|||
``` |
|||
|
|||
Or add them to your local `.env` file: |
|||
|
|||
``` |
|||
REDIS_URL=redis://localhost:6379 |
|||
CACHE_BACKEND=redis |
|||
``` |
|||
|
|||
> **Note**: the waffle switch `REDIS_CACHE` must also be `on` in your local |
|||
> database for `start.sh` to activate the Redis backend. Run: |
|||
> ```bash |
|||
> python manage.py waffle_switch REDIS_CACHE on --create |
|||
> ``` |
|||
|
|||
--- |
|||
|
|||
## Update `redis.conf` without redeploying |
|||
|
|||
```bash |
|||
# Edit the ConfigMap |
|||
rancher kubectl -n sapl-redis edit configmap redis-config |
|||
|
|||
# Restart the pod to pick up the new config |
|||
rancher kubectl -n sapl-redis rollout restart deployment/sapl-redis |
|||
``` |
|||
|
|||
--- |
|||
|
|||
## Rate limiting — two layers, two jobs |
|||
|
|||
SAPL enforces rate limits at two independent layers. They use different |
|||
algorithms and protect different things; their thresholds must be tuned |
|||
separately. |
|||
|
|||
### Layer 1 — nginx `limit_req` (leaky bucket) |
|||
|
|||
Defined in `docker/config/nginx/nginx.conf` (zones) and `sapl.conf` (burst). |
|||
|
|||
``` |
|||
sapl_general rate=30r/m # 1 token every 2 s |
|||
sapl_heavy rate=10r/m # 1 token every 6 s (PDF/report endpoints) |
|||
``` |
|||
|
|||
`burst=N nodelay` means nginx accepts up to N requests instantly above the |
|||
current token level, then enforces the drip rate. Requests beyond the burst |
|||
cap return 429 before reaching Gunicorn — **zero Python cost**. |
|||
|
|||
Burst values are set at container startup via env vars: |
|||
|
|||
| Env var | Default | Location | |
|||
|---------|---------|----------| |
|||
| `NGINX_BURST_GENERAL` | `60` | `location /`, `location /media/` | |
|||
| `NGINX_BURST_API` | `60` | `location /api/` | |
|||
| `NGINX_BURST_HEAVY` | `20` | `location /relatorios/` | |
|||
|
|||
Defaults are 2× the zone's per-minute rate, so a user can spend a full |
|||
minute's quota in a single burst before the leaky bucket takes over. |
|||
|
|||
### Layer 2 — Django `RateLimitMiddleware` (sliding window) |
|||
|
|||
Defined in `sapl/middleware/ratelimit.py`, backed by Redis DB 1. |
|||
|
|||
Requests that pass nginx reach Python. The middleware counts them in a |
|||
60-second sliding window per IP (anonymous) or per user (authenticated): |
|||
|
|||
| Env var | Default | Scope | |
|||
|---------|---------|-------| |
|||
| `RATE_LIMITER_RATE` | `35/m` | Anonymous IP | |
|||
| `RATE_LIMITER_RATE_AUTHENTICATED` | `120/m` | Authenticated user | |
|||
| `RATE_LIMITER_RATE_BOT` | `5/m` | *(reserved — bots are currently blocked outright, not counted)* | |
|||
| `RATE_LIMITER_UA_BLOCKLIST_REFRESH` | `60` s | How often each worker re-fetches `rl:bot:ua:blocked` from Redis | |
|||
|
|||
When the window count hits the threshold the IP/user is written to a Redis |
|||
blocked-set with a 300 s TTL and subsequent requests return 429 with |
|||
`Retry-After: 300` — without touching the database. |
|||
|
|||
Decision flow inside `RateLimitMiddleware._evaluate()`: |
|||
|
|||
``` |
|||
1. IP in whitelist? → pass (no further checks) |
|||
1a. UA matches BOT_UA_FRAGMENTS list? → 429 reason=known_ua |
|||
1b. UA token hash in rl:bot:ua:blocked SET? → 429 reason=redis_ua |
|||
2. IP in rl:ip:{ip}:blocked? → 429 reason=ip_blocked |
|||
3. Authenticated user? |
|||
3a. User in rl:{ns}:user:{uid}:blocked? → 429 reason=user_blocked |
|||
3b. Suspicious headers (no Accept/AL)? → 429 reason=suspicious_headers_auth |
|||
3c. User request count ≥ auth threshold? → SET blocked, 429 reason=auth_user_rate |
|||
4. Anonymous: |
|||
4a. Suspicious headers? → 429 reason=suspicious_headers |
|||
4b. IP request count ≥ anon threshold? → SET blocked, 429 reason=ip_rate |
|||
4c. NS/IP window count ≥ anon threshold? → SET blocked, 429 reason=ua_rotation |
|||
→ pass |
|||
``` |
|||
|
|||
### Why they are not the same number |
|||
|
|||
| | nginx burst | Django threshold | |
|||
|-|------------|-----------------| |
|||
| **Algorithm** | Leaky bucket — token refills over time | Sliding window — hard count per 60 s | |
|||
| **Protects** | Gunicorn workers from being flooded | Per-client fairness, business policy | |
|||
| **Tuned by** | Capacity of the server | Acceptable request volume per client | |
|||
| **Failure mode** | Workers overwhelmed | Legitimate user over-browsing | |
|||
|
|||
A user loading a page quickly may fire 5–10 Django requests in two seconds. |
|||
With `rate=30r/m` (1 token/2 s) and `burst=60` they absorb that fine; the |
|||
leaky bucket refills before they click the next link. The Django threshold |
|||
(35/m sliding window) catches sustained automated traffic from a single IP |
|||
that looks like scraping even if it arrives slowly enough to beat the nginx |
|||
burst cap. |
|||
|
|||
--- |
|||
|
|||
## Request routing — how nginx reaches Django |
|||
|
|||
`proxy_pass http://sapl_server` forwards the HTTP request — with the original |
|||
path intact — to the Gunicorn Unix socket. Django doesn't know or care that |
|||
nginx is in front; it sees a standard HTTP request. |
|||
|
|||
``` |
|||
GET /media/foo.pdf |
|||
│ |
|||
▼ |
|||
nginx (sapl.conf) |
|||
location /media/ → proxy_pass to Unix socket |
|||
│ |
|||
▼ |
|||
Gunicorn (WSGI server) |
|||
receives raw HTTP, calls Django WSGI application |
|||
│ |
|||
▼ |
|||
Django middleware stack (settings.MIDDLEWARE) |
|||
RateLimitMiddleware → pass or 429 |
|||
│ |
|||
▼ |
|||
Django URL router (sapl/urls.py) |
|||
r'^media/(?P<path>.*)$' → serve_media |
|||
│ |
|||
▼ |
|||
serve_media(request, path='foo.pdf') |
|||
returns HttpResponse with X-Accel-Redirect: /_accel/media/foo.pdf |
|||
│ |
|||
▼ |
|||
nginx sees X-Accel-Redirect header |
|||
/_accel/media/ internal location → reads file from disk → sends to client |
|||
``` |
|||
|
|||
nginx does no routing beyond picking a `location` block. The mapping from |
|||
URL path to Python function lives entirely in `sapl/urls.py`. `proxy_pass` is |
|||
just a pipe. |
|||
|
|||
--- |
|||
|
|||
## Media file serving — `serve_media` and X-Accel-Redirect |
|||
|
|||
All `/media/` requests (public and private) are routed through Gunicorn so that |
|||
Django middleware runs on every hit. Nginx serves the file bytes via |
|||
`X-Accel-Redirect` — the Gunicorn worker is freed as soon as it sends the |
|||
response headers. |
|||
|
|||
### nginx locations (`docker/config/nginx/sapl.conf`) |
|||
|
|||
```nginx |
|||
# Proxied to Gunicorn — Django middleware + serve_media() run here. |
|||
location /media/ { |
|||
limit_req zone=sapl_general burst=${NGINX_BURST_GENERAL} nodelay; |
|||
proxy_pass http://sapl_server; |
|||
} |
|||
|
|||
# Internal — only reachable via X-Accel-Redirect, not by external clients. |
|||
location /_accel/media/ { |
|||
internal; |
|||
alias /var/interlegis/sapl/media/; |
|||
sendfile on; |
|||
etag on; |
|||
} |
|||
``` |
|||
|
|||
### Django view (`sapl/base/media.py`) |
|||
|
|||
`serve_media(request, path)` — registered at `^media/(?P<path>.*)$` in `sapl/urls.py`. |
|||
|
|||
Per-request steps: |
|||
|
|||
1. **Path traversal guard** — `os.path.abspath` check; raises 404 on escape. |
|||
2. **Auth gate** — `documentos_privados/` paths require an authenticated session; redirects to login otherwise. |
|||
3. **Path counter** — increments `rl:{ns}:path:{sha256}:reqs` in Redis DB 1 (TTL = `MEDIA_PATH_COUNTER_TTL`). |
|||
4. **Content-type cache** — reads `file:{ns}:{sha256}` from Django default cache (DB 0); on miss, calls `mimetypes.guess_type`, stores result (TTL = `MEDIA_FILE_CACHE_TTL`). |
|||
5. **Serve** — in DEBUG: `django.views.static.serve` directly. In production: `X-Accel-Redirect: /_accel/media/<path>`. |
|||
|
|||
### Settings |
|||
|
|||
| Setting | Default | Purpose | |
|||
|---------|---------|---------| |
|||
| `FILE_META_KEY` | `'file:{ns}:{sha256}'` | Key template for content-type cache (DB 0) | |
|||
| `MEDIA_PATH_COUNTER_TTL` | `60` s | Per-path counter window | |
|||
| `MEDIA_FILE_CACHE_TTL` | `3600` s | Content-type metadata TTL | |
|||
|
|||
--- |
|||
|
|||
## Key schema reference |
|||
|
|||
| DB | Use case | Key pattern | TTL | Constant | |
|||
|----|----------|-------------|-----|----------| |
|||
| 0 | Page / view cache | `cache:{ns}:*` | 300 s (default) | `CACHES['default']` KEY_PREFIX | |
|||
| 0 | Static file cache (logos) | `static:{ns}:{sha256}` | 3 – 24 h | *Future* (requires OpenResty/Lua) | |
|||
| 0 | Media file content-type cache | `file:{ns}:{sha256}` | 1 h | `FILE_META_KEY` | |
|||
| 1 | IP rate-limit counter | `rl:ip:{ip}:reqs` | 60 s | `RL_IP_REQUESTS` | |
|||
| 1 | IP blocked marker | `rl:ip:{ip}:blocked` | 300 s | `RL_IP_BLOCKED` | |
|||
| 1 | User rate-limit counter | `rl:{ns}:user:{uid}:reqs` | 60 s | `RL_USER_REQUESTS` | |
|||
| 1 | User blocked marker | `rl:{ns}:user:{uid}:blocked` | 300 s | `RL_USER_BLOCKED` | |
|||
| 1 | Namespace/IP sliding window | `rl:{ns}:ip:{ip}:w:{bucket}` | 120 s | `RL_NS_WINDOW` | |
|||
| 1 | Path counter (`/media/`) | `rl:{ns}:path:{sha256}:reqs` | 60 s | `RL_PATH_REQUESTS` | |
|||
| 1 | Path counter (`/static/`) | `rl:{ns}:path:{sha256}:reqs` | 60 s | *Future* (requires OpenResty/Lua) | |
|||
| 1 | UA deny list | `rl:bot:ua:blocked` | permanent SET | `RL_UA_BLOCKLIST` | |
|||
| 2 | Django Channels | `channels:*` | session TTL | *Future* | |
|||
File diff suppressed because it is too large
@ -0,0 +1,96 @@ |
|||
""" |
|||
serve_media — X-Accel-Redirect gate for all /media/ files. |
|||
|
|||
Production flow (nginx proxies /media/ to Gunicorn): |
|||
1. Django middleware runs (IP rate-limit, bot UA check, etc.). |
|||
2. serve_media() runs auth check for documentos_privados/, writes |
|||
per-path counter to Redis DB 1, caches content-type in Redis DB 0. |
|||
3. Returns an empty 200 with X-Accel-Redirect pointing to the nginx |
|||
internal location /_accel/media/<path>. Nginx serves the bytes |
|||
directly from disk — Gunicorn worker is freed immediately. |
|||
|
|||
Development flow (DEBUG=True, nginx absent): |
|||
Falls back to django.views.static.serve for live file serving. |
|||
|
|||
Redis side-effects per request: |
|||
DB 1 rl:{ns}:path:{sha256}:reqs — per-path access counter, TTL=MEDIA_PATH_COUNTER_TTL |
|||
DB 0 file:{ns}:{sha256} — content-type metadata, TTL=MEDIA_FILE_CACHE_TTL |
|||
(sha256 is of the URL path, e.g. sha256('/media/2024/01/doc.pdf')) |
|||
Key template: FILE_META_KEY (sapl/middleware/ratelimit.py); TTLs in sapl/settings.py |
|||
""" |
|||
|
|||
import hashlib |
|||
import mimetypes |
|||
import os |
|||
|
|||
from django.conf import settings |
|||
from django.core.cache import caches |
|||
from django.http import Http404, HttpResponse |
|||
from django.views.static import serve |
|||
|
|||
from sapl import settings as sapl_settings |
|||
from sapl.middleware.ratelimit import ( |
|||
_NAMESPACE, |
|||
FILE_META_KEY, |
|||
RL_PATH_REQUESTS, |
|||
_incr_with_ttl, |
|||
) |
|||
|
|||
|
|||
def _safe_resolve(rel_path): |
|||
""" |
|||
Return the absolute path of rel_path inside MEDIA_ROOT. |
|||
Raises Http404 if the resolved path would escape the root |
|||
(path traversal guard). |
|||
""" |
|||
abs_root = os.path.abspath(settings.MEDIA_ROOT) |
|||
abs_path = os.path.abspath(os.path.join(abs_root, rel_path)) |
|||
if not abs_path.startswith(abs_root + os.sep) and abs_path != abs_root: |
|||
raise Http404 |
|||
return abs_path |
|||
|
|||
|
|||
def serve_media(request, path): |
|||
""" |
|||
Registered in sapl/urls.py for both DEBUG and production. |
|||
Route: ^media/(?P<path>.*)$ |
|||
""" |
|||
# Path traversal guard — raises Http404 on escape attempt. |
|||
abs_path = _safe_resolve(path) |
|||
|
|||
# Auth gate for private documents — redirect to login if anonymous. |
|||
if path.startswith('documentos_privados/'): |
|||
user = getattr(request, 'user', None) |
|||
if user is None or not user.is_authenticated: |
|||
from django.contrib.auth.views import redirect_to_login |
|||
return redirect_to_login(request.get_full_path()) |
|||
|
|||
# Per-path rate counter (DB 1) — key uses URL path so that storage |
|||
# location changes in the next PR don't reset existing counters. |
|||
path_hash = hashlib.sha256(f'/media/{path}'.encode()).hexdigest() |
|||
_incr_with_ttl( |
|||
RL_PATH_REQUESTS.format(ns=_NAMESPACE, sha256=path_hash), |
|||
ttl=sapl_settings.MEDIA_PATH_COUNTER_TTL, |
|||
) |
|||
|
|||
# Content-type metadata cache (DB 0) — avoids mimetypes.guess_type |
|||
# and os.path.isfile on every hit for hot files. |
|||
file_key = FILE_META_KEY.format(ns=_NAMESPACE, sha256=path_hash) |
|||
content_type = caches['default'].get(file_key) |
|||
if content_type is None: |
|||
if not os.path.isfile(abs_path): |
|||
raise Http404 |
|||
guessed, _ = mimetypes.guess_type(abs_path) |
|||
content_type = guessed or 'application/octet-stream' |
|||
caches['default'].set(file_key, content_type, timeout=sapl_settings.MEDIA_FILE_CACHE_TTL) |
|||
|
|||
if settings.DEBUG: |
|||
# Development: no nginx present; serve the file directly. |
|||
return serve(request, path, document_root=settings.MEDIA_ROOT) |
|||
|
|||
# Production: tell nginx to serve the file from the internal location. |
|||
response = HttpResponse(content_type=content_type) |
|||
response['X-Accel-Redirect'] = f'/_accel/media/{path}' |
|||
response['Cache-Control'] = 'public, max-age=86400, stale-while-revalidate=3600' |
|||
response['X-Robots-Tag'] = 'noindex' |
|||
return response |
|||
Loading…
Reference in new issue