SPDT/sapl - sapl - Gitea: Git with a cup of tea

5917 Commits

62 Branches

269 Tags

233 MiB

Tree: 5086491157

Commit Graph

Author	SHA1	Message	Date
Edward Ribeiro	9140f3e4b8	Phase 1-4: Redis infra, rate limiter middleware, cache layer, nginx hardening docker/Dockerfile: - GeoIP offline build with MaxMind secret; optional build args for graphviz, poppler, psql client; envsubst for nginx burst vars docker/docker-compose.yaml: - saplredis service (redis:7-alpine, allkeys-lru, 512 MB) - REDIS_URL + CACHE_BACKEND wired into sapl service docker/startup_scripts/start.sh: - configure_redis_cache(): builds CACHES dict, sets REDIS_CACHE waffle switch, falls back to file cache gracefully - POD_NAMESPACE resolution (k8s Downward API → hostname fallback) - DATABASE_URL exported before migrate docker/k8s/redis/ (moved from docker/k8s/): - redis-configmap.yaml, redis-deployment.yaml, redis-service.yaml - ClusterIP service on port 6379, sapl-redis namespace docker/k8s/sapl-k8s.yaml: - REDIS_URL env var injected; app.kubernetes.io/name=sapl label for fleet-wide discovery sapl/middleware/test_ratelimiter.py: - Unit tests for RateLimitMiddleware with mocked Redis scripts/test_ratelimiter.py: - CLI smoke-test: fires N requests and reports first 429 Removed: rate-limiter-v2.md (content migrated to plan/RATE_LIMITER_PLAN.md), scripts/test_ratelimiter.sh (replaced by .py), docker/k8s/README.md (merged into plan/RATE_LIMITER_PLAN.md), docker/scripts/redis_populate_test_data.py (renamed to redis_inject_test_data.py) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2 months ago
Edward Ribeiro	bea9d84373	Phase 5: X-Accel-Redirect for /media/, UA Redis deny list, per-path counters nginx: - /media/ proxied through Gunicorn (sapl_general rate limit) instead of direct alias — Django middleware now runs on every media request - /_accel/media/ internal location serves file bytes via X-Accel-Redirect sapl/base/media.py (new): - serve_media() gate: path traversal guard, auth redirect for documentos_privados/, per-path Redis counter, content-type metadata cache, X-Accel-Redirect response; falls back to Django serve() in DEBUG sapl/middleware/ratelimit.py: - RL_PATH_REQUESTS, RL_UA_BLOCKLIST, FILE_META_KEY constants - _incr_with_ttl() extracted to module level (reused by media.py) - Runtime UA deny list: _refresh_ua_blocklist() fetches rl:bot:ua:blocked SET from Redis (SMEMBERS, cached per worker, TTL=RATE_LIMITER_UA_BLOCKLIST_REFRESH); _is_redis_blocked_ua() tokenises UA and checks sha256 of each token sapl/settings.py: - RATE_LIMITER_UA_BLOCKLIST_REFRESH, MEDIA_PATH_COUNTER_TTL, MEDIA_FILE_CACHE_TTL added (all env-tunable via config()) plan/RATE_LIMITER_PLAN.md: - Key schema table updated; media file serving section added; decision flow documented; UA deny list seed section expanded Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2 months ago
Edward Ribeiro	b321b57b78	GeoIP offline build; Redis inspection tools; smart_rate/smart_key; cache KEY_PREFIX GeoIP (docker/Dockerfile): Remove at-build-time MaxMind download (required BuildKit secrets, caused cache-miss issues). Replace with COPY from docker/geoip/GeoLite2-ASN.mmdb (git-ignored binary). If absent, build succeeds with ASN blocking disabled. Add docker/geoip/update_geoip.sh — run before each build to refresh the database from MaxMind using MAXMIND_LICENSE_KEY from env or .env file. Redis inspection / synthetic test data: Add docker/scripts/redis_populate_test_data.py — injects synthetic rl:* entries into Redis DB1 to validate key schema and blocking thresholds without waiting for real traffic. Supports DRY_RUN and CLEAR modes. Add §4.5 (Redis CLI quick-reference + RedisInsight guide) to rate-limiter-v2.md. Auth-aware @ratelimit decorators (smart_rate / smart_key): All 51 @ratelimit decorators across 9 files used rate=RATE_LIMITER_RATE (35/m) regardless of authentication, silently over-throttling logged-in users compared to what RateLimitMiddleware allows (120/m). Add smart_key() and smart_rate() to sapl/middleware/ratelimit.py: - smart_key: user pk for authenticated requests, masked IP for anon - smart_rate: RATE_LIMITER_RATE_AUTHENTICATED (120/m) for auth, RATE_LIMITER_RATE (35/m) for anon — mirrors middleware thresholds Update all 51 decorators across crud/base.py + 8 view files. Remove now-unused RATE_LIMITER_RATE imports from those files. Cache KEY_PREFIX (settings.py): Change KEY_PREFIX from POD_NAMESPACE ("sapl") to f"cache:{POD_NAMESPACE}" so DB0 cache keys are unambiguously prefixed cache:{ns}:* — distinct from any future static or file cache key patterns. Update key schema table and code examples in rate-limiter-v2.md to match. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2 months ago

Author

SHA1

Message

Date

Edward Ribeiro

9140f3e4b8

Phase 1-4: Redis infra, rate limiter middleware, cache layer, nginx hardening

docker/Dockerfile:
- GeoIP offline build with MaxMind secret; optional build args for
  graphviz, poppler, psql client; envsubst for nginx burst vars

docker/docker-compose.yaml:
- saplredis service (redis:7-alpine, allkeys-lru, 512 MB)
- REDIS_URL + CACHE_BACKEND wired into sapl service

docker/startup_scripts/start.sh:
- configure_redis_cache(): builds CACHES dict, sets REDIS_CACHE waffle
  switch, falls back to file cache gracefully
- POD_NAMESPACE resolution (k8s Downward API → hostname fallback)
- DATABASE_URL exported before migrate

docker/k8s/redis/ (moved from docker/k8s/):
- redis-configmap.yaml, redis-deployment.yaml, redis-service.yaml
- ClusterIP service on port 6379, sapl-redis namespace

docker/k8s/sapl-k8s.yaml:
- REDIS_URL env var injected; app.kubernetes.io/name=sapl label for
  fleet-wide discovery

sapl/middleware/test_ratelimiter.py:
- Unit tests for RateLimitMiddleware with mocked Redis

scripts/test_ratelimiter.py:
- CLI smoke-test: fires N requests and reports first 429

Removed: rate-limiter-v2.md (content migrated to plan/RATE_LIMITER_PLAN.md),
         scripts/test_ratelimiter.sh (replaced by .py),
         docker/k8s/README.md (merged into plan/RATE_LIMITER_PLAN.md),
         docker/scripts/redis_populate_test_data.py (renamed to redis_inject_test_data.py)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

2 months ago

Edward Ribeiro

bea9d84373

Phase 5: X-Accel-Redirect for /media/, UA Redis deny list, per-path counters

nginx:
- /media/ proxied through Gunicorn (sapl_general rate limit) instead of
  direct alias — Django middleware now runs on every media request
- /_accel/media/ internal location serves file bytes via X-Accel-Redirect

sapl/base/media.py (new):
- serve_media() gate: path traversal guard, auth redirect for
  documentos_privados/, per-path Redis counter, content-type metadata
  cache, X-Accel-Redirect response; falls back to Django serve() in DEBUG

sapl/middleware/ratelimit.py:
- RL_PATH_REQUESTS, RL_UA_BLOCKLIST, FILE_META_KEY constants
- _incr_with_ttl() extracted to module level (reused by media.py)
- Runtime UA deny list: _refresh_ua_blocklist() fetches rl:bot:ua:blocked
  SET from Redis (SMEMBERS, cached per worker, TTL=RATE_LIMITER_UA_BLOCKLIST_REFRESH);
  _is_redis_blocked_ua() tokenises UA and checks sha256 of each token

sapl/settings.py:
- RATE_LIMITER_UA_BLOCKLIST_REFRESH, MEDIA_PATH_COUNTER_TTL,
  MEDIA_FILE_CACHE_TTL added (all env-tunable via config())

plan/RATE_LIMITER_PLAN.md:
- Key schema table updated; media file serving section added;
  decision flow documented; UA deny list seed section expanded

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

2 months ago

Edward Ribeiro

b321b57b78

GeoIP offline build; Redis inspection tools; smart_rate/smart_key; cache KEY_PREFIX

GeoIP (docker/Dockerfile):
  Remove at-build-time MaxMind download (required BuildKit secrets, caused
  cache-miss issues). Replace with COPY from docker/geoip/GeoLite2-ASN.mmdb
  (git-ignored binary). If absent, build succeeds with ASN blocking disabled.
  Add docker/geoip/update_geoip.sh — run before each build to refresh the
  database from MaxMind using MAXMIND_LICENSE_KEY from env or .env file.

Redis inspection / synthetic test data:
  Add docker/scripts/redis_populate_test_data.py — injects synthetic rl:*
  entries into Redis DB1 to validate key schema and blocking thresholds
  without waiting for real traffic. Supports DRY_RUN and CLEAR modes.
  Add §4.5 (Redis CLI quick-reference + RedisInsight guide) to rate-limiter-v2.md.

Auth-aware @ratelimit decorators (smart_rate / smart_key):
  All 51 @ratelimit decorators across 9 files used rate=RATE_LIMITER_RATE
  (35/m) regardless of authentication, silently over-throttling logged-in
  users compared to what RateLimitMiddleware allows (120/m).
  Add smart_key() and smart_rate() to sapl/middleware/ratelimit.py:
    - smart_key: user pk for authenticated requests, masked IP for anon
    - smart_rate: RATE_LIMITER_RATE_AUTHENTICATED (120/m) for auth,
      RATE_LIMITER_RATE (35/m) for anon — mirrors middleware thresholds
  Update all 51 decorators across crud/base.py + 8 view files.
  Remove now-unused RATE_LIMITER_RATE imports from those files.

Cache KEY_PREFIX (settings.py):
  Change KEY_PREFIX from POD_NAMESPACE ("sapl") to f"cache:{POD_NAMESPACE}"
  so DB0 cache keys are unambiguously prefixed cache:{ns}:* — distinct from
  any future static or file cache key patterns.
  Update key schema table and code examples in rate-limiter-v2.md to match.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

2 months ago

3 Commits (5086491157a70c97e6cff6e119e86e64d1ac592b)