SPDT/sapl - sapl - Gitea: Git with a cup of tea

5872 Commits

62 Branches

269 Tags

232 MiB

Tree: e580f8ec8d

Commit Graph

Author	SHA1	Message	Date
Edward Ribeiro	b321b57b78	GeoIP offline build; Redis inspection tools; smart_rate/smart_key; cache KEY_PREFIX GeoIP (docker/Dockerfile): Remove at-build-time MaxMind download (required BuildKit secrets, caused cache-miss issues). Replace with COPY from docker/geoip/GeoLite2-ASN.mmdb (git-ignored binary). If absent, build succeeds with ASN blocking disabled. Add docker/geoip/update_geoip.sh — run before each build to refresh the database from MaxMind using MAXMIND_LICENSE_KEY from env or .env file. Redis inspection / synthetic test data: Add docker/scripts/redis_populate_test_data.py — injects synthetic rl:* entries into Redis DB1 to validate key schema and blocking thresholds without waiting for real traffic. Supports DRY_RUN and CLEAR modes. Add §4.5 (Redis CLI quick-reference + RedisInsight guide) to rate-limiter-v2.md. Auth-aware @ratelimit decorators (smart_rate / smart_key): All 51 @ratelimit decorators across 9 files used rate=RATE_LIMITER_RATE (35/m) regardless of authentication, silently over-throttling logged-in users compared to what RateLimitMiddleware allows (120/m). Add smart_key() and smart_rate() to sapl/middleware/ratelimit.py: - smart_key: user pk for authenticated requests, masked IP for anon - smart_rate: RATE_LIMITER_RATE_AUTHENTICATED (120/m) for auth, RATE_LIMITER_RATE (35/m) for anon — mirrors middleware thresholds Update all 51 decorators across crud/base.py + 8 view files. Remove now-unused RATE_LIMITER_RATE imports from those files. Cache KEY_PREFIX (settings.py): Change KEY_PREFIX from POD_NAMESPACE ("sapl") to f"cache:{POD_NAMESPACE}" so DB0 cache keys are unambiguously prefixed cache:{ns}:* — distinct from any future static or file cache key patterns. Update key schema table and code examples in rate-limiter-v2.md to match. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2 months ago
Edward Ribeiro	e85f5e0847	Fix remaining get_client_ip stale imports; split Phase 5 to work_queues.md Import fixes (all three imported get_client_ip/ratelimit_ip from sapl.utils which no longer exports them — causing the ImportError at startup): - sapl/materia/forms.py: move get_client_ip to sapl.middleware.ratelimit - sapl/materia/views.py: move get_client_ip + ratelimit_ip; keep RATE_LIMITER_RATE in sapl.settings (used by @ratelimit decorators) - sapl/base/views.py: same pattern as materia/views.py Docs: - rate-limiter-v2.md: remove Phase 5 section (§8); renumber Open Questions to §8; update Table of Contents - work_queues.md (new): Async PDF via Celery + Django Channels WebSocket voting panel, with full context, Redis B topology rationale, k8s manifest list, and open questions. Planned start: after rate-limiter-2026 is stable. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2 months ago
Edward Ribeiro	8df2861799	Phase 0 hardening: nginx GeoIP2, rate limits, Gunicorn tuning, N+1 fix - nginx: sendfile on, tcp_nopush, reduced keepalive/proxy timeouts - nginx: GeoIP2 ASN-based bot blocking (cloud providers + known scrapers) - nginx: UA blocklist (GPTBot, ClaudeBot, Chrome/98.0.4758 impersonator, etc.) - nginx: rate-limit zones (30r/m general, 10r/m heavy/relatorios), 429/500 error pages - nginx: proper ETags + Cache-Control on /media/ to stop 30GB logo re-transfers - Dockerfile: install libnginx-mod-http-geoip2; download GeoLite2-ASN.mmdb via BuildKit secret (key never baked into image layers); ARG GEOIP_CACHE_BUST for forced re-download without --no-cache - Gunicorn: workers 3->2, threads 8->4, timeout 300->120, max_memory 300->400MB - Django: FILE_UPLOAD_MAX_MEMORY_SIZE=2MB, FILE_UPLOAD_TEMP_DIR for large uploads - relatorios/views.py: fix N+1 in get_etiqueta_protocolos with bulk-fetch MateriaLegislativa + DocumentoAdministrativo using select_related + dict lookups - Add robots.txt, 429.html, 500.html static pages - docker-compose.yaml: use sapl:local for local dev - docker/README.md: build instructions with MAXMIND_LICENSE_KEY - rate-limiter-v2.md: canonical planning document (Architecture through Phase 5) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2 months ago

Author

SHA1

Message

Date

Edward Ribeiro

b321b57b78

GeoIP offline build; Redis inspection tools; smart_rate/smart_key; cache KEY_PREFIX

GeoIP (docker/Dockerfile):
  Remove at-build-time MaxMind download (required BuildKit secrets, caused
  cache-miss issues). Replace with COPY from docker/geoip/GeoLite2-ASN.mmdb
  (git-ignored binary). If absent, build succeeds with ASN blocking disabled.
  Add docker/geoip/update_geoip.sh — run before each build to refresh the
  database from MaxMind using MAXMIND_LICENSE_KEY from env or .env file.

Redis inspection / synthetic test data:
  Add docker/scripts/redis_populate_test_data.py — injects synthetic rl:*
  entries into Redis DB1 to validate key schema and blocking thresholds
  without waiting for real traffic. Supports DRY_RUN and CLEAR modes.
  Add §4.5 (Redis CLI quick-reference + RedisInsight guide) to rate-limiter-v2.md.

Auth-aware @ratelimit decorators (smart_rate / smart_key):
  All 51 @ratelimit decorators across 9 files used rate=RATE_LIMITER_RATE
  (35/m) regardless of authentication, silently over-throttling logged-in
  users compared to what RateLimitMiddleware allows (120/m).
  Add smart_key() and smart_rate() to sapl/middleware/ratelimit.py:
    - smart_key: user pk for authenticated requests, masked IP for anon
    - smart_rate: RATE_LIMITER_RATE_AUTHENTICATED (120/m) for auth,
      RATE_LIMITER_RATE (35/m) for anon — mirrors middleware thresholds
  Update all 51 decorators across crud/base.py + 8 view files.
  Remove now-unused RATE_LIMITER_RATE imports from those files.

Cache KEY_PREFIX (settings.py):
  Change KEY_PREFIX from POD_NAMESPACE ("sapl") to f"cache:{POD_NAMESPACE}"
  so DB0 cache keys are unambiguously prefixed cache:{ns}:* — distinct from
  any future static or file cache key patterns.
  Update key schema table and code examples in rate-limiter-v2.md to match.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

2 months ago

Edward Ribeiro

e85f5e0847

Fix remaining get_client_ip stale imports; split Phase 5 to work_queues.md

Import fixes (all three imported get_client_ip/ratelimit_ip from sapl.utils
which no longer exports them — causing the ImportError at startup):
- sapl/materia/forms.py: move get_client_ip to sapl.middleware.ratelimit
- sapl/materia/views.py: move get_client_ip + ratelimit_ip; keep
  RATE_LIMITER_RATE in sapl.settings (used by @ratelimit decorators)
- sapl/base/views.py: same pattern as materia/views.py

Docs:
- rate-limiter-v2.md: remove Phase 5 section (§8); renumber Open Questions
  to §8; update Table of Contents
- work_queues.md (new): Async PDF via Celery + Django Channels WebSocket
  voting panel, with full context, Redis B topology rationale, k8s manifest
  list, and open questions. Planned start: after rate-limiter-2026 is stable.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

2 months ago

Edward Ribeiro

8df2861799

Phase 0 hardening: nginx GeoIP2, rate limits, Gunicorn tuning, N+1 fix

- nginx: sendfile on, tcp_nopush, reduced keepalive/proxy timeouts
- nginx: GeoIP2 ASN-based bot blocking (cloud providers + known scrapers)
- nginx: UA blocklist (GPTBot, ClaudeBot, Chrome/98.0.4758 impersonator, etc.)
- nginx: rate-limit zones (30r/m general, 10r/m heavy/relatorios), 429/500 error pages
- nginx: proper ETags + Cache-Control on /media/ to stop 30GB logo re-transfers
- Dockerfile: install libnginx-mod-http-geoip2; download GeoLite2-ASN.mmdb via
  BuildKit secret (key never baked into image layers); ARG GEOIP_CACHE_BUST for
  forced re-download without --no-cache
- Gunicorn: workers 3->2, threads 8->4, timeout 300->120, max_memory 300->400MB
- Django: FILE_UPLOAD_MAX_MEMORY_SIZE=2MB, FILE_UPLOAD_TEMP_DIR for large uploads
- relatorios/views.py: fix N+1 in get_etiqueta_protocolos with bulk-fetch
  MateriaLegislativa + DocumentoAdministrativo using select_related + dict lookups
- Add robots.txt, 429.html, 500.html static pages
- docker-compose.yaml: use sapl:local for local dev
- docker/README.md: build instructions with MAXMIND_LICENSE_KEY
- rate-limiter-v2.md: canonical planning document (Architecture through Phase 5)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

2 months ago

3 Commits (e580f8ec8d302fdb88c6a3a62da9bbdf586798fe)