BioD Hub

The service exposes secure endpoints for managing users, environmental sensors, animal detections, hydrological and soil measurements, OCR workflows, and analytics that help conservation teams monitor biodiversity in real time.

Overview

Auth & authorization: OAuth2 password flow with JWT tokens; fine-grained collection permissions checked on every request.
Data ingestion: REST endpoints accept sensor readings, environmental metadata, and animal detection payloads (including image uploads streamed to MinIO/S3).
Storage: MongoDB acts as the system of record; MinIO stores raw media; Supabase endpoints enrich plant and insect digitization statistics.
Monitoring: Middleware captures per-request transactions and an APScheduler job summarises daily analytics.
Automation ready: Dockerfile and docker-compose.yml are provided for reproducible deployments.

Tech Stack

FastAPI & Starlette
Motor (async MongoDB driver) and Pydantic v2 models
MinIO (S3-compatible object storage) for media assets
APScheduler for background jobs
OpenAI Responses API for OCR text extraction
pytest + FastAPI TestClient for basic regression tests

Project Layout

root ├── main.py # FastAPI app entry point ├── apps/ │ ├── config/ # Middleware, auth helpers, exception helpers, Supabase integration │ ├── database/ # Mongo client bootstrap and Pydantic models │ ├── router/ # FastAPI routers grouped by domain (users, sensors, animals, OCR, statistics, …) │ ├── scheduler/ # APScheduler transaction analytics │ └── utils/ # Shared helpers (media paths, preprocessing, dedupe logic) ├── test/ # pytest suites ├── Dockerfile ├── docker-compose.yml └── requirements.txt

Getting Started

Prerequisites

Python 3.11+
MongoDB instance accessible via connection string
MinIO (or any S3-compatible storage) bucket for media files
Optional: Docker & Docker Compose for containerised runs
Optional: OpenAI API key (OCR), Supabase endpoints (statistics enrichment)

Installation

git clone <repo-url>
python -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate
pip install -r requirements.txt

Environment Variables

Create a .env file (or export variables) before running the API. Keys marked (required) must be set for core functionality.

Variable	Description
`SECRET_KEY` (required)	JWT signing secret used for access tokens.
`ALGORITHM` (required)	JWT algorithm (e.g. `HS256`).
`MONGO_DETAILS` (required)	MongoDB connection string.
`MINIO_URI` (required)	Host:port for MinIO/S3 endpoint.
`MINIO_ACCESS_KEY` (required)	Access key for MinIO/S3.
`MINIO_SECRET_KEY` (required)	Secret key for MinIO/S3.
`BUCKET_NAME` (required)	Name of the MinIO/S3 bucket used for uploads.
`LOCAL_BUCKET_DOMAIN` (required)	Base URL served to clients on local/private networks.
`EXTERNAL_BUCKET_DOMAIN` (required)	Base URL served to external clients.
`DEVELOPMENT`	When truthy, uploaded paths are prefixed with `dev/` for sandbox buckets.
`SHOW_USERS`	Controls whether user-management endpoints appear in the OpenAPI schema.
`MEDIA_DIR`, `ANIMALS_DIR`, `INSECTS_DIR`, `PLANTS_DIR`, `DRONE_DIR`	Optional filesystem paths used when mounting volumes in Docker deployments.
`OPENAI_API_KEY`	Required for OCR endpoints; leave unset to disable OCR.
`MODEL`, `ROLE`	OpenAI model ID and role string for OCR requests.
`PLANT_SUPABASE_URL`, `INSECT_SUPABASE_URL`	Supabase Function URLs consumed by statistics endpoints.
`WEB_APP_EMAIL`, `WEB_APP_PASSWORD`	Credentials used to fetch Supabase access tokens.

Run the API

Local Development

uvicorn main:app --reload --host 0.0.0.0 --port 8000

Interactive docs are available at http://localhost:8000/docs (Swagger) and /redoc.

Docker Compose

docker compose up --build

Update docker-compose.yml or your .env file with the correct environment values before starting.

Scheduler & Analytics

Startup hooks register the transaction processor (apps/scheduler/transaction_processor.py), which:

Logs each API call in api_transactions via TransactionMiddleware.
Once per day, aggregates metrics (latency, error rates, transfer volume, top endpoints, user agents) into api_analytics.
Writes human-readable logs to transaction_processor.log.

Ensure the MongoDB user has write privileges on both collections.

API Usage

Authentication Flow

POST /auth/token with form data (username, password) to obtain a bearer token.
Include Authorization: Bearer <token> in subsequent requests.
Authorization rules are enforced per collection via allowed_collection and collections_permissions stored on the user record.

curl -X POST http://localhost:8000/auth/token \
  -H "Content-Type: application/x-www-form-urlencoded" \
  -d "username=demo_user&password=Pa55word!"

Core Routers

Prefix	Tag	Purpose
`/user`	User	Manage API users (CRUD, password updates) with permission checks.
`/auth`	auth	Issue OAuth2 bearer tokens for authenticated access.
`/camera_trap`	Camera Trap	Retrieve and update camera trap metadata (auto-created during animal detection uploads).
`/animals_detection`	Animals Detection	Ingest detection payloads, upload images to MinIO, filter detections by camera, species, and date.
`/sensors`, `/env_data`, `/hydro_data`, `/soil_data`	Sensors	Store and query sensor metadata plus environment, hydrological, and soil readings with pagination and filtering.
`/ocr`	OCR	Submit base64-encoded imagery and receive extracted text via OpenAI.
`/statistic`	Statistic	Aggregated counts for sensors, animal detections, and published datasets pulled from Supabase.

Additional routers for drones, legged robots, plant, and insect digitization exist under apps/router/ and can be enabled via apps/config/include.py when the supporting datasets and storage are available.

Media Uploads

Image uploads are streamed directly to MinIO via the shared client in apps/database/db.py.
LOCAL_BUCKET_DOMAIN versus EXTERNAL_BUCKET_DOMAIN ensures returned URLs are reachable from both private and public networks.
Object keys are organised under dated folders with UUID filenames to avoid collisions; enabling DEVELOPMENT keeps uploads in a separate dev/ namespace.

Logging & Troubleshooting

Request/response metrics land in MongoDB collections api_transactions and api_analytics.
Scheduler output is written to transaction_processor.log.
MinIO warnings are printed when buckets are missing during startup (apps/utils/media.check_bucket).
For OCR issues, verify OPENAI_API_KEY, MODEL, and ROLE values and confirm outbound network access.

Testing

To access the API interface for testing and usage, please click here to open the interactive documentation page.