Anyone who knows me knows that I self host everything. My entire company's technology stack is run on Open Source, Forked Open Source or developed from scratch (in that order). My email, my invoicing, my web servers, my git servers - everything is self hosted. I run a couple of Proxmox clusters (one personal, one for business) in EU Data Centres on my own server hardware and I run a TrueNAS server locally for stuff I need running on my LAN (mostly for security reasons).

That’s fine for normal server stuff but what about AI? Do I use AI? The answer is yes and I use the same principles for AI as I do for the rest of my tech stack with one exception (Claude) which is used for code and security reviews (never used for client work, only internal development) - other than that my AI stack is also self hosted and it runs 24/7 and is managed entirely via Gitea (a self hosted alternative to Github).

I have an (almost) fully self-hosted, local-AI research-and-content pipeline where gitea is the operating system: Gitea issues are state, labels are the program counter, webhooks are the event bus - and every bit of actual intelligence (other than code reviews/security reviews) runs on my own Apple Silicon.

The layers (bottom to top)

1. Hardware (office)

  • Mac Studio (M1 Ultra 128GB Apple Silicon), the GPU workhorse - this runs heavy weights and high context models such as Qwen3.6-27B MLX 8-bit (dense) and Qwen3-Coder-Next MLX 8-bit as well as Flux.2 dev and ComfyUI MLX models/nodes;
  • Mac mini (M1 16GB Apple Silicon), headless and used for embedding models;
  • MacBook Pro (M4 Pro 48GB Apple Silicon) - this is used for smaller context jobs and is very capable (Qwen3.6-27B MLX 8-Bit).
  • TrueNAS box - storage + Docker host for the always-on services.
  • Storage - 8TB NVME RAID0 (2x4TB NVME Drives) for fast model loading and 32TB RAID5 Array for everything else

2. Serving models locally

  • omlx - an MLX LLM server, in simple terms it is optimised to use Apple Silicon’s MLX system for AI models
  • shim - a thin JSON adapter in front of omlx that patches some tool calling issues with Qwen
  • ComfyUI - diffusion: FLUX.2 for images, LTX for video.

3. Gitea (the clever bit)

Self-hosted and authoritative this is basically the orchestrator. It isn't just where code lives - it's the state machine for the whole system:

  • Issues = durable state. Each unit of AI work is an issue.
  • Labels = the current step. A scoped label like Research/Run or Idea/Restate says where the work is in its lifecycle.
  • Webhooks = the event bus. Every label change, assignment, or repo creation fires an HTTP event.
  • Nested JSON = the program. A workflow file describes the steps; the dispatcher flattens it into those scoped labels.

It's self-propagating: finishing a step re-labels the issue > fires a webhook > runs the next step. No external queue, no separate database - git is the source of truth.

4. The orchestration engine

A small Python (stdlib-only) HTTP server that receives Gitea webhooks and routes them to targets:

  • qwen / claude - headless coding/reasoning agents (run in -p mode)
  • vane - the research engine
  • mattermost - a human-approval gate
  • omlx / comfyui / inference - GPU executors
  • scheduler / script - the control plane (deterministic, no LLM)

It's multi-host: a target the local instance owns runs locally; anything else is proxied to a peer. Triggers route on issue assignment, and auto-open an "Idea" issue when a repo is created (built from its description) so a new project bootstraps its own pipeline.

5. The GPU scheduler

A job queue built entirely out of Gitea issues + webhooks + scripts:

  • Each job is an issue whose body is a JSON spec; the issue's mere existence = "queued."
  • One GPU job at a time (the Studio's GPU/RAM is shared between LLM and diffusion).
  • Every control is a label or field: Do Next (single-use queue-jump), off-peak windows (22:00–09:00), issue due-dates as deadlines, dependency links, a pause switch, multiple endpoints, and automatic fan ramp while a job runs (to keeps machine cool when running inference).

6. The agents

A core principle: an inference server like omlx can generate but can't touch the filesystem or commit code. So the dispatcher fires qwen -p / claude -p - headless agent processes - to do the real work: write files, run tests, push to git. The shape is "enqueue returns 200 immediately; the agent acts on that response." Claude even has its own Gitea account for attributed actions.

Qwen-Code (qwen - p) is running in it’s own docker container and writes all code Claude Code (claude -p) runs on the Mac Studio where the dispatcher runs

I also have several PydanticAI agents (27 in total) which do various tasks (basically an AI workforce).

The design principles (the spine)

  1. Gitea is the OS - issues/labels/webhooks replace a bespoke workflow engine and database.
  2. Sovereign by default (mostly) - all inference (apart from code reviews and security reviews) is local MLX on Apple Silicon; no third-party APIs;
  3. Separation of thinking and acting - models generate, agents execute, the dispatcher routes.
  4. Deterministic control, probabilistic work - the queue/scheduler is plain scripts + webhooks (no LLM in the control loop > no runaway agent loops); only the actual work uses models.
  5. Humans hold the gate - everything is ultimately human gated - nothing gets pushed to production without human review.
  6. Everything is git - code, state, the job queue, and the research output all live in and are versioned by git.

Final Comments

What do I use it for? Well I mostly use it for research and development and the reason I set it up this way is I have limited options when it comes to hardware - I cannot use third party solutions for client work due to the highly sensitive nature of the work I do, so I had to use what I had to hand and I just happened to have all this hardware to hand when the AI shift started so I figured I may as well use it.

Is it perfect? No - I am literally climbing the walls wishing I had an M3 Mac Studio Ultra with 512GB unified memory so I can run bigger self hosted models and have bigger context windows - but availability is literally non-existent at the moment so I have to make do with what I have.

But it is manageable - so long as you normalise your tasks first (break them down into the smallest parts) so the LLM doesn’t need a massive context window and doesn’t suffer from context drift, these self hosted models are perfectly capable.

It is also not as fast as using frontier models (obviously) so normalising the tasks makes them run faster as well - but this is primarily why I added the Gitea queue management system so that everything is prioritised appropriately and the machines are running 24/7 (so inference continues when I sleep). These are also incredibly low power devices, so much more friendly on your energy bills and the environment.

Obviously this is not for everyone, it takes a lot of time to setup and you need to be technically competent to manage and secure these systems - but for geeks like me, it is doable and functional. I have spent 20 years shouting about 3rd party risk and self hosting rather than using public cloud - so it is only natural that I would want my AI stack to follow the same principles and it (mostly) does.

I also fall back to Runpod Serverless GPU instances sometimes if I desperately need a large model for something - which follows the same principles because they are zero knowledge and zero logs - I basically setup an inference engine and rent the GPUs from Runpod for the inference - it is accessed over a TLS enabled api endpoint, spins up the container, runs the inference, sends back the results and destroys the container - the Runpod billing is per second of GPU use so it is quite economical and allows me to maintain control and confidentiality in circumstances where I just don’t have enough local juice.

All of this works - you don’t need to push everything to a third party that is going to use that data for whatever purposes it chooses (or a Judge chooses for them) - I built an entire website compliance management stack using Qwen2 models - a platform capable of saving millions of euros a year by automating compliance monitoring, audits and reviews - it is what gave me the idea for WebSentinel in the first place.

Now, if only my friends at Apple would gift me an M5 Ultra with 1TB of Unified Memory - that would be a very nice addition to my current stack :)