Xur

Put local models to work.

Select a model, assign its GPUs and connect a client once the engine is ready.

Choose a workload

Add a model workload to a profile and select the appropriate engine and model recipe. The editor offers llama.cpp, vLLM and vLLM-Omni selections. Check the model's VRAM needs and compatibility; being listed by a model publisher does not guarantee it works with every engine or GPU.

Use Settings to save an HF token when a repository requires one. Obtain any necessary model access from its publisher. The first load can take longer while weights and engine images download and warm up.

Engine updates and recovery

Every model start checks the latest upstream engine image. Running engines keep their current image until restarted; selected model revisions and launch settings stay unchanged. If a download fails, Xur can use a compatible locally cached engine. Models in the loaded profile recover after an exit or reboot, while intentionally unloaded models stay stopped. See the model lifecycle and compatibility guide.

Qwen MTP

The supported Qwen MTP recipe includes its concurrency and speculative-decoding settings. Loading or resuming an older saved recipe repairs the known missing settings automatically.

Qwen inference has passed live smoke tests on RTX 3090 hardware; this does not guarantee every model or configuration. See the dated results and test limits.

Connect to an active language model

  1. Wait for the profile to report the model ready.
  2. Open LLM endpoints and copy its Base URL.
  3. Create a Testing key under Settings → API keys. Use it as your client's API key.
  4. Open the model list to find the exact served model ID, then select that ID in your OpenAI-compatible client.

Use the Tailscale HTTPS address for clients on another network. Keep the API key private; revoke it when it is no longer needed.

A running container alone does not mean a model is ready. If the endpoints page is empty, check the profile operation and workload logs.

Try a conversation or benchmark

Model lab provides basic test chat and saved benchmarks. Results retain model identity, the running workloads and sampled VRAM alongside timing and response results. Compare runs under similar workload and prompt conditions.

Model lab showing the model selector, benchmark settings, saved runs and Test chat tab.
Select a ready model, then choose Benchmarks or Test chat. No measured results are shown. Real interface with synthetic example data; select to enlarge.