Local LLM Setup for Abrest Translation & Matching

This guide covers setting up a local LLM server for the abrest import translation and embedding-based item matching features.

Hardware Requirements

Component	Minimum	Recommended
RAM	8 GB	16+ GB
CPU	4 cores	8+ cores
GPU	Not required	NVIDIA with 8+ GB VRAM
Disk	10 GB free	20+ GB free

CPU-only inference works for both embedding and text generation. A GPU significantly accelerates text generation (translation and match confirmation).

Installing Ollama

Ollama is the recommended LLM server. It exposes an OpenAI-compatible API and manages model downloads automatically.

# Linux
curl -fsSL https://ollama.com/install.sh | sh

# Verify installation
ollama --version

Ollama runs as a systemd service on port 11434 by default.

Pulling Required Models

Abrest uses two models: a text generation model for translation and match confirmation, and an embedding model for vector similarity search.

# Text generation model (translation + match confirmation)
ollama pull qwen3:8b

# Embedding model (vector similarity)
ollama pull nomic-embed-text

Other compatible models:

Purpose	Model	Size	Notes
Translation	`qwen3:8b`	~5 GB	Good multilingual support
Translation	`gemma3:12b`	~8 GB	Strong multilingual, needs more RAM
Translation	`llama3.1:8b`	~5 GB	Good general purpose
Embedding	`nomic-embed-text`	~275 MB	768 dimensions, multilingual
Embedding	`mxbai-embed-large`	~670 MB	1024 dimensions
Embedding	`all-minilm`	~45 MB	384 dimensions, fastest

Verifying the API

Test text generation:

curl -s http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3:8b",
    "messages": [
      {"role": "system", "content": "Translate the product name from uk to en. Return only the translation."},
      {"role": "user", "content": "Молоко пастеризоване 2.5%"}
    ],
    "temperature": 0.1
  }'

Expected response contains "content": "Pasteurized milk 2.5%" (or similar).

Test embeddings:

curl -s http://localhost:11434/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{
    "model": "nomic-embed-text",
    "input": "Laptop HP ProBook 450"
  }'

Expected response contains a "data" array with an "embedding" array of floats.

Abrest Configuration

Add or update the translation: section in abrest.yaml:

translation:
  enabled: true
  api_url: 'http://localhost:11434/v1'
  model: 'qwen3:8b'
  embedding_model: 'nomic-embed-text'
  match_threshold: 0.80
  timeout: 30

Key	Default	Description
`enabled`	`false`	Set to `true` to enable translation and embedding features
`provider`	`openai_compatible`	API dialect of the generation endpoint
`api_url`	`http://localhost:11434/v1`	Base URL of the OpenAI-compatible API
`api_key`	(empty)	Sent as a bearer token; leave empty for a local Ollama
`model`	`qwen3:8b`	Model name for translation and match confirmation
`embedding_provider`	(same as `provider`)	API dialect of the embedding endpoint
`embedding_api_url`	(same as `api_url`)	Set only when embeddings are served separately
`embedding_api_key`	(same as `api_key`)	Credential for a separate embedding endpoint
`embedding_model`	`nomic-embed-text`	Model name for computing vector embeddings
`match_threshold`	`0.80`	Cosine similarity threshold for embedding match candidates (0.0-1.0)
`timeout`	`30`	HTTP request timeout in seconds

The four embedding_* fallbacks mean a single local Ollama needs no extra keys: set them only when generation and embeddings live on different servers.

Restart abrest after changing the configuration.

Initial Embedding Population

After enabling translation for the first time, existing items in R_TOVAR need their embeddings computed. This is a one-time operation:

curl -s -X POST http://localhost:4603/api/tools/embeddings/rebuild \
  -H "X-API-KEY: your-api-key"

Response:

{"processed": 12345, "failed": 2}

This runs synchronously and may take several minutes for large catalogs (~15ms per item with nomic-embed-text). New items imported after this point get their embeddings computed automatically during import.

Troubleshooting

"Translation is not enabled" Check that enabled: true is set in the translation: section of abrest.yaml and restart abrest.

Translation returns empty strings - Check that Ollama is running: systemctl status ollama - Verify the model is pulled: ollama list - Check abrest logs for "LLM API call failed" warnings - Test the API manually with curl (see above)

Embeddings rebuild shows many failures - Items with empty names are skipped (expected) - Check Ollama memory usage — the embedding model needs to be loaded - Increase timeout if the server is slow to respond

Import is slow with translation enabled - Each item requires 1-2 LLM calls (embedding + potential match confirmation) - Consider a GPU for text generation, or a smaller/faster model - Embedding computation is fast (~15ms/item) even on CPU - Items already translated or matched are skipped (idempotent)

Changing the embedding model When switching to a different embedding model, existing embeddings become incompatible. Run the rebuild endpoint to recompute all embeddings with the new model. The old embeddings are overwritten automatically.