bifrost-prompt-rewriter

Bifrost Prompt Rewriter Plugin

A Bifrost PreLLMHook plugin that uses a small, fast local model (e.g. Qwen3-4B-Instruct) to rewrite and improve user prompts before they are sent to a larger model (e.g. Qwen3.6-35B or Qwen3-Coder-Next).

This is useful when you want better prompt quality without manually engineering every request.

How it works

1. Intercepts the outgoing chat request in PreLLMHook
2. Extracts the last user message
3. Sends it to a small model with a system prompt that asks it to rewrite the request into a clearer, more structured prompt
4. Replaces the original user message with the improved version
5. Lets the request continue to the target (larger) model

If the small model fails or times out, the plugin fails open and uses the original prompt.

Requirements

• Go (same major version as your Bifrost binary)
• Bifrost gateway running
• A local OpenAI-compatible server serving your small model
  (llama.cpp server, Ollama, Lemonade, vLLM, etc.)

Project Structure

prompt-rewriter/
├── main.go          # Plugin source
├── go.mod
├── Makefile
└── README.md

Build

# Download dependencies
make deps

# Build the shared object
make

# Or rebuild everything
make rebuild

This produces prompt-rewriter.so.

Important: Build the plugin on the same OS + architecture as the Bifrost binary (Linux/macOS, amd64 or arm64). Cross-compilation is not supported for Go plugins.

Bifrost Configuration

Add the plugin to your Bifrost config.json:

{
  "plugins": [
    {
      "name": "prompt-rewriter",
      "enabled": true,
      "path": "/absolute/path/to/prompt-rewriter/prompt-rewriter.so",
      "config": {
        "small_model_endpoint": "http://localhost:8081/v1/chat/completions",
        "small_model_name": "qwen3-4b-instruct",
        "enable_logging": true,
        "timeout_seconds": 12
      }
    }
  ]
}

Config Options

Key	Type	Default	Description
small_model_endpoint	string	http://localhost:8081/v1/chat/completions	OpenAI-compatible endpoint for the small model
small_model_name	string	qwen3-4b-instruct	Model name sent to the small model server
rewrite_system_prompt	string	(see source)	System prompt that instructs the small model how to rewrite
enable_logging	bool	true	Log original + rewritten prompts
timeout_seconds	int	15	Timeout for the small model call

Example Flow

Original user message:

make a function that reverses a linked list

After rewrite (example):

Write a clean, well-documented function that reverses a singly linked list. 
Include the node definition if needed, handle edge cases (empty list, single node), 
and provide both iterative and recursive versions if practical. Use clear variable names.

The improved prompt is then sent to your larger model.

Notes & Tips

• The plugin only rewrites the last user message. Conversation history is left intact.
• Calling the small model directly (not through Bifrost) avoids recursion and keeps latency lower.
• Keep the small model fast (4B–7B class GGUF on GPU/CPU is ideal).
• You can customize the rewrite system prompt for coding, reasoning, or general use.
• If you want the rewrite to only happen for certain target models, you can add a simple check on req.ChatRequest.Model inside PreLLMHook.

License

Use and modify freely.