Using LLM tools in Emacs

In order to fulfil the requirements, the container running an LLM backend must be isolated from most of the host file system, it cannot have access to the external network and any changes within it should be harmless. A way to do it is:

• ECA stuff must be mounted into the container read-only, with the exception of the cache.
• Source code directories can be mounted read-write assuming they are under git control. .git directory must be mounted read-only in order to protect the version control data from unwanted changes and to be able to revert file changes easily.
• Each container is a fresh scratch container started from a common image.
• The container network must be a Podman internal network. This way, it cannot access the host or any external network.

If you don’t trust Podman container isolation for the purpose, you can choose other solutions, like virtual machines or combining Podman and virtual machines, e.g. with libkrun (podman run --target=krun). The same principles apply, the implementation will be different.

I use three containers running:

• ECA, a separate container instance for each ECA session
• LiteLLM proxy, a common container with a static IP address
• network proxies, a common container with a static IP address

The containers share a Podman internal network:

podman network create --internal --disable-dns llm

The container image should ideally contain all the system and development tools needed. The agent can be instructed to install whatever it needs itself but this is just a waste of time and tokens.

The container can be run as:

podman run --rm \
       --network=llm \
       --add-host=proxy:PROXY-IP \
       --add-host=litellm:LITELLM-IP \
       -e http_proxy=http://proxy:8080 \
       -e HTTP_PROXY=http://proxy:8080 \
       -e HTTPS_PROXY=http://proxy:8080 \
       -e LITELLM_API_KEY \
       -v $HOME/.emacs.d/eca:/home/pdm/.emacs.d/eca:z,ro \
       -v $HOME/.config/eca:/root/.config/eca:z,ro \
       -v $HOME/.cache/eca:/root/.cache/eca:z \
       $*

where PROXY-IP and LITELLM-IP are the static addresses of the corresponding containers. To get an idea what addresses can be used, run a Podman container with --network=llm, run podman inspect on it, look for Networks entry in the output and llm network to see the network IP address and range.

Further arguments are provided from the Elisp side:

(defun my-eca-wrapper (command roots)
  (append (list "SCRIPT-FILE-STARTING-THE-CONTAINER-LIKE-ABOVE"
                (if roots
                    (file-name-nondirectory (directory-file-name (car roots)))
                  "default")
                "-i")
          (cl-loop for r in roots
                   for dir = (expand-file-name r)
                   for gitdir = (format "%s.git" dir)
                   append (list "-v" (format "%s:%s:z" dir dir))
                   when (file-directory-p gitdir)
                   append (list "-v" (format "%s:%s:z,ro" gitdir gitdir)))
          (list "CONTAINER-IMAGE-NAME")
          command))
(setq eca-process-wrapper-function 'my-eca-wrapper)
(setq eca-send-process-id nil)

This ensures file system isolation:

• ECA stuff is mounted under /root, read-only (with the exception of the cache).
• The source directories are mounted read-write, to the same location as on the host to avoid path confusions.
• .git subdirectories are mounted read-only.

The primary danger are ECA tools. If there is a new tool introduced or installed then it may make a new way to access something that shouldn’t be accessible. For this reason, the common default tool access policy should never be allow. Let’s be clear in the ECA configuration file:

"toolCall": {
    "approval": {
        "byDefault": "ask"
    }
},

ECA sets allow permission on some read-only commands by default, but even read-only commands shouldn’t be able to look anywhere uncontrollably.

I use rootless Podman containers. ECA is running under root user there. It’s possible to create an ordinary user in the container, let’s say agent and to run shell commands under this user:

runuser -u agent -- env -u LITELLM_API_KEY bash -c 'properly escaped commands'

This can be set up in the ECA configuration as follows:

"hooks": {
    "sandbox": {
        "type": "preToolCall",
        "visible": false,
        "matcher": "eca__shell_command",
        "actions": [{
            "type": "shell",
            "file": "/root/.config/eca/my-sandox-command.sh"
        }]
    }
},

where my-sandbox-command.sh can be something like

#!/bin/sh

command=$(jq -r .tool_input.command)
wrapped="runuser -u agent -- env -u LITELLM_API_KEY bash -c $(escape $command as best as you can)"
jq -n --arg cmd "$wrapped" '{"updatedInput": {"command": "$cmd"}}'

Then the shell commands don’t have access to files private to ECA and hopefully also not to the ECA process memory (I’m not sure about memory isolation in rootless containers).

There are serious caveats though:

• If the hook fails or doesn’t work correctly, ECA executes the shell command without being wrapped, i.e. with all the permissions.
• If the hook succeeds, ECA prints a message informing about modification of on each shell command execution. This cannot be disabled and is somewhat annoying.
• Escaping the shell commands is tricky; consider commands like ls /home & ls /root. I think I’ve solved it, but I’m not sure enough about my solution to present it here. Do your own research how to escape properly.
• If the source directory on the host is not accessible to all the users, it won’t be accessible to the shell commands.
• git may not like the fact that the source directory is owned by somebody else. git safe.directory option should be set for agent user to disable the protection.
• The worst is that the LLM may grab runuser seen in the previous commands and run it itself, resulting in double runuser invocations and the LLM getting crazy to look for a working command version.

These mean there are no real security guarantees and there are significant inconveniences. Whether running commands under a different users or not, the basic rule is not to put anything sensitive into the container and to be careful about mounting read-write directories into the container. And it’s hard to protect the LiteLLM API key and the ECA cache without the shell command wrapper.

git ECA tool is basically a specialised shell command runner and .git is read-only anyway. It’s best to disable it in the ECA configuration:

"disabledTools": ["eca__git"],

To protect the LLM credentials, using a LLM proxy is advisable. (On the other hand, adding another component increases risks such as supply chain attacks.) I use LiteLLM, for no particular reason, it’s a free software proxy supported by the tools I use.

The LiteLLM can be run in a separate container, then the actual LLM API credentials are protected from the agent. The agent can steal at most the credentials to the LiteLLM proxy, which are not that useful.

ECA configuration for a LiteLLM proxy can look like this:

"providers": {
    "litellm": {
        "api": "openai-responses",
        "url": "http://litellm:PORT",
        "models": {
            "model-1": {},
            "model-2": {},
            "model-3": {},
            …
        }
    }
},
"defaultModel": "litellm/model-2",

LiteLLM can be run from the LiteLLM image:

podman run -dit --rm --name=litellm \
       --network=llm:ip=LITELLM-IP --network=podman \
       -e LITELLM_MASTER_KEY \
       -e YOUR_LLM_PROVIDER_API_KEY \
       -v …/config.yaml:/config.yaml:z,ro \
       docker.litellm.ai/berriai/litellm:main-latest \
       --port PORT --config /config.yaml

LITELLM_MASTER_KEY contains the password that can be used as LITELLM_API_KEY to access the proxy if no separate LiteLLM API key is set up. The key must start with sk- prefix.

config.yaml can specify just the available models:

model_list:
  - model_name: some-model
    litellm_params:
      model: some-provider/some-model
      api_key: os.environ/YOUR_LLM_PROVIDER_API_KEY
  - model_name: another-model
    litellm_params:
      model: some-provider/another-model
      api_key: os.environ/YOUR_LLM_PROVIDER_API_KEY
  …

The container should be fully isolated from the host and external networks. At the same time, there must be means to talk to the outside world, to be able to use the model (unless it’s local), MCPs and to access information on the web.

The ECA container uses only the llm internal network and is thus fully isolated. LiteLLM and the proxy containers share this network to be accessible to ECA, and also use podman network to access the outer world.

To connect to the outer world, mitmproxy HTTP proxy can be used. I haven’t explored it much yet, but the basic features are there:

• Displaying and logging network connections.
• Interactive approval of selected connections.
• Redirections.
• Etc.

The proxy container can be run as follows:

podman run \
       --add-host=host:host-gateway \
       --add-host=litellm:LITELLM-IP \
       --network=llm:ip=PROXY-IP --network=podman \
       -p 127.0.0.1:8081:8081 \
       -v $HOME/.mitmproxy:/root/.mitmproxy:z \
       …

Port 8081 serves as the mimtproxy web interface on the host. mitmweb can be started like this:

mitmweb --set web_host=0.0.0.0 --set web_password=STATIC-PASSWORD-IF-DESIRED

Point your web browser to http://127.0.0.1:8081 to access mitmproxy and configure it according to your requirements. Which network connections to allow and which not is a difficult question, I don’t have much idea about the possible answers yet.

The ECA container is already set up with HTTP proxy environment variables for the controlled internet access via mitmproxy. Note that ECA itself honours the environment variables to some extent but not completely. For example, they may not apply in MCP setup and there is no support for no_proxy, which means the LiteLLM traffic travels through the network proxy.

If anything else than HTTP is needed, ad-hoc socat forwarding should do the job. See below for an example MCP setup. For more complex setups or permissions, consult your firewall.