Technology Artificial-intelligence
8 min read

Autonomous coding agent with open-weight models - Foundation

First article in a series on building an autonomous coding agent with open-weight models on Jenkins with a GPU-enabled node.

This is the first article in a series on building an autonomous coding agent with open-weight models. This one lays the foundation — the next article will cover code examples and a deeper dive into the configuration needed.

Introduction

In recent times, the gap of intelligence between open-weight models and closed source frontier models is becoming very thin. Open-weight models such as GLM-5.3 and Qwen3.8 are performing with efficiencies nearing frontier models like Claude Opus 5 or GPT-5.6. While the edge offered by frontier models in terms of performance and complex multi-step reasoning cannot be negated, open-weight models carry their own advantages.

The Operational Expense

The cost of running an open-weight model is much lower compared to frontier models. Instead of paying per token or per request inference, we only pay for the infrastructure cost that hosts the open weight model. This allows us to tailor the infrastructure to our workload and pay for what we want. The savings by hosting an open-weight model in our own infrastructure can be in the order of 10x.

Data Privacy and Control

Self-hosted open weights ensure total data sovereignty and compliance, completely removing third-party data exposure risks associated with external APIs. The infrastructure that hosts the model can be hardened to avoid any potential data leakage to the cloud.

Customization

Open-weight models provide access to the trained hyperparameters and, obviously, the weights. This available data allows further customization of the model to specific needs.

But are they any good?

For this section of the article, I tried to grab a lot of information around benchmarks of open weight models compared against frontier models. I did find some content which I wanted to present here. But instead of doing that, I am sharing the summary of what I found instead of the other links themselves.

On general capability, Epoch AI’s Capabilities Index puts the best open-weight models about four months and 8-9 points behind the best closed models — roughly the same jump as between two point releases like GPT-5 and GPT-5.5. It’s a real gap, but not a different league.

That gap narrows a lot once you look at coding specifically instead of general reasoning. On Epoch’s software-engineering-focused index, models like GLM-5.3 and Kimi K3 land within 2-3 points of Claude Opus 5 and GPT-6 Astra. Artificial Analysis’s Coding Agent Index, which scores full agent harnesses like Claude Code end-to-end rather than just the raw model, tells a similar story — GLM-5.3 and Kimi K3 sit close behind the leaders, and even a smaller, cheaper model like Qwen3.8 turns in a usable score running the exact same Claude Code harness that tops the chart.

None of this makes open-weight models the obvious first choice for every task. But it’s a large enough share of frontier capability, at a small enough fraction of the cost, to earn a real shot at something like an autonomous coding agent — especially once you stack it against the operational expense, privacy, and customization wins from earlier in this article.

Where frontier models still pull ahead

The scores above don’t tell the whole story. A few gaps show up directly in how these benchmarks are built and run, and they’re worth calling out honestly rather than glossing over:

  • Context window. Frontier closed models generally ship larger context windows, and just as importantly, hold quality better across the full window instead of degrading as it fills up. An agent reading through a large repo across many tool calls leans on this a lot.
  • Long-horizon agentic coherence. Staying on-task across dozens of tool calls without drifting, looping, or losing earlier context is only partially captured by benchmarks like Terminal-Bench — real agent runs often go longer than any benchmark task does.
  • Tool-calling reliability. Malformed function-call arguments and similar failures are a weaker spot for a lot of open releases, since closed frontier labs iterate heavily on this as core infrastructure for their own agent products.
  • Benchmark contamination. Worth flagging since it comes from Epoch’s own write-up on the ECI gap: “open-weight models tend to perform worse on private benchmarks compared to closed models, plausibly because they more aggressively optimize for public benchmarks.” The real gap could be a bit wider than the public leaderboards suggest.

None of these are disqualifiers for the setup this article is actually about. This isn’t an open-ended agent freely roaming a codebase for hours — it’s triggered by Jenkins for a scoped job, working against one repo and one task at a time, with a constrained toolset and a pipeline that reviews its output before anything merges. A smaller context window matters a lot less when the task itself is bounded. Long-horizon drift matters a lot less when the agent isn’t expected to stay coherent for hours unsupervised. Tool-calling hiccups matter a lot less when the harness validates and retries rather than trusting blindly. And the contamination caveat, if anything, just reinforces the framing from earlier — treat the benchmark numbers as directional, not gospel, and let the operational, privacy, and customization wins carry the rest of the argument.

The Agent Setup

Now that we have established that open-weight models can be given an opportunity to handle moderately complex coding tasks, let’s look at what it actually takes to implement and run the autonomous agent. In the subtitle, I already mentioned that Jenkins would be used as an orchestration engine. There is no hard rule to go only with Jenkins for this approach. But certain features of Jenkins make the orchestration easier to manage.

The Jenkins Node Agent — Jenkins itself doesn’t run the job on its controller — it delegates. When a pipeline asks for an agent matching a given label, Jenkins hands that request off to a cloud provisioner, which spins up a node to match. Once that node registers back with the controller, Jenkins schedules a pod onto it. When the job finishes, the pod and the underlying node are torn down.

Our requirement is to run an LLM model in the cloud. So, this node has to be a GPU Node. The size of the instance and the amount of GPU that is required depend on the model.

With this fundamental building block in place, let’s take a look at the core components and their role in the setup.

Component Purpose Example Tool/Framework
Inference Server Inference server hosts the Large Language Model. Ollama, vLLM, etc. We will use vLLM going forward as an example in this article
Coding Harness Coding harness is a piece of code that interacts with the LLM and makes changes to the code base Claude Code, opencode, pi.dev etc. We will use opencode as an example going forward
Orchestrator Script Handles git operations, invokes the coding harness, and opens the pull request. Can be any scripting language/technology

Let’s look at how these components stack up when deployed.

flowchart TB
    Controller["Jenkins Controller\n(Jenkinsfile)"]

    subgraph NodeAgent["Jenkins Node Agent (GPU Instance)"]
        subgraph InferencePod["Inference Server Pod"]
            Inference["vLLM"]
        end

        subgraph CodingPod["Coding Agent Pod"]
            Orchestrator["Orchestrator Script"]
            Harness["Coding Harness\n(opencode)"]
            LocalRepo[("Local Git Clone")]
        end
    end

    Repo[("Git Repository")]

    Controller --> NodeAgent
    Repo -->|"clone"| LocalRepo
    Orchestrator --> |"create"| LocalRepo
    Orchestrator --> |"invoke"| Harness
    Harness --> |"work upon"| LocalRepo
    Harness --> |"call"| Inference
    LocalRepo -->|"push"| Repo

The Coding Agent pod consists of the orchestrator script and the coding harness bundled into it. The workload on this pod is primarily CPU intensive. It does not use any GPU at all. The inference server, on the other hand, hosts the actual LLM model and is a completely GPU-bound workload. Both the pods can remain on the same node as most of the GPU nodes also carry CPU capacity.

The Coding Workflow

As already called out before, Jenkins is our primary orchestrator. Having said that, some orchestration such as cloning the repository, creating the branch, etc. is still carried out by the orchestrator script. When everything is put together, the following are the steps of execution.

Pipeline is triggered with a prompt, repository to work upon, and branch name

  1. Jenkins controller requests a node agent matching the GPU label defined in the Jenkinsfile; the cloud provisioner spins up a matching GPU node.
  2. As the node comes up, its own startup script starts the Inference Server pod. vLLM pulls the model from cloud storage and starts serving it.
  3. Jenkins controller schedules the Coding Agent pod onto the same node as the actual build agent.
  4. Orchestrator script in the coding agent pod clones the remote repository to the local file system.
  5. Orchestrator script invokes the coding harness with the prompt provided and waits for it to complete.
  6. Coding harness iteratively works upon the code, calling the LLM hosted on vLLM as needed.
  7. Once the harness finishes, the orchestrator script creates the branch from the agent’s commits and pushes it to the remote repository.
  8. The orchestrator script opens a pull request.

Here’s the same sequence visually:

sequenceDiagram
    participant Controller as Jenkins Controller
    participant Inference as Inference Server (vLLM)
    participant Orchestrator as Orchestrator Script
    participant Harness as Coding Harness
    participant Repo as Git Repository

    Note over Controller: Triggered with a prompt, repository, and branch name
    Controller->>Inference: Start pod, pull and serve the model
    Controller->>Orchestrator: Schedule the coding agent pod
    Orchestrator->>Repo: Clone repository
    Orchestrator->>Harness: Invoke with prompt
    loop Until the task is complete
        Harness->>Inference: Send prompt
        Inference-->>Harness: Return completion
        Harness->>Harness: Edit local files
    end
    Harness-->>Orchestrator: Signal completion
    Orchestrator->>Orchestrator: Create branch from the commits
    Orchestrator->>Repo: Push branch
    Orchestrator->>Repo: Open pull request

Conclusion

This article laid the foundation: a case for why open-weight models are good enough to trust with a scoped, self-hosted coding agent, and the Jenkins-based architecture — controller, GPU node agent, inference server, and orchestrator script — that ties it together. It’s still a diagram and a workflow on paper.

The next article in this series will get into the actual code — the Jenkinsfile, the pod definitions, the orchestrator script itself — along with a deeper dive into the configuration each piece needs to work together.

Continue reading: Autonomous coding agent with open-weight models - Setup