Troubleshooting Podman Pod Port Publishing: Why Jenkins Wasn’t Accessible from Host IP

When deploying Jenkins in a Podman pod using Ansible, I encountered an issue: the Jenkins container was accessible on its internal pod network IP but completely unreachable from the host’s external IP. Here’s how I diagnosed and fixed the problem.

The Problem

I had set up a Jenkins pod with the following configuration:

podman_pods:
  - name: jenkins-pod
    ports:
      - "8080:8080"
      - "50000:50000"
    networks:
      - frontend-net

The pod was created successfully, and Jenkins was running. However:

• ✅ Working: curl http://10.89.1.33:8080 (pod internal IP)
• ✅ Working: curl http://10.89.1.33:50000 (pod internal IP)
• ❌ Failing: curl http://192.168.1.5:8080 (host IP)
• ❌ Failing: curl http://192.168.1.5:50000 (host IP)

Even after disabling the firewall with sudo ufw disable, the ports remained inaccessible from the host IP.

The Diagnostic Journey

Step 1: Verify the Service is Running

First, I confirmed Jenkins was actually running and responding:

sudo curl -i http://10.89.1.33:8080
# HTTP/1.1 403 Forbidden (Jenkins login page - working!)

sudo curl -i http://10.89.1.33:50000
# HTTP/1.0 200 OK (Jenkins agent port - working!)

The service was fine, so the issue was clearly with port publishing.

Step 2: Check Pod Configuration

I inspected the pod to verify port bindings were configured:

sudo podman pod inspect jenkins-pod | jq '.InfraConfig.PortBindings'

Output showed the ports were properly configured on the pod:

{
  "50000/tcp": [{"HostIp": "", "HostPort": "50000"}],
  "8080/tcp": [{"HostIp": "", "HostPort": "8080"}]
}

Step 3: Inspect the Container – Finding the Smoking Gun

This is where I found the problem:

sudo podman inspect jenkins | jq '.Config.CreateCommand'

The output revealed:

[
  "/usr/bin/podman",
  "container",
  "run",
  "--name", "jenkins",
  "--network", "frontend-net",  // ⚠️ Problem!
  "--pod", "jenkins-pod",       // ⚠️ Conflict!
  "--user", "root",
  "--volume", "/opt/jenkins-data:/var/jenkins_home",
  "--detach=True",
  "docker.io/jenkins/jenkins:lts-jdk21"
]

The Issue: The container was specifying both --network frontend-net AND --pod jenkins-pod. This creates a networking conflict!

Further inspection confirmed:

sudo podman inspect jenkins | jq '.HostConfig.PortBindings'
# Output: {} (empty!)

The container had no port bindings, even though it was in a pod with port bindings.

Step 4: Understanding the Root Cause

When containers join a pod in Podman:

1. Pods manage the network namespace – all containers in a pod share the same network
2. Port publishing happens at the pod level – not at the container level
3. Containers should NOT specify their own network – they inherit the pod’s network automatically

By specifying both --network and --pod, the container was creating a conflicting network configuration that prevented proper port forwarding from the host to the pod.

The Solution

Fix the Ansible Playbook

The issue was in my container creation task. I needed to remove the network specification when creating containers that belong to pods:

Before (Broken):

- name: Create containers
  containers.podman.podman_container:
    name: "{{ item.name }}"
    image: "{{ item.image }}"
    state: started
    pod: "{{ item.pod | default(omit) }}"
    network: "{{ item.networks | default(omit) }}"  # ❌ Remove this!
    volumes: "{{ item.volumes | default([]) }}"
    user: "{{ item.user | default(omit) }}"
  loop: "{{ podman_containers }}"

After (Fixed):

- name: Create containers
  containers.podman.podman_container:
    name: "{{ item.name }}"
    image: "{{ item.image }}"
    state: started
    pod: "{{ item.pod | default(omit) }}"
    # Network is inherited from pod - don't specify it here!
    volumes: "{{ item.volumes | default([]) }}"
    user: "{{ item.user | default(omit) }}"
  loop: "{{ podman_containers }}"

Recreate the Container

# Remove the old container and pod
sudo podman stop jenkins
sudo podman rm jenkins
sudo podman pod stop jenkins-pod
sudo podman pod rm jenkins-pod

# Recreate using Ansible with the fixed playbook
# Or manually:
sudo podman pod create \
  --name jenkins-pod \
  --network frontend-net \
  --publish 8080:8080 \
  --publish 50000:50000

sudo podman run -d \
  --name jenkins \
  --pod jenkins-pod \
  --user root \
  --volume /opt/jenkins-data:/var/jenkins_home \
  docker.io/jenkins/jenkins:lts-jdk21

Verify the Fix

After recreating without the --network flag, the container configuration looked correct:

sudo podman inspect jenkins | jq '.HostConfig.NetworkMode'
# "NetworkMode": "container:549805b8a87f..."  ✅ Using pod's network!

sudo podman inspect jenkins | jq '.NetworkSettings.Ports'

Output:

{
  "50000/tcp": [{"HostIp": "", "HostPort": "50000"}],
  "8080/tcp": [{"HostIp": "", "HostPort": "8080"}]
}

Now the ports are properly published! Testing from the host:

curl -I http://192.168.1.5:8080
# HTTP/1.1 403 Forbidden (Jenkins login - working!)

curl -I http://192.168.1.5:50000
# HTTP/1.0 200 OK (Jenkins agent port - working!)

Troubleshooting Checklist for Future Issues

When debugging Podman pod networking issues, use these commands:

# 1. Check what's listening on host ports
sudo ss -tlnp | grep -E ':(8080|50000)'

# 2. Verify pod port bindings
sudo podman pod inspect <pod-name> | jq '.InfraConfig.PortBindings'

# 3. Check container port bindings (should be empty for pod containers)
sudo podman inspect <container-name> | jq '.HostConfig.PortBindings'

# 4. Check container network mode
sudo podman inspect <container-name> | jq '.HostConfig.NetworkMode'

# 5. Look for the conflict in creation command
sudo podman inspect <container-name> | jq '.Config.CreateCommand'

# 6. Verify network namespace sharing
sudo podman inspect <container-name> | jq '.NetworkSettings.SandboxKey'

# 7. Test from Pod infra container using network namespace
INFRA_PID=$(sudo podman inspect -f '{{.State.Pid}}' f08c08e55b4c-infra)
sudo nsenter -n -t $INFRA_PID curl -v http://10.89.1.33:8080

Key Takeaways

1. Never specify --network when using --pod – containers in pods inherit the pod’s network automatically
2. Port publishing is a pod-level operation – configure ports on the pod, not individual containers
3. Check the CreateCommand – it reveals exactly how the container was started and can expose configuration conflicts
4. Compare PortBindings at both levels – empty container PortBindings with non-empty pod PortBindings indicates the issue
5. Network namespace sharing is key – look for NetworkMode: "container:<infra-id>" to confirm proper pod networking

Conclusion

This issue taught me an important lesson about Podman’s pod networking model. Unlike standalone containers where you specify network and ports at the container level, pods handle networking centrally through their infrastructure container. When you add a container to a pod, it automatically joins the pod’s network namespace, and any attempt to override this with explicit network configuration creates conflicts.

By removing the network specification from my Ansible container creation task, the ports published on the pod level now properly forward from the host IP to the Jenkins container, making it accessible from anywhere on my network.