When running containerized applications with Podman or Docker, one of the most common challenges is establishing network connectivity from external machines to services running inside containers. This guide walks through the diagnostic process and solutions for firewall-related connectivity issues.
Understanding the Two-Layer Firewall Problem
When connecting from an external machine to a containerized service, you need to consider two potential firewall barriers:
- Host-level firewall: The firewall on the machine hosting the containers, which may block incoming traffic to ports mapped to container ports
- Container-level firewall: The firewall configuration within the container itself, which may block traffic to the service port
Both layers must be properly configured to allow traffic through.
Diagnostic Techniques
Testing Connectivity
The first step in troubleshooting is determining where the blockage occurs. From your client machine, use netcat to test connectivity:
nc -vz <host-ip> <port>
The response tells you exactly what’s happening:
- “Connection refused”: The host is reachable, but nothing is listening on that port. This typically indicates a service configuration issue or the port isn’t properly mapped.
- “Connection timed out”: Usually indicates a firewall is blocking the connection.
Verifying Port Listeners
On the container host, check if the port is actually being listened to:
sudo ss -tlnp | grep <port>
A successful configuration will show output similar to:
LISTEN 0 4096 0.0.0.0:61208 0.0.0.0:* users:(("conmon",pid=237164,fd=5))
If you see no output, the container may not be properly configured with port mapping.
Checking Firewall Logs
System logs can reveal if traffic is being blocked. Use journalctl to inspect firewall activity:
sudo journalctl -xef | grep UFW
Blocked traffic will appear with messages like:
[UFW BLOCK] IN=eth0 OUT=podman3 SRC=192.168.1.49 DST=10.89.2.3 PROTO=TCP DPT=61208
This clearly shows UFW (Uncomplicated Firewall) blocking traffic from the source to your container network.
Configuring UFW for Container Networks
Basic Host Firewall Rules
To allow traffic from a specific external IP to a port on your host:
sudo ufw allow proto tcp from <source-ip> to any port <port>
Example:
sudo ufw allow proto tcp from 192.168.1.49 to any port 61208
Routing Rules for Container Networks
For traffic that needs to be routed to container networks, use UFW routing rules:
sudo ufw route allow in on <external-interface> out on <container-network>
Example allowing HTTP traffic into containers:
sudo ufw route allow in on eth0 out on podman2 to any port 80 proto tcp
For bidirectional communication between your host interface and container network:
sudo ufw route allow in on eth0 out on podman2
sudo ufw route allow in on podman2 out on eth0
Managing Firewall Rules
To remove rules, use the delete command with the same syntax:
sudo ufw delete allow proto tcp from 192.168.1.49 to any port 8888
sudo ufw delete allow in on eth0 to any port 8888 proto tcp
After making changes, reload UFW:
sudo ufw reloadOr
sudo ufw disable
sudo ufw enable
Container Management Best Practices
Listing Running Containers
Use custom formatting to see essential container information, such as the display network name:
sudo podman ps --format "{{.ID}} {{.Names}} {{.Networks}}"
Inspecting Container Networks
To view detailed network configuration:
sudo podman network inspect <network-name>
Example:
sudo podman network inspect monitoring-net
Creating Systemd Services for Containers
For production environments, managing containers as systemd services ensures they start automatically and integrate with system management tools.
Generate Service Files
Podman can automatically generate systemd unit files:
sudo podman generate systemd --name <container-name> --files --new
Example for a PostgreSQL container:
sudo podman generate systemd --name postgres --files --new
Install and Enable the Service
Move the generated service file to the systemd directory and enable it:
sudo mv container-<name>.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now container-<name>
The --now flag both enables the service to start on boot and starts it immediately.
Verification Checklist
After configuration, verify your setup:
- Check UFW status:
sudo ufw statusshould show your allow rules - Verify port listeners:
sudo ss -tlnp | grep <port>should show conmon or your service - Test external connectivity:
nc -vz <host-ip> <port>from client machine should connect - Monitor logs:
sudo journalctl -xefshould show no UFW BLOCK messages for your traffic
Conclusion
Network connectivity issues with containers often stem from firewall configurations at the host or container level. By systematically diagnosing where traffic is being blocked and properly configuring UFW routing rules, you can ensure reliable connectivity to your containerized services.
The key is understanding that container networks are separate network namespaces that require explicit routing rules, not just simple port allowances. With the diagnostic tools and configuration patterns outlined above, you’ll be well-equipped to troubleshoot and resolve container networking challenges.