"The service can't reach the partner API" is one of the most common tickets an on-call engineer gets, and one of the easiest to waste an afternoon on. The usual pattern is to jump straight to the most familiar suspect: restart the pod, blame the firewall team, or flush DNS. Sometimes that works. When it does not, you have learned nothing.

What works better is following the packet in the order it actually travels: resolve the name, pick a route, open a TCP connection, negotiate TLS, then speak HTTP. Each layer has one or two commands that give a definite answer, and each answer eliminates whole categories of causes. This post is the toolkit we use, with the output that tells you which layer is broken. Our running example is a worker on a Linux host that cannot call https://api.partner.example.com.

Start with the exact failure

Before running any diagnostic, get the precise error from the client, because it usually names the layer:

Error text Layer that failed
Could not resolve host, Name or service not known, NXDOMAIN DNS
Network is unreachable, No route to host Routing
Connection timed out during connect Packets dropped (firewall, security group, routing)
Connection refused Host reached, nothing listening or actively rejected
certificate verify failed, handshake failure, unknown CA TLS
HTTP 4xx/5xx, or a timeout after connecting Application or proxy

curl -v against the same URL from the same host shows how far it gets, and is our first command every time:

curl -sv -o /dev/null https://api.partner.example.com/v2/health

The * lines show resolution, the connect attempt, the TLS handshake and the request. Where they stop is where you start.

Layer 1: name resolution

There are two ways to resolve a name on Linux, and they can disagree. Applications use the system resolver via getaddrinfo, which follows /etc/nsswitch.conf (files, then DNS, sometimes more). dig bypasses all of that and asks a DNS server directly. Check both:

getent ahosts api.partner.example.com      # what the application sees
dig +short api.partner.example.com         # what DNS says
resolvectl query api.partner.example.com   # on systemd-resolved hosts, shows which link and server answered

If getent and dig disagree, look at /etc/hosts first; a stale entry from a past migration is a classic. If dig fails but dig @1.1.1.1 works, the problem is your configured resolver, not the record. If both resolve but to different addresses than the partner documents, you may be hitting split-horizon DNS or a stale private zone.

Also check which address family the client picks. If the name has an AAAA record and the host has a broken IPv6 path, some clients try IPv6 first and time out. curl -4 versus curl -6 settles that quickly.

Layer 2: the route

Once you have an address, ask the kernel how it would send a packet there:

ip route get 203.0.113.45
# 203.0.113.45 via 10.0.1.1 dev ens5 src 10.0.1.23 uid 1000

This shows the interface, the gateway and the source address the kernel will use. Surprises here include traffic leaving through a VPN or a secondary interface, or a source address that the remote firewall does not allow. On hosts with policy routing, add mark or from to the query to match how the application's traffic is classified.

To see the path beyond your own host, use mtr in TCP mode on the real port, because many networks drop or deprioritise ICMP and UDP probes:

sudo mtr --tcp --port 443 --report --report-cycles 20 api.partner.example.com

Loss that appears at one hop and disappears at later hops is usually that router rate-limiting replies to probes, not real loss. Loss that starts at a hop and continues to the destination is real.

For path MTU problems, the symptom is connections that open fine but hang when large responses or TLS certificates are sent. tracepath reports the discovered MTU along the path:

tracepath -n 203.0.113.45

Layer 3: the TCP handshake

Next, can we open a TCP connection at all, without any TLS or HTTP on top?

nc -vz -w 3 api.partner.example.com 443

Three outcomes, three meanings. Success means routing and firewalls allow the connection. Connection refused means a host answered with a reset: you reached something, but nothing listens on that port, or a firewall actively rejects. A timeout means packets are being dropped somewhere without a reply.

For timeouts, watch the handshake itself. In one terminal:

sudo tcpdump -ni any 'host 203.0.113.45 and tcp port 443'

In another, retry the connection. If you see your SYN going out repeatedly and nothing coming back, the drop is outside the host: a security group, a network ACL, a firewall at the partner, or an allowlist that does not include your egress IP. If you see no SYN at all, something on the host is dropping it before it leaves; check nft list ruleset or iptables -S. If you see SYN, SYN-ACK and then your host sending a RST, a local firewall or connection tracking is interfering.

While a connection is hanging, ss shows its state from the kernel's point of view:

ss -tnp state syn-sent '( dport = :443 )'

A pile of sockets in SYN-SENT confirms that handshakes are not completing.

One more thing to confirm at this layer is the egress address the partner actually sees. Behind NAT, it is not your host's address:

curl -s https://checkip.amazonaws.com

Partners often allowlist a specific IP, and a NAT gateway change is enough to break that.

Layer 4: TLS

If TCP connects but the client fails during the handshake, use openssl s_client. Always pass -servername, because without SNI many servers present a default certificate that will not match your hostname:

openssl s_client -connect api.partner.example.com:443 \
  -servername api.partner.example.com -showcerts </dev/null 2>/dev/null \
  | grep -E 'subject=|issuer=|Verify return code'

What to look for:

  • Verify return code: 0 (ok) means the chain validates against the local trust store.
  • unable to get local issuer certificate usually means the server is not sending an intermediate certificate, or your trust store lacks the root (common in minimal container images without ca-certificates).
  • certificate has expired speaks for itself; check the host clock too with timedatectl, because a wrong clock produces the same error.
  • A subject that does not match the hostname suggests the wrong SNI or a proxy intercepting traffic.

To see expiry dates and names on the leaf certificate:

openssl s_client -connect api.partner.example.com:443 -servername api.partner.example.com </dev/null 2>/dev/null \
  | openssl x509 -noout -subject -issuer -dates -ext subjectAltName

If the partner requires mutual TLS, add -cert and -key and watch for alert certificate required or handshake failure, which usually means our client certificate was not sent or not trusted.

Layer 5: HTTP and timing

When the connection and handshake work, the remaining questions are about the request itself and where time goes. curl's write-out variables give a timing breakdown per phase:

curl -s -o /dev/null https://api.partner.example.com/v2/health -w \
'dns=%{time_namelookup} connect=%{time_connect} tls=%{time_appconnect} ttfb=%{time_starttransfer} total=%{time_total} code=%{http_code}\n'

Each value is cumulative from the start. A large dns points back to the resolver, a large gap between connect and tls to a slow handshake or MTU issues, and a large gap between tls and ttfb to the server or anything in between doing work.

To test a specific backend or bypass DNS entirely while keeping the correct hostname for SNI and the Host header, use --resolve:

curl -sv --resolve api.partner.example.com:443:203.0.113.46 https://api.partner.example.com/v2/health

Also check for proxies. env | grep -i proxy on the host and inside the service's environment reveals an HTTPS_PROXY that the application honours and your shell does not, or the reverse.

Testing from inside the right network namespace

A command that works from the host proves little if the failing process runs in a container with its own network namespace, DNS config and routes. Run the same tools inside that namespace without installing anything in the image:

PID=$(docker inspect -f '{{.State.Pid}}' payments-worker)
sudo nsenter -t "$PID" -n ip route
sudo nsenter -t "$PID" -n curl -sv -o /dev/null https://api.partner.example.com/v2/health

Note that nsenter -n switches only the network namespace; the binaries and /etc/resolv.conf come from the host. For DNS checks, read the container's resolver config with docker exec payments-worker cat /etc/resolv.conf and query the same server explicitly with dig @<server>.

How our example ended

For the record, the partner API case was layer 3. getent and dig agreed, the route was normal, nc timed out, and tcpdump showed our SYNs leaving with no reply. The checkip call showed a new egress address: the platform team had replaced a NAT gateway the week before, and the partner's allowlist still held the old one. Ten minutes of following the packet, one email to the partner.

Common mistakes

Mistake Why it hurts Do this instead
Testing from your laptop or the host instead of the failing workload Different DNS, routes, proxies and egress IP Test from the same namespace, or with nsenter
Using ping to prove reachability ICMP is often blocked while TCP works, and vice versa Test the real port with nc or curl
openssl s_client without -servername Default certificate, misleading mismatch errors Always send SNI
Trusting dig alone Applications use nsswitch and /etc/hosts Compare with getent ahosts
Reading per-hop loss in mtr as real loss Routers rate-limit probe replies Look for loss that persists to the destination

Key takeaways

  • Read the exact error first; it usually names the failing layer.
  • Walk the layers in order: DNS, route, TCP, TLS, HTTP. Each has a decisive test.
  • A timeout means drops, a refusal means you reached something; tcpdump tells you which side drops.
  • Always send SNI when testing TLS, and check the clock when certificates look expired.
  • Test from inside the failing workload's network namespace, and know your egress IP.

References