那些年我们终于找到的缺失防火墙规则

背景与摘要

系统管理中的故障排除往往充满了巧合与直觉。本文讲述了一个真实的故障排查案例：一台提供热门数据的新Web服务器因为遭受大量并发请求而过载，导致共享同一交换机的其他机器出现NFS性能问题。管理员们随后使用 tc 和 Apache 的 mod_qos 添加了带宽和并发限制。在这一系列调整中，他们意外发现将新服务器加入边界防火墙的黑名单拦截规则后，并发请求出现了第二次大幅下降。这次下降揭示了一个之前被忽略的缺失防火墙规则，并解释了为何旧服务器在没有 tc 或 mod_qos 保护时也能免受类似流量的冲击。

Summary

在系统管理中，根本原因的发现往往是混乱的，它受巧合和直觉的驱动，不亚于有条理的调试。本文回顾了导致在一个高需求 Web 服务器上发现缺失防火墙规则的一系列真实世界故障排除事件——强调了指标中出乎意料的下降是如何充当关键诊断信号的。

In system administration, root-cause discovery is often messy, driven as much by coincidence and intuition as by methodical debugging. This article recounts the real-world troubleshooting chain of events that led to the discovery of a missing firewall rule on a high-demand web server—highlighting how unexpected drops in metrics can act as critical diagnostic signals.

1. The Overloaded Web Server and Collateral Damage

当我们最初为一组极高需求的数据部署新的 Web 服务器时，Apache 并没有被配置为处理海量的并发连接。它立刻就被 HTTP 请求淹没了，但我们一开始并未理会这些警报。

When we first deployed our new web server for a highly in-demand data set, Apache wasn't configured to handle a high volume of concurrent connections. It was immediately overwhelmed with HTTP requests, but we initially shrugged off the alerts.

然而，由于这不断触发我们的监控系统，我将 Apache 的连接数上限调到了一个极其离谱的高数字。虽然这让监控系统不再报警，但它将服务器的出站带宽推到了 1G 线速，我认为这是可以接受的。

However, because this constantly tripped our monitoring system, I turned up Apache's connection limits to an absurdly high number. While this kept monitoring happy, it pushed the server's outgoing bandwidth to 1G wire rates, which I figured was acceptable.

不久之后，一台不相关的机器开始遭受严重的 NFS 性能问题。经过调查，我们意识到它与那台极度活跃的 Web 服务器共享同一个 1G 交换机，并且它的网络流量被完全挤占了。

Shortly after, an unrelated machine began suffering severe NFS performance problems. Upon investigation, we realized it shared the same 1G switch as the hyper-active web server, and its network traffic was being completely crowded out.

2. Bandwidth Limits and Quick Fixes

为了紧急止损并恢复正常的 NFS 性能，我匆忙地手动添加了一些基于 tc 的带宽限制——这意味着我直接在一个活动的 shell 会话中输入了 tc 命令。

In a rush to stop the bleeding and restore normal NFS performance, I hastily added some tc-based bandwidth limits by hand—meaning I typed tc commands directly into an active shell session.

几天后，我们找出了如何直接在 Apache 内部配置 mod_qos 限制。我们引入的参数之一是对来自单一 IP 地址的并发连接设置严格上限。效果立竿见影且十分显著：

A few days later, we figured out how to configure mod_qos limits directly within Apache. One of the parameters we introduced was a strict ceiling on concurrent connections from a single IP address. The effect was immediate and dramatic: * Apache 错误日志显示来自触碰阈值的一小部分 IP 的大量流量。 * The Apache error log showed heavy traffic from a subset of IPs hitting the threshold. * 我们的指标系统显示，并发请求直线下降至大约之前水平的一半。 * Our metrics system showed concurrent requests plummeting to roughly half their previous levels.

3. Connecting the Dots: The Missing Firewall Rule

眼前的危机解决后，我们不得不决定是让基于 tc 的限制永久化（这将需要编写启动脚本，因为这将是我们第一台在生产环境中使用 tc 的机器），还是完全依赖 mod_qos。

With the immediate crisis resolved, we had to decide whether to make the tc-based limits permanent (which would require writing startup scripts, as this would be our first machine using tc in production) or rely entirely on mod_qos.

在权衡选项时，我想起我们的外围防火墙包含了一个通用的“拦截暴力破解行为”的系统。将其扩展到针对我们新 Web 服务器的 HTTP 和 HTTPS 请求，作为一层额外的保障，是非常合理的。

While weighing our options, I remembered that our perimeter firewall included a general-purpose "block brute-force things" system. It made sense to extend this to HTTP and HTTPS requests targeting our new web server as an extra layer of insurance.

就在应用此更改后，我们的指标系统记录到了并发 Apache 连接的另一次大幅下降，再次缩减了一半。

Immediately after applying this change, our metrics system registered another major drop in concurrent Apache connections, shrinking by half yet again.

（外围防火墙通过受限于 HTTP/HTTPS 限制的明确 IP 地址允许列表来运行；因为我们的新 Web 服务器有一个全新的 IP 地址，所以直到现在它才被加入到该列表中。）

(The perimeter firewall operates via an explicit allowlist of IP addresses subjected to HTTP/HTTPS limits; because our new web server had a fresh IP address, it simply hadn't been added to the list until now.)

这第二次下降是令人恍然大悟的时刻。它解释了为什么我们的旧版主 Web 服务器在很大程度上避免了被这些流量淹没：那个外围防火墙规则正是旧服务器拥有而新服务器缺乏的保护。我们的主服务器没有 tc 限制，而 mod_qos 早在几年前就被禁用了。

This secondary drop was the "aha!" moment. It explained why our main legacy web server had largely avoided being overwhelmed by this traffic: that perimeter firewall rule was the one protection the old server possessed that the new one lacked. Our main server had no tc limits, and mod_qos had been disabled years prior.

Afterword & Post-Script

最终，我们决定同时保留 Apache 的 mod_qos 限制和基于 tc 的限制，作为一项纵深防御措施。我们希望在这台特定的服务器上严格执行带宽上限，利用两个独立的机制可以确保只有当两者同时失效时才会发生溢出。

Ultimately, we decided to keep both the Apache mod_qos limits and the tc-based limits as a defense-in-depth measure. We want to strictly enforce bandwidth caps on this specific server, and utilizing two independent mechanisms ensures that both would have to fail simultaneously for an overflow to occur.

我们还不足够担心到需要直接在 FreeBSD PF 中实现带宽限制——部分原因是防火墙级别的配置错误造成的破坏范围要比搞坏单台主机大得多。

We aren't yet worried enough to implement bandwidth limits directly in FreeBSD PF—partly because a misconfiguration at the firewall level carries a much higher blast radius than breaking a single host.

来源: Finding Our Missing Firewall Rule

Source: Finding Our Missing Firewall Rule