DevOps Server Optimization for TCP Closewaits

My production box is a high-traffic Linux server that keeps thousands of TCP connections open at any given moment. It is already running Nginx, but connection spikes are starting to expose bottlenecks and a few security gaps. I need a DevOps specialist who can step in and deliver three things in one engagement: noticeable performance tuning, solid security hardening, and smarter resource management.

Here is what I have in place:
• Ubuntu-based host on a dedicated machine
• Nginx handling both HTTP and long-lived TCP streams via stream{} blocks
• Basic monitoring with top/iostat and Nginx stub_status

What I need from you:
1. Tune the kernel (sysctl, ulimits, epoll, TCP backlog, etc.) and Nginx worker settings so the server continues to serve traffic smoothly even when connection counts jump.
2. Lock down obvious attack surfaces—TLS configuration, headers, fail2ban, firewall rules, and any Nginx-specific best practices—without sacrificing throughput.
3. Rebalance CPU, memory, and disk I/O so the machine stays under sensible thresholds during load tests and can self-recover if one resource saturates.
4. Handling Closewait numbers

Deliverables (all version-controlled or clearly documented):
• Updated configuration files (sysctl.conf, limits.conf, nginx.conf)
• A before/after benchmark report showing latency, throughput, and resource usage improvements
• Security hardening checklist with each item marked off and verified
• A lightweight monitoring or alerting script/dashboard (Prometheus, Grafana, or similar) that visualizes key metrics

Acceptance criteria: all changes must pass a 30-minute stress test at 1.5× current peak traffic with no dropped connections and average latency no worse than present baseline.

If that sounds like your wheelhouse, tell me which specific tools you would reach for first and how quickly you can begin.