C&Thoth · Azure MSP
MIR4 Global: massive Azure infrastructure and automated DDoS blocking
Dedicated operation of a global game with 3,500 servers and 650 databases on Azure, plus a pipeline that finds DDoS sources in logs and adds them to an NSG automatically.
- Role
- Cloud Engineer · Main engineer (client: WEMADE)
- Period
- Jul 2021 – Feb 2023
- Cloud
- Azure
- Scale
- 3,500 servers · 650 databases · Massive
- Key tech
- Azure · Dedicated Host · PowerShell · Log Analytics · NSG
- 3,500
- Servers
- 650
- Databases
- 0
- Human steps from detection to block
At a glance
Problem
At this scale, a physical failure or an attack could spread across the global service. Night and weekend DDoS attacks were slow to answer when a person had to block them by hand.
Approach
Involved from the first design stage, I helped set a high-availability layout on Azure Dedicated Host. Mitigation flow logs from Azure DDoS protection are queried on a schedule in Log Analytics to find abnormal TCP SYN sources, and PowerShell adds those IPs to an NSG deny rule.
Result
Detection to block runs with no human step, which removed the delay at night and on weekends. Resource deployment, automated incident fixes, and asset collection were also automated in PowerShell.
Up close
What I learned
I learned that for the service to survive one physical host failing, spreading dedicated hosts across separate fault zones is the baseline. At this scale, failures and attacks spread across the global service, so such decisions had to be made from the first design stage rather than added during operation. For DDoS, so that response speed was not tied to when someone was awake, finding abnormal sources in the logs and adding them to the NSG block became one automated flow. And the more blocking is automatic, the more each block needs its reason recorded, so a person can review the decision later.