Starting approximately 2:58pm 23rd April, we began receiving reports of abnormal activity within a segment of our network.
The decision was made to assemble our action team at 3:00pm, which was completed at 3:06pm, and status page incident created at 3:11pm.
Investigations concluded at 3:17pm that a single network element was creating issues, which had full connectivity within the network, however lost connectivity to both of our metro route reflectors via IPv4, resulting in links appearing up and active, but return path traffic being unable to locate the destination path.
This impacted some select downstream peering sessions, and internal tools using this element as a primary path, with the largest impact being felt by internet subscribers and attempts to renew their DHCP leases being unsuccessful.
At 3:19pm the decision was made to reboot this network element, and also simultaneously complete a minor firmware upgrade.
This was completed at 3:21pm and our action team began confirming service restoration, which was completed by 3:31pm. Monitoring continued showing no further outlying impacts or issues.
Investigations will continue into why connections to these metro route reflectors will continue and any appropriate adjustments completed to ensure this doesn’t reoccur.
We apologise for any inconvenience experienced to our customers as a result of this incident.