Due to a firmware bug, unfortunately a single routing filter line which distributes routes between bng3-quic and our core was not being respected by our core network elements.
This line resulted in a subset of customers on bng3-quic, within a single /24 subnet, to have reduced, or no connectivity while this issue was located and routing filter line removed and re-added.
We have been in touch with our vendor and advised this bug has been fixed in a beta firmware release. Once released into a stable version, we will complete the necessary upgrades to this version to ensure this doesn’t reoccur.
We apologise to impacted customers for the inconvenience caused by this incident.
Issue confirmed resolved as at 10:00am today. No further occurrences since monitoring.
We are observing impacted circuits recovering and are monitoring for any further occurrences.
We have detected an issue impacting some customers connected to bng3-quic are experiencing connectivity issues.
We are working through draining these customers onto an alternate BNG to provide resolution.