We have had 502 error from Google since we can remember. We know our cluster has enough machines and we know they are not returning errors nor are they overloaded (we have metrics to check). We know these requests (502) never make it to our machines in the first place. I am not Sherlock Holmes, but after lengthy discussions about this topic we literally have no other explanation.
As for major IAAS providers, I worked for GCP for some time so I know for a fact (since I know how it used to work) this is totally possible. As a client I used Azure for a quite some time (definitely worst than GCP) and we had troubles every 2 weeks. It was a nightmare.
So to me, and with my experiences and knowledge I have, it sounds not just likely, but normal.
Google manages their own load balancers, which send traffic world wide to all the regions. We don’t have direct control over them (we have some limited control). This is the charm of the offer - you don’t need to worry too much with balancers, Google does.
It is worth mentioning that after a long stress test, we concluded these errors were around 0.0015% of all requests. This is well withing the terms of service Google providers iirc, so there is little that can be done. We are leaving this matter as 0.0015% is not a sufficient error rate to invest more time and effort.
