front end – Failing to publish changes to code on Sundays

You need a way to manually bypass some of these checks and balances for emergency fixes. This incurs a lot of risk, especially for financial institutions. But then again, not fixing the problem also incurs a lot of risk.

The manual bypass can be a person logging in to whichever tool you use to automate deployments, and then push a button to start the deployment. Some tools allow you to specifically record approvals from other people (sometimes called “gatekeepers”). Worse case scenario, do it over e-mail and save the e-mails some place accessible by people at your organization. Saving them right in your automation tool would be ideal.

The “manual bypass” should absolutely not be a single person. If you need to deploy code on the weekend, and your automated tests cannot run, because external resources are closed for business reasons, then you and management need a way to bypass this and get the fix out to production. Something like this would probably need approval from someone pretty darn high up in the corporate hierarchy.

Get ready to be on a first name basis with the CEO. This might be a good thing or bad thing.

Does this show a problem in how your unit tests are written? If so, what is the problem?

If your unit tests are failing because the banks are closed, then you have not written true unit tests. A unit test runs in-memory and operates on data in its own thread without reaching outside the thread.

More likely you have written integration tests. An integration test can certainly get data from outside the test itself, be it from a file, web service or database. There is nothing wrong with integration tests failing on Sunday because the banks are closed. The tests are not written incorrectly. In fact, it sounds like the tests are written correctly. You just need a way to bypass all of that automation for emergency fixes.

And get to know your CEO, because they should probably be the one to approve this blatant circumvention of extensive and well-planned checks and balances.