Vendor Contracts

INTRODUCTION

One of the interesting aspects of my job as a senior technology manager in banking was to participate on vendor contracts reviews and negotiations. The idea was that by attending I could ensure that technology requirements for supporting the application were considered in the contract. My team would be supporting the product once it was accepted in-house. Naturally, most of this section is focused on contractual technology requirements in banking when using service providers.

The nature of the contracts my team reviewed were for vendor applications that we would subscribe to or license in-house for a business user base and support it. These were not in-house developed products; these were applications a vendor built and made available to the bank as a software package, but more often as a vendor hosted application accessible by the bank through a web browser.

I'm writing this section to share some of the technology features my support team would require in vendor contracts and hoping these experiences will benefit others in working with contract negotiations.

Build or Buy

Over the years in banking I've noticed an increasing trend to buy versus to build applications. It used to be that most firms, banks included, would build the applications they needed. However, there is a high cost to build and then maintain an application with an in-house team. A vendor could build an application and spread the development and maintenance costs across many clients; thereby reducing the cost of the application to any one client. As well, a vendor will likely have a longer product life cycle for an application as it is how they continue to drive revenue. A vendor will also be more enticed to develop new features to continue locking-in the client.

What I've observed with in-house applications, once they are built and moved into production, there is little interest in maintaining them with new features over time. Those applications that are impacted by regulatory changes, necessarily receive more attention. But often the application and underlying infrastructure stay stable for many years until the operating system or server is no longer supported or a regulatory change requires a code update. Even so, sometimes it's difficult to find the people who worked on the application. Staff turnover of 11% is normal at most Canadian banks.

Vendor Technology Management

The reason my team was created was because as application were brought in-house from vendors, there was no one in-house who could be the technology liaison between internal bank clients (consumers of the applications) and the vendor. Our VP at the time, stipulated that every technology application brought into the bank had to have technology ownership to best support the business users. Every application, be it vendor hosted or in-house built, is an asset to be managed.

One example, was a desktop tax software product that had to be deployed to several branch desktops. During tax season, the team had to coordinate getting the new versions from the vendor, orchestrating the testing of the application with the business and then package the application for deployment and have the distribution team deploy it (through automation) to the business desktops.

Another case was supporting new releases of a trading system the bank used but was hosted and developed by a vendor. This was a more complex suite of applications, that supported thousands of Wealth management clients. It was a full time effort for one resource as three releases were deployed during the year. Each release required a small army of QA resources to review the vendor's release notes, prepare QA test cases, inform the business on changes to the application, coordinate with internal IT teams if any technology impacting changes were needed, manage the change records to notify the bank of the upcoming release weekend and then test again the application on the deployment weekend before it was available to clients on Monday morning.

Monitoring of the application over the week to ensure its stability. Once the release was stable, it was time to start preparing for the next release. A full time job considering the team also had to address any security audit deficiencies found! When the release manager retired, she was replaced by three people! This was the most client critical application we managed.

TECHNOLOGY SUPPORT REQUIREMENTS

• Security

  The security aspects to be considered when working with a vendor are significant. When a bank outsources services or uses applications built by a third party, there is a chance that an adverse event (incident) at the vendor will reflect negatively on the bank; this is third-party risk. It is one of the reasons that banks, justifiably, are very demanding of their vendors on application security requirements.

  For example, exfiltration of client data due to a vendor's improper data management controls, will result in damages to the bank and could result in fines from financial regulators. These type of security lapses are very harmful to the bank as it erodes the client's trust. Although the bank trusts that the vendor has adequate controls over their data, lapses in security impacts the bank very publicly whereas the vendor rarely gets the same level of attention in the news.

  An example; in 2019 Desjardins Group experienced a data theft affecting 9.7 million clients. A series of technology and administrative gaps caused the data leak; lack of controls. At the time this was the largest data leak in the Canadian financial services sector. The data breach cost Desjardins $108 million (CAD). A subsequent class action lawsuit cost an additional $201 million.

  1. CSAE 3416 Report

     This is the Canadian Standard on Assurance Engagements report. It addresses audit engagements undertaken by an auditor on the controls of a service organization (vendor). It provides independent assurance of controls at a service organization relevant to security, availability, processing integrity, confidentiality or privacy. This is also known as a SOC 2 report. There is also a SOC 1 report but that has a financial focus that includes a service organization’s controls relevant to an audit of a service organization’s client’s financials.

     From a technology perspective, the SOC 2 report is important; especially for client critical applications. It is normally produced annually by an independent auditing firm that the vendor hires. It is however costly for the vendor to do this on an annual basis and some will push back on the need to do so. As technology owners we would annually review several SOC 2 reports provided by our vendors. Both technology and business would review the report and provide feedback on any identified gaps or simply provide sign-off indicating that it was reviewed and there are no concerns. These reports are highly confidential because sometimes it highlights gaps that vendors have in their processes; issues that they would not want to be made public.

     I've never seen a vendor's SOC 1 report. Likely it could be valuable for the procurement department. The SOC 2 report is what the technology and procurement (contracts) teams focus on.

     Recall that the intent of the report is to give an independent assessment that all is well with the vendor's controls so that as a bank, we are comfortable that there is little chance of third-party risk. It's extremely important to minimize this type of risk.

     Of the SOC 2 reports I reviewed, some of the gaps that I've seen reported were:

     • No monitoring of application error logs. Application errors were detected but there is no evidence of follow-up action taken. The risk being possible improper calculations on data or corrupt data.
     • Gaps in removing employee access to systems on a timely basis for those who have left the vendor.
     • No process for monitoring alerts from intrusion detection.
     • Inadequate business continuity planning (BCP).
     • Weak password management on production systems.

     This is just a sample, there were many more.

     A practice to watch out for is when a vendor provides the SOC 2 report of their own service provider. For example, a vendor who has their application hosted at IBM, Rogers or an Azure data centre may provide the SOC 2 report of their hosting provider. I've seen this several times, especially with smaller firms (startups). This is not acceptable as it does not address the vendor's own internal controls around their own firm's processes for managing data, security, BCP, etc. Vendors must implement their own set of controls and provide their own SOC 2 report.

     For business critical vendors, it's important to demand in the contract that a CSAE 3416 report be provided annually by a certified independent auditor.

  2. Vendor Site Visits

     This is another check that a bank may demand prior to signing a contract to ensure client data will be housed in secure facilities with the proper controls. I've participated on one of these visits and reviewed the auditor's findings on other visits they conducted. Typically, the site visits are conducted by the bank's security auditor. In another case an independent auditor provided an assessment on the vendor's hosting provider; factors such as building access controls, physical access to servers and who can access, how the firm's servers are physically isolated from that of other hosting clients, proximity to a fire station, the safety of the location as regards disruption due to bomb threats, fire, etc.

     Some of the issues noted with vendor sites were:

     • As a result of dumpster diving at the vendor site by the bank's security auditor, printouts were discovered with client information. These should have been shredded prior to disposal.
     • At one site the staff had a warm bean cooker inside the computer room; a fire hazard.
     • Inadequate CCTV coverage of the entrances to the data centre.
     • At another vendor, the bank's audit staff checked into the building as a valid visitor, then checked out of the building, walked back in without a visitor badge and were let into the premises.
     • At a vendor site, the security auditor roamed the building freely without any constraints on where he could go. No one questioned his presence. He had access to confidential documents on desks as he wandered about the facility.
     • At one data centre the electric power transformer supplying the facility was in plain view from the street. Without a surrounding protective wall, the building's power source was exposed to vandalism/disruption.

     Vendor site visits are expensive for both the bank and the vendor. These visits are usually reserved for the most business critical clients. One vendor we dealt with was an on-line travel provider. Employee corporate credit card, passport and other restricted data was to be kept on their servers. A site visit was warranted due to the sensitivity of the data; this is where we discovered the bean cooker!

     From a support perspective this is not something the technology staff would demand in a contract but a technology owner may be asked to accompany the auditor on-site. The audit reports of such visits are always an interesting read. A security auditor attending a contract meeting would be demanding the site visit if they felt it was necessary due to the data classification.

  3. Data Encryption

     Recently this has become a hot topic. Data at rest and data in motion need to be encrypted. From a technology support perspective, this must be in place already at contract signing. If not, it becomes the responsibility of the technology team and contract management to ensure the vendor does this post contract sign-off. If not done, a security finding is issued against the application. Once contracts are signed, there is little motivation from the vendor to remediate issues expeditiously.

     If a vendor does not have data encryption at rest, this could become a big expense for them to remediate. Subsequently they may push back on this or promise to deliver it at a far off release date. Encrypting and decrypting data adds overhead, often requiring disk and server upgrades. If the encryption is performed through out of the box database functionality then often a higher licensing cost is required to obtain that functionality from the database vendor. The vendor may offer to only encrypt any confidential or restricted data as a solution. However, the bank's security representative and business needs to approve what can remain unencrypted.

     Data in motion is often easier to implement by encrypting data transfers either through secure channels or ideally a combination of encrypting data (e.g. PGP) and sending it over secure channels (e.g. SSL). However, there is also server overhead in this scenario.

     Often overlooked is ensuring that the vendor's off-site data backups are also encrypted. Be it on tape or disk. Any client data leaving their premises must be encrypted.

     In summary, data encryption should be a contractual requirement. If not present, then a date by which it will be implemented and remedies for not doing so by that time need to be specified. Given the challenges and cost of data encryption, be ready for vendor push-back regarding financial remedies. If the bank must have the application for its clients, despite the gaps, then the vendor really has the negotiating leverage and the issue is up for discussion. It also becomes more difficult to close these gaps once the bank signs long term contracts (5 years and greater) with a vendor. The technology support teams should not own lack of encryption as a vendor item to manage post contract.

• Application Release Scheduling and Notification

  Vendors release new updates to their applications constantly. However, before the application is available to the business it needs to be tested either formally by the bank's QA group or lightly tested by the business area using the application. In either case, no application is released to the broader user audience without being first tested.

  Depending how many vendor releases are made, this could have an impact on resources. In one bank we had an application upgrade every two weeks. The upgrades were scheduled with the vendor during the weekday evenings, as it was their preference. Often these were minor bug fixes but required a business user to test the application and provide sign-off each time.

  Another large application that was used in trading had four annual releases that required extensive testing by the QA group. On average most vendor applications had a monthly release cycle. One exception was vendor software updates during tax time. Whenever an update was made to the tax rules, a new vendor application release was made available. It had to be immediately installed on the tax user desktops. An exception to the process was made using a blanket change agreement during tax time.

  Whenever a vendor application is released, the technology representative has to create a Change Ticket (CT) that is reviewed by the bank's Change Board. Every technology change, must be reviewed before being approved for production. On being notified of a new vendor application version, the technology rep creates a CT which is then presented at the Change Board. The CT remains open until the application is successfully tested. Once tested the CT is closed. If the testing fails, the technology rep needs to explain to the Change Board, at a subsequent meeting, the failure case.

  Every bank, and most organizations have a defined production release schedule where the Change Board meets to review application or infrastructure changes; often every week or, at one bank, every two weeks.

  If issues are uncovered during testing the vendor can either restore the previous working version of the application or attempt to repair the application during the scheduled maintenance window.

  More complex applications, such as those involved in trading or other books of record, have longer release cycles allowing for sufficient time to create CTs and present to the Change Board. I recall one application that had a weekly application release schedule; this was an extreme example and kept one resource quite busy. The frequency and complexity of the vendor releases also drives the need for resource capacity, which needs to be considered prior to the technology support team accepting it into their support portfolio. If the technology teams don't have the resource capacity to assume frequent releases, then the vendor must change their release schedule.

  So what does this have to do with contracts? If your organization's change management cycle is two weeks, your organization cannot take weekly releases. So, at contract time the release cycle needs to be discussed so that there is no mismatch between the vendor's release cycle and the organization's ability to take the releases.

  Another aspect of vendor releases is the Release Notes a vendor needs to provide with each release. The Release Notes describe what has changed in the application. My experience with these is that Release Notes are written at too high a level and don't provide sufficient detail for the QA teams to create adequate test cases. This poses a risk to the users if there are lapses in testing due to lack of feature information. Ensure the vendor is aware of the need for detailed release notes.

  As regards notification, vendors have notified the business users of new releases. However, the vendors should also notify the technology representative for the application as they have to assess the technical impact of the change, create the CT and represent the change at the Change Board meeting. It has happened that the business failed to notify the technology team of a new vendor release.

  In short, ensure vendor releases match the organization's release cycle and capacity to take the change. Each release must have detailed release notes.

• Application Support

  Provisions in the contract need to be considered when there are application issues. The level of application severity needs to be defined. What are the factors that determine if an application failure is a Severity 1 versus a Severity 2 and what is a Severity 3? As well, how quickly will the vendor respond to a Severity 1 notification versus lower severity levels. If the application failure has client impact, how long is the vendor contractually obliged to contact the organization once they have been notified? These parameters need to be defined at contract time.

  Likewise, in the event of a failure, what is the service level agreement? Is vendor support only through business hours or is the application sufficiently critical to have 24X7 telephone support?

  Some contracts also specify the resolution time to fix the problem for different severity levels. For a Severity 1 issue, the contract may demand the issues be fixed within 60 minutes. For a Severity 3, issue the contract may stipulate a fix within 30 days. The resolution time depends on the criticality of the system to the business.

  When the organization experiences an application a failure, who will the support people call? There needs to be an escalation matrix within the contract to denote the first level of escalation, typically a vendor support desk, the next level up (support manager) and the next level beyond that; the vendor's executive. Their office telephone numbers, mobile numbers and e-mail need to be available to the organization's technology support staff. Vendor support over WhatsApp or Messenger is not allowed in banking.

  It's happened on my team whereby a vendor release was available for testing in the evening, a problem with the application was encountered during testing, then we contacted the vendor's first support level only to receive no response. Incidents such as these require next level escalation contact information.

• Service Level Agreements (SLAs)

  The Service Level Agreement defines the level of service expected by the organization from its vendors. Metrics are used to define the expected level of service.

  The SLA section in the contract will specify the services expected from the vendor and the required performance parameters of the services. If the requirement is not in the SLA then it's not contractually binding.

  Generally this section speaks to the metric associated with the various services. Some examples are; the application's UI response time to a user query, the service availability (up-time), the vendor's support hours, the availability of reports or special processing by a given fixed time or the response time of vendor hosted APIs.

  Service Level Agreement Components

  

  Some SLAs can be measured by the bank whereas others have to be measured by the client. Up-time can be measured by the bank as the bank will know when the application has gone off-line or when report delivery does not meet the SLA specification. Performance metrics for API response time, UI response to user queries, can only be accurately measured by the vendor. This is because wide area network latency between the bank and the user's application can degrade the response time. The vendor cannot be held responsible for delays by the network service provider in delivering their service. For example, the vendor can prove that an API turnaround took 500 milliseconds by logging the request/response pair, but the bank may experience the request taking one second. The additional time is due to the network travel time to send the request and the time to receive the vendor's response at the bank's application.

  Note, that vendor hosted applications are often in different geographical locations from the bank. The vendor's data centre distance from the bank's data centre will incur a variable network travel time which will impact the response time experienced at the bank. At one downtown Toronto bank, their trading system depended of several vendor hosted applications located in Laval (Quebec) New York (USA) and Barry (Ontario).

  In the case where the vendor provides the metrics, the norm is to do so on a monthly basis. A performance report would be provided to the technology owner and the contracts management staff. The monthly report is to ensure it is reviewed and the vendor is in compliance with the service parameters everyone agreed too.

  As a practice, each SLA will be numbered as in the example above. It will indicate in clear terms the service description, how it will be measured for compliance, who measures it (the bank, vendor or a third party), how often is it measured (monthly, quarterly, etc.) does failure of the service trigger the option to terminate the contract and what are the remedies (financial penalties) if the service fails.

TERMINATION AND REMEDIES

If it's important to have SLAs in contracts, then it's equally important the client consider non-performance penalties and contract exit conditions when a vendor consistently fails to provide the service.

The contract termination conditions and remedies are usually located in a separate section of the contract. If failure of the SLA can trigger a contract termination, then the conditions need to be specified.

Referring to the SLA example above, a trigger for contract termination could be:

"If the up-time (SLA 7) performance parameter is not met for three consecutive months, or if the up-time is below performance for any four times within a calendar year (January 1st through December 30th), then the client has the option to terminate the contract as per the termination process outlined in Appendix C."

Note that the client has the option to terminate, not the obligation. The vendor may be sufficiently critical to the business that terminating the relationship is not an option. Other venues need to be explored in these cases; senior executive management escalation or vendor acquisition in the extreme.

There will be variations of termination triggers, this is simply an example. If the client decides to terminate, then the vendor has to be given notice. The client will need time to replace the vendor with another one; this will take time often due to the application complexity and number of interfaces with the bank's systems. Additionally, written guarantees that client data has been purged from vendor's systems, and premises including off-site storage, is required. This is never a pleasant discussion.

The other aspect of SLA performance failure is remedies. The purpose of remedies if not to derive revenue at the peril of the vendor but to entice the vendor, through financial penalties corrects the performance. It's an unpopular discussion with the vendor because they lose revenue.

Again referring to the example above, a remedy for the up-time service failure could be stated as:

"If the up-time for SLA 7 is below the service performance metric for any one month, the vendor will credit the client 25% of the monthly maintenance fee for that month, to be credited the following billable month."

So, if the vendor has a performance failure in June and the bank pays the vendor a monthly application maintenance fee of $10,000, then in July the bank is only billed $7,500, calculated as; (10,0000 - (10,000 X 0.25)).

Remedies

As previously stated, the goal is not to use remedies to reduce operating costs by clawing back vendor maintenance funds but to entice the vendor (by financial penalties) to change their behaviour to contract compliance. In summary to entice vendors to fix their applications to eliminate the issues.

Service Level Agreement Components

Normally for a client with a few SLAs the remedies are on a 100% scale. For example, a client with 5 SLAs will have penalties based on the level of criticality with the service impacted. The up-time SLA is often the most critical and will have the highest penalty. So, assume five SLAs have the following claw back percentages of the monthly maintenance fee of $10,000 paid to the vendor.

1. SLA 1 - 40%
2. SLA 2 - 25%
3. SLA 3 - 15%
4. SLA 4 - 10%
5. SLA 5 - 10%

SLA 1 and 2 are the most critical, with the first being an uptime SLA. So, if the vendor misses the uptime performance metric, then the client can recover $4,000 of the monthly maintenance cost. Assuming this application causes lost client revenue when it's unavailable, we penalize the vendor so they are motivated to fix their infrastructure to ensure uptime compliance.

The second SLA has a 25% penalty. This SLA is for receiving a file from the vendor at a specific time of day. If the file is received late, it has a problematic downstream impact on several other client consuming applications. When it's late, several batch jobs on the client side need to be re-scheduled to account for the delay. The delay risks the timely delivery of the firm's daily large transaction end of day report to the client's regulator for which there is a penalty for non-delivery. As you can see there is a valid reason for enticing the vendor to change their behaviour.

So as the business grows, the client asks the vendor to create 10 new APIs that the client calls from their trading system. These are critical business APIs, some dealing with order fulfillment, order status, trading history inquiries, etc. Each API requires a different performance requirement. For the order entry API, the client demands a maximum API response time of 300 milliseconds (ms), whereas for the order status inquiry API, the maximum response time cannot exceed 600 ms.

As the vendor provides new services to the firm, the monthly maintenance increases to $15,000.

The client now has 15 SLAs to calculate percentage remedies against. On a 100% scale the 15 SLA performance remedies would have to be scaled down. For example, SLA 1 and SLA 2 represent 55% of the remedies leaving 45% of remedies to be allocated to the remaining 13 SLAs.

The 250% Scale

So, how do you entice the vendor to remediate any deficiencies in their application without watering down the weight of each remedy to fit a 100% scale? The answer is to increase the scale beyond 100%. I've seen a 250% scale in use for those cases where the client has a large number of APIs that have critical performance requirements; such as the case of uptime, files or API response time. It does not have to be 250%, it could be less or more, but this is a suggested approach for managing remedies on contracts with many SLAs.

Vendors hate this approach as it allows the client to maintain higher penalties for critical services and not have to reduce the penalties on a 100% scale. Vendors prefer the 100% scale because it waters down the remedy of any SLA when there are many.

I learned of this approach at a procurement seminar. I worked with the contract management staff at one bank to implement this approach for a vendor application contract. There was enormous push back from the vendor, because, frankly they could not always maintain SLA performance and would be subject to high monthly penalties.

Using the example above, the remedies for the 15 SLAs, using a 250% scale could be stated as:

1. SLA 1 - 40% - uptime
2. SLA 2 - 25% - file reception
3. SLA 3 - 15%
4. SLA 4 - 10%
5. SLA 5 - 10%
6. SLA 6 - 30% - order entry API
7. SLA 7 - 15% - order status inquiry API
8. SLA 8 - 15% - order cancel API
9. SLA 9 - 10% - order history API, 12 months
10. SLA 10 - 10% - order history API, 6 months
11. SLA 11 - 15% - order history API, 3 months
12. SLA 12 - 15%
13. SLA 13 - 15%
14. SLA 14 - 15%
15. SLA 15 - 10%

In this example, the remedies above amount to 250%, but in practice the client would not demand more than 100% of the monthly maintenance amount as a penalty. The client may even cap the monthly maintenance amount to never exceed 80% or less of the monthly amount as a penalty. Lower penalties will facilitate acceptance of the remedy terms by a vendor.

What the SLA remedy table above implies is that SLA 6 is critical to the firm. A performance failure would result in a $4,500 penalty ($15,000 X 0.30). Typically the APIs more critical to the firm are those that have financial impact and have higher call rates than others. In an actual trading scenario, 20% of the vendor APIs were the most frequently called by the client's application.

When I was helping the contract manager with a vendor contract we did a call rate analysis on the APIs to determine which ones the bank called more frequently and which ones would have a higher financial impact during a service failure. These two factors determined the percentage penalty associated with each API.

Vendor Transaction Rate and Concurrent Users

Vendors will agree to the performance parameters if it's based on a maximum performance threshold for provisioning their services. What this means is that a vendor will also set limits on how they can provide good service. A vendor will agree to SLA remedies if they specify that it only applies to a maximum concurrent user count or a Transactions Per Second (TPS) rate.

Consider the example of a vendor application that the client's users log into. The uptime SLA is valid if the client's concurrent user count is 60 users. Beyond 60 concurrent user sessions the vendor can no longer guarantee that the system will remain stable. This is due to the fixed capacity of the vendor's infrastructure. If the client requires 100 concurrent user capacity, then the vendor needs to upgrade their networks, servers and possibly software to accommodate that need.

So, the SLA stipulates a 98.5% availability with 60 concurrent users, but on a particular day, there are 100 concurrent users and the vendor's system crashes. You could blame the vendor for allowing more users beyond the 60 limit, but the reality is that the client exceeded the documented vendor limit of 60 sessions, so the vendor cannot be penalized as the client is at fault for causing the downtime.

Similarly with APIs there are limits on a vendor's systems how many times the API can be called within one second; referred to as the TPS. With APIs, the vendor will define the number of TPS their infrastructure can tolerate while meeting the required response time. For example, the vendor may state that if the client needs a 300 ms response time this can be met if the call rate does not exceed 120 TPS. So, as long as the client calls the API less than 120 times per second, the vendor guarantees an API response rate less than or equal to 300 ms.

What happens if the client call the API 140 times per second? In this case, given the fixed capacity at the vendor it's possible the API response time will increase beyond 300 ms. The API response time will be slower. The vendor cannot be penalized because the client is calling the API more frequently than was agreed to in the contract. The worst case scenario is that the client's excessive API calls crashes the vendor's system. This would normally trigger the uptime SLA penalty, but because the client exceeded the contracted TPS rate, the vendor is not liable.

The summary is that as the technology representative on a contract review meeting you need to specify the response time expected of the application, the required TPS or concurrent users your firm requires from the vendor. For some vendors, infrastructure upgrades are needed to meet client requirements, resulting in higher monthly maintenance costs to the client.

VENDOR PERFORMANCE

Most vendor contracts that have critical service requirements for the bank will have a quarterly or annual vendor performance review meeting. If the vendor is local, the meetings are held face to face. Occasionally, the vendor's representatives will fly in from other cities to attend in person.

This is an opportunity for the vendor to show the key metrics by which their performance is evaluated. For example, their application uptime was 99.9%, they processed 10 Million client transactions in the last quarter, printed 50,000 tax statements and mailed them out on time meeting all SLAs, etc. Depending on the vendor the metrics will be different.

The technology representative is usually invited to these meetings. This is also an opportunity to drill down into any SLA failures and next steps to remediate those.

Some vendors will use this opportunity to present what new services or products they are working on. It's an opportunity for them to up-sell more services. This is also an opportunity for the technical representative to understand any technology direction the vendor is planning on that may conflict with the bank's own technology standard.

In contract negotiation be aware if the contract management group requires the vendor to submit periodic performance reports, you may be called upon to review these and attend the meeting.

THIRD PARTY CONTRACT FACILITATORS

Occasionally, a contract may be significant enough to the bank, from a cost and strategy perspective, that a law firm specializing in technology contract negotiations is engaged by the bank to help finalize the contract terms. As a client technology representative, you are invited to the meeting.

In such cases the vendor and the client usually meet at the law firm's premises while the firm walks through the contract terms and facilitates clarification between the attending parties. The vendor may have their own lawyers attending.

The caution here is not to overly trust that the facilitating law firm will have covered all the SLAs needed by this time. There may still be SLA gaps at this time. It's easy to trust experienced contract lawyers but they are not aware of all the services the firm needs. As a technology representative you need to know in advance the services and SLAs you expect from the contract before attending the meeting; be prepared and look for gaps. The law firm will ask pertinent questions and formalize the contract language. They expect the vendor and client to be in agreement by this time. This is the last chance to add or update any technology requirements.

SOFTWARE ESCROW AGREEMENTS

Some contracts will have a provision for a software escrow agreement. The software escrow agreement outlines the responsibilities of all the parties and includes pre-defined release conditions. Generally, these agreements state that if the vendor files for bankruptcy or insolvency, or no longer supports the application, then the bank is entitled to the source code of the vendor's application. A third party firm specializing in escrow "deposits", is involved in securing the latest source code release from the vendor.

These are three party agreements, involving the vendor, the escrow agent and the bank.

The escrow agent will securely retain the latest release of the software provided by the vendor and only release it if any conditions in the escrow agreement are triggered. This implies that every time the vendor has a new software release of their application, the source code is sent to the escrow agent for safekeeping. It's important to have the source code, the documentation and the scripts that build the source into an application.

There is a cost to the bank for maintaining the escrow. Every time the vendor sends source code to the agent, there is a cost to the bank for the agent to maintain that repository on behalf of the bank.

I've seen a couple of cases where escrow agreements have been part of the contract. In one case, the vendor decided to no longer support the product; this was a document management application. Although the bank had access to the source code, it did not have the expertise to re-build the application from source.

If the application is highly critical to the business, then an escrow agreement could be beneficial. The bank must be committed to taking the code and re-creating the application, otherwise it is just another expense. As the business technology representative, should the escrow ever be triggered, you would be responsible for finding the resources who can re-build the application. As well, consider if the application uses technologies (e.g. public domain code libraries) that are not compatible with the bank's technology stack as this could be another barrier.

CHALLENGES WITH VENDORS

There are many cost benefits to outsourcing applications and services to vendors. However, consider that vendors are in the business of generating revenue for their firms by licensing products or selling you more product.

Work Orders

Often our business partners would complain that they were being nickel and dime'd for every small thing they would ask of the vendors; after all the organization was already paying a monthly application maintenance fee, could the work not be performed under that? Keep in mind that the monthly maintenance fee is for vendor application support, it also pays for on-going application development and R&D. Any enhancement the organization needs becomes a work order request.

We had some vendors who gladly did small works at no charge and this was truly appreciated by the business. This was true for our smaller vendors. However, the large vendors charged for every request, no matter how small, the business made of them. Those larger vendors have more formal processes and more expenses for performing work requests. Larger vendors also have a higher hourly rate for any work to be performed. There should be no surprise when a vendor asks for a work request to be submitted.

Typically, hourly rates for any vendor work are specified in the contract. Some vendors may also have a block of free monthly work hours or at preferred pricing built into the contract to accommodate small requests such as custom reports.

Negotiation Challenges

Overall during contract negotiations the contract management representative from vendor management usually takes the initial lead. The bank's business reps are always present and often lead the discussion. However, for technical issues the firm's representatives may defer to the technology person present. Initial negotiations can take many days and go through several bank stakeholder group reviews before it even reaches the contract stage.

Most of the contract negotiations I've attended, both vendor and client argue their points logically and calmly before reaching consensus. Occasionally sticking points are taken off-line and discussed at a more focused meeting. However, sometimes cool heads don't prevail. Contract negotiation discussions may become heated.

I attended one meeting where the bank required a flexible transaction rate; meaning they could demand as many API calls as they needed while retaining the same response time. The vendor pushed back as they required a costly technology uplift to meet this demand. The bank's view was that this was the vendor's problem to solve. This was a business critical vendor. The technology AVP leading the discussion was silenced as the vendor, red in the face from arguing his point, refused to budge. The vendor held their position and the bank had to accept that there would be upper limits on the API call rate.

I've also witnessed the opposite when the bank threatens to walk away from a vendor negotiation. If it's a new application for the bank and there are market alternatives, then the bank has the leverage. When the vendor has a market monopoly, or is already integrated into the bank's application ecosystem, then the leverage is often theirs.

It works both ways. Negotiations can be contentious. Generally, technical people are not good in conflict settings. As part of your role you have to know when to stop the debate and take the contentious issue off-line. Fortunately, most technology representatives are brought into negotiations to consult rather than to negotiate. Skilled negotiations are left to the business and the bank's contract managers.

Contract Length

The length of time the organization signs a contract with a vendor denotes the level of commitment the organization is making to the vendor. A long contract term denotes that the vendor's services are critical to the organization and the organization is committing to support the firm for the long haul.

One advantage to the organization is typically better vendor pricing for long-term contracts. Contracts I've seen typically are 2 to 3 years in length with the option of extending by 1 to 2 years. Very few contracts have longer than a 5 year term. One contract did have a 10 year term but that was for an application that was key to the bank's business; this is not the norm.

For long term contracts, if the organization misses any SLA requirements, then they are stuck with the provisions agreed to for the term of the contract. As an example; we had a case whereby several time critical business reports from a vendor's system were needed by 2:00 PM daily on business days. However, there was no SLA defining that requirement. What resulted was that some days the vendor delivered the reports before 2:00 PM but on days (month-end, quarter-end) where there was additional processing on their systems, the reports could only be delivered by 4:00 PM.

To deliver the reports earlier required software changes at the vendor and an infrastructure upgrade, which the organization would have to pay for. Longer contracts require more scrutiny before signing.

Code Quality

When a vendor provides an application to the bank, the vendor will have tested the application internally to ensure there are no known defects with the code. However, occasionally there are defects which the bank's QA staff discover when testing the vendor's application.

Defects are reported to the vendor. One of two things happens in this case; if the defect is minor, the application is promoted into production and the defect is corrected on the next release. If the defect is major, then the vendor needs to correct it before it goes into production.

When the vendor completes the correction, the bank's QA team once again tests the application to ensure the error is corrected and that no other functionality has been affected by the correction. In effect, the QA team has now tested the application twice before going into production. This has a cost to the bank. When technology support budgets are planned annually, there is funding to cover one QA test cycle per vendor release. Contingency funding supports some releases that could take multiple QA cycles to be production ready.

However, what happens when a vendor releases a product that has so many defects that it requires six QA cycles before it's ready for use. Is the bank expected to assume the increased QA costs because the vendor has done a poor quality job of creating the release code? This specific vendor had a worsening record of code quality ever since they off-shored their application development. The increasing QA costs put a strain on technology budgets as funds had to be re-directed from other applications to sustain the on-going QA effort on this one application.

This was not the only vendor plagued by numerous release errors, which required several QA iterations before being production ready. Often correcting a series of errors results in the vendor adding new errors. This then becomes an increasing frustrating cycle of correcting old errors and fixing new ones.

At contract negotiation we tried to add an SLA on code quality that would impose remedies for application releases requiring more than two QA cycles due to defects, however the vendors pushed back vigorously. I believe contracts should have a code quality SLA with remedies for recovering excessive QA costs that burdens fixed technology budgets. It makes sense, but vendors hate the mention of this. It puts the cost of delivering robust code back on them. With some vendors I felt as if they relied on the bank's QA to find their code defects in order to get the product out faster and minimize their QA effort.

An approach used by one bank with their Microsoft vendor was to capture the costs associated with vendor patch releases. Most patch releases are to correct errors and security gaps with Microsoft's applications. Issues created by the vendor for which the client is encumbered with unplanned costs. There is a cost to the bank to release and patch thousands of desktops and servers. This bank believes the vendor is responsible for these unplanned costs, so at contract negotiation the bank presents the vendor with the costs they incurred to maintain their applications.

I'm sure if contract managers were writing this section it would be a longer dissertation. These are simply some of the challenges I've experienced from a technology perspective.

SUMMARY

Being a vendor to a bank is not easy. There are many demands a bank imposes on their vendors. The level of oversight, security reviews and performance reviews adds increased cost. Larger, established vendors will have higher prices for their services than smaller startups; generally. However, third-party risk remains a major concern for banks as they buy more vendor services rather than build applications and services in-house. Recent high-profile corporate security breaches also places more due diligence and reporting on the vendor by the bank. Contracts are complex legal documents, this section just scratches the surface on the various aspects of contracts that a technology representative should be concerned about.

Reference Links

Below is a list of links relating to the subject discussed above.

• *click here* - 7 myths about SOC 2 compliance (Onetrust.com).
• *click here* - CSAE 3416, Reporting on Controls at a Service Organization (Deloitte).
• *click here* - Office of the Privacy Commissioner of Canada, Investigation into Desjardins' compliance with PIPEDA following a breach of personal information between 2017 and 2019.
• *click here* - Faster Capital: Key Components of A Service Level Agreement.
• *click here* - CBC News - Desjardins settles 2019 data breach class-action lawsuit for up to nearly $201M.
• *click here* - Escrow London - example of a company specializing in software escrow deposits.

Compiled on 05-27-2024 17:39:36