Understanding the “black box”: big data analytics and artificial intelligence in the banking sector

This article was written by Urszula McCormack, Evan Manolios and Jack Nelson.

By implementing big data analytics and artificial intelligence ("BDAI") applications, businesses can enhance – and even automate – everyday processes and decision-making. 

Given this potential, it is unsurprising that BDAI is already a key component of the banking industry's technology stack, powering everything from chatbots to risk management systems.

BDAI applications also present novel legal questions and challenges. In particular, great care and effort is required to integrate BDAI applications with banking systems while remaining within regulatory expectations.

And critically – what does it take to build a system that will stand the test of regulatory audit?

Recent guidance

Against this backdrop, the Hong Kong Monetary Authority ("HKMA") has released three documents that will shape the banking industry's use of BDAI systems:

• "High-level Principles on Artificial Intelligence", stating 12 principles (each a "Principle" and together the "Principles") that the HKMA expects banks to take into account as part of designing and implementing BDAI applications.  
• "Consumer Protection in respect of Use of Big Data Analytics and Artificial Intelligence by Authorized Institutions" ("Consumer Guidance"), providing specific guidance on consumer protection in the BDAI context.
• "Artificial Intelligence (AI) in Retail Banking" ("Survey"), setting out the high-level results of a survey that the HKMA conducted on the use of BDAI in retail banking, showing that 89% of Hong Kong* banks "have adopted or plan to adopt" BDAI applications.

These documents follow earlier "robo-advisory" guidance provided by the Securities and Futures Commission ("SFC") – the Guidelines on Online Distribution and Advisory Platforms ("Guidelines"), as well as plentiful international guidance, such as the Organisation for Economic Cooperation and Development's artificial intelligence principles.

About this alert

This alert synthesises the key aspects of the HKMA's regulatory guidance on BDAI, together with insights based on our experience. We focus on Hong Kong developments, but the themes can be extrapolated to multiple jurisdictions and sectors. In particular, the Principles largely represent emerging best practice when it comes to BDAI development, deployment and operation.

We cover three key areas:

What are BDAI applications?	BDAI principles and practices	What you need to do

Overall, our key message is that while BDAI applications lend themselves to seemingly impenetrable "black boxes", regulators will not accept this as the status quo. They must be thoroughly understood by internal stakeholders, implemented with appropriate governance oversight, and explainable to customers.   

Numerous other developments are occurring in tandem – for example, the "New Personal Lending Portfolio" ("NPP") initiative of the HKMA is an excellent example of how BDAI may be applied in practice.

Reach out to us if you would like to learn more about integrating BDAI applications in your business.

What are BDAI applications?

BDAI applications are often highly complex, and represent the cutting-edge of computer science. Examples of such applications in the banking sector include:

• chatbots and robo-advisory;
• automated fraud detection; and
• simulations and stress-testing.

Hong Kong is no stranger to BDAI. For example, a number of BDAI applications have also been accepted into the HKMA's Fintech Supervisory Sandbox, which generally covers technology solutions being deployed by Hong Kong licensed banks.

Unsurprisingly, their operation can be difficult to understand, with an all-too-common representation of a BDAI application being a simple "black box":

Many often argue that this is realistically the way it needs to be represented, either because it reflects a bank or technology provider's "secret sauce" or because it is argued that it is impossible to unbundle. More about this in our "All About the Data" alert.

But going forward, the HKMA has made it clear that there can be no "black box" excuses with respect to BDAI. Rather, all relevant stakeholders must have a level of understanding of BDAI that enables them to effectively supervise the use of BDAI applications in their organisations. Training and hands-on interaction with BDAI applications will be an essential part of this, and should form the starting point for a bank's BDAI journey. 

BDAI principles and practices

Overview

The 12 Principles are spread across three categories. These are summarised as follows, with further detail about each category and Principle below.

Category 1 - Governance (and accountability)

Accountability is at the crux of BDAI and is a theme across sectoral guidance and international best practice. As set out in the Consumer Guidance, an "appropriate governance, oversight and accountability framework" must be established and documented. 

The HKMA's commentary on this Principle emphasises that automated decision-making does not absolve a bank's Board and senior management of their general responsibility to supervise the bank's operations. 

Our take

Long gone is the ability to blame an algorithm. There must be (human) accountability for all decision-making, even when that decision-making is undertaken by a BDAI application.

In practice, this requires senior management and Board members to understand how BDAI applications work and the risks inherent in them, and the right people to advise them. 

To this end, we suggest that banks implement the following steps before deploying a BDAI application. Principle 2 also delves into expertise in further detail.

1. Promote knowledge and awareness. Ensure there is a sufficient level of knowledge and awareness of BDAI generally, and the Principles specifically.

2. Allocate senior responsibility. Allocate responsibility at the Board / senior management level for the other two categories of the Principles – that is, Category 2: application design and development and Category 3: ongoing monitoring and maintenance.

3. Implement frameworks. Implement governance, accountability and risk management frameworks and clearly document these frameworks, in line with the Consumer Guidance, and socialise these frameworks among all relevant stakeholders.

Boards and senior management must remain abreast of evolving customer and regulator attitudes to BDAI, which can have significant practical, operational and reputational implications. Responsibility for monitoring changes in regulation and expectations can be delegated, but should feed through relevant management information systems and applied to specific product proposals.

Category 2 – Application design and development

Demand outstrips supply for BDAI talent – as is true for any burgeoning area – making it challenging (and expensive) to attract and retain the right people. However, before implementing any BDAI application, a bank must have appropriate and specific expertise for developing and supervising BDAI applications.

It is therefore unsurprising that, as the Survey finds, the biggest impediment to BDAI adoption among Hong Kong retail banks is lack of appropriately-skilled staff. To this end, the HKMA's commentary states that banks interested in BDAI need "to recruit, train and retain employees with suitable skillsets".

This Principle is also reflected in paragraphs 4.10 and 4.11 of the Guidelines. These paragraphs emphasise the importance of having "adequate staff who have sufficient expertise and understanding of the technology, operations and algorithms (including the rationale, risks and rules behind the algorithms), and who are closely involved in the design, development, deployment and ongoing supervision of the operation of the algorithms".

Our take

Hiring is an option for any bank with a sufficiently deep wallet and good recruiters. However, it is likely that drawing upon existing experts and skilling-up other staff will be at the heart of building a robust talent pool.

A key step to ensuring sufficient expertise exists inside any organisation involves identifying interested personnel internally, and then working out what will be most useful for them to do their jobs.

Accordingly, we suggest that training and testing should expand beyond staff that interact with BDAI applications during their day-to-day work. In particular, those in reporting lines where there are BDAI applications deployed should have particular training.

By way of example only:

• for those tasked with development functions, in-depth technical training and even structured formal courses are likely to be necessary unless they have the right skill set already;
• for those in the "lines of defence" and other risk-related functions such as compliance, legal, audit and risk, the training need not be technically granular, but should still provide a sufficient grounding in relevant technologies and a thorough understanding of legal boundaries, regulatory expectations and emerging risk areas;
• relationship managers need to have a working knowledge of how BDAI applications affect their customers practically, and what needs to change in terms of customer engagement (including any necessary disclosures and choices); and
• the Board and senior managers must have enough competency, information and resources at their disposal to make appropriate business and risk decisions.

The aim should be that a cross-section of the business, from sales to HR to accounting to legal, will have a degree of BDAI fluency.

This will also feed into compliance with Principles 1 and 3.

BDAI applications often suffer from a lack of explainability, due to their technical complexity. But as this Principle and the Consumer Guidance emphasises, banks deploying BDAI applications need to tackle this complexity head on – there must not be any "black box" excuse when it comes to BDAI applications in the banking sector.

However, the level of explainability required to comply with this principle will inevitably differ according to the circumstances. As the HKMA's commentary explains, the level of explainability should be "appropriate and commensurate with the materiality" of the relevant BDAI applications.

The disclosure obligations set out in paragraph 4.2 of the Guidelines are also relevant here. This paragraph requires the disclosure of sufficient information to enable customers to make informed decisions regarding whether to use a particular service.

Our take

Not every BDAI application needs to be explainable all the way down to its source code. Indeed, explaining a BDAI application or its algorithms in terms of its fundamental mathematics is unlikely to be useful.

A useful benchmark is what is currently expected in the banking industry. For example, a financial advisor is expected to be able to provide reasons to support a particular recommendation. Accordingly, a financial advisor that relies on a BDAI application, must be able to explain the data points that the BDAI application relied upon, and how it processed and weighted them, in order to produce a particular output.

In other words, the financial advisor should be able to explain a BDAI application by reference to how it handles and balances inputs to arrive at an output.

Based on our experience, fintechs at the forefront of BDAI take an iterative approach to algorithmic design, which significantly assists explainability.

In a similar vein, taking the time to explain, in plain language terms, how the bank adopts BDAI as part of customer engagement can help ameliorate adverse attitudes to BDAI, and is also relevant to compliance with Principle 8.

A BDAI application is only as good as the data that is used to develop it. Risks relating to data quality are compounded if the BDAI application will "learn" on the job – that is, it will change its behaviour overtime and learn from the data that is inputted into it during operation. Accordingly, this Principle requires banks to use good quality data when developing its BDAI applications.

Our take

All data that is to be relied upon when developing a BDAI application should be thoroughly interrogated. But what is good quality data?  This is a specialist area, but four areas of inquiry are set out below:

Further, if the BDAI application "learns on the job" (ie from inputs provided by customers during normal operations), strict monitoring must in place to understand what exact data the BDAI application is learning from. A particular concern here is "data poisoning" – a type of attack that involves feeding specific data to a BDAI application in order to influence or degrade its performance.

Data also has history. It can have biases. For example, data used in connection with creditworthiness may discriminate against certain groups not because they are inherently less creditworthy, but because of lending patterns and other factors that should be controlled against. Increasingly, regulators (and the public) expect a high degree of probity in relation to potential discrimination.

This Principle mandates that banks conduct rigorous validation before launching a BDAI application, to ensure the "accuracy and appropriateness" of the models that underpin that BDAI application.

As stated in the commentary to the Principles, the HKMA's preference is that banks "involve an independent party (eg the second or third line of defence or an external consultant) in the model validation process."

The Consumer Guidance further provides that BDAI applications require "proper validation … and thereafter ongoing reviews, to ensure the reliability, fairness, accuracy and relevance of the models, data used and the results". In addition, banks must ensure that "[t]he models used for BDAI-driven decisions are robust and have appropriately weighed all relevant variables."

Our take

The Survey shows that banks in Hong Kong are aware of the potential for BDAI bias – the risk of biased decisions is identified as the second biggest BDAI risk for banks. This risk can be met by careful validation, which is essential to not only determine that a BDAI application is functional and reliable, but also that it is functioning fairly.

Testing should be calibrated to catch disparate impacts early, by identifying whether BDAI applications delivers results that differ among groups.

In addition, the HKMA's requirement for rigorous testing means that "edge case" testing is essential. That is, the BDAI application should be able to handle unexpected, rare or incomplete/corrupt data inputs.

As for involving an independent party in the validation process, the appointment of such a party must be made in line with Principle 4 - that is, the party must be appropriately qualified, experienced and skilled to participate in model validation.

BDAI applications require a proper audit trail. Specifically, the HKMA's commentary establishes that banks should be able to:

• "track the outcome of [BDAI] applications on a continuous basis and where necessary gather evidence to support investigations when incidents or unfavourable outcomes arise."

Our take

Comprehensive audit logs and documentation are key to compliance with this Principle. In addition, BDAI applications should be well-documented, including in respect of:

1. datasets that were used in its training and testing (see Principles 4 and 5);
2. any external checks or reviews that the BDAI application has undergone (see Principle 5);
3. monitoring mechanisms (see Principle 9).

A key question is: how long should audit logs and documentation be retained?  The Guidelines provide differing time frames according to the exact processes in questions, but we generally suggest adopting the longest of these time frames – that is, at least a 7-year retention period – across the board.  The precise retention period will also depend on each bank's internal record-keeping policies.

Importantly – relying on staff members' memory is likely to result in very poor outcomes on regulatory audit, investigation or other action. Clear and simple documented explanations prove valuable time and time again.

Due diligence should be performed on vendors, in line with the Principles, prior to appointment and throughout the course of the engagement.

Our take

Banks will be responsible for their vendors' compliance with the Principles. Accordingly, they must be made known to vendors, and consideration should be given to including them in RFQs and other vendor documentation / agreements. Consideration must also be given to the outsourcing requirements set out in SA-2 of the HKMA's Supervisory Policy Manual.

Good contract drafting with vendors can significantly help limit downside financial and reputational risk, but regulatory and criminal liability generally cannot be passed on.

Ethics, fairness and transparency are now well-accepted norms of BDAI development.

At the core of this Principle is ensuring that BDAI applications do not discriminate or show bias against any particular customer or group of customers.

As set out in the Consumer Guidance, banks are also subject to the customer protection principles set out in the Code of Banking Practice, Treat Customers Fairly Charter and other regulatory and industry guidance.

Our take

Principle 8 is not easy to operationalise - there is no bright line test, and compliance with this Principle is invariably multifaceted.

However, a strong compliance framework, clear disclosures, well-documented and reasonable rationales for decisions, and an awareness of the potential for BDAI applications to discriminate and exhibit bias is key. Just as the bank must understand the BDAI application, customers should generally be aware when they are interacting with a BDAI application, or when a BDAI application will be used to make a decision that affects them.

Principles 9 and 10 will also be relevant to compliance with this Principle.

On-going monitoring and maintenance

As an emerging technology, BDAI lives at the cutting edge and must be treated accordingly. Periodic reviews and ongoing monitoring are essential.

Furthermore, the ability of certain BDAI applications to learn means that periodic reviews and ongoing monitoring are especially important. This Principle reflects this concern, with the Guidelines also stating that procedures should be in place to monitor and test the relevant algorithms, including with respect to any advice that they provide. To this end, the Guidelines suggest that regular and random samples be taken and reviewed to ensure that the BDAI application is functioning as intended.

Our take

Staff should be encouraged to provide feedback on BDAI applications, particularly where the output appears to be wrong or incomplete. Further, feedback systems should be easy to use to encourage comments. A system of external review for emerging risk areas and proper arrangements with vendors should also be in place.

BDAI applications are data-intensive. Therefore, privacy law compliance is mandatory. To this end, the HKMA's commentary on this Principle focuses on banks deploying effective data protection measures, and where possible using de-personalised ("sanitised") data.

The Consumer Guidance further states that compliance with the Personal Data (Privacy) Ordinance (Cap. 486 of the Laws of Hong Kong), and its six Data Protection Principles, is essential, as is consideration of the Privacy Commission for Personal Data's ("PCPD") guidance and emerging best practice.

The Consumer Guidance also endorses "privacy by design", "data minimisation" and stresses the importance of informed consent when it comes to use of personal data by BDAI applications.

Our take

Full disclosure and responsible use of personal data is key to fostering trust in a bank's use of BDAI applications, and strict compliance with the PDPO and the available guidance is essential. Fortunately, extensive guidance on privacy by design is provided in a joint publication of the PCPD and its Singaporean counterpart, the Personal Data Protection Commission.

The "no surprise, no harm" philosophy, as well as "privacy by design / default" and "data minimisation" principles are gaining traction across the board. When we speak to clients and regulators alike, these tend to resonate strongly.  

All BDAI development should take this guidance to heart, as it provides strong avenues to reduce the personal data compliance burden and risk post-deployment.  If nothing else, it helps protect against risks still not fully known.

Cybersecurity is key for banks, given the assets and data that they control. The risks are amplified when it comes to BDAI, given that some decisions may be made without human intervention – accordingly, this Principle emphasises the risks in the BDAI-context, requiring vigilance and sound defence measures to manage the risk.

Our take

As we have previously written, cybersecurity is a serious business risk that needs to be managed. Fortunately, there is plentiful guidance available, and the HKMA's Cybersecurity Fortification Initiative is key resource – and framework – through which to implement cybersecurity controls.

"Cyber hygiene" is an increasingly important part of a comprehensive cybersecurity plan, and generally refers to measures taken to control administrative accounts, security patching processes and user authentication requirements – the Monetary Authority of Singapore has provided specific guidance on this topic that is generally applicable in the BDAI context.

Finally, Principle 12 requires appropriate risk mitigation and contingency planning.

No BDAI application is perfect, and hiccups are likely to arise. Accordingly, the HKMA's commentary on this Principle focuses on the need for risk-mitigating controls, including "human-in-the-loop" requirements, such that manual intervention is always possible.

The HKMA's commentary also emphasises the importance of having fall-back processes available should a BDAI application fail or otherwise need to be disengaged.

Our take

A key challenge with BDAI is how to make BDAI applications sufficiently reliable and secure so that both banks and customers can feel confident in their use. To this end, we suggest that strict compliance with the Principles is key to managing the risks that arise in respect of BDAI applications – and that effective contingency planning (in line with the HKMA's business continuity requirements set out in TM-G-2 of the HKMA's Supervisory Policy Manual) will reduce the impact of any BDAI failures. This is essential for gaining staff and customer support for BDAI deployments.

What you need to do

Almost every bank is deploying or at least considering BDAI applications.

The rules that will govern BDAI are still being developed – and those that currently exist will undoubtedly evolve as use-cases expand. Despite the hurdles, industry interest in BDAI applications shows no signs of abating. We recommend the following:

• An understanding at the Board and senior management level regarding the nature and scope of BDAI applications within the bank.
• A targeted compliance framework that specifically addresses BDAI.
• Designated staff within each relevant division who have specific BDAI-related responsibilities, with a steering group or committee.
• A training program with modules for relevant staff, commensurate with their needs and day-to-day functions.
• Engaging with your regulatory contacts – this can serve two purposes: any necessary regulatory disclosures, as well as helping inform regulatory best practice thinking.

*Any reference to "Hong Kong" or "Hong Kong SAR" shall be construed as a reference to "Hong Kong Special Administrative Region of the People's Republic of China".