2.1 The Main Principles in Non-binding Guidelines

Companies follow ai principles that typically address some of the main topics of equality and non-discrimination using wording derived from the diversity, inclusion, equality, and non-discrimination context.²⁵ As such, most guidelines issued by the major ai companies (gafam)²⁶ address human rights issues²⁷ but tend to avoid specific issues of gender equality or non-discrimination.

It is good that companies address these issues in their ai principles, but in practice they also need to follow up on those promises to ensure that diverse mindsets of developers lead to less biased algorithm designs. Such general statements in ai company policies resemble marketing statements and should similarly be complemented by measurable targets, such as achieving 40% representation of underrepresented groups. Statistics on the representation of women in ai companies could inform customers of these ai projects whether the priority goals are pursued in hiring and training. In addition, the decision-making process should be made more transparent in order to verify whether the ai lifecycle indeed includes the broad range of backgrounds claimed in the company policy.

On biases Google outlines its position on avoiding biases in its ai Principles under the heading “2. Avoid creating or reinforcing unfair bias.” It states that

ai algorithms and datasets can reflect, reinforce, or reduce unfair biases. We recognize that distinguishing fair from unfair biases is not always simple, and differs across cultures and societies. We will seek to avoid unjust impacts on people, particularly those related to sensitive characteristics such as race, ethnicity, gender, nationality, income, sexual orientation, ability, and political or religious belief.

While it can be acknowledged that efforts to address biases and unjust impacts on humans is welcomed, merely repeating legal requirements is not sufficient in the algorithmic age. Discrimination based on protected characteristics is prohibited in most jurisdictions around the world and should therefore clearly guide the company’s actions. Amazon dedicates one paragraph to biases in its guide on Responsible Machine Learning³⁰ and utilizes a tool called Amazon SageMakerClarify³¹ to detect biases, the company defines biases as “imbalances in data or disparities in the performance of a model across different groups. Amazon SageMaker Clarify helps you mitigate bias by detecting potential bias during data preparation, after model training, and in your deployed model by examining specific attributes.” Rather than merely stating policy goals, such examples present concrete tools that could be used by the company or as elements in a legislative framework to address the issues. Meta also disposes of tools to address biases, stating that Facebook “is continually improving our Fairness Flow tools and processes to help our ml engineers detect certain forms of potential statistical bias in certain types of ai models and labels.”³² These efforts are a notable step in the right direction, as they can serve as tools to achieve the objectives inside and outside legislative frameworks.

On fairness, Google acknowledges that

First, ml models learn from existing data collected from the real world, and so an accurate model may learn or even amplify problematic pre-existing biases in the data based on race, gender, religion or other characteristics. For example, a job-matching system might learn to favor male candidates for ceo interviews, or assume female pronouns when translating words like “nurse” or “babysitter” into Spanish, because that matches historical data.

Discussing concrete examples of biases and potential discriminations is useful as it draws attention to the risks and shows the company’s awareness of the problems. Microsoft, for example, set a goal to minimize stereotyping that states that “Microsoft ai systems that describe, depict, or otherwise represent people, cultures, or society are designed to minimize the potential for stereotyping (…) identified demographic groups, including marginalized groups.” Facebook also states that: “To help us consider these issues from a broad range of perspectives, Facebook’s Responsible Innovation team and Diversity, Equity & Inclusion team both facilitate input from a wide range of external experts and voices from underrepresented communities.”³³ Here again, broad political statements are difficult to verify for consumers or regulatory authorities. Credibility would be added if accompanied by statistics or concrete examples of how the policy goals foster and impact the development of ai systems.

While the ai principles appear to be a marketing product in its own way – the wording is appealing and appears to be addressing a popular issue –, there are still many limits and shortcomings if companies aim to address and effectively diminish algorithmic discrimination. Notably, many company guidelines remain vague and, while they pose general principles or ideas, they do not concretely show how the company attempts to avoid discriminatory outcomes (see Table 5.1).

2.2 Limits and Shortcomings

In contrast to a national or regional legal standard, while most guidelines have many similarities, no single guideline addresses the same issues. This is true not only for the general issues of transparency, accountability, and responsibility, but also with regard to issues of equality and non-discrimination. This creates different situations for consumers using a given service, as each business is free to set up their own particular guidelines which leads to legal uncertainty. Contrary to legal standards, an analysis of several guidelines also reveals the broad and imprecise formulation of some of these standards (see Table 5.1). Finally, while these guidelines can be invoked vis-à-vis the company itself and might eventually give rise to the ability to “litigate” rights within a system set up by the company,³⁴ they do not guarantee not guarantee the kind of gain cause that accompanies a lawsuit in a national court.

2.3 Concrete Guidance Addressed at ai Developers

Some non-binding documents, like the Fairness, Accountability, and Transparency in Machine Learning (fat/ml), can be of added value, if companies use them while developing algorithms. For example, the Principles for Accountable Algorithms try to “Ensure that algorithmic decisions do not create discriminatory or unjust impacts when comparing across different demographics (e.g. race, sex, etc).” And the Social Impact Statement for Algorithms,³⁵ asks

that algorithm creators develop a Social Impact Statement using the above principles as a guiding structure. This statement should be revisited and reassessed (at least) three times during the design and development process: design stage, pre-launch, and post-launch. When the system is launched, the statement should be made public as a form of transparency so that the public has expectations for social impact of the system.

Such non-binding principles can incorporate non-discrimination principles into algorithmic design. Using specific design questions templates is another effective approach to improve companies’ impact statements. The method can be used by designers, developers, and tech companies during the model and design phase.³⁶ It can impact everything from the automated decision (Who is the audience and who will be most affected?), to bias detection (How to test the algorithm? What is the threshold for measuring and correcting biases notably for groups protected against discrimination by law), to diversity considerations (Is the design team representative enough?).³⁷ Such self-regulatory approaches can be used regardless of the existence of any current or future regulatory requirements and to mitigate bias and discrimination.

Specific guidance could take the form of a simple din-A4 page that is handed out to developers: it would recall some of the main risks of biases, stereotypes, and discrimination in relation to design of models, algorithms, and datasets. Even if the knowledge is already present in the developers’ mindset or has been disseminated by training and seminars, it can make a difference to directly address the person in charge of developing the algorithm. Similar to the airline industry where pilots have been thoroughly trained, checklists are used to ensure the application of some fundamental principles and to ensure that key procedures are properly implemented. Such concrete guidance, specifically designed for those who create algorithms, can be made a requirement in legislative frameworks, which could support the reduction of biases and discriminatory potential at the design stage.

3.1 Business Responsibilities to Avoid Human Rights Harm and Discrimination throughout the Lifecycle of ai Systems

Many guidelines, business consultancies, or legal standards contain references and links in their ai principles concerning the role of companies in the protection of human rights.⁴⁵ To avoid human rights violations, many companies actively foresee processes at company level.

The oecd Guidelines and Recommendations call on to respect the principle of non-discrimination and promote equality between women and men. The oecd Guidelines for Multinational Enterprises, for example, explicitly refer to non-discrimination in hiring practices as well as promotion practices (paras. 5), non-discrimination in employment and occupation (para. 51), the principle of non-discrimination, and refer to obligations contained in this regard in the International Labor Organization (“ilo”) Conventions (para. 54).

The oecd/legal/0449, Recommendation of the Council on Artificial Intelligence, which is understood to complement existing oecd instruments, includes recommendations both for governments and for ai actors, including businesses.⁴⁶ The oecd “calls on all ai actors to promote and implement, according to their respective roles, the following Principles for responsible stewardship of trustworthy ai.”⁴⁷

Within the principle of human-centered values and fairness, the oecd states that “ai actors should respect the rule of law, human rights and democratic values, throughout the ai system lifecycle. These include freedom, dignity and autonomy, privacy and data protection, non-discrimination and equality, diversity, fairness, social justice, and internationally recognized labour rights.”⁴⁸ On transparency and explainability, the Recommendation states that “ai Actors should commit to transparency and responsible disclosure regarding ai systems. To this end, they should provide meaningful information, appropriate to the context, and consistent with the state of art.”

Under the principle of robustness, security and safety, it is recommended that “ai actors should ensure traceability, including in relation to datasets, processes and decisions made during the ai system lifecycle, to enable analysis of the ai system’s outcomes and responses to inquiry, appropriate to the context and consistent with the state of art” and that “ai actors should, based on their roles, the context, and their ability to act, apply a systematic risk management approach to each phase of the ai system lifecycle on a continuous basis to address risks related to ai systems, including privacy, digital security, safety and bias.”

Finally, on accountability, it is recommended that “ai actors should be accountable for the proper functioning of ai systems and for the respect of the above principles, based on their roles, the context, and consistent with the state of art.”

Similarly, the Council of Europe originally assigned some obligations to states and others to businesses in the proposed Draft Framework Convention on ai. While Chapter ii and specifically Articles 5–7 addressed public authorities, Article 8 for example specifically addressed private actors.⁵⁰ However, in the final version of the Framework Convention, the scope is restricted to public actors and includes only private actors acting on behalf of public authorities (Art. 3(1)(a)).

While the European Union’s ai Act, adopted by co-legislators in 2024, takes the legal nature of an EU Regulation, it assigns obligations to companies, states and supervisory authorities, but in particular to the developers and users of ai systems that want to use systems within the EU market or where the impacts of the ai system affect the EU market. In that sense, the EU ai Act can be seen as a sort of product regulation which produces compliance requirements and costs for companies.

Major ai companies use tools throughout the ai lifecycle to ensure fairness and/or detect biases in their algorithms or machine learning tools.⁵¹ Other resources used by ai companies include specific guides for ai developers and programmers as well as education and training⁵² to ensure fair and non-biased algorithms. It would be wise for global ai regulators to take note of the existing assessment infrastructure and capabilities among ai companies when they design new rules. They should also work with independent researchers to assess the concrete needs and feasibility of regulatory obligations, for example to detect, diminish, and eliminate biases from the design, training and other datasets in order to reduce discriminatory outcomes.

3.2 The Tools of the Soft Law Instruments and Proposed Regulatory Frameworks to Achieve Non-discriminatory ai Systems

Among the soft law instruments, the tools suggested can be regrouped in three categories based on the amount of involvement required by businesses and potential compliance efforts: information, active and passive obligations, and far-reaching involvement. Information requirements include providing information to ai systems users and publishing relevant information such as reports on bias audits on websites. Active obligations include conducting ex ante assessments, such as bias audits, to check algorithms for potential biased datasets, biased design, or discriminatory impacts. Passive requirements include all regulatory actions by public authorities that check or verify compliance with a given regulation and granting access to businesses for certain activities, like sourcing code or creating datasets.

Among the proposed hard law instruments, several tools are usually proposed to ensure that regulatory goals like bias mitigation and reducing discriminatory impacts of algorithms are met. First, several proposed international frameworks⁵³ suggest bias audits, a tool that can be used either before market entrance or for monitoring the ai system while in use. Typically, the aim of a bias audit is to detect any biases or discriminatory outcomes and correct them before releasing an ai system on the market. Second, known from environmental impact or human rights, algorithmic impact assessments are considered a comprehensive tool used to check for compliance with bias and discrimination requirements. Companies need to provide detailed information to show that they made all required efforts to ensure no or less biases and discrimination in their algorithms.⁵⁴ Third, transparency and documentation tools enable potential victims of discrimination and authorities to verify any violations. Fourth, prohibition is a strong tool used to prohibit the use of an ai system in very specific circumstances in an effort to avoid human rights harms. Fifth, an essential tool to ensure compliance is the possibility to impose fines on companies that develop or use ai systems.

4.1 UN Level

The Report of the Special Representative of the Secretary-General John Ruggie on the Issue of Human Rights and Transnational Corporations and Other Business Enterprises, or the “Guiding Principles on Business and Human Rights” (“ungps”), is a major reference document for business and human rights.⁵⁶ The ungps, adopted by the UN Human Rights Council by Resolution 17/4 on 16 June 2011, are a set of principles directed at governments and businesses that clarify their duties and responsibilities in the context of business operations. For instance, Pillar 2 spells out foundational and operational principles that specify businesses’ responsibility to avoid social impacts wherever they operate and whatever their size or industry, and address any impact that does occur. The ungps rely on three pillars focused on business’ obligation to respect, states’ obligation to protect and the possibility for accountability and remedies.⁵⁷

The operational principles clarify the general human rights commitments and show ways in which these can be implemented at company level, for example through policy statements,⁵⁸ Human Rights Due Diligence,⁵⁹ assessment and identification of actual or future human rights risks,⁶⁰ prevention and mitigation of adverse human rights impacts,⁶¹ monitoring and tracking of human rights response,⁶² and external communication of human rights efforts.⁶³ Considering that the ungps have been designed with both businesses and governments in mind, they incorporate many concrete steps, mechanisms, and procedures to mitigate human rights impacts which are also applicable in the context of algorithmic discrimination. Thus, they can inspire the drafting of national and regional legal frameworks to address algorithmic discrimination. The operational principles are particularly well suited to achieve this end. While policy statements might be less adequate to mitigate human rights harm, human rights due diligence and the assessment and identification of actual or future human rights harm are suitable tools to prevent and mitigate algorithmic discrimination.⁶⁴

The Report of the Special Rapporteur on the rights of persons with disabilities⁶⁵ is one of the rare UN reports that addressed issues of ai and is therefore of relevance in this context. Three aspects are highlighted in the Report’s conclusions that are relevant to the current analysis. First, the “unprecedented power of artificial intelligence [which] can be a force for good for persons with disabilities,” and that the “Profound advances for humankind must be properly harnessed to make sure that the farthest left behind can at last benefit fully from science and its advancements.” Second, it acknowledges that “the well documented negative impacts of artificial intelligence on persons with disabilities need to be openly acknowledged and rectified by states, business, national human rights institutions, civil society and organizations of persons with disabilities working together.” In accordance with the present analysis, it pointed out that

At the development level, those negative impacts arise from poor or unrepresentative data sets that are almost bound to lead to discrimination, a lack of transparency in the technology (making it nearly impossible to reveal a discriminatory impact), a short-circuiting of the obligation of reasonable accommodation, which further disadvantages the disabled person, and a lack of effective remedies. While some solutions will be easy and others less straightforward, a common commitment is needed to work in partnership to get the best from the new technology and avoid the worst.

Third, and finally, the document calls for “a fundamental reset of the debate (…) based on more evidence and greater consideration of the rights and obligations contained in the Convention on the Rights of Persons with Disabilities and other human rights instruments.” The recommendations specifically address businesses and the private sector,⁶⁶ notably in relation to transparency and information obligations,⁶⁷ disability-inclusive human rights impact assessments for ai,⁶⁸ human rights due diligence,⁶⁹ accessible and effective non-judicial remedies and redress for ai caused human rights harms,⁷⁰ and realistic and representative datasets.⁷¹

In the UN Report on Racial discrimination and emerging digital technologies⁷² racial discrimination is analyzed from the perspective of ai. The Report analyzes different forms of racial discrimination in the design and use of emerging digital technologies, including the structural and institutional dimensions of this discrimination and outlines both human rights obligations of states and the responsibility of businesses to combat this discrimination. The report highlights the “monumental influence in the design and use of emerging digital technologies”⁷³ of private corporations but also emphasizes them as “key intermediaries between governments and their nations, with the capacity to significantly transform the situation of human rights.”⁷⁴ It is recalled that human rights law “will by no means be a panacea for the problems identified (…) but it stands to play an important role in identifying and addressing the social harms of artificial intelligence and ensuring accountability for these harms.”⁷⁵ In addition, the report emphasizes that “[t]hese obligations also have implications for non-State actors, such as technology corporations, which in many respects exert more control over these technologies than states do.”⁷⁶ With regard to the diversity crisis discussed earlier (Section 2), the Report mentions states’ obligations to prevent and eliminate racial discrimination in the design and use of ai but stresses that “states must work together with private corporations”⁷⁷ to ensure meaningful diversity and equal representation and suggests including opportunities for “co-design and co-implementation with representatives of racially or ethnically marginalized groups”⁷⁸ when it comes to conducting impact assessments. In this regard, the Report recalls that human rights law is only legally binding on states, who in turn must ensure that effective remedies for racial discrimination attributable to private actors, including corporations, exist in national or regional laws.⁷⁹ The Business and Human Rights in Technology Project (B-Tech Project) is mentioned as an example “which applies the Guiding Principles to digital technologies, due diligence should apply to the conceptualization, design and testing phases of new products – as well as the underlying data sets and algorithms that support them.”⁸⁰

The recommendations notably state that

States must ensure that human rights ethical frameworks for corporations involved in emerging digital technologies are linked with and informed by binding international human rights law obligations, including on equality and non-discrimination. There is a genuine risk that corporations will reference human rights liberally for the public relations benefits of being seen to be ethical, even in the absence of meaningful interventions to operationalize human rights principles. Although references to human rights, and even to equality and non-discrimination, proliferate in corporate governance documents, these references alone do not ensure accountability. Similarly, implementation of the framework of Guiding Principles on Business and Human Rights, including through initiatives such as the B-Tech Project, must incorporate legally binding obligations to prohibit – and provide effective remedies for – racial discrimination.⁸¹

Finally, the report suggests that corporate human rights due diligence needs to be implemented by states based on human rights law prohibitions on racial discrimination and refers to the European Commission’s proposal for mandatory due diligence for companies.⁸³

The two most recent work-streams at the UN level concern the Commission on the Status of Women (“csw”). Each year, the csw adopts agreed conclusions as a result of the 67th session which was dedicated in 2023 mainly to digital and ai topics in the context of women’s rights.⁸⁷ In addition, the current work of the Committee on the elimination of all discriminations against women (“cedaw committee”) regarding the elaboration of a General Recommendation 40⁸⁸ on equal representation in decision making systems is recognizing ai systems as game-changing technology.

4.2 Council of Europe Level

The Council of Europe (“CoE”) is a major force that sees itself as the guardian of human rights and democracy. Equality and non-discrimination have been on its agenda and its core instruments since the beginning. The CoE adopted a Framework Convention on ai and Human Rights, Democracy and the Rule of Law⁸⁹ in May 2024 and is currently working towards a framework for ai regulation that specifically addresses equality, including gender equality, and non-discrimination angle which will be published by 2025.⁹⁰ The clear advantage of CoE legal frameworks on ai are deeply rooted in human rights, enforceable by the European Court of Human Rights in Strasbourg, and has a wide potential reach within the CoE’s 46 Member States. In addition, other states are following the CoE regulatory process on ai carefully as “representatives of the observer states,” such as Canada, Israel, Japan, and the United States of America which all have many ai companies within their jurisdictions.⁹¹

First, the Framework Convention on ai includes rules on the principle of non-discrimination (Art. 10). Previously, Article 12 specifically stated that “Each Party shall (…) ensure that the design, development and application of artificial intelligence systems respect the principle of equality, including gender equality and rights related to discriminated groups and individuals in vulnerable situations.” Now only the Preamble makes reference to “the risks of discrimination in digital contexts, particularly those involving artificial intelligence systems, and their potential effect of creating or aggravating inequalities, including those experienced by women and individuals in vulnerable situations.”

Second, the equality and non-discrimination legal instrument is too far in the future to make specific remarks, but it will be surely based on and incorporate the work of the general framework Convention and consider the work of the EU ai Act.

4.3 European Union Level

The EU is currently the most advanced jurisdiction in the world in terms of addressing adverse effects of ai and algorithms on fundamental rights. Following the EU General Data Protection Regulation (“gdpr”), the EU adopted the Digital Services Act (“dsa”) and the Digital Markets Act (“dma”) and set up the European Center for Algorithmic Transparency (“ecat”), tasked with helping the European Commission enforce the dsa with relevant expert knowledge.⁹² Considering that in principle, all EU legislation is based on a human and fundamental rights approach, EU law can be an interesting reference point because it offers an example of legally binding nature and model character that is often imitated by other jurisdictions.⁹³

The most important EU legislation in the area of ai, the ai Act, was adopted by both co-legislators (European Parliament in March 2024 and Council of the European Union in May 2024). The ai Act is a legally binding instrument for regulating ai on the basis of a fundamental rights approach which foresees several very specific obligations for ai companies designing or using ai systems. In that regard, it differs from the previously discussed soft law instruments because, far from being broad and political, the relevant articles detail the legal requirements imposed on companies.⁹⁴ The ai Act also involves private stakeholders, via the specification of technical standards. It allows for the European Standard Setting Organizations to specify requirements for certain concepts and mechanisms used in the legislative framework of the ai Act.⁹⁵

Specifically, on algorithmic discrimination, the ai Act could make a fundamental contribution to controlling and diminishing discrimination caused by ai systems. While not obvious at first sight, the horizontal proposal on ai regulation could address the issue of discrimination through the regulatory requirements for High-risk ai systems. The scope of High-risk ai systems would include areas such as education or labor market use cases, for example ai recruitment systems.⁹⁶ If an ai recruitment system is classified as High-risk, it would need to comply with the requirements, like mandating a certain level of transparency and documentation. As such, if ai applications fall under the scope of the Directive, they would need to fulfill the detailed requirements for High-risk ai systems. This would enable control of the design and datasets of those ai systems, which in turn could help identify and diminish the risk of discriminatory outcomes. The European Parliament wanted to include the principle of discrimination in the operative part of the legislative text.⁹⁷ This would have been an improvement compared to the original European Commission proposal which merely foresaw the issue of biases and discrimination by ai systems in the recitals.⁹⁸ In addition, the European Parliament suggested strengthening the enforcement mechanisms of the ai Act, notably by including a complaint mechanism (which has been integrated in Art. 85),⁹⁹ upgrading the EU ai Office in the future to the status of an EU Agency (not reflected in the ai Act),¹⁰⁰ foreseeing joint investigations between national and EU authorities (now Art. 74 (11))¹⁰¹ and strengthening the access to the models and algorithms underlying ai systems (now for example included inter alia in Art. 77).¹⁰² These suggestions would help detect algorithmic discrimination and facilitate claims by victims of discrimination. While it is clear that the aim of the EU ai Act is not to address the issue of algorithmic discrimination per se, it can be a useful complement to classical non-discrimination laws and can facilitate detection and enforcement of non-discrimination claims as specified. Because it only partially addresses potential problems of biases and discrimination in relation to the ai applications, there is room to fully address the issue of algorithmic discrimination in any future review of the EU Gender Equality and Non-discrimination Directives.¹⁰³ Together with other future legislative frameworks, such as the proposed EU ai Liability Directive,¹⁰⁴ enforcing non-discrimination claims in the age of ai might be strengthened, notably when it comes to available remedies and compensations for violations of EU non-discrimination rules in conjunction with the liability principles of the future EU ai Liability Directive. A joint legislative framework that includes elements of ex ante regulation of ai systems, reformed non-discrimination laws to consider algorithmic discrimination¹⁰⁵ as well as ai liability frameworks that would be applicable to remedies for non-discriminatory ai systems would enable victims of human rights violations and discriminations to have effective remedies.

5.1 Advantages of Involving Businesses: Expertise

There are several clear advantages to involving businesses in the fight against discrimination from the start, including any positive changes to the conception of models, design of algorithms, and ai systems to their deployment, and the post-market monitoring. The added value that is provided by the ai businesses and their researchers can take the form of pure knowledge (sharing) or amount to a gradual involvement in development and regulation.

First, and foremost, the developers of ai systems that program the underlying algorithms have the knowledge of what views and concepts shaped their design of the algorithm and how it functions. Sharing this knowledge with regulators enables the latter to make informed decisions more easily, considering that some relevant information is only held by the companies. Often, regulators depend on the flow of information from businesses to administrations to enforce the rules. While the state can impose access to information under specific circumstances, voluntary collaboration is preferred and less costly.

Second, including ai businesses from the outset – when algorithms are being designed – could help to incorporate the principle of non-discrimination by design.¹⁰⁸

Third, soft law and hard law proposals often prescribe specific obligations and responsibilities for companies, ranging from transparency, information, and documentation requirements to the obligation to conduct ai impact assessments,¹⁰⁹ audits, or monitoring. In each these scenarios, a close collaboration between the state and businesses is essential. Only if businesses are involved from the start of the legislative process and provide information about the feasibility of specific regulatory ideas, can regulation can be designed appropriately and effectively implemented. While there might be differences of opinion to the extent and depth of regulation, early collaboration can ensure that both actors are aware of the regulatory environment which helps companies adopt the future regulation in terms of compliance and helps states to better enforce their new regulatory framework.

Fourth, regulation can envisage a specific role for business in the development and implementation of regulatory content. Business representatives might be present in oversight bodies or be entrusted with specific roles or responsibilities to help shape the legal rules.¹¹⁰ An example can be found in the proposed EU ai Act, which foresees the specification of some of the content of the future Regulation within the framework of a standardization request entrusted to European Standard setting organizations, the European Committee for Standardization (“cen”) and the European Electrotechnical Committee for Standardization (“cenelec”).¹¹¹ Nevertheless, considering the risk of regulatory capture, the state needs to be aware of potential imbalances of knowledge and technical expertise, which could lead to a situation where states have to blindly follow the analysis or assessment of companies due to a lack of expertise on the administration’s side.¹¹²

While all these various roles and types of involvement can have beneficial impacts for good and effective design and implementation of regulation, there are some important limits that should be considered when involving business in the regulatory process.

5.2 Limits of Involving Businesses in the Regulation of ai: Lack of Legitimacy and Private Interests

First, business decisions are usually steered by private and economic considerations rather than common good considerations. The businesses’ interests might be in conflict with citizens interests’ or the state’s interest. Businesses are not democratically legitimized and accountable in the same way as public bodies.

Second, lack of resources on the side of government institutions might create a temptation to over-rely on business inputs and expertise without being able to understand or duly verify its impacts.

Third, independence can be a problem even if a lot of research is of high quality and standards.¹¹³ In addition, high salaries facilitate talent to be attracted by business rather than the resource-drained public administrations or academia, which further concentrates knowledge and power on ai systems within the industry¹¹⁴ on which often regulators can only rely because they lack the means and the information. Even though academic knowledge can also foster ideas, review, and improve company processes¹¹⁵ to preserve human rights and the principle of non-discrimination, the ultimate aim of each company employee remains to work in the interest of the company.

Fourth, considering that ai companies develop and sell ai systems, one needs to be aware of the interest in avoiding any regulatory obstacles that impede the market for ai systems and go against the business interest of companies.