How to fix Big tech’s diversity problem

After years of big promises and little change, Silicon Valley experienced a tiny breakthrough in raising diversity among its workforce, where women, Black, and Hispanic workers have long been underrepresented. On 12 January, Twitter said that it had boosted the proportion of Black employees at its US locations to 9.4 percent from 6.9 percent in only one year and the share of Hispanic workers to 8.0 percent from 5.5 percent. 

Even if the company hasn't revealed the seniority levels and functional areas where the hiring took place, the numbers attest to substantial changes, especially considering the lack of progress on diversity at other tech companies. How did it pull it off? Can others do the same? And can Twitter do even better? The answer is yes to all.

The reason Twitter could raise these numbers so quickly was summarised by the company's vice president of inclusion, diversity, equity and accessibility, James Loduca. Because of Twitter's switch to allowing flexible work from anywhere, Loduca said, "We were able to hire folks in markets that we know have high populations of Black talent, markets that we know have high populations of Latinx talent." Black candidates were more than twice as likely to accept a job offer from Twitter compared with the previous year, while Hispanic candidates were five times as likely.

If this strategy were used more widely, Twitter's diversity gains could be copied elsewhere. The problem is that firms need to know where to look, as they would be straying far from the familiar recruiting channels and locations close to their headquarters. In this, the tech sector's own innovations could help: Searches assisted by artificial intelligence (AI) can help prioritise the best remote locations for hiring people representing a diversity of demographics.

There are three reasons why this issue is not just a human resources problem. We should care about the persistent underrepresentation of women and minorities for reasons of inclusion and equity alone, but the workforce gap in tech comes with wider societal consequences.

First, we're already familiar with social media algorithms producing biased results, but the problem will only get worse with the increased use of AI. Humans pick the applications and data sets to train algorithms and use their judgement to translate these results into outcomes. When this is done by a narrow cross-section of developers, it raises the likelihood of implicit bias in the outcomes. Given the growing role of algorithms and AI in so many spheres of society, from financial access to health care to personal security, such biases have a wide impact.

Second, there is a growing concern that the technology sector is exacerbating wealth and income inequality. The concentration of tech work in a handful of cities with an outsized role played by just one super-cluster, the San Francisco Bay Area, is a key contributor to such inequalities. Not only are tech professionals drawn from a narrow demographic profile, but the concentration of highly paid tech workers in a few urban clusters make those clusters unaffordable for others. While workforce diversity is no panacea, it certainly helps mitigate growing inequality.

Third, the nature of work itself—and what it means to be employed by a Silicon Valley company—is undergoing some profound changes. The pandemic has made the industry more open to remote and hybrid working. Employers can thus go where potential workers live, not the other way around. Such trends also help deconcentrate the tech workforce away from the traditional supercluster cities.

Twitter seems to have used this opportunity. While its actions are commendable, they need to be replicated at scale, particularly across the larger tech giants. Google was an industry pioneer in publishing its workforce demographics to hold itself publicly accountable, but its track record on the actual numbers has been less than stellar. Even after the controversial firing of a prominent Black female AI-bias researcher put Google in the spotlight, Black women make up only 1.8 percent of Google's workforce. Worse yet, California's civil rights regulator is investigating Google's treatment of Black female employees.

The good news is that Twitter's approach can be amplified. It starts with building a process to identify the best locations to spot talent from specific underrepresented demographic backgrounds. That can be done with the help of AI tools, as we tested at Digital Planet using data from Seek Out, a talent search platform that applies AI tools to search for candidates who fit desired demographic profiles by scanning over 700 million resumes and profiles on the major recruiting platforms used by the tech industry, such as LinkedIn, GitHub and others.

Consider the talent needs for the broad field of AI itself, where the imperative for diversity is particularly high. For one, a diverse AI workforce is the best insurance against implicit bias in AI applications. Moreover, the gap that needs closing in AI is especially wide: for example, only an estimated 17 percent of the global AI talent pool is female, compared to 27 percent of workers in science, technology, engineering and mathematics overall. 

For our analysis, we picked the 60 US cities with the largest pools of AI professionals—people whose resumes note skills in AI, machine learning, deep learning, big data, data science, speech processing or natural language processing—and sorted them by the share of women and minorities as identified by the AI tools.

Similarly, we considered the top non-US cities for AI professionals and sorted them by two markers for diversity: immigrants in the workforce and gender. Of course, these are simple tools and categories that don't shed much light on more complex issues of discrimination, such as bias based on class and social background, or pernicious systems of discrimination like India's caste divisions. Nonetheless, it's a very useful first step to help solve the tech industry's worst problems of underrepresentation.

We found that the San Francisco Bay Area has the third-largest pool of Black AI workers in the United States. But that merely reflects the sheer size of the Bay Area workforce—as a percent of total candidates, the Black AI talent pool was the lowest among all 60 cities. Ideally, a recruiter needs both a large pool to draw from and a high proportion of the underrepresented demographic. The perfect city might be Atlanta, which ranks first out of 60 in terms of the size of the Black AI pool and third in terms of share. Other top cities that rank high on both attributes include Washington, Baltimore, and Houston. In terms of share alone, the Winston-Salem and Greensboro metro area has the best representation of Black AI professionals across all 60 cities.

For Hispanic talent, the top locations in the United States are Miami, Houston and Los Angeles, followed by San Francisco and New York. To recruit female AI talent, the location to target would be Boston, which has a large total AI talent pool and ranks second after Providence in the proportion of women. Other cities to consider would be Washington, San Francisco, New York and Seattle.

Worldwide, the cities to target for AI professionals with immigrant backgrounds—combining both high share and large overall talent pool—are Montreal, Toronto, Vancouver, Melbourne and Sydney. According to our analysis of the talent data, Tel Aviv ranks first globally on gender diversity with just above 25 percent women in its AI talent pool, though the overall size of its AI pool is still small. Edinburgh, Buenos Aires, Mumbai and Melbourne should also be top locations for recruiting female AI professionals. But in general, North American cities led their international peers in terms of gender diversity.

The message is clear: high priority recruiting locations can be mapped more easily using data and analytical power. That should make it easier for tech companies to fulfil their promise to diversify.

2022 will be a year of intense public scrutiny for Big Tech. Unless the industry makes some serious efforts to change, its poor track record on diversity will only compound its problems. Public pronouncements on diversity are little to go by. 

Consider a recent study that found the tech companies making Black Lives Matter pledges or statements in 2020 have 20 percent fewer Black employees than companies that did not make similar pledges and statements. A recent report on gender bias by New View Strategies found that one in three women in tech say they experience workplace bias, with 38 percent planning to quit in two years.

It is time for tech companies to move beyond making virtuous pledges and publishing diversity reports with little real change in the makeup of their workforces. If Twitter can make a change, so can others. Work can be done from anywhere, and workers can be drawn from anywhere. In the meantime, AI tools can help map out the geography of diverse AI talent. The industry's innovations are ready and available to help solve one of the tech industry's deepest problems.

Bhaskar Chakravorti. Illustration: TBS

Bhaskar Chakravorti is the dean of global business at Tufts University's Fletcher School of Law and Diplomacy. He is the founding executive director of Fletcher's Institute for Business in the Global Context, where he established and chairs the Digital Planet research program.

Disclaimer: This article first appeared on Foreign Policy, and is published by special syndication arrangement.