Trabajo 798efcdc87b7

Vista previa (transcripcion.txt)

[00:00:03] I don't see your
[00:00:08] video okay so the recording should be on now um so good morning good afternoon good evening uh everyone thank you so much for joining us for today's Global Network webinar my name is p Yu from the United Nations statistics Division and I
[00:00:29] will be Modera in this webinar together with my colleague Alexander lki so as many of you know this webinar is organized by the global network of data officers and statisticians we use our online discussion group on Yammer to
[00:00:45] share knowledge on various areas of data and statistics if you're not a member already we invite you to join us and we'll share a link to do so in the chat box shortly today we are very delighted to welcome Anders hlum from the boo School
[00:01:02] of Business of the University of Chicago Anders will be presenting on the topic of the adoption of chat gbt so in a recent paper Anders and his co-author investigated the adoption of Chad GPT the icon of generative AI using a large
[00:01:20] scale survey experiment linked to comprehensive registered data in Denmark serving 18,000 workers from 11 exposed occupation they documented that chat GPT is widespread but substantial inequalities have emerged for example
[00:01:39] women are 16 percentage points less likely than men to have used the tool for work furthermore despite its potential to lift workers with less expertise users of chat gbt already earned slightly more before its arrival
[00:01:55] the workers see a substantial productivity potential intivity are often hindered by employer restrictions and the need for training uh Anders is going to kindly cover these and other key findings from the survey
[00:02:11] uh in the next hour or so uh let me also quickly introduce our speakers Anders is an assistant professor of economics and Fuji Mori Mo faculty scholar at the boot School of Business at the University of
[00:02:25] Chicago He studies the impact of technological change on the economy and how educational policies may help workers adapt to new technologies Anders received this PhD in economics from Princeton in 2020 uh just a few housekeeping
[00:02:43] reminders uh Anders we have 35 to 40 minutes to present we will then have a Q&A session following his presentation uh throughout the presentation please feel free to write your comments and questions in the chat
[00:02:58] box at any time and during the Q&A session after the presentation you are encouraged to ask your questions directly by using the raise hand function if you unable to turn on your microphone or camera you could also
[00:03:13] reach out read out your questions to Anders on your behalf just let us know and as usual this webinar is being recorded and will be made available on the global Network after the event we invite you to continue the discussion
[00:03:29] there uh after this webinar and with that out of the way let me now turn over the floor to you Anders uh for your presentation thank you thank you so much for the kind introduction uh let me share my uh screen here and full screen
[00:03:45] um yeah so with that introduction let me just uh jump right into it um this is a a research project that is a joint work with Emilia vestard who's a a brilliant um phsd student at the University of Copenhagen um so the title of the paper
[00:04:01] is the adoption of chat DBT um and um the majority of my talk will focus on that uh first paper this is ongoing work and we are actually now able to um track these workers in uh their labor market outcomes so will at the end be
[00:04:20] able to um study actually the short run effects uh of these adoption decisions do we actually see that workers who have adopted uh generative Ai and a particular chat DBT are doing better uh in the labor market um okay so starting
[00:04:38] with the first uh question of uh the adoption of chat DPT so chat DPT really marks the error of generative AI uh in which uh several high-skilled occupations may be disrupted so in this uh research project
[00:04:54] we ask three fundamental questions so first um who has adopted chat TBT second how do workers anticipate um this tool to impact their jobs and finally why do some workers use chat TBT and others not so to answer
[00:05:15] these three questions um we conducted a large scale uh survey of 11 exposed occupations um in Denmark and then we took these um survey answer and linked them to very uh detailed administrative data uh on
[00:05:35] workers earnings experience demographics allowing us to get a really um nuanced picture of who have embraced this Frontier technology and then in this paper we will examine in particular how workers beliefs and potential adoption barriers
[00:05:53] are ER deciding or determining who has uh used this uh tool and who still not have adopted it thus far okay so how did we collect this data um this was a a um a collaboration with Statistics Denmark where we invited
[00:06:13] 100,000 workers um from November last year to to January this year and these uh 100,000 workers all were employed in 11 exposed occupations and I'll I'll give you the list of occupation in a second um now the why why the Danish
[00:06:36] context it really unlocks two key advantages for this study the first uh key Advantage is that statistics Denmark has very detailed occupational uh codes on every individual worker in Denmark so that uh
[00:06:54] allows us to make this survey very targeted to the workers that we actually think this this tool is relevant for so these could be software developers journalists marketings professionals and so forth we were able to really Target
[00:07:08] uh these these occupations the second key Advantage for this study is that every Dan uh has a so-called digital mailbox that the government uses to um send important informations about their taxes and so
[00:07:24] forth but that statistics Denmark also has the authority to send survey invitations to so that means that we can really get um a very large and representative uh sample of workers in these exposed occupations so I think
[00:07:41] that really is important because um otherwise we may end up in very with very selected samples for example you could imagine that everyone who have signed up for this uh talk are uh have some interest in uh Ai and in chat in
[00:07:59] particular but we want to go beyond the people that are just intrinsically interested in the technology to get a more rep representative and comprehensive picture of how many workers actually using the tools and how
[00:08:12] the workers anticipate this tool to impact jobs so uh we collected a 18,000 uh complete and valid responses and we uh can show and show in the in the paper that this these 18,000 responses represent uh representative
[00:08:33] samples so they are balanced on the co-variates like earnings gender age um and furthermore there's several variables such as workers experience and expertise and and their task that they hold that we both uh measure in the
[00:08:51] survey as well as in the register allowing us to validate that these two U measures line up for the for the variables that um that that we can measure both in the survey and and in the register so just to take stock of
[00:09:04] the data here we really have a very a large scale representative and high quality uh survey linked to comprehensive admin data uh on the on the backgrounds of these workers I think it's a really exciting data set that
[00:09:20] allows us to get a a really unique picture of who have adopted this tool okay so jumping uh right into uh the first key result so here um I plot uh for each of our 11 occupations the share of workers who are aware of um the
[00:09:44] tool who have used the tool who have used it at work and who have used it at a core task where the core task definition is based on um the onet survey that some of you may may know where we asked uh about several uh task in their um in
[00:10:04] their occupations then we ask whether this task was a an important task for them and then whether they were using it for this important core task okay so what does this um uh figure tell you the first takeaway is that chat
[00:10:22] activity is already pervasive in these uh 11 exposed occupations so uh across the board just on average more than half of these workers have used chat TPT at any given point now the adoption rates vary um
[00:10:43] across occupations and in particular you see that it's um it's occupations that are either have writing text as a core task like marketing professionals or journalists or who are into the technology sector themselves and uh
[00:11:00] software uh programmers who have adopted this tool faster in particular in their jobs then at the bottom RS of of this uh of this franking we have workers uh such as accountants financial advisers that are either uh handling
[00:11:18] very confidential data uh financial advisors for example handling your banking statements or where being correct and is is is a core part of the task like for example with a legal uh in legal work meeting legal standards is of
[00:11:36] course uh uh um a necessary condition for for the value added that they're providing so we're going to dive into an understand what are driving these uh differences uh across occupations in a bit so it's uh not all workers who have
[00:11:56] ever used uh chat DBT who are still currently using it so here we ask what are the share of workers who have used chat TBT in the last two weeks so these are active users of this tool and here we see that it's about um about 40% who
[00:12:15] are still actively using the tool and then we can go even further and ask who has an active Plus subscription so this is these are the Plus Account where your paid subscriber to this tool so these these are in in some sense super users
[00:12:32] uh they they're willing to pay uh for for using the tool so we should expect them to also really use it um uh more intensively and we see that among these uh among all workers uh it's only around 7% who are plus
[00:12:50] subscribers so there's a large share of workers who have played around with this tool and who are using it somewhat but is the smaller share who are actually paying
[00:13:05] for the second thing we want to do is that we want to go inside each of these 11 occupations and ask what characterizes workers who have jumped into this new technology so this is what this um regression table shows here so
[00:13:24] in the first five columns I just show Univar correlations within occupations so for example in the first uh column I take um all paralal and then I ask do younger paral leals are they more likely to have
[00:13:44] adopted this tool and then in the last two columns I run a multivariate regression where I'm throwing all the variables in there and seeing in a horse race which are the long EST predictors of the adoption of
[00:14:01] chat TBT and then in the very last column I um I even control not only for workers occupations but for workers uh workplaces and tasks so now in this last column uh we are comparing two workers uh for for example two paralegals
[00:14:23] working in the same legal firm handling the same type of legal task okay so let me walk you through what the key insights are from this table the first two rows shows that it is younger H and less experienced
[00:14:40] workers who have adopted chat TPT uh at work so um for example every year of age or experience is associated with um uh or every year younger is associated with uh around uh 7 percentage points higher
[00:15:03] likelihood of uh having used chat TPT at work so this pattern of selection of younger and less experienced worker makes sense from what we know about this tool we have um good uh evidence from randomized control trials showing that
[00:15:23] it is workers with less experience and expertise who have the most to gain from this tool so for example um if I'm just starting to learn a programming language uh I have much more to to gain and to learn from having access to this Tool uh
[00:15:40] helping me uh with with the guidance into the syntax of of the programming language and we see this showing up that it is the workers with with less experience in their jobs who have Embrace this tool
[00:15:56] faster now despite the fact that uh users of chat DBT are younger less experienced they actually earned slightly more already before chat TBT was even invented um so this shows that um workers who are uh High lower achieving
[00:16:21] workers within their cohort of similar age and experience may need some further um guidance and assistance to help them H adopt this tool so I think this second finding is is a surprising one because um there had been predictions that this
[00:16:40] tool could help alleviate existing inequalities exactly because it helps workers with less expertise and experience but thus far we don't see that showing up in the data in the sense that this third row showing that workers
[00:16:55] who are using the tools are the high achieving workers were um were earning more already before chat TPT was even
[00:17:07] launched now the last row here of this table shows a staggering a gender gap in the adoption of chativ in particular take two workers in the same occupations we take for example two software developers then we compare
[00:17:27] males to females we see that the female software developers are 16 percentage points less likely to uh have used chat TBT as well this is a a really large and substantial gender gap that we had not anticipated
[00:17:45] before um conducting this survey it was a real surprise to us uh but it seems to uh uh now replicate in many other context it seem to be a very general uh phenomena uh uh and we can uh study this gender gap within each of these 11
[00:18:04] occupations and in all of these 11 occupations uh the female workers are substantially less likely uh to have uh use chatti and if we focus in on more intense adoption measures such as whether you're using it for core task uh
[00:18:23] at work these the relative importance of gender only increases so I think this is a a really surprising finding and and I think there's much more uh follow-up work to be done really unpacking what are driving these gender
[00:18:38] differences I'll get into uh it a little bit in in in in this talk okay so this was the F first part of the of the paper which was in some sense a a descriptive study of who have adopted this tool the second thing we
[00:18:55] want to uh ask in this project is how do workers uh anticipate this tool to impact their jobs so now we're asking real journalists or teachers or software de developers how this Tool uh might impact their jobs so the first finding is that
[00:19:16] workers in these exposed occupations see a substantial productivity potential in this toour so the average worker um states that the chat TBT can half their working times in a third of the job tasks so this is really an astounding
[00:19:37] number um I had not anticipated these large magnitudes it it it really shows that um chat D and generative AI is not just hype among Tech expert if you ask uh a representative worker in the in the exposed occupation they do confirm that
[00:19:57] there seems to be uh a potential to improve productivity and save time uh in in the relevant job tasks the second finding is that workers in these um exposed occupations understand that the way that chat DBT uh
[00:20:15] benefits workers is by helping workers with less expertise so we see this showing up in that when workers are asked about well how large are the times they means for workers would greater expertise they say
[00:20:31] that the time savings are are smaller so this is uh I think also remarkable that that workers have this understanding that that the way that chbt is boosting productivity is mainly by substituting for uh expertise in the relevant
[00:20:47] task the third finding uh is that 40% so or 38% so substantial share of the workers who say that can save time with chatti say they will not perform more of the task that chat helps them save time in um so one example uh where this sh is
[00:21:13] very high is for example School teachers um so uh More than 70% of school teachers um say they will not uh perform more of the task that chat sa times in so I think I think this makes sense because uh suppose that I could use chat to help
[00:21:32] me create essays that might actually speed up the the grading um uh but in the short run I will not have more essays to grade so I think this is this is important because it shows that in at least in the short run before firms and
[00:21:49] and sectors have reorganized their work around chat DBT we should maybe not expect these large expansions in in output um so I I think this is uh could partly reflect that this is a short run picture maybe in the longer run as as firm start
[00:22:06] to to reorganize around these tools we will see larger expansions also in output in response to the time savings from this tool now the fourth um finding uh and final finding uh about the beliefs of chat TPT is that workers say they're
[00:22:23] very uncertain uh about these assessments um so there's a a lot of heterogeneity in their beliefs and they they they say they're highly certain about the estimating Time Savings and I think this makes sense uh in the sense
[00:22:38] that just the the large language models that are powering chat gbt are evolving continuously new versions are being launched and also new use cases are being found all the uh all the time I think there's a a good case to be made
[00:22:54] that chat TBT or large language models more generally uh falls into the class of so-called general purpose Technologies um which has a use cases in a vast domain of occupations and sectors okay so the third thing we want
[00:23:16] to do is is to understand well how do these uh optimistic beliefs about chat and its time savings then map into workers's actual use of the tool so let me uh let me walk you through how uh what these charts show so
[00:23:35] here I show on the left I split uh left and right I split workers into whether they have ever used chat TBT or have never used chat TPT so people on the left here have some experience with chat TPT this could be at home it could be at
[00:23:52] work they are uh have some experience using the tools the workers on the on the right here have never used the tool then I uh take all the job tasks of these workers and then in the inner ring I ask what is the share uh of uh the job
[00:24:12] task where workers say uh that chat DBC can deliver large Time Savings and lot timing Time Savings here are defined as um being able to half your working time at equal quality so what does equal quality means uh the
[00:24:33] definition we we used is that if someone reviewed your work they would not be able to assess whether you have completed it with or without assistance from chat TBT okay so this is what the the the the
[00:24:49] inner ring uh Circle shows here then in the outer ring we take each of these job task and ask where the workers are in tending to use chat TPT in in each of these tasks okay so what what does this uh chart shows the first uh finding is that
[00:25:10] workers see large potential Time Savings from this tool so this is what the blue segment shows here regardless of the actual experience with the tools so this shows that there's there's a general sense that uh this this tool might be
[00:25:25] beneficial regardless of whether people have actually used it or not um okay the second finding is that if you uh look at workers who have never use the tool very few of them are actually looking to to to use it in the future so
[00:25:43] for example uh even among the 31% of the job task where these never users are saying that chat DBT could have their working time only around 3% of these job task uh or these workers are intending to use chity going
[00:26:04] forward so uh I think this shows that um sometimes in the in the in the technology diffusion literature we talk about s shapes or S diffusion curves uh I think this what this uh chart on the right shows is that we are beyond the
[00:26:20] inflection point of of this s shape and and and and the speed of adoption for chat D might be slowing in the sense that workers who have not used this tool thus far are not right now looking to start using it anytime
[00:26:37] soon now the the the third uh H and maybe more surprising finding is that even among workers who have used the tool so they understand the technology they're able to log into to chat gbt and and have played around with the prompt
[00:26:55] and they say that chat gbt can um half their working times still more than 60% of these workers say they're not intending to use chat DBT at work so this seems like a big puzzle here uh and er these workers know the technology say
[00:27:16] it could save them time not intending to use it uh it suggests that this must be some something that prevents these workers from from reaping the the full benefit of this tool in that jobs so this is a the next thing we do
[00:27:34] in this survey is we ask them why um why are workers not using chat TBT um if H they say it could save them time so here uh on the on the just a second sorry um yeah um so here on the um on the uh on the next slide we'll
[00:28:04] ask what are preventing workers from using chat TBT even uh when they say or despite the fact that they say that chat TBT could help them save time okay so this is what this plot shows we're zooming in on this gray
[00:28:21] segment here that I highlighted with the red uh red uh lines and asking what a preventing work workers from using chat DBT despite saying that it could save them time in their jobs and then we splitting the answers
[00:28:37] Again by whether workers have ever used chat DBT this is the the red bars or where whether workers have never used the tool so the first finding here um is that a substantial fraction of workers uh say they need some training
[00:28:56] to use the tool this is particularly the case for workers who have never used the tool so for example um all the workers and uh especially also uh female workers say that they need some training before
[00:29:12] they can really leverage and and take advantage of the time savings um uh in in their in their jobs the second finding is that there's a substantial fraction of workers uh who are simply not allowed to use
[00:29:30] chat DPC at work um so we saw this very early on um for example with the financial advisers a lot of banks simply shut down uh the use of chat DBT early on and of course that's a very IL legitimate reasons for not using using
[00:29:50] the Tool uh using the tool at work uh if you're simply your employer are not allowing it and we see that this is particularly the case with workers who have ever used the tool they may have tried it at home but they're still not
[00:30:04] able to really leverage uh Leverage The Tool at work simply due to employe restrictions so I think this shows that some of the key um barriers to um adopting chat at this point uh might be related to um firm policies or at least
[00:30:22] something firms and employers H have uh under their control in the sense that it is employers of course that are directly controlling whether whether workers are allow allowed to use it but employers might
[00:30:35] also be in in a particular uh well uh positioned place to uh facilitate for example employer training of these tools and in some sense it's quite surprising that such a large share of workers say they need training to use chat DBT
[00:30:52] because some of the genius elements of chat DBT is that it is fairly easy to use H is that it it is this large language models but it has it's it's a a chatbot interface um it's very easy to to uh to interact with ER just using natural
[00:31:10] language you don't need any particular Technical Training in terms of programming Lang language to access this tool so I think this this really shows that there might be some lwh hanging fruit for employers just to provide that
[00:31:22] training one day crash course and using chat TBT uh and then helping workers integrate create the tool into that daily work now the second uh and and and and um and final finding from this uh bar uh shows that uh what are the types of
[00:31:43] barriers that didn't seem to be very relevant and here uh we are plotting more existential fears of for example being replaced or being redundant in your job or becoming dependent on this technology um only or or less than 10% of
[00:32:01] workers report that as a reason for not using chap TPT so this really shows that these more existential fears of of uh unemployment or or technology dependencies at this point doesn't really seem to be the main barer for for
[00:32:15] using T activity it's much more these very practical reasons such as Employer restrictions or need for training okay so um in the in the in the last part uh of of the talk here um I will now shift gears H and ask whether these
[00:32:35] adoption decisions that we've uh now uh documented do we actually already see them uh mattering for labor market outcomes like for example we see that women are much less likely to use chat DBT should we already now be worried
[00:32:53] that they're falling behind in the labor market and earning less so I think this is this is is a completely open questions um uh and and my priors were not that strong before uh before diving into this um so let me just give you
[00:33:08] three quick uh reasons why you might expect uh chat DPT to really have mattered for the earnings of workers and why they they may not so um first we already showed and it has been documented or in other service the tat
[00:33:22] DPT is really once in a decade maybe maybe in a lifetime at least Le in a decade uh technology in the sense that it's the fastest diffusing technology ever so this seems to be a really uh a tool that seems to be relevant for a
[00:33:38] large set of workers workers have really jumped into it um second uh as I showed earlier workers uh in the exposed occupations really see a substantial productivity potential of this tool and this uh this
[00:33:54] this finding or these beliefs from workers are backed up by real uh evidence from randomized control trials showing that if you provide access um to chat DPT to for example text writers this is a paper published in in in in
[00:34:11] science or to uh customer support agents this is forthcoming in the qge uh there are also other other papers uh studying consultant it shows that uh access to chat DBT can really help save time without sacrificing on um the quality of
[00:34:33] work and this is especially the case for for workers with L less experience so just from that set of set of uh so these are the three Green Dots here fast diffusion large adoption and R sh uh productivity gain you might expect this
[00:34:49] to to to Really Matter already now for work earnings so three reasons why you might not be skeptical about this is that there's also been been uh been good evidence showing that whether chat DBT is really useful or not for you uh
[00:35:04] depends very much on the task you're using it for so this paper which is um coming out in in in nature um um did the same experiment as the first uh as the first rcts but then they varied the job task um and it particular
[00:35:22] they um gave access to chat TBT for task where chat TBT was not that uh well suited for example at that point downloading a spreadsheet doing some calculations and inferring something from those spreadsheet calculations and
[00:35:36] they showed for those types of task actually having access to chat DBT was detrimental for workers uh productivity so this was an experiment among consultants in the busting Consulting uh group uh and these BCG Consultants
[00:35:51] actually suffered from having access to chat D the second uh two uh uh and and last two reasons why you might be skeptical is that thus far there's really been limited firmwide adoption of captivity it's actually quite remarkable
[00:36:08] that we've seen this fast diffusing uh diffusion of a production technology but it has all been from the bottom up so workers embracing and jumping into this technology whereas workers or whereas employers and firms are actually either
[00:36:22] passive or even regressive regressing uh restricting the Tool uh of restricting the use of this tool at workplace then finally you might just think that the reason why the men are using this tool is that this is the new
[00:36:37] digital gadget and it's not really providing providing U providing U uh any any uh gains uh from this um uh productivity gains for them in their workplace okay so what uh why is this a challenging uh questions to to answer is
[00:36:58] that it's not random uh which workers have adopted CHT in the sense that we already saw that it's for example younger workers uh less te workers with less experience and higher achieving workers uh who have adopted chat
[00:37:16] TPT okay so how are we going to make progress on this so this was a a little bit of a long-winded introduction um what allows us to answer this question and and make progress is that we take now these uh survey
[00:37:32] responses and then we link it to the labor market outcomes of workers after the arrival of chat GPT so now we have data uh on at the individual level on earnings and employment of these workers uh allowing us to see whether they
[00:37:49] workers who have used this tool actually fairing better and then the strategy will uh compare uh adopters versus non-adopters uh before and after chat DB and we also furthermore in order to address this
[00:38:06] selection issue are going to exploit these employer restrictions which in some sense created a natural experiment for us uh in the sense that two legal professionals who are otherwise similar might be employed in in different firms
[00:38:22] that had different policies with respect to the use of chat DBT and in particular the the the pargal that were uh employed in the legal firm where they were not allowed to use chbt is a is a natural control worker who uh were not allowed
[00:38:38] to use the tool despite his uh underlying interest in in using it okay so just starting with these restrictions on use uh we're now asking well what were the firms that were restricting the use of chat TB um and
[00:38:56] here we are again plot in the 11 exposed occupations and then now we're splitting workers based on whether they employed in a large firm so these are firms that are above the median uh size within the
[00:39:09] occupations or smaller firms so the first finding here is that um this ranking is almost the the reverse mirror image of the ranking we saw in terms of adoption so workers who um have or occupations that have used chpt a
[00:39:31] little for example financial advisers or legal professionals are those that are most likely to be restricted in their use from their employers um so again I think this this makes uh makes total sense that these
[00:39:46] these occupations are either handling confidential material um like financial advisors or or or something where being correct is is is crucial but then within occupations we see that the firms that are using uh or have the firms that have
[00:40:03] restricted the use of of the tools are more likely to be larger firms so I think this was a little bit surprising in the sense that usually we think that these larger more productive firms um are are exactly more productive
[00:40:18] because they're embracing new technologies and always at the frontier of the technology curve but with chat DBT we saw the exact opposite and I think the reason is that at least in the short run these larger more profitable
[00:40:30] firms had simply too much to lose from from any scandals coming out from using chat TBT er er or misusing misusing it in terms of uploading for example confidential data for financial advisers okay so um here I just uh show
[00:40:51] um in the first panel or in in the first two two columns here I comp here what are the characteristics of the workers that are either restricted and the difference between restricted versus non-restricted workers and the first
[00:41:06] panel here just shows that there is a first stage effect on their adoption of of chat TBT so workers who are restricted were also less likely to have used the tool at work for job task or even having a Plus subscription the
[00:41:22] second uh panel here shows this is a balance check in some sense showing that workers who um are restricted were on average not that different from workers who are not restricted in their same occupations the only slight B misbalance
[00:41:39] we have here is that they seem to have to um to uh earn a little bit more the workers that are restricted but this is really only coming or fully coming from the fact that they were employed in larger more productive
[00:41:54] firms okay so now turning to the the results here we're asking whether workers who are using chat DBT now have benefited in the labor market from the from the from from the use of this tool so this is a a simple difference in
[00:42:11] means comparing workers in the same occupation age experience and gender category and then asking whether workers who have adopted this tool at work are also earning more and then this uh uh vertical dash line marks the launch of
[00:42:28] chat DBT in uh November uh 22 um okay so what uh what the blue graph shows here is that workers who are using chap DB do indeed earn more and they earn on average six to 7% more in the labor market so the outcome here is lock
[00:42:49] earning so you can interpret the scale here as percentage effects so it seems that just focusing on what workers are earning now uh there could be something to to this tool actually have boosted their their
[00:43:02] earnings but then the key benefits of of of our data is that we have the panel dimension of this admin data so we can follow these workers back in time and then when we do that we get this picture showing that workers who are using chat
[00:43:19] DBT now yes they're earning more but they earned more already before the arrival of chat TPT these are simply just higher aiing workers uh selecting and jumping into this tool and that does not seem to be at least now yet uh a
[00:43:35] clear break that after the the arrival of this tool they're starting to earn more so one way to do that is that we can uh do a difference in difference where we just indexing this difference in means relative to the launch of chat
[00:43:48] DBT and we see that uh chat DBT has really thus far not have a uh significant impact on the earnings of um of of workers we can reject earnings effects that are larger than um like two or three percentage
[00:44:08] points and again we can do the same things so before we were just comparing adopters to non-adopters we can also do this um design where we in instead comparing restricted versus non-restricted workers uh which
[00:44:23] alleviate this selection effect and now we actually see that workers who are non-restricted so more likely to use chbt they they earn slightly less this is coming from this uh selection effect from The Firm side that that uh larger
[00:44:37] firms were more likely to restrict the use but then again if we do a different diff we get the same picture there's not a sense in which workers who were not restricted in the use of chat DBT and therefore better able to to access the
[00:44:51] tool at work are now earning more in the labor market Okay so so uh with with those um with those uh finding let let me let me wrap up and give you the the key uh insights and takeaways from this study
[00:45:08] so the first half of the of the talk was about the adoption of cativity and here we see that charity is already pervasive in the exposed occupations half of uh workers in these occupations have used the tools but
[00:45:24] substantial inequalities in the use of t has already emerged and particular women uh if you just look at whether they've ever used the tool are on average 20 percentage point less likely to have used the tool relative to a worker in
[00:45:41] their same uh occupations we also see that despite the fact that this tool could uh help alleviate existing inequalities actually workers were using the tour right now earned more already before the arrival
[00:45:54] of chat DBT second we see that workers see a substantial uh productivity potential in this tool um state that it could half their working time in a third of the job task but a substantial share of these workers are hindered in the use
[00:46:11] of chat either from employers explicitly restricting the use of chat TPT or them uh stating and reporting that they need some training before they can take advantage of this tool and I think this really puts uh firms uh and corporations
[00:46:28] in a unique position to facilitate the further adoption of chat TT in the sense that workers who are saying that they never tried a chat DBT um and not intending to use it going forward they they really point to to the need for
[00:46:44] training to help them uh overcome that barrier um so I think the next chapter of the adoption curve here will really put firms in a very important um role of of driv the further adoption of chbt and then the last part here which is ongoing
[00:47:02] work studying the effects of these adoption decisions in the labor market we see that at least thus far we shouldn't be too worried about um nonusers falling behind in the sense that chat DBT thus far has have limited
[00:47:17] causal in effect of worker earnings yes chbt users are earning more but that's because they're just the high ability workers who um who have earned who would have earned more regardless it's really reflecting that selection effect and not
[00:47:32] the coration um so to to be clear these are short run effects uh we see that workers say they can save time using the tool so it could be that it just takes some time for those Time Savings to to Really manifest in the earnings but thus
[00:47:47] far we don't see large effects on outcomes okay so with that um let me uh thank you all for for listening and and let me opening up open up for for questions thanks so much thank you Ander for this fascinating presentation first of all
[00:48:07] congratulations on the research and I'm very happy to hear that this is ongoing work and you will update the findings and hopefully ask new questions uh I think a few thing really struck me as you were presenting and first of all
[00:48:23] just from the perspective of data and statistics You' goal here the ability to reach out to so many people and to get a pretty sizable response rate and that the ability to match quite precisely the answers to your survey to administrative
[00:48:41] data I mean that's that's a researcher's dream uh secondly you the the the question about the main question about productivity gain in chat GPT the finding that you know for people who adopted they could have their working
[00:48:57] times and up to a third of the task it's when was the last technological advance that saved people so much time right I mean given let's say a typical eight hour working day this may translate to a saving of up to an hour and a half out
[00:49:12] of that so that's that's significant and lastly I think your finding about the main barrier to adoption being that the need for training and probably training is just as simple as producing a 15minute video on how to create an
[00:49:27] account and how to get started that's also quite quite baffling um with that said I have already enable the camera and microphone and I want to take a couple of questions from the chat first before we we go to uh the live audience
[00:49:45] I see a couple of questions from uh AA and from jav Clan I wonder if you want to just come on online and and ask your question yourselves yes thank you this is AA thank you very much Anders for this beautiful presentation um I'm going to
[00:50:06] read the working paper as well um just a quick question uh based on your presentation since there is a significant gender gap in the use of the two was there an analysis by gender or reasons for not using the twool and if
[00:50:22] this has not been done um will this be something that you explore in your future plans thanks yeah maybe let me take can we take another question and then okay let me just write let me just write it down
[00:50:37] then before sure hello uh thanks this is J um just to give you a minute to yeah I'm ready thank you awesome thank you thank you first of all wonderful presentation and wonderful depth of uh I would say a
[00:50:55] very hot topic that many would like to use it I think many uses it but on a corporate or organization level it's being challenged so my question relies on that how an organization could overcome this obstacles and once they use it their
[00:51:16] product or whatever their content how do they ensure the transparency with the end user who are consuming those uh content I either it fully or partially processed by let's say tools like Chatta and Gemini thank you once again I'll be
[00:51:35] reading your full paper and happy to hear it's an ongoing work thank you so much okay over to you Anders before we take another round of questions okay yeah the these are two two great questions so starting with aa's
[00:51:49] questions on the gender disparities uh so we do a little bit of work uh in the paper trying to dissect what is going on women are more likely to say that they need some training to use the tool they're also more uncertain
[00:52:07] about their assessments so there seems to be some and this is even within occupation so now we are comparing to pargal or something like that even colleagues uh the women ER are more likely to say that they need some
[00:52:22] training to use the Tool uh in order to um to really take advantage uh of it at work um maybe they're more cautious um in in using this tool and that and and I think that could in itself be a good thing uh because we also know that that
[00:52:41] chat DBT is definitely you can go very wrong with with this tool that it hallucinates uh that it's not clear whether you're allowed to use it uh and and women may be more um cautious with that in those aspects I
[00:53:02] want to reference another study which I think is fascinating and gets to some of the some of the other mechanisms which was a um a survey and an experiment done among econ undergrads in Norway where they um gave um a an experiment where
[00:53:21] they gave a a syllabus to these econ undergrads and then they varied whether the syllabus explicitly allowed the use of chat TBT or explicitly uh banned this allowed the use of the tool in this course and they showed
[00:53:40] that these this could almost fully explain the gender gap in the sense that when the um the tool is explicitly allowed both the uh girls and the boys are equally likely to use it uh in the coursework but when it's
[00:53:59] explicitly banned uh women are much more sensitive to that and actually stop using the tool in the courses whereas still a substantial fraction of the of the male counterparts still use the tool despite the fact that it was actually
[00:54:14] not allowed so I think there's something here about women maybe being more cautious maybe for legitimate reasons uh but that's also showing up in them not then not really embracing this this this tool um so I I think to be honest the
[00:54:30] gender gap was a huge surprise for us um we're still trying to to fully understand it uh and I think there are more work to be done uh really really finding out what what's driving it um yeah um so on the organizational
[00:54:46] barriers to and obstacles uh the second questions for for using for using a chat DBT um or or large language models more generally um I I think this is the the main um issue now for the adoption of these of
[00:55:04] these models is to get firms on board and finding out how to uh structure the work productively uh around these tools uh I want to say one thing which is that my sense uh is that the firms that were earliest to restrict the use of chap TPT
[00:55:24] I think are the same firms that are the first to develop their in-house models and these in-house models I think will be able to address many of the challenges ER that that chat DBT imposed in the sense of for example leaking
[00:55:42] confidential data or you could train your model on your in-house data making sure that at least what it recommends and say is consistent with your internal documents and and what you know about the specific case
[00:55:56] one sector where we see this um definitely playing out is the um financial sector that many many banks have now developed their own in-house language model so for example the largest bank in Denmark is called Dan
[00:56:12] bank and they now have a Dan GPT which is basically just chat GPT uh that language model but then all is all inous so nothing is shared with the with open Ai Ai and now Financial advis can actually use this tool when they're
[00:56:29] providing advice to clients so I think I think firms are already making progress and then I think this uh facilitating the uh the training of of uh of um of employers seems to be a lwh hanging fruit uh fruit to to to do uh but you're
[00:56:48] right that that there are still challenges in the sense of how do you Source or how do you reference this uh this uh this tool tool how do we make sure it sources are are valid I think there will be more work even if it can
[00:57:01] save time for uh for workers in their job task there might be more work to be done now uh factchecking the output and making sure that it's uh that it still meets the the quality standards that different um employers and firms want to
[00:57:17] meet in in in supplying their products to their clients thanks a lot Anders for that let me take a let let let me take the next couple of questions I mean since we don't have a lot of time may I ask colleages just uh
[00:57:34] keep your question short uh I see I have a question yeah uh this is sorry I'm not can I can I go through the uh the list first because I see that habba and then Marcelo have their hands up first let me take those
[00:57:49] two questions and then we'll come back to to you uh in the next round Haba uh over to you thank you thank you thank you thank you very much for this the presentation very interesting work and I hope you continue the
[00:58:04] research I have two question very quickly so first you mentioned in the beginning that you send the survey to that government mailbox in in Denmark right so I'm I was just wondering whether that could have affected the
[00:58:19] sample in that people with stronger feelings about this technology so people who are either very excited or disappointed with the technology whether they're more likely to respond whether do you see it in the data secondly what
[00:58:32] I was very surprised to hear that your finding was that people with less expertise benefit more from this technology because my experience was quite the opposite of course I'm just one data point so what I have seen is
[00:58:47] that so my impression is that I can only use chat GPT in an area where I'm an expert because then I can see the gaps the shortcomings the mistakes that chat DPT makes I can fix those and use it at the same time I have seen people with
[00:59:01] less experience make those mistakes with confidence because they don't know those mistakes are there right so that whether that could have been the case because you also talk about certainty in in the
[00:59:15] in the response right so whether you see this variation in certainty so whether more experienced people are less certain than the the less experienced people thank you thanks uh let's take another question
[00:59:28] for barelo before we give the floor to Anders thank you Anders and thank you for the organizers for this excellent event a very quick question Anders I wondered if you were able to track returns to the firm as opposed to the
[00:59:42] worker I was curious whether the firm the lack of a of a result on the returns that you see from a worker level whether there still might be some some additional returns but has been captured at at the profitability of the
[00:59:57] firm level or if if it's a if that's possible from the data set that you have uh thanks for the excellent paper yeah thanks yeah great great um question so starting on the on the sample selection issue so that's that's
[01:00:14] of course a key concern and something we spent a lot of pages on in the paper making sure that this is actually a representative sample that's the whole goal of this exercise is to go beyond selected samples really to get a
[01:00:25] comprehensive picture something that is very nice with the linking it up to the register data is that we can see whether for example male workers or younger workers or less experienced workers are more likely to just fill out the survey
[01:00:41] and we don't see any of those selection selection issues the second thing uh we do so it's representative on so-called observables um the second thing we do to also um address selection effects is that we uh gave people a price so some
[01:00:59] money for um for filling out the survey but we randomized the amount that were that these respondents were given and then we can show that workers who were randomized into a larger price amount were more likely to fill out the
[01:01:14] survey but their their responses were just similar to everyone's else so this I think this is another way to to examine selection into this survey because these workers didn't respond respond because they were interest
[01:01:27] interested in chat TBT per se they only responded because they had a higher price money for doing so and we see that they their responses are just similar to to to to to the other respondents um so I I think of the expertise question
[01:01:42] that's a that's an excellent one and something that is definitely not settled in the literature H we can we there are these RS that I talked about for example the one published in science by noan S showing that for some task for example
[01:01:58] uh text writing uh it is it does level the playing field in a sense that workers with less ER experience have more to gain um from this from this tool but this is in a very controlled setting where we know that chat DPT was great at
[01:02:14] that test test uh text writing task that it was that they were asked to do I think going Beyond these very plain vanilla task it becomes much less obvious whether uh expert versus non-expert workers are more likely to to
[01:02:29] benefit from this tool and I just want to point to that um one experiment that I talked about among Consultants from BCG where they varied this job task and they showed that if it's not a job if it's not a task where chat TBT is just
[01:02:44] like just excels it's actually you you might be worse off from using the tool and that's exactly where expertise might come in is to assess where the chat D is actually valuable whether it's actually providing informative uh informative
[01:03:00] answers and I think that resonates with your experience that it requires your personal expertise to really make that assessment um and then melo's uh melo's questions about the about the firm outcomes um I think that's that's an
[01:03:16] excellent one and something we actually looking into right now it could be that uh even though we don't see um uh work workers earning more at least in the short run from this tool that employers are still able to maybe uh maybe cut
[01:03:34] down on on on employment save some time in terms of the wage bill by now having fewer for example customer support agents on staff handing the same number of of uh of um customer uh requests H and that's certainly something we can
[01:03:52] look in in in the data uh and this is definitely ongoing work that we will add in the coming months or two so yeah great questions thanks a lot Anders I know it' be like six minutes over but are you willing to take maybe
[01:04:07] one stti on yeah all right let let's take one last question before because I think there are still 80 people in this uh call so clearly people are very interested in the topic so we'll take one last question from armor go ahead
[01:04:23] armor thank you very much p thank you Anders for this presentation I'll be very short because of the time constraints I have a couple of questions one of them is about the context as you mentioned Denmark was a very favorable
[01:04:35] context and conducive and facilitating such study has there been any reflection about tools for other contexts what in less favorable context uh not to mention even in kind of developed countries but even in developing countries which would
[01:04:49] be more challenging to conduct such studies my second question is about the earnings at this stage at the early stage there is no kind of impact as you said or causality between the adoption of the Chad GPT and the
[01:05:05] earnings but could we expect such a thing in five years maybe the Achievers would be more Achievers and would earn more and the laggers would continue to be to be laggers so I'll stop here thank you oh these are great great questions
[01:05:18] yeah so on the context I want to say that um of course it's uh it's much harder to do exactly what we did uh in in other context there are surveys starting in the US context there's a recent survey by Blandon big and Deming
[01:05:36] that did a representative survey Among Us workers uh actually replicating many of the patterns uh that we see in this in this um in this survey in the sense High adoption rates especially for chbt and a substantial gender gap and also
[01:05:53] gaps in in younger and less experienced workers more likely to use the tool so there's some evidence there there's um there's uh also um work by Reuters um where they did a also tried to do a representative a smaller sample um so a
[01:06:12] thousand work a thousand workers from I think five uh different uh countries that included um definitely included Argentina uh actually also Denmark us replicating a lot of these patterns but I I totally agree that many of these
[01:06:31] patterns should should uh should definitely try to be replicated and see whether they hold up in other conts and in particular in developing uh uh um context in some sense this tool might have uh larger Potentials in these
[01:06:46] context because it's it's so easy to adopt and it's uh right now uh costlessly so you don't need a large uh Capital uh and and a lot of fundraising to use this tool at work so I would be very excited to to expand these survey
[01:07:02] efforts to um to other countries outside of uh outside of um Denmark and or the US um yeah and then uh on the short versus long run I totally agree with your assessment that right now this is very short run and there might be many
[01:07:19] frictions in the labor market that uh prevents these Time Savings to really manifest into to higher earnings H and I think it's a total open question whether in five years times these adopters uh then starting to diverge and really
[01:07:36] outperform the nonadopters in the labor market and that's why I think we should still be very worried about for example the gender cap that we see in the use of chat DBT in the sense that even though even if if the women are not earning
[01:07:49] less right now due to chat DBT the fact that they're not like uh on the train and and really and really part of leading the adoption curve might later on um really really hurt their their potential uh earnings in the labor
[01:08:08] market down the road in five years time as you say well thank you and I mean for such a such a an interesting topic unfortunately we don't have a lot of time uh we still have a lot of other questions but I hope in the future uh we
[01:08:24] may be able to revisit it again once you have uh basically more research uh on this to share uh but for now I would like to thank you again so much for taking the time uh for those who don't know Anders had a very early start today
[01:08:40] because you in Chicago so probably we started at 7:30 or time in the morning uh so thank you for for taking the time to speak to the global Network and if possible may I ask the participants to just unmute yourself turn on your
[01:08:56] cameras and join me in giving a big round of applause for Anders for speaking to us today thank you so much Anders and best wish for for the research thank you it was a lot of fun thank you thanks thanks
[01:09:12] everyone see you at the next Global Network webinar have a great day bye now thank you goodbye