{ "meta": { "video_id": "ZEH2_YSgOXI", "source": "transcript-api manual/en", "target_language": "none" }, "segments": [ { "start": 10.32, "end": 25.799999999999997, "text": "So thank you so much, everyone, to join this Global Network webinar. And before we start, please note that the camera and the microphone of all attendees are automatically switched off during the presentation. And also during the Q& A session," }, { "start": 25.8, "end": 35.519999999999996, "text": "we will of course give you the opportunity to unmute yourself and Switch on your camera. And welcome to." }, { "start": 45.071, "end": 64.071, "text": "we have with us Matthias, Dave, Rafael, and Benjamin, who will speak about the privacy enhancing technologies and artificial intelligence. And this is the second of our five webinars in our AI for Official Statistics series." }, { "start": 64.951, "end": 87.11099999999999, "text": "And I would like to introduce the topics and also the speakers of today. National statistical offices face growing pressure to share and link data across organizational and jurisdictional boundaries, whilst navigating legal frameworks and maintaining public trust." }, { "start": 87.831, "end": 113.86699999999999, "text": "Privacy Enhancing Technology or PETS PETS offer a way forward, unlocking statistical analysis and collaboration without exposing underlying data. Since 2019, the UN PETS Lab has brought together NSO's researchers and technology providers to test the path in" }, { "start": 113.867, "end": 135.423, "text": "the real world from cross-border federated learning to secure linkage of international trade statistics. At this event, the expert from the UN Pet Lab Community of Practice will present the role of Pets in the context of responsible AI" }, { "start": 135.423, "end": 157.489, "text": "governance, showcase practical examples of use cases and technologies, and explain how to get involved. and access support from the Pat Lab community. Now, let me say a few words about our speakers. So, Matthias is a strategic advisor for data" }, { "start": 157.489, "end": 177.689, "text": "management of Statistics Netherlands. He holds a master degree in computer science from the University of Jupiter. He works in senior leadership positions in statistics. offices of the Republic of Slovenia and Statistics New Zealand," }, { "start": 177.689, "end": 196.607, "text": "as well as a consultancy role for several international organizations. He is an active participant in the International Statistics Community, leading the test team on PATS, part of the UN committee of experts in big data and the data science." }, { "start": 196.607, "end": 211.875, "text": "Dave is a technology policy consultant specializing in privacy technologies and the data governance. He works in policy and the partnerships at the Open Mind Foundation, a nonprofit building open source" }, { "start": 211.875, "end": 236.495, "text": "infrastructure for privacy preserving AI and the co-chairs of the UN Test Lab. Previously, he worked as a technology advisor in UK government and was a software engineer in private sector. Rafael is a senior data scientist with deep expertise in privacy preserving data" }, { "start": 236.495, "end": 260.471, "text": "analysis, spatial data, and rare event modeling. which he applies to support and advanced public administration. He holds an engineering degree from Mines Paris Tech and a PhD in Statistics from Echo Technique Federal de La Seine." }, { "start": 261.591, "end": 282.871, "text": "Since 2021, Rafael has been working at the Swiss FSO, where he supports Swiss public administrations in leveraging data science to drive innovation. And Benjamin is the lead privacy engineer at Stasix Canada." }, { "start": 283.431, "end": 301.511, "text": "Specializing in research and the integration of emerging paths into AI and the machine learning frameworks, Benjamin champions how paths such as federated learning, synthetic data, and secure multi-party computation." }, { "start": 302.151, "end": 321.99899999999997, "text": "act as a critical accelerator for innovation by unlocking insight from sensitive data that cannot be openly shared, maximizing data utility while maintaining public trust. And before I hand over to Matthias, who will also give us an introduction of" }, { "start": 321.999, "end": 336.451, "text": "this webinar, as usual, some practical remarks. You can write your comments and questions into the meeting chat at any time, but during the Q&A session, we would also encourage you to ask your questions yourself. As always," }, { "start": 336.451, "end": 354.119, "text": "this Global Network webinar is being recorded and will be made available on the Global Network of Data Officers and Statisticians on LinkedIn. So if you haven't joined the New Global Network yet, we invite you to follow the link in the chat that I will share later to join the" }, { "start": 354.119, "end": 371.591, "text": "group on LinkedIn. And please also sign up for our newsletter as well, following the other links in the chat to ensure that you get our monthly updates also in the future. So now over to you, Matthias, who will also give us the introduction of" }, { "start": 371.591, "end": 394.621, "text": "this webinar. Thank you so much. Yeah, yeah. Thank you, Yuxi, for a nice introduction. And also thank UNSD, Alexander, to inviting us to this exciting AI series of webinars. So, yeah, as was mentioned, I have a privilege of chairing the UN" }, { "start": 394.621, "end": 411.73199999999997, "text": "Task team on privacy enhancing technologies. And we have been now for seven years already investigating the role and application of privacy technologies in official statistics. And we have published the Handbook of" }, { "start": 411.732, "end": 431.79299999999995, "text": "Privacy Preserving Technology, as they were called at the time, as well as Pet Guide in 2023. And all our work and outputs are available here in the link that I just shared. But during this investigation, we realized that probably the statistical" }, { "start": 431.793, "end": 451.08299999999997, "text": "organizations really need help and support in terms of practical testing of these techniques on their own use cases. So back in 2022, we established what we call UN Pet Lab Community of Practice, which is actually the network of members" }, { "start": 451.083, "end": 472.323, "text": "from statistical organizations, from academia, so academic researchers, as well as from private organizations and foundations that are actually working on privacy enhancing technologies. And that gave us some sort of more practical focus." }, { "start": 472.323, "end": 491.71299999999997, "text": "And my colleagues today will present, yeah, both insights from this research as well as for some practical projects. So yeah, with that, I would just like I would like to pass to to the first presenter, Dave Buckley," }, { "start": 491.713, "end": 506.631, "text": "who will explain how pets what pets are and and how what is their role in the responsible AI governance. Thank you. Thanks, Francesca. Just give me one moment to share my screen." }, { "start": 512.111, "end": 527.551, "text": "Hopefully, that's not coming through. Looks good. I see nods. Great. Thank you. Yes, as Matthias mentioned, so I co-chair the UN Pet Lab, also worked for the non-profit Open Mind Foundation, who actually were a co-founder of the Pet" }, { "start": 527.551, "end": 539.982, "text": "Lab back in 2019. Yeah, hearing Matthias say seven years was quite scary, but here we are. Yeah, hopefully we're going to talk a little bit about some of the work we've done in that time and some of the work we" }, { "start": 539.982, "end": 552.046, "text": "continue to do through both the Pet Lab and the wider task team that Matthias talked about. So I'm going to try and set a bit of, yeah, maybe more of a high level context of sort of what we mean when we talk about pets and how it might fit in around" }, { "start": 552.046, "end": 564.649, "text": "themes of sort of AI governance. And I guess to start, I would sort of slightly provocatively perhaps say that AI governance is data governance, or at least I think at least you would say data governance is a big part of AI" }, { "start": 564.649, "end": 580.442, "text": "governance, right? So a model is a compression of its data. A lot of the things you want to govern when it comes to AI, so sort of from algorithmic bias, to privacy, to provenance, to consent, to security and accountability, kind of resolves into a question about" }, { "start": 580.442, "end": 594.996, "text": "data. So whose data, you know, where did this data originate from? How was it gathered? Under what terms? With what controls? And this thing kind of exists along you know, a whole life cycle of the whole AI life" }, { "start": 594.996, "end": 609.3919999999999, "text": "cycle, really. So be that where we're referring to pre-training data, data that's used for fine-tuning, sort of context that we bring in, the inference time when using a model, or even like the models themselves, right? So like model weights, you know," }, { "start": 609.392, "end": 624.694, "text": "are just kind of an array of numbers, which could be considered data as well. Um... And I guess kind of at high levels, I was going to talk about some challenges with traditional data governance. So, you know, we," }, { "start": 624.694, "end": 639.06, "text": "as people in the statistical community, will know more than most. Yeah, like disclosing data in a secure and private way and safe way is a challenge. We see sort of pseudonymization and anonymization techniques that might" }, { "start": 639.06, "end": 649.9159999999999, "text": "traditionally been used can fail to meaningfully protect privacy. And you know, this tends to sort of cut through to the public domain when something really bad happens, like a big data leak. And yeah," }, { "start": 649.916, "end": 657.0709999999999, "text": "you can imagine there's been a whole series of such incidents over the past sort of. Yeah, 20, 30, 40 years." }, { "start": 658.951, "end": 666.031, "text": "There's also this idea that if you using using pseudonymization techniques can, you know, can reduce the utility of data, right, if you're applying." }, { "start": 668.311, "end": 681.8789999999999, "text": "Yeah, the identification techniques to try and mask information or redact information that kind of necessarily reduces the utility of that data. So, you have this sort of concept of this of this privacy versus utility trade-off. Yeah," }, { "start": 681.879, "end": 696.114, "text": "perhaps with these more traditional techniques, it can be quite hard to sort of get both privacy and and whilst preserving utility. Even if we could perfectly anonymize data, we can't necessarily prevent misuse when sharing data." }, { "start": 696.114, "end": 709.1510000000001, "text": "So I might share some data with another organization for an express purpose. But as soon as I've shared a copy of that data, it's hard to sort of, in a technical way, ensure and enforce that what that data" }, { "start": 709.191, "end": 724.725, "text": "was shared for, the purposes that it was shared for, is what it ultimately is used for, and that it's not further shared with other organisations or used for different use cases. So we end up relying on sort of legal frameworks and policy-based governance" }, { "start": 724.725, "end": 739.351, "text": "measures, which are very much necessary. But I would say a sort of perhaps insufficient. And as I was just alluding to, can be very difficult to enforce. So we kind of end up with this world where data sharing initiatives can fail due to this sort of combination of" }, { "start": 739.351, "end": 752.492, "text": "privacy, security, intellectual property, costs, legal reasons. In an ideal world, we need solutions that can address some of these concerns by design. So yeah, you're getting this see this image a lot when people give like high level pet presentations," }, { "start": 752.492, "end": 762.55, "text": "but can we sort of move the privacy utility curve upwards so we can have this sort of have our cake and eat it, so to speak, where we can have sort of good levels of data privacy at this without having to" }, { "start": 762.55, "end": 776.4319999999999, "text": "sacrifice utility of the data sets. And this is where, when we talk about privacy enhancing technologies, This is kind of the shared property that we're typically referring to, this ability to enable us to extract utility from sensitive information whilst" }, { "start": 776.432, "end": 790.3109999999999, "text": "preserving privacy. Which yeah, leads us on to what are pets. So which I, yeah, I guess I'm just restating that here. So this is kind of a broad collection of technologies that share this property of being able to shift the trade-off between privacy and utility of data." }, { "start": 791.071, "end": 805.535, "text": "And what this can enable in principle is that different researchers, different organisations, different individuals can sort of collaborate on their data, do collaborative data science without having to see the other organisations' private data assets. And again," }, { "start": 805.535, "end": 817.954, "text": "I'm not going to get into the specifics of any technologies here, But these are, and again, this also isn't like a definitive list, but these are typically the sort of types of technologies and specific technologies" }, { "start": 817.954, "end": 829.3109999999999, "text": "that get talked about. I will just mention, so this distinction between input privacy and output privacy. So this is, I think we, I think the Pet Lab, when we pub, well, the Pets test team published a UN Pet Guide." }, { "start": 830.471, "end": 846.526, "text": "I want to say in 2022, that could be wrong. But I think this introduces this concept of input privacy and output privacy in how we categorize pets. So the idea that there are pets that present privacy at the input layer, i.e. the data that's going into a system," }, { "start": 846.526, "end": 858.196, "text": "and pets that protect the output layer. So that sort of data coming out of the systems. This is things like differential privacy, and you'd also include in there sort of some of the statistical disclosure mechanisms that national statistics" }, { "start": 858.196, "end": 873.1519999999999, "text": "offices would traditionally use as well. So yeah, why that's for official statistics? Again, I think I'm sort of preaching to people who know much more about this than me, but national statistics offices have a mandate to set the public interest by" }, { "start": 873.152, "end": 881.0709999999999, "text": "protecting the data of the citizens that they represent, but also maximizing the public interest benefits that can be derived from that data." }, { "start": 882.991, "end": 893.641, "text": "So sort of the theory here is that with privacy enhancing technologies, this can provide like new avenues for enabling useful data to be more accessible for research and innovation. So you can sort of, in theory," }, { "start": 893.641, "end": 903.631, "text": "we can actually sort of do more to maximize the public benefits from that data while still maintaining like really, really high standards of protecting privacy and the autonomy of citizens." }, { "start": 906.271, "end": 909.711, "text": "So, the role of privacy and has technologies and data governance, so..." }, { "start": 911.591, "end": 915.7909999999999, "text": "This QR code I think links, it will link to a page that I think then links to these papers." }, { "start": 918.511, "end": 934.269, "text": "But yeah, essentially last year we published, so this was a few authors who are collaborating through the UN Pat Lab. We published 4 papers as part of a special edition of the Statistical Journal of the IAOS on privacy and housing technologies," }, { "start": 934.269, "end": 947.841, "text": "and a couple of these focused on data governance. So this sort of first paper, which is quite a brief introductory paper, is explicitly focused on the topic of the role of pets in data governance and linking to some of the high level" }, { "start": 947.841, "end": 958.591, "text": "frameworks that we see through that have come out of the UN, through like the Global Digital Compact and other sort of global international organisations who are interested in data governance as well." }, { "start": 959.471, "end": 972.186, "text": "A second paper kind of goes a level deeper on the role of pets and official statistics and goes into sort of trying to provide a taxonomy of use cases for privacy and housing technologies and talks to the different stakeholders who" }, { "start": 972.186, "end": 985.727, "text": "are involved in doing data projects at national statistics offices and how pets can potentially help to like shift the incentives of those stakeholders in order to make it sort of more likely that a valuable data" }, { "start": 985.727, "end": 1004.506, "text": "collaboration projects can proceed. And that all the work in all the kind of, yeah, all the work in that paper came about through sort of stakeholder consultation through the UN Pet Lab. So through the various privacy tech startups, foundations like Open Mind," }, { "start": 1004.506, "end": 1017.002, "text": "and I think most importantly, representatives and national statistics officers who are exploring these technologies. And yeah, I probably don't have time to go into detail on any of these papers in this" }, { "start": 1017.002, "end": 1030.446, "text": "short session, but I'm happy to follow up with anybody if if of interest. There's also a third paper, which was led by our colleague, Zhiyuan Kim from Fraser Nash University in Canada, which conducts A Adelphi survey that's" }, { "start": 1030.446, "end": 1044.946, "text": "conducted with statisticians from national statistics offices to better understand the role of pets in supporting risk assessment frameworks. with a particular focus on the five states framework. And a final paper," }, { "start": 1044.946, "end": 1060.6709999999998, "text": "which is from Ben, who's going to present to us momentarily, and colleagues at Statistics Canada and ISTAT in Italy, which describes A hands-on pilot that they've done to measure trade asymmetries in a privacy preserving manner." }, { "start": 1061.471, "end": 1063.391, "text": "using secure enclaves." }, { "start": 1065.391, "end": 1072.8310000000001, "text": "So yeah, do check out that journal if you haven't already. I'll just touch briefly on this first paper, given I only have a short amount of time." }, { "start": 1075.231, "end": 1087.6599999999999, "text": "We basically have this idea that this is talking about responsible data sharing, and we frame this as responsible data sharing is kind of finding this optimal balance between maximizing the societal value of data and needing to protect the" }, { "start": 1087.66, "end": 1102.372, "text": "privacy, security, and safety of citizens, which I was sort of describing before, and argue that sort of pets can support this by better navigating this trade-off between privacy and utility. and also providing more precise, granular control over sort of which" }, { "start": 1102.372, "end": 1112.111, "text": "queries and which computations are permitted to run on a certain data asset. This is going beyond just sort of protecting the privacy of data, but also" }, { "start": 1114.591, "end": 1131.065, "text": "enabling it to be governed in a much more granular way and providing data owners like insights into how their data is being used. And in principle, and you'll have to take this with a pitch of salt because I know I know you're describing this at quite an abstract" }, { "start": 1131.065, "end": 1144.009, "text": "level, but taking these two things together, I think this can be like potentially like quite powerful mechanisms for more robust and effective data governance, and therefore more robust and effective AI governance. And as I was mentioning, we kind of linked to the Global Digital" }, { "start": 1144.009, "end": 1160.103, "text": "Compact, which has sort of relevant ideas. on data governance. And I think that's a most relevant privacy enhancing technologies is this objective 4, which has a particular, so is all about advancing responsible, equitable and interoperable data" }, { "start": 1160.103, "end": 1172.5819999999999, "text": "governance approaches and has a dedicated section on data privacy and security. So yeah, we've been engaging and working with the folks within the UN community who are sort of working on operationalizing the" }, { "start": 1172.582, "end": 1182.871, "text": "Global Digital Compact to understand how our community of practice can help support its goals. And hopefully this webinar can sort of work towards that aim as well. Just very quickly, just to wrap up," }, { "start": 1182.871, "end": 1196.299, "text": "I guess I just wanted to give some high level examples of the types of use cases that sort of supported AI governance could enable. And this will then hopefully segue nicely into the subsequent presentations, which will be a little bit more, yeah," }, { "start": 1196.299, "end": 1200.1109999999999, "text": "hands-on in terms of showing the capabilities of these technologies." }, { "start": 1202.591, "end": 1215.267, "text": "But yeah, super high level. So one example, which maybe this might be the most familiar to people, is this idea of doing federated learning. So can I train a model on distributed federated data sets where the training data doesn't have to leave the" }, { "start": 1215.267, "end": 1230.695, "text": "institution from which it originates? I think Ben is going to talk a little bit about a project we've been exploring there through the UN Pet Lab. There's also things at inference time. So again, we seem to be moving to this world where a lot more of the information flow" }, { "start": 1230.695, "end": 1247.541, "text": "involved with models is actually at inference time, a lot more people using sort of retrieval systems. So again, can there be ways of doing sort of federated retrieval where you can sort of leverage leverage data from different organisations at inference time whilst" }, { "start": 1247.541, "end": 1262.511, "text": "retaining and maintaining, I suppose, data sovereignty. And there's also approaches around sort of third party evaluations of models. So again, AI evaluations is, I think, is like a big and growing ecosystem." }, { "start": 1262.991, "end": 1277.859, "text": "And this idea of sort of third party evaluations to be able to verify the claims made about AI systems is again an interesting mechanism where we've seen privacy enhancing technologies employed to protect the privacy and intellectual" }, { "start": 1277.859, "end": 1295.071, "text": "property of sort of models as well as evaluation data sets. that are used in those processes. So that's kind of a bit of a flavour of what's possible. I will pause there and I suppose pass to pass back to Matthias to maybe lead us into the to the next session." }, { "start": 1296.991, "end": 1310.031, "text": "Yeah, thank you. Thank you, Dave, for your insight. And yeah, I'll just pass then to Rafael, who will explain a bit more how pets can mitigate certain confidentiality and privacy risks." }, { "start": 1312.671, "end": 1317.951, "text": "Thank you very much, Matthias. So do you hear me well and on the screen, the slide full screen?" }, { "start": 1319.791, "end": 1337.142, "text": "Getrude. Yeah. Yeah, all good. Perfect. So thank you so much for the invitation. It's really a good pleasure to be here today. So the idea is to give you some idea on basically how privacy enhancing technology can help mitigate what we call the confidentiality risk," }, { "start": 1337.142, "end": 1353.966, "text": "and especially in artificial intelligence. So just let me start by setting up the scene to give you an idea of how and where privacy and handling technologies can help protecting this confidentiality. And let me first by describing what we" }, { "start": 1353.966, "end": 1369.585, "text": "usually consider as being a standard workflow in AI and machine learning in general. So usually, When you would like to use this model, the very first step is actually to start preparing your data. It's a step that is also known as feature" }, { "start": 1369.585, "end": 1381.442, "text": "engineering, where basically you try to manipulate and transform your data such that it can be leveraged as well as possible by your AI or machine learning model. The second step is actually to combine" }, { "start": 1381.442, "end": 1396.787, "text": "the data that you prepared at the first step with this AIML model, so it's usually a mathematical representation of some specific combination of parameters. And during the training step, the idea is to find the best parameter" }, { "start": 1396.787, "end": 1411.773, "text": "that correspond to the data that you have been preparing, so this input data. Once you have been successful in combining those those two components. So that basically means that you have estimate, you add value for the parameters of your" }, { "start": 1411.773, "end": 1425.109, "text": "model. Then obviously, having a trained model is not a goal in itself. What you would like is to make this model available to a potential user. So this is step 3, where you actually serve the trained model through these estimated parameters" }, { "start": 1425.109, "end": 1439.971, "text": "and give the opportunity to a user to bring his or her own data, input data, and then give it to the model such that in the fourth step, actually, they can manage to obtain a prediction out of the model. So along this whole workflow," }, { "start": 1439.971, "end": 1454.337, "text": "we see that we have multiple stakeholders. So the very first one is the data owner, and he is responsible in general to protect the data. And when he or she wants to actually use this data to start the training process," }, { "start": 1454.337, "end": 1468.231, "text": "it will request some computing power that is usually provided by an IT provider. That can be internal to the organization, but it can also be external of the organization. And finally, once you have the trained model," }, { "start": 1468.751, "end": 1480.167, "text": "you will make it available to users, and those users can also be within your organization or outside your organization. So you have multiple stakeholders, multiple steps, and the idea is try to see how privacy" }, { "start": 1480.167, "end": 1492.183, "text": "enhancing technology can actually help preserving the confidentiality of the input data throughout this process and for all the different stakeholders. But first, let me start to explain actually why I" }, { "start": 1492.183, "end": 1507.471, "text": "was speaking about confidentiality so far and not directly on privacy. The reason is that confidentiality is actually something that is requested by the Principle 5 of the National Code of Good Practice for Official Statistics." }, { "start": 1508.431, "end": 1525.981, "text": "And it has a specificity that confidentiality is a property of data. So it's basically assessing if data that we have collected at the National Statistical Office remain secret. So it's objective. It's something that we can measure and that we can quantify." }, { "start": 1525.981, "end": 1542.8639999999998, "text": "It's basically under the question, is this data secret? Yes or no. And to be able to maintain the status of the data, hopefully in the secret status, there's two sides to confidentiality. The first one is security, which basically revolves around" }, { "start": 1542.864, "end": 1562.963, "text": "preventing any kind of unauthorized access to the data. So this is more traditionally the role that is given to IT security. And the other side of confidentiality is disclosure control, which is basically disclosing the risks that when we actually make willingly" }, { "start": 1562.963, "end": 1576.625, "text": "available the result of a computation. So for instance, a machine learning or AI artificial AI algorithm that we would have trained with the data, We would like to limit the risk that this data product," }, { "start": 1576.625, "end": 1591.711, "text": "so the result of this computation, might disclose the data that has been used in inputs. So really enforcing confidentiality is providing solution for both of these aspects and privacy enhancing technologies are here to protect those two aspects" }, { "start": 1592.031, "end": 1604.298, "text": "of confidentiality. So going back to the standard workflow for machine learning and AI, the very first natural step to be able to protect confidentiality of the data is protect data directly at the very first" }, { "start": 1604.298, "end": 1618.031, "text": "step, so at the data preparation site. Because if confidentiality is protected at step one, then we know that the data will be protected against all of the stakeholders throughout the whole process and the whole workflow for" }, { "start": 1618.031, "end": 1632.591, "text": "machine learning. So here, two main examples of privacy enhancing technology that could be used in this setup are synthetic data and local differential privacy. Synthetic data is very simple. The idea is to randomly generate" }, { "start": 1632.991, "end": 1650.0349999999999, "text": "data sets that mimic the statistical properties and the structure of the original data set. Local differential privacy is similar, but has a little slightly different strategy, where here, instead of randomly generating new data, we'll take the original data and we will" }, { "start": 1650.035, "end": 1667.3619999999999, "text": "partially randomize this input data to make it unreliable to be used at the individual level. So what does that mean in practice? So for instance, if you have a categorical variable with two possibilities, yes or no, we will replace the true answer 50% of" }, { "start": 1667.362, "end": 1683.511, "text": "the time by the true answer. And 50% of the time, we will also replace the answer by just the cost, the toss of a coin, which take 50% of the time the answer yes, and 50% of the time the answer no. So really the idea here is that with this" }, { "start": 1683.511, "end": 1696.111, "text": "kind of technologies of pets, we can ensure that the data, if it were accessed by someone who has no rights over the data, would not be able to use the data against the individual because it's not reliable." }, { "start": 1697.071, "end": 1712.872, "text": "The second strength of the two technologies is that it's really, in general, easy to set up and also easy to communicate in terms on how they can offer protection with respect and confidentiality. Of course, there are some challenges." }, { "start": 1712.872, "end": 1729.152, "text": "The first one is that, for instance, with synthetic data, the sector and the methodology is today very, very wide. And the privacy guarantees that this jungle of provider of synthetic data generators offers widely differ today." }, { "start": 1729.152, "end": 1744.392, "text": "The second challenge is that Dave mentioned this privacy utility trade-off is when we start applying the protection So early into the chain, into the workflow and machine learning and AI, it's extremely difficult to actually" }, { "start": 1744.392, "end": 1757.489, "text": "assess the utility, the reliability, and marginally the impact of the protection on the model that you are training on the protected data. So this is why very naturally the idea is not to enforce the protection on the data" }, { "start": 1757.489, "end": 1771.559, "text": "preparation, but at the training step. And so here comes into play potentially other kinds of privacy enhancing technologies. The first one is known as confidential computing. So here the idea is to offer some" }, { "start": 1771.559, "end": 1789.663, "text": "security protection against the IT provider by offering a computing environment that is completely isolated from outside of the world. So the idea is that nobody can access and can read the data that is fed for the computation, except the data owner." }, { "start": 1789.663, "end": 1801.995, "text": "And this also includes the data provider. So for instance, if you do not trust your IT provider, so if you go to public cloud services, then with confidential compute, you have the guarantee that even this" }, { "start": 1801.995, "end": 1816.696, "text": "provider cannot access the data. Differential privacy will offer other kind of guarantees. It will offer protection against the risk that the trained model might disclose the input data. How does it work? It's fairly simple. So here," }, { "start": 1816.696, "end": 1831.991, "text": "instead of partially randomizing the data, we will partially randomize the training algorithm itself, such that we will limit the risk that the algorithm will disclose some of the input data. at the later stage. So the strengths of this combination of" }, { "start": 1831.991, "end": 1844.393, "text": "pets in this particular context, where we actually have a solution that offers input privacy and output privacy, so confidentiality, both on the security with confidential computes and disclosure control with" }, { "start": 1844.393, "end": 1862.031, "text": "differential privacy, they offer improved model performance. So improved level of utility. The challenge in applying this technique is that those are required usually very strong technical expertise, especially expertise that you need to have internally of your organization." }, { "start": 1863.391, "end": 1878.584, "text": "Finally, as you can guess, it's also possible to protect the model at serving time, so whenever you make it available to the public. So here, how does it work? It's again possible to use differential privacy, where here, instead of partially randomizing the" }, { "start": 1878.584, "end": 1893.0629999999999, "text": "training process, you will partially randomize the model's parameter such that it will limit the risk that this trained model will disclose the input data. One of the strengths to this kind of approach is that here you can apply the" }, { "start": 1893.063, "end": 1910.351, "text": "workflow exactly as you would have been applying without privacy confidentiality protection measures. So it simplify everything. And also this as the consequence that you can have a very fine control on the impact of this privacy protection on the performance of your model." }, { "start": 1911.231, "end": 1926.298, "text": "The challenge is naturally always the same, because here you're still applying very advanced technique, namely differential privacy, for which you require internally very strong technical expertise. Secondly, in terms of challenges," }, { "start": 1926.298, "end": 1942.154, "text": "applying differential privacy in this context is not accessible and possible for all kind of algorithms that exist. So only a restricted number of models is available. Now, with this workflow, I have described to you how privacy" }, { "start": 1942.154, "end": 1961.096, "text": "enhancing technology is an example of privacy technology can help protecting confidentiality throughout this process of training and serving a machine learning algorithm. Now, there's plenty of use cases where actually we would like to be able to use" }, { "start": 1961.096, "end": 1978.4479999999999, "text": "multiple sources of data between multiple data owners such that we can train a machine learning algorithm and an AI algorithm that on the mutual different sources. And so for that, also kind of privacy enhancing technology can be used," }, { "start": 1978.448, "end": 1994.047, "text": "in particular federated learning where you train locally some model and then you can pull together those parameters such that you obtain in the end one federated model that relies on all the data sources and benefit from all the" }, { "start": 1994.047, "end": 2007.8709999999999, "text": "multiple sources while preserving confidentiality. But here, I'm not going to speak more about it now because I leave the floor to my colleague Ben, who's going to get into the detail of the federated learning approaches." }, { "start": 2008.831, "end": 2024.913, "text": "Looking forward to the Q& A if you have any questions. Otherwise, I leave the floor to Ben. Thank you very much for your attention. Yeah, thank you, Rafael, for this explanation. And now we go to the next presentation, which is showing practical projects and" }, { "start": 2024.913, "end": 2028.351, "text": "practical examples. Yeah, Ben, feel free to start." }, { "start": 2034.671, "end": 2051.795, "text": "Okay, thank you. Can you see my screen? Yes. Yeah, OK. Thanks, Rafael, and hello everyone. Thank you very much for this opportunity. I'm a lead privacy engineer at Statistics Canada, and I've been working on the Petland" }, { "start": 2051.795, "end": 2061.311, "text": "since 2021. And on behalf of the UN Petland, I would like to present some of Examples and projects on. that we work on." }, { "start": 2064.591, "end": 2083.3109999999997, "text": "Okay, the first one will be on federated learning. So we had a federated learning work stream. This started on the high level group for modernization of official statistics from the UNICEF. And then, because all the team was on the Ped Lab," }, { "start": 2083.791, "end": 2104.351, "text": "So, we, we naturally, we move these work stream to the to the human path lab. So, so what is federated learning? So, federated learning allows you to train models collaboratively while you're protecting the data that is locally stored in in several, I would say." }, { "start": 2104.511, "end": 2119.861, "text": "nodes or workers that are shown here on the image. So, ferated learning, so you start like with local data, you train these local models, and then you can send it to a central authority that can aggregate those models." }, { "start": 2119.861, "end": 2138.135, "text": "In this case, we are talking about model weights. And then this updated model is sent back to everyone. that they can continue the training. So you have some sort of continual learning during this process. Also, further learning is not orthogonal to" }, { "start": 2138.135, "end": 2153.806, "text": "other privacy and hazard technologies. As Rafael exposed a little bit about, explained a little bit about the differential privacy, and so we could use differential privacy during the training of those models locally." }, { "start": 2153.806, "end": 2168.391, "text": "And then we can do the aggregation after. Also, we can use homomorphic encryption, which is a cryptographic technique that allows you to compute mathematical operations on encrypted data. So in this case, you can send" }, { "start": 2169.631, "end": 2177.871, "text": "encrypted models, we showed you in the next slides, you can send encrypted models and you can aggregate them. So." }, { "start": 2180.031, "end": 2198.794, "text": "The first thing that we we we start exploring was cross-silo horizontal federated learning. So, in this case, multiple NSOs hold or held at different separate data sets, but they were like uniform in the in the" }, { "start": 2198.794, "end": 2216.111, "text": "schema. So, we have... every NSL have the same features. In this case, the data set was related to human activity, like walking, sitting, standing, or laying. And the application was to train an activity recognition model on this data. So" }, { "start": 2217.831, "end": 2231.999, "text": "What this allows us to train these local models, each and so will train the local model and then send it back to the aggregator. For the tech stack, we use Flower, which is a popular learning library," }, { "start": 2231.999, "end": 2252.533, "text": "and Opacus, because we apply it after differential privacy to the training and homomorphic encryption to. due the secure aggregation of the models. So as an example here, I'm showing you how it looks like the case study repository for the UN Pet Labs." }, { "start": 2252.533, "end": 2273.1099999999997, "text": "So this was the case number 15 for the UN Pet Guide. And if you can scan this QR code if you want to read more about it. In this image at the right, what I want to show you is like each and so they held a data set. We do some kind of local training and" }, { "start": 2273.11, "end": 2290.041, "text": "then we send encrypted models to the aggregator. So in this case, the aggregator, they don't own the private key. So it was a way to protect. the model so the aggregator blindly aggregate these models and then send it" }, { "start": 2290.041, "end": 2292.5910000000003, "text": "back to each of the NSO. Sorry." }, { "start": 2294.551, "end": 2310.481, "text": "Then we went to the phase two. This is published here. You can scan the QR code if you want. And what we added to the project was to add noise during the training, right? So for that, we use differential privacy," }, { "start": 2310.481, "end": 2326.48, "text": "which on a high level, you're just injecting noise during the training. And you have a parameter that is called the privacy budget or epsilon, and that the control that that can give you control on the amount of noise," }, { "start": 2326.48, "end": 2347.802, "text": "so the higher the value epsilon you have, you have a better performance, but the lower epsilon you have, you you can achieve. higher privacy. So it's like a compromise to choose your epsilon to see how your task perform. The other thing that we wanted to study" }, { "start": 2347.802, "end": 2366.071, "text": "was the aggregation strategy. So there are different ways to add those models. So we tested. And, as you can see at the bottom, we we tested different strategies like Fed AVG, Fed AdaGrasse, Fed Adam, and Fed Yogi," }, { "start": 2366.071, "end": 2382.47, "text": "and we compare with different values of epsilon. So, nothing new here. What we saw is like for the higher epsilon values. we match more or less the performance of the models when we're training without" }, { "start": 2382.47, "end": 2383.951, "text": "differential privacy." }, { "start": 2386.591, "end": 2406.2309999999998, "text": "Then, now we are working on the on the phase three. This is in progress. So, one of the use cases here is cross-silo horizontal for farm management survey. So, the difference here is that we are considering that multiple..." }, { "start": 2406.671, "end": 2424.016, "text": "agencies or NSOs. They hold separate data sets with the same features, but this is related more on a national or regional survey. And the idea is to predict best management practices with a strong privacy guarantee." }, { "start": 2424.016, "end": 2444.518, "text": "So we will try to train a model in a ferated way. And, and for the tech stack, we're using here Civ Box, which is an open source developed by OpenMind, integrated with Flower, and this will be deployed on the UN global platform," }, { "start": 2444.518, "end": 2448.111, "text": "and it may be in a source infrastructure." }, { "start": 2451.311, "end": 2467.417, "text": "And so, so the the flow of of this particular work stream is is is this, so we we start with the micro data, the farm management survey. This is from from a StatCan survey, and then we we are we are right now we" }, { "start": 2467.417, "end": 2481.316, "text": "are training a baseline model for predicting the the best management practices implementation level, and once we got this baseline, we will generate synthetic data and we we we generate this synthetic data with with" }, { "start": 2481.316, "end": 2500.663, "text": "privacy guarantees, so we we will we will train it with differential privacy. And the idea is for this synthetic data that we have here, we will try to tweak it to perform as good as the baseline when you don't use" }, { "start": 2500.663, "end": 2523.038, "text": "any any any privacy any differential privacy. So, We will do some privacy assessment to ensure that this synthetic data is safe to share. And once we do that, we will split this synthetic data to different NSOs or nodes that will be distributed on the UN global platform or" }, { "start": 2523.038, "end": 2543.99, "text": "maybe on-prem. And, at the end, we will use this synthetic data to to train the learning model. So, the thing here is, we are using AI, I mean, a generated model to help us to test the technology, but in a real world use case, we will jump directly from the from the" }, { "start": 2543.99, "end": 2547.311, "text": "micro data to define the learning phase." }, { "start": 2550.991, "end": 2571.631, "text": "So another thing that we can do instead of splitting the data, which uniform or homogeneous features across the NSOs or the nodes, will be to split the data in terms of the features. So in this case, it could be like to have different departments, national departments, or" }, { "start": 2572.031, "end": 2591.2309999999998, "text": "or agencies that holds different features for the same subjects. Let's say for a particular farm, an agency has, let's say an NSO has some kind of demographics and another agency has more details about the management of the farm. So the idea will be to" }, { "start": 2591.551, "end": 2611.18, "text": "The data vertically, and then again repeat all the experiment to do the federated learning phase. So, for that, we are considering the same software stack, and I didn't say before, but we plan also to use Opacus because this is a differential privacy library" }, { "start": 2611.18, "end": 2615.7909999999997, "text": "that... That can allow you to inject the noise during the training." }, { "start": 2618.991, "end": 2643.277, "text": "Finally, this is another study I think Dave mentioned. One of the papers I will give you, the link is here on the QR code. So what we did was a private linkage inside of an enclave. In this case, the enclave was provided by Oblivious." }, { "start": 2643.277, "end": 2664.007, "text": "And we had two inputs, one from Statistics Canada and one from CBS. So this was mock data. This was not real data, but the idea is that each one of the NSO wants to protect their inputs. So we will do all the operations inside this secure enclave that protects the" }, { "start": 2664.007, "end": 2682.549, "text": "data from anyone that wants to look at it. So neither StatCan, neither CBS can look what's happening inside here, but they agree on some common base code to be run against the data. Right, so there was some linkage that was" }, { "start": 2682.549, "end": 2699.551, "text": "performed using the Python record linkage toolkit, some error estimation that was computed there, and then at the end you got like a link data set. So it was used for two different purposes. One was to do regression and the output." }, { "start": 2699.631, "end": 2717.173, "text": "that we got from here is just the estimated coefficients. And the second one was to use, to generate synthetic data in a privacy way. So we generate the synthetic data and then we can, we get as an output a public data set" }, { "start": 2717.173, "end": 2739.158, "text": "that can be used. Um... So finally, I would like to, let me see if I can share this. I would like to share some links about the, about the UN Pet Lab, so how we can help. So we have, we collect these case studies as the ones" }, { "start": 2739.158, "end": 2759.254, "text": "I just presented. These are listed in a in a in a web page. Also, we publish the pet guide in 2023, but we are working on a new and a new and an updated UN pet guide, and... Also, the UN Global Platform is supporting us" }, { "start": 2759.254, "end": 2776.3509999999997, "text": "for the practical experimentation. As I said, we want to deploy these nodes on the UN Global Platform. And also, we have some events and technical presentations that are periodically given in the community. So feel free to look at those links." }, { "start": 2776.511, "end": 2778.111, "text": "And, and thank you very much." }, { "start": 2780.911, "end": 2789.631, "text": "Yeah, thank you, Benjamin, for this presentation. So yeah, now passing back to Yuxi for the Q&A." }, { "start": 2793.191, "end": 2811.246, "text": "Yep, thank you so much, Matthias, and also thank you so much, Ben, for sharing these practical examples from Pat project, and I think this way is very useful to bring together the themes that we have discussed today, and now we will move to the Q&" }, { "start": 2811.246, "end": 2832.831, "text": "A session, and for... participant, please feel free to raise your hand to ask your question. And I just enable you guys to open your microphone and camera. So please, when asking your question, you may also indicate if this is directly to a specific speaker." }, { "start": 2833.231, "end": 2842.751, "text": "or whether it is open to the panel. So, please. So, I see Amar, please go ahead, Taimur." }, { "start": 2845.431, "end": 2858.979, "text": "Thank you, Yuxi, and thanks all. I'm Amr Nour. I'm the Chief of Detail Innovation and Capacity Branch, actually the organizer of this series. And I really don't have a specific question, but I do have first. I wanted to thank all of you for this" }, { "start": 2858.979, "end": 2870.946, "text": "wonderful presentation. I mean, really, really the series has been and the sequence has been great. All right, so thanks to all of you and it also shows the fantastic work that has been done within the UNCEBD." }, { "start": 2870.946, "end": 2882.564, "text": "We've met probably with some of you there and I really, really want to put on record my appreciation and the appreciation of my branch for all the work and UNLD for all the work that you've been doing as part" }, { "start": 2882.564, "end": 2898.479, "text": "of the PEPs. task team in the UNCVD. Now, one of the things that came to my mind is obviously it all comes, I think there are two things, as Dave said at the beginning, it's kind of AI governance is or within" }, { "start": 2898.479, "end": 2914.9069999999997, "text": "the pets is part of the overall data governance. And I think really, that's important. And I think really, that's an important point that I would like to reemphasize again, because we do have a working group on data governance, all right," }, { "start": 2914.907, "end": 2928.176, "text": "which is mandated by the UN Statistical Commission. And they are actually exploring all right, how AI can fit within the broader data governance ecosystem as well. So I think you are providing an insight" }, { "start": 2928.176, "end": 2943.034, "text": "and a valuable insight for this that I would like to put on record and will certainly tap into you as well as in as part of this project or the work of the working group on data governance to situate the AI governance within the" }, { "start": 2943.034, "end": 2959.8709999999996, "text": "broader ecosystem of the data governance. So thanks for that. The second thing is this balance ultimately between the utility of the data and the previous. All right. And I think Rafael has just made this point very, very nicely. All right." }, { "start": 2959.871, "end": 2975.589, "text": "And I'm just wondering, as we move forward with the learning and the practical case uses, do we have better insights about this balance, all right, or how we can alter or improve this balance based on the nature of the data" }, { "start": 2975.589, "end": 2990.11, "text": "itself, the nature of the input data, the more, whether it's more sensitive or less sensitive and we can yield on the utility, for instance, Do we have any insights on that already in terms of measuring this balance" }, { "start": 2990.11, "end": 3000.591, "text": "somehow from the case usage that you had experienced? So I'll stop here and really, really thank you again. You've been a fantastic presentation. Thank you." }, { "start": 3005.151, "end": 3022.031, "text": "Yeah, thank you, Amr, for these nice words. And of course, yeah, we are working already and we will continue working on the data governance with the data governance group. Probably your question, maybe Rafael can say a bit about the measuring the balance." }, { "start": 3023.311, "end": 3042.215, "text": "Sure, with pleasure. So I can speak about our experience as the Swiss Federal Statistical Office. So we've been running some pilots with using differential privacy. So we really advocate the use of differential privacy because it enables us to have a transparent and mathematical" }, { "start": 3042.215, "end": 3062.3199999999997, "text": "quantification of this privacy utility trade-off. Now, the challenges are always the same when we're trying to balance is how much risk is enough and what is it okay or not. So we don't have a good answer to that. And it's potentially the most challenging" }, { "start": 3062.32, "end": 3078.545, "text": "to answer if we take the question in this direction. Actually, our solution was to reverse, because we usually think about balancing this trade-off by starting by the privacy and then going to utility. What we did," }, { "start": 3078.545, "end": 3094.122, "text": "and we are not the first one, actually we got inspiration from the US Census Bureau, is to start from utility and then see what was the maximal protection we could offer. while preserving the utility. So for instance," }, { "start": 3094.122, "end": 3110.511, "text": "instead of thinking about this is this production that I want to give, and this implies this utility, we discuss with researchers and ask, okay, what is it the question that you are trying to solve? This is the numbers that you need to generate." }, { "start": 3110.911, "end": 3125.627, "text": "And this is the maximal noise that we can inject such that you can still draw a reliable conclusion for your research. So our solution for now is basically to apply this utility first principle such that we can ensure utility and offer the" }, { "start": 3125.627, "end": 3131.791, "text": "maximal protection in terms of privacy. Hope it helps in a way. Give insight on how we could help. Gods." }, { "start": 3135.911, "end": 3136.511, "text": "Thank you." }, { "start": 3139.471, "end": 3155.951, "text": "Okay, thank you so much, and uh... Anyone want to ask any further question regarding the presentation today and also the topic? And so far I don't see any question being posted in the chat box." }, { "start": 3158.191, "end": 3162.2709999999997, "text": "I see the hand from Omar. Please go ahead, Omar." }, { "start": 3173.071, "end": 3179.151, "text": "Omar, I'm not sure if you are speaking, but... From my side, I cannot hear you." }, { "start": 3181.871, "end": 3182.1910000000003, "text": "Nope." }, { "start": 3184.191, "end": 3194.271, "text": "It looks like your microphone is open, it's on, but... We couldn't hear you. Maybe, Omar, you can put it in the chat, the question, because we cannot hear you. Yeah." }, { "start": 3196.635, "end": 3204.111, "text": "In the meantime, if somebody else wants to ask a question, I think that would be a good opportunity." }, { "start": 3216.191, "end": 3216.7509999999997, "text": "And..." }, { "start": 3237.951, "end": 3258.703, "text": "Yeah, I can see. Maria, Maria F posted very interesting the work everyone is doing. Indeed, I feel like the work presented today is very interesting and I have also feeling that the presentation is also fruitful" }, { "start": 3258.703, "end": 3276.791, "text": "and I guess all the participants are more or less satisfied with the content. Yeah. So there's a last question, which is also a good sign for us. And, uh, I, ohh, uh, Omar, I see your hands up. Please go ahead." }, { "start": 3276.791, "end": 3283.591, "text": "Yeah, well, I mean, I'm trying to really benefit the maximum from our presenters, honestly, here, and since I didn't want to come back Yeah." }, { "start": 3286.251, "end": 3300.312, "text": "again, but since there is no other questions, I just want to ask, okay, you apply this pets for the NSOs. How do you go? Do you also... replicate them in other statistical systems at your own country, you know," }, { "start": 3300.312, "end": 3314.1110000000003, "text": "like within the Ministry of Agriculture, within the Ministry of Trade, within etc. How do you go about that? I'm just thinking also about the capacity building component of this kind of pets at the national level as well. Thank you." }, { "start": 3316.751, "end": 3331.988, "text": "Yeah, yeah, Ben, I thought you would be good to answer that, or at least begin. Yeah, thank you. Yeah, no, no, very interesting question. So in my opinion, given that data that is split across country," }, { "start": 3331.988, "end": 3349.14, "text": "it won't be as uniform as having their own data for your own country. I believe that the potential in pets, it's more, at least at this point, is more collaboration between agencies, having data with the same for the same" }, { "start": 3349.14, "end": 3363.8309999999997, "text": "population, but for different features, kind of the last thing I was talking about, the vertical learning setting. So I believe that, I mean, what I'm trying to do is just to build these use cases and" }, { "start": 3364.111, "end": 3379.0589999999997, "text": "And show to to the agencies and show internally and externally to to you know to to get interest from people outside StatCan and and other organizations, so so they can see the potential, because if we can do this kind of if we" }, { "start": 3379.059, "end": 3392.007, "text": "can use these paths to you know to to get insights for this data that is. locked right now, we gain A lot. Even if we have some kind of agreements between agencies, but they sometimes they take a long time," }, { "start": 3392.007, "end": 3408.589, "text": "you can do something something quicker, something faster using pets and they allow you to, you know, to do to do more. So at this point, we are still experimenting with the technology. but we are pretty sure that it will work. And just to going back to your first" }, { "start": 3408.589, "end": 3422.738, "text": "question also about the utility and privacy balance, also, I would say that is very case dependent. So the stronger and cleaner data that you have, you will have more, you know, a better quality outputs and you can" }, { "start": 3422.738, "end": 3428.031, "text": "apply, you know, privacy measures better. Thanks." }, { "start": 3432.351, "end": 3449.821, "text": "Yeah, I know I felt. Yes, I can follow up on the question of Omar and then answer maybe Omar's one. So, for us, the main use case for which we leverage privacy enhancing technology is secondary use of data and especially for research. And so," }, { "start": 3449.821, "end": 3466.234, "text": "this is a use case that is not only relevant for us as National Statistical Office, but for whatever office that might collect some data that is potentially sensitive or needs protection for confidentiality, and that can be used or could be relevant" }, { "start": 3466.234, "end": 3481.26, "text": "for researchers in Switzerland. So the way we try to disseminate these good practices is that we are a center for innovation for the federal administration. And so everything that we do, we do it such that we provide tools that" }, { "start": 3481.26, "end": 3495.7110000000002, "text": "can be used and reused by others. So we do everything. We do, we develop software that is open source, that is hopefully portable and deployable on standard infrastructures such that we facilitate their task." }, { "start": 3495.751, "end": 3510.654, "text": "by for adopting this kind of technology outside National Statistical Office. And so for the question of Omar, who was actually wondering if we have applied this kind of technique for our national integrated data system," }, { "start": 3510.654, "end": 3524.181, "text": "I would like to say yes, but actually, as I mentioned, we are an innovation center. So for now, we are We are using this technology to try to see if they are relevant, but we are slowly moving towards production," }, { "start": 3524.181, "end": 3531.151, "text": "but we are not quite there yet, unfortunately. But hopefully next time we meet, I can say, yes, we have something in production." }, { "start": 3533.551, "end": 3546.645, "text": "Thank you. Yeah, I would just like to add also perspective from the Netherlands before we conclude. So in the Netherlands, yeah, we have also a lot of collaboration, the National Collaboration on Pets." }, { "start": 3546.645, "end": 3564.546, "text": "We also have the National Support Centre for Pets that is called Nick Pet. And one area that actually Rafael opened its secondary use of data. And in the Netherlands, what we see is there There are also a lot of kind of emerging use cases with the research on health" }, { "start": 3564.546, "end": 3580.209, "text": "data that often requires a combination of very sensitive health data, as well as statistical data that provides some sort of background information. And in Europe, there are also projects such as European" }, { "start": 3580.209, "end": 3596.131, "text": "data. health spaces, which also has secondary use of data for research purposes. And just to finish, also within the EU and Eurostat, European statistical system, there is a project that is currently just" }, { "start": 3596.131, "end": 3614.556, "text": "finished the feasibility study and proof of the concept of using pets in the context of European National Statistical System. So these are the links. And yeah, but things are changing and we will be updating the pet guide and we are also" }, { "start": 3614.556, "end": 3624.511, "text": "collecting new case studies as was mentioned. So yeah, if you keep checking our resources, we will be adding new examples. Thank you." }, { "start": 3626.591, "end": 3651.087, "text": "Thank you so much, Ben, Rafael, and Matthias for your responses. And it also lead us to conclude this Global Network webinar on the AI, specifically for the paths of work. of the privacy enhancing technologies and the artificial intelligence." }, { "start": 3651.087, "end": 3671.106, "text": "And I would say before we leave the meeting, I would like to encourage everyone to just open your microphone and turn on your camera to give a round of applause to our people. presenters today and to thank them for this great and fruitful presentation." }, { "start": 3671.106, "end": 3677.471, "text": "And thank you so much. Thank you so much. Thank you. Thank you. Degrade." }, { "start": 3682.431, "end": 3682.751, "text": "Yeah." }, { "start": 3697.471, "end": 3698.351, "text": "Thank you all." } ] }