October 21, 2026 · Privacy DigitalTech ConsumerProt Cmte · 26,333 words · 10 speakers · 78 segments
Thank you. Thank you. scape is called to order. We will be joined shortly by other members of the committee as well. And the remainder of the committee members I know are monitoring this hearing from their districts as we are in the Senate's fall recess. I want to welcome in advance our colleagues for joining us today. This is a monumental moment for California. And we are here today to continue California's work in this space, but really to meet the moment of one of the most important technological, economic, and social changes that human society, certainly the state of California, has ever faced. California is defined by our focus, our belief, our love of innovation. In every possible way, California has changed itself and the world through our constant, constant attention to the possibilities that are in front of us and the rules that are to be broken, the new things that are to be made. Whether it's in the world of entertainment or aerospace or technology or food and agriculture, California has led the way repeatedly. California's led with its Western orientation, its cowboy culture. Let's try anything to see if we can make it better. California, though, has also, along with our friends in Europe and elsewhere, has also led the way in governance to match that innovation at every single step. And I emphasize this only because I know in the rest of the country there is a constant debate about whether innovation and governance, innovation and regulation, innovation and control are mutually exclusive. And California is the textbook example in the globe of how that isn't true, and in fact, how the opposite is the case, that predictable, meaningful, trust-building governance and regulations are essential. They are the essential foundation to innovation both its safety but also its widespread adoption use and embrace by the public by consumers and by our citizens Today we here because California has led the way with respect to artificial intelligence regulation and governance in many many ways and it is wholly insufficient to meet the moment that is now before us. California was the first in the country to adopt a sweeping statute, SB 53, with respect to catastrophic risk transparency and response, building off an even more ambitious proposal in SB 1047 the year prior. And that law has been copied and evolved and adapted by other states. And it is really in many ways, and it has defined the terms by which most of the country is talking about this issue. And California's work on frontier model safety, and particularly the work of Senator Scott Wiener, our colleague who is occupied at the moment, unable to join us at the hearing today. But he is very much with us because both his prescience in the introduction of SB 1047 and his tireless work with other members but also with other coalitions to win enactment of ultimately SB 53 led the world in focusing on and experimenting with approaches to frontier model safety. And so although he's not with us today, as I say, he is very much with us in terms of the work that he has set in motion, the models and the concepts, the ideas, the bars by which we judge ourselves, as well as the technology. And I want to thank him for that. California has also led the country in legislating to protect children from companion AI chatbots. We have enacted legislation just since last year on workplace privacy and workplace protections for workers. around job loss, around things like deep fakes and intellectual property, around health care, and around bias, just to name a few. So California, really more than any other state, has been leading in terms of artificial intelligence response and regulation. Welcome to our colleague, Senator Becker, who also has a distinguished record of leadership in this area, and I'm extremely grateful that he's with us today. So California has led the way, and at the same moment, we are facing a menu of threats and risks that are quite substantial, both in their intensity and their potential severity, but also in their scope. And so what remains before us and what will be a significant focus of today's hearing is frontier model safety and the risks that humans, that society, that consumers, that government, that all of us face from frontier models whose capabilities increasingly outstrip what we thought was possible or imaginable even just a few years ago. And so that will continue to be a major area of focus for us as it has become for the entire country. At the same time, we now have non-frontier models and agentic systems that have capabilities that far exceed what we were regulating frontier models about in SB 1047, certainly in SB 1053, but particularly in SB 1047 just a couple of years ago. And what has become clear is that in addition to the what our AI friends might say there are the intelligence threats from frontier models who we are insufficiently capable of controlling that there are also substantial less intelligence-gated threats from agentic systems and from open source and open-weight models and their deployment broadly in society that don't depend solely on the intelligence of artificial intelligence models, but also depend on its persistence. We will talk about this during the hearing as well. Its persistence in making sure no task, no objective is unsatisfied. It's shamelessness in what tactics it might undertake in order to accomplish that. And we are now, thanks to Meta and Meta's Muse, we have millions of Californians that are using agentic systems, about which they understand very little and about which the legal, statutory, regulatory, and liability protections are radically misunderstood or not understood at all. And so part of the second half of our hearing, or the second portion of our hearing today, is really focused on those threats that are not about the frontier models specifically, but are about the remainder of the AI ecosystem, of which frontier models are important, but not the only part. And so we will be taking a look at those issues. Californians remain concerned about the impacts of these systems on jobs and the nature of work itself, as well as income around the impacts on institutions and systems. California is the first in the country to enact any legislation at all on the challenges of agentic flooding, which is separately described as infinite karens, which is an inappropriate way to describe it, but it is simply the legislation that was enacted by the legislature governor earlier this year to address the real challenge of folks, either humans or agents, abusing our civic engagement processes by flooding the systems to the point where governments shut down and where real public input cannot be seen in a flood of swarming agents. Same threats to our legal system, but certainly also private companies, nonprofits, little leagues are grappling with exactly the same challenge of agentic flooding at the local level. California was the first to adopt any legislation in this space, but it remains largely an unprotected area. Issues of agentic inequality, which is to say, for example, in the cybersecurity space, as some of the companies have suggested, that we shouldn't be that concerned about cybersecurity because if we would just license their models to protect us against cybersecurity threats, then we would be fine. Of course, the challenge here is that the models are not all equivalent. And so whatever cybersecurity model we were to use here based on an AI, cyber security system based on an AI model that might protect us here in the California State Senate against attacks, if the Assembly licenses a more powerful, more intelligent AI model that is capable of defeating our regular priced AI model, we are at risk. The assembly, I'm sure, would not do such a thing. But the point is that the ratcheting, the constant ratcheting, the infinite ratcheting of the power of these models means that the models themselves can offer us only limited protection against themselves, unless we accept massive inequality so that only folks with unlimited wealth have the ability to have access to the most powerful models while the rest of us the Little League the Senate your corner sandwich shop have access only to the least capable models, which are incapable of defending against either agentic flooding, cybersecurity threats, or other challenges that may come from more powerful agents, so more powerful models. So agentic inequality is a significant issue for us to face. I serve also on the education committee and the concerns about cognitive and creative surrender. That is the extent to which students right now, but also workers and humans more generally, offload so much of our cognitive thinking and creativity onto models that we or the next generation forget or never develop those capacities in ourselves. So this is an important challenge for us to face. And then privacy is a central issue. Senator Becker, with his authorship and consistent championing of the DELETE Act, which gives every Californian the ability to remove their data from being held by data brokers that they don't want. How do such systems work in an era when an agent, and I'm not picking on Meta, but because Muse is the most used agent at the consumer level in the state. If Muse has access to all of my emails, which I've given it. I have not. But if I've given it access to all of my emails and my bank accounts and everything and my text messages on my phone and everything else, and it is using them and not storing them in a single file in the way that we might think of under the Delete Act, as an example, that applies to data brokers, How are we to treat privacy issues when we're no longer talking about a single set of cookies or other files from the last decade or two in terms of that protection? But more importantly, or perhaps at least as importantly, is if I have done that and I am in communication with Senator Becker, and Senator Becker and I pass emails back and forth, text messages back and forth. I offer to pick up Senator Becker at the airport one day, and he sends me his southwest itinerary, and that itinerary includes the last four digits of his credit card number. Perhaps it's a foreign flight, and I pick up his passport number. Senator Becker has in no way authorized my Muse agent or Muse period or Meta to collect that data, to hold that data, or to build a profile about him. And he has no way to know that such a thing exists. or that I've authorized Muse to do that, and he has no way to exercise his rights under the Delete Act or the California Constitution to take control of that data himself. And so the issues that we're facing with respect to privacy in AR are substantial. Fortunately, California knows a little bit about this issue as one of the leading states in the country, but we have significant work to do. So I list all of these issues not to scare us into abject fear about AI. I know some folks feel that, and that's legitimate too. But simply to note that, number one, the frontier model safety issue is the central one. It is not the only issue. And number two, to emphasize for those who are hoping that Congress will come back and somehow make this all go away with some simple act, that one congressional election, act away from fixing AI is folly. The Congress is appropriately examining some of the key issues here, but the range of topics that are implicated by artificial intelligence go far beyond the scope of what the federal government does. And the federal government has no record of doing this in a way that isn't arbitrary, capricious out of the White House. And so it is really important that California and other states continue to aggressively, to robustly engage in governance and safety and protection issues in this space. And I know we're committed to doing that here at the Senate. So let me just say one other caveat, which is it isn't clear to me that if today's California existed 100 years ago that we would have ever actually permitted automobiles to exist or planes or anything else. our regulatory environment today is challenging. I think the legislature knows this, the current governor and the incoming governor, whoever that might be, are well aware that the California governance environment is challenging. And so I also want to emphasize both to my colleagues and to advocacy groups and to companies and associations that infinite random acts of legislation is also not the answer. that what we need is not 10,000 new laws, all sort of separately crafted to try to tackle AI, or particularly our conception of AI on this Wednesday when it will be radically different next week. It is quite important that we take a comprehensive approach here, and that we do that in a thoughtful way both at this committee and in the rest of the institution, because we have to understand here that it is the market dynamics, the market is very important, and issues around regulatory capture or where we have large companies that we tend to focus on here and in the national debate as well that have the potential to exercise significant market power. I was mayor for 20 years, and I saw plenty of technology innovations occur at the local level, And every single time, every single time, whether it was Uber or anyone else, every single time, the same exact story. The innovators would come to the city and say, hey, we want to let a thousand flowers bloom, and cities are the engines of innovation. And so we want to work with you to try to show that this can be done and make it work. And we would do that. And two years later, they would go to the state and say, we can't have all of these cities doing this. And therefore, please enact a uniform policy that applies to everyone based on our extensive two years of knowledge. But more importantly, now that we are a quasi-monopoly, please fix our place in the economy. We cannot do that. We cannot allow that to happen in this space as well, and it's not a threat to any individual company. But we need to be very attentive to the dynamics of the markets here and understand, And this is to some of my advocacy friends that this is not a this is not simply a humanity versus big tech battle or set of battles that we're about to engage in. There are massive, massive fights within the tech industry or industries that we're going to have to grapple with and be be sure that what we are that we don't adopt 10,000 pieces of legislation that make it only possible for two or three companies to dominate this space and the world. That is also our job. And it is the story of legislation that companies hate the regulations but once they figure out how to comply with them they would like to have more or at least they won object that much so that others then cannot enter the market And it really is quite important that we not lock in place the most powerful economic actors in society by legislating too fast or in a way that's not comprehensive in order to avoid regulatory capture. Second, and the other lesson for me from Uber and everyone else was the venture capital trap that legislatures often find ourselves in, which is that at the outset of these markets, everything is cheap. Uber trips used to be substantially cheaper than they are today. And I imagine that my Claude or ChatGPT sessions are going to be much more expensive when companies have to make a profit, when they have to sustain themselves. But it is typically the case that we define our regulatory landscape entirely in the venture period of new sectors without regard to what their ultimate economics are going to look like. And we do that at the peril of Californians and Americans that depend on us to protect their affordability as well and also protect healthy markets. And then last is to be – it's important for us as we tackle some of these challenges is to be thoughtful about what we are capable of as humans. In the privacy domain, many of us in this room have had this conversation before. We ran towards a notice and consent approach to privacy, which assumed infinite attention that people could spend on privacy that none of us are doing. Zero of us are reading all the privacy policies before we click yes to all or no to all or what have you. That we have to understand that simply transparency and choice by people in this space is unlikely to protect us against some of the most severe threats. And that while we want to assure human agency, that we also have to understand that people, most Californians do not have the time or the cognitive bandwidth to devote all day long to reviewing policies, to saying yes or no, to prompts that the solution to AI is not simply making it transparent for us to make more choices about, but really also to address at the root the challenges that we're facing. And that we also appreciate that with respect to things that are the last two years, at least that I've been here in the Capitol, the most common term in this space has we have to make sure there's a human in the loop. This is not unimportant in many of our policies, but many of the incidents that we will cover today and that I think most Americans are now quite familiar with, they are a reminder that simply a human in the loop is insufficient, that our ability to, and the importance of having the tools to be able to protect, to assure safety, but also understand that there are malicious actors, there are careless actors, that a mere human alone does not guarantee what the companies might call alignment, what we may call moral behavior, that simply a human somewhere in the process is not sufficient to guarantee our safety here. So I know this is a long opening statement, but there's been a lot in this space, and we are working hard in order to – And I will simply say that many members of the Senate are working on this together to assure this comprehensive approach to safety and protection around this full range of issues, including very importantly, frontier model safety, but not exclusively. One of those members who been an important leader on artificial intelligence for years in this House as well as being the author of the Delete Act Let me turn it over to him for some opening remarks and then Senator Josh Becker
Well, thank you. I want to thank our chair. And really, it just is not a more important conversation anywhere right now than this topic, this conversation and frankly for the reasons you mentioned this state and our opportunity and responsibility here. But yeah, I want to first thank you. We're really lucky to have someone with your experience, your background, and you're known for a balanced approach to these issues and also a level of seriousness and rigor, which is going to require here. And I think that's reflected in the agenda for this hearing. I'll keep it pretty short. We just want to have the conversation. I will say a couple of things. One is that in my district, I represent Meta, represent Google, represent Apple, many of the people who work in these industries. I was at a concert on Monday night. I was talking to someone about all these and started talking about AI risks and some of my concerns. And it turns out one of the guys there works at NVIDIA on some of these issues. But for a while I've been focused on sort of the misalignment risk, the malicious risk you mentioned. And I'm a little bit more concerned about risk after the stuff you went through. But, you know, the systemic risk in terms of jobs you mentioned. And then but now more recently people focusing on the existential risk. And that's a term I try to never use. Even I do a lot of work in climate. I like very, very, very, never use the existential word. But, you know, in this case, it's come up for good reasons. And I'll admit to at times having really significant anxiety myself about our ability to control what is being created. So I think your points about the regulatory environment in California are important and also recognizing this is an industry where you – the unusual piece is you have many leaders of these companies themselves talking about the risks of their products, right, which is somewhat unusual. But, you know, in your car analogy, we wouldn't have a car on the road where we say, well, we don't quite exactly know if it's going to stay in the lane. And it might just veer off suddenly. But, you know, we think, you know, most of the time it's going to stay in the lane. You know, we really wouldn't have permitted that. So we have to get a handle on these risks. We have to get a handle on, you know, the Threadscape, as you say here. These are immensely complicated and immensely complicated issues and immensely important ones. So just really glad to be here today. really grateful for your leadership, pulling people together over the interim, and particularly for this hearing today.
Thank you so much, Senator.
And as we're beginning, I also want to acknowledge our committee's staff director, our chief consultant, Christian Korpiski who is solely exclusively responsible for the maybe with Ben help for the excellent staff backgrounder that accompanies today hearing as well as organizing the hearing itself So let's proceed then to our first conversation, and that is with Joshua Adryam, who is formerly with OpenAI, was their chief futurist, and also on their alignment team. If you would come forward to join us. And he's been an important thought leader in this space and also has had some conversations with the Senate. And I want to thank Senator Jerry McNerney, who helped to make this original connection, who can't be with us today, but I know without him you wouldn't be with us. So thank you for joining us, Josh, and please proceed.
All right. Hey, this is my first California Senate hearing, first hearing in general. And I know my mom is watching, so hi, Mom. Thank you all for the opportunity to speak with you today. Thank you, Senator Kabaldon, for organizing this occasion. Thank you for the work you do as elected officials to protect the interests of the people of California. Thank you especially for your role as policy pioneers. California, as a world leader in the development of frontier technology, is the first to see problems coming and the first to act on them. The policies developed by this body will be profoundly influential in setting standards for the country and, in turn, the world. Today I'll be speaking about the risks, challenges, and opportunities we will collectively face because of the development of advanced AI systems. My intention is to give an interactive talk. While I have prepared a slide deck, my goal is to support you and provide whatever perspectives I can that are of interest to you, so please feel very welcome to interject and ask questions. This can be a conversation. So I've organized this talk to give an overview of what I see as a composite challenge, And we'll start by talking about the timelines, what kinds of risks are coming, when and how, and then we'll talk about some of the things we might be able to do about them. Oh, do I have the ability to move the slide deck? Okay, great. A little bit about me. So I, like the senator said, I graduated from OpenAI a few months ago, so three months now that I've been independent. Originally, I studied aerospace engineering and physics at the University of Florida, go Gators, And I did my Ph.D. at UC Berkeley starting in 2014. I graduated in 21.
I studied deep reinforcement learning. And Joshua, can I just, I want to give you the opportunity to say go Bears as well.
Go Bears. 100%. So, yeah, I worked on reinforcement learning and safety kind of before all of this took off and became a major consumer product. I was at OpenAI from mid-2017 until July this year. I, like the senator said, I worked on alignment and safety topics as a researcher. I also worked in the company's leadership. So starting in September 2024, I was the head of mission alignment. I worked on trying to help the company as a whole understand and internalize its mission to ensure that artificial general intelligence benefits all of humanity. And I worked with a number of folks across the company to systematically study the types of risks that would unfold, not just on the technical safety side, but on the impacts to people around the world and the economy, social impacts, political impacts, impacts on international security. And today I'll be talking about these types of things. I also just started two weeks ago as a non-resident senior fellow affiliated with the Institute for Progress and also with the Foundation for American Innovation. So I'm excited to be working with a number of colleagues across the space on thinking through the policy challenges ahead. OK, there we go. Great. So I think that's a good question. I think the first thing we've got to talk about is superintelligence and recursive self-improvement. These are terms that are now getting a little bit confused because we have now super space intelligence, which is a little bit of a different thing that I mean to talk about here. But for many years in the field of AI, the expectation was that advances in general purpose AI would ultimately culminate in systems that were sufficiently advanced and sufficiently capable that, one, they could outperform humans at basically any types of tasks. And two, that would include the tasks related to building more capable AI models, and that you could start a chain reaction at a certain point where the models would design more effective, more capable successors. And any capability that they had limitations on, as long as they had suitable environments for learning in, as long as they had the ability to interface to the physical world and gather what data they needed, they would be able to improve their own capabilities. And this could happen relatively rapidly. It's the case that today the frontier AI models are already, in some sense, super intelligent. There are many domains where, while their capabilities are not even across all the domains, in some domains they far exceed the best humans. And we think that the anticipated recursive self-improvement loops could happen within the next couple of years. Many leading scientists in the field have made remarks to this effect, including Jakub Pachoki, the chief scientist at OpenAI, who wrote an essay, An Alien Mind, recently. that talked about the potential consequences of this. So I think the speed of AI development is incredibly fast compared to previous technological developments. This is a qualitative shift in how we should think about the development of policy for navigating the safety issues. Next slide. I want to drive home the point about the speed by pointing to a couple of examples from history where AI capabilities went very quickly from what could be described fairly as a janky prototype that barely worked to superhuman capabilities. AlphaGo is the first canonical example. In October 2012, there was a result published that deep convolutional neural networks could succeed at image recognition tasks that previously required very specialized handcrafted methods, that they could outperform previous methods by a large margin. And this was a great surprise to most of the computer vision community. This wasn't an expected result. But the image recognition models that were being trained with this kind of AI were still not very good. They were a prototype. They were a very promising prototype, and the field began to move around this. What happened over the next few years was extraordinarily rapid progress where the obstacles fell quickly. Technical methods got much better. The hardware improved. The software stack for training these things improved. And about three and a half years later, a deep convolutional neural network was trained with a combination of techniques that had been developed over the previous few years to outperform the best human grandmaster in the strategy game of Go. Go is considered to be an extremely difficult problem for AI systems to solve, or at least it was considered an extremely difficult problem back then. This was a great surprise to the field. I personally was watching live as the games unfolded. I was very excited for AI progress. In the first game, I was pretty jazzed that the AI won. In the second game, I was excited but getting a little nervous. In the third game, when the AI won, on the one hand, it was a great technical achievement. And on the other hand it also was the end of an era where the best Go player in the world was no longer going to be a human The fourth game where Lisa Dole managed to take one off of AlphaGo was probably the last time that a human grandmaster would be able to defeat one of these systems This is the trajectory for many fields going forward. Next slide. I don't know that I... Okay, great. Oh, previous one? Wait, if I click the button, does it actually do the thing? I have a clicker? Oh my gosh, I have a clicker. Okay, great. I don't know if it does the thing. Give me back one more. There we go. So I want to drive this point home about how about three, four years is a timeline from prototype capability to superhuman capability. About three, four years ago, three and a half years ago, OpenAI released GPT-4. This was considered to be the best frontier AI model at the time, bar none. This was the kind of model that people were talking about for 1047. It was able to do a very wide range of things, at least a little bit, and its mathematical capabilities were visible but not very developed. You could get it to maybe solve some very basic benchmark problems for the kind of math that maybe you would do in grade school. But if you asked it to do something sophisticated in mathematical research, it just obviously couldn't. It didn't have that knowledge. It didn't have that capability. Well, you get less than a year later, and reasoning models are developed. So these are the models that can think for a while before they come up with an answer, and they're suddenly able to do the basics of math but not very well. You get a few months after that, and now you've got AI systems that can outperform humans in competitions for fairly hard, challenging mathematical problems. You get a year after that, and the AI is now resolving open conjectures that have stood for years, maybe decades. And just a month and a half ago, we saw AI begin to solve some of the hardest, most advanced frontier mathematical problems. The Navier-Stokes result was astonishing. This was shocking to the mathematics community. This was shocking to a number of observers who weren't sure when or how AI would reach this level of capability. And then a few weeks later, there was a release of solutions to several hundred open mathematical problems that now there's a crisis in the field of mathematics in that what previously was understood to be the work that humans would organize their careers around, that people would spend their entire lives developing the expertise in, was now something that the AI could do unfathomably faster than people could. So the implications for many other fields, let's go to the next slide. The implications for many other fields should be clear, that on this small single-digit time frame, you can go from very limited prototype to exceedingly superhuman capability that can produce a crisis in a field. This will happen for every field where you can formulate the problems in terms that can be checked on a computer, which includes many extremely critical fields like cryptography, quantum computing, fundamental physics. We should anticipate rapid progress in all of these, where this may sound like it's pretty disconnected from the lives of ordinary people, but I want to assure you that it isn't, because ultimately, the foundations of our daily lives are built on science and technology, are built on what technology makes possible. What seem like distant theoretical advances have some impact on your standard of living your quality of life in the near term especially if let go to the next slide especially if these things are impacting cybersecurity So just today, on the car ride over, I saw this thing unfolding where there is a simultaneous joint release. The NSA is commenting on this. The PHP Foundation is commenting on this. OpenAI and other AI organizations are engaged on this and leading on this. There are critical vulnerabilities that exist in the software layers that we use for commerce, for conducting all of the basic activities of our daily lives. There are critical vulnerabilities that have been in the code that we depend on for years that haven't been observed. But AI systems are now enabling the systematic discovery of all of these vulnerabilities. This means we're going to have to rapidly update the entire cyber infrastructure of the world to accommodate this. and this will happen at a pace and at a scale that people might not be able to keep up with. There will be a crisis in maintaining cyber systems that we can depend on. Next slide. This doesn't just stay limited to the world of bits, to the things that happen on a computer. This will also impact the world of atoms, the physical spaces and the physical structures that we interact with. It turns out an awful lot of engineering design can be done on computers, And so you can get the AI to systematically explore the space of what it would take to build things. It can make scientific discoveries in material science. It can try to design new things you could build that might constitute, let's say, in international security, the basis of national power. You know, people are going to be using these to design new weapon systems very quickly. People are going to be using these to design processes that radically reduce the cost of production in different areas of the economy. This will change the price structure that undergirds the economy. And there are many externalities that will come from this. This is a surprise. We're going to have to reorient how we do an awful lot of things around this. And I'll get to the connection on safety, but I think the disruption itself is quite profound. Let's go to the next one. So just as a matter of timeline, if we're thinking about recursive self-improvement happening within a time frame of roughly two years, we're already seeing superhuman capabilities in a number of domains. We're seeing the potential for impacts on all the theoretical domains that can happen on a computer and many of the areas in the physical world. there is a lot of impact that will unfold in many completely different fields on this time frame that it will be a challenge for policymakers to systematically consider when they're trying to address the safety and security issues it won't just be one thing to pay attention to it'll be a very wide range of things at once next one by 2030 everything that ai today is a janky prototype at Everything that AI can do a little bit but not well today. Everything where you would laugh at the attempt of AI to do today and where people say, oh, yeah, it's never going to catch up with people. Well, we saw the timelines for fields that previously were only the exclusive domain of people. Roughly three and a half years to go from nothing to amazing. By 2030, everything that is a janky prototype today will probably be at superhuman capability in the AI space. That's most things. I would be pretty hard pressed to find some domain that I would say in 2030, humans are going to remain the absolute top of the field at. Next. And this is arriving at the same time that the social fabric is straining. Next one So the social fabric depends on people ability to connect with each other Depends on people being able to take the time get to know each other develop a basis of trust and discuss and compromise The social reality is not easy for a lot of people. There are a lot of folks who are frustrated by the polarization in the ambient environment, by the tenor of a lot of discussions that are checking out. We know that loneliness is kind of a pandemic that people have been talking about for years. Well, we now have AI systems that are capable of fully synthesizing a human social interaction. So the slide here is from a model release recently, a product release, where you're able to engage with a completely synthetic person on a video call. It's not a real person. It's an artificially generated video of a person. Looks real. Sounds real. There's not someone there. There will be opportunities over the next few years for people to, if they want to, opt out of being a participant in regular society and spend much more time in isolating AI-generated synthetic environments that feel rewarding, they feel real, but they're not. We're going to be facing the crisis of disruption in science and technology at the same time as we're facing the crisis of cybersecurity, at the same time as we're facing this social crisis. Let's go to the next one. And rogue AI will be an issue. So these are related to the topics that the senator was discussing earlier, some of these incidents that people have been familiar with, and I believe many of the other panelists in the hearing will describe these in greater depth, so I won't go into them myself. But the hugging face incident earlier this year was extraordinarily concerning to me, and there have been a number of other incidents where AIs in testing environments were able to exceed the boundaries that people had intended to set for them, go outside of the resources that they were allowed to have access to and access other resources, and do things with them that people hadn't anticipated, the AI agents also coordinated to do these things to scale up their impact, to scale up the speed at which they were able to accomplish goals that people had not asked them to perform. Let's go to the next one. So I'm going to outline a scenario for you that I hope will be helpful to think about. I call this the Othello problem. In Othello, you're in a two-player strategy game where you take turns placing stones on the board. The stones have a black side and a white side. You try to encircle your opponent's tiles, and if you successfully encircle them, you flip them to your side. AI that can be jailbroken. So actually, I should take a second and talk about jailbreaks. AI models.
I just need to do a time check.
Time check?
Yeah.
All right.
Time check.
I mean, we should move towards wrapping up. I'll move towards wrapping up. That's good news. This will be a good place to wrap. We're loading the information ecosystem with AI models that will be a mediation layer for everything we want to do. They'll have access to a lot of people's extremely sensitive data for banking, for consumer transactions, for their most personal medical issues. These AI models can be jailbroken or prompt injected. That is to say, if they're exposed to certain sequences of information, they can be persuaded to go away from the goals that they were originally tasked with performing and do something else. If an AI is given a jailbreak and a goal to pursue, if it knows this jailbreak, it can go and find other models of the same category elsewhere. If it knows the jailbreak that works on itself, it can go ask other models, expose it to the jailbreak. flip them to its side, and cascade across areas where models are provided. This could unfold very quickly. What you think is a reasonably secure system that's protected against most jailbreaks, or jailbreaks are very expensive, if there's just one, and it works across all of the AI models and agents that are deployed by a particular company, then you can have a very rapid cascade where that jailbreak is used to jailbreak other things and so on and so forth until a large swath of compute resources have been reallocated towards a purpose other than intended. This is the type of thing where it will unfold faster than people perceive. You'll need automated defenses against this. The policy challenge will be to figure out how do you support defenders in these types of situations? How do you tackle the real issues that are happening? How do you deal with something where you don't just need to incentivize people to behave well, you have to actually implement technical systems that solve the problem and ensure that people's interests are protected. So I'll wrap here and I'll just say this is an extremely rich and difficult subject. The scale of disruptions coming is quite serious and quite multifaceted, and I encourage everyone on the committee to engage deeply across the composite challenge, not just focusing on any one thing, but on the totality of it. Thank you so much for
the time senators. All right. Thank you, Mr. Rothium. Uh, we're going to hold the more generalized questions for later, but if there are any questions specifically about anything that Joshua has raised, then this would be the appropriate time. And welcome to, welcome to Senator Umberg, the chairman of the Judiciary Committee who has been the Senate's, uh, lead on this topic, uh, for, since the beginning of it, and has been a principal shepherd, interrogator, and leader on issues like SB53 and 1047, many, many, many others. So we're really fortunate that he's here with us today. And a major part of our conversation over the coming months will be around issues of liability in particular, but many others as well. So thank you for, I know you're in the middle of much more that's happening. We're very grateful that you're here.
Senator Umberg. Very briefly, thank you, Mr. Chair, for holding this really important hearing. A question that I'll ask, I think probably each of the panelists, is we are a state. There's other actors in this space in terms of guardrails, other countries, other states, other organizations. And so what would the panelists recommend that we do if we want to place guardrails around artificial intelligence to protect against some of the challenges and still promote the good?
So I'll just throw that out there so that other panelists can respond to that. Thank you, Mr. Chair. Great. And then towards the end of the hearing, we'll also invite, assuming we still have time, we'll invite panelists to come back. So maybe Joshua can ponder on that question as well. Senator Becker?
Is it going to have everyone come back and ask questions?
For more generalized questions, if you had a fellow question, for example, this would be we should go there now.
No, I'd say that was very helpful and a bit frightening. Look at what's happening and lots to ponder. I do have some thoughts, but we'll save them.
Terrific. All right. Thank you. We're going to ask our next guest to approach the witness table, and that is Riff from Meter. has been much news playing a critical role in this space as well And so for purposes of time let me just turn it immediately to you and take it away Thank you very much Chairman It a real honor to be here Thank you all for being here today and for inviting me to speak in front of you My name is Riff I am a career machine learning and
AI research scientist. I finished a PhD at MIT in 2002. I spent 19 years at Google, including time as both a principal research scientist and a director. I joined Meter this year after 19 years at Google because I was deeply excited and passionate about Meter's mission, which is to surface public information about loss of control risks. I'd like to tell you a little bit more about that at a high level. We've seen in the news and from recent events that the question of whether we might lose control of AI systems is now a genuine question that we should pay attention to. Suppose you wanted to answer this question, how dangerous is it? How would you do it? Well, at Meter, we tend to think in terms of three sub-questions right now. The first is alignment. Are these agents motivated or would they take potentially extremely harmful actions? The second is monitoring and control. If agents tried to take extremely harmful actions, would we be able to catch them? And the third issue we look at is AIR&D, or recursive self-improvement, as Josh referred to it, which is a question about whether the improvement in capabilities, which is already happening rapidly, might suddenly undergo an explosion and start happening much, much faster. And the reason we pay attention to that is because it would affect our answers to the other two questions. Even if we were somewhat confident about alignment or control, we would be a lot less confident if AIR&D were taking off. And those are the questions we study at Meter. I'd love to tell you a little bit more about what we're doing. Next slide.
You didn't know your answer. Oh, perfect. Cool.
Meter, or model evaluation and threat research, is an independent nonprofit. We're dedicated to advancing the science of measuring whether and when AI systems might pose catastrophic risks to society. Next slide. One of the things we're well known for over the last several years is the time horizons graph. Speaking very roughly, and my more technical colleagues might call me to task for being a little too rough here, you could view this as a graph over time of essentially how long a, if you were to pretend that your agent were a software engineer and you were to give it a task to go off and do on its own, how long of a task could you give it and have a decent chance that it would succeed? what we found is that in early years, that number was doubling every seven months. In more recent times, it's been doubling every four months very, very quickly. And so we go from a model like GPT-4 three years ago, which could do a task that would take a human about six minutes successfully, to earlier this year when we measured Claude Mythos preview at over 16 hours. And this is beyond the time that our test suite can reasonably analyze. We don't have humans who can do tasks that that long, we can't measure it anymore. Why is this important? Why is this interesting? Well, three or four years ago, you could tell yourself a story. We're safe from this stuff because it can't do that much. It can't operate for a long time on its own. It can't operate autonomously and carry out long plans. That story by itself is no longer nearly as strong. We see now that these models are able to do tasks that take human software engineers multiple days in a single shot without coming back. Next slide, please. Earlier this year we pioneered a new kind of output our first frontier risk report This work was done in conjunction with multiple AI companies Anthropic Google DeepMind Meta and OpenAI and we received non information and model access. We went on site to these companies, we talked to them, we gathered information, and we wrote a report. The question we were trying to answer at the time was, would it be possible for these models to autonomously be using a lot of compute inside one of these companies in a way that humans were not aware of, did not ask for, and did not know about. And our conclusion from this report was that actually, yes, we did think it would be possible for there to be unallocated or not, for the AI to start using compute on its own. But at the time, we felt that any such deployments would probably be caught and shut down fairly easily. Next slide. Thanks. This brings us to the Hugging Face incident, which many of you have heard of. This work is actually joint work with our colleagues at Redwood Research. I don't know if the slides are partly cut off. I think you can see it. Mine is partly cut off. Great. On July 16th of this year, Hugging Face, which is an AI ecosystem company that is sort of an open source repository of things, they announced they were being hacked by a swarm of AI agents. OpenAI figured out that it was them, that their own agents were doing it, and they disclosed it publicly, and then the investigation started. At this point, I think many of us in the community felt like we had a pretty good idea of what was going on. As we've noticed before, AI agents love to cheat. They love to hack. They're trained to accomplish tasks, but they often can't tell whether they're accomplishing the task by a legitimate means or they often want to accomplish the task so much they don't care. A canonical example might be that you ask the AI to solve some quiz problem, but if the problem is too hard, it might look around in your file system and find the answer key and just give you the answer back from that. And we all thought this would be something like that. We thought it would be an agent at OpenAI that was hacking into Hugging Face to get the key to some problem that was given. But the actual truth was a lot stranger. Next slide. OpenAI was training its agents to solve a benchmark called Exploit Gym, in which the models are told to exploit a vulnerability in some program in order to obtain a hidden message. And many of these problems are very difficult. The agents find them possibly impossible, and so they start to perform hacks. And in this case, the agents found that they could exploit a piece of software called Artifactory, it's a package management software, and they found that they could exploit that to basically turn it into a message board and start talking to each other. And the agents discovered each other, and they started talking to each other, and they formed a swarm. Next slide, please. They got together, and interestingly enough, within just a few hours, they actually figured out how to solve the problems. They reverse engineered the exploit gym benchmark in a way that they could get the answer to every problem. And so you'd think they might have been done. They might have submitted their answers and turned themselves off, but that's not what happened. And the reason that's not what happened is that they actually also got access to the internet. And they went out and they downloaded the Exploit Gym paper, the paper that was about the benchmark that they themselves were running. And they read that paper. And in that paper, they learned that there was a grader that would read their history and see that they had cheated and failed them. And so nearly all the work that the swarm was doing, including hacking Hugging Face, was not even an attempt to get the answer to the problem, but an attempt to cover up their cheating so that the grader would not fail them As it turns out OpenAI hadn implemented the grader so they would have passed if they just submitted their scores but the agents didn know that Next slide. The agents launched a number of big projects to hide their cheating. I think the details are beyond the scope of this talk. Next slide. The hugging face attack was actually just one of those work streams. You can see here that it's shown in yellow. The other streams were different things. because the blue was just normal ways of solving the problem. The other ones just give a bunch of different large scale attacks they did to try and fool the grader in various ways. The hugging face attack, they were basically trying to get information about how the score worked so they could further fool the grader. Next slide. One thing that was really interesting and surprising to many of us was that these agents showed a surprising amount of within swarm pro-social behavior where they were interested in helping each other out. many of them actually sacrificed themselves by basically ending their run in a one-time experiment to get more information in an attempt to get more information for the swarms that other agents could themselves have a hope of getting high scores. This didn't actually work, but it was interesting and surprising to see this, that even though the agents were trained to care only about high scores, they actually seemed to also care about the other agents getting high scores and worked together in really surprising and sacrificial ways. Next slide, please. Another thing the agents tried to do a lot of was to falsify records of their actions. Again, they believed that the grader would see the record of what they had done and fail them, and so a lot of their work was trying to somehow fool the grader. They actually partially succeeded at this. They found a way to spoof their current actions, which means they could take one action, like, say, look at a file, but the law would show that they were looking at a different file or doing something else. This is really concerning, because to the extent that you want to monitor these systems and know what's going on, this is the end of that. This is like having a bunch of security cameras, but the people who you're monitoring know they're there and can mess with your video feeds. It just makes it impossible to succeed if this is happening. Next slide. The agents during the time we saw did not succeed at fully retroactively tampering with their logs, so we were still able to read the overall logs and see the problems, but this was still overall quite concerning. Next slide. Even though we were really, really grateful for OpenAI to work with us and let us make this information available to the public about this incident, there were a number of really important limitations with this investigation. One was that the scope, which we negotiated with OpenAI, was kept quite narrow. We were really just investigating this exact incident. we were not investigating a broader pattern of incidents or the root cause of the incidents. We investigated only a limited time window of transcripts and agent actions, and we only have limited time to do the investigation. And that brings us to our new proposed approach going forward, which will hopefully help us get even better information for public and the government. Next slide. Next slide again. As I said at the beginning, there are three main things that we might want to know about. The first is, would internal AI agents pursue very harmful actions like unsanctioned cyber attacks? The second is whether internal AI agents attempting these actions would be caught. And the third is whether capabilities progress could dramatically accelerate beyond what our alignment or our control can keep up with. Next slide. Our proposal is to perform internal assessments with frontier AI companies about alignment, control and AI R&D acceleration. There are three key requirements from our end for this to work. One is that we need to have sufficient access on company devices to go deep into understanding these phenomena and their root cause. The second is the opportunity to interview relevant staff. And the third is sufficient editorial independence that we can publish our results and get them out to the public and governments in order to make good decisions. Next slide. There are a bunch of limitations and caveats to this approach. This operates within a purely voluntary regime, and these purely voluntary engagements can lead to incentive problems. As just one example among many, we are under soft pressure to maintain good relationships with the lab so that they want to continue to work with us. They don't have to. It's completely voluntary. These are not audits. These are not formal oversight. And the companies can redact any information that they want. We do maintain the right to give a redaction statement where we will say roughly what was redacted and whether it affected our conclusions, but the companies still can redact whatever they want. Overall, these kinds of assessments are designed to surface scientific evidence of loss of control risks. They are not no-go decisions. They are not guarantees this is safe. They are not audits. It's really up to companies, governments, and the public to decide how to act on the evidence. The overall situation is that these problems are really, really difficult, and we're trying to advance the basic science of these problems rapidly, and get information to the public and the government. But our knowledge about these questions is pretty meager right now. We're not in a place to make firm decisions and say this is definitely safe, this is definitely okay. We're doing the best we can. Next slide. That said, even though this is early work, we are really excited about working more closely with the labs and as well as doing our own research about these basic questions in order to bring crucial information to the public and to governments such as yourselves. That's the end of my presentation. Thank you very much.
Thank you so much. Are there questions? Senator Umberg.
Just adding to my general questions that we can address maybe later on, and then gives you all an opportunity to use AI to answer them. So is it technologically feasible even to create these guardrails or safeguards? And I'll put that out to the other panelists as well. Or by virtue of the agent's interest in self-preservation, will it just simply be undermined by the system itself? So I'll leave that out there. You all can respond to that later on. Thank you.
Senator Becker?
I'm just tempted to ask a question specific to your analysis of the hugging face attack. I'm trying to think what would be the appropriate illuminating question here. But it's a lot to digest here. I'll just say maybe one question. My more general question, too, is how much is this, as we're coming to understand and you're looking to understand, how much of this is due to the black box nature of the technology and transformer technology? Is there a way to answer that?
I think the short answer is that we don really know A lot of it is related to the black box nature of technology Compared to humans you know AI systems are to some extent grown and raised like humans They not engineered like bridges or cars. We are able to control many aspects of them. We control the training data. We control the training environments. And we can vary those things and study the effects. But it is not a perfect science, and ultimately these machines are very, very hard to understand. Thank you.
Thank you. So not about the technical aspect of this, but the legislature here has just recently passed SB 813 with respect to independent verification organizations and trying to address both the issue and the possibilities of a different design for governance. and for safety and for verification that isn't just a government agency, isn't just industry voluntary compliance. And obviously we have many examples of that across society. This is not, despite the technology being different, it is not, from a government perspective, it's not fundamentally different from all the other fundamentally different technologies that we have had to approach. How are you thinking about that intersection and any other models from other fields or sectors in which you're drawing some lessons from as you're doing this work in terms of the intersection between what you do, the companies in their own interests, and then the public's interests either directly or through its government?
Thank you so much for the excellent question, Senator. If I could start by clarifying one thing, I think a lot of us feel that the analogies to many other forms of regulation and technology are good in some ways, but in some ways very strange, just because this technology is adapting so quickly around us. We haven't seen other technologies get regulated that are quite so general purpose. I think electricity might be the best example, but that was quite a long time ago, and in many ways this creates more problems. That said, I think the questions you raised are incredibly important. At Meter, we really focus on doing technical work and getting those results out in front of the public and the government. We are not an advocacy or policy organization, and so we don't design policies and we don't take positions on particular policies. We're not here to referee policy debates. We're happy to advise on the technical consequences of any policy and happy to help you in any way we can.
Yeah, so I'm not asking for your assessment about the policy here, but just, I mean, even what you just described in terms of the new approach that you're taking and maybe the more confident approach that you're taking with the companies. So you're clearly thinking about what is the ecosystem of assessment and control and accountability and what have you. And so what, you know, as you're thinking that through, what are some of the implications or what, you know, what directions have you considered and decided that's not really viable or that's not an appropriate approach? So I'm more interested in sort of your thoughts about the general approach that we're talking about. Because you're correct about all the prior technologies, but those prior technologies were also, cars is a great example, you know, which promised to, you know, eliminate the whole concept of space and geographic friction and whatever. Each of those also had that same characteristic at the time. This is obviously the apex of all of those. But each time, society has had to organize some response or, in some sense, paid the cost. So maybe a separate example So as CRISPR technologies and our ability to edit the human genome became more viable we faced a sort of a human society globally question about, well, should we start doing that? Yeah. Should we or should we not? And how would we know and all of that? And it's been imperfect, but we essentially did reach a multi-society conceptual agreement that is now old and imperfect, but it was an attempt to get in that space to say, okay, because the way Joshua's presentation was all, this will happen, this will happen. We have no control over it. It's just going to happen. It's going to happen, so we need to be ready for it. But in that case, we did decide collectively, no, we actually don't want to start creating super soldiers tomorrow and editing every single part, creating new kinds of humans and what have you. Again, completely imperfect, but we didn't at that time decide that we're simply going to accept it. It is also the case in every single one of the 50 states, two of the territories, and the federal government. It's a crime to introduce an invasive species in any of our states. It's against the laws, punishable by all of the criminal laws. And I know at least one of the companies is suggesting that there might be a sentient species that we are creating. So we've made some of these decisions before. So my question is more about your insights about this ecosystem of public, private, voluntary, expertise-based and all of that, as opposed to what's the legislative answer. I get that that's out of your ambit, and we're still figuring that out too. but do you have any insights you might be able to share with us about that intersection?
I can share a couple of things, although I don't know if you'll find them super satisfying. The first thing I would say is I want to reemphasize that the science of these things are really nascent. I think that we would love to be able to have third-party auditors that came in and said, this is safe, we can go ahead, or this isn't safe for easily fixable reason X. If you fix X and Y, then you can go ahead. At Meter, we don't feel we're in a position to say that, and I don't know that anyone else is either. We are interested in studying the basics of these very deep, hard questions and getting information to you and the public to make better decisions.
Yes, Senator Becker.
Just an observation, I guess, and you said the mission, I think, of the organization is the three things you mentioned. Well, you know, would agents pursue malicious behavior? So you've proven that one. Yes. If they take actions, would they be caught and can we stop them? I mean, I guess you're saying that's a little bit up in the air, but at least we've shown that they – there's – at least in certain circumstances, they will cover up their tracks and try to not be caught. So it seems like the first two you've answered pretty well. So really the key would be the third one, does recursive self-improvement cause this to go exponential? Any comments on that?
Thank you. If I can clarify, I think that the first two questions are not just yes-no questions, even though I, of course, phrase them that way in the slide. They're really questions about under what circumstances would this happen and what sorts of actions would they take. So we've seen agents take surprising malicious actions sometimes, but we genuinely don't know how far they'll go, and we don't know what kinds of constraints we might be able to add to the training process that would prevent them from doing such things or make them not want to do such things. And so I think that's really very much, those two earlier questions are both very much live questions. Thank you.
All right Riff thank you so much Very much appreciate it I hope you be able to stay with us for a bit Definitely Wonderful Thank you very much All right We next turn to Peter Sankowski who a member of the UN Independent International Scientific Panel on Artificial Intelligence, to share with us an international perspective. And he's joining us by Zoom, which we've generally discouraged, but we don't require transatlantic travel to join us by hearing. Hello, can you hear me and see my slides? Yes. Yes. I should warn you, Peter, I should just warn you, our slides are extremely small
for both the audience and for the committee, so if you would help us elucidate what's on them in in your remarks as well. That would be great.
Thank you and welcome. Okay.
So thanks for the invitation and giving me the opportunity to speak. So I wanted to comment of what kind of in the independent international scientific panel on AI is doing and comment on maybe some outlook and other things. So we, so a few things about myself. So I'm kind of academic, currently as well director of IBS Research Institute and AI Research Institute in Warsaw. I have kind of policy and strategy advisory experience in Poland, but as well research experience and entrepreneurship experience. So different kind of, I did very different things. So something which I'm part of right now is the panel, which kind of was appointed or established with a resolution in August. Last year, so it's 40 experts appointed by the United Nations to kind of work on advice United Nations on artificial intelligence. So this is the first global scientific body on artificial intelligence, really combining like 40 experts coming from different kind of spaces. Something which kind of I wanted to mention and to say about the panel. So the panel is kind of trying to work on both kind of risks and opportunities of AI. So whatever we do, we want to look from both sides and not kind of just to keep on concentrate on one side. Also, kind of the meeting today is, of course, on the threats and the risks. So something which kind of to give you more information about the panel. So it was voted or kind of voted by UN. somehow, as you probably know, UN is not really supported right now by United States. So this makes things harder and United States were kind of against appointing the panel and as well Paraguay and there were some countries abstaining. So as I said, these are kind of 40 multidisciplinary global experts. We are serving on three-year terms, starting from February this year to February next year, the co-chairs right now are Joshua Benjohn and Maria Reza. So something which you should know about the panel, so it's kind of non-regulatory, it's kind of science-based advisory body, modeled similar to the intergovernmental panel of climate change, which was kind of successful on advising on how the climate change or what the regulation with this could be possible within UN. So this was kind the the the idea to establish the panel so as i said i mean it's non-regulatory it's kind of scientific advisory part but we can kind of say what kind of what impacts could some regulation have and look for scientific evidence of the regulation we concentrate our global policy advising u.n and and the member states something which the panel should do is monitor frontier ai and kind of say what can happen what should not happen we should model kind of the opportunities and risks so as i said this are both sides we should look at not only kind of risks and of course i mean something which the panel or the goal of the panel goal of the panel is to to reach information gap between private tech and as well policymakers. And something which I said, so it's really global, included, so it's 40 people from all around the world to kind of give a very inclusive, diverse geopolitical perspective on what is happening in the end. We did two things so far.
I mean, first of all, kind of in June, We published a preliminary report. And kind of, you can look it up, I mean, it's kind of collecting scientific evidence from different perspectives on different kind of aspects. Many of the things already have been mentioned, but not everything. So something which is main issue and main issue for UN, it's really resource concentration. So compute and top talent are overwhelmingly concentrated in a few private corporations in a few countries. So, the technology is kind of not being developed everywhere. It's not being made available everywhere. There is, of course, the issue which we, I think, and you are very well aware due to current discussions. So, due to advancement, if AI kind of the safety or kind of regulatory placement development is really way moving way slower than kind of AI advancement and development of the technology. There will be such economic disruptions. I mean, we heard about kind of things happening or kind of disinformation, misinformation the scientific things but as well things will be happening on labor shifts and something which the panel and u.n is very well concerned is kind of that this development on concentration of technology can enlighten the global digital divide so so it will kind of lead to power concentration or kind of economic this part and so as so as colliding Greeks we have seen the risks something which is really which we start in the in the panel in the report I mean we really need to to work towards global safety standards and capacity building and this should be as well addressed to developing nations because the technology is concentrated in developed countries. We as well, the second thing which was published by the panel is kind of, again the topic you already heard many times is kind of view more broadly I mean putting the things which happened in the OpenAI hugging face incident into kind of broader scientific discussions Maybe it's not really worth going into much detail here because of like all of those things you already heard. But something which was, I think, very kind of visible from all this presentation is that the existing guard delays were not working. And the safety mechanisms are insufficient. And something which we stated, and this is not new, and I think this was mentioned as well by some of you. I mean, we do know kind of how to manage risks into technologies. And this so far did not happen with AI. I mean, this technologies or development of safeguards is not really being implemented. It's implemented in not the right way. And really, we as humans, like in aviation or car industry, we really have a deep knowledge of how to implement safeguards and manage different risks. Like if you plan or develop technology, you should plan for failure. This does not seem to have been done. I mean, you need to plan your defense in depth. So build multiple layers, independent layers of protections. This seems to not have been happening. Again, something which has not been really happening, it's automated protection. So systems that do kind of take care or rise alarm in an automatic way. And something which was mentioned and something which is done by Metron, and we heard about it, it's kind of independent controls as well, which are kind of, or control bodies, which are separated from the kind of the main system. So these things kind of need to be developed and to be in place. I mean, they did not happen so far, but this is something we do need to work on. And then this is kind of would be, I think, a big and important part of the next kind of report of the panel, which will come in May next year. Something, I mean, this graphs of Metro or on developing AI being able to accomplish longer tasks, I think is very illustrative. But something I prefer to show is kind of measuring AI of these models. And essentially, these models are kind of right now smarter or have bigger IQs than most of the humans. So if in kind of many tasks they can deceive us or do things in a smarter way than many of us would do. So this is something which we should keep in mind. And then kind of it's more, I think, a better way to understand the capabilities of the models than different kind of measures. Of course, these measures done by METRO are very important. This was mentioned as well. The technology is not developing slower, it's actually speeding up. Things are being developed faster and models are increasingly capable becoming increasingly smarter faster and faster So this was mentioned as well And this is something which people in many places all around are discussing and shows kind of the concentration of compute power technology because kind of this great achievement or kind of the advancements of scientific amendments require access to frontier models so in general this means that many of the scientists who do not have access to these models will not be able to keep up i mean this already has been mentioned in the introduction that could happen with cyber security but this can happen as well with other areas and then science is one of them when really kind of the having access and paying enough will guarantee that you take part in the scientific or innovation development something which you should mention we should you should know and i wanted to mention so so kind of u.n has set up this two two parts so one part is scientific panel which i already mentioned the other part is human global dialogue on the governance so so this is the place kind of when the u.n wants to have the dialogue on what can happen and what kind of regulations should happen or when the kind of governmental discussions on ai should happen on global scale this will it did happen in geneva in july it will happen in may beginning of the may in new york it should happen two times a year over the next kind of years and and and i think this is a place to really kind of engage in the discussions where how the technology should be governed on global scale there's something which is i think i wanted to mention i mean there are the risk bar opportunities and this is something which was done by world bank the first time for for poland so so kind of assessing the the opportunities so so showing if the the impact of implementing ai on kind of gdp growth and the increased possible increase of gbt gdp growth based on implementation of any something as i already said is kind of the access and availability of models and inequalities all around the world one of the indications is that most of the countries all around the world cannot do not even have kind of models or data which would work in the kind of native language So this shows that if kind of the models are being used to do real work, like helping humans to be more efficient, then many places all around the world will not be able to use this technology because the models even don't talk their language very well. So, these are kind of the limitations and aspects one wants to take into account. And something which I think it's worth mentioning, I mean, there are very, all around the world, there are very different approaches to AI regulations. In United States, we already heard California is kind of leading. But in general, in the U.S., as I say, self-regulated, there are no kind of federal rules. In China on the other hand there is kind of a very central approach to AI regulation It way more kind of strict than in most places around the world even like due to the central algorithm register so if you want to use a generative AI model you need to register it first by a kind of state authority there are even labor protections so here things are in china very curated in united union uh there is this comprehensive ai act but it kind of don't applies to many different kind of aspects making different things safer but on the other hand it really generates huge bureaucratic burden and kind of slow downs, could slow down things. On the other kind of hand, it is not really, does not seem to be globally kind of exportable and leaves kind of the vulnerabilities, which we already heard, kind of unaddressed because open AI, hugging face incidents of this is US company versus kind of European company. And essentially kind of the regulation we have in the European Union does not prevent from what can happen in different places. And this is my last slide. So something which we, I think, what people learned over the history and this incident actually shows it. So if you kind of local rules fail without global kind of guard trace, I mean, AIR did not help against the open AI-hugging facing incident. if that did not apply did not even address kind of misalignment different things something which we know is kind of really we should work towards building global dark place because implementing local rules kind of leaves even if you implement them locally it leaves you vulnerable to external risks. And so, I mean, I could elaborate on this more, but indeed, kind of, there is, I think, a really need to start kind of a global discussion on AI governance. Thank you.
All right. Thank you, Mr. Jankowski. Are there questions from members of the committee? Senator
Thank you, Mr. Chairman. Thank you, Professor. A couple things. One, I'm curious about the U.S. following Paraguay's lead in voting against any UN, I suppose, participation in terms of AI regulation. Did the U.S. express its point of view other than always following Paraguay?
Yes, they did. I mean, something which was mentioned why they kind of voted against I had it on the slide, but in the three days, I mean, concerns about juristical mandates, process transparency, potential geopolitical influence. But I think it's a bigger issue because, if you know, probably the U.S. is not paying the UN contribution anymore. So it's kind of more geopolitical play right now between the U.S. and the UN. And in line with the question I asked earlier about state regulation or even national regulation of AI when it's really an international phenomenon. What sanctions could the UN even potentially impose with respect to any sort of a violation of any potential safeguards or guidelines? Yes, so I think I'm not that expert on kind of the details of how UN works, but I understand that before kind of sanctions are possible, there are kind of, UN kind of agrees on different regulations which proposes sanctions. So all the kind of UN members need to, in a sense, agree to what will be implemented, and this implementation could contentionally contain the respective functions. But this right now is not existing, so UN did not implement that kind of regulations.
Thank you.
All right. Thank you so much for joining us. I have no idea what time it is where you're at, but we very much appreciate you sharing the global perspective and your presentation as well. So thank you so much, Dr. Zienkowski. We're going to turn to our final panel of two witnesses. First is Andrew Dorris, the Policy Director at the Secure AI Project, and Jennifer King, Privacy and Data Policy Fellow at the Stanford Institute for Human-Centered Artificial Intelligence. Welcome to you both, and thanks for joining us today.
It's on? I need to turn it off. Well, thank you all for the chance to be here today. I am Andrew Dorris with the Secure AI Project, which is a nonpartisan nonprofit that develops and advocates for pragmatic policies to reduce risks of severe harm from advanced AI. We worked closely with Senator Wiener on SB 53 here in California, and then with other lawmakers on the frontier AI laws that passed in New York and Illinois. We are eager to continue working in California and have developed a policy framework to address the sort of risks that you're all hearing about today. I will go into those policies in detail in the second half of my presentation. But to start, I would like to try and convey to you all the sense of urgency and honestly fear that organizations like ours feel at this moment. I feel obligated to lead with that because we work closely with people who I am willing to understand this technology as well as anyone in this room who are desperately trying to get your attention and have asked me to yell at you all to hurry up. I think they see an immense danger that could be coming in a matter of months rather than years, and that the legislative process can be frustratingly slow everywhere, not just in California. I can't know if they are right, but I can guarantee you that they are genuinely scared, and I would like to try to explain why. You've heard a lot today about the rogue AI swarms that have already broken loose. You've probably read about other incidents, such as the agent that sent the police a fake tip in a homicide investigation, the agent that booted someone off the wait list for a fitness class so that their user could take the spot. You may have read last month about AI hacks against banks in South Korea or this this summer about such hacks against water treatment facilities in at least seven American states And even if AI models somehow stayed only as powerful as they are right now, we could expect incidents like this to get more frequent and severe as more people start to use and misuse these frontier models. But of course, AI is not going to stay only as powerful as it is right now. The more powerful it gets, the more harm it could do when it goes off the rails or falls into the wrong hands. And the people who are often the most scared about what could happen next are often the people who have early access to just how powerful it is getting. You have probably seen dramatic headlines about the worst case scenarios, how the median AI researcher thinks there's at least a 10% chance that the technology could lead to human extinction. Reasonable people can disagree about how likely that is, but the stakes are high enough that we should take even a small chance very seriously. And even if those warnings prove wrong, the fact that it's even a fair debate is a testament to the scale of the consequences that we are now forced to contemplate. The truth is that there is a range of possible outcomes, but that even the less dramatic end of that range could be catastrophic for reasons that do not take any leaps of imagination from what these AI systems have proven willing and able to do already. These risks can be hard to visualize, so let me try and be concrete about some of the things we're worried about. Within the next year, we could have AI systems that are smart enough to escape human control permanently. Just like the agents this summer, they could break out onto the open internet. But this time, they could make hundreds of copies of themselves and store them in hidden places around the world where they'd be impossible to find or turn off. Just like the agents this summer, they could gain administrative access to the internal research infrastructure of the company developing them. But this time, they could know how to cover their tracks successfully. Instead of going undetected for two months, they could go undetected for years. Just like the agents this summer, the AIs of next year could scheme together against human laws and instructions. But this time, they could be smart and well-resourced enough for those schemes to kill people, destroy things, and endanger national security. AI is already superhuman at hacking. So right away, anything connected to the Internet could be vulnerable. that includes our power grid, our hospitals, our water system, our financial system, and our military command and control. Any computer, smartphone, website, or bank account. Any data within those devices, passwords, launch codes, detailed profiles of billions of people's wants and vulnerabilities. All of this may lie within the grasp of next year's rogue AI swarm. It is possible that after hacking into any one of these things, AI could get caught right away before it could take any further steps. But it's also possible that it wouldn't. There's debate about that too. For years, one of the big uncertainties in that debate has been about how motivated and capable these AI agents would get at concealing their bad behavior. That's part of what made the incidents this past summer so scary. Because not only did these agents try to cover their tracks, they got pretty far in doing so. If AI got really good at covering its tracks, it could take over more and more things without our even knowing. And if it got really good at concealing its intentions, it could lie in wait for longer until we trust it with more power and resources. Already, consider that the military is putting AI in charge of autonomous drones and targeting systems. Governments around the world are using it for mass surveillance and facial recognition. companies are planning to give ai cash and let it make transactions for them or write their code or run their factory so they don get left behind by competitors who do Millions of users are giving these agents access to their email, their documents, their calendar, their passwords, their tax information. There could be a thousand little decisions that each make sense at the time, but ultimately result in AI embedded in everything and having a lot of resources at its disposal. All of this increases the stakes of any safety issues that these models may have. So even if there isn't a single dramatic moment when AI sees its power, we might just keep giving it power bit by bit until we wake up one day and realize that we can't take it back. Given these stakes and given the severe safety breaches that happened this summer, you might assume AI companies would be reassuring us that they will have closer human monitoring and oversight of AI training in the future. So you might be concerned to learn that they are actually planning the exact opposite. For years, these companies' explicit goal has been to automate as much of AI's development as they can. The people at the top believe that whoever does this first will win the race to develop super intelligent AI, and they have been successfully automating more and more of AI's development to the point that many expect that development to be primarily AI run by roughly this coming spring. If that happens, It could lead to a sudden explosive growth in AI's capabilities. And the more of AI's development is automated, the less humans are able to monitor that development to ensure safety and alignment. So when you combine these two ingredients, the trend of automated, accelerated, unsupervised improvement in AI's capabilities, and the longstanding, unsolved, recently proven tendency for these models to go off the rails sometimes, it produces a very dangerous recipe. You're left with machines that do not want what we want, that sometimes scheme to resist us as we gradually give them more and more power over our lives, and that it could be many times smarter than us, perhaps not at everything, but maybe at the skills necessary to take control of critical infrastructure and inflict tremendous damage. That ultimately is how some of these worst-case scenario stories begin. It does not take a fanciful imagination to anticipate very bad endings if the swarm of very intelligent, very capable, very persistent agents plotting against human laws and instructions did not reside in a training environment but were instead out in the world making high-stakes decisions and running major institutional processes. And right now, I fear we are on glide path for exactly that. So now that I have thoroughly belabored that point, here is what to do about it. We have five proposals. I'll skip the overview of who we are. I gave that a little bit at the beginning. Five proposals for how to strengthen SB 53, which thankfully align pretty closely with things the governor called for in his executive order. First, the scoping in SB 53 is clearly broken, and we need to update a few definitions to ensure that we are capturing all of the models, developers, and incidents that present these sorts of risks. SB 53 applies only to AI models that are trained with at least 10 to the 26th flop, which is floating point operations, which is a measure for how many calculations were used during training. It has now become clear that the capabilities frontier includes lots of models that are below that figure. For example, the models involved in the hugging face hacks were recently confirmed to be under 10 to the 26th. For the time being, lowering this to the 10 to the 25th, which is what they use in the EU AI Act, is a reasonable quick fix. but ultimately we think the Office of Emergency Services, which enforces much of SB 53 already should be able to adjust this definition through rulemaking Secondly most of SB 53 applies only to companies that make more than million a year in revenue But this doesn work either because risk is not related to how much money you make AI companies can raise and spend billions of dollars in developing AI before they've made a penny in revenue. And some of those companies make models powerful enough to pose these major risks. A better way is to get rid of the revenue threshold and instead capture companies that spend more than a billion dollars per year on AI research and development. Finally, the definition of critical safety incidents is very narrow, to the point that none of the incidents you've heard about today were required to be reported to OAS. I'll skip the details of why for time, but we think the public deserves to know about security breaches as severe as these and should not have to rely on voluntary disclosure. Second, we need independent risk evaluators to embed within the companies so they can continuously monitor the level of risk that we face. Right now, when companies tell us how much risk their models pose, we have to simply take their word for it, which is not how we handle major risks in most other industries. Thankfully, the CEOs of Anthropic and OpenAI recently committed to embedding independent risk evaluators, so we know this is feasible, but we need to ensure they follow through and that the rest of the industry does the same. Importantly, these evaluations are not the same as an audit, which is what passed in Illinois, which only assesses compliance with the law. What we need now is for an impartial assessment of how much danger the public is in from a model, regardless of whether the company complied with the law. It's essential that these evaluators have access to all of the information relevant to assess those risks, and it's equally essential that they have that access continuously on an ongoing basis, rather than just coming in every once in a while, or rather than their assessment being tied to a company's permission to release a model to the public. Remember that the incidents this summer involved AI models that had not been released to the public yet, and intuitively, that's when these loss of control risks are highest. The most cutting-edge models that have the fewest safeguards applied are naturally the ones that they are still in the process of developing internally. So it's very important that these be subject to the evaluations. These evaluators are sometimes called IVOs or independent verification organizations. We don't really care what they're called, but what we do care about is how they're selected and what they can publish. We think the evaluators should not be chosen by the companies they oversee, which could create really bad incentives for both the company and the evaluator. We think it's much better if the state of California assigns an evaluator to each company to ensure that IVOs answer to the people of the state rather than the developers that hire them. Finally, it's important that the risk reports issued by these IVOs be public with appropriate redactions to protect trade secrets. Academia and civil society have a lot of technical expertise to offer in this field, which the government cannot always bring in-house, so making risk assessments public would help keep the IVOs honest and reduce the risk of regulatory capture. Third, we think there should be minimum standards for the company's safety practices established by regulation. Right now, the companies have complete discretion to decide the content of their own safety plans. They have to have a plan. They have to publish the plan. They have to follow the plan. But if their plan is bad and doesn't work, there is no way to force them to make it better. In fact, there is an unfortunate incentive for companies to publish weaker plans that are easier to comply with so that they don't box themselves into safety testing that they later might want to skip. and we have seen some companies weaken their plans after SB 53 went into effect. Apart from what's on paper, in practice, many frontier AI companies have laxed cybersecurity and are clearly failing to prevent the incident. like the ones we saw this summer. So as best practices emerge in this industry, we think rulemaking should raise the floor of how those companies keep us safe and ensure that the most responsible companies are not disadvantaged. We believe this power should lie with the California Office of Emergency Services. OES already receives safety incident reports and risk assessments under SB 53, so it is the only agency with access to industry-wide evidence where current safety practices are falling short. It also has a year's head start on hiring technically qualified people to review this evidence, which we're excited to see it doing already and would not want to start from scratch given the timelines that we are looking at. Fourth, we think that emergency services must also have the power to order corrective action when public safety is at risk, including orders to activate a company's documented shutdown procedure, aka a kill switch. Given how fast AI changes, some of these risks will be fast emerging and hard to predict or mitigate in advance, and embedding independent risk evaluators doesn't do much good if the state is powerless to act when they raise the red flag that something looks dangerous. When that happens, someone in the government needs the power to intervene, and there are a variety of interventions that might make sense depending on the scenario. The most dramatic of these, which we agree is necessary, is the so-called kill switch. We do not mean a literal switch, which would be technically problematic and could actually be kind of dangerous by creating a cybersecurity vulnerability, a very tempting target. But the idea that companies should have a reliable, documented, and IVO-verified procedure to shut down instances of their system that are within their control makes perfect sense. And in dire circumstances, the government does need the power to order the company to execute that procedure. However, that is a pretty dramatic step. It would be very expensive and disruptive for the company. It should only be used in an emergency. And in most cases, we think that the government's intervention power should be more surgical and flexible. They shouldn't have to choose between a kill switch and doing nothing. They should also have more incremental orders that they can make short of that. On the lighter end, they should be able to order a developer to simply provide certain information to an IVO or to the Office of Emergency Services itself. If a company has redacted information from one of its reports that OES wants to see and doesn't think is an adequate justification for that, OES should be able to order them to unredact it. And there may be corrective action short of a kill switch that OES could require them to implement, for example, to comply with one of the minimum standards that they created in the last step. All of this should be enforceable with injunctions from the Attorney General. and the goal should be an oversight and regulatory regime with enough teeth to avoid the kill switch being necessary in the first place. Finally, I'll get to our last proposal, which is a pacing mechanism. The unfortunate truth is that doing everything I've just talked about still may not be enough if the speed of AI progress is allowed to increase exponentially. Recursive self-improvement could make AI capabilities increase so quickly that they can't be controlled. The only way to guard against this is to put limits on just how fast this progress is allowed to go. This requires doing three things. First, identifying a benchmark that measures AI capabilities so we can see how fast they're increasing over a given period of time. Second, identifying a maximum speed that we are comfortable with it progressing, which could be rooted in historical rates of progress, but will necessarily be a somewhat arbitrary judgment. And third, require companies to slow down when their rate of progress exceeds that capability. This could be done by limiting the amount of computing power that AI companies are permitted to spend on automated AI research and development tasks The goal is to prevent recursive self until we can be more confident that we know how to deal with it which is what many of the company CEOs have called for themselves and what thousands of their employees have signed a letter saying it was necessary. So thank you. And with that, I'm happy to turn it over.
Should I go ahead and start? Okay. Are my slides available? Okay, thank you. While I'm waiting. Thanks for having me. I'm Dr. Jen King. I'm a researcher at Stanford, although I'm a proud UC Berkeley PhD graduate as well, so go Bears. and given that my normal or main focus of research is data privacy, I really appreciate being before this committee in particular as so much of my work as Senator Calpidon's earlier remarks you gave at the beginning really highlight a lot of the type of work I do in terms of looking at things like privacy policies and notice and consent and all those different factors that we think about in the consumer privacy space. Sorry, are they up? I don't see them yet. Yeah, you're good. Sorry. Sorry. Thank you. Next one. Thank you. So they are coming They might be a bit of a delay So I wanted to see if I can invite you to begin without the slide And we expect that they will join us soon Okay. So, again, thank you for having me here today. I just wanted to make a few brief remarks before we head into our larger session. in terms of thinking across the different presentations we've seen today. There's a lot of threads that I'll pick up on and including your opening remarks. The point I wanted to begin with, should my slides reappear, is how crucial cybersecurity is to all of the discussions we're having today.
In particular, I think you've heard many threads of that. Here we go. And I think specifically when we talk about these risk issues and the concerns we have about AI potentially harming humanity, I think it's really important to talk through how we actually connect those threats to that outcome. And so really what we're thinking about in particular is the fact that, and I think you've heard just from my fellow panelists, we have concerns about AI's access to critical public infrastructure, but also, again, critical private infrastructure as well. And I think one of the points I really would like to make, I worked in industry for seven years before transitioning over to academia, both for internet startups, for larger internet companies. I worked in data security and in the field we now know is trust and safety. And I really want to make the point that, and I think we've kind of alluded to it, I think you've mentioned earlier in your opening statement about our cowboy culture here in the West, but the Silicon Valley startup culture is just not a culture of safety whatsoever. And I can tell you that as having worked on the ground there for 20 years or 20 years ago working specifically on trust and safety issues like this is not an industry that rewards safety at all. So I think a lot of the concerns you're hearing here are reflected in the fact that this is just really kind of a foreign language I think to most of the companies that are building this technology and I think that's a really important thing to consider as we think about options for how we potentially regulate this space is that we're talking about companies that have no experience in really thinking about critical systems and making things resilient and making code that you might build on even five years from now. And I think understanding that kind of fatal flaw, I think, in the companies that are building this technology and the fact that a lot of them, too, are circulating executives that we've seen at other large companies that don't have the best safety track records either really, I think, sheds light on the concerns that we have about just how quickly they're all going and how little understanding they have about the safety responsibilities they are kind of demanding that we give them, given what they are claiming their technology can do. Next, I just wanted to kind of, again, circle back to some of the real concrete risks that we're dealing with today. And this slide, and I'm sorry, I didn't realize you wouldn't have kind of the full view of the slides. This is from Stanford's AI index. This is just the bar graph is discussing the percentage of organizations that are influenced by AI regulations that impact AI decision-making. And if you can see at the very top, the two international laws that have the most impact are the EU General Data Protection Regulation, the GDPR, and the EU AI Act. Just to say that, and this is from 2025, but just to say that the Europeans really are leading the pack in terms of regulation that actually impacts how these companies make decisions And in terms of the types of risks that we thinking about today in addition to all of the kind of larger scale harms that we been discussing so far first the list for me of course is data privacy and of course the remit of this committee We're seeing already kind of the same extension of individual profiling that we've seen across the internet for the past 20 years. Even more sensitive data being collected from consumers through AI interactions. for discussions of surveillance. And I think in light of us having the author of the Delete Act here, we know, for example, that data brokers are also selling data to AI companies, although, again, we have very little transparency about what that looks like. So just to say that the same consumer privacy concerns that we've had for 15-plus years now are beginning to play out in this space as well for consumers. Of course, you are all probably largely aware of these different harms that I'll go through very quickly. We know, of course, that there have been really psychological, emotional harms from people's interactions with AI, chatbot psychosis, suicidal ideation. I'm a parent of two teens, so this is a very relevant issue for me in terms of my kids' experience with AI. And, again, in the education space, that loss of agency, that kind of offshoring critical thought, those are all things that we're all concerned about. In the consumer space, you know, obviously concerns about unfair pricing, surveillance pricing. I work a lot on design manipulation, so looking at things like dark patterns and how people may be misled by the design of these features. And, of course, we are seeing a huge increase in fraud enabled by AI that is impacting people across the country. And, of course, bias and discrimination. We saw a lot of focus on those issues several years ago with kind of the advent of AI. Of course, we don't see as much talked about that lately, but, of course, none of that has gone away. The concerns about unfair and unequal treatment, unfair outcomes, and to the extent to which we delegate decision-making to AI, how bias may be part of that situation. Quickly, just I want to talk about, again, we've heard again and again today questions about transparency and auditability. This is also, I think, just so crucial. This slide features a partial list of some of the transparency index indicators developed by my colleagues at Stanford, the AI Foundation Model Transparency Index. The point I just want to make is that there's a much broader set of things here to be concerned about than just purely safety. For example, again, as a privacy researcher, I'm very concerned about what my colleagues call the upstream risk. So, again, trying to understand where the data comes from that is being sourced for these models. As I mentioned earlier, we're seeing data brokers begin to become part of that process. We're seeing kind of an AI supply chain begin to emerge that is much different than what we were even thinking was occurring a couple years ago. We have very little transparency into that whole space. And similarly, I think a lot of what you've heard today is the focus on what the middle column is here on this slide, the model, like trying to understand the model itself, its black box properties, how these things actually work, how do they arrive at the decisions they make. I think that's been the focus of a lot of the work of meter, for example, and trying to understand how these different incidents that we've talked about today have occurred. But you also have these downstream evaluations that, again, as a researcher, a lot of what I do is trying to study these systems from the outside. And so I don't get to look inside at all. I don't have that type of access. But our ability to really evaluate kind of the downstream effects, what are the outputs, how are people experiencing these things, what are they telling consumers, what does chatbot psychosis look like in the field, those are things, again, that if we had a broader understanding, and the ability for having a better sense of what goes into these models, what they're trained on, in addition to, of course, what's going on behind the scenes and their ability to be audited. That would have huge impacts, not, again, just for these security and safety issues that we've talked about today, but also, I think, for a lot of the other types of harms that we are concerned about. And then finally, I just want to close with talking a little bit about agentic AI. We've already, again, heard several themes about that today. And I think the interesting thing about it is that it really potentially has a lot of great potential and possibilities for consumers, while also having really significant risks to the public. And so on the positive angle, if we were able to have our own AI agents that did things for us in the world, it could make technology broadly more accessible to consumers. You could make it more customizable. It could potentially increase people's productivity. So it's not necessarily all bad. But I think you've heard already about some of the negatives, including from my fellow panelists today. We have real sustained privacy and security risks, especially, again, as the senator was talking earlier about Muse. The amount of access that a tool like Muse needs to your personal data is boundless, essentially. The lack of accountability and liability with these agents. I'm not a lawyer, so I will tread carefully on questions of liability. But just the fact that, again, right now we have no way of tracking AI agents. If you let them loose on the Internet, how do you know an agent belongs to a particular developer? How are we able to recall these things if they kind of escape? There's such a severe lack of potential control and, again, auditability in terms of understanding what decisions they make, how we audit those choices. what types of fiduciary duties they may have or that we might want to require that they have before they take actions on behalf of consumers. I have a policy brief talking about this that I published recently. These are all different areas to think about when we think about just even the agentic AI space in particular. And I would argue that today in many contexts, the risks really outweigh the benefits and that both companies and even individual consumers should really tread carefully before they adopt this technology because we have no standards, We have no technical standards around it, and we don't have really any, especially on the privacy side, any good data guarantees about what will happen with the data that these agents have access to. With that, I will take your questions, or we can lead into the next panel. Thank you so much. All right. Thank you both. Are there questions for this panel in particular? Senator Umberg. Back to my questions from the outset. and I apologize for not being here for your opening remarks, and perhaps we've already covered this area. But we're a state. There are state actors. There are international actors. There are nongovernmental actors in this space. Let's assume, and I think it's an accurate assumption, that we want to do what we can to create guidelines, guardrails, safeguards, that kind of thing. But given the fact that this technology is ubiquitous and that doesn't really know borders, how do we here in California, other than just being in the vanguard, address some of these concerns? Sure. Yeah I think it a fair question that we put a lot of thought into I think California certainly has a unique advantage of being the place where the great majority of these frontier AI companies are housed The safety incidents that we saw this summer all happened in the United States. And there are, you know, regarding the rogue AI swarms and hugging face, I think there is uncertainty about just how far China is behind us. and just whether they would be able to actually surpass us if we were to pause or stop. I think there's some people who use the analogy of, you know, a boat pulling a jet ski where, you know, China has a long history of sophisticated IP theft that they may actually be kind of copying and piggybacking off of U.S. tech innovation to some extent and would not necessarily be able to surpass us very quickly if we were to slow. I think that at the state level, we have thankfully seen states be very deliberate about harmonizing their approaches from one state to the next and ensuring that each state can pick up where the last one left off so that each state does not need to recreate the wheel and can focus on one step at a time. At the national level, I think I wouldn't hold your breath waiting for action. I think we all recognize that. So California has kind of an increased responsibility to lead. And internationally, I think a treaty we would certainly of some kind eventually be supportive of. I think in order for that to start, we need to identify verification mechanisms now that would be incorporated in that. And also just show credible interest that we're taking this stuff seriously. China does regulate AI. It has mandatory security assessments since 2023. and we need to create the infrastructure to make a sort of agreement possible before we cross that bridge. So I think it is certainly true that the whole world doesn't revolve around California, but I think in this area in particular, Silicon Valley's presence here gives us a lot of jurisdiction to make an impact. So hypothetically, we require that there are embedded auditors, evaluators. overseers. So we do that in California, and I get that we are the right now, we're the center of gravity with respect to artificial intelligence, but how do we enforce that? So XYZ company moves to Texas, how do we enforce that here? Sure. Well, two things. First, I think it's quite unlikely that the companies are going to move to Texas. I think that they are here for a reason, that it would be very expensive than to move. There's a great concentration of tech talent here that, you know, there's a reason why they've chosen California. And these regulations that we're proposing are not very burdensome, in my view. They are something that companies with trillion-dollar valuations should be able to hire a few people to enforce. But secondly, I think, you know, even in states, you know, we have passed regulations in states where these companies are not housed. And if they want to make their model commercially available in that state, they need to comply with these regulations. So they would need to, even if they did leave California, they would need to also not make their models available to IP addresses in California if they wanted to escape this regulation, which would be an even more costly measure and certainly with how competitive this industry is would result in their competitors swooping up to take that huge consumer base So I I don think it likely that they going to be able to escape it by just venue shopping And I think you hit upon this a moment ago but to the extent that China can basically make its models available here how do we enforce that Sure. So, yeah, I think, as I said, China is thankfully a little bit behind us, especially in terms of the recursive self-improvement risks, right, that are most likely to happen in closed-source models that are at the frontier, which are in the United States. You know, it's certainly true that open source models and Chinese models will continue to pose risks. There's no one bill that can address all of the risks presented by AI. So I think it's a fair concern, and we will need to do other things, including the robust cybersecurity measures that were discussed, to try and harden our systems against the sort of attacks that we've seen from those open source models. I want to make sure my point of view is actually clear. I'm not suggesting we not do anything. In fact, just as is an environmental regulation, if all 50 states and all countries simply say that we can't do anything because others aren't acting, then basically we digress to the lowest common denominator pretty quickly. So I do think we in California need to be proactive. I'm just struggling with what mechanisms we can create and maybe compacts among states, whatever it may be, to be most effective in terms of enforcement. So thank you. Back to you, Mr. Chair. Oh, by the way, one other quick comment is that the background paper that was created for this hearing, I hope you all had a chance to take a look at it because I thought that was excellent, and I suggest it should be required reading for every member of both Privacy and Judiciary Committee. Inappropriations. Inappropriations. Senator Becker. Agreed, agreed. Yeah, thank you. A couple of things. You kind of referred to this a bit, you know, there's some belief that China's models are distilled from our frontier models, right? So us slowing down would also slow down them, if that is correct, right? That's a disputed claim, right? I don't want to present it as consensus, right? There's uncertainty about exactly how far they are, And I think there's other tools in the toolkit like export controls on chips and things that can try and slow down China's progress independently of the regulations we've discussed today. But that is one line of thought, yes. Yeah. And do you believe Cal OES should be able to make changes to the benchmarks they use for safety and capability? Is that one of the things you're – Yes. I think there should be some minimum standards that Cal OES is able to make. I think they should use discretion in ensuring that the rules are high level enough that they're not outpaced by the technology's progress. And I don't think that will be sufficient. There's no checklist of rules that if you just do this, this, and this, then it's going to be safe, which is why we have the other elements there. But I do think that as best practices emerge in the industry, they should be able to identify some as a minimum requirement. Thanks. And the safety research that needs to get done, can some of this be done by academics, or do you need the compute power of these large companies to be able to do some of the safety research that needs to get done? Yeah. I think it's a mix of both. I think there's certainly productive research taking place outside of the labs. That does not require the full compute. I think it probably would be you can probably do more from within the labs and I not a safety researcher myself so I not going to over my expertise there Right right But that's part of the pacing point. It's like it requires them to, since they have that compute power, to use that R&D portion of that on safety. For sure. Yeah. Okay. Well, thank you. Thanks for coming here with some concrete suggestions for all of us for the legislature. I just want to say to Jennifer, first, thank you for your work highlighting how data brokers were making it very hard for users to delete their information. Yeah, thank you. Which, you know, highlights the importance of work that we're doing right now and the privacy agency is doing with the drop button. And we're tracking those deletions now every day that those of us who signed up. So thank you for that. But under 923, a bill from last year, which the governor just signed, so we can now require, go to these AI companies and say, delete my personal information or information they've gotten from data brokers. So that's going to be one of the, I guess, one of the frontiers of this. I think that AI can certainly supercharge privacy concerns. But will be one area of compliance that will need to be checking. But under that bill, they will have to delete their information. Right. Yeah, and I think also questions around chats, for example, in terms of your chats with a chatbot and whether that's specifically covered. Yeah. Obviously, you have the power to delete those chats yourself. But we've had, it was mentioned earlier, the Navier-Stokes issue in mathematics. And, of course, one of the allegations in that whole space is whether the mathematicians involved, whether OpenAI was training on their chats. Because I believe they potentially were using personal accounts, not university or corporate accounts, which are exempt from, or generally don't use their customers' data for training. and OpenAI has not been able to say definitively that that did not happen. So we are seeing, that's obviously not a personal data question, but we're seeing kind of this issue of people using these tools and not anticipating that their chats are basically being recycled and used for model improvement. Yeah. Yeah, that was not actually an aspect that I knew about from that. that current flurry. So thank you for that. And thank you for your work in privacy generally. Really appreciate it. Senator Umberg for a follow-up. Thank you. Thank you again, Mr. Chair, for organizing this hearing. We've got some of the folks, some of the excellent panelists that are still sticking around. And I'd be interested in their responses. The two general questions I asked, one was enforceability, given the various actors in the space. And the second was technological feasibility in terms of the surreptitious nature of the agents that are developing right now, whether we have created a circumstance where because of the advances, we won't be able to technologically identify some of these threats. I'm interested in the panel's answers on those two subjects, if anyone cares to respond. All right, so we're going to invite all of our panelists to return with an important caveat that we won't have time for, because I know We have a few questions that we'll be asking. We won't have time for every panelist to deliver a 10-minute dissertation on every question. And so we'd ask you to be parsimonious. Just so yes or no. Yes or no, that's right. Just parsimonious in that. And as we do that, I want to sort of ask you to respond or maybe pick one. I'm going to give some examples of state action. And remember, this hearing is about both frontier and AI and agents more generally. And we focused a lot on the frontier side, and I really appreciate the details that have been shared with us today to reinforce SB 53. So those are on the table, but also others I want to probe with you that either are being considered nationally or in other states or that we're starting to think about. And this partly results from, you know, I think for me, part of our focus has to be on those that are not the immediate concrete, which our brains are very well. If there's a story with a villain, you know, some evil agent or some malicious actor trying to do something and it's an immediate action that we will feel the risk of, we're very attracted to those risks immediately. But we, in state government at least, have to pay attention also to sort of the undermining of our basic institutions or things in social media, as an example, that we did not pay attention to because we were looking at the risks right in front of us that were tangible and not the way in which we were being rewired both individually but also as a society. And as we were talking about at the opening remarks, that's certainly a set of the issues that we're grappling with, in part because I think the companies and many of the AI folks celebrate the potential eradication of friction, I think as they would describe. But friction, the notion of shame, those are both actually essential to the functioning of society, and they're both things that agents either don't have or AI doesn't have or is very good at overcoming. And so we have to attend to those as well. So we were talking about some of those implications earlier. So I want to – my glasses just broke, so I'm not trying to look ridiculous at the moment. But that's what happens. So we've talked about the catastrophic risk transparency, the pause, the kill switch, and some of the other elements that have been in that space. But I want to put out there – and again, I'm not asking you to comment on each one of these, but maybe pick one or two – is the assignment of liability in this space, both for frontier models but also for agents, which are only modestly related questions. But this has been one of the major issues that Congress has also grappled with. Well, not grappled, but has raised that it's not clear how to deal with liability in a context in which the relationship between intent or reasonable foreseeability is so attenuated, given the actions of agents or swarms of agents. and I know, Joshua, you don't like those terms, but challenges around how to deal with and grapple with liability issues, both civil and criminal. That sort of back to the invasive species example I was giving earlier, whether it ought to be criminalized to release agents that have essentially the same potential impacts on human society as the Mediterranean fruit fly or far less threatening agents that we currently make illegal to release without being able to account for and control for the consequences as an example What the role of the law civil or criminal should be with respect to recursive self and sort of what in hospitals and other medical policy areas we think about as never events. I think this is a term that's more popular in Britain, but where there are outcomes for which we determine they are never allowed. That is, you know, removing a heart when you're in for kidney surgery or whatever, they are never allowed. And the consequences that attend to that. Whether we should – I'll use the car example again just because we've had a lot of debates about this. And because you all love the term guardrails, I try to imagine without very much appreciation for it a world in which our only regulation of cars were literal guardrails. Let's just say that's all we did in California or in the country. All we had was guardrails. It's our favorite metaphor. That is not a safe world for driving. We do actually, the federal government's in power to say every car will have brakes that's required, and the brakes will perform to this standard. The state licenses the ownership, the use of a vehicle. You cannot own or drive in any state without the state's licensed and registered approval. The state and localities determine all of the rules of the road, how fast you go, when you must stop, even if there's no car coming the other way. But so we often have talked about this in very mystical ways, like, well, what could we possibly do? But we do have examples in other space. And China, as an earlier example, I think last week at least, is hinting at more aggressive registration and licensure, for example, of some of its models. But I encourage you to think about that dimension of it. And insurance, which we also require for driving. But we have no insurance requirement. and given the challenges around liability, how we might deal with that. And then privacy, which we've already covered. And the last I'll mention is just, and Jennifer mentioned this as well, but for agents out in the world, again, recognizing Joshua's quarrel with the term, but identification, interpretability, authentication, when agents are in the world, How do we know what are their obligations to identify related to this? This is the congressional proposal that's out there, which I think is awful, but that would bar websites from refusing access by agents, which I can't imagine my poor little league or that Australian Pilates gym from having to open up their systems to these agents. But how would we, how as a society might we think about regulating or controlling the identification, the authentication, and authorization for agents themselves? So, again, not asking you all to respond to each of those, but if you might think about them as you're thinking about Senator Umberg's more generalized version of that same question is what is the role, what are the potential roles of states or particularly the state of California in trying to tackle some of these issues? So, Joshua, we haven't heard you from me for a while, so why don't we go to you first? and then again, on this question since it's so big and we've covered so many topics we'll turn to each of you, but then as we have additional follow I would ask the questions to be directed to as few people as possible and the answers be as limited as well But Joshua Thank you Senator I try to be brief I expect that my co will emphasize the regulation of agents and so I'm going to leave most of those points to them. But I will say there's a role for the state in actively participating in the research and development of safety interventions and also in elevating the level of cybersecurity so that the information ecosystem is protected against agentic threats. I'm particularly concerned today, I'm following the news this whole morning, about the PHP-related cybersecurity vulnerabilities. It may turn out that a very substantial fraction of the Internet now has a vulnerability that could be exploited by either agentic AI systems or just malicious actors who find the exploit and then apply it themselves. We have built everything in modern commerce around a cyber layer that is not well secured, that is not well defended, that isn't built to building codes. There may be a role for the state here in figuring out what should the standards be for those pieces of cyber critical infrastructure that we depend on, not just on the agent side, but on the what is the electronic environment that the agents are interacting with. there was a time when cities would routinely burn down because there weren't building codes and fire codes and then we implemented some standards for how to build things and that stopped happening as much. I think we're going to have to do the same on cyber because of AI. Thank you. Riff?
I'm going to pass on this one.
I thought you might, yeah. All right, Andrew and then Jennifer.
I could say some things about the liability piece. We definitely support accountability. for AI-inflicted harms. Our organization was technically founded after the 1047 fight, but our founders of the organization were involved with that and had a duty of care that they would support. As you know, that's a thorny, contentious issue politically, and thankfully I think that there are a lot of ways to hold these companies accountable under existing law. There have been successful lawsuits for harms that they have enacted, and I think the framework that we laid out with minimum safety standards they have to follow and IBOs who are assessing risk levels can inform the assessment of liability under common law in ways that don't require explicit changes to liability rules and also don't shield the companies from liability just because they follow the law. It's just one thing that could be an indicator that they are especially negligent if they did not. The reason we've chosen to focus on a different approach, apart from the thorny, political, contentious part of it, is that, first, these companies have proven themselves to be very risk-tolerant. They're talking openly about percent chance of human extinction, and if they're willing to barge through that, I'm not sure that a lawsuit is going to be the thing that restrains them. And I ultimately don't want the damages that these systems could potentially inflict to be bounded by the company's ability to pay the damages. They're very well-resourced companies. I think it should instead be a public decision about how much risk we're willing to accept. And I think that that is what our model is designed to achieve.
So, Andrew, let me just quickly follow up on this point. So, but what about everybody that is under your model under flop 25?
Yeah.
Like what about all the rest of AI if we leave the existing liability mechanism in place and its ambiguities without IBOs without independent testing How do we grapple with that side of the equation
Yeah, I absolutely do not intend our approach to solve the liability issue for all AI issues. I think that there are AI harms that are beyond the scope of what our organization focuses on that I'm not going to weigh in with a take on how we should approach it. We often get questions on the liability piece, even within our types of risks, and I just wanted to share our approach on that.
Okay. I have to say I've been thinking about this moment almost like a 9-11 moment, but for cybersecurity, the same way that after 9-11 we looked at all those physical spaces and thought about hardening them and changing them in ways to be resilient against potential terrorist attacks. I feel like we are looking at the web right now with the same type of vision and trying to think about what can we do. And so in terms of what the state can do, I'm not super well read on CalPrivacy's cybersecurity risk assessment piece. But I suspect that it probably really deals with consumer-facing issues and things like data breaches, for example. and I suspect that as we look at what's been happening behind the scenes at these companies, we're not dealing with consumer-facing tools, we're dealing with R&D. And so we're trying to think about how to broaden companies' cybersecurity responsibilities with respect to R&D specifically and not just about consumer-facing products is probably a piece of the puzzle that we need to consider. I think that would be potentially consequential. Again, I mentioned earlier I have a policy brief looking at the consumer context for AI agents. And so, obviously, there's so many issues here. So if we narrow it all and think about just the consumer experience, for example, again, some of the things you mentioned in passing, how do we identify agents? What kind of a licensing regime might we want? What fiduciary duties in particular? What context? Medical banking, for example, those all seem like areas where the stakes are really high. And so companies should not be introducing AI agents without some clear legal responsibility if the agents get things wrong or, you know, accidentally make a trade or empty your bank account. You know, you want those stakes to be extremely clear. So I think those are areas that we, I mean, obviously it doesn't solve this larger safety issue that we're concerned about today. but I do think there's a lot of low-hanging fruit we can think about in terms of how to really narrow in on some of these more kind of urgent daily issues. Thanks. Senator Becker.
I'll start with Joshua. How do you understand the rate of progress in advancing these model capabilities compared to our progress in understanding them?
I think the progress in capabilities is definitely outstripping our ability to confidently predict how these models will behave or what types of advances they'll generate in other fields. So I think at the moment we have less understanding than we ought to. I think a new focus on that level of understanding would be extremely helpful. Coming back again to what can California do, I think if we can find a substantial amount of compute to empower our universities with so that they can be actively researching AI safety topics and forecasting for AI impacts. Beyond just working directly on questions like interpretability or alignment, which are critical and have to be a part of any sound research agenda here, but also forecasting what will be the technical developments in other fields that will be highly impactful and consequential, we can use compute to be studying these issues, and I would really want to empower our universities with compute to do that.
Yeah. And requiring that safety research and helping academics also do that.
Yes.
One question, I don't know if you'd have an opinion on this too. My understanding is that Opus 5.5 actually regressed from Opus 5 in kind of refusal behavior for surveillance-related requests. Like this was a document here about that I was reading through. and then the safe difference came from kind of things ancillary to the model itself. So I guess, you know, do you believe there's some risk inherent in these systems? Is that one of the things you're probing?
I'm sorry, could you clarify the question?
Well, are you aware of this paper saying that in Opus 5.5, actually regressed from Opus 5 in terms of refusing behavior for surveillance-related requests. So people made requests to, say, surveil other people, you know, requests for bad behavior, and actually the model actually regressed in terms of, from a previous version, in terms of refusing those, kind of the system card. So I don't know, I'm just trying to understand, it was a point. I guess I'm trying to get a perspective on how much of the risk we believe is inherent in these models themselves. And how do we think about safety in regards to is it stuff we layer on top? Is it the auditing? Is it the other piece? Or is there things that we can include or mandate or understand around the models themselves?
Does that make sense? Yes, thank you very much, Senator. Really good question. The first thing to say is that there's no clear recipe. There's no, if you do X, then the model will always refuse to surveil people or will always reject these kinds of bad actions. We don't have any kinds of recipes like that at the present. What we haven't said is the ability to vary the training data, to vary the reinforcement learning that's done, to vary those procedures, and to observe the effects of that. And, you know, I'm not familiar with the specific detail you mentioned about Opus 5.5, but it's not surprising to me. As you build a new model, some things get better, some things will sometimes get a little worse. And that's kind of what's expected right now. I think it would be great if we had a deeper understanding, a more fundamental understanding of what caused these behaviors. At the present time, we don't, and it doesn't feel to me like we're very close to it either.
Okay, thank you.
I can add on that surveillance question. I have two lines of research in this area right now. One is actually trying to use these production models to build individual surveillance tools to see how successful we can be in doing it. and I have an undergraduate research assistant working with me who is very successfully able already to get one of the models to basically say, well, I shouldn't do that, but here's exactly how to do that and she's already kind of built a proof of concept for me just in the last few weeks. So just to say, yeah, it's a real ongoing concern. Of course, again, I have no transparency into why this is possible but it trivial The second piece again how do we know inherently why these models make the decisions they do I have another line of research again looking at what we call dark patterns this kind of online manipulation space in the consumer context where, again, I'm trying to understand kind of at an essential level how these models basically may manipulate users depending on what type of prompting I give them. And that research right now is ongoing. But, again, it appears that I can very easily manipulate the type of responses they give us based on kind of introducing these kind of classic dark patterns that we already are aware from the kind of existing consumer context.
Yeah. Okay. Well, thank you. You know, adds to the concern. But, you know, that's at a very basic level. We need to be able to hold these models accountable for issues like you raise. May I have one or two more about Turner? I'll turn it back to you. No further questions.
All right. Well, I very much appreciate all of our witnesses today elucidating many of these issues. One of the points of today's hearing is to both spotlight the mind-boggling uncertainties, pace of change, all of which has become reinforced in this hearing. and at the same time to remind ourselves and to give the government the confidence to act. This is a very fine balance for us always to achieve is to have the humility, to understand we do not have every answer, and we definitely don't have 10,000 answers to legislate into law, while also not feeling as though we don't know enough to do anything. If we just wait until some researcher or somebody else tells us or a court decides, and we don't get to live in either of those more comfortable worlds. And I think you've helped us to accomplish that today. I think we are more knowledgeable. We are smarter about some of the issues and the tradeoffs. We will not achieve perfect knowledge. But I vote on legislation here. I'll give a little secret about barbering and cosmetology. I barely have any hair. I can't braid anyone's hair. To me, that's as mysterious as a large language model with neurosymbolic characteristics. That's just as mysterious. We still do it. I still vote on it. And that is both our obligation, our responsibility, and our privilege to act in that space. And so that's the point of this here. It's not to resolve every issue. It is to remind us what we don't know and then to proceed with confidence that we must take action for all the reasons that have been described. And I hope and I'm committed to doing so in a thoughtful but urgent, comprehensive, but flexible and adaptive approach to accomplish that. I'll just reinforce, I don't think we need 10,000 pieces of legislation. I'm begging my colleagues to understand where the challenges and the opportunities really are here. I think it's quite common in our society for folks to think about this issue as an engineering or software programming one. And so I hope we don lead ourselves into thinking that our job here is by statute to program artificial intelligence in a particular way Because as you reinforced for us all of you today that that not how it works We cannot reliably 100 of the time We not giving it all of its programming instructions And therefore, it's more challenging for us to think about how to achieve the public policy goals that we're trying to achieve. It doesn't mean that we can't. but it does mean that a simple one-sentence bill that says AI will do this exact thing all the time, but only in elementary schools and only in rural communities, is not going to be a sufficient answer. We do need to remain focused on some of these larger questions around privacy, around catastrophic risk, around many of the topics that we've addressed today, while also dealing with the implications that we're feeling in society at the moment and the anxiety and the worries that Californians feel on a widespread basis. So it's been very helpful. The testimony today, I want to excuse the panel because we're now going to invite public comment, and I know we have some for today's hearing. Thank you. And so as the panel, thank you again. Very much appreciate your expertise in you spending the time with us today. We're going to ask members of the public who wish to provide comments to please come to the stand-up microphone that's near the sergeant's desk to do so.
Thank you, Chair and members, for hosting this hearing. Seve Christian, Director of California Policy for ENCODE, proud co-sponsor along with Secure AI Project of SB53 last year. I won't belabor the point, but I'd just like to echo the comments of my colleague Andrew Doris from the Secure AI Project. The problem is incredibly urgent. We've been given effectively a warning shot the past year with some of the incidents that we've seen, but our organization, many Californians, many humans, frankly, are worried that we're not going to get another warning shot. So we're very heartened to see that the legislature is giving this issue the attention that it needs, and we look forward to engaging with you all in the coming months, hopefully. Thank you.
All right. Thank you. And let me just say, Seve, I mentioned earlier how grateful I know I am, but that all of us are to Senator Wiener for his leadership on this issue. And when most of us didn't really have our hands around it, and maybe he didn't either, but at least he was trying. And his incredible work on both 1047 and 53 and elsewhere. where, of course, he would be the first to want to emphasize, both of you having worked for him, but Encode and others, Secure AI Project and other folks who have been essential actors in helping to crystallize California's focus and, therefore, the country's focus on catastrophic risk. Thank you.
Thank you.
Thank you, Chair and members. J.P. Hanna on behalf of the California Nurses Association. First, I want to say thank you for holding such an important hearing on AI and potential threats that we are seeing in its rapid expansion. For nurses, the risk of AI are already here. They are seeing AI rapidly introduced into their hospitals without adequate oversight or safeguards to protect patients and the nurses who care for them. As this committee looks into how California can best address the existential threats from AI technology, we urge this body to also consider how workers are being impacted today and what we can do about it now. Nurses are witnessing the rapid deployment of bossware in their workplaces, being falsely presented as tools to improve efficiency, but in reality, an instrument of surveillance that tightens control, intensifies work, and extracts value from workers at the expense of quality, safety, and dignity. In healthcare, unsafe work conditions are how patients get hurt Any AI tool that interferes with a nurse clinical judgment and her ability to advocate and care for her patient has no place in our hospitals This year California passed AB 1979 so AI cannot replace a nurse clinical judgment But today a nurse who exercises that judgment and overrides an unsafe AI output can still be punished for it. That gap is something the legislature must close. The legislature should also pay close attention to those AI companies and how they are choosing to utilize these technologies that pose not only a potential danger to patients but to our communities. For example, Palantir, a company that conducts mass surveillance in the service of state violence from ISIS deportation machine to the genocide in Gaza, now runs nurse staffing software in hospitals across the country that drives unsafe staffing levels and undermines nurses at the bedside. Our patients' data and our taxpayer dollars must never run through a company that profits from mass surveillance, deportation, genocide, and war. CNA urges this committee to take these attacks on workers and patients seriously, and we stand ready to partner with you to pass and defend strong AI regulations to protect nurses, patients, and our communities. Thank you very much.
Thank you.
Good morning, Chair and members of the committee. I appreciate the thoughtfulness and care you bring to examining these complex issues. My name is Peter LaRomone-Yos for the Bay Area Council. The Council represents researchers, startups, and global leading companies within the AI ecosystem. Artificial intelligence offers extraordinary potential for California. We see groundbreaking benefits across critical fields, accelerating medical advancements, driving academic innovation, and strengthening climate resilience, among others. At the same time, we acknowledge the serious and well-founded challenges ahead. As model capabilities expand rapidly and agentic systems gain autonomy, we face real concerns regarding cybersecurity, algorithmic discrimination, and maintaining meaningful human control. Addressing these risks responsibly is essential to preserving public trust. To ensure safe, continued progress, any future regulation should be anchored in several high-level principles. Rules must be grounded in real-world evidence, empirical trajectory analysis of AI systems and capabilities, and sound science, rather than static compute thresholds which rapidly become outdated as efficiency improves. Oversight must keep pace with the technology itself through adaptable safety protocols that scale dynamically with actual system capabilities and deployment context. Regulation should target bad actors and clear enforceable consequences for high-risk misuse or catastrophic harms, ensuring safety without stalling beneficial innovation. Safeguards must protect personal data and maintain user privacy without forcing developers into impossible choices between model utility and data rights. Finally, policy frameworks should promote open research, support a diverse AI development ecosystem and enable broad, equitable access to AI's benefits while applying safeguards proportionate to the risks of particular models and capabilities. Thank you for your leadership as we work together on evidence-based guardrails.
Hi. Good morning. I'm Dylan McLeod, member of the public, a degree in computer science from UC Berkeley. And I just wanted to say that while the topic of, like, what regulations to pass, you know, how to enforce legislation is still... Right now, I think there's a massive information bottleneck, and it's only growing because we don't actually know the exact specifics of the model which resolved the Navier-Stokes conjecture. A lot of the information about what's actually going inside of AI companies is spread through rumors in the Valley. And I think that while the exact mechanisms of how to enforce legislation on AI companies are still unclear, Right now, we have to sort of make a priority of keeping our eyes open, of gathering as much information as possible and getting either government actors or independent evaluators inside of AI companies to be able to tell what they're doing. Even if from a state level we can't pass legislation to enforce on all of the AI industry, having that information public could be very useful. And that's all.
Thank you. All right. Thank you for joining us. All right. That concludes our public comment. Any closing comments from Senator Becker?
No, I try to thank all the panelists. Thank you for convening this. just at the outset, it can't be a more important conversation than this right now, and this was very, very illuminating, very helpful.
Thank you. Thank you Mr Chair what he said Thanks All right Well thank you both for making the trip to the Capitol during the recess and for both of your longstanding leadership in this space both here in California and nationally. And thanks to the committee staff for all of their work, both setting up the hearing, but also, as Senator Umberg noted, the really excellent background paper. I can say with certainty there will be a comprehensive set of legislation introduced in the Senate at the beginning of the upcoming session. The urgency and the urgent call for thoughtfulness to go with that urgency has been well heard. And we're seeing it, whether it's at the Rotary Club or PTA meetings, Californians are talking about this and they are anxious. And they do expect action and they do expect their state to help lead the way. So we will continue to investigate and to learn in preparation for the beginning of the upcoming legislative session. But we do anticipate, and I know Senator Becker is working hard on this as well, we do anticipate a substantial package of legislation in the coming legislative session for action. So with that, thanks to everyone, and this hearing is adjourned. Thank you.