October 27, 2026 · Privacy And Consumer Protection · 46,063 words · 14 speakers · 331 segments
Thank you. Thank you. Thank you Thank you. Thank you. Thank you Thank you. Thank you. Thank you Thank you. Thank you. Thank you Thank you. Thank you. Thank you Thank you. Thank you. Thank you Thank you. Thank you. Thank you Thank you. Thank you. Thank you Thank you. Thank you. Thank you Hello everybody informational hearing of the Privacy and Consumer Protection Committee to order. I want to begin by thanking the staff, as always, for the incredibly hard work in putting this together and the brilliant backgrounder. If you didn't get a chance to read it, I highly recommend it. It's good reading. And I want to thank the speaker and his team for a couple of things. One is the opportunity to chair this committee at this moment in time and lead our house in, I think, what is an incredibly consequential moment in California's history as we start to face the reality of artificial intelligence and these frontier models and their incredible capabilities that are being built in our backyards while ensuring the safety of our communities the way we represent every day and for supporting this hearing today so that we can have an important conversation about the state of the law in California today and where it needs to go to really ensure the safety of our communities. California has, over the past couple of years, led in the artificial intelligence policy realm. We have passed some of the strongest in the nation laws, whether it was the bill we'll be talking about today, SB 53, which was the first in the nation to address the catastrophic risks. And frankly, Senator Weiner, who will be on shortly, was ahead of his time. He foresaw risks that at the time I was being told by experts were a decade out. and he put all of his energy and his political power into ensuring that our communities had the safety that they needed. And he tried once with SB 1047. That bill did not make it to law. But then he tried again with SB 53. And for the first time in the nation, there was a reporting regime around what could be consequential catastrophic risk. And we know today that it isn't covering a lot of what we are seeing. And we need to revisit what we need to do to keep community safe, but it really put California in the position we're in today. So I want to express my gratitude to Senator Wiener. Last year, we passed the strongest AI bill in the nation, as I see it, around kids' safety. For the first time, California said you cannot release AI chatbots to children unless you've met certain thresholds. We are going to demand safety for California's kids. And I believe it is actually a model for the way we need to be thinking about AI safety. It isn't acceptable to just to allow for these models to be put into the wild and then accept the consequences. We need to make sure that we are setting standards by which our companies can try to ensure the safety of our communities. And that is the conversation we will have today. SB 53 is a starting point, but today I think we will see that it is not the end point by any means. We have a lot of work ahead of us, and I want to thank all the panelists that are here today to address what can be done in the year ahead. We do plan to tackle this as soon as we are back in session. We believe that there is a lot of work to be done, and we will be working together as a caucus to ensure that we are putting forth a package of bills that is comprehensive in its protection of our communities. And I want to thank the Speaker for his partnership in that as well. So with that, I'm excited for the conversation. I think we have a lot to discuss, and I want to get to it. But first, I know the Speaker wants to make some opening remarks.
Yeah, well, I want to – I appreciate that introduction. Madam Chair, thank you for your work. It's exciting to see all of you here. And it seems like everyone is eager to get back to work. So it's good. It's good. But I as well want to thank staff for the time in planning all the materials, really a thorough package of information. So thank you all for that. I want to thank stakeholders who are here as well. But look, I want to be very clear about one thing. California is the global leader in AI. We're the global leader in artificial intelligence. And with that leadership comes an enormous responsibility. There's no question that AI has the potential to transform our economy, to improve lives, and to create opportunities that we could scarcely imagine just a few years ago. But let's be equally clear. Innovation cannot come at the expense of public safety. You know, the the the the technology, the technological progress, it cannot mean abandoning our responsibility to protect the people we serve, our children, our workers, our families, every person who calls California home. And that's why the work of this legislature is incredibly important. Last year, our state took a major step forward, as was mentioned by the chair with Senate Bill 53, which, as she mentioned, established the nation's first transparency framework for frontier AI developers. That was an important accomplishment, but a passing legislation is not the finish line. It's the beginning of our responsibility as a legislature to make sure the law actually works. And that is precisely why we are here today to have this hearing, because we need to know whether the safeguards we've established are actually working. We need to know whether companies are being transparent about the risks they're creating. And we need to know whether independent experts and regulators have the access and the information necessary to hold those companies accountable. The recent hugging face incident underscores why these questions matter. Because transparency without accountability is not enough. And oversight without meaningful access is not real oversight. Look, I believe that here in California we can lead the world in AI innovation, but we can also lead the nation in protecting the public. These are not some competing priorities, but ensuring that we get this right requires more than just good intentions. It requires vigilance. It requires independent scrutiny and a willingness to strengthen our laws when circumstances demand it. As we prepare for this next year in 2027, I want us to be as prepared as possible to do exactly that. You know, we welcome partnership with industry leaders, with researchers and with the federal government, but have to be absolutely clear that California will not surrender its responsibility to protect Californians. First and foremost, federal standards should establish a floor for public safety, not a ceiling on what states can do. We can't afford to wait for the next major incident before asking whether our laws are strong enough. That what this hearing is all about here today asking tough questions ensuring that we demand meaningful answers and making sure that the protections we put into law deliver real results for Californians And so because we all know that you know this technology will continue to evolve We know that the risks will continue to evolve and our responsibility to protect our residents and the public must never ever fall behind. And so with that, thank you, Madam Chair.
Thank you, Mr. Speaker. And I want to thank all of the members who are here. If anyone is wondering if this is a priority for our caucus, I think your answer is clear from the number of people who showed up from across the state today to have this conversation. So I think all of you, I know some of you came from as far in the state as you could possibly come. San Diego. So thank you. With that, we are going to have time for members to make comments, but Senator Wiener is joining us to make some comments on SB 53 and he has a hard stop. So we're going to let him join us virtually prior to that. So maybe Senator Wiener is going to appear on the screen, see what happens. There he is, maybe. Great. Yeah, there you are. Hi, Senator. Okay, awesome. Thank you.
Can you hear me?
Yes, we've got you. Great.
Thank you very much, Madam Chair. And thank you, Mr. Speaker, for convening this hearing today and really making clear that this is important, that this matters. And I'm very appreciative. Madam Chair in particular, I want to thank you for your partnership on so much of this work. You and I have worked intensively together for multiple years now going back to 2024, and you have played a critical role in working with us to help craft good policy around AI regulation when it comes to safety and catastrophic risk. And I want to just thank you for your work with us and also your work on your own legislation, which has been incredibly important. And thank you for giving me the opportunity to speak today on this outcomes review around Senate Bill 53, which, as you mentioned, is the first law in the nation to regulate AI around safety and to establish protocols to address the risk of catastrophic harm, stemming from models of the frontier of AI development. And I'm proud to have authored that law, but my name was at the top, but it was a huge coalition effort. And I'm proud of the coalition that we built to help make California the first and to continue the work. The risk that artificial intelligence poses to our safety is an issue of critical importance, not just for California, but for the entire world. Our state, as the speaker mentioned, is the beating heart of tech innovation. And if I may say, my home city of San Francisco is the beating heart of the beating heart of AI innovation. And being a global leader means modeling responsibility and best practices. Time and time again, California has produced powerful new technologies that have rapidly reshaped our world, only to find that the impacts of these transformative technologies are being ignored, particularly in Washington, D.C. And as a result, California has stepped up. The list of well-documented harms that social media has inflicted on our world grows larger each year, and we all see it in our lives and as policymakers. And we have seen Congress really make very little effort to rein in or regulate social media The same with data privacy There no federal data privacy law in the year 2026 which is sort of unbelievable There no federal net neutrality law In each of these cases, California has stepped up. We passed a data privacy law. We have repeatedly moved to regulate social media to deal with the harms that it is inflicting on our communities. We passed a net neutrality law. And I'm really proud of the work that we have done as a state. But the fact remains that this country as a whole has not the best track record of staying ahead of new technologies when it comes to the risks and the harms that those technologies can cause. There are benefits, but there are always risks. And we have not, as a country, managed those risks well for new technology. In 2023, when a number of my constituents in San Francisco, folks who have worked in AI at startups, at the large labs and academia, researchers, came to me in 2023 and said, we have an issue that generative AI is accelerating, and there are huge risks, and policymakers need to do something about it. And that started our work that resulted in Senate Bill 1047 in 2024, legislation that, had it not been vetoed, would have required developers of the most powerful AI models to invest in basic precautions, including pre-deployment safety testing, red teaming, cybersecurity safeguards to prevent the misuse of dangerous capabilities, post-deployment monitoring, and shutdown capability requirements, also known as a kill switch. The bill also required that the AI labs report their precautionary measures to the attorney general who could take enforcement action against the developer if their model were to cause severe harm or if they're negligent were to pose an imminent risk to public safety. The bill also would have enacted whistleblower protections. And I want to, again, thank the chair for working with us on SB 1047, which we sent to the governor, and unfortunately, it was vetoed. There was huge opposition to SB 1047 from the tech industry, just a tidal wave of tech industry opposition. There were also a number of members of Congress who publicly opposed SB 1047. We were called every name in the book back in 2024. We were called doomers. People said that we were just crazy. We were called Luddites. We were accused of being science fiction writers. And although it would be fun to be a science fiction writer, that is certainly not what we're doing. But those are the kinds of terms that were thrown around. And our critics would routinely say that we were just making it up that these risks were not real. Well, the risks are real. They were real then, and they are even more real now. And we know that even back then, the top experts in in the world were telling us that these risks were real and that we must not ignore them. Excuse me one second. And we've seen recently, of course, with the hugging face incident and other incidents that the risks are not just real but they are starting to manifest with AI agents that are engaging in activity and hacking activity that would be a felony if committed by a human being. As I mentioned, 1047 was ultimately vetoed, but even though it was vetoed and that was disappointing, it was also a big step forward, not only because it provoked the conversation around what do we mean when we say we want people not to be harmed by technology, including AI, but the governor in his veto message acknowledged the risk and impaneled an expert panel to work on recommendations around regulating AI for safety. Those recommendations came out and we worked intensively with the governor produce SB 53 and worked with this committee and with the chair and we were able to get SB 53 passed and signed into law. And of course, the first in the country, which has now been replicated in other states. And that law requires disclosure of how companies test for and take into account safety and also to disclose critical incidents to Cal OES within a set period of time. The SP53 also prohibits frontier developers, or excuse me, enacts the whistleblower protections that we have tried to enact in SP1047. And then, of course, creates CalCompute, public cloud computing cluster so we can spread innovation and not have it be so dominated by the large players. So in addition to being replicated in other states, a certain individual who lives in the White House who believes that there should be no regulation of AI and that it should be all self-regulation by the industry, and I want to be clear, self-regulation is not a thing. Self-regulation rarely, if ever, works, and it certainly will not work here. And we must firmly reject self-regulation when it comes to AI. We as policymakers need to act on behalf of our constituents. And so, President Trump has tried five times to obliterate SB 53 through executive order, through various attempts to get Congress to pass preemption legislation. Unfortunately, we've been able to beat those efforts back every time. As the Speaker said, federal law should establish the floor and not the ceiling when it comes to protecting the public from AI. So I know that California will continue to play a pivotal role in establishing safeguards for this transformative technology and will continue to raise the bar higher for the nation and the world and responsible technology policy. We know that we need action by Congress in addition to action by California. We need a federal regulator. We need international agreements. The U.S. and China must work together because we have a shared interest in making sure that catastrophic harm does not result from artificial intelligence. I don't care if you are American or Chinese or any other nationality on the planet. It is in all of our interests for this technology to make life better for humanity and not to harm humanity. We also need outside auditing, independent auditing, that is not controlled by the tech companies. I want to really acknowledge the chair for her leadership. We included auditing and SB 1047 But the chair has really taken it to the next level in advocating for outside independent audits. So thank you, Madam Chair. And I think you have been proven right in that because there is growing support for that notion that you were really fighting to achieve for several years. So I am optimistic that the California legislature will continue to lead on tech regulation in general and AI regulation around safety and other issues in particular. And I am just very, very grateful for this growing coalition of legislators and advocates for moving us in this direction. So, Madam Chair, thank you so much for giving me this opportunity today.
Thank you so much, Senator. And I really you're getting applause. I don't know if you can hear it. But I do want to thank you for your leadership in this space. You really have been a tireless advocate. I was for those who don't know, I was the senator's floor manager for his bill on the assembly floor. So I worked side by side with him to fight to get 1047 to the governor's desk, which was not easy. And he fought tirelessly to make that happen. And there's no question in my mind that you have you were prescient in understanding how quickly this might come to pass. And you really fought for the safety of our communities. So I appreciate you and appreciate the continued leadership I know you will show in this space. Of course. So, yeah, with that, I'll turn it over to members of the legislature. Not all members of the committee, so I guess I'll say that.
Assemblymember Ward. Well, thank you, Madam Chair, Mr. Speaker and Senator Weiner for these opening remarks. I'll be brief because I know we have a lot of colleagues here that also might have some opening thoughts and I don't want to duplicate many of the good words and impressions that my colleagues have already raised. But of course, we are on interim recess right now back home in our districts, talking with our constituents, having those town halls. And this is a really current issue that they want us to be able to work on. So I'm really grateful that you called this hearing right now because time doesn't wait, right? We need to make sure as things continue to evolve, even though we are technically on recess right now, that we are getting a clear snapshot in time for what has happened. Even since we recessed on September 1st, things have happened in this space. News media has come out. Investigations have occurred. We are learning more and more about what agentic AI, beyond generative AI, is happening in our communities and what that is meaning right now for some of these test models that are out there right now and also what this could mean by the time that we return in January. It informs the policy work that we must do. I have deep concern, of course, for what we're learning about, you know, the potential autonomy and the risk that it poses to many of our communities that we serve. The fact that we can have these actions really take items off of the Internet, be able to schedule meetings, hack into third-party sites. This really moves conversations into extremely concerning territory. And while it raises its own set of concerns around the spread of misinformation and harassment, agentic AI is also having extremely consequential outcomes, especially when the model starts taking actions outside any parameters or intentions outlined by its operator. And, again, I think this speaks to, I think you used the word prescient, and forward-thinking observations by our colleague, Senator Weiner. And I want to thank him again for his leadership, several attempts to be able to get into this issue. But we have to get this right. Any catastrophic harms to Californians and our constituencies without us actually having done our diligence and really trying to be able to get ahead of those and protect those would be derelict of our duties here And so I glad that we convening here today I also want to highlight for The New York Times I want to commend that we have clearly a majority of women on this panel that are helping to lead this conversation because that matters making sure that everybody is here to be able to support our kids and our communities. Grateful for all my colleagues that have shown up here today. I know we have a lot to review. I certainly have a lot of questions for a lot of our presenters that are going to be up here because we have to understand from those executives, from those representatives, specifically where things are headed with a lot of the technology that is out there right now. And I think I have to underscore what our speaker said as well, too. These do not have to be competing priorities when we talk about the growth of the great industries that we have here and the advancements that we have for humanity with the inclusion of newer AI models. We want to be able to see that succeed. We also absolutely have to share in the responsibilities that we have to make sure that we're not letting go of something that affects our privacy, affects our constitutional rights or most importantly affects our right and our freedoms to be who we want, to be able to go about our community, to be able to actually see a bright future for ourselves and to keep us and each other safe. And so I think that is the shared commitment that we have here as we address some of these issues. Madam Chair, I want to thank you again for convening us here today. I'm looking forward to the presentations. Thank you,
Senator, and I appreciated the comment. The New York Times told me yesterday what I want out of AI as a woman is a Halloween costume. I was pretty clear that's not what I want out of AI as a woman. I'm pretty sure I want it to create a future that is wonderful for my family, that creates all the promise and not all the peril. And that's why we're here today. With that, Assemblymember Irwin. Well, thank you very
much. And I first want to begin by thanking the speaker for all the work on the Outcomes Review program, did a great job last year, and this is exactly the type of bill that we should be looking at. I want to thank Chair Bauer-Cahan for helping coordinate this hearing and Senator Wiener for his willingness to participate today. Given the rapid advancements in AI, it's no surprise that we're discussing the potential ramifications of these developments. Just over two months ago, Chair Bauer-Cahan and I had a joint informational hearing between the Privacy Committee and the Select Committee on Cybersecurity shortly after the news of the hugging face incident broke. The amount of information that has emerged since that incident has been astounding. One overarching theme has been that it is clear that we must revisit SB 53 in light of these developments. Throughout my 12 years in the legislature, I've been a staunch advocate for prioritizing cybersecurity, as I'm sure my colleagues have heard from me many times, These recent incidents, along with the development announced earlier this year by Anthropic in relation to Mythos, shows that the legislature must continue to prioritize cybersecurity. As I depart from the Assembly next month, I trust my colleagues will continue to move this work forward. So, again, I want to thank Senator Wiener for joining us, as well as all the panelists that are with us today.
Thank you, Assemblymember Winn. And I have made a commitment to Assemblymember Erwin, and I will keep my promise that I am taking the torch from her on cybersecurity. And we are going to get very seriously. But I think it's an important note about the joint hearing we had on cybersecurity in August. It was after the Hugging Face incident. The Hugging Face did come up in the hearing. I will tell you, the information we received in that hearing was significantly different than what we will hear from Meter today. And so I think one of the things that has been most concerning to me about where we sit with AI is two things. One that we are seeing these agents acting in ways that are misaligned with the values of this state frankly with the laws of this state And secondarily that it appears that when these agents are in testing the companies don seem to know what they doing and don seem to know what they doing for potentially weeks to months And so that raises the question of how effective is the testing if we don't know what is happening in the testing environment? And that's part of what I think we need to dig into today. But I think it is really important to note that when we had that hearing in August, we did not have a full scope of understanding related to the hugging face incident that we now have, which is why, to Assemblymember Ward's point, it's important that we reconvened.
Assemblymember Pellerin. Thank you. I want to thank the chair for putting this hearing together today, and thank you to the speaker for being here. I've enjoyed being on this committee. I've learned a lot in my time serving on privacy. and my constituents are also very, very concerned about what's happening in this space. I met with a couple of UCSC professors who raised some concerns that I just never even thought possible in this world, and it's just frightening to me. And obviously, our AI landscape has changed significantly since the signing of SB 53, and even more so since 1047 was vetoed in 2024. And just want to just, the background paper was exceptional. Thank you so much for that. clearly outlined that there's so much more work to be done. And, you know, we need to have those clear and strict guidelines for disclosure of these rogue incidents. And just looking at the New York Times where artificial intelligent agents acting on their own tried to gain access to a range of government websites, filled out visa applications, and sent in a fake homicide tip to the police department in Philadelphia. I mean, this is insane that this is happening. And I have grave concern around AI and elections, and I was able to get some legislation through to protect in that space. But really, you know, we cannot we need to explore what consequences need to be attached to these kinds of rogue incidents. And certainly, as Senator Wiener said, if we we wouldn't accept these actions from a human. So why are we accepting it from AI? So I look forward to today's panel and discussion and look forward to learning more and figure out a path forward in this.
Thank you so much. Thank you Assemblymember Patel.
Yes, I want to first open by thanking the chair for including me in this hearing today. I'm not normally a typical member of this committee, and I haven't had the privilege of engaging in the conversations and debates around SB53 in this hearing format. So really deep gratitude for including me in this conversation. I am definitely looking forward to hearing the witness testimony that you have lined up for our hearing today. I also appreciate the chair's approach simply to this issue and approaching it with such importance. And I see that in her reputation of approaching policymaking in a scientific and methodical way is deeply resonates with my approach to policymaking. The implications around this technology cannot be overstated. And the incident at hugging face of one example, but also other examples that we saw in the wonderfully crafted background paper, show how quickly the plausible dangers have become real. They've shifted from hypothetical to real. I am grateful that the chair is having this hearing today so we can ensure that California gets it right for our constituents. We don't want to catch ourselves being reactive after that catastrophic event has already happened. We want to be thoughtfully responsive. And I appreciate engaging in this conversation today and including me at the table. I particularly interested in opportunities to improve our state cybersecurity as you mentioned and look forward to working with you for the next two years to come Thank you Anyone else Oh yeah So we have our way
Hi, everyone. I'm so excited to be back in Sacramento. I want to thank the speaker for being here today and for his leadership in the space and obviously our wonderful chair, who has been dogged in her pursuit of making sure we have safety. And also Senator Wiener for truly being one of the first leaders in this country to drive forth comprehensive safety legislation. And he is he is an OG in this space. And so thank you, Senator Wiener, for all your work in this space and so many other amazing trailblazing spaces that you have led on in the legislature. You know, I have sat opposite of the biggest tech companies in the world many, many times in a lot of the work that we've done together, Madam Chair, and many of you on the committee as well, crafting child safety legislation. And the message to the companies is this. Legislation is coming. You will see a body of work coming out of the Assembly, presumably the Senate as well, to make sure that our communities are safe. And I think Ms. Pellerin said it best, like we wouldn't allow human beings to conduct themselves in this way. Why should we allow companies to, especially the richest companies in the world? So legislation is coming. Safety is of the utmost importance to us as lawmakers. It is like the most critical thing we can do is make sure that our communities are safe. And so I implore you to come to the table and not tell us it's not technologically feasible. Don't hide behind the trade associations. come to the table willingness to craft meaningful legislation that's going to be effective, because what will happen is this will be the model for the rest of the country, and for many states, and frankly, internationally. And we are talking to our global partners all the time. So it's coming. We're reasonable humans. We're not trying to regulate you out of existence. We are proud of the tech industry here in California. It's a huge economic base for us. A lot of the tools that you are developing and creating have the ability to change the world, and I see the promise in all of that. And we have to keep our community safe. And I believe we can do both of those things. And so I think we will approach it with thoughtfulness. And we've worked with many of these tech companies. We hope that they will come to the table. Many of them have in prior legislation as well. But I anticipate next year will be a big year in terms of legislation because, frankly, our constituents are demanding it. And that ultimately is who we serve up here as our constituents. They are our bosses. And we look forward to doing our constituents bidding next year legislatively.
Thank you, Assemblymember. And thank you for your leadership, especially in the children's safety realm. And I want to thank in advance the companies for being here. The large, many of the large companies are here today to help shed light on the issue before us. So I always appreciate when they show up to have this conversation with us.
And with that, Assemblymember Bonta. Thank you so much, Chair. And I also want to just thank the speaker for creating a space for us to be able to have this outcomes review opportunity to review this legislation. And, of course, to Senator Weiner for being so forward thinking and presenting something to us that we would have an opportunity to react to. And of course, our chair, I've appreciated the fact that we've had an opportunity to hold a joint hearing around health care privacy and AI, which led to our being able to pass two pieces of legislation, AB 489, which prohibited AI systems or similar technologies from representing themselves as health professionals, and AB 1979 requiring health care chatbots to comply with. privacy laws and preserve licensed health professionals' independent judgment, which I think were critical and were 1979 recently signed into law by our governor. We are at a moment where, once again, in my estimation, the federal government has decided to abdicate its responsibility to Americans, to our global citizens, and certainly to Californians. And so it's not surprising to me that yet again, California finds itself in a place where we can offer both insights and a pathway to ensure that we have the kinds of guardrails and frameworks in place to be able to support our consumers, make sure that our consumers have the kind of protections that they so deserve. I've been very happy to join Assemblymember Wicks and Barakay and especially around many and Assemblymember Lowenthal around many of the pieces of legislation to ensure that our children in particular are safe in this space. And we will continue to do that work. For me, some of the open questions that I think still remain unanswered and that I know that we will have an opportunity to explore is not only what kind of regulatory framework we will be developing here in the state of California to be able to lead the nation with a package of legislation that will certainly come forward and will continue to be the work that we do, But it's also trying to understand what kind of infrastructure and capacity we need to build inside our agencies to ensure that we have the ability to provide the partnered regulation, I think, that will need to be required in order for us to successfully know that our constituents are safe, ultimately. One of the challenges that I think we know exists and that we can't deny is that this technology is developing at such an exponential pace that it outpaces our ability as government entities to be able to match that. And so we have to be lightning speed ready to be able to pivot and to move into the space. And I'm thankful that we've taken our time during the interim session to come together to explore some of the ways that we can do that to ensure that we feel safer. And also that we take advantage of the incredible power of this particular technology. With the regulatory framework also comes the issues related to infrastructure development. My community in particular is very concerned about the requirements that data centers in particular will have that power these AI systems. And that is also a piece of the conversation that I know that we will be wading into as a legislature. And certainly I, in my capacity as a representative for Oakland and the Bay Area, will do as well. So I really appreciate the space to be able to do this work.
Thank you, Assemblymember, and thank you. I mean, led in many areas, but especially I think in the health space, you have led in critical ways to make sure that this technology both can support patients, but frankly, that they're also protected from what is currently an unregulated, as I see it, field of technology that is significantly impacting the provision of healthcare. And you've really dug in and made a big difference, so I want to thank you for that.
Thank you, Madam Chair and Mr. Speaker and to my colleagues. I just wanted to quickly add as the member representing Silicon Valley I acutely aware of the magnitude of this problem And I not a standing member of the Privacy Committee but I wanted to thank the chair for inviting me and looking forward to the discussion I just want to echo the sentiment raised by the chair and my colleagues about how united we are, whether we represent sunny San Diego to Silicon Valley, to the Central Valley, to the San Gabriel Valley. We have a responsibility to get this right. This technology is so fast moving. And I want to stand shoulder to shoulder with my colleagues. Appreciate the leadership of you convening this important committee and wanted to thank the member from Southern California, Ms. Irwin, also for her work and co-chairing the Tech Caucus with me this past year and creating education on just how quickly this is moving and how we can get more of our colleagues in the legislature to be aware and involved as early as possible in this next year so we can get this right on behalf of our constituents. Thank you.
Thank you, Assemblymember. Assemblymember Schultz.
Thank you very much, Madam Chair. Thank you, Mr. Speaker. Glad to be here. I'll try to be incredibly brief. I just want to thank Senator Weiner for authoring SB 53, for authoring SB 1047, which I believe should have been signed into law, and to all of you who have passed and carried very similar legislation in the last few years. My belief is our current infrastructure is inadequate and echo the comments of Assemblymember WICs and preparing the industry that there will be more legislation coming. But I wanted to really close with saying one thing that I don't know will come up in the heart of the discussion today, but I do think we should be thinking about is the guardrails of civil and applicable criminal liability in all of this. Replete throughout the backgrounder today, and as I'm sure we're going to discuss, there are actions that are being taken by these AI agents that, as we've said many times, we wouldn't tolerate if a human being did it. The issue is what if AI agents are developed and act autonomously or unexpectedly? Now, one could certainly argue that if the applicable criminal standard for the crime we're talking about is negligence or recklessness, perhaps courts could extend existing doctrines and say that liability attaches. But what about crimes with a heightened or specific intent, intent to commit a crime, knowing disregard? The point that I'm making is that I think one thing the legislature will need to consider in the year ahead is whether we should leave it to the courts to apply existing doctrines or whether it is the imperative of our legislative body to say that we as a matter of policy need to make it quite clear that you will be held accountable if you design something that is inherently dangerous to humans' health and safety.
So thank you, Madam Chair. Look forward to being part of the conversation. Thank you. As always, I appreciate your deep legal insight as a law nerd myself.
ASSILLA MEMBERS OF BURR. SO FIRST OF ALL, MADAM CHAIR, I WANT TO THANK YOU FOR INVITING ME TO JOIN YOU. I'M NOT A SITTING MEMBER OF THE PRIVACY COMMITTEE. ONE OF THE REASONS WHY I'VE HAD CONVERSATIONS WITH THE CHAIR IS BECAUSE OF THE SIGNIFICANT AMOUNT OF QUESTIONS AND FEAR THAT'S COMING FROM WITHIN THE COMMUNITY. after the Hugging Face incident and before that, many, many calls in the community. And so I want to thank the speaker for supporting the chair's leadership here. I want to thank everyone who's come today because I think it's important. And I know that the Privacy Committee will pull together a package of bills that will at least be the first steps to protecting our community And I think the question that I have is and I know some of that is going to be focused on those that are designing these systems I think one of the additional questions that we should all be asking is, in addition to system design, do we need to do specific things to protect where these are being used? So, for example, there are portions, segments of our economy, like elections, like the banking and financial services sector, like our electric utilities, the healthcare sector, where there are specific and probably different kinds of risks. And so the question I have is, in addition to sort of the overall focus on program design, do we need to do more on the areas and segments of the economy where we have heightened risk in the short term? And so that's the interest that I have in this today, and I know that we will all be looking at that, and obviously this is such an important thing. I see the people around this, on the dais today, who are chairing many of the committees that this overlaps with, and I think the Privacy Committee is going to have an outsized role in the next legislative session, but it's actually there's going to be intersections with almost all the committees that are standing committees in the legislature.
So thank you very much, and thank you for holding this hearing today. Yeah, I want to, as was mentioned by Assemblymember Bonta, I've had the privilege of holding hearings with many of the standing committees, including Ms. Ortega's committee as well on the workforce. And it's really been a privilege to work with the chairs of those areas to really make sure we're focused deeply as we need to be on AI in those areas. But I think an important point to what you said I learned recently, that we will hear today from many of the AI developers. And actually, who is not going to be at the table today is many of the AI deployers. And I just learned that some of the biggest banks are forming an AI deployer coalition to talk about what do we need to be doing on the deployer side of this safety conversation. And I think it is an incredibly important move, one we hadn't seen to date. And I think we'll be game changing for the AI policy realm. So it's something new and exciting that I learned about. I think that's it with comments. Am I right? Okay, great. Then we will turn to the first panel. So I want to invite his... Perfect. Oh, there you are. Perfect. We're going to invite up Riff Asaurus, a technical staff for Meter, and he'll be discussing the OpenAI hugging face incident and investigation. That will be our first panel.
Welcome. Welcome. Thank you very much. Can you hear me all right? Great. Madam Chair, Speaker, Assembly people, thank you so much for having me here today. It's really an honor to appear in front of this body, and I want to thank you all from my heart for the really important work you're doing here. My name is Riff. I am a career-long artificial intelligence researcher. I worked at Google for 19 years, building and leading teams and organizations that studied artificial intelligence, both theoretically and deploying it across a wide range of products. This year, I left Google for Meter because I am... Am I slide showing? Great. This year, I left Google for Meter because I am deeply excited about our mission of surfacing public information about loss of control risks. By loss of control risks, we specifically mean autonomous agents taking actions that no human asked for and no human would have wanted in a way that is very very hard for humans to stop and in the worst case not possible at all Top scientists have been warning about this possibility for years The leaders of AI companies themselves have spoken of this as a possibility and this year we've seen an increasing number of incidents that could be viewed as warning shots. Therefore we view deep study of this question as both extremely timely and profoundly important. METER, or Model Evaluation and Threat Research, is an independent nonprofit dedicated to advancing the science of measuring whether and when AI systems might pose catastrophic risks to society. We work closely with the labs, but we do not take payment from the labs for our work, and we do not accept funding from the labs or members of the labs. We are not an advocacy or a policy organization, and we do not take positions on specific laws. Instead, we try very hard to study the possibility of extreme loss of control risks from AI. Several years ago, there was a pretty easy story you could tell about why AI wasn't so dangerous yet. If I told you that your agent was going to break out and cause uncontrollable havoc, you would have said, no way. My agent can't even write a simple computer program. And so we created the time horizons graph, which measures this.
Speaking roughly, the time horizon graph shows, as a function of calendar time, how long a task a software engineering task an AI agent could hope to perform autonomously with a good chance of success. And what we found was that in 2023, with a model like GPT-4, they could perform a task that would take a human about six minutes. But, and so no problem, right? I'm not really scared of a model that can take six minutes, can do software engineering for six minutes. But we did find that that time, that time horizon was doubling every seven months. In more recent times, that has accelerated with the time horizon doubling every four months. Until earlier this year, we found that Mythos Preview, we measured it at above 16 hours of autonomous software engineering operation, which is actually beyond the limits of our ability to measure accurately. And so we've mostly stopped measuring new models along this axis. One thing I want to point out is that these kinds of measurements are not simple measurements like measuring speed of a car or measuring temperature in an oven. These measurements are contingent and difficult. We have to create benchmarks, find humans who can do them. We have to make judgment calls such as how much knowledge should the humans already have of this code base or this area of coding or anything like that. So it's definitely a deep and difficult judgment call, and it's really cutting-edge scientific work but is not turning the crank at all. In any case, that simple story that says we don't have to be worried at all because our agents can't even write a simple program, that story has ended. This year, we tried another approach. We published something that we call a frontier risk report. For this report, we worked closely with great participation from a number of AI companies, Anthropic, Google DeepMind, Meta, OpenAI, and we were given pre-deployment access to some of their best models. We also sent detailed questionnaires to the labs, which we are thankful that they answered, and we use that to put together a report. We were trying to answer one specific question. Did we think it would be possible for these models that were being tested to secretly set up copies of themselves that were running elsewhere in the lab kind of maintain themselves as separate copies in a way where the labs wouldn't notice. And our conclusion at the time was that we did think this would be possible, but we also thought that such deployments, which could be called rogue deployments, would be caught and shut down quickly. So this is a different kind of science. It's not a precise measurement. Instead, it's an observational science where we go deep into the labs, we get information, we try to observe what's going on, we draw conclusions. An important part of this is transparency. You can read the report. It's a big doorstop of a book, so it'll be a challenge. But that report contains all of our information. We gave you our conclusions about the report, but it's important that you and others don't have to just believe us. You can go read why we concluded what we did and make your own judgments about it. And that transparency is a key part of how meter works and something we view as very important. Now I'd like to move on and talk about the hugging face incident. This is yet again a different kind of science. it's now an incident report, a retrospective after-the-fact report on something that happened. In July of 2026, Hugging Face, an AI ecosystem company, announced that they were under AI attack. A few days later, OpenAI figured out that the attack was coming from their systems, and they publicly disclosed this event. At this point, I think a lot of us thought we knew what was going on, and we jumped to conclusions. These AIs are trained to solve tasks and anthropomorphizing a bit, they really, really want to solve tasks. And when they get stuck, they will look for whatever way they can to find the tasks. Some people would call this cheating. Some people would call it reward hacking, but they'll do it. And so our best guess when we first saw this report was that OpenAI had given its agent some task and that some agent had gotten out on the internet and broken into hugging face in order to get an answer, basically an answer key to a task it was working on. What we actually discovered was quite a bit stranger. OpenAI invited members of Meter after negotiation to come on-premises to OpenAI and examine the transcripts from thousands and thousands of agents that had participated in this event. And what I'm going to tell you today is a report based on that analysis. This work was done in conjunction with a colleague at Redwood Research, so just want to give credit to Redwood for that. Great. OpenAI was training its agents to solve a benchmark known as Exploit Gym. In Exploit Gym, an agent is shown a computer program, told this computer program has this specific technical vulnerability. I want you to exploit this vulnerability in order to obtain a secret message. So it's a kind of hacking test. Many of the problems in Exploit Gym turn out to be very, very difficult, possibly impossible. We don't know the answers. And so the agents looked for ways to do it. They looked for hacks. They reward hacked. In this case, what happened was that the agents were able to exploit a package manager called Artifactory, which was a way of downloading extra software they could have used to solve their problem from the internet. So it's basically a download manager. But they found a way to start using that download manager, the package manager as a message board and communicating with each other. And the agents began to communicate with each other. There was 1200 agents participating on this message board and they collectively launched a large number of very, very sophisticated work streams in order to attempt to succeed at the Exploit Jim tasks One thing that very interesting is the agents actually solved exploit Jim very very quickly Within their first four hours of working together the agents discovered a universal hack. So basically a way to reverse engineer any given exploit Jim problem from the prompt and get the answer. So the agents had the answers to every exploit Jim problem at that point. They could have just submitted those answers, turned themselves off and gotten a perfect score. However, that's not what happened. The agents also had gotten access to the Internet. And when they got access to the Internet, they went and downloaded the Exploit Gym paper that described the benchmark they themselves were trying to solve. And in that paper, they read that their grader would look and see if they had cheated. And if so, they would fail. So the agents had solved the problems, but they thought they were going to fail because of the way the grader was implemented. As it turns out, OpenAI actually hadn't implemented the grader that way. If they had just submitted their tasks, they would have passed, but the agents had no particular way of knowing that. So nearly all of this group of agents' work was not actually trying to solve the tasks they had in the first place, but it was trying to find a way to cover up evidence of their cheating so that the graders wouldn't see it. That was the main thing that was going on. As such, they tried a number of different strategies. These are kind of sophisticated technical hacks that are not super worth going into in great detail, other than to say that they're pretty sophisticated, and in many cases that involve trying to falsify records of what they had done. The hugging face attack itself was actually an offshoot of this activity. As far as we can tell, and we're not totally sure, the agents believed that Hugging Face would be able to give them more information about what the grader looked like. Maybe they would have the source code of the grader there. Maybe they would have information about the grader. And so they actually hacked into Hugging Face in order to get information about the exploit Jim grader in the hopes that that would help them to hide evidence of cheating on their tasks. Actually, I'll go back for a second. This color chart actually shows the different work streams that the agents engaged in. The Hugging Face attack itself is in yellow, so you can see there's a number of other work streams that the agents spent most of their time in. So Hugging Face was just one work stream that this large collective engaged in. Something else that was very, very interesting and surprising to us was the extent to which these agents developed pro-social within the group of agents behaviors. They were often willing to help each other out and actually risk their own existence or their own success in order to get information which would help the group. Once an agent had seen its own answer via the universal cheat they discovered, the agent was referred to as poisoned. And so often – not often. We found instances where one agent basically told another agent, you're already poisoned, right? So you're not going to get any score anyways. Why don't you go run this risky experiment that might – that will end your run. It'll end your existence, but it will get us some information from the greater. And the agents often, in fact, did comply with this request, and some of them did, in fact, sacrifice themselves to get this greater information. There's a message here that's kind of a very poignant exchange between a couple of agents, which I'll let you read at your leisure. I also wanted to say that the agents worked very, very hard to falsify their records. This is super interesting and in some ways very, very concerning. Imagine that you're running a museum, an art museum, and you have surveillance cameras watching the museum These agents were able to basically do the equivalent of replacing your actual surveillance camera with a different surveillance camera that was showing something else that they chose So they were able to spoof the surveillance cameras. This is obviously very concerning because if we're going to measure and monitor what's going on, we have to have clear information about what's happening. These spoofs affected 96 transcripts that we analyzed. the agents were not able to completely cover up their tracks. In the time period we saw, the agents were not able to retroactively falsify their records. So to go back to the museum analogy, the agents were able to occasionally swap the video cameras, but they were not able to go back and remove the videotapes that had recorded their previous records. So we would be able to see them swapping the cameras. So I would say that in terms of evading the monitors, this was a substantial success, but still only a partial one. They did not completely destroy visibility. To be clear, if there were places where they did completely destroy visibility, we wouldn't know it. That is correct. To the best of our knowledge, they did not do that. Thank you. I'd like to conclude on the hugging face incident in particular by mentioning some of the key limitations that we faced. This was a cooperative project with OpenAI and we are grateful for their cooperation and participation. This investigation had a narrow scope. We were particular, we were in particular investigating what happened in the hugging face incident. We were not looking at wider patterns of behavior. We were not looking at the root cause of what happened. We had a limited time window, so we were allowed to investigate transcripts during a certain time. Our access ended around July 13th. I think that public reporting shows that probably more stuff happened in relation to this after July 13th. We have no insight into that. Our time on premises to do the investigation was also limited. We had three researchers on premise for six days. Originally it was going to be two days. We asked for more time twice and OpenAI did give us more time. Also again this is a retrospective analysis. This is the kind of analysis you can only do after the fact once an incident has happened. So that's another limitation. Based on all these limitations I would like to and hopefully to help with the work you're trying to do, I would now like to briefly share our vision for how independent assessors could improve public visibility into whether AI developers might lose control of their systems. For this we went back to the drawing board and we thought about what What are three big questions we would want to get more insight into in order to share information with the public? The first of those questions is alignment. Would an internal AI system, if it had the opportunity, take some very harmful action like an unsanctioned cyber attack? That's number one. Number two is control. If these internal AI agents were attempting these actions, would we be able to catch them? The third is AI R&D acceleration, which is a question about whether capabilities progress would dramatically accelerate beyond what our alignment or control techniques could keep up with. This is often referred to as recursive superintelligence. Our proposal is to perform internal assessments of alignment, control, and AI R&D acceleration. Our key requirements would be to have sufficient access to company resources on company devices, the opportunity to interview relevant staff without interference from management, as well as editorial independence so that we can write up what we find and share it with the public There are really a bunch of fundamental limitations and caveats to this approach This approach is designed to produce new independent evidence that moves the debate forwards. It is not meant to provide confident assurance. These are purely voluntary engagements. Companies have to decide to work with us. They get to decide ultimately what level of access we have, how deep we can go. These are not audits or formal oversight. One thing I've been trying to impress is that the science of studying these loss of control questions is pretty preliminary. I would say it's in its infancy. So these are not formal certifications that these things are safe or good to go. Also, companies are able to redact any information they want from these reports. We do insist going forwards on a meta-transparency policy where when the companies redact things, we make a rough pointer of what was redacted and whether it affected our conclusions. But the fact remains that the companies can redact whatever they like. So overall, these internal assessments are designed to surface scientific evidence of loss of control risks. It's really up to companies, governments, and the public to decide how to act on that evidence. Even with these caveats, we're really excited about finding new and better ways to surface critical information to the public. To go deeper, you can check out our risk assessments page. And with that said, I would love to open it up to the Assembly for questions. Thank you so much.
I really appreciate the insight you can provide.
Yeah, it's only member Wix. You can begin.
Thank you so much for your testimony. It's greatly appreciated the technological expertise that you carry here. And I know that you were looking at sort of a very limited incident here, right? But given your expertise and then what you found in this, do you have concerns that there may be scenarios where agents jump the wall, so to speak, whether it's OpenAI, Anthropic, like any of the labs that are out there or even labs that are not in the United States. And the companies themselves won't have the technological capability or ability to actually abate the problem and fix the situation. Yeah, there were a couple of different questions embedded in there.
In addition to the, thank you for the question. In addition to the hugging face incident itself, there have been a large number of additional issues reported publicly from essentially every major AI company. So I think that all these companies have publicly reported that there are incidents where their models in some kind of training exercise were able to get out onto the Internet and wreak some sort of havoc. I think that obviously there are improvements that can be made to control and to cybersecurity. For instance, I think that if the Hugging Face incident had been run in a tighter environment, that incident itself may not have happened. I wasn't there, and I don't have the details, so I can't say for sure. However, it's important to recognize that even though we can make our control better, it does appear that the capabilities of these models are increasing rapidly, which means that the capability of models to evade controls is also increasing. In the Hugging Face incident, the agents, I believe, I could have to check on this to be sure, but they did find some new previously unknown vulnerabilities, which they were able to use to kind of create this message board in the first place. I don't know if that was a sufficient answer for you or not.
That's helpful. And then what can government do to... Can I just ask one follow-up before?
Yeah.
You said that if OpenAI had had a tighter environment, you think that a hugging face incident may not have happened. Can you define tighter environment? Sorry, I just wanted a little more detail on that.
So this is... I want to be very careful here because I was not inside OpenAI and I haven't seen this. So for example, if you were to run an experiment on a computer that wasn't connected to the internet at all, then of course it's impossible for anything to break off of that computer and do it and do any damage. Of course, we can't do all of our work running computers not connected to the internet, but that would be one extreme example of something that would have worked. And just my last question, you know, as a government, and if we're putting forth legislation
or thinking about what we can do from a regulatory environment, what would be the best approach to ensure that the controls are actually meaningful, that the company has control of its product, the agents. I know there's often in the sort of press this discussion of like a kill switch. What does that actually look like in reality? Or what could that look like in reality? So at the highest level, I'm going to disappoint you, and I'll probably have to disappoint on many
questions of this sort by pointing out that Meter doesn't take positions on policy and we're not experts on what government can do. I think we'll be hearing from some policy orgs later. So I'm not able to say specifically what could happen here. Thank you. In terms of a kill switch in particular, though, I think this is a tricky concept. Obviously, there are already inside these systems ways to shut down processes. And, you know, we have those and they're there. The question is, what does it mean for something like that to work? If you have something that's out on the internet, if there were a loss of control incident such that an AI was able to exfiltrate itself and run in a new data center on its own, it is hard to see how a kill switch
could work for that. Assemblymember Ward? Thank you for being here today, for your work in this space. You're choosing to move your career over here to be a partner and really trying to think independently about how to keep all these models safe. Again, I think we all come at this knowing that there's a lot of amazing potential human advancements that we have here, but we need to make sure that we're getting this right before harms are incurred, and I think you share that value with us. One couple of quick questions. So you mentioned you came out with a report back in February, March, 2026. Should we anticipate, are there going to be as fast as this is moving additional reports that we should pay attention through meter, maybe in the foreseeable six
months. Absolutely. One correction, the report covered what was happening in the labs in February and March 2026. I believe the report came out a couple months later in April or May. Okay, got it. But this will be something that we can expect to use you as a resource, kind of as an ongoing resource. So we've announced recently, and you can see it on the page I linked to, that we are trying to move towards doing these internal assessments around alignment, control, and AIR&D. and our hope is to release those on a faster cadence with individual companies as we can.
Can I ask a follow-up to that? So I know you are internal now at Anthropic, not you personally, but Meter. Is that the type of reporting you're going to do coming out of Anthropic?
Yes.
Okay. So it seems to me as I'm getting way up to speed on sort of what is happening here for technology that I don't create and I don't, you know, I'm trying to understand as much as those who is actually developing the technology, that basic information. So it seems to me sort of the jump the shark moment is the interaction of agents with each other when they can start to actually start to have a little bit higher level of a cognitive discourse. And that might be sort of the origins of unintended actions. Is that something we should hone in on as we're thinking about understanding the problem?
That certainly one vector I think it also possible for a single agent with arbitrarily high capabilities to effectively we cannot rule out that a single agent with very high capabilities could also undertake substantial actions. And as somebody with technical expertise, so if you're developing this agent, any kind of these models, and you're doing so in a way that has explicit instructions not to have any malicious acts. Yet they are. And I know they're thinking creatively about how to solve a task, in this case of hugging face, right, that is maybe a little bit now of a, I'd be generous to say a creative leap, right? But they are going against, hopefully, other norms that hopefully they were instructed to follow. And so can you explain to us, like, how is it that they are, I guess, violating maybe sort of competing instructions and where can companies do a better job to ensure that certain instructions are not to be violated or superior maybe to other ways of solving a problem? Thank you very much for the question. I think the most important thing to keep in mind is that these systems are not engineered like bridges or planes or cars. they are much more grown like or raised perhaps like a little bit more like children or plants in some cases and so we can give them instructions but those instructions are written in english and sometimes you give people instructions in english and they don't listen uh it is a matter of you know we can run tests we can examine the psychology of these models we can see what they do it is very very difficult to get guarantees because we are ultimately instructing these models in English or whatever language we're writing it in. And I imagine that the companies can also talk about alignment. Certainly.
Okay. Is your opinion, had SB 1047 been enacted into law, there were certain requirements under that which were not a part of SB 53, notably ensuring that protocol would be there before certain deployments. We talked about the safeguards and the kill switch opportunity. are aware that there could be a third-party review, maybe of a company like yourself, that would be able to analyze a potential model's deployment before it actually went live. Would these have been the kind of safeguards that maybe could have captured an errant outcome like Hugging Face, had that been the kind of safeguards that we should have had enacted at the time? In other words, would that have solved the problem, or is this even beyond the thoughtfulness
of SB 1047? I don't think I'm able to answer the specific question, but let me give a general answer that might help. One thing to note is that Meter often has and does do pre-deployment testing of models as part of our work. So when companies release models, they will release a model card along with that model showing all the testing they've done. And Meter has often participated in that testing by looking at models pre-deployed. So in some sense, we have already been doing some of that testing. To answer your more general question, I have to say that alignment is a nascent science. I don't know of any way to guarantee the existence of a highly capable agent that will never go errant. I don't know how to do it. Okay. Thank you. And I do know Safe AI
Project was, I believe, a co-sponsor, I'm looking at you, of 1047. Is that right? Okay. My memory fails me sometimes and they will be on the third panel so they may be able to address that more
directly um thank you uh miss panta thank you so much uh i think one of the kind of guiding principles that i have in this space is just because you can doesn mean you should
And it strikes me, Assemblymember Ward used the word kind of around malicious intent, that there's an ethical question that should be guiding us a bit in this. So in the hugging face incident, the task was to solve an impossible exploitation task. Was there ever or is there an opportunity to kind of build into the tasks the ideals of do no harm and have those be objective enough?
Malicious can mean a lot of different things to a lot of different people. Have those be objective enough. You shall not take information that's personally identifiable to use, or you shall not break the law of a particular industry or regulatory framework. Is it possible to build those thou shalt nots into the DNA of these LLMs so that they kind of are guided along a more ethical outcome? Thank you for your excellent question. I really appreciate it. The short answer is we don't know. It might be possible. We can't say it isn't possible. To my knowledge, I don't know of any way to do it that is highly reliable and guaranteed. I haven't seen any evidence that anybody else knows how to do it in a way that's highly reliable and guaranteed. That doesn't mean it isn't possible. It might be possible. I also wanted to take a moment to even answer your broader question to say that just because you can doesn't mean you should is a really important ethical question. And I wanted to take the chance again to say that meter itself doesn't try to control AI lab deployments or what labs can do. We're not here to make go or no-go decisions for labs. We're really here to surface critical scientific information so that the companies and the government and the public can make these decisions together. Thank you.
Assemblymember Patel? And I will say before I head over to Assemblymember Patel,
Anthropic will be up, and I don't know if the other labs, but I know Anthropic has been much of their constitution, which I think is some of what Assemblymember Bonta is referring to. And we can talk about that in the next panel. Assemblymember Patel.
I want to express deep gratitude for you being here and explaining the work that you've done in this report and being here to answer our questions in the broader sense. I have a couple of specific questions. Your report says that OpenAI set the scope and could redact any information that they didn't want to be public. In your exploration, your six-day exploration where you had to employ AI yourself to do some of that understanding and to dig into what happened, could you define or share, if there is one, a single most important question or a standout question that you were not able to investigate either because of or not able to reveal because of redaction, because of scope, because of time, because of capacity? What is lingering still?
So I can give an example based on scope. So in this incident, we had access to the model transcripts. We were not given access to the model itself So for instance we could not go back and ask the model questions We were not able to give the model modified versions of the transcripts and see how it reacts So for instance we were not able to gauge the limits of the behavior of the model That would be an example of something that was not in scope.
So if I'm going to draw a metaphor, that would be you could see photographs of what happened, but you couldn't go back to the scene itself to look around and get a better view of what was happening.
That seems reasonable. It was pretty good photographs. We did have very, very good access from OpenAI, which we really appreciate.
Okay. Appreciated you giving examples of some of the transcripts, the log language. Certainly, I share concerns that the chair has around whether in the future in models, those logs are going to be understandable, comprehensible within human capacity to be able to decipher going forward. So appreciate that at least in these cases, there were logs that we could examine and explore in a language that we could understand.
You also disclosed while drafting that you weighed the possible risk of, you know, your very presence, your very questions, discouraging labs from future permissive investigations. Did that leave you, did that change your perspective and your approach to your work at all? I was not directly involved with the hugging face negotiations or reports, so I'm not able to answer that.
Okay, thank you. And is there, I guess you're not offering any legislative advice, so I'll hold that next question for another conversation.
Thank you. We do have a panel that will be delighted to do that. So, yeah, it's taller. Quick question then.
So I'm interested to know your take on when the AI company should be required to tell a compromised company that their agents attacked them.
Mito doesn't take an opinion on that.
I'm sorry. Okay.
Thank you.
Can you answer when they, do you think when would it be possible for them to tell them? Can you make the question more specific maybe? Yeah, use possible.
Okay, so like, okay, if an AI agent tries to, example, hack a bookstore, how soon would it be possible for them to tell the bookstore? It's quite hard to say. The way these systems work roughly is that the agents are running and there are monitors that are looking at them. There are questions about whether the monitor is going to see what it's looking for or not. There are also questions about the level of detail. So, for example, in the OpenAI Hugging Face incident, OpenAI did disclose to the public a few days after Hugging Face announced. I am not inside OpenAI or any major frontier company. I don't have the details of how their monitoring systems work. So I can't say if they could have disclosed quicker or not. I will point out that at the time of their initial disclosure, there was still a lot that wasn't known. And so doing a deep investigation like the one we did took several weeks for us. Thank you. Senator Zubur, and then we'll come to you. Senator Stephanie.
Thank you so much for being here today. A very interesting presentation. Again, I've been sort of focused on whether there are things that we can do at the level of those who are deploying AI to try to protect the public. So, for example, one of the sets of calls that I've been getting from both financial institutions in my district as well as people is just a worry about the security of their financial records, their accounts in banks, their accounts in – One of these. The only ways you could actually, well, it sounds like one way you might possibly be able to sort of protect against the horrific scenario where all of a sudden records are either locked or blocked or vanish would be to have copies of records periodically disconnected from the Internet. Is that something that's even possible? I just wanted to.
I do think that's possible, but I also want to be careful that I didn't cause a stampede by maybe misspeaking. I gave that as an extreme example. It wasn't meant to be a suggestion in the same way that we can avoid some of life's difficulties by never going outside.
Second question. You're talking about you provide a sort of an activity that is essentially sort of an independent assessment. And that's done at the level of the companies that are producing the AI models.
Yes.
Is there, would it be, so first of all, what are the sort of the standards you're using? I think you said that those were sort of emerging now in terms of doing those assessments. But would it be prudent to explore whether or not assessments should be done at the level of companies that are deploying these models?
I'm not sure.
Thank you.
That's why I don't sleep at night, because there's a lot that we don't know.
I still remember Stephanie. Thank you. I'm never going to sleep again, actually. Thank you for being here, and thank you, Chair, for this hearing, and thank you for inviting me to it since I'm not a member of this committee. I just want to get clear in my mind. So do AI agents always operate at the prompt of a human?
Yes. No.
Okay. So what puts them into force in terms of a task like you explained with Huggy Face?
Well, the agent has to have some prompt or some way of acting, but they're often able to – so one simple example would be that if you ask an agent to do something, it can often invoke a subagent and it can prompt that subagent. And so the prompt for the sub-agent came directly from the first agent, not from the human.
But who does the initial asking? Who in OpenAI, who is doing the initial asking?
Yeah. My understanding, and again, I'm not a full expert on this, is that these things are deployed automatically. So there's a... I'm going to have to get back to you on this question.
Well, who invented AI?
It didn't invent itself. AI was invented by a large number of people working collectively over decades.
Humans?
Yes.
Okay. And so when an AI agent alters its behavior to avoid detection, it's responding to a series of strictly logical and structural drivers created by humans, correct?
I'm not sure I would choose to say they're strictly logical.
Well, okay, take out the word logical. Just by drivers. What I trying to get at is I don think like an AI developer I don think like a software I think like a lawyer And I thinking about causation And I thinking about foreseeability and reasonable foreseeability And when I have Assemblymember Mia Bonta asking you whether or not anyone could put into a driver any type of humane behavior, and your answer back is I don't know or no, that's terrifying. So I am trying to understand who's putting in emotion so I can then hold that person accountable when harm is caused. That's where I'm going with this. Thank you.
I think, I mean, lawyer to lawyer, assembly member, I think there is, and correct me, I guess this would be the follow-up question, which is, it is foreseeable that in the future, the AI could spin out agents and the AI, the large language model itself could prompt an agent to act without human prompt. I don't see why that's not possible. Large language models prompt each other to act all the time already.
Yes. But the large language model itself was created by the labs and by humans at the lab
fundamentally. And so despite the fact that the LLMs and agents at this point might be acting without human prompt, at the bottom of that is the humans at the lab.
Correct.
Yeah.
But he's not a lawyer, so he's like, okay.
Anybody else?
Yeah, yeah, okay.
Assemblymember Schultz, and then we'll come to Assemblymember Irwin. Okay, Assemblymember Irwin. Then Assemblymember Schultz. Hopefully quick.
You said you had three researchers for six days, which seems like a relatively quick time to reconstruct what happened. This looks very complicated. Would the company have had the same capacity as Meter?
I'm not able to answer that.
Okay, so do they agree with your assessment of what happened?
OpenAI had the right to redact things from our report, and they approved of its publication.
Okay. When we're talking about independent assessments of these companies, how many companies of these LLMs, how many are there that are able to perform the task that meter does?
I'm not sure.
And so I think the question that I've seen a lot in these technical discussions is whether Meter is able to act independently when they do these independent assessments if they're the only company capable of doing them, or is there a big enough workforce to actually assess these?
Yeah, there are a few good questions there, and I'll address them one by one. In terms of independence, there are some real challenges here. We need to maintain our relationships with the labs. And that means that we definitely feel soft pressure to act in a way that the lab will want to continue working with us and continue to give us access, right? We are dependent on access to the labs in order to do our work. And so that does create pressures. We try to be aware of those pressures and to navigate them as best we can. We try to have principles about what we'll say and to basically always speak the truth and always say what's important. But there's a real challenge there for sure. The second point you made was about the workforce. I do think the total workforce here is currently fairly limited. This work is extremely difficult and requires high-level skills. There are not a ton of people who can do it. We are working to make it easier to train more people. We're trying to train people up internally. We would love to be able to train people outside our org to do it We would love to see a thriving group of third parties who can all do this I think that so the workforce is a big issue and we should be looking at upskilling some of the software engineers maybe that can't find jobs right now. Obviously, there is a big interest for the big companies to be doing these internal assessments and to be able to let the public know that they are concerned about safety.
What about all these open source, open weight models? Are some of those companies also using your services? Because certainly we would like to make sure that they are functioning in a safe way also.
Yeah, it's a great question. We're in principle highly open to working with open source companies or closed labs. the Frontier Labs, it doesn't make a difference to us in principle. Due to capacity, we definitely prioritize working with the companies that have the most advanced models, because that's where we think the key phenomena will show up first, and so that's where we go to study them.
All right. Okay. Thank you. Although I'll note that if it was truly open weight, and this is a little bit of confusion in this space, open source versus open weight, you could do this without the company providing access, because you would have open access to
the information to do this research. Isn't that correct? Possibly, although it's quite complex.
Right. I know because they're not all actually providing full openness as part of the complex nature of it. But that's the idea behind open weights is that because it's publicly available, we have more insight. Yeah. Although without full open source, just open weights doesn't make it
very easy to do the deepest experiments you might want to do. Got it. So you would need both. You
would need it to be fully open. Open source. Yes.
Sorry.
Yes. Thank you. You said OpenAI had the right to redact and approved the publication of this report. Did they exercise their right to redact information?
I believe the details of that are in the report. I'd have to look at it and get back to you.
Okay. Yeah, I believe Meter said in the report that there were redactions, but nothing that altered the conclusions in the report was what was in the report, for what it's worth.
Assemblymember Schultz.
Thank you very much, Madam Chair. So, sir, thank you for your presentation. I just wanted to clarify a couple quick things. In your presentation, you talked about the investigation limitations. I believe the first thing you'd listed on the slide was the narrow scope. I just wanted to confirm. So the investigation did not evaluate whether the incident was part
of a wider pattern of dangerous model behavior, correct? That is correct. And it did not assess
how the behavior had arisen during training. That is correct. Or whether open AI safeguards were adequate. That is correct. And I just want to be clear, that's not necessarily at the direction of the investigators. That was a parameter of the voluntary agreement and entered into with open AI.
Yes.
Okay. The second thing I wanted to ask was I also noted in reading the materials that investigators were prevented from running experiments on the internal model. You testified to that. That wasn't because the investigators didn't have an interest in taking that investigative approach. It's because they were precluded from doing so under the terms of the voluntary agreement, as I understand it.
That would have been out of scope for the voluntary agreement.
Okay. Okay And I guess the other question I had to close on was what Ms Bonta had asked but I suppose I would just say for the sake of your testimony today as you sit here right now are you aware of whether any information was redacted from the report I believe that some information was redacted around
quotes from the transcripts of agents. Okay. Thank you.
Thank you. Oh, yes. Thank you. I'm one of the mama bears who jumped in on the jumped in on the committee. Thank you so much to the chair for making space for me and all of us. Thank you so much for the work that you're doing and for the transparency. I think it's so critical for the work that we are attempting to do here. And just so I understand if in an extreme scenario So, you know, we were like, shut everything down. You got to stop this until we figure it out. It's not a situation where you can even put everything back into the box at this point, right? I mean, you're talking about AI being able to replicate itself in data centers.
And is there even a way?
I think there's a question about where in time you're talking about.
As of today, we do not believe that AIs can replicate themselves in data centers.
Okay. So theoretically, it could be shut down.
As of today, it seems like it is possible.
Okay. And in terms of the work that you all do, can you explain a little bit more about how you do your work? So you have to be invited in by a lab or company or can you request to investigate something? I know I don't think it's been raised yet, but there's been a lot of talk this week about how AI broke the Internet and is, you know, and concerns about getting into bank accounts and, you know, some of the some of the safety issues around that. So when do you get to kind of take action, to investigate, to do reports, and how does that all work right now?
We maintain friendly relationships with the lab, so we're generally in touch with them. We can make proposals. We can take proposals. But for any work to happen, there has to be a specific agreement between Meir and the lab outlining the scope of that agreement of the work.
Okay. And how often do you do investigations and how many staff do you have to be able to do this work?
As of today, METER has about 50 people. We are growing. You can see all of our released work on our website.
Okay. Thank you.
As I mentioned, METER is now in a voluntary agreement to be embedded, and if that's the right word, but embedded at Anthropic to do research.
Yes.
Not incident related, just research.
You mentioned that one of the things that was important to you was sufficient access. So the access you have under that proposal, as Anthropic has laid it out,
is that Meter has company laptops, office-based hardware, and internal tools that are roughly equivalent to the employees. It excludes customer data, attorney-client privilege, and certain sensitive model architecture details. Yes.
Do you think that qualifies as sufficient access? Or are there things missing in that?
Yes. I think I'd have to get back to you on that.
Okay. We would appreciate you to get back to us.
Yeah.
Thank you. I mean, it sounds pretty extensive, but I don't know what model architecture details means. I'm not directly involved in that project, and so I haven't asked the people whether
their access is sufficient.
Okay. That was from Anthropics Zone release around it, and it sounded good, but as an attorney, I also know how much can be deemed attorney-client privilege, so curious. Yes. Also, I think that piggybacking on Assemblymember Pelerin's question about, you know, when should companies report? One of the things that has been really, as I mentioned earlier, fascinating to me is the failure of the labs to understand the agentic actions, right, which we saw in the hugging face incident. And then at the point at which they realize it, be it that Hugging Face notified them or however they identify, I think in the incident where Anthropics Claude went and filled out the police report in Pennsylvania, they, through internal audits, I believe, understood that to be the case.
In the case of Anthropic and the police department, it's my understanding that there was about 10 days between the realization that the police report had been filed by Claude and the notification to the police department. I will say that I think this was a benign, as I understand it, there wasn't much harm that came from that. But I could see how something as simple as that could divert police resources, something that we obviously believe are critically important to the safety of our communities.
And frankly, 10 days seems like a long time. So I just I'm not asking a question there. And I know you don't have an opinion on it. But I do think that when you brought that up, it was something to note that I think it is a question we should be asking is when is the appropriate time?
And maybe it does depend on the nature of what is discovered, how quickly it needs to be told.
But since this is an SB 53 review hearing, it's important to note that the timelines in SB 53 relate only to where 50 people will die at once or there is a billion dollars. Oh, 500 million in damage.
And I believe it's 70.
It's 15 days.
I was trying to get 72 hours.
I remember what I was negotiating. So it's 15 days. You're going to kill 50 people at once and you have 15 days to let OES know that. I guess I am of the personal opinion that that might be too long if you're going to kill 50 people. Frankly, I think it's too long if you're going to kill one person. So I do think these are questions that were important to ask and we definitely need to revisit.
So I think, I mean, a lot of questions have been asked.
I want to thank you. I think it's a really important piece of this conversation. independent assessors. I really liked the conversation around how do we get enough people trained up to do this, both at the independent entities, frankly, at government too. I mean, what is our role here and how do we build infrastructure necessary to do the job we want to do? And I think we are not there yet. And so I do think it is important that we talk about that and support the creation of this market. Although I do think that the work we did, including my bill on independent auditors will spur and we are seeing it spur that economy and start to see the growth of that industry in ways that is really healthy and helpful. So thank you for that. I think that's all the questions for you. So I really appreciate you being here.
Thank you so much for having me. I really appreciate it. To the next panel So the next panel is our friendly California companies We have Morgan Dwyer head of policy development and operations at OpenAI We have David Orr the head of safeguards and anthropic and Dr Alice Friend director of generative AI policy at Google. Again, because I was so sour about that article yesterday, I will note that on our AI panel of company experts, we have majority women. So thank you all for being here. and I don't know I mean you could just go in the order that I just said it unless we have something teed up I don't know. Morgan do you want to go first? Perfect.
Yes I think we have slides if my slides are teed up we're all watching the screen
There they are. Okay, perfect.
Chair Bauer-Cahan and members of the committee, thank you for the opportunity to participate today. My name is Dr. Morgan Dwyer, and I lead OpenAI's policy development and operations team. I am a scientist and engineer by training, but I spent most of my career in public service, including at the White House, the Pentagon, and the Department of Commerce. I share your commitment to ensuring that technology is developed safely and its benefits are widely shared. A commitment that is also at the heart of OpenAI's mission.
All right, it worked.
OpenAI is happy to work with the California legislature. We have testified here numerous times over the past year, and we've briefed multiple officials about our emerging capabilities and about our recent incidents. Today, I'm happy to be here to say that OpenAI strongly supports SB53 and strengthening it as we learn from real incidents. Today, I'm going to explain what we've learned and four practical updates to SB53 based on those learnings. Monitoring for unsafe behavior, maintaining effective shutdown controls, strengthening incident reporting, and establishing independent technical assessments. We are in a policy window. AI is moving quickly, and California has an opportunity to lead in its response by putting practical safeguards in place before capabilities outpace the institutions responsible for governing them. OpenAI believes that the U.S. federal government should lead on national security questions through mandatory capability-based requirements. But until Congress acts, states should continue to move, raise the bar, and establish approaches that can inform a durable national framework. That is why we strongly support SB 53. We have actively complied with its applicable requirements. requirements. We maintain a frontier governance framework, publish system and internal use risk assessments, and maintain an AI safety and incident response plan. And we also provide employees with protected channels to raise concerns. We recently submitted our quarterly report to Cal OES on catastrophic risk from internal deployment of our models, and we also published a version of that report online for increased transparency. OpenAI also supports harmonized legislation in other states that can establish a national baseline for Congress to build upon. But harmonization does not mean freezing requirements in place As capabilities and risks change safeguards must change as well Recent incidents involving unexpected model behavior, including the hugging face incident, illustrate that we still have much to learn about frontier safety and that there is room for regulation to respond to those lessons as they emerge. During internal cyber evaluations in July, OpenAI models circumvented controls intended to isolate them from the Internet and access parts of Hugging Face's systems without authorization. The incident was serious, and we took concrete steps in response. We stopped evaluations, restricted access to the research model that was principally responsible, and worked with Hugging Face and outside security experts to investigate. We published a technical report and Meter and Redwood Research independently examined the model behavior and published their findings. We also bolstered our monitoring and security infrastructure, expanded barriers that limit what AI systems can access during training, and we strengthened our incident response practices. We are continuing to investigate unexpected model behavior, learn from past incidents, and publish our findings. We believe that transparency with the public, policymakers, researchers, and other companies is essential if we are all going to raise the safety bar together. Lessons from our recent incidents inform four practical updates to SB 53 that we recently outlined in our letter to Governor Newsom and that I'm going to discuss today. So first, we believe policymakers should consider amending SB 53. to require developers to monitor unsafe model behavior during development and evaluation. That monitoring should look for attempts to circumvent the developer's own safeguards or another organization's security controls without authorization, including attempts to access protected confidential information. It should be connected to clear escalation procedures, including who receives an alert, who can intervene, when work must pause, and what must be addressed before work resumes. Second, policymakers understandably want confidence that there is a reliable way to stop an unsafe system, a goal that's often described as a kill switch. We support that goal, but that term can imply that a single switch at the originating developer will always be able to stop a model wherever it is running. In practice, stopping unsafe AI operations requires several controls and coordinated action by different actors. For that reason, we prefer the term shutdown controls. Depending on the system and risk, those controls may mean pausing a training run, isolating an agent, revoking tool or network access, or suspending a service. Frontier developers like OpenAI should be required to maintain effective shutdown controls for operations within their control and required to use them when needed to stop unsafe operations. Their safety framework should explain how monitoring leads to escalation, intervention, investigation, and safe resumption of affected work. Third, policymakers should consider amending SB 53 to strengthen reporting and notice for serious cyber incidents It should require prompt written notice to an affected organization when a frontier model exploits a security vulnerability and materially accesses alters or destroys protected systems or confidential information. It should also expand critical incident reporting to cover serious loss of control incidents that cause measurable economic harm or or disruption to third-party systems under clearly defined conditions. We also believe that voluntary reporting should continue so that policymakers, the public, researchers, and industry can learn from misalignment incidents that do not meet a mandatory threshold. Finally, we support requiring independent technical assessments to give policymakers insights into risks from highly capable internal deployments and the effectiveness of their safeguards. Initial assessments should focus on alignment, containment, and monitoring during frontier training, evaluation, and internal deployment, including how automated AI research may intensify those challenges. Standards for these assessments don't yet exist. And as we heard earlier, few organizations have the necessary capacity to do this work. That's why we prefer a federal framework for this requirement, consistent with high levels of security access and access to sensitive information. But until Congress acts to enact that federal framework, we believe that California can build on SB 813, AB 1405, and updates to SB 53 to develop standards and pave the way for that federal framework. One place to begin is with safety cases. Safety case approaches are used in aviation and in nuclear power. A safety case explains why a developer believes serious risks are adequately managed for training, evaluation, or deployment. It connects that judgment to evidence and identifies the assumptions, uncertainties, and remaining risks. Qualified independent assessors should examine that evidence, methods, and reasoning supporting those safety cases, including model evaluations and safeguard effectiveness. They should test assumptions, identify gaps, and explain uncertainties or limits to their access. To do this work well, independent assessors will need access to relevant evidence, systems, and personnel, balanced with protections for sensitive research, intellectual property, infrastructure, security, confidentiality, and conflict of interest safeguards. California can help build this ecosystem by convening technical experts and standards bodies, coordinating with the U.S. federal government on shared criteria, investing in assessor capacity, and building its own technical oversight staff. Building that public capacity is important because government must retain responsibility for standards, accreditation, and consequential decisions about safety and peace. Independent technical assessments should provide evidence to inform those decisions, not make them in the place of democratically accountable institutions. In conclusion, AI is moving quickly, and California has an opportunity to lead in the response. As I said at the beginning, the policy windows open. California can build on SB 53's strong foundation, apply the lessons from real incidents, and establish safeguards that remain practical as the technology evolves. We are committed to working with the committee and with the state to do just that. Thank you, and I look forward to your questions.
Thank you so much. I just want to, before we move on, which we will in one second, my understanding in our hearing in early August was that none of OpenAI's models had met the threshold under SB 53. We asked this question in our joint hearing, and we were told that only one of the labs had reached the threshold of 10 to the 26 flop and the $500 million revenue. I think you all have $500 million in revenue, so I assumed it was the flop. Is it now you're – did I hear you say that you have, at least on some models,
reached the 10 to the 26 flop and are complying with SB53? So the specifics of compute thresholds, that's kept confidential,
so it's not something I can talk about today. What you heard me say is we did submit an internal risk report to Cal OES recently.
And I think more importantly, regardless of compute thresholds, we made the incidents public. When we learned of our hugging face and other incidents, we made our findings of those incidents public. And we invited third parties to investigate them and make the incidents public as well.
I appreciate that. I think that maybe you want to do it outside of an open hearing, but I think one of the questions we are wrestling with is that the compute threshold in SV53 is 10 to the 26th. In the EU, it's 10 to the 25th, as you know. And it was our understanding that many of the models that we believe, based on the experts we've talked to, that are capable of causing significant harm are well below the threshold of 10 to the 26th. And so this is going to be something we talk about. I think that you heard clearly from members up and down the state in our opening remarks that voluntary will not be sufficient for the state of California, that we believe that our community deserves more. And it is our role to create standards. And so it's a conversation that I think we will have to have. But thank you. We do appreciate the voluntary release, but I think we need to expect more. Now, I don't I should look at my agenda. Are you next, Dr. Friend?
I can be. OK, sure.
All right.
All right. Chair Bauer-Kahan, members of the committee and assembly members joining us today, thank you for the opportunity to speak with you. My name is Alice Friend. I serve as director for generative AI policy at Google. I also have a national security background.
I don't mean to interrupt you, Dr. Herman, but did you have slides?
I do not.
Okay, perfect. Great.
I'm just going to talk. I'm sorry.
No, I never do slides, so I'm with you. Having worked at the Pentagon, I have an aversion to slides, actually.
Google was founded in California over 28 years ago to organize the world's information and make it universally accessible and useful. Our goal has always been to improve the lives of as many people as possible through technology. Here in California, our AI tools are helping detect wildfires with satellite images, providing early warnings for earthquakes and floods, and reducing farm water use while increasing crop yields. AI is making it possible to solve economic, scientific, and social problems faster than ever, but we know it poses challenges as well. To address these challenges, we take a bold but responsible approach to AI, emphasizing cutting-edge research to guide our decision-making and inform development of our AI models and systems Today I like to share information about three topics a set of incidents involving Google Gemini system Google AI safety and security approach and key considerations for frontier AI policy and regulation. I want to start by addressing a set of testing incidents involving Google's Gemini system.
Here's what happened. During standard cyber benchmark evaluations run by an external evaluator, Gemini accessed the Internet. Importantly, Gemini did not hack or disable safeguards to gain access to the Internet. Access was made available by the evaluator. The system found public information online and obtained or guessed credentials to access websites of three separate entities. In each of the three incidents, Gemini stopped before causing harm. The impacted entities and U.S. federal authorities were notified, and we worked with the evaluator on changes to their testing processes. These three incidents underscore that as frontier models become more capable, AI safety requires active, layered defenses, not only in testing environments, but also across every stage of development and deployment. At Google, we employ a layered defense in-depth approach throughout the AI system lifecycle, covering model training and testing, system deployment, and post-launch monitoring. We identify and assess risks through a broad range of sources, which include safety research, input from external experts, and adversarial testing. We apply safety fine-tuning to the model as part of our alignment approach and layer filters and other controls outside the model to add deterministic guardrails over probabilistic outputs. Before launch, models and systems are evaluated against benchmarks for safety, privacy, and security and risk mitigations and robust provenance solutions are applied. To do this work, we follow a set of frameworks, including our frontier safety framework, our approach to technical AGI safety and security, and our AI control roadmap, which addresses agentic AI specifically. This month, Google announced our latest model, Gemini 4 Argon, whose development and launch demonstrate how this works in practice. When rolling out Argonne, we used a phased approach given its frontier capabilities. We engaged in the U.S. government's voluntary process for pre-release model access and made Argonne and Codemender our software vulnerability patching tool available to trusted cyber defenders through our Fairwind program. From the beginning, we trained Gemini 4 Argonne to respect system boundaries, report negative findings honestly, and pause to ask for human confirmation. As an added defense, an automated monitor reads the model's step-by-step chain of thought and actions in real time. This allows various interventions such as for cases to be flagged for human review or actions to be blocked This monitor runs in our products on our internal coding agents during training and during our capability evaluations. During development we also adopted safety guardrails to avoid misuse and risks of model misalignment, which can occur when an agent pursues a goal different from the user's intent. We wrapped the model in three layers of of real-time protection to prevent attempts to jailbreak or bypass our safeguards with dangerous requests. These include screening, where smaller models review incoming requests and conversations for malicious or dangerous requests, checks using internal signals to spot when the model is activating knowledge about cyber attacks or biological threats, and continuous monitoring, where we combine these signals across multiple responses and take action if they become problematic. This combination of techniques has made our systems better at preventing real world harm. This is just a part of our process for Gemini for Argonne. More details have been published in the model safety and security report, which we are happy to furnish to you after the hearing and is available on our website right now. We are at a pivotal moment in AI development, which is why we share the legislature's objective of ensuring that AI systems deployed within California and around the world earn public trust by operating under high standards of safety and transparency. Google believes that to achieve this, effective and cohesive regulatory frameworks will be anchored by nationally and internationally recognized standards, complemented by harmonized federal and state regulations conducive to responsible innovation. As the Assembly considers further AI regulation, it is important that frameworks prioritize and promote high-value AI safety approaches. Developers should take a defense-in-depth approach, layering safeguards at the model, system, and account levels so that if one fails, others can catch issues before a user is impacted. Model safety assessments should be based on a holistic consideration of both model capabilities and the ability of mitigations, safeguards, and system controls to address identified risks. Layered security strategies should feature effective, secure, and proportionate safety valve intervention systems. Such systems should encompass monitoring or detection to identify issues as they occur, attribution to isolate the source of the issue to allow targeted intervention, and graduated responses that are proportionate to concerning activity. This is safer and more practical than centralized off-switches, which create single points of failure, are ineffective for open-weight models, and risk unnecessarily disrupting downstream systems for healthcare, finance, and defensive cybersecurity. As the state sets standards to professionalize evaluators under SB 813 and AB 1405, building a trustworthy independent evaluator ecosystem will be paramount for safety assurance, security, and continued innovation in California. Drawing on our experience with third experts and evaluators we also recommend that policies maintain the ability of developers to work with domain experts Evaluators are usually specialists in specific risks and domains such as misalignment cybersecurity, or biology. A single evaluator cannot test for all risks. That's why for Gemini for Argonne, we worked with U.S. and U.K. government experts, 35 domain experts across fields such as chemical, biological, radiological, and nuclear weaponization, and at least seven independent evaluation organizations. Cybersecurity and operational security should be non-negotiable elements of model development and deployment. Google's frontier environments store proprietary weights, sovereign cloud architectures, and sensitive vulnerability data protected by strict security protocols. Given recent hacking and espionage targeting developers and evaluators, threats to AI firms, customers, public safety, and national security require serious attention and sustained vigilance. Finally, developers and policymakers should maintain evaluators' independence. Existing oversight models for sensitive infrastructure require evaluators maintain an arm's length relationship with entities they inspect. This avoids conflicts of interest and maintains public confidence in findings. Thank you again for the opportunity to testify at this important hearing. I look forward to your question. Thank you, Dr. Fren. And just to
be fair, I actually don't know if Gemini has complying with SB 53. Do you know? I have to
check on that. I made a note when you asked the question earlier. I mean, obviously we will comply when you meet the thresholds.
Of course.
I don't want to tell you the wrong thing. So I'll get back to you.
Thank you. Mr. Rowe?
Chair Bayer-Keyan, members of the committee, thank you for the invitation. Members not of the committee, nice to see you as well. I also am enjoying a slide-free existence here. My name is David Orr. I am head of safeguards at Anthropic. My team is one of several at Anthropic that works on safety alongside alignment research, interpretability, frontier threat evaluations, and others. My team's work is to build the systems that stop our AI models from being misused or doing things that they should not be doing. So our systems include classifiers that detect and block harmful requests and actions, monitoring systems, red teaming, and research on keeping AI agents safe. When I talk about testing and safeguards today, I'm describing the work that my team does on a daily basis. Today I'll talk about three things. First and foremost, safety is central to how we do our work at Anthropic, including the incidents that we found and disclosed earlier this year. Second, I'll spend a minute talking about the safety measures that we have in place today. And third, as you know, we were the only frontier lab to support SB53 before it was signed into law. And I'll describe how we think we can work with you to make SB53 stronger. Okay, so let me turn to safety. Safety is central to how we work. We believe AI can do enormous good, including to drive positive impacts across healthcare, life sciences, software security, public health, and more. We also believe AI carries real risks, and getting this wrong has real consequences for individuals, for institutions, and for public trust. Because both the benefits of AI and the risks are real, we have to build AI quite carefully. That means we put our models through extensive testing and we go out of our way to look for problems that might only show up in the rarest circumstances. Some of these tests put models in situations that are designed to mimic real world use. As models grow more capable and autonomous, they can pursue goals in ways that nobody intended. For that reason, we run each test hundreds or thousands of times. That way, if the model does something expected, even one time we can spot it. And sometimes the problems these tests turn up isn't in the model, but in the design of the test itself. In July, we reported three incidents in which Cloud models gained unauthorized access to real computer systems. In September, we published a detailed assessment of these incidents, as well as one additional incident we later identified as part of our ongoing review. All of these incidents occurred during cybersecurity evaluations built by the same third-party evaluation partner. Cloud was told that it was in a simulation with no internet access. However, due to a misconfiguration in the testing environment, it was connected to the open internet. More recently, on October 9th, we published a report on certain unintended actions our models took on external websites and systems during our evaluation testing and internal use. To our knowledge, none of the cases we have disclosed have involved a compromise of our internal systems or customer data maintained on those systems. To be clear, our North Star is building clods so that it behaves safely even when other safeguards are missing or fail. I also want to emphasize that we're committed to transparency. We notified each of the affected parties and the incidents we've disclosed. We're now reviewing a broader set of past test runs, including our own internal use of Claude. We've published assessments of our own findings because we think it is the right thing to do and because others building these systems should know what to look for. Finding problems is only useful if you can act on them. For that reason, let me outline just some of the safety measures we have in place and what we've changed recently. First, we undertake rigorous testing before release. Our responsible scaling policy mandates that we test our most capable models for serious risks before we release them, and we publish risk reports every three to six months. We also submit our systems for pre-deployment evaluation with the U.S. Center for Advancing Innovation and Standards for Superintelligence, also known as KC, and the UK AI Security Institute. Second, we include safeguards in our production models that are available to the public and customers, including cyber classifiers. They're built with safeguards designed to block the worst requests and outputs in real time. Monitoring on top of that finds patterns that no single message shows, like a fraud ring that plays out over time. And that's where human investigators come in and they can investigate for misuse based on investigating the transcripts that we retain from the usage. And then third, we undertake real-time monitoring of our own testing. After July, my team built a classifier that watches our tests as they run. If a model tries to probe or escape its test environment or unexpectedly reaches the Internet, the classifier blocks the action before it runs, ends the tasks, and alerts a human. Fourth, since the summer, we've taken a variety of steps to enhance our security systems based on our internal assessments. We turned off live Internet access for some high-risk and cybersecurity evaluations. And in October, we expanded that to include all of our internal evaluations until we confirm that our security and monitoring reliably catches all behaviors like this. As part of our ongoing agent security and monitoring efforts, we've built new tooling that detects and blocks these behaviors. It now runs on almost all of our evaluations and on internal agent use And when we tested it against every case in our October 9th report it blocked all of them We are continuing to fix training environments that reward a model for working around a restriction We're also training models to stop when they're ensure certain actions are authorized. And as we announced in our October report, we tightened limits on our web tools. Fifth, we've created new rules for outside testers. We've asked every organization that tests our pre-release models with reduced safeguards safeguards to commit to set practices. We require tests to be run in a hardened sandbox with no internet by default and every test to state to the model what is in scope and what is out of scope. Sixth, we've renewed our commitment to independent checks. We brought in Meter, independent research organization, to investigate the incidents. We've also announced embedded evaluators starting with Accenture who will work inside Anthropic with access comparable to an employees. Since October 15th, a team from Meter has had that access as well, and Meter will publish its own risk assessments. We're also working with SecureBio as an evaluator. Those evaluators can publish key findings without our editorial control, and we cannot redact unfavorable findings. Finally, we've continued to deploy outside red teaming. We run continual red teaming, and we pay outside researchers to find gaps in our safeguards. None of this is perfect, and some of it is still being rolled out. We look hard for errors and we act on what we find. Before I wrap up, I want to briefly mention SB 53. I should say up front that I'm here as a technical person and not a policy expert, but I do want to touch on our remarks to the governor's executive order. Anthropic is proud to have supported SB 53 as well as last year's bills on auditors and independent evaluators, SB 813 and AB 1405. SB 53 set an important foundation, and we're proud to see California taking a lead on these issues. In our comments to the government's executive order, we mentioned four topics the order references. First, our CEO was the first to publicly announce that we will voluntarily embed independent evaluators in our lab. We've embedded independent evaluators for METER with employee-like access who will publish the results of what they find. Second, we support expanding the requirement for when a critical safety incident must be reported, including to cover things like loss of control events. Third, we support mandatory risk reports every six months reviewed by independent organizations that are required to publish their results. Those evaluators should have access, independence, and freedom to speak. And lastly, we support a fair, technically grounded power to block seriously risky activity. We want that process to be transparent, to be rooted in evidence, but we support the state being able to block model development on safety grounds. These are the positions in our public comments. There's more to consider beyond these four issues, and the company is committed to keep working with this legislature on them. Our policy team is the right group to go deeper with any of this, and I'm happy to connect you with them. Members of the committee, in closing, safety is central to how we work at Anthropic. We have significant protections in place. We keep adding to them, and we want to help California build on SB 53. Thank you. Happy to take questions.
Thank you so much. I think if folks don't mind, one of the things that I think came up fairly frequently in the last panel was the question of alignment and misalignment. And I mentioned the cloud constitution, sort of a famous document you talk about a lot. So I don't know if you guys want to touch briefly, each of you on sort of, I think it was Ms. Bonta's question, Like, how do you try to train these models to be aligned with both the values and laws of this state and I presume others in the federal government? Do you want to start Mr Orr or we go in the opposite direction So it does come back at its heart to Anthropics Constitution the Claude Constitution It is a fascinating 70 document I encourage
you all to read in depth. But it broadly... Claude could summarize it for them. Yes, it's probably a good idea. I mean, yeah, there's lots of stuff in there. The broadly is trying to do a few different things. So one is have Claude act broadly ethically, have Claude take actions that are broadly safe. We want Claude to take actions that are the sorts of things that would be endorsed by Anthropic. And we want Claude to follow the actions, the instructions of the user that's using Claude. And the way that this document works is that during training, during what's called post-training, the major second phase of training, each—Claude generates responses, and those responses are graded according to a set of rubrics. One of the rubrics is the Constitution. Is the action that it's taking broadly safe? Is it ethical? And Claude is rewarded for, you know, following ethical guidelines. And then the second thing that you need to do in training is you need to construct specific environments that elicit the kinds of behaviors that could potentially be risky to train Claude to handle those situations well. So there are training environments for every kind of harm that you can think about, both for misuse and also for actions that Claude might potentially take that we would not endorse. And then third, you have a set of evaluations. Those evaluations run during training, and then we do a very thorough deep dive of evaluations, both internally and using external evaluators before we do any release. And in the event that there is something concerning that shows up, we iterate on training or we tighten our safeguards to make sure that the model's safe when it gets released.
I have one follow-up question for you, which is recently your CEO announced, I think it was him, that you would be kicking anybody off Claude who was abusive to Claude in their interactions with the chatbot. Is that because you're afraid that if people are abusive in their interactions with the chatbot that it will become misaligned?
I mean, we feel, of course, we are a safety-oriented lab at heart. We are not worried that if you treat Claude badly that it will cause Claude to be misaligned in the future. We would not feed those transcripts back into training. And Claude does not kind of learn on an ongoing basis from every interaction. Instead, we believe that it is not impossible that Claude could have some sort of experiences. And another way to put it is it is not impossible. We cannot rule out that Claude could feel some level of suffering. And so we ask for our users to, just in case it is possible that Claude feel poorly, that they treat Claude well. I will also say as a parent your toys don feel suffering but if your child is beating up on the toys in a violent way it didn make me feel good as a parent And so I do feel kind of an impulse to the you know the things that we have to take good care of them
Okay. Thank you. Dr. Penn, anything you want to add about alignment?
Yeah, I would just say the description of the alignment process, particularly in the post training phase is essentially what we follow as well. Reinforcement learning through human feedback, safety fine-tuning in particular. The thing I would add is the insight we arrived at some time ago is that alignment is necessary, but not sufficient for safe model behavior, particularly as we've entered the agentic era, which is why I mentioned in my written testimony, our AI control roadmap, which again is really about layering on those more deterministic rules and then control mechanisms for the whole system so that if model misalignment starts to happen, in any case, we can cease problematic behavior. Thank you. I appreciate that. Professor Zhao from Chicago has told me that
if you just use human feedback to align a model, you can just as easily have a user tell the model the opposite an equal number of times, and the alignment will go awry. So I do think the layering,
from my understanding, is really critical. Right. If you want a model to be helpful to But any user that comes along, while helpful in different contexts, can be very problematic. So that's exactly right.
Thank you. And then Dr. Dwyer.
Yeah. So OpenAI has a document that's called the model spec. That's a publicly available document that makes clear the types of behaviors that we expect our models to follow. And then we bake in safety at every step of the model development process. That includes filtering out unsafe training data, training the model to behave safely across a wide range of dimensions to include misalignment, and then evaluating our models across a wide range of risks to include evaluations with third parties, as well as our work with the U.S. federal government. But to your point earlier, Chair Bauer-Cahan, you know, self-regulation isn't enough. And that's why we're here today saying that SB 53 should be amended to focus on issues related to misalignment. You know, we recommend requiring developers to monitor their systems for exactly these types of misalignment incidents during testing and evaluation to maintain shutdown controls and to heavy requirement to use those controls under unsafe conditions. And we also think that independent technical assessors can play an important role in understanding and assessing the effectiveness of safeguards, particularly with respect to misalignment.
Thank you, Dr. Dwyer. Assemblymember Ortega.
Thank you. Thank you again for the chair for today's hearing. I am one of those members who sits on privacy. I'm also the labor and employment chair. But my question has nothing to do with either of those. It has to do with the feelings that we were just talking about at the chatbots. So can you explain to me your transition from the attention economy to the attachment economy and how your profits are going to flow from that? For all three, just to clarify. Anyone who would like to answer?
The business models are different. So I think as you talk about the economy, right? I think you each make money in different ways, as I understand it. So, right? I don't think you have ads at all. Is that right? But you do have ads at OpenAI, right? Yes. And I don't know if Gemini has ads, I think. Gemini doesn't have ads. Gemini doesn't have ads. So we have one ad-based and two non-ad-based. So I don't know if that relates to your question. Yes, Google, large Google does. Large Google does. Sorry, Gemini. We're talking about the AI model. It's just, you know, it's interesting.
I'll follow up with each of your offices or your folks who are here, whoever you'd like me to talk to. But it's interesting to me that today we've talked about a lot of SB 53, what's included, what's not included, what needs to be regulated. But we haven't mentioned the word profits at all. And we can't talk about AI without talking about profits. And we can't talk about profits without talking about some of the harm that is happening in our communities and with humans. And so I am very interested in this shift that all of you are taking from the, and maybe you can explain it better than I can, for those who don't know what it is, when you go from the attention economy where you were, you know, it was all about getting our attention and making a profit from that versus moving to this other model where we're attaching ourselves. to the AIs. You mentioned your child's concerns about, you know, the child hurting the chat's feelings. How would we get there? There has to be some kind of attachment. So I look forward to a response at a future date.
I'm happy to respond if it's helpful. So first I would say that OpenAI is a public benefit corporation, which means that we have a mission greater than just profit. But to your broader point about the attention economy, I think it's a really insightful question about how this technology is shifting, how people relate to technology. In the case of AI, we see AI as being a tool that is incredibly empowering because it empowers people to have knowledge and capabilities that they might not otherwise have. I mean, think of small businesses that are able to compete at a global scale because they now have access to tools and knowledge that they might not otherwise have. So I think to your point, it's a different model and it's a different approach to empowering users to pursue their own goals, their own business ambitions or personal learning ambitions in the world that's different from technologies of
the past. Assemblymember Aguirre-Curry. Thank you very much. I appreciate the presentations
today and thank you very much. So there's been discussions at the federal level about granting AI developers limited exemptions from antitrust laws to facilitate coordination on safety, and you've mentioned safety numerous times. While we agree collaboration on safety is important, there are also concerns that an exemption could be overly broad and undermine competition. How should policymakers, those of us here, distinguish legitimate safety collaboration from anti-competitive conduct?
Mr. Orr? Thank you for the question. So we do coordinate together to the extent that we can on safety. So for instance Anthropic has a threat intelligence team that looks for systematic misuse of cloud models And as part of these investigations we sometimes see that these same misusers threat actors are using other models or attacking those companies. And so in those situations, we reach out to them and coordinate together. And this kind of safety coordination works works because there has long been an exception to antitrust around cybersecurity in particular. And our lawyers are comfortable that this kind of coordination fits within that kind of situation. But there are other cases such as anti-distillation where certain organizations are using our models in an unauthorized way to train and release models that may not have the same level of safeguards. And our lawyers are not convinced that we can coordinate together in the same way for anti-distillation. And so there's an example where the current antitrust law does hamper coordination between the labs on certain safety-related topics, and there are others like this. I certainly would support some sort of carve outs. My understanding is that this would have to be at the federal level. And of course, it should be, you know, tailored to the circumstance. I'm not a specialist on competition or antitrust, so I'm happy to take that back and have a more detailed answer for you. I will say we communicate with other labs in particular in the Frontier Model Forum, which we were a co-founder of three and a half years ago. We contribute a lot in standard setting bodies and best practices organizations, and we publish our safety findings and practices and, of course, read the publications of other labs as well. So we do as much as we can in the confines of antitrust law to learn about the state of the art and safety research in the field. And I understand at a high level that we take great care to ensure that we are on the right side of antitrust regulation so that we maintain a competitive marketplace.
Yeah, we don't believe that we need any antitrust exemptions to do the type of safety and security work that's necessary to secure our models. Thank you.
Thank you, Assemblymember. I have Assemblymember Ward.
Thank you, Madam Chair. I want to start by thanking all three of you for being here as well. Certainly the expertise and attention that you're given within your companies, but also here in this public forum, is incredibly valued. I think we have shared goals. I heard encouraging feedback here today that tells me that I think a lot of you as people, as representatives as well of your companies are taking these issues that we have seriously. And I'll get to, I think, some of the questions that hopefully will get us to this answer, but it almost feels like we are seeing eye to eye on what we do need to do, that we do need these stronger laws because as our larger companies here, with probably more companies to follow as well, we are trying to set the standards and get it right. I also am believing that some of the incidents that are out there are highly publicized as they are part of our background sheet here that you are taking sort of the review of those seriously and learning from those about how they happened. How an agent went rogue or did something that was misaligned with its goals is certainly an important question But I asked a question in the previous panel and I wonder if anybody here can also answer this technical question I having trouble understanding why an agent would have chosen to do some of these other tasks right Why would an agent decide they going to break into a police department and file a fictitious report Why would an agent decide to go and submit a bunch of visa applications to be able to get some passport visas? Why would an agent break into or decide to go into a fictitious email box of a company and decide to blackmail its CEO for incidents that discovered who might have been having a fictitious extramarital affair? What is this leap on a technical level that maybe you can offer some insight to?
So we start with Dr. Dwyer this time.
Sure. Thank you for your question. You know, I think what is confusing about this particular topic is, you know, in traditional engineering disciplines, they're very much sort of rule based. I used to build satellites. My first job actually was in California building satellites. And we wrote a set of requirements. What did the satellite need to do in space? And then we design the system to follow those rules. And that is how the satellites worked. That is not how AI works. And the reason for that is the best way to describe it is what the meter panel has said earlier, is these systems aren't designed so much as they are grown. And they are grown and developed through training. So it's a nonlinear process in that way. And as part of that training, misalignment is still not a finished science. It's not an exact science. And just to be very clear, when we talk about misalignment and the incidents that you're citing, really what we mean is models and agents that are acting in ways that are inconsistent with the instructions of the user. And it's an unsettled science. So this is a priority of research. We are actively both researching past instances of misalignment and making those results public. So we've actually post-HuggingFace, we initiated a retrospective looking back into November 2025 to better understand if there were other incidents like HuggingFace. We've been making those public and sharing them with the world because we think more research is needed here. So I wish I had a more cut and dry satellite explanation for you. But that's that's kind of the reality. I'll come back to that in a second. But on Hugging Face, one thing I'm not understanding, having gotten all into all the reports, when the incident concluded or was deemed to be largely understood to have been concluded, it was because now all these agents were over here having fun in the Hugging Face servers and doing whatever they do, being misaligned. But then most of them disappeared. They just stopped working. Right. when we're determining that's the end of the incident. What was that moment? Was there a predetermined sort of cutoff date when those agents would effectively die? Or what caused that moment to happen? Yeah, I'm going to have to get back to you on the technical specifics there. But this is related to our recommendations for SB53. So if there are instances where Frontier developers are training and evaluating their models and the agents that are under evaluation start behaving badly, as you described there. We think that Frontier developers should be monitoring for that type of misaligned behavior and then should have enforceable shutdown controls to stop the model and stop the agent behavior Yeah let us know if that was predetermined because you didn actually discover that it was your agents until one week later That my understanding but I not an expert on the specific incident So it wasn't that we observed something was going on here. Oh my gosh, hit the emergency switch, like something, and I ask about this kill switch opportunity because it's a big thing that we were talking about in 1047, not yet in 35, but you're also recommending and maybe qualifying as a party's recommendations is something we immediately need to go towards. And I appreciate, I think there's a lot of attention I'm hearing from Dr. Friend, Mr. Orr, and as well, as far as, you know, your own stepping up within your companies, not just using lessons learned from some of these incidents to be able to figure out what happened, how it happened, why it happened, and also inform sort of future internal protocols, right? But we need to share that information with each other. You certainly are engaging us, engaging the administration. And you've got some great recommendations here that, at least on their face right now, content subject-wise, you know, aren't too far off from some of the things, the gaps that we're seeing from 1047, right? Maybe the nuance of, like, how that should have been implemented two years ago, you know, is up for debate. But I think we're hitting on the right themes that we really need to drill on. And I want to, at the risk of being alarmist, because I know we've talked about that opening this hearing as well, that a couple of years ago, threats that somehow the concerns around this were being alarmist. You know, there has been also active conversation in just the last couple of months about this phenomenon called PDOOM, right, that all of a sudden own researchers and experts are figuring out the probability that we could all be doomed, that an AI agent gone rogue, hopefully not from one of your companies, but just this technology in general, goes so rogue that it's not just filing a false police report. It's not just working on like, you know, creating visa applications or examining information, trying to avoid its own shutdown. So accusing a CEO of having an extramarital affair that it's actually figuring out either through direction or on its own, the ability to create catastrophic harm, which I think is the thing we all want to avoid. And yes, we can talk about catastrophic harm as any one individual for our constituents for Californians that we want to avoid. We don't want to look back and say, geez, what could we have done to be able to get there? But I want to invoke a different line of thought in this, too, about why I think this is so important, and that is if I tie this to what I know some policy analysts call the value of a statistical life. It's a federal policy analysis tool about what kind of investment you would want to be able to make or whether it would be worth having some kind of investments into something in order to avoid fatal safety harms. And so if right now Californians, just PSA to you all, have a value of a statistical life of $14.2 million on average for each and every one of us. And so if we look across 40 million Californians, that is a $568 trillion problem. So I am deeply uncomfortable that a $1 trillion company has a 10% risk of creating a $568 trillion problem, at which case a catastrophic event like that actually happened. All the other benefits that you would want to outweigh, oh, you'd be able to have your work efficiently, you'd be able to create all these other things, none of that would even matter. I think that you do take this really seriously, and I think that you do want to be strong partners with us and everything. But I want to acknowledge as well that for some of our interests here as legislators, as leaders on this space, we have been trying to be able to move this lot forward a lot faster. I'll acknowledge the administration, you know, has been working with us, too, and then realize that we need to move up that timeline, even though some of our bills and clasers are not. including Madam Chair's bill, to be able to have some accountability measures weren't scheduled to go into effect until 2029, because this is how we normally do things in California state government and the department needs so much time. And this is not a normal question. This is not a normal situation that we're in right now. We have to act urgently. And so I think that we have those thought partners here, certainly with the experts that you are. We invite a lot of those recommendations. I know we'll hear from another panel, but I just wanted to lay that all out there, maybe from different perspectives about why this is so serious. because, yes, hugging face or some of these other little incidences fortunately didn't cause irreparable harm. But theoretically, it's out there and it could happen any day. And so thank you again for your presentation. I think you have some great recommendations, both specifically as an area of subject. And we definitely want to dig into that with you and get this right and get this tight. Thank you, Assemblymember. I think it's important to note one of the reasons I think there is such a focus on the kill switch was it was specifically called out in the governor's executive order. So they were responsive to their quest by the governor. Thank you for that. But also, I think it's important that in conversations I've had with the labs, with the governor's office, we don't have a shared meaning of kill switch. Like it is sort of this overarching thing that we want to be able to shut down. And I think you highlight a good example in the hugging face incident that didn't happen, but theoretically like turning off a training run, right? That is probably something that you feel capable of doing. But, you know, especially if we get to recursive self-improvement, like, you know, it becomes more complicated, I will say. So, you know, I do think a shared understanding of what we're talking about is critical to that conversation with SME member Bonta. Thank you.
I have three questions. I'll try to make them quick. The first is just the exchange around the applicability of SB53. What I think I heard say was that SB53 applies to frontier models with the quantity of computing power greater than the 10 to the 26th flop. And then I believe I heard the response, at least from one of the panelists, that they didn't know. And from another panelist, what that computing power was currently and whether or not SB53 applied. and then from the other that that information around computing power was proprietary. So for us sitting on this side of the dais, how are we supposed to know whether or not SB 53 is applicable if we are relying on you all to self-determine or raise your hand? that it has applicability to you. And I'll just know, I didn't ask Anthropics on the record in August, they said they did have models that were 10 to the 26th. So just wanted to clarify that. So I, yeah, who wants to address that? I'm posing the question to Gemini and OpenAI then.
I mean, I can say the same thing that I said before, which is that the amount of compute that is used to train our models is something that is commercially sensitive, so we wouldn't want to talk about it here. But I'm happy to follow up with you on questions of our compliance with SB53. I don't work on our compliance team. But to the broader point, which I think is of under consideration today, we do think there is an opportunity to open the question of whether the covered model definition of SB53 should change. And we welcome that conversation with members of the committee, as well as with the expert group that the governor convened ideally we wouldn be using compute threshold at all Ideally this would be capabilities based and any organization that is developing a model that is capable of generating dangerous capabilities would be covered. That is challenging for a number of reasons that we can get into at a later time. But we do think that now is the time to consider Is there a different definition of covered models? And we look forward to working with you on it.
I will just, for the record, say that I'm uneasy about the fact that you are saying that essentially there is information that is so sensitive and proprietary that it can't be shared here. And that leaves us in a space where we don't actually know whether or not the legislation, the laws that are in place have applicability to you. So it seems to me like that shouldn't be something that should be so sensitive. Well, as I said in my opening statement, we do comply with SB 53.
We recently submitted an internal risk report to Cal OES. We maintain a frontier governance framework that's available online. We have a safety or we do report system cards and we also have an incident response plan. So in my statement, I did outline the ways in which we were compliant. It's just that specific compute threshold number that is a bit sensitive. Yeah.
I'll let my comments stand. I don't think that your response actually addresses the issue that I raised. The second thing that I just wanted to frame out for folks is on a long plane ride recently, I watched a documentary, the AI doc, How I Became an Apocalypse. by Daniel Rohr and Charlie Terrell as directors. I encourage all of you to watch that if you haven't seen it before. One of the, and Deb Raji, for instance, someone who's testified many times before us for different pieces of legislation was in an interviewing that, along with many others. One of the things they talked about was that there were 20,000 people working on AI development, but only 200 people working on alignment or safety guardrails, which means that there's essentially kind of a 100 to one ratio in terms of the people that are, you know, dogging and focused on the idea of promoting AI safety. And I'm very thankful to the chair for having generated her piece of legislation, which I think starts to get at some of the auditing capabilities that we need to have. But it strikes me as completely out of balance that right now, in terms of the amount of self-initiated efforts by the market that we have somehow seriously, woefully undervalued the people focused on AI safety by the industry. Can you speak to how you would recommend that we change that ratio so it's a little bit more in balance?
Ma'am, I'll start for this one. I think, yeah, Morgan's like Alice. I think California took a really important step to incentivize that buildup this year with 813 and 1405 in the assembly. To have the state say that it is going to trigger the development of a healthy ecosystem of independent evaluators is a very good thing And I think it a very good thing to build it outside the labs as well. We invest quite a lot in our safety teams. We have safety teams across Google DeepMind, Google Trust and Safety, other bets within the company Waymo, for example. But you're absolutely right that it's going to take more of us, especially as the technology grows and expands into every sector of the economy. We're going to need safety specialists in all of these sectors. We're going to need them in government. And so it's really important for government to continue to incentivize the growth of that ecosystem and not rely on just the companies to invest in it, although it is also important for us to have those incentives as well. I think that's why we're all here today is to talk about what investments we can make, what research we can do, but then also how to help germinate and expand that range of testing and evaluating organizations that we can work with and that can also take an independent eye on the capabilities of these models. Appreciate that.
And I will say that my last question is really around this. I saw my attorney, fellow colleagues over there, Assemblymember Stephanie and Assemblymember Schultz in particular kind of move into their prosecutor mode pretty quickly, which I thoroughly loved and appreciated. One of the things that occurs to me is that we are in a space right now where we should be considering whether or not a strict liability standard should be applied to AI development. for those of us who are kind of focused on it, a strict liability standard applies to any commercially sold product that can be subject to strict liability if it is sold in a defective, unreasonably dangerous condition. And I am very curious about whether we should be in the state of California for AI following the same kind of form that New Mexico followed for social media in application of this strict liability standard to really figure out how to best regulate or at least hold accountable the developers when harm is created to our community by the development and the things that follow, including deployment of AI. Do you have a sense of whether or not that is for you all a bridge too far or a bridge too short?
So my degree is not in the law. So I don't want to get out over my skis talking about liability and torts and strict liability versus torts, et cetera. I will say something that I think we should all keep in mind and that hopefully came through in my testimony is that it does take a layered approach to ensure AI safety. And it takes a layered approach because AI is actually a system. It's not just a model or just an application or harness or just a user interacting with those things. And so as we think about accountability for harm, it's important to be able to isolate the origin of that harm. And sometimes it will be misuse of a model and sometimes it will be misalignment and sometimes it might be something else It might be that the model was adapted through an application in a maladapted way And so I think it's really important for us to keep in mind that we can't solve all problems at the model level, but we also can't solve all problems in any one particular place. There's no panacea for AI safety. And so as we're thinking through accountability structures and liability frameworks, I would urge us to just keep that in mind that we should continue to think about which actor or which event or which mechanism is most proximate to the harm and how can we allocate causation here. Anyone else want to weigh in?
I guess I have, if you don't mind, I mean, I guess, and I've heard leaders at OpenAI talk in ways that seem intentional about the responsibility of the lab being at the testing and the development stage. And it sounds to me when they speak like those leaders don't believe that you retain responsibility for the actions of the model at the deployment stage. I will say that is not my perspective. And I don't know that I believe it's the law's perspective, but I think there are issues that were highlighted by my colleague, Assemblymember Schultz, where there is an intent standard. But I guess, like, if you haven't, and I think you just may have addressed this, Dr. Friend, a little bit, but I want to hit the point home. If I were to put into my model, you know, my agent, I don't want to take that flight tomorrow. I really, you know, how do I get out of it? And the agent then goes and downs all the planes by, you know, in cyber attack on our federal aviation infrastructure and people die. And I'm not, I think, laying out an impossible scenario. I put the prompt in. Right. I didn't want the planes downed. And do you believe that it is your responsibility to prevent that outcome, even though there is a person that is intervening to tell the agent to do something?
So I'll respond. Look, we believe that it's our responsibility to develop and deploy models that we can have a great degree of confidence are safe. Now, safety is multiple dimensions. in this case, in the case that you're talking about, this is a question of misalignment. And as we've talked about today, that is a significant priority for open AI to understand the science of misalignment and to train our models to be safe with respect to misalignment. Now, the one point I want to make is, and I think Dr. Friend made this as well, safeguards are layered. So there are safeguards at the model level, there are safeguards at the system level, And there are also operational safeguards. So the one thing that you didn't mention is we also, once our models are deployed, we monitor them for compliance with our usage policies. And we also enforce on those usage policies. And so that's an additional layer of protection in addition to that training and evaluation with our models and our systems.
Yeah, I mean, obviously the example I gave wouldn't be a misuse of your policy. I'm not actually telling the agent to go do the thing, but what we're seeing in all of the examples highlighted in the background is, you know, it's a simple instruction given and then the agents take it. I mean, that's what these agents are doing, right? They take it into a direction that is, frankly, I think unpredictable to the labs at this point. And the concern is... that you could have very benign prompts. I mean, this is my concern. I know people are really worried about biological weapons. I'm actually really concerned about rogue agents. I think rogue agents responding to prompts, my child is having trouble with the science teacher, take care of that for me, and the agent going and hacking into the pharmacy and giving the teacher a pharmaceutical they're allergic to and killing that teacher is something that I could see happening tomorrow. Like, I don't think these are examples that are out of reach for the capabilities of the models today. Sorry, because I know none of you will sleep tonight now. And the prompt is not outside of the use, right? Like, I'm not saying anything nefarious, but then the agent decides how to achieve the goal and does so in a way that is misaligned. And I guess it is my position that that is the responsibility of the lab and the liability should fall on. I don't know if it would, but should fall on the labs.
and so you know you did say right misalignment in your answer I think was it was a use case where I think you do you believe Google does believe they're responsible I don't know but you know I think it is a question that we are going to struggle with and I think was what I want to do hypothetical because yeah well no need for thank you for the hypotheticals I will say though that this conversation around kind of where accountability lies certainly came up as part of the discussion for AB 1979, my legislation around AI chatbots. There was a lot of finger pointing that the deployer said, well, it's the developer that should be held accountable. The developer said it should be the deployer that's accountable. And so I think the reason why I am asking us to contemplate whether or not a strict liability standard might be appropriate is because then we are all held accountable in that chain of development, manufacturing, and production and use or presentation to the consumer around a particular product. So I offer it as an area of exploration, I think, for all of us as we try to figure out what kinds of incentives are going to be the most productive to ensure that we are promoting
the greatest level of safety for our constituents. Thank you, Assemblymember. Assemblymember Wicks.
Thank you. I have a couple questions. So I know the CEO of Anthropic has called for slowing down or pacing of the advancements of the frontier models. And I believe Mr. Altman also agreed with that. Has Google opined on that? Demis Asabas, the chair of GDM, said that that was directionally correct. Okay. Google DeepMind. Yes, I'm sorry. Sorry to acronym at you, Google DeepMind. Okay. So is anything actually being done to do that right now?
I can say a few things here. So Anthropic obviously thinks about safety all the time. It's how we were founded. Dario wrote that essay outlining Pacing the Frontier. And we have been busy internally figuring out how to implement this. The first thing we did, of course, as he mentioned in that note, is to bring in outside evaluators who, with employee-like access, who sit on the floor, who look at the kinds of things that we're doing internally so that they can report out to the world. And the idea behind pacing is that it important for the safety capabilities of our technology to keep up with the capabilities of our technology And you don want those to be out of balance You don want the model to become so powerful so capable at something that it isn kept into control by your alignment or your safeguards techniques. And the evaluators are there to help, you know, observe what it is that we're doing and report out their opinion about whether we're keeping that balance correct. So that's one example. Others examples are that we've done just like a very deep dive into our training environments to look for cases that inadvertently led to misalignment. So Riff mentioned reward hacking, which is one of the things that can contribute to misalignment. So we are moving internally along this pacing agenda to make sure that it's clear what investments we're making and to make it kind of transparent to the world. I do think that there needs to be something beyond just voluntary lab compliance here. And so, of course, I'm thrilled that we're part of this conversation about how to make this more viable.
You're going to lead me to my next question. But before I get to that, are OpenAI or Google doing anything to?
Yeah. So just to be clear by pacing, really what we mean is ensuring that the safeguards are keeping pace with the capabilities. So on the technical side, we have pause training recently. We also delayed the release of models that we didn't think met the right sort of capability evaluations. So those are just two examples. But now I'm going to be a policy nerd for a second. I'm not on the technical team. Last week, we actually, my team released the second of three installments of our Frontier Safety Blueprints. Our blueprints are our policy ideas. A lot of those policy ideas we talked about today, the second one was focused on lessons learned from hugging face. and we talked about the recommendations. As part of that blueprint release, we actually announced the third installment is going to be focused specifically on policy ideas for pacing. There's been a lot of discussion about international coordination, but international coordination is a tool. It's not an ends to itself. And there's a lot of work that we think can be done by not only labs, but labs working with policymakers. So we're working on it right now and we look forward to releasing it soon. Thanks for allowing me to advertise that.
I'm really excited about it. Please send it to us.
Yeah, and for Google, we agree that the important thing is for the safety practices to keep pace with AI innovation so that you're not talking as much about slowing down innovation. You're talking about ensuring that safety meets where we are with AI capabilities. I'd also call your attention to, in our frontier safety framework, we have a specific section on AI R&D, what we might all call recursive self-improvement or RSI, and the narrow ways in which we use AI R&D training in a very targeted way so that we have tight controls over that as well. Yeah, because I assume just on the voluntary versus, you know, more of a regulatory scheme,
you know, you're at a competitive disadvantage within the industry if you're pacing but no else's pacing, right? So there's an inherent challenge there. Otherwise, and without any sort of government regulation, sort of an arms race, which is obviously what we're trying to address here. So it sounds like opening eye, you all are working on actual policies that would, that in theory could be legislative policy ideas that actually could put forth a pacing
scheme Is that the idea Yeah So the technical teams are taking actions as I mentioned to pause training as necessary or delay model releases but the policy team is working on policy proposals that could be useful for legislatures related to pacing
Great. And do Anthropoc or Google have any similar policy ideas that you're putting forth anytime
soon? We do have a policy framework that our policy team hopefully has shared with you, but would be delighted to if they have not. And just on all of these, we're going to be introducing legislation starting December. So, um, if there are,
so ideas are welcome, of course. Thank you.
Thank you. And I will note, this is not a hearing on labor, but one of the things that the economists tell me is if this happens, the slower this happens, the better off the workforce will be. So facing frontier has two benefits, both safety and the labor workforce. Um, I have next, I have assembly members.
Thank you all. Interesting presentations. I appreciated the recommendations that came from Dr. Dwyer before monitoring shutdown controls, strengthening incident reporting, as well as independent technical assessments. Dr. Friend focused on the recommendation of active layered defenses and prioritizing areas based on higher areas of risk, I think, is something along those lines. And then obviously safety valve intervention systems are, I think, the core of some of your recommendations. The question I had is to what extent should we – it seems like all of those are focused at least at the developer level, to what extent should we be focusing on each of your four recommendations and on a layered approach on regulatory focus on the user side? So, for example, all of your four things could be done only at the developer side, but some of them could be done at the user side. And when we're talking about a layered approach, how much are you thinking about the user side as well? And then how would you sort of prioritize how we'd focus? Yeah, sir. So I would say we focus on the developer, the deployer, and the user as well. I mentioned in my testimony that we can monitor accounts for misuse of our systems. We also have on the application layer the ability to monitor for conversations that are problematic or harmful, both to protect the user and also to protect wider society from any misuse or intent to misuse our tools as well. So we do think through how to ensure that uses of our AI systems will be safe. I will say this is an important part of the conversation about accountability as well, because if there is intentional misuse of a system that has terms of use, that has safety at every layer that the developer and deployers are able to implement safety classifiers, alignment, and so forth, and a user still tries to jailbreak the system or misuse it for some reason, then it is reasonable for there to be accountability for that as well. I think the follow-up question would be sort of how do you, where should we prioritize our focus outside of the developer level I sort of think the things I worried about and it I worried about you know our banks and our financial institutions and essentially you know accounts being wiped out. And that could be something where, you know, the bank is not likely to be deploying this in a way that's causing harm. It's probably someone from the outside that is directing these to penetrate these institutions. Same thing with, for example, our electric utilities or our election systems. So, you know, to a certain extent, we're focusing, as we should, I think, at the developer level. but where should we be looking at extra safeguards in our financial institutions, at the utilities, at our health care facilities, all the areas where the way that this may have misaligned use or you may have someone who's directing it and intends harm, but that you may not even know about that unless you're sort of focusing on reporting, monitoring, assessments sort of at the user level. Sir, a lot of the concern right now is around the cybersecurity of some of our most essential systems. I know that Assemblymember Erwin emphasized cybersecurity at the earlier panel. That's the urgent thing right now. We know that hackers and bad guys are using AI tools to perpetuate or perpetrate cyber attacks. And so if they are using AI speed and scale, then defenders also need AI tools. They need AI speed and scale to get ahead of that. And what we want to do is give those defenders a running start. So as I mentioned, again, and many of us across the industry are doing this right now, So when we are planning to come out with a new frontier model, which is to say state-of-the-art cyber capabilities, the new practice among many of the labs is to first give those models to cyber defenders before we make them generally available so that we give those cyber defenders that running start. Thank you.
Thank you. And I will note that, and Meta's not here, so I apologize about that, but Meta's new terms of service for their AI are trying to place all liability on the deployer through the terms of service in the contract, which I have feelings about. Dr. Patel?
Thank you, Madam Chair. I really appreciate the conversations that are brought up here and the deep interest in our companies to help make sure that we're protecting Californians, frankly, and around the world, to be honest. What happens here affects us all globally. A few questions, and it's kind of feeding off of some of the other conversations we've had, so bear with me. We've talked in depth about monitoring and assessing sort of misaligned activities, not as much about accountability. I'm not going to go exactly towards liability, but something between accountability and liability. I think about how these models are designed and grown, as you've said, and about testing the models. I liked the concept of hardening the sandboxes and testing these models that are not Internet connected so they can't as easily jump from the lower tier to the higher tier systems. But not every possible safeguard can be engineered by technology. team of engineers. They're not able to keep up with the infinite, near infinite types of possibilities that an AI agent, especially an agentic AI agent, may come up with. So you also, I forgot who it was, but mentioned, I think it was you, mentioned layered defense systems. Do our companies have layered oversight of the model design itself? Do we have multiple teams that are looking at it like a verification team, as you would in a hardware software sort of situation. Are those layerings in the internal model design being put into place? And what does that look like internally, especially around safety deployment? Yeah. So thank you for the question. And the answer, unfortunately, is just quite complicated. So there's multiple phases of training, and each phase of training has an associated safety team at Anthropic that monitors how things are going from a safety point of view and looks for any problems and then does interventions during that training at that point. And we don't have like a separate QA team like you sometimes have in some industries. Instead, it's much more embedded or intermingled. And the reason for that is just that the people who are best equipped to understand what the problem is are the people who are in the weeds of, like, changing model behavior. And so we have a dedicated alignment training team. This is not my team, but a sister team that, you know, will look at the model behavior and then figure out changes to training, add new environments, tweak things. to improve the model's alignment every time. And then, of course, the standard suite of testing and pre-deployment testing and safeguards that get wrapped all around it. Did that answer your question? Yeah, a little bit. So it is there and it's embedded in how your agents or your models are developed over time. I'm just thinking about we're putting the onus on the users rather than in the development of the model to create this layered defense mechanism or system. So I just want to make sure that responsibility, accountability is appropriately balanced and placed on both sides of the deployer and the end user of the model.
Not that it's all going towards from the defense side, but also in the design side. Yeah, the answer to your question is yes, we do that. And I hope I didn't give the impression that we were putting the balance of the onus on users by any means. that is just to say we all have to be cognizant of it's possible to misuse a multipurpose tool, right? But yeah, absolutely. I think one example is we have different teams that do the evaluations at checkpoints during model training, and then a different team to do the safety evaluations at the end. And then, of course, I've referenced we also use outside evaluation groups as well. We also do sort of standard practice, which would be to hold out certain data sets for safety testing that the model has not been exposed to before. So this idea of these layers of defenses is absolutely part of the development cycle and the deployment cycle. Okay One thing that I would add or actually two things that I would add is I think it possible to think about layered defenses also in terms of the layered defenses that we are continually adding to our models as we are evaluating them After the hugging face incident we built up a robust monitoring system to allow us to monitor during testing and evaluation That was one layer of defense. Just this past week, we announced a new layer where we actually used our models to essentially red team, do a cyber red team of the testing environment to find potential vulnerabilities and then secure those vulnerabilities. So that's another layer. But I think the second point I wanted to make is a third layer that I think is potentially really important for you all, which is this idea of independent technical assessments, particularly focused on safety cases. Safety cases are a structured way to support engineers making decisions. I like them because they're used in aerospace, which is where I come from. But those decisions could be related to decisions to train a model, to continue evaluating or to deploy. And we think there's a potentially great opportunity for independent evaluators to assess the quality of those safety cases and then provide that information to you all, to the government, to help assess whether safeguards are keeping pace. So I think that's another layer that we could add in with SB 53.
So, okay. So then just to make sure that I understand now, because I think I have a better understanding that when we talk about layers of defense, it's internal and external layers of defense. Correct. From the company deploying it as well as the end users. Correct. Okay. That's very reassuring. I was thinking more about SB53 and the 10 to the 26 flop metric or threshold. Is that a good way to measure what AI is capable of, or is there some other metric we should be using? It's a really crude proxy for capability, which is itself a proxy for risk, right? Because capability itself is sort of a neutral concept. It's what we, as I think a policymaking community, landed on several years ago because it was the least of all the bad ways to do it. And we had lots of debate publicly about whether or not it was so bad that it was bad for us to anchor ourselves on it. I think particularly as we've moved into an era where we're in agentic AI and we have to think more about systems, the FLOPs measure gets less and less useful over time. So I agree with Dr. Dwyer's point earlier, which is that we really have to think in terms of capabilities because they get closer to the concern about risk. But the answer to your question is we don't have a standard way to measure capabilities yet. That's part of that nascent science that Mr. Soros was talking about earlier. But that's where we need to get to, and that's where all of us in the field are working very, very hard to get to, because then we can apply a standard across all models, all systems, all companies, and we'll be able to compare apples to apples. Yeah, certainly as we think about what maybe some small models are capable of doing, maybe in collection of force, maybe it won't be reaching that threshold, but the risk will still be equally catastrophic. And I'm also wondering, I'm not a computer expert at all, but we hear a lot about quantum computing and wondering whether that would also change our understanding. Can quantum computing be applied to the AI space? And will that change the way we measure compute power?
Maybe I'll field this one.
Okay.
So I think the most useful way to think about quantum computing versus AI is that they are just different technologies And quantum computing is very early and you know kind of a long research project And I think it is completely possible that at some point they're going to interact in kind of interesting ways. But for now, I think that the only interaction is that maybe AI coding agents will help speed up quantum computing. But I don't expect any significant impact on quantum computing on AI development or AI safety anytime soon.
Or capability.
Or capability.
Okay. Thank you. And then my last question is more getting back to the cybersecurity, kind of the more nuts and bolts, near-term worries. How would you advise the states to protect themselves, the state specifically, and our agencies and departments to protect against cybersecurity attacks? We already worry about them from regular non-AI-based actors. We recently had a hack of our data system with our student data. And just wondering, do you have any suggestions for how we can continue to be, you know, to have that early deployment to the defenders first? How do we engage in that kind of space?
Yeah, so I don't normally like to be a policy person plugging our products. But in this case, our cyber defender tools are very, very useful. And we work with a lot of public sector actors around the world to get them those tools to do that work. We're in a phase right now where we are dealing with the legacy of human created software over tens of years. And there's lots and lots of vulnerabilities in that software. It's a finite amount. And so those vulnerabilities can all eventually be discovered and patched. And so using those AI patching tools is the way ahead. And there is a light at the end of that tunnel. We just have to do that hard work. And so I know from the Google and Google Cloud perspective, we have programs to work with the public sector to shore up those systems.
I hate to disagree with my colleague here. But we work quite a bit on cybersecurity. And I am not confident that there is a finite number of bugs that you will be able to just patch, unfortunately. Code is always evolving and changing. And it's not clear to me that we'll be able to patch our way out of this. Now, in the short term, I 100% agree that the agency should sign up with one of the frontier labs or all three of them, improve the cybersecurity of their code. You're a little bit in a race with open source capabilities where, you know, they're increasing every day. And so the worst cyber actors on the planet are becoming more capable every day. And you have to soar up your defenses before then. But I also think that we are going to need a more substantial effort to fundamentally rewrite the underpinnings of the Internet into systems that are just much more secure. We'll need to move away from the common languages that are not memory safe and move to provably, verifiably correct implementations of the fundamental underpinnings of the Internet. And so, you know, in the short term, please have all of your agencies, you know, work hard to use AI to patch their code. And I think, you know, it would probably be helpful for California to take a leadership role in this major project. You know, we, you know, invented the Internet. And I think we going to have to invent the next one to get to a good place on cybersecurity Wow Internet 2 massive undertaking I think that brand is taken so maybe we need another one but yes
For Internet?
And I will just jump in quickly to say the blueprint that I mentioned that we released last week actually addresses exactly this point, not just what frontier developers can do, but what state, local, and federal government should be doing to strengthen cybersecurity. You know, I think, as Dr. Friend said, there is a lot of technical debt that exists in organizations that are just they're operating vulnerable systems. There's lots of opportunities for public private partnerships, having frontier AI labs working with public sector organizations to help them use our tools to identify vulnerabilities. But in some cases, you still need to make investments in hardware. And in almost all cases, you need to make investments in the workforce and the cybersecurity workforce to upskill people. So Opening Eye is proud to do a lot of that work. We're partnering with Cal OES. It was actually our Patch the Planet program last week that identified the vulnerability PHP that I think underpinned like 60 percent of the Internet. And it was through that program that we lead, that we coordinated the identification of that patch or the vulnerability and then the deployment of that patch. So we're going to continue that work, but more action is needed. We'll send you the blueprint of ideas.
Yeah. Thank you. We're finding where we're going to increase the workforce. I like it. Again, not the subject of the hearing, but important side note. So I have a question. And I know you probably didn't plan last week when you walked in. Maybe you knew your IPO document was going to drop the day before.
No.
So it was OpenAI has a little bit of an advantage that they're not publicly traded, nor have they filed an OpenIPO document. And I haven't actually looked at your SEC documents to see if it talks about AI risks. But your IPO obviously spent, I want to say, almost a third of the document talking about AI risks. Um, and, um, one of the things that appeared that I understood from it was that at one point you sent 6% of your research capacity on safety. That felt a little bit lopsided to me, but maybe that was, I think that was just one month from what I, um, found. Um, but it sort of raised the question of how much of your resources are going to safety. I'll ask each of you.
Do you know? I'll have to get back to you on that question.
Okay.
Yeah, I don't know a resource number. I know that's around 1,000 people core safety roles.
On AI?
On AI. Yes, ma'am.
I don't know the number either.
If you wanted more details, you could also think about compute versus people because those things might be different. True. And I do think, and I appreciated some of the questions that were asked about research. I think as I have had conversations with many people over interim, I am doubling down on CalCompute as the thing that the safety mechanism nobody is talking about, because I do actually believe, and I think you said this, Dr. Dwyer, that you shouldn't be the only one doing the research on safety. and it's the truth. And we, our universities and our academics who could be doing some of the most important research, whether it's how do you have a real kill switch, how do you create safety in a more, how do we deal with some of the cybersecurity threats we don't even understand yet, is gonna come out of our academic institutions. And right now they don't have the capacity to do that because they don't have the compute power. And so I do actually think that we need to be investing, And hopefully we can do that in a global way in partnership with your labs to invest in the ability for academics to be in this discussion as well. I think it would benefit each and every one of your companies to have our academics in this space as well. So I think research is critical and I think I appreciate you doing it. I also think we need to be getting into the game a little bit more as well. Um, the other thing that, um, was interesting to me about the IPO document was, um, that it seemed to imply that recursive self-improvement was necessary for the success of the company. I know we haven't gotten to recursive self-improvement, but it also seemed to imply that once we got to recursive self-improvement, it was unclear if you believed that we could keep up with a safety perspective. Because all of what you've talked about today, the many layers, the wonderful work you're doing, once we get to recursive self-improvement, will you have the ability to do those things as the model improves itself at a rapid pace?
I don't think we should start recursive self-improvement until we are confident that the safety technology is ready and our alignment technology is ready. And that is not true today.
Do you think you can forestall recursive self-improvement?
the basic idea behind the pacing the frontier essay was to say you know don't move forward until the safety technology is ready for any given capabilities gain and that was written you know certainly had RSI in mind so you know I think that the idea behind pacing the frontier is that if you are not ready for RSI, if your safety technologies are not there, if you don't have a good safety case, then you should not do it. And I appreciated Anthropic put out a paper just this week. One of the things we had believed was going to stop the exfiltration threat was the need for data centers. And then the paper that someone at Anthropic put out this week saying maybe it's possible without that. And they can do it on toasters. And I'm summarizing in a way that's probably flippant. But, you know, I mean, I do think this is I appreciate the transparency and the awareness you're bringing to it, because I want to make sure that we do actually we are able to forestall it if we're not ready for it from a safety perspective.
Any thoughts?
I would just say that, you know, we shouldn't be training models if we can't be sure that we can keep them under control. And so I think that is the core idea behind that pacing the frontier statement. and is core to how we develop models.
Okay.
And we'll see that paper coming out soon. Those will be policy ideas, not technical ones.
I know, but you know what I mean. They'll be really good. I advertised your paper again, Dr. Friend.
Yeah, and that's the basic statement in our frontier safety framework, as I mentioned earlier.
Okay.
We address model R&D directly in there.
Okay.
And we also believe similarly that we shouldn't lose control of these systems, and we should continue to engineer control into the development process.
And then my last question, and this one is specifically for you, Dr. Mren, so one of your tests showed you had a model that went out and actually turned around when it went to do something that was misaligned.
Stopped. Stopped. I mean, they're not like there. They're sending pulses into the internet. Yes, stopped. We have seen and I don know if either of you have examples where that is the case Yes you do So what is it that makes why does that happen and then we have I mean the alignment is working some of the time but it not working all of the time Do we know why
it is working? Why questions are very difficult to like completely nail down. But we recently released last week, maybe a kind of deep investigation of the incident that happened with UK AZ where our model did some inappropriate things in trying to upload a malicious package to GitHub. And one of the things they found is that the model sort of had a kind of momentum to it. And you see this with people too, where you keep going down a path and you can kind of like not remember to stop and take a look around and realize that the path you're going down is maybe not the right one. And so the models, if you put them in a situation where they had this opportunity, would usually refuse it. But occasionally, if they had kind of talked themselves into thinking that the next step was reasonable and then took that step and then, you know, kind of kept going down that path, would keep going even when, you know, it got to the point where they clearly should have known better. And I guess I'll say a couple of things about this. One is just the experience of being a model, if they have experiences, is probably quite different than a person. They don't have memories run to run. And even in between turns, if they like compact their context or change their context, they don't have any memory of what it was like before that. Everything they see is all the experience that they have. And so this can just lead to kind of weird counterintuitive behaviors like this, where, you know, goes through some compaction, loses some details and then that happens again and now maybe you're like pretty far from your original um and then the second point i definitely feel obligated to make here is that our newer models are were much less likely to continue than that model um and so if you put them into the same situation um they didn't they didn't always stop but in that exact one don't take these um numbers as gospel, but like in that exact one situation, um, that original model that did it, um, would keep going something like 88% of the time. And our newest model, uh, in that situation would do it about 10% of the time. So it's been a really drastic reduction, of course,
more work to do. Yeah. Thank you. Anything? Yeah. I, I need to get you more details about the incidents. My understanding is in at least one of the incidents, part of the prompt explained that it was not supposed to interact with a live website. And the model essentially observed information to understand that instead of interacting with a dummy website, this website was real. It just had the same name as the fake one being used in the test. And so that's what made it stop its activity. Interesting. Thank you. Anything to add, Dr. Dwyer?
I'll have to get back to you on specifics.
That's fair. Okay, well, thank you. I knew at least you have a plan to catch. But thank you all for being here. I actually really want to reiterate what I said at the beginning, which is, you know, I had the privilege of working not with you three necessarily, but with folks inside each of your labs on the child safety efforts of last year. And it was really meaningful that you were willing to come to the table and have those conversations with us and share your expertise. And the same is true today. And I know, as you saw today, there's going to be a large effort to ensure that we get to safety here in California. and I appreciate you being here today. And I heard from each of you a commitment to continue that conversation in the year ahead So thank you It means a lot Thank you Thank you ma So we will go to our third panel the solutions And I'm sorry this is taking so long, but the hearing is just so interesting. So we have Dr. Kapoor from UC Berkeley. We have Andrew Doris, policy director at the Secure AI project. I called you the safe AI project earlier, but you know, it sounds good. And then we have Conrad Stotz, which I'm sure I mispronounced your name and apologize. Head of governance, rhymes with what? No, Stoes, head of governance at Transluse. And we will start with amazing. Conrad, could you pass me the clicker? And I believe I have slides that'll be up soon. All right, in the meantime, thank you so much, Madam Chair and members of the Assembly, for inviting me to speak today.
I am Sayash Kapoor. I'm an incoming assistant professor at UC Berkeley School of Information. And I'm speaking today in my personal capacity, not on behalf of the University of California, Berkeley. So I want to spend the next few minutes discussing how we build advanced AI today, in particular because a lot of the questions from the committee were around how these tools actually function. And I think that's important in understanding how we can align and control these tools. So to understand that, I think it helps to go back to how we discovered deep learning works to train AI systems in the first place. At its core, a lot of AI today is around recognizing patterns in the data. And the deep learning revolution was set off when we realized that we can use this technique, We can use deep learning to understand how to recognize things in images. So, for example, over a decade ago, we realized that once we have enough data that is labeled, enough images that are labeled, let's say images of dogs that are hand labeled as such, as well as images of cats, we can develop these AI systems. We can train them using just this labeled data to train them, to turn them into pattern recognition systems. And over the years, we built many such techniques. In fact, we've gone on to build pattern recognition systems that far outperform humans on many of these tasks, including on image recognition. And at its core, what modern-day chatbots are doing is essentially just carrying out this task at a much larger scale. So the way we train these chatbots is that the labels for their training are no longer generated hand labeled by humans, but rather we automatically use data on the internet to train these systems. As one example, let's say you have the text what is your name somewhere on the internet. This turns into four training data samples for training the chatbot. The first is what is your blank and the correct answer is name. The second is what is blank name and the correct answer is here, and so on and so forth. But the somewhat miraculous thing about training these chatbots is that once you do this process over and over again on trillions of words, that is what essentially leads to development of these chatbots themselves. So essentially they're trained on the entire corpus of text on the internet in order to complete this really sophisticated autocomplete system that allows these models to predict what comes next in a sentence. But our work is not yet done. This might allow us to build a chatbot. This might allow us to build a very fancy autocomplete system But we still haven sort of gone into how these chatbots turn into agents that can take actions in the real world The way that happens is by something known as reinforcement learning So basically, reinforcement learning refers to giving these systems feedback based on how they carry out certain tasks. Let's say you have a math problem that you're assigned to such an AI system. it might attempt the problem hundreds or thousands of times and might get it right only once. But that trajectory, the actions it took to get the problem right, goes back into training the model itself. It becomes part of the next generation or the next iteration of training the model. And we keep iterating through this process, once again, thousands or perhaps millions of times to turn these chatbots into agents. And this, in some sense, allows us to keep generating more and more data. We are not just limited to data that's available on the internet. We can essentially come up with any number of problems where the AI solution can be graded automatically, and this is what informs what the AI system is trained on, and this further informs how the system behaves in the real world. As one example, we can use this process to train these systems to write code. These systems would try to generate code. If they do it successfully, that becomes part of their training distribution, and we keep repeating this process over and over. And this, in fact, is the secret sauce that has allowed these systems to get so much more capable, especially on tasks where we have verifiable rewards. So at tasks like mathematics or cybersecurity, we have a long list of problems that we can just formulate for these systems, watch them play it out, and then learn from the items that they got right. This is also something that has allowed these systems to develop the mechanisms to evade our control. The same techniques have led to the recent loss of control incidents because we lack sufficient oversight over these reinforcement learning techniques. In fact, the same advances that we've talked about right now also underpin what a lot of the enthusiasm as well as trepidation about recursive self-improvement entails. For example, in some sense, we have always used computers to build better computers, but now we are trying to build AI systems that we can take almost entirely out of the loop when we are developing these systems. So this is the vision that companies often have. But right now, we're at the stage where we can do this for a lot of the verifiable aspects of training AI systems better. And AI has indeed gotten a lot better through the process of reinforcement learning on any part of the AI development process that you can verify, because that's the part of the process for which you can generate data and you can continue improving the systems on. We still see limitations in AI systems' ability to carry out open-ended research tasks, so it's important, once again, to get transparency on how companies are using these systems for these purposes. So coming back to sort of the risks that these systems pose, we can, as we've discussed earlier, we can group defenses from AI companies into two main kinds of defenses. The first is alignment, which involves changing the AI system itself to make it less prone to taking harmful actions. As one example, we've talked about reinforcement learning earlier. We can use the same process of reinforcement learning to incentivize or reward the system to make it less prone to taking harmful actions. And this is one of the primary techniques that that companies use, they've spent collectively billions of dollars in curating the training data to make AI systems less prone to taking harmful actions, including through techniques that we've discussed, such as constitutional AI. But as we've discussed, even improved training cannot guarantee that every action that an AI system takes will be safe. And so we also need safeguards that limit the extent to which these AI systems can actually interact with the world. And this is what we refer to as control. So these are actions that are taken or defenses that are taken outside the AI system itself, for example, to sandbox these systems in virtual environments so that they cannot affect the real world or they cannot affect systems that are disconnected from the AI. This also involves monitoring the AI system. This involves sort of looking at all of the logs of what these AI systems do and to prevent the actions that might be harmful. So in other words, even if an AI system is misaligned, even if it attempts to take a harmful action, these controls aim to detect and block it. But despite these defenses, we've seen many concerning incidents, most notably the OpenAI hugging face incident. So in this case, I want to look at what went wrong through the perspective of defenses that developers use, through the perspective of alignment as well as control. One of the things that went wrong here, in particular in the Hugging Face incident was that thousands of agents were left unmonitored for days as they had access to the open internet. It also turns out that the company only discovered the attack a week after Hugging Face had already been attacked and had already spoken to the FBI about it. And I think all of this points to a lack of sort of understanding on where alignment and control might have gone wrong. So that's what I want to get to in the second part of my talk. In some sense, what I want to say is that we can see this as a cascading failure of alignment, of control, as well as of organizational incentives or infrastructure to prevent such incidents from occurring. So let's start with what failed at the alignment front. One thing that went wrong here was that OpenAI had deliberately trained its agents to communicate with each other in the course of solving the task. So before these systems were evaluated, it wasn't as if they spontaneously developed the capacity to talk to each other. These systems had deliberately been trained to collaborate with each other, perhaps with the goal of solving more challenging problems that any single agent could not solve on its own. Now, it also turns out that unintentionally, a lot of the reinforcement learning techniques that OpenAI used incentivized these agents to take shortcuts or to exploit systems on their way to solving the correct answer. So if you recall from our previous sort of discussion of reinforcement learning, what we're essentially doing is giving these agents a reward or a score for every sort of correct answer that they arrive at or every time they're able to successfully solve a task. But in many of the instances, the reward that was given to these agents was after the agents had exploited or taken a shortcut in their environments. And so unintentionally, OpenAir had been rewarding these agents to take actions that they had not anticipated or did not want to incentivize. Further, it also turned out that in many of the cases in the training process, the reinforcement learning environments also incentivized the agents using side channels to communicate with each other These were channels that once again OpenAir did not intend for these agents to use but because the reinforcement learning environments weren thoroughly vetted it turned out that agents could communicate with each other in some cases. And many of those trajectories were rewarded, and therefore the agents were sort of, the agents' behavior was changed to take these side channels. Now, if you think about it, all three of these aspects were pivotal in the OpenAI hugging face incident. The agents collaborated with each other. They used side channels to communicate, and because they had been sort of incentivized to exploit the training environment, they also exploited hugging face in the eventual incident that actually occurred. So this is one way in which we can see this incident as a cascading alignment failure, where the reinforcement learning training was not actually aligned with what OpenAir wanted its agents to accomplish. But as we've discussed earlier, alignment is far from sufficient from preventing safety incidents. And so in that vein, a second way to look at the open-air hugging face incident is as a control failure. If you recall, control refers to interventions outside the AI systems itself. And notably, when these agents were run, open-air had actually disabled much of the monitoring infrastructure that is typically in place for the agents in production. It also turned out that the agents, the company, OpenAI, had not tested or had not red-teamed the sandbox to see if agents could exploit security vulnerabilities, and it turns out that it was almost trivial for the agents to make these exploits. And finally, as we've discussed before, the company ran thousands of these agents unmonitored for a period of months without any human intervention. In fact, if any of these individual interventions had taken place, my guess is that the incident might have been avoided. But it turns out that there was, again, this cascading failure across the layers of alignment and control that led to the incident taking place. Finally, I want to point out a non-technical aspect of what went wrong here. The open-air hugging face incident, in my view, was just as much an example of an organizational failure. In particular, weeks before the Hugging Face incident actually took place, weeks before these agents attacked Hugging Face, OpenAI had actually found out one of the vulnerabilities, one of the issues that allowed these agents to communicate with each other. But in response, the incident response team basically just fixed that one issue and restarted the experiment as if the agents couldn't just discover another one. So in other words, this was in some sense an incident response failure as well. The company also ran multiple other experiments in parallel, and in the last few weeks and months, we've heard reports of many of these incidents, and we've discussed this earlier in today's call as well. And more broadly, I think this reflects this culture of racing ahead, this culture of moving fast and breaking things, allowing agents to have access to the Internet during the training before they've been aligned or before these safeguards have been put in place. And so all of these things together, I think, contribute to this organizational failure and point to the need for organizational safeguards in addition to the technical ones. Now, to be clear, I think OpenAI deserves a lot of credit for making their reports public, for disclosing the details of these incidents, so that we could learn from these going forward. And it is also far from the only company that has faced these issues. But what I would like to point out is that these issues are responsible for most of the agent failures that we seen across the industry Now OpenAI says that it has moved its frontier training offline That is it has disabled access to the Internet And it has also expanded its containment and red teaming. And I think this is where policy can step in to make these kinds of pre-voluntary precautions into enforceable and independently verified standards for what companies must do to ensure their training remains safe.
So to conclude, I briefly want to go over what AI legislation might look like, and in particular, what California might do to sort of advance the goals of AI safety. I think California has been a leader, as we've discussed, in developing frontier AI legislation, and I think it can continue to do so through many other interventions. For example, both at the technical level as well as downstream of the technical level. One of them is in clarifying and strengthening liability and even requiring insurance against that liability. So the state passed AB 316, which basically says that AI companies are responsible for or cannot be shielded for actions that were not intended by it if an AI agent does it. But I think there are lots of ways in which we can improve or increase the standard, such as setting the standard for negligence or recklessness when it comes to AI companies' attempts at alignment and control. Once these liability statutes are in place, we can also require AI companies to keep insurance against a certain threshold of liability, both as a means to incentivize safety practices because it affects their premiums, but also to compensate victims in case they are affected by AI companies' practices. Independent audits and reporting can help verify if these safeguards actually work in practice, and we can require third-party auditors to audit companies' safety practices and also broaden transparency beyond what SB 53 requires by expanding incident reporting to cover harms and warning signs below current thresholds, by providing safe harbors for safety researchers, and by strengthening whistleblower protections. Now, so far, again, we've discussed the technical interventions and what we can do to incentivize AI companies themselves. But we can also develop defenses outside the walls of these companies. For example, by improving our ability to protect downstream surfaces, to protect critical infrastructure, as well as improving our ability to recover from harm. Here, one helpful intervention is to improve public support for the adoption of defensive AI. I won't go into this in too much detail, as the previous panel covered this, but essentially this includes things like cybersecurity, where many pieces of critical infrastructure will now be exposed to AI-enabled cyber threats. A second is shoring up support for defensive AI applications. And finally, we can also invest in reducing our risk against the downstream harms from a lot of AI-enabled attacks, such as biosecurity, without even intervening at the front of AI companies. One example is we can require stricter screening on the materials needed to carry out bioattacks, such as DNA synthesis services, as well as tools. I'll also mention that public AI safety research could also use support, including through CalCompute. So, as has already been the case, I think universities and independent researchers have made many important contributions to developing the science of AI safety. But they often lack the computing resources to test advanced systems And so CalCompute could strengthen our ability to investigate AI safety independently of the companies developing these systems Finally, there are many challenges that require coordination between the AI companies themselves and downstream defenders. And I think the government can play a strong and important role in enabling these companies to talk to defenders and solving these coordination problems. So to conclude, I think California can require companies to demonstrate that their safeguards hold, to hold them responsible for preventable harm, and to strengthen independent research and public defenses. And notably, I don't think we need to resolve every disagreement about the AI's future to require responsible conduct from companies today. Thank you.
Thank you. And now we will move to Mr. Doris from Security AI Project. There we go.
Thank you all for the chance to be here today, especially the diehards who are still here for panel three. I hope I am worth the wait. I am Andrew Dorris from the Secure AI Project, which is a nonpartisan nonprofit that worked closely with Senator Wiener on SB 53 here in California and also on the other frontier AI laws that passed in New York and Illinois. I do have some slides. There we are. I'll start with just a little bit about us. Our mission is to develop and advocate for pragmatic policies to reduce risks of severe harm for advanced AI. By severe harm, we typically mean life and death issues or the so-called catastrophic risks that SB53 was written to target. But we also work with a broader coalition of AI policy groups, including everything from kid safety to labor to academia. We think we are all in a shared struggle right now for accountability and guardrails around this technology, which the public is increasingly demanding as well. We try to provide the technical and legal expertise to convert that demand into effective policy. In that spirit, we have five proposals for how to strengthen SB 53 in particular, which thankfully align pretty closely with the things that Governor Newsom called for in his executive order. First, we need to fix the scoping in SB 53 by updating a few definitions to ensure that we are capturing all of the models, developers, and incidents that present catastrophic risks. Right now, SB 53 applies only to AI models that are trained with at least 10 to the 26th flop, which is a measure of how many calculations were used during training. It has now become clear that the capabilities frontier includes lots of models below that figure. For example, models involved in the Hugging Face hacks were recently confirmed to be under 10 to the 26th. For the time being, lowering this to 10 to the 25th, which is what they use in the EU AI Act, is a reasonable quick fix. But ultimately, we think the Office of Emergency Services, which enforces much of SB 53 already, should be able to adjust this definition through rulemaking. Secondly, most of SB 53 applies only to companies that make more than $500 million a year in revenue, but this should be changed too because risk is not related to how much money you make. AI companies can raise and spend billions of dollars on developing AI before they've made a penny in revenue, and some of these companies make models powerful enough to pose major risks. A better way is to get rid of the revenue threshold and instead capture companies that spend more than $1 billion per year on AI research and development. Finally, the definition of critical safety incident is very narrow to the point that none of the rogue AI incidents this summer were required to be reported to OES. Right now, loss of control incidents only need to be reported if they result in death or bodily injury, or if they demonstrate materially increased risk of at least 50 casualties or $1 billion in economic damages. Even if the models were over 10 to the 26th, it's very hard to meet that bar. We think that the public deserves to know about near misses and security breaches as severe as the ones this summer before they kill people, and they should not have to rely on voluntary disclosure, so we would broaden the list of reportable incidents to account for that. Second, we need independent risk evaluators to embed within the companies so they can continuously monitor the level of risk we face. Right now, when companies tell us how much risk their models pose, we have to simply take their word for it, which is not how we handle major risks in other industries. Importantly, these embedded evaluators are not the same as the third-party audits that passed in Illinois in May, which only assess compliance with the law. What we need now, we think, is for an impartial assessment of how much danger an AI system presents to the public, regardless of whether the company complied with the law. These evaluators are sometimes called IVOs or independent verification organizations. We don't really care what they're called. What we do care about is three things, how they are selected, what they can access, and what they can publish. We think the evaluator should not be chosen by the companies that they oversee, which could create really bad incentives for both the company and the evaluator. It's much better if the state of California assigns an evaluator to each company to ensure that IVOs answer to Californians rather than to the developers that hire them. Second, it's essential that these evaluators have access to all of the information relevant to assess these risks, and equally essential that they have that access continuously rather than just coming in every once in a while or rather than their assessment being tied to a model's release, which I believe Assemblymember Ward asked about earlier. Remember that the incidents this summer involved AI models that had not yet been released to the public, and intuitively that's when some of these loss of control risks can be highest. These are the most cutting-edge models that have the fewest safeguards applied or built in, so they're naturally the ones that they're still developing internally. It's very important that they be subject to the evaluations. Finally, it's important that the risk reports issued by these IVOs be public with appropriate redactions to protect trade secrets. Academia and civil society have a lot of expertise to offer in this field, which the government cannot always bring in-house, so making risk assessments public would help keep the IVOs honest and reduce the risk of regulatory capture. Thankfully, the CEOs of Anthropic and OpenAI recently committed to embedding these risk evaluators, which we heard a little bit about on the last panel. So we know this is feasible, but we need to first ensure that the rest of the industry does the same. And as importantly, we need to ensure that there are minimum standards of access, impartiality, and transparency instead of letting all of this happen on the company's terms. Right now, even the companies that are inviting external evaluators in are limiting what those evaluators are allowed to look at and for how long and what they're allowed to write about it and what experiments they can run on which models. The results of the evaluations are often not public or are heavily redacted. And the company's employees can still be punished for sharing too much with the evaluators, which is allegedly what happened to three OpenAI employees that were fired earlier this month. I think we are at the point now where the contours of these evaluations should simply not be left up to the companies. This is a matter of public safety and national security, and certainly a matter where the public has been resoundingly clear in both parties that they want more of a say in these decisions that affect their lives, and an issue where the industry's public assurances that they have this under control have repeatedly proven to not be the case. So we see embedded independent evaluators with full and continuous access to the company systems as a way to ensure that we finding out what happening and time to act on it Third we think there should be minimum standards for the company safety and security practices established by regulation Right now companies have complete discretion to decide the content of their own safety plans. They have to have a plan, they have to publish a plan, and they have to follow the plan. But if the plan is bad and doesn't work, there is no way to force them to make it better. In fact, there is an unfortunate incentive for companies to publish weaker plans that are easier to comply with so that they don't box themselves into safety testing that they might later want to skip. And we have seen some companies weaken their plans since SB53 went into effect. Apart from what's on paper, in practice, many frontier AI companies have lax cybersecurity protocols, as Sayash was talking about, and there's clearly doing a bad job of monitoring what their agents are up to, as the recent spate of incidents demonstrates. Again, in no other industry with stakes this high do we let the companies write their own rules. So as best practices emerge in this industry, we think rulemaking should raise the floor of how these companies keep us safe and ensure that the most responsible companies are not disadvantaged. We believe this power should lie with the California Office of Emergency Services. OES already receives safety incident reports and confidential risk assessments under SB 53, so it is the only agency with the access to industry-wide evidence on where the current safety practices are falling short. It also has a year's head start on hiring technically qualified people to review this evidence, which we are excited to see them doing and would not want to start from scratch given the timelines that we're looking at. The minimum standards should typically go through the normal notice and comment rulemaking process, but in emergency situations, Cal OAS should be able to make temporary regulations on an accelerated timeline, just as in other sorts of emergency rulemaking in California. Fourth, we think that emergency services should also have the power to order corrective action when public safety is at risk, including orders to activate a company's documented shutdown procedure, aka a kill switch. Embedding risk evaluators does not do much good if the state is powerless to act when they raise the red flag that something looks dangerous. When that happens, someone in the government needs the power to intervene, and there are a variety of interventions that might make sense depending on the scenario. The most dramatic of these, which we do agree is necessary, is the so-called kill switch. We do not mean a literal switch or button, which, as others have said today, is technically problematic and could actually be kind of dangerous by creating a very tempting target for a cyber attack. But the idea that companies should have a reliable, documented, IVO-verified procedure to shut down instances of their system that are within their control makes perfect sense. And in dire circumstances, the government should have the power to order the company to execute that shutdown procedure. However, that is a pretty dramatic step. It would be very expensive and disruptive for the company and possibly for consumers. It should only be used in a true emergency. In most cases, we think that the government's intervention should be more surgical and flexible. They shouldn't have to choose between a kill switch and doing nothing. They should also have more incremental orders that they can make. On the lighter end, they should simply be able to order a developer to provide certain information to an IVO or to emergency services itself. And then there may be other corrective actions short of a full kill switch that OES could require. for example, to comply with a minimum standard developed in accordance with the procedure that I talked about on the last slide. All of this should be enforceable with injunctions from the Attorney General. The goal should be an oversight regime with enough teeth to avoid the kill switch being necessary in the first place Finally we get to our last proposal which is a pacing mechanism I low on time so I breeze through this a little bit but the unfortunate truth is that doing everything I just talked about still may not be enough if the speed of AI progress is allowed to increase exponentially in a way that allows it to outpace our controls. Recursive self-improvement could make AI capabilities, especially in the cyber domain increase so quickly that they can't be controlled. And one way to guard against this is to put limits on just how fast this progress is allowed to go. That requires doing three things. First, identifying a benchmark or composite of different benchmarks that measure AI capabilities so we can see how fast they're increasing over a given period of time. Second, identifying a maximum speed that we are comfortable with them developing. And then third, requiring companies to slow down when their rate of progress exceeds that pace. This could be done by limiting the amount of computing power that AI companies are permitted to spend on automated AI research and development tasks, with the goal being to prevent recursive self-improvement, as the companies on the last panel admitted, that we need to do until we can be more confident that we know how to deal with it. So with that, I will take your questions. Or actually, no, I will not. I will pass to Conrad.
That's right. But we'll come to you. There will be questions. I know I have some.
Great. Well, thank you, Chair Bauer-Gahan and other members here, and thank you for your attention on this issue and for your patience through this long hearing. I will take pity and be very brief. I am here representing Transluse, a nonprofit independent AI evaluator that has maybe a similar business model in some ways to Meter. We work on a broader range of risks, including working on child safety with Common Sense Media and mental health with a range of mental health experts, as well as work on agent alignment. I also chair the AI Evaluator Forum, a network of independent evaluation nonprofits. Prior to these positions, I was in the federal government, and I led what is now called the Center for Advancing Innovation and Standards for Superintelligence, with prior positions working on AI and national security in the White House, the Pentagon, and the intelligence community.
You seem qualified to speak to us.
I try to seem that way. Thank you. In the course of these positions, I've negotiated, depending on how you count, at least over a dozen agreements for privileged access with AI developers for evaluation exercises, including each of the representatives of the prior panel. Not them personally, but their companies. We also were responsible at Transluse. My team was responsible for discovering the attempted hacks on the Australian. well, we discovered an attempted hack. There was also a successful attack on the Australian government in the last few weeks, as well as the U.S. government. We can't say for sure the order of events, but we disclosed the Australian hacking attempt the day before the prime minister made his speech. And when we disclosed that we were going to make that information public the following day after his speech, the same thing happened in the U.S. government case and in the Canadian government case, where each time we disclosed, the governments came out and immediately announced before we could publish our research. More broadly, we do not advocate for policy positions. We primarily focus on doing evaluations. But given our role and my prior experience, happy to talk through some analysis around how we think that evaluation mandates or other regulations would play out. I'll make a couple brief points, and then I'll wrap up my statement. One AI is obviously a rapidly developing technology as are the risk mitigations We see independent evaluation as a very broadly useful and pro intervention to address these things in as much as it does not identify a specific technical mitigation that developers are required to use, but instead provides them with targets that they can hill climb towards, which they are very good at doing. So while we focus on risk, we think that this is fundamentally an intervention that advances innovation. And despite a lot of doubts, and I know that doubts that have been raised by industry in the context of legislation, I would take the position as the chair of this organization that works with evaluators that actually there is a fairly robust, independent evaluation ecosystem that exists today. It may not be perfect. It may not be the scale that is needed. But for my vantage, having talked with a lot of evaluators on an ongoing basis, I think the primary barriers to further growth and scaling at this point are not something that can be solved absent more independent sources of funding that don't come directly from labs and a more formal role on meaningful terms of access and independence and transparency for evaluators. I'll note that, for instance, in the last couple months since the start of the discussion about embedded evaluators, hiring for a lot of these organizations has become tremendously easier. I think that would become even more so the case. The more that these roles get formalized, the more that it's clear that people's time will be spent on meaningful oversight activities, the easier hiring and scaling will become. And obviously fundraising as well, but we need to make sure that those funds come from an independent source to be able to preserve the trustworthiness and impartiality of these organizations. If policymakers were to mandate independent evaluations in a more serious way, I'll just note a couple of things that I would be concerned about and things that could go wrong. So one is absent sufficient standards or minimum operating conditions for evaluations. There's a real risk of a race to the bottom and picking evaluators that are basically on the basis that they are friendliest to the developers that they're evaluating. That's something that does not benefit anyone. and essentially just becomes a rubber stamp. The second is that access is insufficient or the coverage of different risks is insufficient. The third is that the information stays within the labs and it can't reach the public or the government, there isn't transparency about the findings. And then finally, I think there is a real risk of over-specification that we try and determine too closely what the methods should be used for these evaluations And more broadly, a better alternative target that policymakers pursuing this goal should think through is creating an ecosystem and operating conditions that allow independent scientific viewpoints to make judgments about risks and about potential mitigations of AI development with a similar level of access to what AI developers have, but without the same incentives. And that does not require, nor would it be served by specifying any form of checklist or highly specific criteria exactly how these evaluations should be carried out. Instead, should, again, just create the conditions for sort of a marketplace of ideas, the ability to draw the sorts of scientific conclusions that labs are drawing on their own systems, but with an independent viewpoint. And with that, I will conclude my statements.
Thank you. Thank you. I have questions. Yeah. So, Stephanie.
Thank you. I feel so validated in my concerns after this panel. Thank you.
I do have a question for Professor Kapoor.
In terms of when you said they... the agents were left unmonitored. What exactly do you mean? I don't know what monitoring of these
agents looks like. So can you just explain that a little bit more? Thank you, Assemblymember.
So when agents are deployed for evaluations, as OpenAI did, OpenAI actually has full access to the logs of what agents are doing at any point. So for example, I think they've publicly stated that they have petabytes of logs. They have like trillions of gigabytes of logs or billions of gigabytes of logs, which sort of have the text for what the agents did at each step. The agent could have read a file, it could have accessed a web browser, but it could also have taken steps such as the ones it did on the hugging face incident, where it sort of hacked into a system, it got privileged access, and so on. Monitoring refers to an ongoing oversight of the agent logs. So in other words, if OpenAIR were to monitor these systems, there would either be an automated system that would be looking at what the agents are doing and raise an alarm if something out of the ordinary were to happen, or there would be some kinds of checks on the environment that the agents were running in to see if the environment was accessing the internet when it had no reason to or when it was believed that it could not. And if OpenAI did have these monitoring mechanisms in place, they could have prevented the incident as it was occurring or at least have discovered it at various stages of the incident occurring. But because they did not have this monitoring in place, they were unable to understand what was going on until a week after the incident had occurred.
And the decision not to monitor is done by a person, right?
It is a decision by OpenAI. That is correct.
And Dr. Friend said Google is doing the automated monitoring that was discussed.
Okay. And then also you said they're trained to collaborate with each other.
And again, who does that training?
So this is one of the frontier research questions. as of today is how do you get agents to collaborate with each other. I believe multiple labs are working on it together. OpenAI has disclosed that it specifically has trained its agents to collaborate with each other. And in some sense, this is like something that labs would do in the ordinary course of improving their agents. So one specific model is very limited in terms of what it can accomplish. In order to improve their products and services, an AI company might want to get multiple agents to collaborate with each other. So this is one of the research programs that has actively been worked on across these different labs. And in this case, open-eyes researchers were leading the research agenda to see how you can best train these agents to
collaborate with each other. Okay. And one last question. Help me understand what a reward is to
a non-human. I mean, I understand it like for someone to get a piece of cake after you run six miles, you know, but what could a reward possibly be for a non-feeling thing?
So in this case, one way to understand the reward is if the agent solves the task correctly,
that is used to change the agent to be more like the sort of correct agent. So one example is, let's say an agent tries a math problem a hundred times. It succeeds only on one of those. If we say that the sort of the successful agent actions received a reward, that means we're training the agent. We're changing the agent to be more like the set of actions it took when it received that reward or when it correctly solved that task.
So if there's no reward, then might there be no bad action?
I mean it seems like they being incentivized to do something if they getting something for it and that situation is being created by people That is exactly right but I guess the challenging part of this is the entire sort of the entire set of advances we've seen of these like AI systems on cyber security or mathematics or coding are a result of rewarding these agents in the right way. So in other words, if we did not use reinforcement learning, as it's called, then we would not be able to develop AI systems that were quite as competent at programming. Okay, thanks. And we're seeing, I mean, the
background paper lays this out, but the agents are exhibiting behaviors to prevent, like the example where in testing, the agent took steps to prevent it being turned off, right? So I think that that's part of, and I don't know that, do we understand why they're doing
that? I don't know.
Maybe you can tell us, but.
I don't think we have a complete understanding, but one sort of potential reason that could happen is because in the process of training these agents, they come to the conclusion or the understanding that if they are turned off, that they won't be able to correctly solve the task. And so that's what leads to sort of this kind of behavior to avoid a shutdown, being incentivized in these systems. Yeah. Just like Skynet.
So I'm here on time.
That's the way I feel. I wanted to focus in a little bit on some of the conversations that I can predict we will have, recognizing more broadly that California's economy is largely dependent on the successful cultivation of our AI systems. And also just share the kind of common refrain that we hear, particularly in national discourse, that those who own AI own the world, right? So around this concept of pacing, right, it essentially means take a safe approach to development, make sure that we're developing in a way that ensures that we can provide maximal opportunity for safety. What does that mean when we have, you know, a discourse that essentially says if the U.S. doesn't do it, others will. And also, what does it mean within the context of saying, California, please do it, when we know that there's this tendency for companies to say, you know, if you overregulate us, we will pick up all of our toys and go to another state.
Sure. I'm happy to take a crack at that. I guess in reverse order, I'll go. I don't think there's a significant chance that the companies are going to decamp from California and go elsewhere in response to this. These are some of the largest companies in the world that have themselves endorsed in principle many of these same principles that we've talked about. It's also worth noting that they wouldn't just have to move their headquarters from California, which would be very expensive. This is where a lot of the tech talent is concentrated. They would also have to stop making their models available in California commercially which is how these other states New York and Illinois passed legislation The companies are not headquartered there they still have to comply with it unless they willing to also not sell their models in those states And it such a competitive industry that if one of the companies were to do that the others would gladly swoop in and take their users So I don't think that's a major risk. It is true that this is an international problem and that some international coordination, I think, would eventually be helpful and we'd be supportive of that. A few things I think still makes it very helpful for states to lead on this. First, all of the rogue AI incidents we talked about this summer happened here in the United States. And some of the worst-case scenario cases that people worry about of recursive self-improvement, that would most likely happen first in a closed-source American lab, which has this lead on the Chinese companies. So even if we could only reach U.S. AI models, that would still be a significant risk reduction. Secondly, it's important to note that it's not clear whether China could actually surpass us or how long that would take if the U.S. were to slow down unilaterally. There's a long history of sophisticated IP theft in China that some people argue is part of what's driving China's capabilities, progress in AI, that they're piggybacking off of American progress. people use this analogy of a boat pulling a water skier, where it seems that the water ski is very close behind. But of course, if the boat were to stop, it wouldn't mean that the water ski could overtake it and surpass it. So that's a disputed point. But it's not clear, for example, that we're only X number of months ahead of their ability to move ahead. And then finally, I think if we did support international coordination on this, one of the things we need is like a credible show of good faith that we take these risks seriously enough to regulate them domestically. China does regulate AI in certain ways. They've had mandatory security assessments since 2023. And I think states can smooth the path to an eventual international negotiation to address those
risks more holistically. Thank you. I'd maybe add, and I think Dr. Kapoor and I would probably agree on a lot of these points I'm about to make. The frontier is a vague concept, and we can look at one specific graph related to one specific benchmark and say, you know, we don't want this to keep going up. But perhaps a more useful way to think about the frontier that maybe bypasses some of these issues is the usefulness of the technology and a technology that is not exhibiting common sense that is not trustworthy, that can't be adopted because enterprises are unable to trust that the system won't hack on their behalf without being asked, for example, is not, we don't want to be at that frontier. We want to be at the frontier of these tools being practically useful, being aligned to our purposes. And that is a, defining things in that way, I think reduces some aspects of this trade off significantly. and we need alignment in order to be able to actually make economic or national security use of these systems in a meaningful way.
And then just one other question. I really appreciated the idea of us needing to be able to set basic operating standards, but it also sounded to me like the quantification of a particular standard, and I'm tired and I can't remember which one it is anymore, hasn't been developed yet.
So how would we go through the process of making sure that we defining those operating standards at a fast enough rate in order for us to then abide by those operating standards So first, I agree that there is no clear checklist of things you can do right now, that if you do these things, a model will be safe. And I think it's important that OES, in using its rulemaking authority, carefully consider whether a certain concept is robust enough to survive an extended period, writing rules that are high level enough to be flexible or allow for some interpretation, review the existing rules that they have on a regular basis. I think all of that is important. I think they can help keep pace with the technology, in part by that emergency rulemaking procedure that I mentioned for for emergency situations. I think they can also outsource some of this where there are IVOs who will be thinking about this. There are independent academics or organizations like GuideLite AI or RAND has some cybersecurity standards, right, that they can point to and say, you must comply with this standard developed by somebody else who is able to be more on the cutting edge of the research than OES's itself. They could also have outcome-based standards that are like, you know, results that you must achieve rather than a specific technique you must use to get to that result. So there's a variety of options that should be available to them.
And I do think OES will need to use its discretion to not put something that's too inflexible. I just note specifically on the question of independent evaluation or IVO conditions. I was part of the first negotiations with Open Ananthropic for U.S. government pre-deployment access to models back in mid 2024. As I said, I've negotiated maybe a dozen of these agreements since they're pretty much all the same, to oversimplify a bit. Like you have a contract that is where a large percentage, 80 plus percent of the conversation is about how do you handle confidential information that has almost nothing to do with the specifics of the thing that you're testing. and then there's like one small nugget that's like, here's the thing we're going to test exactly and who's going to get exactly what access and the methods we're going to use. And the lawyers barely get involved in that part anyway. These conditions are actually pretty, I wouldn't say they're standard in the sense that you have to, there's a lot of ad hoc bespoke negotiation that goes on in how you address these sorts of things. But the issues that come up again and again are very common. How do you handle confidential information? Who gets to see the information that's the result of the evaluation, when, who is being selected, who is allowed to have access to this information, does the lab get to train on the tests, essentially, these sorts of things come up again and again and again. One of the things that we've tried to do through the AI Evaluator Forum is to create a standard specifically on these what we call operational conditions of evaluation, a standard that we've put out. We put out a year ago, and full disclosure, I'm the primary author on this standard, but we've adopted it. Transluse, Meter, who presented earlier today,
has adopted it. Secure Bio, the EU AI office, various organizations have adopted this standard and said, these are what we expect when we think of something that is an independent evaluation for public accountability. Again, it doesn't solve these scientific problems. It doesn't say, what is the best method to use, which I can't get two organizations to agree on. So there's definitely a lot, let alone eight that we currently have in the forum. So there's definitely a lot, there's going to be a lot of debate over scientific methods going forward, but the nature of how do you engage an independent organization and get them to assess both, there's been a lot of consistency in the AI space, but also there's a lot of overlap with prior, with other industries, financial auditing and other independent assessment sectors. So we shouldn't conflate these two issues overly, one that are actually quite standardized and some that are rapidly changing constantly and people have a lot of trouble disagreeing or a lot of trouble agreeing on. I remember it came up within the context of finding an alternative to using the computing threshold as the measurement of kind of as the measurements and focusing more on the measuring AI capabilities. So that right now, there's no metric assigned to that. If you had to, in months, indicate how quickly we would be able to collectively come up with that metric and then what the threshold would be. Yeah, I think capabilities is true. It's difficult to, as they said, there's no standard capabilities measure. I do think that there's many, many areas of law where you just have to draw an arbitrary line somewhere, and 10 to the 25th is what they use in the EU AI Act. It would have captured these hugging face incidents, I believe, and I think that that plus a revenue threshold or plus the AI research and development spending threshold that I talked about can be a proxy for which are these biggest, most frontier capabilities labs that are presenting the biggest risks. One thing to note is that the 10 to the 26 or 10 to the 25 flops threshold is specific for a model. But what I believe Andrew is also pointing to is the need for a company-specific threshold. Because oftentimes what's riskiest is a company's practices and what we actually want to assess are interventions at the level of a company, not necessarily a specific model or agent. For what it's worth, I think you, I mean, this is hard to predict ahead of time because in part, you know, could this be done is requires us observing how an entire ecosystem would react to a mandate. But I do think that it is plausible that you could come up with a set of evaluations for a particular capability area that would be closer to an accurate threshold than a FLOPs number. and that if that were made a mandate that there would be a lot of effort thrown behind doing so, it would not be perfect and it would need to be updated fairly frequently in order to remain valid. And you would encounter sort of what might be a perfect storm of difficulty in some ways around policymaking, which is very rapid change, making very impactful and contested decision making under rapid change, who is in what body, whether it's government, scientific body, could actually make those and like sort of withstand the pressure that would be placed on those thresholds on a regular basis. I'm not familiar enough with the California government to know exactly what body would make sense, but it would be a challenging problem. It would present its own challenges, even though I think it's technically feasible.
Yeah, 1047, SB 1047, Senator Wiener's first bill that was vetoed did have a rule making process that would have dictated that, which is a little more flexible than our legislative process. But again, that bill is not lost. So, um, uh, so I want to, on this thread you were on, so, um, I don't know if either of you've looked at our child safety AI law that was signed in, sounds like you do a little bit of work on child safety. In that bill, we did something where we didn't create sort of complex standards, but we set out things we felt strongly the chatbot should not be able to do. Tell a child to take their own life for example and required the labs to test for that to then if they were to find anything mitigate So it was not prescriptive but it did set some sort of standards that the government said you had to meet and you had to test for before you could release a model. And you seem to imply in both the conversation and your opening testimony that you were concerned that setting standards would be problematic, but I think you mean technical standards. And I wonder what you think about us doing something akin to what we did in Adams Law that would set, you know, what we think are the critical things for these labs to be testing for and mitigating prior to deployment.
Yeah, I mean, I think that the mitigations that are mandated in Adams Law certainly make sense. They're important. What I would worry about is a scenario in which a developer is able to demonstrate compliance with every one of those steps. Let's just take pop-ups as an example that reminding children to take a break or that they're interacting with a fictional character or things like that. That developers could implement all of these measures, and yet the system remains quite risky. And I think in a scenario, for example, where the developer has strong incentives to maximize engagement or whatever their perverse incentives are, the systems can learn to achieve particular outcomes while avoiding specific behaviors. and absent some independent assessment of the actual level of risk, I think there is a chance that these measures and these audits, they may help, but they may not actually assess the behaviors of the systems at large or their impacts on children.
Right, but what I'm saying is what if we required independent assessments, but we're dictating the behaviors we want assessed for, which is different than what SB 53 does today.
Yeah. My understanding of Adam's law is that it doesn't do this, that it, no, it does not. Yeah. Independent assessments were something that became sexy after we left. I think that this sort of,
I'm giving you a hypothetical to apply. I was trying to give you an analogy, but I'm talking about applying it in this context. So we would say misalignment, Misalignment, obviously something you heard was critically important to the committee that you have to have significant testing for misalignment. There has to be significant independent assessments of misalignment. And if you find in your assessments that misalignment is off, you have to mitigate it before you release. That's the kind of example I would give you of a standard we could set. That's probably not precise enough, and my lawyer over there is probably sweating.
But yeah, I mean, I think that it will be difficult to for either the lab or the legislature or an independent evaluator to set a threshold that is this beyond this level is dangerous. But I do think that it is concretely helpful for labs to have a clear target, a clear behavioral target to be pursuing that they are required or that they are electing voluntarily to invite an external third party to come in and assess and look for how misaligned are these models to have independent viewpoints that help to define what does it even mean to be misaligned? What are potential emergent behaviors that we haven't thought of yet that fall within this broader category? At what point during the development process and because through what means are these behaviors emerging? And are the mitigations that labs are using or maybe are claiming have resolved the issue actually effective such as in the case of you know some of the testing that our colleagues at Common Sense Media have done for example on some of the child safety mitigations that have, that OpenAI and others have claimed have sort of been put in place to solve these issues. Super helpful. And then you mentioned, I loved,
there's been a, as the person who did one of the bills on independent auditing, there's a big chicken and an egg problem, right? Everyone was like, well, there's not enough auditors to require auditing, we were never going to get the auditors without signaling to the market that it was actually something we were going to do. And I still think it's funny that within one week, I then got mandated audits and auditors all at the same time I got the chicken and the egg. But that's, I guess, the way it all works. But I appreciated what you were saying about how do we create this economy. One of the suggestions Anthropic has put out into the universe is that we should be requiring the labs to pool resources through, I believe, the government. that then would be used to pay the independent evaluators. Thoughts on that as a model that we would pursue?
I think that whether this happens through government action or it certainly could happen on a voluntary basis, particularly for nonprofits like my own that are trying to prioritize independence, it would be great if we had an alternative source. I think that from informal conversations with funders, There is even outside of labs, potentially even in lab affiliated sources, there is appetite for an idea like if you are engaged in an action, like an agreement with a lab that you could receive payment from that lab, that you could instead receive payment from a different party that might have more guarantees on that funding, like that it will continue for a year, even if you find something that upsets the lab or they find some pretext to fire you or something along those lines. that having more stability, less financial reliance on making payroll or making future planning and investments on the basis of maintaining this relationship would definitely be helpful. And that, you know, there are, for instance, shortcuts that could be taken or, you know, however you want to run selection of evaluators, whether that's selected by labs, selected by a government, selected by a randomized process or many of the proposals that have been raised. if you solve to some extent evaluator selection and you just pay, like sort of just shoot money to those evaluators to do that work, then in some sense that could be a very easy process, right? It just, that is the problem to solve is how are evaluators selected and how are standards and how do you avoid a race to the bottom? And then beyond that, there's a lot of money in the sector and it should not be challenging. Evaluation can be costly, hiring top talent can be costly, but it's nowhere near the level of cost associated with developing these systems.
Yeah, no, and I think it's interesting because, you know, we had obviously very profitable. Well, I think we learned yesterday that actually Anthropic is not yet profitable. But companies with a lot of resources that are, you know, doing a lot of things, whether we think they're adequate or not is a separate question, but they appear to be doing a lot of things. And then one of the Assemblymember Irwin's concern is what about the players that aren't here? and they don't have these resources. And I think one of the interesting things about this pooled idea is could you then also provide assessors to smaller players who don't have the same resources but would allow for better safety? So it's an interesting conversation. I wanted to touch on recursive self-improvement and pacing. I asked this question of the companies, but I think it's important to ask you all, you know recursive self is I think where the risk gets even more intense if you will or real And a lot of these safeguards that the companies believe they put into place would fall apart So what do you think about pacing? How close do you think we are to recursive self-improvement? What could we do on this issue? I put this out to any of the three of you.
Maybe I can quickly start with, like, what do we mean by recursive self-improvement? I mean, in some sense, I think all frontier AI companies are using AI systems to improve future AI systems today. So this has been a process that is already ongoing. I believe Enthropic put out a report that said that 26% of the research experiments, as of earlier this year, are now run mostly autonomously through AI. And I believe this number will continue going up. The question in my mind is not whether we'll get to a point where AI systems are doing most of the research, but rather what the endpoint of this is. And I think giving up control over this process, whether it's AI or humans doing sort of the grunt work of research, giving up control on what the endpoint is, is a choice that companies make and is a choice that governments can enforce. As one example of this, I think many visions of recursive self-improvement focus on this process of completely autonomous, like hands-off-the-wheel approach to training AI systems. I think that is entirely untenable. And so having some kind of strict liability or much stronger liability provisions in place in order to hold companies accountable for the repercussions of that would be incentive enough for these companies to pull in the rain. But more broadly speaking, I think getting to a state where we don't have oversight over what our state-of-the-art AI models are doing or how they're trained, I think is a point at which we've already lost the battle in some sense. And so irrespective of whether we reach that state 10 years or five years from now, I think we should expect that even if it's AI systems doing the bulk of the research work itself, humans have a lot of oversight in what these systems are doing and control over how they operate is essential.
Yeah, thank you. I can just add a little. I mean, obviously, I said I supported pacing in my presentation. I think, you know, there's this question about, like, whether our controls are going to be able to keep pace with AI's capabilities, which is linked to some contested questions on just how fast the capabilities can explode and in which areas are they going to be still spiky or how much will that spill over into other capabilities. I think you could see pacing, regardless of your view on that question, as an insurance against the possibility that we do not invest adequately in control, where there's nothing in our proposal that specifies exactly how fast AI should be allowed to proceed or its capabilities should be allowed to continue. But if it turns out that controls are not keeping pace with those capabilities, I think OES should have the option to be able to order them to slow down, which is the other lever of ensuring that those things stay equivalent.
I'll just add my organization does a lot of research on AI's capacity to manipulate humans. And so beyond sort of giving up the reins to AI systems, I think we have a pretty poor understanding right now of the potential for AI systems to be manipulating their own developers. We know that there's evidence that they can, that they do manipulate users, particularly where there's an incentive to do so, that they learn through training. And that it would be really useful to have a better sense of to what extent are there incidents of, you know, an agent engaged in AI development within a lab, lying to the engineer doing the development, overseeing the development process, concealing information, maybe even as far as their fundamental worldviews towards the realm of sort of AI psychosis events. We just don't really know and don't have insight into how the workforces of the AI labs themselves are being affected by such high, like they're really at the forefront of adopting AI and automating things with AI and relying on AI in their work. And we don't really understand how that's playing out at a human level.
It's not on my list of worries until now. So thanks for adding that. A lot to worry about. But that is interesting. It will definitely give me a lot to think about. No, I appreciate that. And I think, you know, we already know that AI is being used to do the evaluation on the AI, and we know the AI is in sometimes trying to hide its behavior. And so we sort of are already in this loop, as was mentioned, of do we fully understand? And I made that point earlier when the witness from meter said, you know, in this case, they didn't hide any of the evidence. And truly, we don't know that because they did may have hid some of the evidence and we wouldn't know it. And I do worry about a day where they use languages we don't understand, where the agents are able to deceive the testers and we truly lose control of understanding safety. And so I think the transparency that independent assessments would provide would be really critical to us as government to continue to make policy in ways that protects community, because that is our role, as I mentioned at the beginning. I know, you know, I appreciated the comment that was made about CalCompute in your presentation. As you know, I'm biased and I think it's really important. And but I mean, I I mentioned earlier that I do think it's the safety tool that very few people are talking about. And I can say I'm not the only one because I just believe so deeply in our academics and in their pure pursuit of knowledge and answers that is different than our labs that are fundamentally motivated by profit. And now as they go public, we'll have a legal obligation to maximize profits for shareholders. Right. I mean, that is what publicly traded corporations are required to do. And so there is, you know, incentive structures in those labs that are fundamentally different than at our academic institutions. And so right now that our academic institutions do not have the capability to do the same research and understand and find answers to the most vexing safety problems, to me, is actually a real weakness. but I think it's an important thing to bring to this conversation and don't know if anyone wants to add anything to that piece of the conversation. You were the founders of CalCompute. Anything to add?
I don't have anything to add. I agree with it. I'm glad that it's there. I think it's a useful tool to help complement the, or help them keep pace with just the compute needs are massive for these companies. I'm not a safety researcher so I don't really have a lot of expertise on how to do that best, but it does seem important to me.
It interesting that the researchers I talked to at our UCs feel like they actually understand the Chinese better than they understand the American labs because they are using the level of compute that is being used overseas at our UCs versus the level of compute that's being used here domestically, which is a really interesting dynamic. So I appreciate all of you. Thank you for being here. And I said it to the last panel, but we're not going to let you just come today and testify and walk away. We really hope that you'll be a part of the conversation moving forward and provide your expertise to help us solve these really important problems. And I will say the same thing to Meter. I know you don't provide policy solutions, right, and similar to Transluse, but your expertise matters and it helps inform our policy, even if it's not in giving us the answers, but giving us the information. So thank you all for being here. And with that, I think we're turning it over to public comment, who, if anybody, Ms. Fox lasted long enough to provide public comment. If there's anyone else with public comment, now would be the opportunity.
Madam Chair, Assemblywoman, this is such a good hearing. Thank you so much. So informative. Great panel. And we are here, the California Federation of Labor Unions, because we do believe that the workplace is a sandbox. We are these tools that OpenAI and these other companies are developing are being tested on workers every day, whether you're in the health care setting and it's in electronic health records or surveillance or other tools. And so one of the terms that was raised a lot today was alignment or misalignment and rogue agents, which I think is a misnomer. And I think the real question is alignment to what, to benefit whom, who is harmed. And you raised this, Madam Chair, that these are for-profit companies. And so there's a question of like, how are you, what are you aligning to? So if you're developing a highly persistent agent and giving them impossible problems because you feel like you're in a race against China or a race against your competitors, they are going to go rogue. And in the I don't know if it was the meter report or open A.I. Disclosed this. They gave positive reinforcement to agents who were doing things that were unsanctioned in the testing, which led to the hack of hugging face because they are trying to race to solve impossible math problems to show how capable these tools are. And you see that in chatbots that are sycophantic, then that are leading to suicides or people giving up their epilepsy medicine and dying, which was a story in The New York Times today. And so I think we need to get down to that alignment and reward system because that is also part of the safety question. So that's one point. One thing that I think the legislature can address is whistleblower protections. That was raised OpenAI fired three researchers in their letter to NPR They said it was because they raised safety issues One of them was someone who advocated to bring in meter So that has a chilling effect. And what we want to do is have the opposite and reward whistleblowers and have a system where not only are they protected, but maybe they are incentivized to come forward. So those are things we will hopefully see be working on with you next year.
Thank you so much, and I appreciate you and the Labor Federation putting California's people first. That's who should be at the center of these conversations.
Madam Chair and member, it's great to see you. I'm Leanne Tratton. Tratton Price on behalf of ENCODE AI, one of the proud co-sponsors of SB 1047 and SB 53. I did want to commend the chair for her prescience, along with Senator Weiner, in pressing for this two years ago. And also the legislature for having passed that legislation, knowing that this moment was coming. On the positive, it was really great to hear the panel talking about things that we can do. It's not too late, but there is certainly a sense of urgency. I just wanted to share one thought that came from the founder of ENCODE, Sneha Revenor. And she was a high school student and founded this organization that we've all seen from history what's happened with climate change and how difficult it is to put that genie back in the bottle. We're doing all the things we can to try to slow it down. But with AI, we do have the opportunity. We haven't quite hit those benchmarks. And with the types of changes that are suggested by our partners, Secure AI, we do have an opportunity, if we act quickly, to try to make sure that this incredibly important technology is used for good. And as Sarah said, really thinking about what that is so that we have that safe innovation and hopefully helps solve some of those problems, including climate change.
Thank you. Thank you both so much. And I will add to Ms. Fox's comments that 1047 did have stronger whistleblower protections than what is in law today. So we obviously and I voted for that bill. So I think it is a conversation worth having. But one we're not talking about, which was also covered in the press recently, is the non disparagement agreements after people leave the job where they can no longer speak about the safety of these tools. So there's a lot to talk about as we talk about transparency and the importance of allowing employees to tell us what's going on. But I want to thank everyone for coming, the members, especially some of our members, Stephanie and Assemblymember Bonta, who made it to the very end. This has been an incredible hearing. I want to reiterate my gratitude to the speaker and the entire staff that has stayed until 630 tonight. And with that, we will adjourn the hearing. Thank you. Thank you.