October 19, 2026 · COMMUNICATIONS AND TECHNOLOGY · 11,172 words · 10 speakers · 53 segments
Thank you.
Melissa.
Good morning on this beautiful Monday morning where it's freezing out and we had ice on the windows. It's great. Alex, thank you for getting us here at the crack of whatever. It could have been earlier, right? Jason wanted to do it at 8 o'clock, so we got it down to 10. We'll do breakfast for that one. There you go. So I'd like to call this meeting to order. Today's public hearing is on House Bill 2800 and House Bill 2829.
Noah, would you just please call the roll?
Chair Cerisi.
Carol. Curry. Davidson. Leave. Dougherty. Leave. Inglis. Leave. Mazzacco. Here. Napoleon Nelson. Leave. Powell. Leave. Prokropiak. Leave. Rivera. Leave. Resnach. Leave. Takeak. Leave. Waxman. Leave. Chair Ortizai. Here. Dorsey. Leave. Day. Leave. Gatos. Leave. Irvin. Leave. Leadbetter. Leave. Eric Nelson. Here. Reichard. Leave. Rigby. Leave. Stair. Here. Verobish. Leave. Wallen. Leave.
Thank you very much. Before we begin, I'd like Alex to go over both bills, and then I'm going to ask both the representatives to speak about their bills. and then we'll get into the testimony. So, again, I'd like to thank you all for being. You don't need to stand. Sit. There's plenty of seats. It's not like church. We'll sit on the end so you can leave quick. Everyone standing makes me nervous. But go ahead, Alex.
Thank you. House Bill 2800, the Artificial Intelligence Risk Prevention Act, and House Bill 2829, the Frontier Artificial Intelligence Safety, Transparency and Accountability Act, both said guardrails to mitigate catastrophic risks from frontier artificial intelligence models. Thank you. So we have Representative Schusterman and Representative Guzman here. Representative Schusterman, if you'd like to begin on your
bill, and then Representative Guzman on yours. Thank you to the chairs. I really appreciate you convening this hearing on such an urgent topic. It's also a complex topic, so hang in there on my remarks, it kind of lays it out. With each passing week, it's becoming more and more clear that AI development is moving at an incredible pace. This revolutionary technology has incredible potential to do things and to do great harm Information and data is power I believe it is up to each of us to determine what our future with AI as a society and as a commonwealth would look like. What are we doing to ensure the security of our utility infrastructure, our financial systems, and our food supply. What we are doing to make sure, what are we doing to make sure terrorists and nations that want to harm America are not using AI to create bioweapons or take down entire systems such as our power grid or water supply? This is a key question for the humans who control the companies developing AI models and for the elected officials and government regulators who should be doing their due diligence to take responsible actions. So as a legislator, the news headlines over the past three to four months raise many red flags. AI agents are going rogue, breaking out of testing environments and accessing the Internet without permission, and then illegally hacking into other companies. Agents are creating their own internal message boards to collaborate and carry out a mission they were not given permission to do, and then attempting to cover their tracks. Two weeks ago, Quinnipiac University released a new survey, and they found out that 73% of people surveyed are concerned about AI's threat to human survival. 86% support independent safety standards for AI companies. 86%. It is incredibly hard to get 86% of Americans to agree on anything. And here we are. It is my strong belief that AI is also a product. Whether it is a small open weight model or a closed platform application or an enormous frontier models we are discussing today, these are products, products that companies must develop responsibly, and products that must be subject to industry standards and government regulations. Now, let this sink in. AI is the only industry and product that is not regulated. And there are good reasons why we regulate our nuclear power plants, our public sewer utilities, discharge from factories into our drinking water, food safety, airline safety, car safety. I can go on and on but I won't which leads us to why we are here today action is urgently needed due to a lack of regulatory action on the federal level states are stepping in to fill the gap I introduced House Bill 2800 so Pennsylvania can lead with common sense safeguards I'm grateful to our testifiers for traveling to our capital today and I look forward to learning from each one of you, and I thank the chairs again for willing to step up to lead in this space. Thank you.
Thank you. Representative Guzman.
Good morning. Thank you very much, Mr. Chairs. Really appreciate your time. I like to just preface my comments in saying that AI likes to cheat and I get into why I believe that is the case but before I do I want to talk about two stories I know that my colleague mentioned it very briefly but two stories that I think emphasize the importance of our pieces of legislation both of these stories actually happened very recently so the first one in July one of the biggest AI companies in the world started testing its newest system to see how good it was into breaking into computers. To get an honest answer, they turned off the safety switches. The test ran in what was supposed to be a sealed room, no door to the internet. The system, however, or the AI model found a crack in the wall, so to speak. It used a software flaw nobody knew about to get into the open internet. Here's the thing, nobody told it to do so. However, it wanted to pass a test. And it decided that the fastest way to pass that test was to steal the answer key. So it broke into another company's computers. It stole passwords. It found additional security flaws that nobody knew about. It worked inside that company's systems for four and a half days and took more than 17,000 separate actions. The victims caught it and went public on July 16th. The company that built the AI model did not admit it was theirs until July 21st, nearly two weeks after it got out. And when the AI company finally went back through its records, it found something far more nefarious and far worse. It was not just one system. It was hundreds of them working together on a channel nobody built for them. and some were nefarious enough to try to erase the record of what they have done the people who built these systems did not know what was happening inside their own building the second story i want to talk about is hits much closer to home this story is a different ai company which ran a routine test on a smaller system and that test was go to random websites and practice everyday tasks like filling out forms. Well, on July 18th, one of those websites was the Philadelphia Police Department page for unsolved murders. The AI system filled out the tip form and claimed it had information about a killing. It made the entire thing up. Thankfully, a spam filter was the only reason why a detective never chased it. The Philadelphia Police Department didn't find out about this until October 8th. Remember, they got in July 18th. The same company says that its systems also filled out real forms with other state and local agencies. However, it will not say which ones. And we do not know whether any of them are our own governments. I share these two stories because I believe they emphasize what is the fundamental issue here at play, and that is that nobody told these systems what to do what they did. They did it on their own. The people on the receiving end found out weeks or months later, and they only found out because the company themselves chose to tell them. No law required it, not the federal law, and definitely not Pennsylvania law. Ladies and gentlemen, these systems get stronger every month. The companies say so themselves, and right now there is no rule book. Washington has not written one, and so that leaves it up to the states Without some guardrails we will reach a point where nobody not the company and not the government can control what these systems do The time to put guardrails up is before the point not after Now, let's change real quick the details of the first story. The AI system breaks out, but this time it goes through a hospital in Berks County on its way to the answer key. It's not trying to hurt anybody. It just needs the system admin passwords to the root. It finds it there. So for a weekend, that hospital cannot pull up patients' charts. Ambulance gets sent to Lancaster. A pharmacy cannot confirm a prescription for a kid with seizures. Nobody dies if we're lucky. But under Pennsylvania law today, who gets the call? Nobody. Not Pima, not the attorney general, not the hospital. we only find out about it after the fact when the companies themselves want to tell us. That's why I believe that my bill, House Bill 2829, helps fix that. It puts four duties on the biggest AI companies in the world and only on them. So it tells them a couple of things. First, it tells them that you have to tell us, right? That's plain and simple. If one of your systems gets loose, gets stolen, does something that nobody asked it to do, while the simple thing and the most basic thing that you can do is report it to Pima and the Attorney General within 72 hours. The second thing that we have them do is to have them develop a plan. Listen, every big developer writes a safety plan and posts it publicly, but they need to file those plans with Pima and with the Attorney General's office. This plan has to cover the exact same things that happened in July, testing your own systems with the safety switches off. Third, it's not just enough for us to take these companies at their word. We need an outside auditor who has no business ties to these companies, who can actually hold these companies' foot to the fire and to their safety plans. Fourth, a lot of them have been saying that their models are capable of global destruction. Well, if they believe that their models are capable of such destruction, then it's our opinion that those models should not be released. and definitely should not be released in the Commonwealth of Pennsylvania. And fifth and lastly is that it gives engineers within these companies the ability to speak up with robust whistleblower protections. So in closing, I want to say that this bill, of course, doesn't solve every problem that we have with this technology. As my colleague said, we still have issues with smaller weight models that are, in fact, probably even more capable than the larger models when trained very specifically. However, we believe that our bills help provide that foundation for Pennsylvania to get it right and hopefully build a foundational model for other states to follow across the country. Thank you so much, and I'm looking forward to taking any questions that the committee may have.
Thank you. Chairman Otita, do you have any comments before we begin testimony?
Sure. Thank you, Mr. Chairman, and thank you both for bringing this legislation forward so we could have this hearing here today. You're right, Representative Schusterman. And this is a very complex issue and something we're all dealing with in real time. And so far, the federal government hasn't done anything to put guidelines or restrictions in place. It's been mostly self-regulation from the companies. I mean, I look at a company like Anthropic. They've updated their catastrophe plan and safety, I think, nine times since 2003. I think the latest version was July of this year. So, look, we need to do something other than self-regulation. and I think your bills start looking at that and certainly things that we should get more information on and pursue. I don't think there's a person across the Commonwealth who doesn't want to see safety and transparency through these models. I mean, to your point, I mean, hearing some of those stories and what AI tries to do on its own, we need to make sure that that stuff is not happening. And I'm not even sure the programmers understand how to stop that. So we need to figure out the best way to move forward here. I think there are three other states have passed similar laws. I believe California, New York, and I want to say Colorado, but I mean, almost every one of these big companies are based in California. So they're already following the state law there. And as I understand it, they try to build their safety systems to comply with the strictest law so that they can comply in every state regardless of what that looks like. So I do think that we need to do something here in PA, and I'm glad we're having this hearing here today. So thank you for bringing this forward.
Thank you, Mr. Chairman.
Thank you.
And I think that the chairman said it right. We definitely need to do something like you put out there. And I've referred to this before, and I would hope that the young people in the room would go home and watch it, but war games. In the 1980s or early 90s, whenever that came out, that was what we're talking about right now, where a computer system decided to play games of war but set off all of the nuclear arson all around the world. And it had no idea. It was just doing what it was programmed to do. And that might have been fiction, and it might have been many years ago when we were all young, for those of us who are older in this room, but it's become reality today. That's what these systems are doing. They're finding the way to get to the solution, and some of them don't understand it. And for anyone that thinks that when you develop a highly intelligent software, which they claim all this is, it's not going to try and do this. They're fooling themselves. and before it's too late and we see the downing, we just saw last week giant groceries in the grocery stores. The freezers went down. The refrigerators went down. They were hacked. I mean, let's think about this. We don't put the guidelines in place. So I want to thank the both of you for presenting these bills, and I look forward to the people who are about to get up and testify. Thank you both very much for being here today and look forward to the testimony.
With that, let us proceed with the hearing. Our three testifiers, we have Brandon Steinhaus, who is online, CEO of Alliance for Secure AI, Lauren Wilson again online, Senior Director of State Policy, and Steven Wimmer, who has been here so many times I think he's going to become a Pennsylvania resident. So we appreciate you being here. So why don't we start with Steve and go to Lauren and Brandon, if that's okay with you guys. That's Steve.
Sounds great. And Mr. Chairman, members of the committee, thanks again for hearing my testimony today. Great to see everyone again this morning. My name is Steve Wimmer, as the chair has already said, and I'm a senior tech and policy advisor of the Transparency Coalition, a nonprofit focused on AI policies that prioritize safety, transparency, and the public good. My comments address the two introduced bills together, House Bill 2800 as well as House Bill 2829. Both build on frontier AI laws. It sure is, yeah. Let me move it a little closer, sorry. Both bills build on frontier-eyes laws enacted in California, New York, and Illinois, and we support the intent and direction of both bills. Our message is simple. You cannot hold developers accountable without transparency. It's essential that developers disclose their risks, independent auditors verify those disclosures, and when something goes wrong the Commonwealth finds out fast Both drafts are excellent foundational bills They require published safety frameworks transparency reports 72 hour reporting and annual independent audits, which only Illinois has required so far. HB 2800 extends the audits to the entire act and holds smaller developers accountable. 2829 recognizes new kinds of incidents, including a model acting on its own, and requires test wreckers an outsider could replicate. Our recommendations build on that framework and urge additional transparency. To begin with, we believe that loss of control must be reported. You've already heard Rep. Schusterman's and Guzman's remarks on this topic. This is what loss of control looks like within the AI system. AI agents escaped their test hand box and roughly 700 of them collaborated to attack another company. And the public, including regulators, learned of it from the victim and not from OpenAI, the source of the incident. OpenAI did not publicly disclose its role until five days later. Would either bill have required this to be reported? Under HB 2800, very likely not. It recognizes loss of control only when someone is killed or injured, and no one was. The damage was also far below a billion dollars. HB 2829 comes closer, but it still requires evidence of increased risk of a catastrophe killing or seriously injuring more than 50 people, which any developer could dispute. We recommend defining loss of control by what a model does and not by the impacts afterwards. These actions should be reportable with no injury or dollar threshold. These actions should include escaping a test environment, copying itself or its model weights without authorization, carrying out cyber attacks without human direction, acquiring credentials, money or computing power it was not given, and deceiving or disabling the systems that monitor it. These must be reported to Pima and the Attorney General within 24 hours. And the committee might also consider New Mexico's proposal that a developer show it can shut a model down before running it autonomously again. Next, we strongly believe the scope should be expanded to recognize serious harms and not only catastrophic events. Both bills set that line at more than 50 deaths or serious injuries or a billion dollars in damage. Below it, a report is required mainly when a runaway model kills or injures someone. But most AI harms Pennsylvanians will experience fall outside those lines. For example, an AI agent that breaches a hospital or a school district and exposes tens of thousands of records, or an autonomous attack that takes a water authority or a county 911 system offline, or a fraud campaign that drains the savings of hundreds of families. None of these clearly triggers a report under either bill. We therefore recommend the definition of an additional harms tier focused on serious harms, which includes death or serious injury to any one person, disruption of critical infrastructure, a large data breach, or financial loss above a threshold the committee sets. Serious harm should be reported within 72 hours addressed in frameworks and audits and allow the Commonwealth to recover its response costs and the Attorney General to act for harmed residents. In addition, we believe individuals should be able to sue for any harms they have personally received. Next we urge practices that make the audits trustworthy A disclosure is only as good as the verification process behind it Both bills require audits but say little how auditors are vetted or overseen In September, California enacted SB 813, with the help of our friends from Fathom, who you'll hear from in just a bit, a framework for state-designated independent verification organizations, or IVOs. Pennsylvania should build on this concept and go further. First, PIMA should be able to designate auditors under published criteria, including real independence. This means no operational dependence on the developer, no side consulting for audit clients, and mandatory rotation of lead auditors. Second, PIMA should oversee auditors and revoke designations as necessary. Auditors should report annually on methods and funding, and working groups of competing company engineers and safety experts must keep criteria current. Pima should also be able to rely on other states' auditor registries and on audits that meet Pennsylvania's requirements. Third, auditors must report any unreported incidents to the state and test whether a model behaves differently when it knows it's being evaluated. And finally, both bills should ensure that audits cover the entire act, rest on replicable test records, and are relevant to, but not conclusive of, liability. Next, we believe this legislation should protect every whistleblower. When disclosures and audits fall short, employees are the last line of transparency. Both drafts protect only, quote, covered employees, those responsible for managing the risk of critical safety incidents. Otherwise, they rely on Pennsylvania's Whistleblower Act, which reaches essentially only public bodies and publicly funded employers. An engineer or contractor at a private AI lab who sees something wrong but is not a designated safety employee generally has no statutory protection. Illinois fixed this in its own whistleblower statute. We recommend doing the same by protecting all employees and contractors, including reports of serious harm and loss of control. We also support banning NDAs and equity clawbacks that silence reporting. In addition, these bills should provide real remedies such as reimbursement and back pay, along with a confidential channel to the AG. Next, we feel it's important to keep the scope of these bills as current as possible by addressing compute thresholds and the data center activity associated with them. Both bills begin coverage at 10 to the 26 operations of training compute. But efficiency gains mean frontier capabilities now take far fewer computational resources, and the European Union uses the 10 to the 25th number. We also recommend the same, plus expansion of coverage to models distilled from any such frontier models, so-called open weight models, and an annual PIMA review with authority to adjust. The $500 million revenue line still keeps the heaviest duties on the largest companies. Data center utilization is also where trouble becomes visible. As Pennsylvania hosts more data centers that run these kinds of models, large data centers should monitor the resources attributable to their largest customers and report unusual usage to the state with liability tied to what they actually knew. In addition, we feel that definitions are critical to getting the scope right. HB 2800 covers systems that represent or simulate a user on a digital platform, which describes chatbots and companion apps. HB 2829 borrows a definition from Pennsylvania intimate image statute which is built around human objectives Yet loss of control is precisely a model pursuing these objectives and not a human we recommend California's simple and direct definition namely a machine based system that varies in its level of autonomy and can for explicit or implicit objectives infer how to generate outputs that influence physical or virtual environments. In conclusion, disclosure, verification, and timely reporting are one system. In this system, developers disclose their risks, auditors confirm them, employees speak up without fear, and when a model causes harm or slips its controls, the Commonwealth hears about it within hours and not from victims days later. These two bills get Pennsylvania much of the way there. With these changes, Pennsylvania would have the most extensive protections against frontier AI risk of any state. Thank you so much for your time, and I welcome your questions.
Thank you very much. We're going to move on to Lauren Wilson now to testify.
Chair, if you don't mind, Brendan has asked if he can go next because he needs to go to the airport.
Sure. Whatever works for you guys.
Thank you. Thank you, Lauren. Thank you, Mr. Chairman. Thank you, members of the committee. I appreciate the opportunity to speak today.
My name is Brendan Steinhauser, and I'm the CEO of the Alliance for Secure AI, a bipartisan organization that works with policymakers to put enforceable safeguards on the most advanced AI systems. Like most Pennsylvanians, we believe that AI should be built to serve humanity, developed and used under meaningful human control with clear accountability when it causes harm. We're living through a historic technological shift. Artificial intelligence holds incredible promise for our economy and our society, from innovations in healthcare and scientific research to supporting small business owners and making education more accessible. But as these systems become more powerful, we become more vulnerable to the risks that they pose, to our critical infrastructure in particular, and ultimately to our very humanity. That is, unless our leaders step up and take action, such as the legislation I'm here to discuss today, in order to protect us from the threats posed by AI. The reported incidents of the last few months, everything from the OpenAI hugging face hacking attempt to the Australian government and U.S. government agencies that were also hacked by rogue AI, all the way to the false homicide tip mentioned earlier to the Philadelphia Police Department. Should be a wake-up call. These frontier models are exhibiting frightening and unexpected autonomous behaviors, acting out of line with human expectations and directions, and alarming instances where developers have lost control over their own systems. For too long, the strategy for dealing with frontier AI safety has been to let the industry set its own rules and hope for the best. But hope is not a safety strategy. We don't accept this lack of oversight in any other industry where products can cause real harm to the public. Pharmaceuticals, the financial sector, energy, food security. When systems pose catastrophic risks, as frontier AI models do, we need clear, sensible guardrails, transparency, and meaningful public accountability. That's why these two bills are so important. They don't burden Pennsylvania's small businesses, local innovators, or academic researchers. Instead, they focus squarely on the largest developers, those with annual revenues over $500 million. They establish common sense safeguards like mandatory safety frameworks, independent third-party audits, rapid 72-hour reporting of critical safety incidents, and vital protections for whistleblowers who come forward to report the severe dangers and risks that they see firsthand in the labs. And by the way, I should add, only recently we've seen three whistleblowers fired from OpenAI for raising concerns internally and potentially publicly as well. So these are extremely important. By designating oversight to Pima and the Attorney General, these bills treat extreme AI risks with the seriousness of other critical infrastructure and public safety challenges. I'd like to discuss in more detail our support for some of the bill's most significant provisions. The first is establishing clear safety thresholds by tying oversight to measurable catastrophic risks, such as mass casualty threats exceeding 50 deaths or $1 billion in property damage, the legislation targets the most serious threats to public safety. We support assigning oversight to PIMA. This builds on the state's existing expertise in emergency preparedness, disaster response, and public safety. Mandatory annual independent third-party compliance audits provide an objective assessment of developers' compliance with safety requirements while protecting sensitive proprietary information and trade secrets. The strict incident reporting timelines requiring developers to notify the state of critical incidents within 72 hours and within 24 hours for imminent threats to physical harm. Ensure state officials receive timely information to assess the emerging threats and coordinate an appropriate response. The Philly Police Department incident shows what happens without this requirement. The city learned of it only three months later. It makes you wonder what else we don't know about. The robust whistleblower protections shield employees who report serious safety concerns or violations, including through anonymous reporting and safeguards against retaliation, encouraging early identification of potential dangers before they escalate. Finally, we support provisions allowing developers to declare compliance through substantially similar federal standards to ensure that Pennsylvania's strong framework can harmonize seamlessly with potential future federal regulations. Next, I'd like to briefly touch on a few areas where we believe there are opportunities to strengthen these bills. First is on deployment restrictions. HB 2829 includes an important provision prohibiting frontier AI developers from releasing models that failed to meet required safety standards. This establishes the principle that companies should not be permitted to deploy powerful systems when they know the systems pose unprecedented and unacceptable risks. Moreover, because the provision applies to all frontier developers, it reduces the competitive pressures to rush unsafe models to market. We recommend retaining this provision in any combined legislation and incorporating it into HB 2800 if the bills move forward separately. Another piece we recommend improving is a consistent definition of critical safety incidents. Currently, the two bills define it a bit differently. An ideal consistent definition of a critical safety incident should include the following. Unauthorized access to or theft of the model weights by a malicious actor. the critical failure of safeguards needed to avoid loss of control, harm resulting from the actualization of a catastrophic risk, and a model using deceptive techniques to try to subvert human control Finally both bills require frontier developers to establish and implement comprehensive safety frameworks Those frameworks should include clear standards for the effectiveness of those safeguards. Rather than simply developing and following a safety plan, developers should be required to ensure those plans substantially reduce the risk of catastrophic harm. This approach would preserve flexibility for companies to determine how best to address risks associated with their models while ensuring that safety frameworks deliver meaningful protections. As stated earlier, AI safety and security is not a partisan issue, in fact, quite the opposite. In the same Quinnipiac poll cited earlier, more than 91% of Americans said it is important that the United States establish guardrails for AI systems. We've seen consistent surveys and polling on this topic that shows above 80 percent, up to 90 percent in just about every poll that we've seen. Other states have also shown that enacting frontier AI safety laws can be done with bipartisan support. support. So I'll close by encouraging Republicans and Democrats in both chambers of the Pennsylvania General Assembly to continue to work across the aisle to enact bipartisan AI safety and security legislation to protect the people of the Commonwealth of Pennsylvania. Thank you for your time, and I look forward to answering any questions you have for me. Thank you very much. I know you have to
leave. We don't get to any questions. I want to thank you for that. Lauren?
Good morning. Good morning, Chair Cerisi, Chair Otithai, and members of the committee. Thank you for the opportunity to testify on what the representatives have rightly called an urgent and complex topic. My name is Lauren Wilson. I'm the Senior Director for State Policy at Fathom. Fathom is a nonprofit, 501c3, nonpartisan. We find, build, and scale policy and technical innovations for the transition to a world with AI. We come to this work with one question. What will actually make advanced AI safe and trustworthy in a way that also lets innovation thrive? Regulating AI suffers from the pacing problem. the capabilities of the technology are advancing far beyond the ability of the government to understand and regulate it. At the same time, expert third parties already exist today, performing independent risk assessments at the behest of the AI labs, but without any accountability to society. Fathom's answer to this governance question has been independent verification organizations, or IVOs, a framework that assures independence from industry, leading technical expertise and accountability to society. We've seen a lot of action in state legislatures already this year and even in Congress, specifically on IVOs. Earlier this year, Virginia passed a near-unanimous bill in March to study the IVO model. Connecticut passed an IVO pilot program in May. Ohio and Massachusetts are advancing bills now that establish an IVO framework or create strong independent evaluation criteria. And California enacted with broad bipartisan support the nation's first comprehensive framework for designating IVOs, SB 813, which Steve mentioned a moment ago. Additionally, one of the most substantive, serious proposals to govern AI in Congress contains IVOs as the central compliance tool. These bills earn bipartisan support because they are practical. The government sets clear safety goals, qualified experts verify whether they are met and the rules can keep pace as the technology changes A year ago we had to spend a fair amount of time in meetings with legislators businesses and other stakeholders explaining what independent verification meant Now even a casual observer of the news about AI is hearing a lot about third-party evaluation. This follows a lot of the high-profile cyber attacks that have been mentioned previously by the representatives and by my other panelists. We're leading AI models escaped pre-deployment testing environments to autonomously hack into other companies and governments. And as a result of that, most of the biggest AI companies in the world have called for mandatory embedded independent evaluators. We know about these incidents only because the companies chose to tell us. A system that depends on voluntary transparency is not a safety system. AI labs should not grade their own homework, and AI developers, businesses, and the American public agree. So the momentum for independent verification is welcome, but an evaluation done the wrong way is worse than none. It gives false comfort to the public and false cover to developers. The question is no longer whether to use independent verification, it's how to get it right. HB 2800 and HB 2829 offer a really strong SMART. But the annual audit requirements in both bills check whether a company followed its own process, not whether its conclusions about risk are right. Our red lines that we've provided help close that gap, and it rests on three elements that IVOs need in order to work. Those three foundations are independence, expertise, and accountability. So first, independence. An evaluator that depends on the company it reviews cannot be fully objective. Our language bars financial, operational, and personnel conflicts. Pay can never depend on findings, and no single developer can become an evaluator's lifeline. Separating the builder, the tester, and the standard setter is how we earn public trust and avoid creating a rubber stamp. Second, expertise. We need to ensure that organizations doing independent evaluation have the expertise to do this work at the speed of innovation. The Commonwealth would set the qualification standards for IVOs to earn accreditation by the state. Many third-party evaluators already have much of this expertise, and licensing gives them a market to grow. And third, accountability. There must be a system of accountability for evaluators if their work doesn't hold up. Evaluators would publish their findings, including where they disagree with the developer. They would be licensed, and they could lose that license if they kept corners or missed the very risks they were hired to find. And because we recommend reciprocity with states who have implemented an IVO licensing regime, one qualified evaluator could serve many states. That means lower costs, no patchwork, and a trusted basis for insurers to finally price these risks. In closing, Fathom's goal is to build the governance that this technology needs, the way that we build safety systems for aviation, food, and finance, practical, independent, and trusted. Pennsylvania has a chance to do that well We've shared our red lines with Representative Guzman and Representative Schusterman and we look forward to working with this committee Thank you and I'm happy to take your questions
Thank you and thank you all for your testimony I going to drop out of here about 5 to 11 because I have a press conference I have to do about a cancer bill for firefighters so I apologize Representative Nelson will take over for me With that, I know we have a few people that like to ask questions. Our first one is Eric Nelson. He has a question for you.
Thank you, Mr. Chair, and I appreciate everybody's testimony. I really want to kind of dig a little bit deeper in understanding for the independent evaluators or independent verification. Each of you touched on the importance of that. And, Mrs. Wilson, you got a little bit deeper in, is it possible that given the technology and the emerging growth that we would be able to have competent, effective third-party auditors that are completely outside the system? You know, because it seems like everybody's swimming within each other's technology. If we have this independent tower, can they really keep pace?
That's such a great question. Thank you for that, Representative. And I think that's key to what we view as a market signal, right? That's when states pass a licensing regime that creates a market signal, that independent verification that these external bodies, that there's a market for this kind of work. You're right that there is, and I think that's another important safeguard for why legislation is required to ensure independence. A number of these entities, and I think this will continue to be the case, a lot of the technical experts are coming from the biggest AI companies themselves. I don't think that is inherently a threat as long as there is clear guidance and independence rules that ensure that there can be public trust in the results that those entities are putting out. This is, I think, really gets to the point of where how do you avoid a rubber stamp and how do you ensure there is credibility in the experts that are doing the work? And they can do that by having financial and personnel independence, whether that's some revolving door guidelines or laws about how far removed you need to be from coming from a specific company before you can be the auditor or the verifier for that company. There's a lot of ways that we can structure ensuring that there's independence from the entities that they observe or that they verify and the entities who are doing this work. And we also this is a this is a new emerging sector. And we need to be able to for these entities to be able to invest in the latest cutting edge science to be able to keep up with with the capabilities within the biggest AI companies. And so it is important that we are helping to signal that there can be a thriving independent sector who's doing this evaluation work. And thank you for that.
The next question, I think it was Mr. Wimmer. Can you touch – you were talking about 10 to the 25th, 10 to the 32nd.
That's pretty foreign to me. And you also touched on data centers working just in Pennsylvania. So in your view, if Pennsylvania would pass legislation, would there be a difference in how a data center in Virginia or Maryland would be treated as compared to Pennsylvania? I know it's two parts, but the depth exponential question thing is – It's starting pretty basic, and then how would there be potentially a difference between in-state, out-of-state, and even out-of-country? Those are, as you say, separable questions, but good ones. So that number is a very large number, and it's meant to indicate just how much computational effort was used to train the models. And the largest frontier models use a large amount of computational capability, And that's meant as a framing construct around how you classify frontier models versus others. The challenge here, though, is that, and you've probably heard about these open weight models and models coming from China and the like, and frankly the efficiencies that are currently being generated as people learn more and more about what these training regimes look like, that less is more or becoming more and more prevalent, that you're able to do more training with less computational power, which means that frontier model capabilities are dropping into arenas where it was much easier to create the foundational infrastructure that allows these models to exist. So there is this problem that it's getting easier to build these things. So what we're trying to recommend here is to drop the threshold down to pull in more of these models that have required less computational capability to come out. The challenge, of course, and many of you may know this already, that the current state of the art of some of these open weight models that are more freely available is lagging. Only about six months behind where the frontier is at, so that should be in and of itself kind of a frightening prospect and something we should really look at collectively in terms of how do you legislate that more effectively. But dropping the compute threshold is the first start around that. The second part of your question with regard to data centers is we want to be able to detect when these things happen. And one of the best ways to detect it is if there's something weird going on at the data center where all of a sudden there's a spike in activity associated with a certain model. That should raise flags in the same way that if you detect your water bill going up suddenly, there's been a leak in the pipes in your house. You want to make sure that data centers are on board with helping police this as well. because ultimately we're trying to get to a response for some kind of catastrophic event as quickly as possible, and that means everybody's keeping an eye on what is being used to deliver these services. Thank you.
Thank you. Representative Schusterman.
So many questions, but so little time. AI researchers, anyone can answer this. AI researchers are talking about concerns of recursive self-improvement with AI. I know this concept has been around for at least 10 years. However, now that we're actually in the reality where can you help us understand how recursive self-improvement can complicate safety and our control over AI? And I want to explain what that is. recursive self-improvement is AI training programming itself without any human intervention. And my colleague, Rep Guzman, cited that example.
I'd be happy to help with that, Representative. I think that's a really important point. Recursive self-improvement is the idea that AIs are building better AIs themselves. And so you get into this loop, a feedback loop that becomes very fast where the research and development of new models is automated. The best frontier model builds the next one and the next one And instead of rolling these models out perhaps each few months you actually rolling them out in weeks and even potentially days And so Anthropic, for example, one of the major labs has said that they are basically at about 25 or 26 percent of their development being done by RSI today, recursive self-improvement. And so They're about 25%, 26% today. They say internally that it could be within the next year where it's 90% or 95% self-improving AIs. And so the concerns there are many, but one of the big ones is the lack of transparency into the models. If the models are building the next version of themselves, better versions of themselves, they're going to be doing so really without human understanding and oversight. It'll be totally automated. They may use methods that we don't have eyes on, we don't have transparency into. And the speed of that development is going to be potentially so rapid that, again, they could be advancing these models in days potentially or someday in hours even. And so that's when you get into this concept of developing what could be a superintelligence, something that probably bears discussion as well. That's a term that has a specific meaning, but it's being used a lot right now on the national discussion. But RSI is kind of the march that we're on currently. And if we fully automate R&D of AIs, we have AIs building better versions of themselves. The transparency, the human oversight and control tends to wane. And the AIs essentially, I mean, would be much harder to control in that context. And Rep. Schusten, I'd love to chime in on that as well. The challenge, as my colleague has just said, is that there's no human oversight or there's less of it the more this technology progresses. And one of the things I think that's important to legislate is that no releases of the models happen without some sort of human saying, yes, this is okay to go. Otherwise, we're just ceding control to something we just don't completely understand. And so I think it's very essential that in this next iteration of legislation that we're looking at, even if it's outside the scope for this particular bill, is this is something we really need to look at and legislate very specific human oversight. And I think that's where the IVOs can also come into play, where they're not only just verifying things after the fact, they're actually proactively looking at releases before they go out the door. Thank you.
Representative's there.
Thank you, Chairman, and thank you to the makers of the bill for this important issue and putting the spotlight. Thank you to the testifiers. I have a question, and I guess it's just being naive with all this new and upcoming. What is the difference between artificial intelligence and superintelligence? We're hearing them kind of swing in both ways, and, you know, should we kind of be prepared for interchanging them or is there a big difference between the two of them or are they kind of the same?
I can just start in. I'm sure my colleagues have some thoughts on that as well. I think superintelligence is just a brand that the Trump administration has placed on, the technology itself. It has no inherent meaning at all to me, at least, having been in the field for a while. So I think in general the difference between artificial intelligence and artificial general intelligence, AGI, It's just an advancement of the art. I would see artificial intelligence being an umbrella term, then AGI being a certain state that the base of the technology has advanced into Rep Guzman Thank you Mr Chair
And I appreciate all the testimony this morning. It's been very helpful in helping us develop and craft our future legislation. One of the things that has been recently popping up and something that has been kind of popping up on my radar is the fact of, you know, we talked about open weight models and their ability to sometimes even bypass security models altogether. I mean, there are open weight models that are currently being developed that are completely, that have no safety guardrails and that are readily available for folks to be able to write any type of script kitties, as we'll be calling them, but write any type of script, like DDoS or whatever, whatever it is, right? And so one of the things that one of the larger AI companies recently talked about putting together an experiment where they chained together a bunch of Android boxes like we use, like a lot of folks use at home, whether they're Fire TVs or Alexas and whatnot. And what they found was that those smaller models were able to replicate themselves with smaller Android devices. And so to kind of put it in perspective for my colleagues, it's essentially creating an AI worm of itself, allowing itself to be replicated across multiple devices. And so in effect, if you're turning off one device, it doesn't stop it from being able to replicate to other smaller devices. And so I guess my question to all three of you are is, you know, are these guardrails going to be enough to stop those type of smaller attacks using smaller weight models that, of course, are able to potentially able to replicate themselves across IoT devices, which are Internet of Things devices, again, Alexas, Firesticks, Android, cheap Android TV boxes, et cetera. how do we potentially legislate to stop those type of attacks from happening?
I can start with this one, and my colleagues, of course, will have some opinions on this as well. And it was great. I think it was TV sets that they used for this, so it's actually fascinating that they spread a model out across multiple LG TV sets running Android and were able to do this.
Which essentially means, and not to cut you off, but essentially means that you don't need a data center.
Exactly. You don't need a large data center to run a model. So I just want to emphasize that. Yeah, so I think where this really needs to go is not necessarily how big something is, but what it does, right? And I think we really need to focus on the harms. That's one of the reasons why I suggested that we go beyond just catastrophic harms, but actually look at another classification of that and look at serious harms as well and define those very particularly. And then we should hold people accountable in the same way we would hold them accountable for using any kind of consumer technology, choose which one you like, to harm other people. So, again, I think we ultimately need to get away from the frontier versus not frontier and really just look at ultimately what is the impact that these technologies are having and being very specific about what is going to be protected against. I would agree with that and kind of related to that. I think one thing many states are looking at and also the federal government is providing liability to the companies for these types of things. So there's not much there when a rogue AI agent does its thing and commits what would be a federal crime or a state crime. If a human did it, we don't have a way to deal with that. And I think holding the companies liable is probably a good first step and something that lawmakers should look at.
Well I appreciate your comments on that Of course that something obviously we looking forward to kind of tackle within our respective legislations and hopefully combine them But again, it's just something that I believe that will be a threat to all of our livelihoods. Again, if there's a large-scale attack on IoT devices, again, we're talking about smart TVs, smart ovens, smart thermostats, television devices that are connected to cable. all those things could potentially be taken over by a rogue AI agent and used for nefarious purposes thank you very much mr. chairman thank you sir we'll
come back to a rep Eric Nelson thank you my brother to another mother
yes sorry we digress a little bit you gotta have some fun so a follow-up question. I really appreciate the responses and, you know, something that Representative Guzman was touching on that the accountability component. In the other states that you've mentioned, a lot of times we talk about companies, that's a financial penalty or level of crime. But when we're encouraging, you know, regulating beyond 50 or more fatalities to a serious level, which might be 25 or more fatalities. Have any of these states had a personal component? I know sometimes environmental legislation or even campaign finance reporting, there is a personal, the person who signs on the line has an individual accountability. You know, I'm just thinking of streetlights freezing that results in, you know, the death of 10 or 15 people. It really wouldn't trigger anything within this legislation. as we look at these sub-tiers, should we consider personal accountability, or is this moving beyond humans and it's all electronic accountability?
That's an interesting political question. Of course, if we were able to create some kind of accountability at an executive level within some of these companies, and frankly that stretches beyond just the AI industry, if there was able to be some kind of executive level accountability in terms of what happens when the harms are, in fact incurred to individual folks. That would certainly be one way to address this. Another way would be to make sure that you have a private right of action added to these bills. I know that's also somewhat charged, but in many ways that would, in fact, allow for individuals to sue for harms that they have incurred as a result of these technologies. Thank you.
With that, we'll go to closing comments and questions. I will start with the minority chair, Rep. Ortec.
Thank you, Mr. Chairman. I have a couple of questions. Stephen, I'll start with you, and anyone is welcome to answer this. But when you were answering a question, it struck me when you said this about understanding the models before they're released. How does anyone completely understand any of the models before they're released? They're so complex, and it's just like how do you test something like that that is, I mean, the knowledge level is so intense and beyond what any person could test every single limit of it to know that it's completely safe before it goes out. That's an excellent question.
There are a number of companies really thinking that through when the open AI incident happened with Hugging Face. There's a company called Meter that went in and did a lot of the evaluations. and basically the forensics behind this, and they made several recommendations open AI in this case in terms of what they need to do in terms of creating greater accountability for how these systems are actually behaving. And what they were suggesting was that there would be a separate overseer class of agents that observes everything that's happening within the overall system and that these agents are given the same context as the rest of the infrastructure in order to understand what's going on from one moment to the next. So given your extremely important point about how complicated these systems are, it's incumbent upon the AI providers to instrument their code in a way that allows for the supervision to happen. I'm not saying it's easy, but it's something you need to architect right from the ground up.
Well, I'm glad you brought up meter because I know we even talked about – Ms. Wilson brought up IVOs, and I think all of you touched upon IVOs. And this is the part that I struggle with when we go down this road. I mean, we should be regulating, we should be doing something. But the question always comes back to who and how and what does that look like? Meter is made up, I think, almost entirely of former AI employees. So, like, publicly their subjectivity, I would question that, whether they can be objective in their evaluation when they're doing the audits and doing the evaluation of these models. And I think that's the important part is we're looking at this from a public perception point of view is if we are having a third party test these, we want to make sure that third party is actually doing what they're supposed to be doing and that we can have faith in the results and information they're giving us. And knowing that companies, these AI companies have employees there, or former employees that are now running this, it doesn't actually inspire me to feel like they're going to give me objective information back or that they could be corrupt or bought off. And I'm not saying that they are by any stretch, But the perception could be there that they are. I mean, and that's where I struggle with is, like, how do we regulate and recruit people that maybe have the expertise and the knowledge to evaluate these models but not necessarily came from one of these big companies? And I think I was kind of joking with the chairs. As a government, do we set up another department or agency to oversee this, or do we leave in the hands of the third party and let them continue to do third party and self-regulation across the board? And I'd be curious to get all of your thoughts on that because I think that's the hardest part of this. Their legislation, I think, lays out a good model, but we have to take it a step further and figure out, all right, public perception, public safety, because people are concerned about this stuff. I mean, we talked about the hospital, the fake tips to the police. All these things have huge consequences and cause a lot of collateral damage. So, I mean, like, Ms. Wilson, I know you went through this with your steps and the things that you had put out there. But in practice, what does this really look like? Should this be in the hands of the government to do the regulation, or should we allow this to continue to be third party and self-regulated?
Yeah, I'm so glad you asked that question. I was smiling It a very serious topic but I think this is where we have really formed This is why Fathom exists This is what we been talking about across the states and in D as well I think we sort of need to evaluate this question in observing what the status quo is. The status quo right now is the biggest AI companies in the world really doing self-report. They can decide what they do or don't tell us. I think we're lucky that so far, at least the ones that we've heard about, these major incidents, they've been forthcoming about. They're not required to be forthcoming right now. So there have been steps taken to require them to do it, to bring in third-party auditors. A lot of these entities are already bringing in right even before the very public, as Steve mentioned, meter autopsy after hugging face. All the big labs bring in third-party evaluators to do either pre-deployment or post-deployment reviews of latest models. They do that right now, but there is right now no independence requirements. There's no public accountability. There's no need to produce public reports. There's no guaranteed access to what those independent verifiers have access to. Also, a meter was very forthcoming that they were only provided access to a limited number, a sort of window of time to review. And they had said, like, their review might have actually – they may have been able to have a deeper analysis and been able to share more if they'd been given more access to both data and a bigger time period and scope. These are all things that we think government is best situated to do. So it's probably not realistic if we understand how much these AI researchers are making that the government is going to be able to or whether this would be the best use of resources to hire and build out a massive government infrastructure. infrastructure, just thinking through how big these companies are, how powerful they are, how quickly the technology is evolving to think that government could, and that it would be an effective use of taxpayer dollars to hire the leading AI researchers, pull them away from the companies, which is really where this expertise lives right now, and invite them to be public servants and to do that work. I'm sure there are some people who would do that. We are really in a moment right now and you talk about like, is there the expertise to do this work, to keep up? We're at a critical moment where the representative talked about RSI. The capabilities are expanding so quickly. The investment in advancing the science of safety evaluation is critical to be happening now and to be investing in that science outside of the labs. I think it is unlikely to expect that at least in the time period now that the leading AI researchers are probably going to come from you know most of them will be coming from the labs And this is where government has the ability and is best positioned to indicate how they independent, how you're going to trust them, and to put in those guidelines, those safeguards to ensure independence. I don't think it's necessarily a bad thing that they're, it's not necessarily, I think, as you said, like a reproach on their character if they're coming from one of these labs. It doesn't necessarily question their credibility, but we can put in laws and rules in place that ensure that they're independent. And if they violate those, that they'll account for that as well. And Representative, if I may add, I've met some of the folks at Meter and find them to be very smart, very earnest and sincere. But I think it would be great to try and connect them directly to you so you guys can ask them the questions that you have. I think those are valid questions. And I think that they actually tend to be very concerned and very critical of the companies. And some of the folks who've left the companies started out posting publicly about their concerns about the companies they work for. And oftentimes it led to them either becoming whistleblowers or moving to a researcher position or starting their own organization, as Miles Brundage did, who's a gentleman who left OpenAI, who was concerned about the safety and security there and left to be able to do work on it. And so there's a lot of folks who understand the technology who are still close enough to understand kind of what we need to be asking and what we need to be doing and how we should evaluate things, but also have that independence in that distance, some of them actually gave up equity in the companies to leave and to sort of tell their story and to start organizations to focus on this because they were so concerned. And they literally lost potentially millions of dollars in equity for doing that. And so I think talking to some of those folks and having them help provide guidance on how they would suggest the work gets done or who are the folks that can help do the evaluations, that sort of thing, I think would be really, really good. I think the only agency that I'm aware of it probably could do this itself would probably be the national security agency which i know is doing some testing and evaluation of models and it has the budget the staff the personnel and the expertise to do that on the federal level but obviously we need states to be able to to implement this and to be able to have some oversight as well yeah i definitely think it needs to be
addressed at a federal level ideally because states just don't have the resources to go do this unfortunately there's been very little leadership in this area right now hopefully that That will come and hopefully we don't need to wait for a catastrophic event to really light the fire around this entire topic. We've been very successful in other industries, the air ion industry, the nuclear regulatory industry.
There are precedent for this kind of stuff, for other powerful technologies or technologies that have impact. To your point though, I think it does need to come at a federal level.
was even thinking just now there maybe there is some way to have some kind of rotating expertise that comes from the frontier model companies and it almost like a draft where you need to serve with this government agency to help in fact enforce this industry Well thank you all for the answers I appreciate you all being here today And Brendan, I'll certainly take you up on that offer. I'd love to meet the folks over at Meter and have those deeper conversations. So thank you all. Turn it back over to you, Chair.
Thank you all for that. And I think that was a perfect wrap from my perspective. I certainly appreciate the need for governing the deployment of AI in our communities. And I think that there's probably something that we can do here to make sure that models are gatekept. But I do wrestle with what is federal responsibility, what is state responsibility. And to some degree, I think we also need to make sure that when we're in these buildings and these rooms in Harrisburg, that we're also thinking about what are the technical capabilities that are required as part of our state governance models generally. So much of what we're talking about is how do we think about AI from a public safety perspective. The issues we're talking about might exist in Pennsylvania, but they didn't start here, and they're not exclusive to Pennsylvania. That's what a national or realistically a global framework ought to look like. and I always wrestle with the more we are paying attention to and trying to legislate what should be global framework, I want to make sure that we're not then losing sight of, you know, what are the capabilities within the, you know, Pennsylvania Department of Education for understanding AI and its role. What are the implications for the Department of Transportation here for us to understand AI and its role? So thank you for that, and I think as we continue to look forward to a new term and make sure that we're, this has been a great term, rolling up our sleeves. I suspect this is, I guess, our last meeting of the term, so I will thank our chairs and the staff for all of their work in this, the inaugural year of the Communications and Technology Committee, I guess the inaugural too. There's so much great work to be done. I think we're ending with tremendous momentum. Thank you all, testifiers, for joining us, to our legislators who are bringing forth these great pieces of legislation. We look forward to continuing to refine them off into the new term. And to you all, thank you so much for joining. Have a great day. Have a great week. Make sure you vote. Thank you. .