June 30, 2026 · Technology Committee · 24,372 words · 9 speakers · 261 segments
Here. Rodriguez. Here. Tatone. Here, ready to go. Madam Chair. Here. And thank you all for being here. More on time than I was. Apologies. There was a lot of traffic getting down from Loveland. We have a lot on our agenda today. We're going to start with, we're going to have Sarah Thunberg and Jill Frazier come up. And we have 20 minutes for operating model and reorg presentation. And then we'll have 20 minutes for Q&A around that time. So if you're ready, you guys can come on up.
Very good. Good.
And Director Thunberg.
Thank you, Madam Chair. Thank you for having us, all of you, especially on an election day. I really appreciate the time you carved out. It's a pleasure to be here with you again. I am now the state CIO as well as executive director for the governor's office of IT, joined with deputy director for security and infrastructure and our state chief information security officer, Jill Frazier, and obviously the amazing Paige Asman.
Okay, cool. Great.
So as we, as you are aware, about a month ago, we undertook a pretty radical step towards transforming OIT to be an organization that is focused on delivering value. As we have talked about many times in this setting and in others, OIT was really struggling to deliver. It was not a result of hard work and effort. We had some very fundamental structural problems that took all of that great effort and made the delivery of actual value and outcomes really, really hard. Additionally, I believe we lost sight of our statutory purpose and our authority and got pretty confused about what our primary role is. And so our primary role, as outlined in statute, is to deliver valuable, modern technology guidance, support, and partnership to Colorado's agencies in service to Coloradans, the taxpayers that pay for our service. That looks a lot like rooting ourselves in oversight and standards, providing excellent advice and expertise to our partners, coordinating with our agencies, and ensuring there is solid security and technology strategy for the state. We are not a vendor. We are a service provider, and we are in service to Coloradans, and we needed to reorient around that foundation. The change we made which we talk about in a minute was quite significant and was pretty all at once And it was important that we do that I think we gotten some feedback and questions Why Why not a more subtle or iterative change And there are two primary reasons. The first is that the threat environment we face as a state has radically transformed in the last 12 to 18 months. Jill Frazier is going to talk in a minute about frontier AI models and what it changed about our security posture as a state. But security vulnerabilities that we previously had a year to discover and patch, we now have a matter of hours to solve for. Additionally, encryption standards are changing very, very quickly, and so we needed to pretty radically become more responsive. Additionally, we, over a course of a year, tried a number of experiments to change OIT in a more iterative way. And every time and every experience we had, the old gravity really hold us back. We tried one pod, and it just got ugh. And we tried and we tried and we tried, and it wasn't working. So we felt it was very important to make a significant shift. And that shift is moving from a traditional IT model that is focused exclusively on the iron triangle of scope budget timeline to one that is focused on delivering value to users. Rather than saying it's shipped and we're done, that is success. It is about delivering continual value for our investment to users. And done, instead of being shipped, done means it's actual working software. It is solving the problem for the user, whether that be a case manager, a call around getting a driver's license. It's not that we deployed it. It's that they can actually get the business of government done. And that we are held accountable over time for it continuing to deliver value. It's not a once and done. This is rooted in the product model. The product model is a large shift for our organization. It is a shift in mindset and culture that is rooted in a number of things, including empowering our teams to solve problems, giving them both responsibility, accountability, tools, and training to own the problem set and deliver that for users. It is one oriented around outcomes rather than outputs, and it is a mind shift. It is a true transformation. There are three dimensions to how the product model works. The first is how we build. Rather than waterfall to Big Bang and calling it done and it being a win, it is about small, frequent releases that are rooted in user research. We are not, at the end, putting a technology in front of someone and saying, does it work for you? Instead, we are bringing them along every step of the way. Again, whether those users are OIT employees, state employees, agency partners, or ultimately Coloradans. It also changes how we solve problems. Again, directly working with users, but also evaluating if solutions are viable. Are they feasible? Not is it a binary here a list of features develop them check them off a box but really understanding if we solving problems It also is a change in deciding how we solve problems. OIT's low trust with our agency partners and our lack of a strategic plan has led to sort of a squishy technology environment. We haven't been clear about when it's appropriate to use Salesforce and when it's more appropriate to use ServiceNow. Instead, we're sort of like, whatever you want to do. And that's led to a lot of expensive technology that doesn't actually work very well. So we are trying to think much more and much more strategically about how we solve the problems and what the right technology solution is. Not everything is a hammer looking for a nail. We need to be right-sizing things. In order to do this, as you were noting earlier, we are realigning our teams into cross-functional delivery pods. Product-oriented delivery is what pod stands for. Leveraging four key roles, product management, service design, delivery engineering, and delivery management. These teams will work in a cross-functional way across our organization. They will be accountable for outcomes. not just process, but that true deciding what problems to solve, what's the best way to solve them, and how do we ensure value. Pods will look different across the organization depending on how they're solving problems, but one I want to take a minute to talk about and sort of dive a little deeper in is the agency delivery pod. We have, as part of our realignment that we announced about a month ago, we moved a number of teams around in an attempt to break down some pretty significant silos we have had. One is our agency applications teams. They are responsible for operations and maintenance for a bunch of custom applications, about 2,000 that serve agencies in Coloradans. They were in an isolated technology org that was completely disconnected from our agencies and the agency users. In order to have a change happen in that application, the agency needed to talk to a person who submitted a form that went to a committee that did a thing and did a thing and did a thing. It could be sometimes 12 weeks to do something like change the color of a button. That is not a good use of resources and it's a terrible outcome for Coloradans. So that team is now directly in the agency pod and meeting sometimes multiple times a week with the program teams and the agency teams in a modern cross-functional team. We also are scaling up in these areas of delivery management, product management, design, and engineering. We didn't invent this model. This is the model by which pretty much every successful technology organization is structured, from big tech to amazing government technology teams like GDS in the UK and Gov Singapore, which are exceptional. And Rep Kelty, I know you're online, but this I think speaks to a question you asked last time we were here, which we didn't have a great answer for, which is why wouldn't we use a validated evidence-based model? We are now. We are using the model that is successful in Amazon two pizza teams again Singapore So we didn invent it We adapting it for Colorado And we working very hard This transition will hinge on the everyday decisions that the thousand people working at OIT and in our agencies make every day. And we need to help those teams be exceptionally high performing. and we are empowering them and working with them in a lot of ways. But some of the really important things is that instead of being a gatekeeper of these requests, we are wanting our teams to be true partners embedded in our agencies. We are measuring their success by outcomes, not outputs. While outputs are important like ticket speed and ticket processing, did we actually solve the problem? Um, discovery is an essential part of this work. We didn't do very good discovery before. Um, and then also, as I've said many times, and you will hear me say many times, done means working for the users. Not do we say it's not, not do we say it's done, not do we check the box, but is there a user on the other side that says, I can solve my problem. This is saving me time. I can apply for Snap more easily. what done looks like. The product model is the framework, and the operating model we've deployed is how we're going to operationalize that within OIT. There are six components to our really radically redesigned system. I'm going to go through them fairly quickly to ensure we have time for questions, but if you ever want to talk about any of them, let me know.
So the, okay. I was just going to make sure that we're good on time. We're more than halfway done,
I'm hoping with your part of the presentation. Okay. I just want to make sure we can hear.
Thank you. Thanks for pacing so well. Yes, of course. So the first thing is, uh, we have
changed our structure and it's an important one that we did a lot of discovery and research about what are the services we deliver and have reoriented ourselves around a service delivery model. So on this slide, um, you'll see also enterprise architecture essential and something we've lacked. So the foundation for our structure at OIT now is this enterprise standards policy and governance. New colleague who is sitting behind me, Casey Cook, we'll introduce him in a minute, has joined us from USDA and will be leading up this new function at OIT. Really a strong foundation is what we need, what we build, how we build it, what standards ensuring interoperability, security, and sustainability of our investments. On top of that is Jill's organization, security and infrastructure. These are those mission critical services that just need to work. Ideally, they are invisible and absolutely working. Broadband. We need to deliver our services to Coloradans. We do that through broadband. And then the top three layers are about service delivery, supporting state employees, supporting our agencies, and ultimately delivering value to Coloradans, all supported by our business operations functions like HR, comms, and finance. This is a pretty radical structure for us. I would argue that for many years we thought of ourselves like a vendor and we're seeking to, yeah, different, about service delivery. The ecosystem is also we are taking a maturity model to how we partner with our agencies. CDEC is here today to talk to you about amazing work they're doing. They are a far more mature digital organization. And so in some instances where it's sort of maturity a level one, embedding as a delivery partner. In CDEC's case, we are about enabling their success, not gating them, not blocking them, but how do we allow them to just own and run towards an amazing future? And all of this on a shared platform that protects the state's dollars. Talent is essential, as I said. OIT needs a workforce designed for modern delivery, and we are actively recruiting new talent, also bringing back old talent, and then focusing on retention. One of the huge changes that's happening as we speak is building a technical ladder. We have a challenge at OIT, which was historically, if you wanted to get promoted, you had to become a manager, which when you are in a technical organization, not everybody wants to be a manager, nor are they well suited for that. We need to have a success, a ladder of career progression that is around technical excellence and technical leadership that doesn't require management. We are investing in straightening out our governance, which looks at OIT like we have stacked sort of policy and compliance on top of policy and policy and haven't retired a lot of the things. So what made sense when we put it in place, now lots of times does not, and we need to pull back some of that so we can be really in the business of risk reduction and risk-informed governance and policy rather than compliance or control. Same as process here. Lots of our processes made a ton of sense when we put them in place maybe a decade ago. We are inspecting do they make sense and truly reduce risk or make work easier now. If not, we're actively and publicly retiring it. And then we are working hard on evaluating how we show up every day, Starting with users is the first place. It's not about us. It's about the Coloradans we're in service to. We want to ask every decision we make, what does this mean for the person that has to use this policy governance technology? And is it really solving their problem? We invite more feedback. It will make us sharp. It will make us question. And it will make us ensure we are delivering excellence. Mistakes are expected. Blame is not. We want to be a learning culture where we can evaluate and look and say, ooh, that didn't work. How do we learn from it and do different? We are wanting teams to own problems, and so pushing decisions down into the teams that are closest to the work. Iterative learning, having spoken hypotheses, and then small releases and feedback loops that test those before we go big. And then centering on principles rather than process. So what does this mean for our partners? In our first 90 days, we've set a pretty aggressive set of targets, and we are hitting them so far. So we just closed out 30 days. Every active piece of work in flight was reviewed with our agency partners and stack ranked. We had a challenge before where everything was a priority, so nothing was a priority. So we've worked with the leadership of every agency and stacked one through X on what their priorities are. Each has a named owner who is accountable for that delivery, and work is stabilized We are seeing that work is actually moving much more quickly which was our goal Our pod teams are taking shape Hiring is underway We can talk another time about mandate letters, which are interesting, and I will share them with you. Additionally, we are making real progress on our audit findings in a faster and more iterative way. We have submitted all 34 of our findings that are due to OSA tomorrow. They were submitted as they were implemented and complete, rather than all at once as we had historically done, which is, OSA is giving us great feedback and much prefers this new way. Additionally, we are working hard. This is it. I'm, yep, perfect. Significant savings. We targeted $8 million in savings in our first 90 days. We've identified $6 million in platform savings already, plus a 10% reduction in our GCP cost. That's Google Cloud. We have activated our partners, some of them who are here, to support risk reduction, the operating model, and an accelerated value delivery model. These are some of our amazing vendors who we reached out to and said, we need to do this differently. Can you support us? And they have said yes. And that looks like cost reductions in our platform renewals, working with us on building new roadmaps, how we deploy our tooling better. On hiring, our velocity is high. We have 1,300 as of Friday, 1,297 applicants for our new roles. We've made 24 offers, five more in progress. About 10 of those are returning OITers. As I said, we've also made a number of key hires. Casey Cook, who is behind me, is leading our enterprise standards policy and governance. He comes with 15 years of experience at USDA, leading complex and strategic solutions, including their cloud architecture. Very important. Sarah Holt comes to us from a very successful career at Google, helping us support our agencies and think differently about how we deliver for them. And on Monday, Jane Young will join us from the Federal Administration for Children and Families, where she was their chief AI officer. She'll join us as our AI strategy lead and our chief AI architect. really good and important progress in just 30 days. Questions?
Very good. Thank you so much for that. I do have a request. I appreciate you sharing all of this, and I would love to see a new org chart if we could, just so we know the key players and that kind of thing. 1,300 applicants, that's a lot, and you said 10 are former OIT employees.
Director Thunberg.
Thank you, Madam Chair. We have extended offers to 24 candidates. Ten of them are returning OITers. So 1,300 applicants, and we've processed, reviewed, I think about half so far. We are doing them in waves. Every other week we launch a new wave. And I just ask one more question When you are looking for product managers I assuming you looking for people who have some understanding not just of IT but of the department that they will be embedded in Is that right or not so much Yes.
Oh, you may go. Thank you, Madam Chair.
Yes. And then additionally, we are partnering directly with our agency partners on the hiring. They are participating in the panels. It's the first time we've ever done that at OIT. Nice work.
Very good. Okay. Vice Chair Tatone.
Thank you, Madam Chair. We'll let Rep. Kelty have a question here, but I have several questions, and I hope that we can spend a little extra time to go through these because I think there's an important conversation that we need to have. That presentation was slightly different than the one we had. I'd like to make sure we get that with the changes that we did not have in there. Okay, so with respect to your pod model, how are you going to prevent this model from creating agency-by-agency silos and shifted costs, weaker statewide security standards, and unclear accountability when services fail?
Director Thunberg.
Thank you, Madam Chair. Thank you, Representative Titone. It is a very good question and one that is at the forefront of our thinking and planning. There are a number of ways. The first is going back to that slide. Hold on one second. This one, the structure. The foundation of all of the work we're doing is this enterprise standards policy and governance so that we are setting the rules of the road by which everyone in the state operates. So no matter which pod you're on, the security requirements and expectations, platforms, the ways of working, the ways we're showing up are rooted there. So while they might have a flavor of CDPS versus CDLE, because those are different programs, the standards are the same. And they will be statewide. Additionally, the next layer is a place of this is where our platforms are. And those platforms are built on the standards. So when you are deploying Salesforce, our guidance, our standards, our governance will articulate how it is deployed no matter which agency we're in. So it's this building customization on a standardized framework. Additionally, we're supporting all of our pods and all of our pod members through enterprise guilds. Guilds will be their professional home, communities of practice. Where those standards around what does good service design look like? How do we ensure it is the same? What tool stacks do we use so we have consistency? it will be living in those guilds so that then we get sort of across good platform, good support that yields good outcomes.
Vice Chair Tatum.
Thank you, Madam Chair. So what formal evidence shows that the agencies that OIT supports were actually consulted with before this workforce reduction and the operating model change? and will you provide us with the agency by agency feedback, the objections, risks, and unresolved concerns to us? We would like I would like to see and maybe the committee would also what feedback did you get I don want anecdotes I want like real I want to know what communications went on Because everybody seems to have said that this came on very suddenly and nobody knew what was happening. So show us that you did this with the agencies in mind and it's not just a dictatorial reorganization without the support and buy-in of the agencies.
Um, Director Thunberg, did you communicate?
I know you've got the agencies working on the hiring process now, but did you communicate before the reorg with departments, um, outside of the regular? I know they give you feedback each year and that is pretty telling, I would believe, but yeah, I'm, I'm, I'm also interested. Yes.
Thank you so much. Thank you both for the question. We have a series of regular touch points with the various layers of leadership in agencies. So executive director to executive director. I meet in my previous role as the deputy, met monthly with each of the deputies. And in that deputy meeting, there was an OIT topic from the time I started in that role in February. every month, bringing a topic about how we were performing, how we could perform better. Additionally, we have a series of meetings with our product directors. Product directors are the people who we work with every single day, sort of OIT's tightest partner. They're members of an organization called the Customer User Group. We use a fortnightly meeting with them to engage conversations about our performance, what's working, what's not working. We have a shared agenda. Those are the common spaces, and those are the ones we're happy to share the notes. In addition to that, a very robust series of one-on-one meetings. I would say I have to go back and count more than in the hundreds, evaluating various pieces. So what would it look like if we moved from project to product? What would a pod look like? What would you need from them? These were the questions from November, really, but especially from February through May, we were engaged really deeply. What was not, I want to be candid about this, We did not ask people, other agencies, for their feedback on our layoff. It's our organization to evaluate the performance of and to structure. And I think that there is a very human and appropriate thing, which is any layoff is a surprise the first time you hear about it. And there's not any way to sort of like slow roll a layoff. The first time is hard, and it is the way of layoffs, and I wish it wasn't.
Okay.
Vice Chair Titone. Thank you, Madam Chair. I just wanted to make a statement, and then let's go to Kelty, and then I have a whole bunch more stuff. I do want to see some of that evidence that there was communications and specifically I want to know what the agency by agency feedback was, any objections that were there, the risks talked about and any of the unresolved concerns that they had.
see that in writing. Great. I'm seeing head nods. Rep Kelty.
Yeah. Hi there. Thank you. Um, I actually have some really good questions already. Some of them, which I actually had in the back of my mind. Thank you for asking those. Um, one question that I have, um, I guess it's pertaining to, um, uh, the question I had or the topic I brought up, um, just recently about the software, about interagency, like who is using what softwares, are there antiquated softwares, are we removing the softwares that we're not using, we're not paying licenses on things that we're not using, saving the state money, and that kind of thing. So I didn't hear within the presentation any verbiage or any discussion on that. I don't know Maybe it just wasn't part of this presentation. Maybe it's part of another one. But I would like to see where we are doing a discovery of things that are not needed, things that are needed, and being able to make some sort of intelligent listing of what the state is actually utilizing as far as software licenses, expenditures, and so forth. So do you have anything like that? Have you started that? Are you planning on starting that? And when can we actually see what was done in that structure?
Thanks, Rep. Kelty.
Director Thunberg. Thank you, Madam Chair, and thank you, Representative Kelty. Yes, we have begun that work. We are working, one of our partners who we are doing this with is actually in the room, Tanium. We are evaluating via our network. what tools people are using, so not just what they say they use, but what they are actually using. At the same time, we have one of our architects evaluating the utility of each of these tools and the cost, and we are working actively to centralize on a smaller number of tools so we can reduce the total cost of ownership of our tools and SaaS platforms and all of those things. Actively underway, and I would love to add it to the agenda, I think, for October or any time you'll have us back.
Okay, very good. Rep Kelty, do you have any other questions?
No, that's all I have for right now. I appreciate it.
Thank you. Wonderful. Thank you. Vice Chair Titone, and we've got about 10 minutes left.
We're going to need more than that, please. No, I've got about 20 questions here that need to be asked, frankly,
and we have until four, technically.
I would really like to take that time.
Go for it.
Yes, why don't you ask? I have really, really, you'll like my questions.
We have to get to them.
Otherwise, we're not going to get the answers.
Okay.
Rep. Pascal. Thank you, Madam Chair. I just wanted to drill down a little bit more on what Rep. Kelty said. I think the next level down of accounting of like we saved $8 million, where did it come from, and included in that, of course, is software licenses, probably other stuff as well. That would be really useful information. I guess it wasn't a question.
Good. Nice. We're seeing thumbs up. Okay. We started behind because of me. That was my mistake. But we did start 10 minutes behind. I was hoping we could catch up at the 2.10, but that would require us to stay on track. We have departments here who want to speak to us and they only have 10 or 15 minutes and then we got some time for questions and I pretty excited to get into the departmental AI stuff as well So I just want to remind us that we cannot change what is going on with hiring and firing in a department. I'm willing to entertain more time but I want to set a time limit. I'm not willing to forego the entire agenda to make this longer. So do you have a certain amount of time maybe that we could do this in? And if not, I'm happy to come up with one. As long as the answers to the questions are concise,
I think we can get through most of these questions in a pretty quick amount of time.
So instead of 10 minutes, 20 minutes maybe? Let's see what we can do in 20 minutes. Okay, we're going to do our best. We're going to talk fast, and we're going to answer fast. Very good. So we're going to do until about 2 o'clock. Let's see how far we get until then.
You're up.
And you're up.
I'll still.
Let's just dialogue so that way I can just keep going.
Okay. How about that?
Okay. All right. All right.
So while we're talking about the money savings, I have a question about that. How will you prove that the restructuring is not shifting costs to agencies by forcing them to pay? OIT, while also hiring staff contractors, third-party vendors to complete the work, OIT no longer has the capacity to deliver.
We're dialoguing, so just go for it.
Thank you. One of the commitments we've made is that we will have a monthly session with each of our agencies to work through and talk through in plain language their bills. Those will also be available to you. So that's how we're going to do it every month, bill by bill.
Okay. So, and then on to billing, I wanted to ask a question because what I understand is that a lot of the billing with real-time billing is very vague and a lot of the departments don't know what they're actually paying for. You're going to be putting more detail into the billing because I've just requested to the controller for all of the records of payments that OIT has had in the past to find out what actually has been paying for. I'm still trying understand the $36 million that you had that you didn't need, but just about the billing detail. Is that something you're going to put into there so the agencies know what they're paying for?
Yes, same conversation. Plain language was really important.
Great.
Okay, so what specific operating model are you implementing? Where has this been proven at a comparable statewide scale? and what direct experience do you personally have implementing that model across a multi-complex agency enterprise?
We used a very similar model to respond to COVID under my leadership in the COVID Innovation Response Team, PS, CDPHE, Gov's office. So it sounds like not really very comparable because this is much, much more complex than just a COVID system.
I would beg to differ.
We built a team from zero to 450 in 90 days, and we had some of the best outcomes in testing containment and disease control in the entire country. Well, I would still disagree that it's not entirely comparable to what you're undertaking now.
If functions are moving back to agencies, what is OIT's remaining value proposition, And how will you prevent the state from recreating fragmented technology inconsistent security duplicated costs and uneven agency capability I not aware of what functions are moving back to agencies that would create that
We are changing the way OIT is functioning and the way we show up to be about value delivery, product management, and long-term value. That's how we're going to do it. I don't see anything going back to the agencies that would lead in a different way. The one thing I will also add is that for agencies who are requesting delegation of authority, we are using a maturity model to evaluate that. And it is a transparent process by which we engage duly. And it's creating a roadmap so in the instances where it is appropriate for them to have delegated authority, we're clear about why, we're clear about the standards, and then we'll review it annually.
Okay.
What baseline metrics existed, if any, before the workforce reduction? And what 30-day, 90-day, 6-month, and 12-month targets will prove this model is improving agency outcomes, not merely reorganizing OIT internally?
A number of them. One is our agency satisfaction score, which has continued to decline. It's an annual survey. We instead this year are deploying a monthly survey. So we get a monthly check-in from our agency power users. It's a grouping of them so that we don't just wait a year. Additionally, we're evaluating various things like completion of our object, like completion of work on time, on budget, and on scope. We weren't ever able to do that before. So while we are evaluating that, which is an output metric, we're also deploying a series of user metrics.
Did we shorten the time that it took to complete a task? Were we able to increase task completion overall?
And you'll be able to demonstrate that and show us those details. Absolutely.
Okay, we'll look forward to that.
What is the full financial impact of the workforce reduction, including severance, unemployment, contractor backfill, consulting, recruiting, onboarding, delayed work, agency side costs, third-party vendor spend, and increased operational risk?
Thirty.
Our own slide that we go.
Because it's not just, you know, there's not just like a savings,
there's a cost to firing people. So how did that offset the savings you're talking about?
So for fiscal year 20, there we are, fiscal year 26-27, it's $27.5 million in salary reductions, one-time payouts as severance.
Sorry, I don't have my glasses on. I don't think we have this slide. No, we didn't have it before.
We made it today anticipating this question. $3.6 million. and then additional salary of 19. So that's the offset. Projected annual savings just in these areas of $4.1 million. Continuing to evaluate the model's value delivery, we are in anecdote land we're 30 days in, so we need more time. We haven't even closed out one month of financials. But what we're seeing is that work is moving faster and with less people involved, which is associated historically with a cost reduction.
So how is OIT going to prove that the claim savings are real statewide savings rather than just moving the costs from OIT budget to agency budgets contractors delayed projects and future remediation work Is there something we could, is there a template or a format or something that would help
engage in that conversation? I'm happy to bring what would work for you.
I mean, I just want to understand, like, if these are thoughts that you've had in the restructuring as to how you're going to prove it. I'm skeptical. I have been a big skeptic of OIT, and this restructuring was dumped on us, and I am not going to just sit here and say that, you know, we can sing kumbaya, we've figured it all out. I want to know that what happened and all the people who lost their jobs is going to be worth it and actually do what you're saying it's going to do, which is to my next question, what independent validation will we receive that OIT's reorganization is improving the service delivery, cybersecurity, audit remediation, agency satisfaction, cost efficiency, and public value? Because we're not getting information from the divisions. They don't want to criticize OIT, and they haven't been able to criticize OIT, except for in their little survey. So how are we going to know that what's happening is actually really getting done appropriately and the divisions are actually being satisfied if it's not just a survey? And how we have an independent validation, not just from you.
So open to feedback and would love to engage in a conversation about what would better meet that independent validation. A couple of things. One SB 185, thank you. Risk reporting, now we are required to bring these risk reports to you. That is a really big shift. It's not just compliance, but actually what is the risk posture of the state and how are we working that down? That will be a huge change. Additionally, I think we can have conversations with folks. I can share with you the feedback that we get. We're doing both anonymous and name surveys. People like telling you what they think in an anonymous survey.
I remember asking Director Edinger for more detail in the surveys, and I never got it. I wanted to know what the questions were and what the answers were, and we never received that information. There's been a lot of obscurity, and I don't have trust. I want to know how we can independently verify and validate that what you're doing is actually working. I worry that that's going to – you know, we had to – we were going to do an audit on the audit after they're done with their work in 185, and we set it up to come out of, I think, the revolving fund. I wonder if this would –
That was for a forensic audit.
Yeah, I just wonder if we're going to have the money to do an independent review.
I don't know where that money would come from. It may not have to come from that, but it's some other source that is not you all telling us everything's great because that's what everybody sat in front of us telling everything is great, and it hasn't been great. Go ahead.
Everything is not great.
No, but before that, that's what people were saying.
there was a lot of things that, you know, obscuring the fact that things were not going well. I can't speak for my predecessors or what happened before. I can only speak for what I'm trying to do, which is to be honest and candid that we weren't delivering and we needed a radical change how doing it. And I would say like the announcement, while there's months of incredible intense, incredibly intense work that led to it. And it was an unbelievable, an unbelievably difficult day, particularly for those people who are laid off. Do not take any of that lightly. It weighs on my heart. And I can only move forward from here and would like to radically change.
But I had asked the question, and you were part of the team at that point, and I said, did you ever confer with any of the people within OIT on the plan to reorganization?
And the answer was no. And the answer was we did this top-down, the previous one, but I mean I'm sure that the previous reorganization had something to do with what ended up happening now. So that was a top-down approach with no input from staff and didn't have anything.
And the question I ask now is that you're going to have a new administrator, a new governor. Was there any conversation with any of the candidates' thoughts on what they thought would be a good direction for OIT to go in? Because somebody is going to take over this, and maybe this is not the way that they want to go. Was there any thought to even speak to any of the folks who were candidates who could win the race that you could get feedback from or input as to what their vision might be?
Multiple of the campaigns have been in conversation with me and other members of OIT about what we are doing, about what we did.
Before or after? Before the reorganization? because I don't think that was true because you...
After, in particular, additionally, it is, we don't know who's going to win the election. And it is not, it is, I would argue, it's not appropriate for us to, given the at least five candidates who have a really different idea of how to run government, to frame ourselves in any of their one image. We are here, and we aren't delivering for a future governor. We're delivering for Coloradans. And we're using the best we know of the evidence and good practice that has made successful technology companies and organizations around the world to inform this. Not a political idea of how to do it.
I mean, maybe there was some way that they all were on the same page, but you wouldn't know that until you told them this is how we're doing it afterwards. But I digress. Did you have a question you want to ask before? I'll let you jump in. Paschal.
Thank you, Madam Chair. I was just going to say the question of whether or not the expenses are just a shell game, like we moved them over there and now look at the savings. I mean, that's, I think, easy to address by having the full picture and showing where the costs are and where they went. And I would say, too, that we are asking as a part of our quarterly review for an update, not like an anecdotal, if you can put together some good data for us, whatever you have, that would be incredibly helpful. I would like us to try to remain flexible and also work with you to figure out what it is that we need. So I hope you engage with the committee to figure out what that is But I will say that we be we love a full review but we will be flexible as we work on trying to figure this out so I like that idea yeah it's a couple more if you'll indulge me yeah can I ask one
sure question that came up for me for the delegation I know we're not doing any
delegation of CISO operations. I just want to confirm that. Yay. No, never will we delegate security or any security related. For example, we just, our friends at CDC, we delegated web development to them. Totally a good thing for them to do. Yep. That makes perfect sense. Okay.
You're up. Vice Chair Tatone. We've got four minutes. Yeah. So how do you measure, well, I'm
sorry, let me go to another one. So what agencies have actually requested some more authority and distance themselves from OIT? Have you seen that? And what might that tell you about what you're doing at this point? I think that is delegating authority. No. You're asking who has asked? Yeah, who's asked to actually have more authority for themselves?
they're going to figure out which departments. We've got a number of agencies. Perhaps a handful. DPA.
Okay.
Mostly for operational stuff related to the state, payroll, finance, those sorts of things. DMVA, military and veterans affairs, largely oriented around security clearance requirements at the federal level. We don't have those security clearances. They do. CDC, as I said, web development, go maturity model. And then, ah, Department of Transportation for their traffic system.
Ah, a fifth one.
HICPF for the CVMS.
CVMS.
No, for the MMRS, the Medicare member.
Okay.
We delegated it.
I don't know about it.
Okay.
That's great. I have one more question because this one is an important question because we were authoring a letter to the governor telling him that we did not believe in the director to facilitate a reorganization. And then they beat us to it, and he resigned or whatever happened before the letter got out. But I'm still bothered by the fact that we had a security audit that was abysmal, and frankly, Ms. Frazier, you're still here after all of that, And that was a big part of the disappointment in the organization. And you are still sitting here and having no responsibility taken. And I think that that's really a shame for a lot of other people who lost their jobs. And you're still in a position that's higher than you were even in before. And I think that I have a big problem with that. because I don't feel that the responsibility and the... what was really taken for that critical mistake which is putting us at major risk Like you said yourself that we have hours to fix things because of the way cybersecurity is going
and we took steps backwards. We're in a worse place now, and now we're not making a big change there, and I find that that's very troubling. So maybe not a question, but I wanted to just put that on the mic.
And I'll just clarify that there was a letter, but we didn't sign it, I don't believe. And we were talking about it, but I didn't sign that letter.
It was ready to go.
Yeah.
Okay.
Wouldn't you agree? No, we won't ask a question on that. Very good. But I will say that I do want those quarterly reviews to include just the metrics that you've heard that would be interesting for us to hear about. And I think we're doing our first one in October. Is that right?
Yes, Director Thunberg. Yes, we are doing it in October.
And my ask of you, we're going to bring it.
And then, as you said, we would love to iterate. If it doesn't meet your needs, if there's something we've missed, if you want something different or more, let us know. We'd love to do it.
All right. Very good. Thank you all very much. Does someone have a question? Oh, Rep Kelty. Sorry.
That's okay. Thank you, guys.
Yeah.
That's okay. So, I think I just wanted to make a comment or question. I do know that I came into this a little later and that the audit had already been done and stuff like that. As far as I'm concerned, you know, an audit is not something to look to punish. An audit is something to look to find areas that need to be fixed. And I see it as a positive thing. I don't want to point it as a negative thing where everyone's like using it as a tool or a weapon to punish rather than we use it as a tool to find where we need to find things to fix and get them fixed. Now, going forward from that, if the items that are found throughout the audit or from the audit are not fixed and they are ignored, which it doesn't sound like they are being ignored, at least in my short time on the JTC. But I think that we should look at this as a positive thing. Let's find the errors. Let's find the problems. And let's get them fixed and plugged and move forward in a healthier environment than it was in the past. And I guess this is my two cents. I'd rather see, rather than punishing, let's just do this together as a team and figure out what we can do to fix things and move forward. Now, they don't get fixed after this, and they're ignored or shoved under the rug or hidden or whatever. That's when punishment comes in. But I think the audit itself is something, it's a tool in a positive nature rather than negative.
Thank you, Rep Kelty. Okay, we're going to let you guys stay here. We are going to move now to the Frontier AI and cybersecurity. We'll have Ms. Frazier kick us off. And just to get our time back, we're going to go for 20 minutes. So you have until 2.20 and there won't really be room for us to ask questions. but we will bring up CDEC after that and then DOC So the floor is yours Thank you Madam Chair Thank you members of the JTC
Go ahead, next slide. So the cybersecurity threat landscape fundamentally changed in April of this year. You've likely seen the news reports that Anthropic, the company behind the AI assistant Claude, announced their new frontier AI model called Claude Mythos Preview, Mythos for short. The model can autonomously find software vulnerabilities at scale and speed that previously required teams of experts years to identify. Anthropic judged Mythos too dangerous to release publicly. Instead, they launched Project Glasswing, which is a controlled program that gave access to roughly 50 major technology vendors so they can identify their own vulnerabilities in their own software and create protections for those vulnerabilities before adversaries develop similar tools. So many of the vendors engaged in Project Glasswing are vendors that we use at the state. Next slide. Yep. This matters because every previous era of cybersecurity had one basic constraint, and that was that finding and exploiting vulnerabilities required skilled humans. So even the best nation-state teams had to hunt manually and then develop exploits for flaws in software. The entire defensive architecture that we and every other organization built was calibrated to that human speed limit. The economics changed. What previously required nation-state funding resources is becoming affordable to far more threat actors. Discovery is no longer a bottleneck. Rather, the ability for vendors to verify the vulnerability to create and ship safe patches, and then for us to deploy those patches is now the bottleneck. With decades of software development behind us, we have accumulated significant numbers of software flaws that no human team ever found. Mythos and Project Glasswing have already identified more than 10,000 critical and high vulnerabilities in every major operating system and web browser. And from the time a vendor releases patches, Frontier AI can now reverse engineer working exploits in minutes. So organizations no longer have the luxury of months, weeks, or even days for installing some of these updates. Next slide, please. One more. Thank you. Even before Anthropics announcement in April, OIT has been scanning all state systems for vulnerabilities and working to address those vulnerabilities by patching, eliminating software that can no longer be patched, and overall operationalizing the patching program. Since the announcement, we've been evaluating our patching standard and are considering requiring a 24-hour patch response on internet facing systems. I want to be clear, we have not yet changed the standard language. We've circulated the plan for feedback, and OIT teams have been actively working towards that 24-hour goal with all newly announced critical and high-risk patches that impact our internet facing systems. Over the coming months, we expect Project Glasswing vendors to begin publicly releasing patches that they've developed for mythos-discovered vulnerabilities. We expect this to translate into significantly larger volumes of patches that will need to be applied quickly to reduce the risk to the state. Chinese state-sponsored actors are estimated to be roughly six months behind anthropic in comparable AI capability. So we have a narrow window before these tools reach our adversaries. The OIT Security Office will continue to identify and report vulnerabilities for remediation, both to OIT teams as well as to agency vendors and non-executive branch agencies. And if these systems cannot be patched, we will also continue to collaborate to identify compensating controls to reduce risk until they can be fully mitigated. So I want to make sure that you all understand we are not approaching this moment unarmed. We have processes, we have tools, and partnerships that we're leveraging. A few, and this is not comprehensive, a few noteworthy examples are that all executive branch agencies and many non-executive branch agencies are using CrowdStrike, an endpoint protection tool. The offering used by most is a 24-7, 365 support model where not only are we proactively using the tool to look for threats, but the vendor is also. CrowdStrike is a project glass-winged partner, meaning they receive intelligence ahead of public disclosures, and they're able to use that information, that advantage, to help protect our systems. Additionally, we use tools to continuously scan all of the technology environment to identify vulnerabilities and report directly to the teams for actioning. Again, both executive and non-executive branch agencies are included here. The tool we are using is AI powered and is both giving us basic information about vulnerabilities but it's also helping us to prioritize and forecast which vulnerabilities are most likely to be exploited. We are partnered directly with Anthropic on two pilots, Cloud Enterprise, which assists our analysts in triaging vulnerabilities and drafting compensating control recommendations. This is helping us to accelerate our patching workflow. We're also piloting Cloud Security. This is Anthropic's AI-powered vulnerability scanning tool. It was released in beta in May, and unlike traditional scanners, this tool can reason about how vulnerabilities are combined into attack chains. What this means is while our focus is on rapid patching of Internet-facing criticals and highs, and that is the correct first step for us to take, we're also aware that AI-powered attackers are now able to combine multiple low-severity flaws. into a single attack that achieves full compromise. This is a technique that is called vulnerability chaining, and Mythos did demonstrate this directly related to vulnerabilities in a browser. This means that systems with zero critical and high vulnerabilities are not necessarily secure from a full system breach. Next slide, please. Thanks. So no single tool process or partnership is going to secure all systems and state data Our approach involves layering disparate activities into a cohesive framework for risk reduction For example, while our scanning tool cannot currently tell us how vulnerabilities could be combined to compromise our systems, as mentioned, Anthropic's clawed security can. And while we are still in trial with Anthropic, we are seeking FEMA and CIS's approval to use state local cybersecurity grant funds to further develop our capabilities with Frontier AI to protect the state. And while CrowdStrike is only as effective as our deployment is comprehensive, it's not our only tool in play. As Sarah mentioned previously, we have a number of vendor partnerships that we're leveraging heavily. And one particular noteworthy example is our partnership with Splunk. They have been helping us to ensure that we're ingesting all of our logs, identifying malicious activity. I want to call out Splunk and Cisco have been a phenomenal partner, and they are also Project Glasswing partners. So while we still have work to do, and there are a few examples of the work, and that is we have a 24-hour patch target. That's the right goal. We need more automation and less manual involvement to make that properly operationalized. We may have blind spots, and we may need help from our contracts team if we run into situations where vendors are not meeting the more aggressive expectations that we have with respect to risk management in this age of frontier AI. And vulnerability patching isn't our only concern. Every automation needs an identity, every integration needs credentials, every cloud workload needs permissions. These identities often have broad and persistent access. Identity has become one of the primary paths of attack in modern cloud environments. So a priority for us is auditing regularly these identities to ensure they are not overly privileged, leading to an attacker being able to use them throughout our environment. So the one-sentence takeaway is that there is no single tool, process, or partnership that's going to fully secure the state. We are chaining together integrated defenses and prioritizing increased automation to address the multifaceted risk of frontier AI to the state. Any questions?
Yeah, Vice Chair Titone.
Thank you very much, Madam Chair. So I understand like you were involved with the Veterans Transition Program for cybersecurity but you didn't follow through with that in the last year or so. and now you're 30 members short on this department and now you're not going to have the manpower to do it? How are you going to accomplish all of these changes that are being made with all of the other problems that still need to be addressed?
Director Frazier.
Thank you, Madam Chair. Thank you, Rep. Detone, for the question. And so I would, from my perspective, I feel like I'm in a fairly fortunate space in that I've had a couple of years to work with the security office. And a lot of the work that we have done over the last couple of years is to get really clear on roles and responsibilities and create a clear line of accountability around those roles and responsibilities So what we've been able to do with fewer people has been to deliver at a much higher level. We actually have a lot of data on the maturity, the number of services that we've been able to deliver in the last couple of years. So we feel very good about where we are as far as staffing. That said, as we learn more, if we identify situations where we need more staff, we need more folks, we will ask. We will make sure that you know that we are in need.
Vice Chair Tatum.
Thank you, Madam Chair. So, you know, flaws in software are only one part of the problem. But with the increased use of AI by staff at the state, there's going to be other entry points. So how are you going to prevent prompt injection, phishing, and other stealth attacks that are happening that are within emails and all kinds of different things? The more AI you use, the more these agents can actually take those directions stealthily, and that can cause problems. How are you going to really prevent all of that? because those things keep coming up, and we don't really have a great stance at this moment, at least the last time I checked, that these are vulnerabilities that we could be facing.
Director Frazier.
Thank you, Madam Chair. Thank you again, Rep. Detone, for the question. So my experience in the security field is that while there are different threats and different threat actors, the message is typically the same, and that is in order to have a full-fledged program that protects the organization, you have many layers, tools, processes, people that you combine together in order to reduce risk. So there's no one answer that I can give you. What I can tell you is with Casey Cook here as our enterprise architect and helping us with governance, we are going to have a solid foundation that ensures that the security office isn't playing a game of whack-a-mole, that instead we are building a solid foundation upon which the services that we deliver to Coloradans and the data that we maintain is secured to the best of our ability. And again, as Sarah mentioned in her presentation, we have a significant number of large vendors that we're working with who are actively helping us to ensure that we're using their tools fully, completely, comprehensively, and that we're a good customer in delivering well, again, for Coloradans.
I see that Rep. Kelty has her hand up, and then I'll come back to Vice Chair Tatone.
So, Rep. Kelty.
Yeah, hi there.
Thank you. Yeah, I just had just a few questions. I mean, if it helps any, I know that with the more modernizing of our tools comes the fact that we'll need less people. You know what I mean? They're greater tools. They'll do more things. And that usually ends up being where we don't need as many people to do the work that the old tools used to do. So that I have witnessed and worked around in my current, you know, civilian environment. Now, if I am understanding the AI that we're using, is that all internal AI? Is it anything that's touching external outsources? And what is our cybersecurity makeup that as far as like you know what tools we using to monitor any types of holes or vulnerabilities that was mentioned by Reptutone that would be nice to know. And you can answer these, some of these now. You can get the rest of them to me later in an email. I know some of these are going to be a little more technical because I would like to know as far as like the stinging and after this, you know, scabs and scans and everything you're doing, what is the stinging timeframe? What is the – is this only on cloud systems? Is this cloud systems on-prem systems? I would like to know that. And also, what are we comparing – the tool that you're using for that, I'm not familiar with, so I just am wondering how comparable that is to the DISA standards that the U.S. government uses, because that I am actually familiar with. And then I would like to know the list of vendors that you mentioned. And I want to make sure that they're all, and I don't know if it's, and I can't remember if it's a requirement for the state, but I noticed for the U.S. government that they are all U.S. companies. We don't have any foreign actors in there that could be embedding anything funky, you know, that could be causing any further types of vulnerabilities or compromises. And I don't remember anyone ever mentioning what type of disaster recovery environment that we have. What tools are we using for disaster recovery? Do we have a disaster recovery plan set in place? And if it is, what kind? Is it a hot site? Is it a cold site? If you could expound on some of that, I'd appreciate it. Oh, my gracious. That was like 17 questions. You did a good job.
I know.
You're good.
Those are great questions.
Why don't you do your best to answer just as quickly as you can so that we can bring up CDEC?
Thank you, Madam Chair.
Director Frazier. Thank you. Thank you, Representative Kelty. I wonder if I could answer your question by circling back around with a visual of all of the tools that we use as they map to the MITRE framework to help explain where we've identified the tools that we have, the protections in place, and where we have identified gaps in our activities. So then that way we can give you the full account to all of what you asked for.
Rep Kelty, does that work for you?
Sorry, I couldn't find my button.
Yeah, that's fine. That'll work for me. I'm sorry. I think a picture is worth a thousand words, and it'll probably help a lot to see that.
So nice thinking. Okay. Yes.
Vice Chair Tatum.
Of course.
Okay, thank you. I appreciate the conversation. I appreciate the effort and what you're showing us. I am very skeptical, as I've said. I believe that there's still a lot of people who are remaining in the organization that are not sure of what's going on, what's happening, if everything is going well. I think that there's a lot of buzzwords and jargon right now. and until we start to see some results, I'm going to continue to be very hypercritical of this until I see some results. I have not heard anything about seeking out what other states have been doing and what their best practices are, how we're going to manage consultants better because we were not managing them at all from what I understand and nobody knew what the consultants were doing. And there are ways that states are taking other contractors and letting them be responsible for the contractors so that way there's an independent person that's responsible for those contractors not doing something and it's not just someone internal. We need to have a lot better thought going into what OIT needs to do by looking at what other states are doing, I haven't seen or heard any word of that is what you've been doing. I've been doing research with NCSL and I have just got it yesterday and I haven't dug into it completely, but I'm going to come up with some other best practices that I think are missing by the time we have our next hearing because this is just thrown upon us and I don't see that comparison and to see what's working. So that's what I wanted to say is I'm still very disappointed. I believe that the accountability is still lacking. I think that people still need to be held accountable and look forward to the next time we see each other here so I can ask some more questions about what's happening because I want to see this organization be successful. I don't want the next administration to have a mess on their hands. And there's quite a mess to clean up.
Director Thunberg, and then we'll go to Rep. Pascal real quick.
Thank you, Madam Chair, and thank you, Representative Titone. I am remiss, and I should have shared, I work very closely with a number of other states. The ones I work most closely with are Pennsylvania, Maryland, and Utah. I would love to talk to you about the alignment we are doing on centralized services, on open standards, on code sharing, working really closely on state-endorsed digital identity with Utah. Colorado is the first state to sign on to the coalition, leading on their work. Additionally, leveraging a lot of what Connecticut has done around standard setting, around how we do good agency support in a centralized model. So I really appreciate that, and I would love to talk more about what other things you are seeing that you think are a good fit for us. Open for that. Of Virginia.
How about Texas? Let's talk about those.
Sounds great. I'm in.
Rep. Pascal.
Thank you, Madam Chair. We did talk earlier about potential metrics to the new operating model, and I think that's a completely legitimate request, and that we figure out what are some metrics and how often of a cadence should we be seeing information. We're going to definitely see it at least quarterly. So if we want to do it more frequently, I sent it the beginning of the summer, and I'm assuming you got it all. I sent it to Dave Ettinger, but a list of all the things that will be in our quarterly reports, and this will be a part.
Okay.
But if we want to do it more frequently is what I'm saying, we can do that as well. I just wanted to mention that.
Okay. Very good. Thank you all. We are now going to welcome CDEC Snowflake Program. And while they're walking up, I just want to mention that if anyone ever wants to chat with any of us, feel free to reach out send us an email and get on our calendar so that we can meet with you before because we never are trying to do really gotcha questions But I wasn able But I also I just want to let you know feel free to reach out to our offices if you like to meet with us. So very good. So I've got, I'm going to ask, I've got Rose Barklow for questions only. Is that human here? No, up. Okay, very good. So, okay, we don't know your name, so whoever would like to start, if you could just start by your name, what your position is. I'm going to just say we've saved about 15 minutes for this part, and then we're going to ask you about 10 minutes of questions. So we only have like 25 minutes, and I'm sorry, But I really appreciate you guys being the guinea pig, and you're very brave. Thank you for getting your data in, and we will be kind. So I will turn to you. Thank you, Madam Chair.
My name is Natrice Bryant. I am the Deputy Executive Director and Chief Administrative Officer for the Colorado Department of Early Childhood. Some of you might remember me from other positions that I've held at the state. I've been around for about 13 years. And we also have Sabina Anderson, who is our Chief Product Officer from CDEC, and then we have our partner, Sarah Senna, who is the Data Architect. She is with OIT at our table today. Thank you for the time. As you all know, at the Department of Early Childhood, we believe that technology is not only the most effective when it strengthens our partnerships, and it makes it easier for people who we serve to have access to our programs and services that they need. Whether it's the family enrolling in universal preschool, a provider managing enrollment, or a state team using data to improve our outcomes, the user experience is our primary concern. Today, we will provide you with an overview of how leveraging Snowflake Cortex to advance the universal preschool program has been beneficial to us. By bringing automation and AI capabilities closer to our data, this work is not just about technology. It's about our own sake and making sure that innovation is possible for our families and children in Colorado. We're excited to share with you the partnership-driven approach. We've been working with CDS. We've been working with OIT and in partnership with several other organizations, as we know that universal preschool is a primary concern as we move forward with the state. So I'm going to go ahead and turn it over to Sabina Anderson, who is our Chief Product Officer for CDEC.
Thank you. Thank you, Administrator Bryant. Ms. Anderson.
Thank you so much, Madam Chair. My name is Sabina Anderson. I'm the Chief Product Officer at CDEC. I first want to start off with saying thank you so much for the opportunity for us to come and share here on the UPK Post Application Income Audit. As most of us know, AI is moving incredibly fast. And as much as we want to leverage new technology to become more efficient, we also want to make sure that we are taking in careful consideration on how we're deploying new technology, learning from it and also deploying it in a safe way. So universal preschool is one of our key primary flagship programs. And within universal preschool, families can apply for 15 hours of preschool. Qualifying families can also apply for up to 15 hours supplemental hours. However, one of those qualifying factors is income. And income is a time-consuming process you can verify. So there are many ways that governments try to solve for this problem And what we often see in government is ask all the questions up front This puts a lot of burden onto families. So we kind of took a step back and asked, well, what is right for Colorado families? So looking at our data, about 59.5% of families apply to universal preschool via their mobile devices. And as you can imagine, having your income documents readily available on your phone is not always the case. We also took a look at having that kind of hard requirements on income documents up front and, you know, realizing that having that type of hard requirement up front leads to an increased abandonment rate of application. So we're thinking, okay, so what if, you know, we have this optional? But as probably many can imagine, busy parents having something that is optional and coming back to it later, it means that you might forget. So it's not an easy challenge to solve for, but we decided we wanted to take a little bit of a different approach to how we want to solve this. We want to take on the burden, and we don't want to put this burden onto families. So instead of adding more steps to families, we saw, well, how can we take on that complexity? So the principle was we should work smarter so families don't have to work harder. So this particular solution has three core concepts to it. Automation, AI, and human control. On the automation part, we really decided to handle the repetitive rule-based type of work that is coming at us at large scales. This is where automation takes place. We know that something is, you know, a data field needs to be moved from one location to another. Nothing needs to be manipulated. It's very expected, predictable in how this is going to show up in our data. where the AI component came in. It's actually in a very, very narrow sliver of work, and that is looking at the income documents the family have uploaded and to be able to retrieve the right information from that income document. So we're leveraging Cortex AI and the text extract features within that, all living within our snowflake environment. And then the last part of this is human-controlled. One of the things that we really wanted to make sure that we are tackling our first AI use case that is in the technology stack. We didn't want to have this as a high risk. We didn't want to have external facing. We didn't want to introduce a lot of complexity into something that we really want to try to understand to the core at the very start. So this tool is very much focused on only being towards UPK specialists. And the AI feature that we have is really only about extracting information, information, not to interpret or manipulate any data, the determination of eligibility is still within humans. So automation and AI is really about pulling data and then presenting it to our UBK specialists to be able to work faster. So in the next slide, we'll see what the high-level architecture looks like. So UBK currently works as the operational database within the SAS provider that we have been working with from the very start. With the SAS provider, we also have robust APIs, and we've been integrating with these APIs for quite some time now. In the last two years, CDC has made some really significant investment into our data ecosystem, and this has really been with a lot of partnership with OIT, setting those kind of enterprise standards governance making sure that we doing things in a secure way that are able to scale So we have multiple programs in here already and we have a large amount of data coming in from UPK into our Snowflake environment So adding one more endpoint was not a large lift for us. So this was really towards that document retrieval. So the data processing is really taking those documents and it's about extracting the right level information on those documents that are like the W2, pay steps, those kind of documents that we often see. This all gets combined into two different types of dashboards. One is kind of like an aggregate of how we are understanding our income documents coming in for various different applications and then we also have an interactive data tool. All of this also, or the interactive data tool also lists within Snowflake on the Streamlit application. So in the following slide, you can see a screenshot of the detail viewed of our follow-up dashboard. So I'm going to talk about what here is AI versus what is automation. There are two different components here that are AI. In the very top left, you'll see that there is a doc confidence that says excellent. That means that our Cortex AI were able to find a document, and the confidence score associated with it was excellent. The second part was the calculated income based from this type of document. Because sometimes we see pay stub, sometimes we see W2, so income is sometimes reported on like a biweekly or sometimes an annual schedule. The rest here is actually going to be all automation. Pulling data from the operational database into this dashboard, because this is really all the data points that a specialist needs to be able to do that determination. So we're doing calculations here to see if there's any follow-up required from the specialist. So since we also have that operational dashboard, we also created that aggregate dashboard to be able to see our findings. So in our data dashboard, we'll see a table of dashboard that gives us visibility into how families' application submission is at a larger scale. And this is really for us to understand how we can improve service delivery and the technology experience. I can imagine that you probably are wondering, well, did this implementation make a difference? So before this implementation, a specialist could reasonably complete about 200 to 300 applications per year because pulling data from all these different systems and to be able to make the calculations was very time-consuming. When we deployed this tool, we pre-processed about 10,000 applications just for this year alone. So now with this tool, a specialist can go through about 175 applications per week, not year. So this means a UPK specialist is no longer buried in finding documents and data, and their expertise can now be focused more on helping families. And this resulted in about a 40x increase in throughput for us. And to be able to get to a 40x increase in throughput, this type of implementation isn't actually that far-fetched. Because we've set up our department to work in a modular and angular way.
Ms. Anderson, we don't have a copy of this presentation, so if you could send that to us, that would be amazing. I'm so sorry to interrupt you, but we're also a couple minutes over, so maybe you can fly through the last few slides, especially if Sarah's going to say anything, because we only have 10 minutes for questions.
Absolutely. I'm almost done, and so yes, this was true Agile in practice. Short discovery with our UPK specialists. We were doing two-week development sprint cycle where we're doing feedback sessions with our specialists because we recognize that hard requirements up front is actually not the way that we learn. We learn through iterative feedback from our users. I'm going to talk about the next slide real quick in that case, and that is towards our call story because all technology implementation, we want to understand the ROI. And cost is something that is very important to understanding that type of investment. So when we first had this problem statement, I want to give this as a comparison point. Our original estimate for this type of implementation was about $100,000 if we went with a vendor. But because we have matured a lot in our technology stack as well as our team capacity, our implementation cost was about $5,600. dollars and when we were seeing the peak load of processing 10 000 application the cost for that was about 12 a month and that was peak load our ongoing o m is in the single digit dollars per month and this really was because we have been able to move in more to a modular perspective have more capacity in-house and really maturing not only cdc but the partnership that we have with Thank you so much.
Thank you so much. I wrote your name down. Senna.
Ms. Senna. I don't know if I have too much additional to add. I do believe the last slide is the overall architectural design. Is that one? We have in the appendix. Oh, in the appendix. Yes. There we go. It's a very, very basic pipeline. Mercifully, the vendor has an API where we were able to retrieve the documents. We were able to call the documents from the vendor's API and place them directly into Snowflake. There was a secure connection. We weren't putting them in a file share. From Snowflake, we were then able to process them using Snowflake's AI capabilities and process that data into JSON and then flatten it out into a table where it actually gets displayed in the user interface to the end users. So it made it a very quick development process. We used out-of-the-box AI. We did not train a language model. We used Snowflake's base document reader, and it did an incredible job. We were able to type the various documents that we received from families to 85% accuracy. So, yeah, I think I'll leave it at that. And if you have additional questions, I'm happy to answer.
I think you guys were the perfect start because I think that your use of AI and your explanation for what it's doing, the human is making the decision at the end. This is just data extraction. It's speeding things up. Your throughput goes up. Your cost goes down. That's amazing. $100,000 versus $5,000, I just want to say thank you. Thank you, OIT, for being able to do that, and thank you guys for being so frugal. So nice work. Vice Chair Titone.
Thank you, Chair. So you still have the vendor, and you still have to pay the vendor yearly cost for that. And on that slide it said that the vendor cost was but you did it in for So I a little confused about what that how that cost actually works and what the ongoing cost is Ms Anderson Thank you Madam Chair Thank you so much for the question.
So our original estimate to going with a vendor was $100,000, but we said no, we're not going to do that. So we are not paying a vendor to support this work. This was all done in-house within our own infrastructure. So the cost that was towards the implementation was $5,600, And the majority of that cost was actually to get a data scientist from OIT to help us understand how do we get to acceptable accuracy in our testing. Because our testing was very rigorous to be able to get to production ready with the approval of OIT as well. And the ongoing infrastructure cost is in a single digit for O&M.
Vice Chair Tatum.
So Snowflake is something you have a subscription to anyway, so it's not an additional subscription you're just using, and you're nodding your head. So I'm going to ask one more question. With this kind of tool and having an online portal where people are putting information in, is this tool set up to prevent the kind of prompt injection which could go into the AI and wreak any havoc internally and do you have the ways of preventing those kinds of incidents?
Ms. Anderson.
Thank you so much, Madam Chair.
Thank you so much for the question. So this is not external facing. There is no way to put in a prompt for a user. The prompting is done on our back end for developers who have pretty strict permission sets. So there is no injection into our code base. This is all about understanding what data is being presented. so users are not able to interact with the AI components. Okay.
Brett Pascal.
Thank you, Madam Chair. So what does the interaction with the users look like? Is that a separate piece of software? Are they handing someone a stack of papers? Or what does this look like? Ms. Anderson.
Thank you so much for the question. So the follow-up dashboard where all the information was presented to the UPK specialist, that is the result of automation and AI. So two fields of that output is generated from AI, but it's not a standalone AI tool that people are interacting with. There are small components of AI that helps facilitate the user interface that is that follow-up dashboard.
And Ms. Senna. Thank you for the question. Following up on the entry path of the documents, which she didn't cover, she talked about the AI process to piece. Our software vendor, BridgeCare, has an application process that UPK families fill out, and the income documents are provided as part of that standard application process. So they submit PDFs, pings, JPEGs of their income documents. Sometimes they give us Medicaid, TANF, SNAP documents that prove that they're an income-eligible program already. Those documents are stored on the BridgeCare vendor side, and it's their user interface that captures those documents and saves and stores them. And we retrieve them via API into Snowflake.
Vice Chair Titone.
Thank you, Madam Chair. And there no way that those documents PDFs and JPEGs and things like that could not have malicious embedded pixels or instructions for the AI that could be picked up at any point Because I mean they getting very clever these days Ms Anderson I think that if the way that we have structured our testing is to look for a specific field
and have that be part of our testing suite. And if that doesn't happen, we don't do anything with that data besides saying there is not the right level of data we're looking for here. Therefore, this needs to go to a human. I don't know if that quite answers my question. And maybe that's something that you can take a look into.
because what I've been hearing in some of the AI space is where there's a lot of things that are embedded into files that are not visible, but when they're fed into an AI, there's an instruction actually there that you can't see. And, you know, I just hate to have that kind of malicious activity be put in there, So if there's some penetration testing you can do to be sure that that is something that doesn't muck up the system.
Ms. Bryant.
Thank you, Madam Chair.
Thank you, Representative Titone, for the question. What I'd like to do is offer us to get back to you on that information. We can go through with our data team and get some information on the quality control process that we have in place, and we can provide that information back to you via email if that will work for you.
That's great.
Okay. Thank you. I appreciate that. And I just want to do a shout out. I see Dr. Lisa Roy here. Thank you for being here. Seeing no other questions, we will dismiss this panel. Thank you for your good work. You are just a perfect example for our very first look at AI. So thank you for being great sports. Excellent. We are going to now call up DOC to speak about the recidivism program. okay okay we have looks like we have Mr. Kaylee on Zoom do we have Sam Zurnaya also for questions only yep I'm sorry are they up there and I'm not seeing it yes I am oh I see you okay great lovely okay just to calibrate here we've got about 15 minutes of you talking and then 10 minutes or so of us asking and you answering questions so if you could please time yourself that would be amazing and I will turn it over to whoever
would like to start first, if you'll just say your name and who you are, then you can go ahead and begin your presentation. Thank you, Madam Chair. My name is Ed Kaley. I'm the Director of Business Innovation for the Colorado Department of Corrections. And I assure you, if you were looking to gain back some time, you're going to get that now. So I don't have a formal presentation prepared but a quick review of Recidiviz and our partnership with that group Recidiviz is a nonprofit technology group who works exclusively with corrections agencies They provide a broad suite of tools and implementations to agencies. They provide their services at a flat fee per year, and agencies are able to prioritize different systems and developments with them. In DOC's specific partnership with Recidivis, their true specialty is data ingestion and consolidation, taking multiple different sources of data, multiple different systems, which the department, as many of you well know, has multiple, multiple different systems. We're halfway or three-fourths of the way into completion of our newest implementation, but we're still running systems that are up to 30 years old as well. And there's just been no way for us to consume that data in a way that we could make real-time decisions from the data. So we're working with them to create a more effective data warehouse. Um, um, we're, we're using them to develop dashboards for, for staff in different roles. And, um, we're also using them to, to help us create, uh, dashboards, much like you would see a tableau on public facing websites and, and those types of things. And we're using them for policy research, um, to help us make informed decisions on ways that we can optimize programs and policies that we have in place, such as we use them in recent months to optimize our Achievement Earned Time program, which affords up to 120 days of Achievement Earned Time credit for completion of specific programs and milestones. We were severely under utilizing that program. On average, recipients were only getting about 20 days of Achima Earn Time credit over the course of a full sentence. We anticipate increasing that to an average of 60 days initially with the recommendations and the updates that we've made thanks to recidivists. They're also helping us with workflow efficiency and automation. We're going to be using them to update our intake processes through the Denver Reception and Diagnostic Center. That effort's about to kick off. And we're working with them to develop tablet and web applications for both residents as well as parolees, a tool that they'll be able to use from the beginning of their incarceration until the completion of their parole status at the end of their term. In terms of AI, we're using very little AI in any of our recidivist tools. The AI tool that we're in the middle of testing and implementing is a meetings transcription assistant that allows our case managers and our parole officers to conduct a live interview with their clients, whether that's a resident inside the facilities for case managers or a parolee out on the streets. And the tool will provide real-time transcription of that interview. Our goal with that is to create significant efficiency for our staff and allow them to better focus on the work at hand and helping their clients achieve the outcomes that they're trying to accomplish without spending the entire time of the interview documenting and transcribing. So this tool will provide real-time transcription at the conclusion of the interview. It will then provide a summary and case note recommendation for the staff who will then use that case note summary to send back into our existing EOMIS database and document that. providing one tool for the parole officer or case manager to work from and not require them to bounce between multiple systems. That is a very, very brief rundown of the work that we're doing with recidivism.
And I welcome any questions that the committee may have. I see Rep Kelty has a question. Rep Kelty.
We have a question. Thank you. I guess my question, Mr. Kaylee, is back in this last year, as you know, we went over this software where it had a 98 percent error rate. That was the risk assessment software that was putting dangerous criminals back on the street. and that was a whole thing. We've now sent stuff using that, re-evaluated, and found that it was an input error, that there was no second look, there was no real management of the software and the inputs and the data that was going into it, which was basically putting dangerous criminals back out in the street, at the 98% error rate. So with the recidivism that you're talking about and the softwares that you're using, what are the checks and balances that you guys have regarding these softwares, the input, the output? How are you managing your checks and balances with that? And is there any type of like an audit plan? Is there also one of the recommendations that we gave was that there would be a second look, that there would be before something was actually implemented as far as that could impact, like you said, their time that they've done, you know, the earned time, that's what I'm sorry, the earned time or being able to put someone back on the street or change their input and output of whether that the individual, the inmate, their outcomes are. What is the management and what is your management plan for that? Because I'd really hate to see something like this have, again, the same impact that that last software we had that had a 98% error rate. So tell me what your
safeguards with all of this is at at this point and what is it going forward? Mr. Cayley.
Thank you, Madam Chair, and thank you, Rep. Kelty, for the question. I really do appreciate it.
One of the biggest advantages of recidivism is that they are very well established. They are the majority of them are former Google and other very high level developers. This project for them started in house at Google as a part time project they were allowed to do And eventually a large group of those individuals decided to branch out and do this exclusively we're actually using recidivism to do evaluation of those risk assessments that you're talking about rep kelty so that we can get a better grasp on what may or may not have been accurate and put a better lens on that 98 percent figure that you have that 98 percent figure doesn't take into
consideration the actual responses and outcomes and whether or not any differences that were identified would have actually resulted in a different score. So that is an evaluation we're taking right now with recidivism on those risk assessments that you were talking about. In terms of outputs and what we're getting back from our recidivism, we do comprehensive testing with recidivism. We're very early in this partnership and tools are not ruled out until they've been comprehensively examined and evaluated to ensure that there's 100% consistency between our systems of record and the output that we get from the recidivism software.
I have a follow-up, if that's okay, Madam Chair.
Of course, Rep. Kelsey.
Thank you. I guess my concern is that I want to know what the checks and balance, like what is the internal DOC management or second look at the data going in, the data going out, making sure that everything is going to be correct. Because even if the last software that had the 98% error rate was, you know, depending on the margins of that, this still caused us and caused us to have to relook at, I think, was over 1,700 inmates and in cases where people were let back out on back onto the street. And some of those actually caused, well, one of them actually murdered an individual in Denver. So I'm trying to prevent all that. So I want to know what the DOC's plan is, whether the software does this or that. I want to know what your plan is as far as managing it and doing a checks and balance of your own to make sure that something like this never happens again. Mr. Cayley. Thank you, Madam Chair, and thank you, Rep. Kelty. No, I totally understand what you're saying. And the department is doing comprehensive testing before any tools are rolled out and implemented. We are using a trusted user model with Recidiviz where we roll out the initial mock-up to one or two individual testers. And we go through numerous rounds of testing with those individuals, identifying where the software is correct, where it may not be grabbing the correct data. And it's very iterative. And then once we have reached a point where those initial trusted testers are reporting 100% accuracy, we're expanding our trusted tester user group to additional users and going through that exact same iterative process. And until we get to a point where we have tested a sample large enough for us to say with confidence that we getting accurate and perfect results as expected from the software So we not rolling anything out until we have reached a point where we are 100 confident that the output is accurate and is what is meant to be collected. Okay, thank you. And I just want to, I won't be there next year, but if you could provide the committee some sort of a report and audit of your own internal audit of that, I think that would do you and them a good service. Yeah, agreed. Thank you, Rep. Kelty. Vice Chair Titone. Thank you, Madam Chair. I'm looking at the Recidivist website, and they have a set of products here. What products are you currently using in that list? Mr. Cayley. Thank you, Madam Chair, and thank you, Madam Vice Chair. Our current live products are, we've used them for our policy analysis, as I said, for Achievement Earn Time. We're now working with them for classification and custody level analysis. the meetings assistant tool that I was discussing that's currently going through trusted user testing with a group of, I believe now, five parole officers that are testing that tool. We've begun the process of mocking up some of the dashboards and tools for our staff to begin using. But as I mentioned, we're very early in our relationship with recidivism. So we do not have a large number of tools that are actually live and in place. Eventually, we expect to use Recidivist to help us build out our Parole Assessment Center program that I'm sure several of you have heard about. We're working with them to partner with the parole board to build a more comprehensive tool for the parole board to use in making decisions and promoting greater alignment between the Department of Corrections and the parole board. And we have taken the tablet application for the residents live. It's a very limited state right now. It provides opportunities for programming and information on achievement earn time for residents to review and get familiar with. And we also have an expanded version of that tablet application live at three facilities, which is expanding what information is available in that tablet application. And we'll be adding milestones and sentence information for residents for them to look up on the tablet application. In terms of things that are actually live right now, that is the full list. Okay. Thank you. Vice Chair Titone. Thank you, Madam Chair. Mr. Kalia, so just because I want to do my own research on what you're using, and I'm not exactly sure what products they are specifically. In the list of products, supervision assistant, facilities assistant, case planning assistant, tablet app, policy forecasting, which ones of those? I'm assuming tablet app you said you're using. I identified that. Out of the other ones, what else are you using out of those so I can do my own research on those applications? Mr. Cayley. Thank you Madam Chair and thank you Vice Chair Tatum We will be expanding to all of those different areas The supervision assistant as well as the facilities assistant those include the meetings assistant tool that I was describing to you. Facilities assistant will be for case managers inside the facilities while supervision assistant is for parolees out on the streets. We're rolling out the supervision assistant first, starting with the meetings assistant tool. Thank you. Vice Chair Titone. Thank you, Madam Chair. You said that you're not using AI except for meeting transcription. I was at the doctor's office and I said, please don't use that because I don't want them to use it because reviewing your notes later, is your memory that good about what was actually said when you're reviewing what was in those notes? so I'm not very keen on a lot of these tools at this point. But it seems to be from the website of the company that AI is like everything that every product uses. It's not just transcription. So are you using these tools and plan to use these tools to determine whether people are going to be ready for parole or not And how can you, you know, we had a law that says these kinds of tools need to be used with supervision of people because there's bias in them. How can you be sure that this is not biased? And how can you be sure that this isn't just a minority report kind of product that's going to just keep people in jail and let certain people out? and supervision becomes something that becomes an afterthought. Mr. Cayley. Thank you, Madam Chair. Thank you, Vice Chair Tatone. Totally valid questions and totally appreciate the concern. We've had the same concerns. What I can tell you is that we've been very clear throughout, and our partners at Recidivates have also been very clear, that this is a human-in-the-loop system. No decisions are made by the system. No outcomes are determined by the system. The system looks at the data to help identify individuals that meet specified criteria defined by the department. But ultimate decisions for parole still lie with the parole board. Classification decisions still lie with our case managers within our facilities. the software will not be making any decisions for any individuals under our care or supervision. And Vice Chair Tatan. Thank you, Chair. I understand that, and I think that when you get a couple good results and it says, well, yeah, this is a good result, and now it's not making the decision for the human, but it's influencing the human to make that decision because, hey, it was right nine out of ten times in that one tenth time that someone said, oh, well, I don't need to really put a lot of thought into this because it's been right nine out of ten times, so the tenth one's probably right too. This is what I'm worried about with these kinds of systems. Maybe it's not nine out of ten. Maybe it's 99 out of 100. But still, there becomes complacency when these kinds of tools are used and the human element becomes less and less oversight and more of a rubber stamp. And that's what I'm worried about with these kinds of systems. So. I think that this is something that next year's legislature should be looking at and seeing if these are the kinds of systems that are ready for prime time and should be used or these should be used in a more limited capacity. Because I don't think that most of these AI tools, especially if they're being run on frontier models, we're seeing very specifically a lot of these frontier models are developing racist tendencies. And there's reports that are in there, and if these are the underlying frontier models that are running behind the scenes in these AI, we've got some serious problems with what these results could be. Okay. I've got a number of questions. I've got a lot of them, so I'm just going to ask three kind of big questions, and then I'm going to email you the others. if that's okay. So do supervisees, we talked about the tablet form. So I'm assuming incarcerated folks know that you're using this information, but do parole supervisees know that you are using this information on them. Thank you, Madam Chair. I'm not sure I fully understand the question. What information specifically are you talking about? When the meetings assistant is used, they're informed that the meeting will be transcribed and the officer will review that. Outside of that, all of the data being used is the same data that already exists in all of our systems. Recidivates is just bringing them together in one place. Okay. So it's taking a bunch of data, putting it in one place, and at some point the people do actually know that you are getting all this information. So we kind of talked a little bit about this, but if some of the data that is flagged is wrong, like an old case record, misclassified medical condition, what is the correction process? And then does an erroneous flag have any procedural consequence in the department? Like if you get a risk flag, do you get an adverse hit on your parole? Thank you for the question, Madam Chair. Obviously, whatever is in the data is what our users rely on. Recidivism, one of the activities we're undertaking with them is an evaluation of our data and looking at that data over time to see whether we can see instances where situations like what you brought up. up have changed where a flag may have not been there and then suddenly was so that we could review that or a case where a flag was there and then wasn't. So those are some of the reviews that we are undertaking with our partners that recidivize to try to eliminate those data gaps that you're describing. Okay and we did talk a little bit about this also but I feel like I appreciated all that you shared. It would have been easier to have seen it in writing, honestly, just to be able to take away from it But I will say given the regulatory complexity with four different federal compliance regimes layered into the one recidivist platform, is there an independent third-party security or compliance audit that happens on a recurring basis? Does the legislature or any oversight body receive any reporting on the system's operations and outcomes. Thank you, Madam Chair. I would defer to OIT to speak to, they work directly with recidivism on the technical aspects of the system, whether that third-party evaluation is happening, And I'm happy to touch base with our partners and friends at Recidivus to identify any reports or regulatory reports that have been completed and would be happy to provide a comprehensive summary of our partnership with Recidivus to the committee in writing. That would be helpful. I just want to confirm then that there is no oversight currently. Like we're not getting reports. No one in the legislature, I think, is getting any information from this. Is that correct? Madam Chair, to my knowledge, that is correct, that there's no reporting directly to the legislature about our partnership with recidivists. Okay. That's going to be a bill. All right. Eligibility recommendations. What specific eligibility determinations does the platform recommend on? Um, is it parole, early release, admission into the program, custody level, eligibility of benefits? Can you help me understand, um, what that's, what that definition means in the way that you're using recidivism? Thank you, Madam Chair. As I mentioned, we're very, very early in our partnership with Recidiviz, and at this time, we don't have any tools in place that are making any types of recommendations on outcomes for any of the residents. The tools we're focused on right now are the resident application to provide information to them, the meetings transcription tool, and our policy analysis. But at this time, we don't have any tools in place that are making any type of recommendations to our case managers or role officers. Is there a plan to bring any of those on in the near term? Thank you, Madam Chair. Yes, there is a plan and intention to bring those tools on to help us identify markers that have been achieved by individuals and determine when they do meet criteria for any of those things that you mentioned, whether that's parole, eligibility, program criteria for job and program placements, any of those types of things. we do anticipate putting together tools that will help us identify when criteria are met and develop eligibility lists. Okay, it would be super helpful if you could include in your written report to us just what is coming next and in what order that you might anticipate rolling some of those out It would be really good for us to understand that My last question is, if a particular field gets flagged, what does that do? Does that either speed things up or do you anticipate that that would slow down or people would be able to make denials or extending of sentences or changing early release? Is that kind of how you envision this going? Thank you, Madam Chair. So recidivist won't be changing any of our data. It'll be adding case note information from the meetings transcription tool, but recidivist reads our data. It doesn't produce our data. So it would find that that flag checked in our systems of record, whether that's EOMIS or DCIS or any of those systems. But particular things being flagged in our data could mean things like ineligibility for certain programs or eligibility for some programs based on flags that are marked. But again, recidivism is not going to be marking any of those boxes. Those are maintained in our systems of record. Okay, and a human, I assume, puts those flags in place. Okay, great. Thank you. I will email you the rest of my questions. Vice Chair Titone. Thank you, Madam Chair. So you mentioned, and I was doing a little research about Recidiviz as a nonprofit company. Is the software open source? Because one thing, I was trying to think of an example of a nonprofit company that turned into a profit company, that's CHED GPT, I don't want this kind of program that would have this kind of tie to all of the states, doing this kind of recidivism, turning into a profit company and then partnering with prison industrial complex. So is the software open source at this time? And if not, do you have any idea why it wouldn't be? Mr. Cayley. Thank you, Madam Chair. Thank you, Vice Chair Titone. It is my understanding that Recidivist does operate with open source code, but I would like the opportunity to verify that before saying that definitively to the committee. Great. We appreciate you checking into that, Mr. Cayley. And again, feel free to just include that in your written report to us that will make public so people understand as a way of, you know, transparency what's going on with DOC, using this AI program to do the things that you've said it would do. So you can just put that in there. I don't see any further questions, so we will go ahead and dismiss you. Thank you so much for joining us, Ms. Naya and Mr. Kaley. Thank you, members of the committee.
Okay. All right. We are going to move now into just some committee discussion about state agency AI. We've had a couple of committees come in front of us, And I would like to figure out what kind of a rubric we want to use as we evaluating these I think it only fair that we use the same rubric for all the departments so that people know what to expect We appreciate, you know, DOC and CDEC being here today as guinea pigs, but I would like to give more information to folks as we move on. So I'm going to pass to Ms. Falco to kind of begin the conversation here.
Samantha Falco, Legislative Council staff. A couple months ago, the committee had, like, started this process by sending out a form or table for state agencies to fill out with the help of OIT. They would kind of ask for a description of systems, if it makes any consequential decisions, who the developer and employers are, who the person overseeing any consequential decisions might be, or other information, if there's any biometric data or identifiers or personal data used, sensitive information, and surveillance. So that's where we kind of started. And then with ongoing conversations with Senator Marchman, we've thought a better system for this would be to have a sort of survey and rubric that could work together to have a consistent analysis of AI systems used by the state. And so we have, through the state legislature, a wonderful technology fellow, Adiba Hari Vivek, who will be helping us with this to kind of structure what kind of information the committee is exactly looking for, whether that be focused on cybersecurity aspects. Or we're just trying to kind of clarify what information the committee is looking to evaluate AI systems on. So I'll pass it off to Diva Hari to ask some clarifying questions.
Good. Would you mind coming to the podium, Ms. Vivek? And you can introduce yourself, and we would love to hear your input. You kind of saw us go through a couple. We didn't have similar – we had kind of similar questions, but they were kind of different, and we know that's how it's going to be when we evaluate the department. So thank you for being here. That's it.
You're on. Awesome. Hi, Diva Hari Vivek. Science and Technology Policy Fellow with LCS. I don't have much to present to you. I'm more here to ask questions about the committee's interest in this rubric. When Sam reached out to me and we started talking about this, she sort of gave me an overview that the committee was interested in exploring the idea of this rubric. And I went ahead and did some digging into just existing frameworks that are out there regarding that do things from classifications, so helping with inventorying products to risk management, which is identifying, categorizing, and mitigating risks, transparency, sector-specific guidance, which I know 1889 outlined specifically, and I think we're waiting on AG rulemaking for more on that. And then this morning she mentioned the committee's interest in cybersecurity and data governance as well. So I think between now and when we will talk about this next, within which I hopefully have more of a concrete rubric to propose to you, I think I'd just like to hear more about it. the committee is looking for and sort of what are the goals of this sort of evaluating framework.
Yes, very good. Thank you so much. I'll start with Vice Chair Tatone.
Did you want to raise your hand? No. Oh, I'm sorry. I'm going to tell you what I think. I've got some questions that I kind of wonder about for everyone, like human in the loop. I want to make sure that the human in the loop is actually not necessarily an IT human, but a subject matter expert in the department. And that was something that you talked about, too, I think. So definitely human in the loop. Independent audit and oversight, we would like to understand what is the oversight mechanism. You know, we're hearing from DOC they just rolled out this AI thing that is now going to be used to determine early release, parole dates, and benefits. And that's big. And so we need to figure out what oversight exists or is lacking. Individual notice and how do people cure the issues. If we have an error in the medical condition or state or benefits, how do we figure that out? For data management, like I had a specific question about recidivism because my understanding is they're trying to get demographic data to understand our prison population and our parole population. So that should be somewhere. That's transparency. Just a full picture of what the system is. We got it, I think, from CDAC, but I don't believe we got it from DOC. But Mr. Cayley is going to work on it. So it would be great to see the full scope of the system. Yeah, just like what is it? Like he said, we're going to implement all five of these. I want to know when, what you're doing before then, you know. What's the process to roll out for early release? What is that going to look like? A couple other data things. There is data that's more highly sensitive than other data. We chose a couple today that were pretty highly sensitive. But we'd like to understand what type of data we're looking at. Do we have HIPAA? Do we have FERPA? Do we have, you know, what are we looking at in terms of the actual data? And then we'd also like to understand if there's a breach, how is that handled? How long and how do we hold the data? and then what is the data deconstruction process? How long are we keeping it? We can't keep it forever, so how long and then will it go away? And then any compliance-related things. So that will vary by department, and it's a little harder to do. But HIPAA and behavioral health data would have to do with like the DOC stuff CDEC was more about income data I don know if that has FERPA in it or Child Find data I just don know what in that data But that's what we need to know is what specific data is being used or added or transcribed. And I think that's it.
Vice Chair Tatone.
Thank you, Madam Chair. The one thing I'm concerned about with a lot of these AI tools we've seen is the cost of entry has been essentially free or very reduced because they're taking a lot of that cost to get people to adopt it, and then once you adopt it, you're basically locked in, and then the costs really start creeping up, and that becomes a real burden on our budget. So that's something that I want. I mean, there's no crystal ball to know how much the costs are going to creep up, but we're seeing a lot of the costs for a lot of these AI tools to start to creep up. And then just risk mitigation and making sure that when these tools are evaluated, as I've been trying to ask questions about, is we have to be on the cutting edge of what people are seeing when it comes to these attacks and how they're using AI to circumvent the safeguards that are in place. So how are we going to keep up with the evolving threats that are happening with a lot of these AI tools, whether they are front-facing on the public where you can inject in through there, or, as I said before, if there's a file that has some kind of metadata that's a prompt injection or something that's not seen, how are we setting things up to be sure we are doing that? because we have a responsibility to the people for going to use these tools, that they are safe, and then are the vendors going to be on the hook for fixing those problems? And then the other question I have is about, in addition to the cost, What is this going to do to adopt all of these AI tools to the cost of our cybersecurity insurance? As we have more risks and more methods and more ways of problems evolving, what is that going to do to that cost? Because that's not something we think about right away. We think about the cost of the product, the cost that we're saving for the people and the employees with their time, but how is this going to affect the premium cost for cyber insurance, and are we going to have to find another way to mitigate a crisis that might happen, and what is the cost of using this tool if a crisis does happen where we have a significant breach? And if these tools do go down because the company that we're using and we're depending on has to shut down their product because they have some kind of hack or incident, that cripples the product that we're trying to use for people. and how are we going to have a contingency to be able to provide services for when these products may be unavailable suddenly if there was some kind of problem So these are the kinds of things that I think we have to think about When we control our own software, we have that control. When we're relying on other companies to do things, then we don't have that control. And then we're relying on that, and we're kind of stuck. So these are the kinds of things that I'm thinking about as we move forward, because a lot of people are relying on our services, and we don't want to have those services be.
Ms. Vivek.
Yes, thank you. So this was all very helpful. This is very helpful because just sort of on our end, too, I'm working with LIS to develop sort of like a responsible use guidelines and like AI policy that the nonpartisan agencies can adhere to, and so we're asking very similar questions. So this is definitely that work will, I think, definitely contribute to what we're trying to do here. Could I maybe get some more clarity on what you intend to use the rubric for? I asked Sam if this was about compliance to 189, but it sounds like it'll be perhaps larger than that, or perhaps it'll help the committee with like considering like future procurement of AI tools. So I'd love to hear more about what you intend to use this for. Yeah, I do think it is for procurement, but I feel like we uncovered a big thing today in that DOC is planning to move all of our early release paroles and, you know, figuring out where people go in housing based on this AI tool.
There should be some oversight. And I think that might come up, like, in our rubric. If we have a rubric that says oversight, audit, there's nothing, that's going to be a place that we need to probably put something in place so that there is some oversight. The right person to do it is I don't know. Sure. Yeah, go ahead.
And so is this, so what I'm understanding is this is to help with procurement evaluation as well as with assisting with oversight of existing use of AI tools. Is this like a rubric that I'm assuming the committee will like sort of evaluate based on their conversations with agencies or will they send this out to agencies to fill out? We're going to do that as a part.
We probably will send it out to them before so they know what we're looking at because, again, I don't like gotchas. but no we're going to evaluate we're going to say oh transparency I'd give that an ABCD or an F or a one two three four five whatever we want to whatever metric we want to go with that's kind of what I picture and then when we see that there is no oversight there is no transparency we can flag that and talk to the department if we can get away with not running a bill I'd prefer to do that And so I'm hopeful that our rubric will highlight what areas each department needs to go back and do a little more digging and a little more work and then where we need to go. Okay.
Okay. Appreciate that. Thank you. I think between our office's exploration of existing frameworks and the work we're doing with LIS, we're already exploring a lot of the questions that you're bringing up. So I think we're hitting the ground running in that regard.
is there sort of a deadline or like for one you'd like to see like a draft of this rubric
definitely before our next meeting I don't know that we have anything scheduled before October
but if we decide we want to so we sent out the survey and not all of the departments responded Sure So I really glad to hear you doing a responsible use of AI
because really I hope that the responsible use of AI goes really closely with our evaluation.
Yeah. And so that they come into it knowing, oh, we're supposed to have some oversight, wonder what this is going to look like. So more informative, but also we're hopeful it's going to improve the security and, you know, just the way we utilize AI.
Well, I'm happy to start working on this. I'll work with our JTC staffers and keep you sort of in the loop as we continue to work on this rubric, evaluation, whatever it is we want to call it. But this is helpful.
Is there anything else I should know about?
Rep. Paschal.
Thank you, Madam Chair.
Yeah, I had some comments. When DOC was talking about recidivism and we kept hearing, no, this doesn't generate any data. It uses our data. But it sounded like it, for instance, it transcribes a meeting. That is data. And I think maybe we need a little bit more clarity about what we're considering to be data. and my thought was when they had this meeting and it transcribes the meeting is there a check then that goes the inmate and whoever else they're meeting with they both read it afterwards and say whether or not this is what they thought happened because that would be one way to get a check on making sure that we're not injecting erroneous data especially if there's multiple steps where data is being generated, there should be some check that it was actually correct. And that, you know, Rep. DeJone was showing me some of the different stages we saw, a handful of places where something is being generated, and, you know, that could be multi-layers of generated stuff, and how far are you drifting from reality if it's not being checked. So that was a concern. And the other one was when Rep2Tone talked about how, you know, you can get in cheap and then it starts to cost you something later maybe, you know, it's like the first couple of hits are free and now it costs you money to come back. The other cost could be data. Like, are we keeping our data private? Or is that data the price we're paying? And I think that should be something that's looked at in the rubric too is do we have full control over our data or are we using that to pay for something?
Understood. Thank you.
Vice Chair Titone.
Thank you, Madam Chair.
Yeah, I mean, I'm just for one of the examples here on the supervision assistant. and auto-generates discharge transfer and supervision level change packets, AI draft summaries, action items, and noteworthy updates from in-person meetings for staff review, assuming those were the ones that were transcribed, maybe incorrectly. Combs through hundreds of case notes in seconds with smart search. Hopefully that's accurate. Maps optimized driving routes. There's like some things that are not that bad, But if you're auto-generating a discharge and transfer with data that's not correct, you know, is it being reviewed correctly? These are the kinds of things I'm concerned about. So I think that we have this illusion that we're actually saving time, but then when we have to do all the review afterwards, is it really saving that time? And how do we know? And so, you know, I don't, I've been guilty of being an early adopter of certain policies and being someone who pushes policies first here in Colorado. But with products like this, I think that looking at some of the track records of other states and seeing what their mistakes are and how they change systems before we adopt them is something that we should do whenever possible because these kinds of systems early on have a lot of promise and a lot of hype, but they don't really live up to the results. And I would hate to have to get locked into these kinds of systems, even if it is open source, which I did verify it is open source, but that's kind of what I'm a little concerned about.
Understood. Thank you.
Okay. Thank you so much for that. And then with the last few minutes, I'm going to run through, I believe I sent an email to everyone, and if I didn't, I might have used the wrong distribution list for LCS. It would have come on May 14th. And it was JTC Quarterly Oversight Standing Expectations. And if you didn't get it, please let me know, and I will go ahead and send it. It's called JTC Quarterly Oversight. See anything? Did you? Thank you. Mr. Gravy is going to go ahead and forward because it looks like I did not send a committee. It looks like I used the wrong one and it went to staff. So it is going to be sent to you as we speak. And it will come through email. But I'll just go ahead and kind of talk about what's in it until you get it. At each meeting, okay, so we're going to have a quarterly meeting schedule. No newsletter needed. we're gonna try to meet each quarter on a half-day basis the month following the close of each fiscal quarter and our first one will be October and we'll do January then we'll do April and then we'll do July this is June 30th a standing agenda every quarter at each meeting I wrote that we'll expect a briefing on eight items but we made it nine today with director Thunberg the 185 implementation status so just a plain language update on every provision we have a vendor contract registry without a delayed effective date we expect to see progress toward our updates from day one walk us through where you are on building the registry architecture diagram compliance the technical standards approval process and CISO duty integrity that's about the delegation. Our two annual reports aren't even due until November of next year, but we'd like to see your implementation roadmap well before then. The second thing is OSA audit recommendation tracker percentage implemented What moved since last quarter what stalled and why We want a number not so much a narrative There a compliance report trigger and recommendations two or more years past their committed implementation date should put a special audit in play. The third thing is active capital IT project status. schedule versus actual, budget appropriated versus budget spent, any deadline changes, if something's heading sideways, we want to hear about it. What happens is we come in December and we have two days or one day of deluge and it's so much and we need more updates than that. It's not functional. Core progress.
Can I ask if these are the things we approved last?
The first one is I just put together what I would like. I sent this to David Ettinger before he left, so we had an idea right at the beginning of interim of what I'd like to see. So with the active capital IT project, is this that you've created?
Yes, it's all the projects that we ranked and ones that have been going on, too. So yeah, you're exactly right.
yeah sorry about that I misunderstood you core progress a brief update per our obligation under 2437.5 123 I'm sorry I don't really know exactly what that is but we'll go look that up at some point broadband office progress deployment numbers grant awards federal funding status each quarter we don't want this to be annual we'd like to see it more frequently given federal vulnerability tech debt and asset refresh which legacy systems are being retired what's on the refresh schedule and where the asset inventory stands we just kind of want a running picture of what's aging out and what will replace it senior personnel changes any changes at the CIO CISO CTO COO CDO or division director level since the last meeting we don't want to learn about departures at a hearing the final thing on my list was federal funding exposure what federal IT funding and and program participation OIT is currently dependent on. What could be at risk and what's the contingency if that federal funding goes away? Frame this as risk, not just wins. Like we want to know the risk area. And then I said just anything else on your radar, new procurement, significant incidents, vendor changes, emerging risks. We did talk about an update that kind of goes along with that personnel changes, an update, so that'll be number nine on the reorg that OIT is going through, going to this product model.
No, yes, product model. Yes, okay.
And then I'm just going to talk real briefly about the IT project budget requests. This is December. So in the past the way this has worked is OSPB puts together what they want and the CFO of IT approves it It's rubber stamp, period. That can't happen anymore. We want your independent technical assessment. We don't want a repackaging of what OSPB wants. It's gotten away from us. It's gotten to a point where we're sending so many projects out that just come out of nowhere and cost us $50 million. And it's kind of problematic. So we'd like that. And I think that'll probably come from Casey in some ways because it's going to be state enterprise architecture. We want to know, does this project align with state IT architecture and standards? What's the security posture of the system being built or replaced? Tier 1, 2, 3, or 4? What does that mean in terms of risk? Like, you guys come up with a number. You don't have to use 1, 2, 3, 4. Is the vendor and contract structure sound, and is it in the registry? What is the realistic timeline, and what are our failure risks? Where does this rank against other IT needs from a technical standpoint? We need to get away from the CFO of OSPB rubber stamping, I'm sorry, the finance CFO of OIT approving and rubber stamping the projects from OSPB. We really need someone to take a technical look. It shouldn't be us. Then they end up, like, trying to negotiate with us. And we need you to do that. And so really appreciate you guys being open to that. You know, OSPB is looking at projects based on what programming they want and how much it costs. That's what their job is. But when you guys come to JTC, we really want to get the architect's view, which is independent from OSPB. It's technically grounded, and it's an honest risk assessment. If a process has a strong OSPB score but a weak security architecture, we need to know that. The HRIS program this year.
You're good. I'm sorry, I don't have my mic on. So where we now, we have this whole set of information forms that we ask departments and people bringing their projects to us to fill out. Are you talking about, this is instead of that, you're creating a new set of things? This is an additional stuff? I mean, I guess I'm not clear what set of data we're talking about.
Okay, let me explain how it goes. So the problem is that OSPB is able to push any project they want to because it's getting rubber stamped by a financial officer in OIT. So the only thing I'm asking to change is I don't need the finance officer to look at it and say, oh, yeah, that's a good project. We need to hear from the architect. Is that security? So if you think about the way the Capital Development Committee works they have projects that come to them but it the architect who actually goes through to evaluate the projects and say technically this is good you know or technically this is not good So they make a recommendation So we get our recommendation from technical people rather than financial people. The information that departments will provide will still be all the same. All we're taking away is the decision-making authority of a CFO to approve technology products. We're going to have the chief architect do that instead, just like CDC does capital development. So when we get our list of things, we have an OSPB ranking.
You want OIT to do a ranking of the quality of the technology? Is that what you're saying?
I just don't want us to just look at the finances when we're implementing technical solutions.
Understood.
Yeah, yeah. Okay. That's generally it. Okay. Okay. Yeah, and we don't need to do anything on that. I just put it into the ether because that's just, it'll change the way things look in December and it'll create a closer partnership with us. We'll be getting together quarterly. We'll get these updates quarterly. So instead of coming back with a brand new list of projects that all the departments want, We already know we've got 15 projects in process. This is where they are. And that goes into decision-making that we're going to use. So, yeah, we'll still get the OSPB. We'll still get whatever the CCAG does that's different than OSPB. We'll get both those rankings. But I expect that we're going to be listening to our architect to make decisions rather than the governor's team, budget team. because he does it just like a part of his budget.
No, I get that. So one of the other things that we have talked about in the past is that when we come into the process of looking at these projects, typically the software has been chosen already. And, you know, we're further ahead or further down the road than we should be before we start looking at it. So where does that come into play anywhere here, and how does that come into play?
It does. It's a part of our quarterly updates. We would expect that we would be seeing active IT project status, schedule versus actual. So if work is going on, if there's an active capital IT, if procurement has happened, if an RFP has occurred, we should know about that in our quarterly meetings. So we'll at least have a bit of a heads up. Okay.
I guess I'm not entirely clear on how something gets into this whole flow. So the list of things are things we've already approved.
Well, it's the list of things they've already approved, but we've also asked them to include new procurements in our quarterly reports.
Okay. So if they do any procuring, we are going to know about it.
Okay.
In the end, I think they can procure without our approval, right? They do the RFP.
Oh, sure. Yeah, yeah, yeah. So at least we'll have visibility of it is what I've added. We added that to the quarterly report. Okay. Looks like Samantha has something. Oh, good. Thank you.
Samantha Falco, Legislative Council Staff. Just for, like, clarity of, like, the actual process, Agencies make their requests through the Office of State Planning and Budgeting, and that's who evaluates, like, what projects are the priority based on what the governor deems, like, the appropriate amount to spend on IT infrastructure. And then with Senator Marchman's recommended plan, OIT would also evaluate those to kind of, like, rank severity of these systems and kind of give us a better idea of, you know, how necessary these systems actually are. And then they get sent to us on JTC on October 1st. And we don't see any of the other systems that might have, you know, submitted an application or whatever for being considered. We only see the ones that OSPB sends to us. And then we're not involved in like the, well, it's kind of project dependent, but a lot of projects, they've already set up a procurement, but we've worked a lot with OSPB to make the procurement aspect of things a lot more prevalent and of more considerations.
Thank you for clarifying. So the departments will continue to do their same process to OSPB. That will come October 1, and then they basically will have a month to, like, is it before November 1? Oh, it's before we meet in December that we would have our architect evaluate these projects.
Yeah, we receive the non-prioritized set of projects. So say there's 20, we receive all 20 of them on October 1st. And then by November 1st we receive the prioritized list And within that the set amount of money that will you know be aimed to spend at obviously that variable but um so it already like confined down to a list of all the requests that they received it's given to us in a list of 20 and then that list of 20 or whatever amount is then prioritized to say that these specific 13 projects are actually the priority We still do an analysis on all 20 or however many, and then meet in December to hear from the departments. And with this new OIT process, I guess we'd have to, like, work out how that would work for us. I guess it wouldn't really inhibit our analysis at all. So if we, you know, meet or get them on October 1st, our process would likely still stay the same. and then OIT would, I'm assuming, receive those at the same time, and maybe we could put a November 1st deadline on them as well. But as far as the October meeting goes, I don't know how much information they'll have at that point.
Yeah, that sounds good. Vice Chair Tatum.
Thank you, Madam Chair. For what it's worth, and you can have our nonpartisan fiscal analysts help verify, But I would like to see what type of impact the costs of some of these projects that are being proposed have been affected by positively or negatively by the reorganization. If that has added additional cost to what the project might have been before or after the reorganization. Just so we can understand a little bit better what the benefit of this reorganization really was. that looks good and yeah I appreciate this
We'll see how it goes. I've only been on this committee a couple years, and it's always really hard in December when all we're doing is looking at the governor's list of priorities, and we're just like, oh, yeah, let's do that one. And then the departments, it's like Hunger Games. So anyway, hopefully we can get back to really objective rather than subjective evaluation of these projects. So, yes, Ms. Falco.
Just a few housekeeping things. We can schedule our meeting for October. We can do that later. But additionally, the statewide Internet Portal Authority, SIPA, asked me to reach out to you guys. They are having their user conference on September 3rd, and they were curious if any of the committee members would like to join a panel of sorts to discuss JTC's current priorities regarding AI regulation implementation costs and fostering innovation and gain clarity on evolving framework for data privacy, transparency, and AI governance, and discuss real-world applications of technology and state and local government services. So that's on September 3rd. If anyone's interested, I'll connect you. Beth Justices, who reached out to me, so I'll get you connected with them to possibly present.
Awesome. That'll be good. Vice Chair Chitone.
Thank you, Madam Chair. Just one point of order for the committee and everyone visiting. If you going to have a presentation provide it 48 hours before the meeting We have a chance to look at it And if there are any changes that you going to present to us just let Ms Falco get that updated version so we can have it in the box so we have the information in front of us. I don't like having people do their homework at the last minute and not have it complete or ready to be presented. So we would like to see that ahead of time if possible. We can prepare better with the information in hand and it's a courtesy that we should be afforded to at least get it 48 hours before the meeting.
Okay. Yes, Rep. Pascal.
No, you're good. Thank you, Madam Chair. Especially if we're going to have a presentation that has a complex diagram, it's up on the screen across the room, we can't see it. So you know, it might as well not be there.
Yeah. Fair enough. Fair enough. Any other housekeeping items? Seeing none, we will get out of here and you get two whole minutes back. Get the meeting scheduled first? Do you want to schedule the meeting or do you want to do it via email? Whatever you prefer. We don't have everybody here. I'll just survey the committee and throw out a few October dates that would work. We appreciate you so much. You guys are amazing. So very good. Thank you guys all so much. And JTC is adjourned. Thank you.