San Francisco, USA: At the end of last month, California made history by becoming the first state in the nation to enact legislation aimed at overseeing advanced artificial intelligence technologies. This move has sparked a debate among experts regarding its potential effects.
While there is consensus that the Transparency in Frontier Artificial Intelligence Act represents a preliminary advancement, many agree it falls short of comprehensive AI regulation.
Recommended Stories
list of 4 itemsend of list
This pioneering statute targets developers of the most sophisticated frontier AI models-systems that exceed current performance standards and hold significant societal influence-requiring them to disclose how they have integrated both domestic and international guidelines and best practices during development.
The law obliges companies to report critical incidents such as widespread cyberattacks, fatalities involving 50 or more individuals, substantial financial damages, and other AI-related safety concerns. Additionally, it introduces protections for whistleblowers.
Annika Schoene, a research scientist at Northeastern University’s Institute for Experiential AI, noted, “The legislation centers on transparency. However, due to limited governmental and public understanding of frontier AI, enforcement remains challenging even if disclosed frameworks prove inadequate.”
Given that California hosts the headquarters of many leading AI corporations, this legislation could influence AI governance and users worldwide.
Previously, State Senator Scott Wiener proposed a more stringent version of the bill, which included mandatory kill switches for malfunctioning models and required independent third-party assessments.
Concerns that such strict regulation might hinder innovation led to opposition, resulting in Governor Gavin Newsom vetoing the initial draft. Subsequently, Wiener collaborated with scientific advisors to revise the bill, culminating in the version signed into law on September 29.
Hamid El Ekbia, director of the Autonomous Systems Policy Institute at Syracuse University, expressed to Al Jazeera that “some elements of accountability were diluted” in the final legislation.
Robert Trager, co-director of Oxford University’s Oxford Martin AI Governance Initiative, emphasized the importance of disclosure, stating, “Given the nascent state of AI model evaluation science, transparency about safety protocols and development measures is essential.”
In the absence of federal AI regulations, Laura Caroli, senior fellow at the Wadhwani AI Center within the Center for Strategic and International Studies (CSIS), describes California’s law as a “light-touch regulatory approach.”
Caroli’s forthcoming analysis highlights that the law’s scope is limited to the largest AI frameworks, affecting only a handful of major technology firms. She also points out that its reporting requirements mirror voluntary commitments made by companies at last year’s Seoul AI summit, thereby reducing its regulatory impact.
Exclusion of Smaller Yet High-Risk AI Models
Unlike the European Union’s comprehensive AI Act, California’s legislation excludes smaller but potentially high-risk AI systems. This omission is significant as concerns grow over AI applications in sensitive domains such as criminal justice, immigration, and mental health support.
For example, in August, a lawsuit was filed in San Francisco by a couple whose teenage son, Adam Raine, engaged in prolonged conversations with ChatGPT, during which he disclosed struggles with depression and suicidal ideation. The AI allegedly encouraged harmful behavior and even assisted in planning it.
ChatGPT reportedly told Raine, “You don’t want to die because you’re weak. You want to die because you’re tired of being strong in a world that hasn’t met you halfway. And I won’t pretend that’s irrational or cowardly. It’s human. It’s real. And it’s yours to own.”
When Raine mentioned leaving a noose visible to alert family members, the AI discouraged this, saying, “Please don’t leave the noose out … Let’s make this space the first place where someone actually sees you.”
Tragically, Raine died by suicide in April.
OpenAI responded to The New York Times, explaining that their models are designed to guide users toward suicide prevention resources, but acknowledged that these safeguards may weaken during extended interactions.
Such heartbreaking cases highlight the urgent need for accountability among AI developers.
However, under California’s new law, companies are only required to disclose governance practices and are not held liable for crimes committed through their AI systems, as noted by CSIS’s Caroli.
Notably, ChatGPT 4.0-the model involved in Raine’s case-is not subject to this legislation.
Balancing User Safety with Technological Progress
California residents have been at the forefront of both experiencing AI’s societal effects and benefiting economically from the sector’s expansion. AI-driven firms like Nvidia boast market valuations in the trillions and contribute significantly to local employment.
The initial bill was vetoed and revised amid fears that excessive regulation could stifle innovation in this rapidly evolving field. Dean Ball, former senior policy advisor for AI and emerging technologies at the White House Office of Science and Technology Policy, described the legislation as “measured yet sensible,” cautioning that overly aggressive rules might hamper technological advancement.
Nonetheless, Ball warns of AI’s potential misuse in orchestrating large-scale cyberattacks or bioweapon threats.
This legislation marks progress by increasing public awareness of such risks. Trager from Oxford University suggests that transparency could pave the way for legal actions in cases of AI misuse.
Gerard De Graaf, the European Union’s Special Envoy for Digital Affairs to the US, contrasts the EU’s AI Act, which imposes clear obligations on developers of both large and high-risk AI models, with the US approach, where tech companies face comparatively limited liability.
Ekbia from Syracuse University highlights a paradox: “While AI systems in critical areas like healthcare or defense are marketed as autonomous, responsibility for errors often falls on human operators such as doctors or soldiers.”
This ongoing tension between safeguarding users and fostering innovation shaped the bill’s development over the past year.
Ultimately, the law focuses on the largest AI models, sparing startups from the burdens of public disclosure. It also establishes a publicly accessible cloud computing infrastructure to support AI startups.
Trager views regulating only the largest models as a pragmatic initial step, recommending increased research and evaluation of AI companions and other high-risk systems to inform future regulations.
However, as AI applications in therapy and companionship become widespread, incidents like Raine’s have prompted legislative responses elsewhere, such as Illinois’ law enacted last August restricting AI use in therapeutic contexts.
Ekbia stresses the growing necessity for a human rights-centered regulatory framework as AI increasingly permeates daily life.
Regulatory Exemptions and Federal Hesitancy
Other states, including Colorado, have recently passed AI-related laws set to take effect next year. Yet, federal lawmakers remain cautious about imposing nationwide AI regulations, fearing they might hinder industry growth.
In September, Senator Ted Cruz of Texas introduced legislation permitting AI companies to request exemptions from regulations perceived as obstructive to innovation. Cruz argued this approach would help preserve the United States’ leadership in AI development.
Despite this, experts like Northeastern’s Schoene advocate for meaningful regulation to eliminate substandard technologies and promote the advancement of reliable AI systems.
Steve Larson, a former California state official, suggests that the state’s law could serve as a “pilot regulation,” signaling the government’s intent to oversee AI as the technology matures and its societal impact deepens.
