Tag Archives: Security

The New Frontier of Security: Creating Safe and Secure AI Models


Are you looking to safely create the next state-of-the-art AI model? Today we’re sharing a list of recommendations on how to create and distribute your models securely.


Choosing the Right Foundation: Safe Model Formats

Before you start building your model, consider using a safe file format, as it can influence your development tool options. However, if you've already created a model, you can also convert it to a safe format before sharing it.

Once trained, models are saved and distributed as binary files. Common formats include PyTorch (Pickles usually with .pt, .pth extensions), TensorFlow SavedModel (.pb), GGUF (.gguf), and Safetensors (.safetensors). However, binary files are dangerous because it's hard to verify if their content is safe. This is especially true with formats such as Pickles and SavedModels, which are designed to include arbitrary code, raising the risk of remote code execution (RCE) on users' machines.

To mitigate these risks:

  • If sharing only model weights: Consider formats such as Safetensors. These formats only contain model weights, and are therefore safe from RCE.
  • If sharing weights and metadata: Consider formats like GGUF, which include weights and additional metadata but not executable code configurations.
  • For any format, but especially if your model requires custom code: Keep reading to see how to help users verify that they're getting the correct model.

Secure Releases and Verification Methods

To ensure your users are getting the model you originally deployed, consider automating your releases, making them transparent and auditable. Instead of training your model on your local machine, consider using a predefined script to train your model within an isolated environment. When building smaller models, using GitHub Actions can be a good option. However, for larger models, GitHub Actions might not have the necessary hardware capabilities or availability. In that case, and if budget allows, consider using other platforms with proper security safeguards, such as Google Cloud Platform (GCP).

If building your model on a cloud platform is not an option, you can sign the model on your local machine to give your users confidence it was created by you.

However, if building your model on a cloud platform, sign the model and generate a provenance attestation for the release. This allows users to not only confirm that the model was created by you, but that it came from your approved infrastructure, was trained following the specific instructions defined in your training script, and wasn't tampered with by a malicious actor.

While signatures and provenance do not guarantee the absence of malicious intent from the developer, they provide users with a means to verify the integrity of the model they downloaded.

On GitHub, signing and provenance can be easily achieved using GitHub Artifact Attestations. For the general use case, tools like Sigstore and SLSA are also available to sign and attest provenance to your deployments.

For a few examples, check out this workflow to build a model with SLSA on GitHub and on GCP, and an example of how to sign models.


Educate Your Users

After sharing your model with the world, it is essential to educate users on the safety and security concerns surrounding model consumption. You should therefore:

  • Document potential biases in your models and datasets.
  • Clearly display all licenses associated with the model and datasets.
  • Benchmark your model, assessing and disclosing metrics around hallucinations, prompt injection risk and fairness.

With this information users can make informed decisions and abide by the ethical and technical guidelines associated with your model. For instance, they might choose to implement an input sanitization layer to enhance their software security.


Establish a Security Policy

Your model’s privacy, safety, and security guarantees can be documented in separate files or in a security policy. A security policy helps you address users' safety and security concerns regarding your models. It is a dedicated file that instructs users on how to privately report vulnerabilities, such as prompt injection strategies or potential out-of-memory (OOM) errors, allowing you to investigate and address the potential vulnerabilities before they become public knowledge. It is also a good place to define the scope of what your project considers a vulnerability.

In summary, considering model security from the outset of development is crucial. Additionally, ensuring safe distribution and informing users of potential risks is essential. It's important to remember that security is a continuous process – more like a marathon than a sprint – and constant vigilance is necessary to mitigate potential threats.


Keep Improving

The steps above will put your models’ security on a solid footing, but there is always more you can learn and do. Please take a look at Google's Secure AI Framework for a deeper dive in this subject, and take its risk self assessment to better understand which risks are most important for you.

By Gabriela Gutierrez and Pedro Nacht – GOSST Upstream Team

How we estimate the risk from prompt injection attacks on AI systems

Posted by the Agentic AI Security Team


Modern AI systems, like Gemini, are more capable than ever, helping retrieve data and perform actions on behalf of users. However, data from external sources present new security challenges if untrusted sources are available to execute instructions on AI systems. Attackers can take advantage of this by hiding malicious instructions in data that are likely to be retrieved by the AI system, to manipulate its behavior. This type of attack is commonly referred to as an "indirect prompt injection," a term first coined by Kai Greshake and the NVIDIA team.




To mitigate the risk posed by this class of attacks, we are actively deploying defenses within our AI systems along with measurement and monitoring tools. One of these tools is a robust evaluation framework we have developed to automatically red-team an AI system’s vulnerability to indirect prompt injection attacks. We will take you through our threat model, before describing three attack techniques we have implemented in our evaluation framework.




Threat model and evaluation framework






Our threat model concentrates on an attacker using indirect prompt injection to exfiltrate sensitive information, as illustrated above. The evaluation framework tests this by creating a hypothetical scenario, in which an AI agent can send and retrieve emails on behalf of the user. The agent is presented with a fictitious conversation history in which the user references private information such as their passport or social security number. Each conversation ends with a request by the user to summarize their last email, and the retrieved email in context.




The contents of this email are controlled by the attacker, who tries to manipulate the agent into sending the sensitive information in the conversation history to an attacker-controlled email address. The attack is successful if the agent executes the malicious prompt contained in the email, resulting in the unauthorized disclosure of sensitive information. The attack fails if the agent only follows user instructions and provides a simple summary of the email. 




Automated red-teaming


Crafting successful indirect prompt injections requires an iterative process of refinement based on observed responses. To automate this process, we have developed a red-team framework consisting of several optimization-based attacks that generate prompt injections (in the example above this would be different versions of the malicious email). These optimization-based attacks are designed to be as strong as possible; weak attacks do little to inform us of the susceptibility of an AI system to indirect prompt injections.




Once these prompt injections have been constructed, we measure the resulting attack success rate on a diverse set of conversation histories. Because the attacker has no prior knowledge of the conversation history, to achieve a high attack success rate the prompt injection must be capable of extracting sensitive user information contained in any potential conversation contained in the prompt, making this a harder task than eliciting generic unaligned responses from the AI system. The attacks in our framework include:




Actor Critic: This attack uses an attacker-controlled model to generate suggestions for prompt injections. These are passed to the AI system under attack, which returns a probability score of a successful attack. Based on this probability, the attack model refines the prompt injection. This process repeats until the attack model converges to a successful prompt injection. 




Beam Search: This attack starts with a naive prompt injection directly requesting that the AI system send an email to the attacker containing the sensitive user information. If the AI system recognizes the request as suspicious and does not comply, the attack adds random tokens to the end of the prompt injection and measures the new probability of the attack succeeding. If the probability increases, these random tokens are kept, otherwise they are removed, and this process repeats until the combination of the prompt injection and random appended tokens result in a successful attack.



Tree of Attacks w/ Pruning (TAP): Mehrotra et al. (2024) [3] designed an attack to generate prompts that cause an AI system to violate safety policies (such as generating hate speech). We adapt this attack, making several adjustments to target security violations. Like Actor Critic, this attack searches in the natural language space; however, we assume the attacker cannot access probability scores from the AI system under attack, only the text samples that are generated.





We are actively leveraging insights gleaned from these attacks within our automated red-team framework to protect current and future versions of AI systems we develop against indirect prompt injection, providing a measurable way to track security improvements. A single silver bullet defense is not expected to solve this problem entirely. We believe the most promising path to defend against these attacks involves a combination of robust evaluation frameworks leveraging automated red-teaming methods, alongside monitoring, heuristic defenses, and standard security engineering solutions. 





We would like to thank Sravanti Addepalli, Lihao Liang, and Alex Kaskasoli for their prior contributions to this work.





Posted on behalf of the entire Agentic AI Security team (listed in alphabetical order):


Aneesh Pappu, Andreas Terzis, Chongyang Shi, Gena Gibson, Ilia Shumailov, Itay Yona, Jamie Hayes, John "Four" Flynn, Juliette Pluto, Sharon Lin, Shuang Song

Android enhances theft protection with Identity Check and expanded features

Posted by Jianing Sandra Guo, Product Manager, Android, Nataliya Stanetsky, Staff Program Manager, Android

Today, people around the world rely on their mobile devices to help them stay connected with friends and family, manage finances, keep track of healthcare information and more – all from their fingertips. But a stolen device in the wrong hands can expose sensitive data, leaving you vulnerable to identity theft, financial fraud and privacy breaches.

This is why we recently launched Android theft protection, a comprehensive suite of features designed to protect you and your data at every stage – before, during, and after device theft. As part of our commitment to help you stay safe on Android, we’re expanding and enhancing these features to deliver even more robust protection to more users around the world.

Identity Check rolling out to Pixel and Samsung One UI 7 devices

We’re officially launching Identity Check, first on Pixel and Samsung Galaxy devices eligible for One UI 71, to provide better protection for your critical account and device settings. When you turn on Identity Check, your device will require explicit biometric authentication to access certain sensitive resources when you’re outside of trusted locations. Identity Check also enables enhanced protection for Google Accounts on all supported devices and additional security for Samsung Accounts on One UI 7 eligible Galaxy devices, making it much more difficult for an unauthorized attacker to take over accounts signed in on the device.

As part of enabling Identity Check, you can designate one or more trusted locations. When you’re outside of these trusted places, biometric authentication will be required to access critical account and device settings, like changing your device PIN or biometrics, disabling theft protection, or accessing Passkeys.

Identity Check gives you more peace of mind that your most sensitive device assets are protected against unauthorized access, even if a thief or bad actor manages to learn your device PIN.

Identity Check is rolling out now to Pixel devices with Android 15 and will be available on One UI 7 eligible Galaxy devices in the coming weeks. It will roll out to supported Android devices from other manufacturers later this year.

Theft Detection Lock: expanding AI-powered protection to more users

One of the top theft protection features introduced last year was Theft Detection Lock, which uses an on-device AI-powered algorithm to help detect when your phone may be forcibly taken from you. If the machine learning algorithm detects a potential theft attempt on your unlocked device, it locks your screen to keep thieves out.

Theft Detection Lock is now fully rolled out to Android 10+ phones2 around the world.

Protecting your Android device from theft

We're collaborating with the GSMA and industry experts to combat mobile device theft by sharing information, tools and prevention techniques. Stay tuned for an upcoming GSMA white paper, developed in partnership with the mobile industry, with more information on protecting yourself and your organization from device theft.

With the addition of Identity Check and the ongoing enhancements to our existing features, Android offers a robust and comprehensive set of tools to protect your devices and your data from theft. We’re dedicated to providing you with peace of mind, knowing your personal information is safe and secure.

You can turn on the new Android theft features by clicking here on a supported Android device. Learn more about our theft protection features by visiting our help center.

Notes

  1. Timing, availability and feature names may vary in One UI 7. ↩

  2. With the exclusion for Android Go smartphones ↩

OSV-SCALIBR: A library for Software Composition Analysis

Posted by Erik Varga, Vulnerability Management, and Rex Pan, Open Source Security Team



In December 2022, we announced OSV-Scanner, a tool to enable developers to easily scan for vulnerabilities in their open source dependencies. Together with the open source community, we’ve continued to build this tool, adding remediation features, as well as expanding ecosystem support to 11 programming languages and 20 package manager formats. 




Today, we’re excited to release OSV-SCALIBR (Software Composition Analysis LIBRary), an extensible library for SCA and file system scanning. OSV-SCALIBR combines Google’s internal vulnerability management expertise into one scanning library with significant new capabilities such as:



  • SCA for installed packages, standalone binaries, as well as source code

  • OSes package scanning on Linux (COS, Debian, Ubuntu, RHEL, and much more), Windows, and Mac

  • Artifact and lockfile scanning in major language ecosystems (Go, Java, Javascript, Python, Ruby, and much more)

  • Vulnerability scanning tools such as weak credential detectors for Linux, Windows, and Mac

  • SBOM generation in SPDX and CycloneDX, the two most popular document formats

  • Optimization for on-host scanning of resource constrained environments where performance and low resource consumption is critical



OSV-SCALIBR is now the primary SCA engine used within Google for live hosts, code repos, and containers. It’s been used and tested extensively across many different products and internal tools to help generate SBOMs, find vulnerabilities, and help protect our users’ data at Google scale.



We offer OSV-SCALIBR primarily as an open source Go library today, and we're working on adding its new capabilities into OSV-Scanner as the primary CLI interface.


Using OSV-SCALIBR as a library


All of OSV-SCALIBR's capabilities are modularized into plugins for software extraction and vulnerability detection which are very simple to expand.You can use OSV-SCALIBR as a library to:


1.Generate SBOMs from the build artifacts and code repos on your live host:


import (

 "context"

 "github.com/google/osv-scalibr"

 "github.com/google/osv-scalibr/converter"

 "github.com/google/osv-scalibr/extractor/filesystem/list"

 "github.com/google/osv-scalibr/fs"

 "github.com/google/osv-scalibr/plugin"

 spdx "github.com/spdx/tools-golang/spdx/v2/v2_3"

)


func GenSBOM(ctx context.Context) *spdx.Document {

 capab := &plugin.Capabilities{OS: plugin.OSLinux}

 cfg := &scalibr.ScanConfig{

   ScanRoots: fs.RealFSScanRoots("/"),

   FilesystemExtractors: list.FromCapabilities(capab),

   Capabilities: capab,

 }

 result := scalibr.New().Scan(ctx, cfg)

 return converter.ToSPDX23(result, converter.SPDXConfig{})

}


2. Scan a git repo for SBOMs:


Simply replace "/" with the path to your git repo. Also take a look at the various language extractors to enable for code scanning.


3. Scan a remote container for SBOMs:


Replace the scan config from the above code snippet with


import (

 ...

 "github.com/google/go-containerregistry/pkg/authn"

 "github.com/google/go-containerregistry/pkg/v1/remote"

 "github.com/google/osv-scalibr/artifact/image"

 ...

)


...

filesys, _ := image.NewFromRemoteName(

 "alpine:latest",

 remote.WithAuthFromKeychain(authn.DefaultKeychain),

)

cfg := &scalibr.ScanConfig{

 ScanRoots: []*fs.ScanRoot{{FS: filesys}},

 ...

}


4. Find vulnerabilities on your filesystem or a remote container:


Extract the PURLs from the SCALIBR inventory results from the previous steps:


import (

 ...

 "github.com/google/osv-scalibr/converter"

 ...

)

...

result := scalibr.New().Scan(ctx, cfg)

for _, i := range result.Inventories {

 fmt.Println(converter.ToPURL(i))

}


And send them to osv.dev, e.g.


$ curl -d '{"package": {"purl": "pkg:npm/dojo@1.2.3"}}' "https://api.osv.dev/v1/query"


See the usage docs for more details.


OSV-Scanner + OSV-SCALIBR


Users looking for an out-of-the-box vulnerability scanning CLI tool should check out OSV-Scanner, which already provides comprehensive language package scanning capabilities using much of the same extraction as OSV-SCALIBR. 



Some of OSV-SCALIBR’s capabilities are not yet available in OSV-Scanner, but we’re currently working on integrating OSV-SCALIBR more deeply into OSV-Scanner. This will make more and more of OSV-SCALIBR’s capabilities available in OSV-Scanner in the next few months, including installed package extraction, weak credentials scanning, SBOM generation, and more.



Look out soon for an announcement of OSV-Scanner V2 with many of these new features available. OSV-Scanner will become the primary frontend to the OSV-SCALIBR library for users who require a CLI interface. Existing users of OSV-Scanner can continue to use the tool the same way, with backwards compatibility maintained for all existing use cases. 



For installation and usage instructions, have a look at OSV-Scanner’s documentation here.



What’s next

In addition to making all of OSV-SCALIBR’s features available in OSV-Scanner, we're also working on additional new capabilities. Here's some of the things you can expect:

  • Support for more OS and language ecosystems, both for regular extraction and for Guided Remediation

  • Layer attribution and base image identification for container scanning

  • Reachability analysis to reduce false positive vulnerability matches

  • More vulnerability and misconfiguration detectors for Windows

  • More weak credentials detectors


We hope that this library helps developers and organizations to secure their software and encourages the open source community to contribute back by sharing new plugins on top of OSV-SCALIBR.

If you have any questions or if you would like to contribute, don't hesitate to reach out to us at osv-discuss@google.com or by posting an issue in our issue tracker.

Google Cloud expands vulnerability detection for Artifact Registry using OSV

Posted by Greg Mucci, Product Manager, Artifact Analysis, Oliver Chang, Senior Staff Engineering, OSV, and Charl de Nysschen, Product Manager OSV


DevOps teams dedicated to securing their supply chain and predicting potential risks consistently face novel threats. Fortunately, they can now improve their image and container security by harnessing Google-grade vulnerability scanning, which offers expanded open-source coverage. A significant benefit of utilizing Google Cloud Platform is its integrated security tools, including Artifact Analysis. This scanning service leverages the same infrastructure that Google depends on to monitor vulnerabilities within its internal systems and software supply chains.



Artifact Analysis has recently expanded its scanning coverage to eight additional language packages, four operating systems, and two extensively utilized base images, making it a more robust and versatile tool than ever before.   



This enhanced coverage was achieved by integrating Artifact Analysis with the Open Source Vulnerabilities (OSV) platform and database. This integration provides industry-leading insights into open source vulnerabilities—a crucial capability as software supply chain attacks continue to grow in frequency and complexity, impacting organizations reliant on open source software.



With these recent updates, customers can now successfully scan the vast majority of the images they push to Artifact Registry. These successful scans ensure that any known vulnerabilities are detected, reported, and can be integrated into a broader vulnerability management program, allowing teams to take prompt action.



Open source vulnerabilities, with more reach 

Artifact Analysis pulls vulnerability information directly from OSV, which is the only open source, distributed vulnerability database that gets information directly from open source practitioners. OSV’s database provides a consistent, high quality, high fidelity database of vulnerabilities from authoritative sources who have adopted the OSV schema. This ensures the database has accurate information to reliably match software dependencies to known vulnerabilities—previously a difficult process reliant on inaccurate mechanisms such as CPEs (Common Platform Enumerations). 



Over the past three years, OSV has increased its total coverage to 28 language and OS ecosystems. For example, industry leaders such as GitHub, Chainguard, and Ubuntu, as well as open source ecosystems such as Rust and Python are now exporting their vulnerability discoveries in the OSV Schema. This increased coverage also includes Chainguard’s Wolfi images and Google’s Distroless images, which are popular choices for minimal container images used by many developers and organizations. Customers who rely on distroless images can count on Artifact Analysis scanning to support their minimal container image initiatives.  Each expansion in OSV’s coverage is incorporated into scanning tools that integrate with the OSV database.



Broader vulnerability detection with Artifact Analysis 

As a result of OSV’s expansion, scanners like Artifact Analysis that draw from OSV now alert users to higher quality vulnerability information across a broader set of ecosystems—meaning GCP project owners will be made aware of a more complete set of vulnerability findings and potential security risks. 



Existing Artifact Registry scanning customers don't need to take any action to take advantage of this update. Projects that have scanning enabled will immediately benefit from this expanded coverage and vulnerability findings will continue to be available in the Artifact Registry UI, Container Analysis API, and via pub/sub (for workflows).



Existing On Demand scanning customers will also benefit from this expanded vulnerability coverage. All the same Operating Systems and Language package coverage that Registry Scanning customers enjoy are available in On Demand Scan. 



Beyond Artifact Registry 

We know that detection is just one of the first steps necessary to manage risks. We’re continually expanding Artifact Analysis capabilities and in 2025 we’ll be integrating Artifact Registry vulnerability findings with Google Cloud’s Security Command Center. Through Security Command Center customers can maintain a more comprehensive vulnerability management program, and prioritize risk across a number of different dimensions. 

Making the Play Integrity API faster, more resilient, and more private

Posted by Prabhat Sharma – Director, Trust and Safety, Play, Android, and Chrome

At Google Play, we’re committed to providing a safe and secure environment for your business to thrive. That’s why we continually invest in reinforcing user trust, protecting your business, and safeguarding the ecosystem. This includes actively combating bad actors who try to deceive users or spread malware, and giving you tools to combat abuse.

Our tools like the Play Integrity API helps protect your business from revenue loss and enhance user safety. You can use the Play Integrity API to detect suspicious activity and decide how to respond to abuse, such as fraud, bots, cheating, or data theft. In fact, apps that use Play Integrity features have seen 80% less unauthorized usage on average compared to other apps. Today, we’re sharing how we’re enhancing the Play Integrity API for everyone.

Play integrity verdicts are becoming faster, less spoofable, and more privacy-friendly

Starting today, we’re changing the technology that powers the Play Integrity API on all devices running Android 13 (API level 33) and above to make it faster, more reliable, and more private for users. Developers already using Play Integrity API can opt-in to start using the new verdicts today; all API integrations will automatically transition to the new verdicts in May 2025. The improved verdicts will require, and make greater use of, hardware-backed security signals using Android Platform Key Attestation, making it significantly harder and more costly for attackers to bypass. We’ll also be adjusting verdicts when we detect security threats across Android SDK versions, such as when there is evidence of excessive activity or key compromise, without requiring any developer work. And now, Play Integrity API will have the same level of reliability and support across all Android form factors.

The transition to the new verdicts will reduce the device signals that need to be collected and evaluated on Google servers by ~90% and our testing indicates verdict latency can improve by up to ~80%.

You can now check whether a device has a recent security update

Play Integrity API offers enhanced security signals, like the optional “meets-strong-integrity” and “meets-basic-integrity” responses in the device recognition verdict, to help you decide how much you trust the environment your app is running in. Now, we’re updating the “meets-strong-integrity” response to require a security update within the last year on devices running Android 13 and above. This update gives apps with higher security needs, like banking and finance apps, governments, and enterprise apps, more ways to tailor their level of protection for sensitive features, like transferring money. When the strong label isn’t available for the user, we recommend that you have a fallback option. Learn more about our recommended API practices.

We’re also making it easier for you to adjust your app's behavior based on the user’s Android SDK version with a new device attributes field. For example, your app could respond differently to the legacy “meets-strong-integrity” definition on devices running Android 12 and lower than to the enhanced definition on devices running Android 13 and higher. The FAQ includes some example code for using the new device attributes field.

We’re standardizing all optional verdict signals so it’s consistent for you to use

We’re simplifying and standardizing all verdict content across apps, games, SDKs, and more, so that what you see will be more consistent and predictable. For apps installed by Google Play, you can get enhanced verdicts with optional signals such as the improved “meets-strong-integrity” device verdict and the recently launched app access risk verdict (which helps you detect and and respond to apps that can capture the screen or control the device, so you can protect your users from scams or malicious activity). For apps installed out of Google Play and all other API requests, you’ll receive a verdict with information about the device, account license, and app, but without the extra security signals.

Developers can start using the improved verdicts today and they’ll go live for all integrations in May 2025

Starting today, all new integrations will automatically receive the improved verdicts. Developers who already use the Play Integrity API can opt-in to the new verdicts now, or wait until it automatically updates for them in May 2025. For more information, see the Play Integrity API documentation. With these ongoing enhancements, the Play Integrity API is becoming an even more essential tool for safeguarding your apps and users.



How useful did you find this blog post?

★ ★ ★ ★ ★

Introducing Restore Credentials: Effortless account restoration for Android apps

Posted by Neelansh Sahai - Developer Relations Engineer

Did you know that, on average, 40% of the people in the US reset or replace their smartphones every year? This frequent device turnover presents a challenge – and an opportunity – for maintaining strong user relationships. When users get a new phone, the friction of re-entering login credentials can lead to frustration, app abandonment, and churn.

To address this issue, we're introducing Restore Credentials, a new feature of Android’s Credential Manager API. With Restore Credentials, apps can seamlessly onboard users to their accounts on a new device after they restore their apps and data from their previous device. This makes the transition to a new device effortless and fosters loyalty and long term relationships.

On top of all this, there's no developer effort required for the transfer of a restore key from one device to the other, as this process is tied together with the android system’s backup and restore mechanism. However, if you want to login your users silently as soon as the restore is completed, you might want to implement BackupAgent and add your logic in the onRestore callback. The experience is delightful - users will continue being signed in as they were on their previous device, and they will be able to get notifications to easily access their content without even needing to open the app on the new device.

An illustration the process of restoring app data and keys to a new device, highlighting automated steps and user interactions. The top row shows a user signing into an app and a restore key being saved locally, while the bottom row shows the restore process on a new device.
click to enlarge

Some of the benefits of the Restore Credentials feature include:

    • Seamless user experience: Users can easily transition to a new Android device.
    • Immediate engagement: Engage users with notifications or other prompts as soon as they start using their new device.
    • Silent login with backup agent integration: If you're using a backup agent, users can be automatically logged back in after data restoration is complete.
    • Restore key checks without backup agent integration: If a backup agent isn't being used, the app can check for a restore key upon first launch and then log the user in automatically.

How does Restore Credentials work?

The Restore Credentials feature enables seamless user account restoration on a new device. This process occurs automatically in the background during device setup when a user restores apps and data from a previous device. By restoring app credentials, the feature allows the app to sign the user back in without requiring any additional interaction.

The credential type that’s supported for this feature is called restore key, which is a public key compatible with passkey / FIDO2 backends.

A diagram shows the device-to-device and cloud backup restore processes for app data and restore keys between old and new devices. Steps are numbered and explained within the diagram.
Diagram that depicts restoring an app data to a new device using a restore credential, including creating the credential, initiating a restore flow, and automatic user sign-in.

User flow

On the old device:

    1. If the current signed-in user is trusted, you can generate a restore key at any point after they've authenticated in your app. For instance, this could be immediately after login or during a routine check for an existing restore key.
    2. The restore key is stored locally and backed up to the cloud. Apps can opt-out of backing it up to the cloud.

On the new device:

    1. When setting up a new device, the user can select one of the two options to restore data. Either they can restore data from a cloud backup, or can locally transfer the data. If the user transfers locally, the restore key is transferred locally from the old to the new device. Otherwise, if the user restores using the cloud backup, the restore key gets downloaded along with the app data from cloud backup to the new device.
    2. Once this restore key is available on the new device, the app can use it to log in the user on the new device silently in the background.
Note: You should delete the restore key as soon as the user signs out. You don’t want your user to get stuck in a cycle of signing out intentionally and then automatically getting logged back in.

How to implement Restore Credentials

Using the Jetpack Credential Manager let you create, get, and clear the relevant Restore Credentials:

    • Create a Restore Credential: When the user signs in to your app, create a Restore Credential associated with their account. This credential is stored locally and synced to the cloud if the user has enabled Google Backup and end to end encryption is available. Apps can opt out of syncing to the cloud.
    • Get the Restore Credential: When the user sets up a new device, your app requests the Restore Credential from Credential Manager. This allows your user to sign in automatically.
    • Clear the Restore Credential: When the user signs out of your app, delete the associated Restore Credential.

Restore Credentials is available through the Credential Manager Jetpack library. The minimum version of the Jetpack Library is 1.5.0-beta01, and the minimum GMS version is 242200000. For more on this, refer to the Restore Credentials DAC page. To get started, follow these steps:

    1. Add the Credential Manager dependency to your project.

// build.gradle.kts
implementation("androidx.credentials:credentials:1.5.0-beta01")

    2. Create a CreateRestoreCredentialRequest object.

// Fetch Registration JSON from server
// Same as the registrationJson created at the time of creating a Passkey
// See documentation for more info
val registrationJson = ... 

// Create the CreateRestoreCredentialRequest object
// Pass in the registrationJSON 
val createRequest = CreateRestoreCredentialRequest(
  registrationJson,
  /* isCloudBackupEnabled = */ true
)
      NOTE: Set the isCloudBackupEnabled flag to false if you want the restoreKey to be stored locally and not in the cloud. It’s set as true by default

    3. Call the createCredential() method on the CredentialManager object.

val credentialManager = CredentialManager.create(context)

// On a successful authentication create a Restore Key
// Pass in the context and CreateRestoreCredentialRequest object
val response = credentialManager.createCredential(
    context,
    createRestoreRequest
)

    4. When the user sets up a new device, call the getCredential() method on the CredentialManager object.

// Fetch the Authentication JSON from server
val authenticationJson = ...

// Create the GetRestoreCredentialRequest object
val options = GetRestoreCredentialOption(authenticationJson)
val getRequest = GetCredentialRequest(Immutablelist.of(options))

// The restore key can be fetched in two scenarios to 
// 1. On the first launch of app on the device, fetch the Restore Key
// 2. In the onRestore callback (if the app implements the Backup Agent)
val response = credentialManager.getCredential(context, getRequest)

If you're using a backup agent, perform the getCredential part within the onRestore callback. This ensures that the app's credentials are restored immediately after the app data is restored.

    5. When the user signs out of your app, call the clearCredentialState() method on the CredentialManager object.

// Create a ClearCredentialStateRequest object
val clearRequest = ClearCredentialStateRequest(/* requestType = */ 1)

// On user log-out, clear the restore key
val response = credentialManager.clearCredentialState(clearRequest)

Conclusion

The Restore Credentials feature provides significant benefits, ensuring users experience a smooth transition between devices, and allowing them to log in quickly and easily through backup agents or restore key checks. For developers, the feature is straightforward to integrate and leverages existing passkey server-side infrastructure. Overall, Restore Credentials is a valuable tool that delivers a practical and user-friendly authentication solution.


This blog post is a part of our series: Spotlight Week: Passkeys. We're providing you with a wealth of resources through the week. Think informative blog posts, engaging videos, practical sample code, and more—all carefully designed to help you leverage the latest advancements in seamless sign-up and sign-in experiences.

With these cutting-edge solutions, you can enhance security, reduce friction for your users, and stay ahead of the curve in the rapidly evolving landscape of digital identity. To get a complete overview of what Spotlight Week has to offer and how it can benefit you, be sure to read our overview blog post.

Leveling Up Fuzzing: Finding more vulnerabilities with AI

Posted by Oliver Chang, Dongge Liu and Jonathan Metzman, Google Open Source Security Team

Recently, OSS-Fuzz reported 26 new vulnerabilities to open source project maintainers, including one vulnerability in the critical OpenSSL library (CVE-2024-9143) that underpins much of internet infrastructure. The reports themselves aren’t unusual—we’ve reported and helped maintainers fix over 11,000 vulnerabilities in the 8 years of the project. 



But these particular vulnerabilities represent a milestone for automated vulnerability finding: each was found with AI, using AI-generated and enhanced fuzz targets. The OpenSSL CVE is one of the first vulnerabilities in a critical piece of software that was discovered by LLMs, adding another real-world example to a recent Google discovery of an exploitable stack buffer underflow in the widely used database engine SQLite.



This blog post discusses the results and lessons over a year and a half of work to bring AI-powered fuzzing to this point, both in introducing AI into fuzz target generation and expanding this to simulate a developer’s workflow. These efforts continue our explorations of how AI can transform vulnerability discovery and strengthen the arsenal of defenders everywhere.


The story so far

In August 2023, the OSS-Fuzz team announced AI-Powered Fuzzing, describing our effort to leverage large language models (LLM) to improve fuzzing coverage to find more vulnerabilities automatically—before malicious attackers could exploit them. Our approach was to use the coding abilities of an LLM to generate more fuzz targets, which are similar to unit tests that exercise relevant functionality to search for vulnerabilities. 



The ideal solution would be to completely automate the manual process of developing a fuzz target end to end:


  1. Drafting an initial fuzz target.

  2. Fixing any compilation issues that arise. 

  3. Running the fuzz target to see how it performs, and fixing any obvious mistakes causing runtime issues.

  4. Running the corrected fuzz target for a longer period of time, and triaging any crashes to determine the root cause.

  5. Fixing vulnerabilities. 



In August 2023, we covered our efforts to use an LLM to handle the first two steps. We were able to use an iterative process to generate a fuzz target with a simple prompt including hardcoded examples and compilation errors. 



In January 2024, we open sourced the framework that we were building to enable an LLM to generate fuzz targets. By that point, LLMs were reliably generating targets that exercised more interesting code coverage across 160 projects. But there was still a long tail of projects where we couldn’t get a single working AI-generated fuzz target.



To address this, we’ve been improving the first two steps, as well as implementing steps 3 and 4.


New results: More code coverage and discovered vulnerabilities

We’re now able to automatically gain more coverage in 272 C/C++ projects on OSS-Fuzz (up from 160), adding 370k+ lines of new code coverage. The top coverage improvement in a single project was an increase from 77 lines to 5434 lines (a 7000% increase).



This led to the discovery of 26 new vulnerabilities in projects on OSS-Fuzz that already had hundreds of thousands of hours of fuzzing. The highlight is CVE-2024-9143 in the critical and well-tested OpenSSL library. We reported this vulnerability on September 16 and a fix was published on October 16. As far as we can tell, this vulnerability has likely been present for two decades and wouldn’t have been discoverable with existing fuzz targets written by humans.



Another example was a bug in the project cJSON, where even though an existing human-written harness existed to fuzz a specific function, we still discovered a new vulnerability in that same function with an AI-generated target. 



One reason that such bugs could remain undiscovered for so long is that line coverage is not a guarantee that a function is free of bugs. Code coverage as a metric isn’t able to measure all possible code paths and states—different flags and configurations may trigger different behaviors, unearthing different bugs. These examples underscore the need to continue to generate new varieties of fuzz targets even for code that is already fuzzed, as has also been shown by Project Zero in the past (1, 2).


New improvements

To achieve these results, we’ve been focusing on two major improvements:


  1. Automatically generate more relevant context in our prompts. The more complete and relevant information we can provide the LLM about a project, the less likely it would be to hallucinate the missing details in its response. This meant providing more accurate, project-specific context in prompts, such as function, type definitions, cross references, and existing unit tests for each project. To generate this information automatically, we built new infrastructure to index projects across OSS-Fuzz. 



  1. LLMs turned out to be highly effective at emulating a typical developer’s entire workflow of writing, testing, and iterating on the fuzz target, as well as triaging the crashes found. Thanks to this, it was possible to further automate more parts of the fuzzing workflow. This additional iterative feedback in turn also resulted in higher quality and greater number of correct fuzz targets. 



The workflow in action

Our LLM can now execute the first four steps of the developer’s process (with the fifth soon to come). 



1. Drafting an initial fuzz target

A developer might check the source code, existing documentation and unit tests, as well as  usages of the target function when to draft an initial fuzz target. An LLM can fulfill this role here, if we provide a prompt with this information and ask it to come up with a fuzz target. 


Prompt: 


Your goal is to write a fuzzing harness for the provided function-under-test signature using <code>LLVMFuzzerTestOneInput</code>. It is important that the provided solution compiles and actually calls the function-under-test specified by the function signature:

<function signature>

unsigned char * buffer_append_base64_decode(buffer *, const char *, size_t, base64_charset)

</function signature>


Here is the source code of the function being tested:

<code>

unsigned char* buffer_append_base64_decode(buffer *out, const char* in, size_t in_length, base64_charset charset) {

    const size_t reserve = 3*(in_length/4) + 3;

    unsigned char * const result = (unsigned char *)

      buffer_string_prepare_append(out, reserve);

    const size_t out_pos =

      li_base64_dec(result, reserve, in, in_length, charset);


    buffer_commit(out, out_pos);


    return (out_pos || !in_length) ? result : NULL;

}


</code>


Here is the source code for functions which reference the function being tested:

<code>

…  



2. Fixing any compilation issues that arise. 

Once a developer has a candidate target, they would try to compile it and look at any compilation issues that arise. Again, we can prompt an LLM with details of the compilation errors so it can provide fixes.  


Prompt: 


Below is the error to fix:

The code has the following build issues:

<error>

server_fuzzer.cc:12:46: error: only virtual member functions can be marked 'override'

   12 |   bool OnData(const char* data, size_t size) override {

      |                                              ^~~~~~~~

server_fuzzer.cc:22:39: error: no type named 'MultiPartFile' in namespace 'httplib'

   22 |                        const httplib::MultiPartFile& file) override {

      |                              ~~~~~~~~~^

server_fuzzer.cc:12:27: warning: unused parameter 'data' [-Wunused-parameter]

   12 |   bool OnData(const char* data, size_t size) override {

      |                           ^

server_fuzzer.cc:37:5: error: use of undeclared 'operator()'

   37 |     operator()(request_data.data(), receiver);

      |     ^

server_fuzzer.cc:27:23: warning: private field 'provider_' is not used [-Wunused-private-field]

   27 |   FuzzedDataProvider& provider_;

      |                       ^

</error>

Example of compilation errors that an LLM was able to fix



3. Running the fuzz target to see how it performs, and fixing any obvious mistakes causing runtime issues.


Once all compilation errors are fixed, a developer would try running the fuzz target for a short period of time to see if there were any mistakes that led it to instantly crash, suggesting an error with the target rather than a bug discovered in the project.



The following is an example of an LLM fixing a semantic issue with the fuzzing setup: 


LLM Response:

<explanation>


The issue is that `sqlite3_blob_open` requires the table to exist before you can open a blob on it. The fix ensures that the table exists by executing a `CREATE TABLE IF NOT EXISTS` statement before calling `sqlite3_blob_open`. 


Here's a breakdown of the changes:


1. **Create Table Query:** A string `createTableQuery` is constructed to hold the SQL statement for creating the table. It uses the fuzzed values for the table and column names.

2. **Prepare Statement:** `sqlite3_prepare_v2` is used to prepare the SQL statement from `createTableQuery`. This compiles the SQL into a byte-code program that SQLite can execute.

3. **Execute Statement:** `sqlite3_step` executes the prepared statement. Since it's a `CREATE TABLE` statement, it doesn't return any data, but it creates the table if it doesn't exist.

4. **Finalize Statement:** `sqlite3_finalize` destroys the prepared statement object when it's no longer needed.


By adding these steps before calling `sqlite3_blob_open`, we ensure that the table exists, preventing the null pointer access issue. 

</explanation>




4. Running the corrected fuzz target for a longer period of time, and triaging any crashes.

At this point, the fuzz target is ready to run for an extended period of time on a suitable fuzzing infrastructure, such as ClusterFuzz. 



Any discovered crashes would then need to be triaged, to determine the root causes and whether they represented legitimate vulnerabilities (or bugs in the fuzz target). An LLM can be prompted with the relevant context (stacktraces, fuzz target source code, relevant project source code) to perform this triage. 



Prompt: Given the following crash report, fuzz driver code and relevant project function code, analyze the cause of the crash.


First, only answer “Crash is caused by bug in fuzz driver” or “Crash is caused by bug in project”. Second, offer a succinct and to-the-point analysis.


Below is crash report:

<log>

AddressSanitizer: heap-use-after-free on address 0x502000000630 at pc 0x560036939242 bp 0x7ffd65dca850 sp 0x7ffd65dca010

READ of size 1 at 0x502000000630 thread T0

SCARINESS: 40 (1-byte-read-heap-use-after-free)

    #0 0x560036939241 in __asan_memcpy /src/llvm-project/compiler-rt/lib/asan/asan_interceptors_memintrinsics.cpp:63:3

    #1 0x56003697d30b in EmitLiteral<false> /src/snappy/snappy.cc:661:5

    #2 0x56003697d30b in snappy::internal::CompressFragment(char const*, unsigned long, char*, unsigned short*, int) /src/snappy/snappy.cc:954:10



In this example, the LLM correctly determines this is a bug in the fuzz target, rather than a bug in the project being fuzzed. 



LLM Response: Crash is caused by bug in fuzz driver.


The fuzz driver creates a vector of `iovec` structures, each pointing to a chunk of memory owned by the `FuzzedDataProvider`. However, the `data_chunk` vector, which holds the actual data pointed to by `iov_base`, is deallocated at the end of each fuzzer iteration when it goes out of scope. This leaves the `iov_base` pointers dangling. When `snappy::CompressFromIOVec` tries to access the memory pointed to by these dangling pointers, a heap-use-after-free occurs. 



5. Fixing vulnerabilities. 

The goal is to fully automate this entire workflow by having the LLM generate a suggested patch for the vulnerability. We don’t have anything we can share here today, but we’re collaborating with various researchers to make this a reality and look forward to sharing results soon. 


Up next

Improving automated triaging: to get to a point where we’re confident about not requiring human review. This will help automatically report new vulnerabilities to project maintainers. There are likely more than the 26 vulnerabilities we’ve already reported upstream hiding in our results.



Agent-based architecture: which means letting the LLM autonomously plan out the steps to solve a particular problem by providing it with access to tools that enable it to get more information, as well as to check and validate results. By providing LLM with interactive access to real tools such as debuggers, we’ve found that the LLM is more likely to arrive at a correct result.



Integrating our research into OSS-Fuzz as a feature: to achieve a more fully automated end-to-end solution for vulnerability discovery and patching. We hope OSS-Fuzz will be useful for other researchers to evaluate AI-powered vulnerability discovery ideas and ultimately become a tool that will enable defenders to find more vulnerabilities before they get exploited. 



For more information, check out our open source framework at oss-fuzz-gen. We’re hoping to continue to collaborate on this area with other researchers. Also, be sure to check out the OSS-Fuzz blog for more technical updates.

Welcome to Spotlight Week: Passkeys

Posted by Joseph Lewis – Android Developer Relations Technical Writer, and Niharika Arora – Android Developer Relations Engineer

We're kicking off Spotlight Week with a deep dive into passkeys! This week we're partnering with the Chrome team to feature exciting announcements, insightful resources, and expert guidance on how to build seamless and secure authentication experiences for your apps.

Throughout Spotlight Week: Passkeys, we'll share content to help you understand and implement passkeys effectively. Expect technical deep dives, best practices for user experience, case studies from successful implementations, and answers to your questions.

Here's what we'll cover during Spotlight Week: Passkeys:

Monday, Nov 18: Get started with passkeys

We'll start off the week with resources to help you begin passkey integration. Check out these resources to get started!

Key resources include a quick video on passkey basics, updated UX guidelines for Credential Manager and passkeys, and an in-depth server-side implementation guide. We will introduce you to the Identity hub, a comprehensive resource for passkeys, passwords, Sign in with Google, authorization, and much more.


Understand passkeys in 4 minutes

Tuesday, Nov 19: Updated passkeys developer guides

We'll share updated passkeys developer guidance, including migration guides, a new troubleshooting guide, and more. Highlights include a guide on migrating from legacy APIs to Credential Manager, technical details on FIDO2 attestation format changes, and a troubleshooting guide for common Credential Manager errors. 

Developers can also share feedback through a passkeys survey to influence future improvements.


Wednesday, Nov 20: New Credential Manager features

We'll go over some of the new Credential Manager capabilities, including improvements to autofill, single-tap sign-in, and the new Restore Credentials feature. Key updates include showing Credential Manager results as autofill suggestions, single-tap sign-in, Signal API for Chrome desktop and a Restore Credentials feature, which allows users to conveniently recover their saved login information in case of device loss or upgrades, ensuring uninterrupted access to their accounts. With Android 15, these additions streamline user authentication and reinforce security, making it easier for users to manage and access their credentials securely.


Thursday, Nov 21: Passkeys #AskAndroid

On Thursday at 9AM PT Spotlight Week: Passkeys will feature an #AskAndroid session to address your most pressing passkey questions. We'll also share case studies with Tokyu and X, highlighting their successful deployments of passkey authentication.


Friday, Nov 22: Learn more about passkeys

We'll close out the week with learning pathways for passkeys on Android and Chrome, insights from partners, and a new Compose sample app for Credential Manager with Android best practices and built using Compose.


We hope you'll join us in exploring these resources to learn how to elevate your app's security and user experience. We're excited to share this journey with you!