Author Archives: Kimberly Samra

Leveling Up Fuzzing: Finding more vulnerabilities with AI

Posted by Oliver Chang, Dongge Liu and Jonathan Metzman, Google Open Source Security Team

Recently, OSS-Fuzz reported 26 new vulnerabilities to open source project maintainers, including one vulnerability in the critical OpenSSL library (CVE-2024-9143) that underpins much of internet infrastructure. The reports themselves aren’t unusual—we’ve reported and helped maintainers fix over 11,000 vulnerabilities in the 8 years of the project. 



But these particular vulnerabilities represent a milestone for automated vulnerability finding: each was found with AI, using AI-generated and enhanced fuzz targets. The OpenSSL CVE is one of the first vulnerabilities in a critical piece of software that was discovered by LLMs, adding another real-world example to a recent Google discovery of an exploitable stack buffer underflow in the widely used database engine SQLite.



This blog post discusses the results and lessons over a year and a half of work to bring AI-powered fuzzing to this point, both in introducing AI into fuzz target generation and expanding this to simulate a developer’s workflow. These efforts continue our explorations of how AI can transform vulnerability discovery and strengthen the arsenal of defenders everywhere.


The story so far

In August 2023, the OSS-Fuzz team announced AI-Powered Fuzzing, describing our effort to leverage large language models (LLM) to improve fuzzing coverage to find more vulnerabilities automatically—before malicious attackers could exploit them. Our approach was to use the coding abilities of an LLM to generate more fuzz targets, which are similar to unit tests that exercise relevant functionality to search for vulnerabilities. 



The ideal solution would be to completely automate the manual process of developing a fuzz target end to end:


  1. Drafting an initial fuzz target.

  2. Fixing any compilation issues that arise. 

  3. Running the fuzz target to see how it performs, and fixing any obvious mistakes causing runtime issues.

  4. Running the corrected fuzz target for a longer period of time, and triaging any crashes to determine the root cause.

  5. Fixing vulnerabilities. 



In August 2023, we covered our efforts to use an LLM to handle the first two steps. We were able to use an iterative process to generate a fuzz target with a simple prompt including hardcoded examples and compilation errors. 



In January 2024, we open sourced the framework that we were building to enable an LLM to generate fuzz targets. By that point, LLMs were reliably generating targets that exercised more interesting code coverage across 160 projects. But there was still a long tail of projects where we couldn’t get a single working AI-generated fuzz target.



To address this, we’ve been improving the first two steps, as well as implementing steps 3 and 4.


New results: More code coverage and discovered vulnerabilities

We’re now able to automatically gain more coverage in 272 C/C++ projects on OSS-Fuzz (up from 160), adding 370k+ lines of new code coverage. The top coverage improvement in a single project was an increase from 77 lines to 5434 lines (a 7000% increase).



This led to the discovery of 26 new vulnerabilities in projects on OSS-Fuzz that already had hundreds of thousands of hours of fuzzing. The highlight is CVE-2024-9143 in the critical and well-tested OpenSSL library. We reported this vulnerability on September 16 and a fix was published on October 16. As far as we can tell, this vulnerability has likely been present for two decades and wouldn’t have been discoverable with existing fuzz targets written by humans.



Another example was a bug in the project cJSON, where even though an existing human-written harness existed to fuzz a specific function, we still discovered a new vulnerability in that same function with an AI-generated target. 



One reason that such bugs could remain undiscovered for so long is that line coverage is not a guarantee that a function is free of bugs. Code coverage as a metric isn’t able to measure all possible code paths and states—different flags and configurations may trigger different behaviors, unearthing different bugs. These examples underscore the need to continue to generate new varieties of fuzz targets even for code that is already fuzzed, as has also been shown by Project Zero in the past (1, 2).


New improvements

To achieve these results, we’ve been focusing on two major improvements:


  1. Automatically generate more relevant context in our prompts. The more complete and relevant information we can provide the LLM about a project, the less likely it would be to hallucinate the missing details in its response. This meant providing more accurate, project-specific context in prompts, such as function, type definitions, cross references, and existing unit tests for each project. To generate this information automatically, we built new infrastructure to index projects across OSS-Fuzz. 



  1. LLMs turned out to be highly effective at emulating a typical developer’s entire workflow of writing, testing, and iterating on the fuzz target, as well as triaging the crashes found. Thanks to this, it was possible to further automate more parts of the fuzzing workflow. This additional iterative feedback in turn also resulted in higher quality and greater number of correct fuzz targets. 



The workflow in action

Our LLM can now execute the first four steps of the developer’s process (with the fifth soon to come). 



1. Drafting an initial fuzz target

A developer might check the source code, existing documentation and unit tests, as well as  usages of the target function when to draft an initial fuzz target. An LLM can fulfill this role here, if we provide a prompt with this information and ask it to come up with a fuzz target. 


Prompt: 


Your goal is to write a fuzzing harness for the provided function-under-test signature using <code>LLVMFuzzerTestOneInput</code>. It is important that the provided solution compiles and actually calls the function-under-test specified by the function signature:

<function signature>

unsigned char * buffer_append_base64_decode(buffer *, const char *, size_t, base64_charset)

</function signature>


Here is the source code of the function being tested:

<code>

unsigned char* buffer_append_base64_decode(buffer *out, const char* in, size_t in_length, base64_charset charset) {

    const size_t reserve = 3*(in_length/4) + 3;

    unsigned char * const result = (unsigned char *)

      buffer_string_prepare_append(out, reserve);

    const size_t out_pos =

      li_base64_dec(result, reserve, in, in_length, charset);


    buffer_commit(out, out_pos);


    return (out_pos || !in_length) ? result : NULL;

}


</code>


Here is the source code for functions which reference the function being tested:

<code>

…  



2. Fixing any compilation issues that arise. 

Once a developer has a candidate target, they would try to compile it and look at any compilation issues that arise. Again, we can prompt an LLM with details of the compilation errors so it can provide fixes.  


Prompt: 


Below is the error to fix:

The code has the following build issues:

<error>

server_fuzzer.cc:12:46: error: only virtual member functions can be marked 'override'

   12 |   bool OnData(const char* data, size_t size) override {

      |                                              ^~~~~~~~

server_fuzzer.cc:22:39: error: no type named 'MultiPartFile' in namespace 'httplib'

   22 |                        const httplib::MultiPartFile& file) override {

      |                              ~~~~~~~~~^

server_fuzzer.cc:12:27: warning: unused parameter 'data' [-Wunused-parameter]

   12 |   bool OnData(const char* data, size_t size) override {

      |                           ^

server_fuzzer.cc:37:5: error: use of undeclared 'operator()'

   37 |     operator()(request_data.data(), receiver);

      |     ^

server_fuzzer.cc:27:23: warning: private field 'provider_' is not used [-Wunused-private-field]

   27 |   FuzzedDataProvider& provider_;

      |                       ^

</error>

Example of compilation errors that an LLM was able to fix



3. Running the fuzz target to see how it performs, and fixing any obvious mistakes causing runtime issues.


Once all compilation errors are fixed, a developer would try running the fuzz target for a short period of time to see if there were any mistakes that led it to instantly crash, suggesting an error with the target rather than a bug discovered in the project.



The following is an example of an LLM fixing a semantic issue with the fuzzing setup: 


LLM Response:

<explanation>


The issue is that `sqlite3_blob_open` requires the table to exist before you can open a blob on it. The fix ensures that the table exists by executing a `CREATE TABLE IF NOT EXISTS` statement before calling `sqlite3_blob_open`. 


Here's a breakdown of the changes:


1. **Create Table Query:** A string `createTableQuery` is constructed to hold the SQL statement for creating the table. It uses the fuzzed values for the table and column names.

2. **Prepare Statement:** `sqlite3_prepare_v2` is used to prepare the SQL statement from `createTableQuery`. This compiles the SQL into a byte-code program that SQLite can execute.

3. **Execute Statement:** `sqlite3_step` executes the prepared statement. Since it's a `CREATE TABLE` statement, it doesn't return any data, but it creates the table if it doesn't exist.

4. **Finalize Statement:** `sqlite3_finalize` destroys the prepared statement object when it's no longer needed.


By adding these steps before calling `sqlite3_blob_open`, we ensure that the table exists, preventing the null pointer access issue. 

</explanation>




4. Running the corrected fuzz target for a longer period of time, and triaging any crashes.

At this point, the fuzz target is ready to run for an extended period of time on a suitable fuzzing infrastructure, such as ClusterFuzz. 



Any discovered crashes would then need to be triaged, to determine the root causes and whether they represented legitimate vulnerabilities (or bugs in the fuzz target). An LLM can be prompted with the relevant context (stacktraces, fuzz target source code, relevant project source code) to perform this triage. 



Prompt: Given the following crash report, fuzz driver code and relevant project function code, analyze the cause of the crash.


First, only answer “Crash is caused by bug in fuzz driver” or “Crash is caused by bug in project”. Second, offer a succinct and to-the-point analysis.


Below is crash report:

<log>

AddressSanitizer: heap-use-after-free on address 0x502000000630 at pc 0x560036939242 bp 0x7ffd65dca850 sp 0x7ffd65dca010

READ of size 1 at 0x502000000630 thread T0

SCARINESS: 40 (1-byte-read-heap-use-after-free)

    #0 0x560036939241 in __asan_memcpy /src/llvm-project/compiler-rt/lib/asan/asan_interceptors_memintrinsics.cpp:63:3

    #1 0x56003697d30b in EmitLiteral<false> /src/snappy/snappy.cc:661:5

    #2 0x56003697d30b in snappy::internal::CompressFragment(char const*, unsigned long, char*, unsigned short*, int) /src/snappy/snappy.cc:954:10



In this example, the LLM correctly determines this is a bug in the fuzz target, rather than a bug in the project being fuzzed. 



LLM Response: Crash is caused by bug in fuzz driver.


The fuzz driver creates a vector of `iovec` structures, each pointing to a chunk of memory owned by the `FuzzedDataProvider`. However, the `data_chunk` vector, which holds the actual data pointed to by `iov_base`, is deallocated at the end of each fuzzer iteration when it goes out of scope. This leaves the `iov_base` pointers dangling. When `snappy::CompressFromIOVec` tries to access the memory pointed to by these dangling pointers, a heap-use-after-free occurs. 



5. Fixing vulnerabilities. 

The goal is to fully automate this entire workflow by having the LLM generate a suggested patch for the vulnerability. We don’t have anything we can share here today, but we’re collaborating with various researchers to make this a reality and look forward to sharing results soon. 


Up next

Improving automated triaging: to get to a point where we’re confident about not requiring human review. This will help automatically report new vulnerabilities to project maintainers. There are likely more than the 26 vulnerabilities we’ve already reported upstream hiding in our results.



Agent-based architecture: which means letting the LLM autonomously plan out the steps to solve a particular problem by providing it with access to tools that enable it to get more information, as well as to check and validate results. By providing LLM with interactive access to real tools such as debuggers, we’ve found that the LLM is more likely to arrive at a correct result.



Integrating our research into OSS-Fuzz as a feature: to achieve a more fully automated end-to-end solution for vulnerability discovery and patching. We hope OSS-Fuzz will be useful for other researchers to evaluate AI-powered vulnerability discovery ideas and ultimately become a tool that will enable defenders to find more vulnerabilities before they get exploited. 



For more information, check out our open source framework at oss-fuzz-gen. We’re hoping to continue to collaborate on this area with other researchers. Also, be sure to check out the OSS-Fuzz blog for more technical updates.

Retrofitting Spatial Safety to hundreds of millions of lines of C++

Posted by Alex Rebert and Max Shavrick, Security Foundations, and Kinuko Yasada, Core Developer

Attackers regularly exploit spatial memory safety vulnerabilities, which occur when code accesses a memory allocation outside of its intended bounds, to compromise systems and sensitive data. These vulnerabilities represent a major security risk to users. 

Based on an analysis of in-the-wild exploits tracked by Google's Project Zero, spatial safety vulnerabilities represent 40% of in-the-wild memory safety exploits over the past decade:

Breakdown of memory safety CVEs exploited in the wild by vulnerability class.1

Google is taking a comprehensive approach to memory safety. A key element of our strategy focuses on Safe Coding and using memory-safe languages in new code. This leads to an exponential decline in memory safety vulnerabilities and quickly improves the overall security posture of a codebase, as demonstrated by our post about Android's journey to memory safety.

However, this transition will take multiple years as we adapt our development practices and infrastructure. Ensuring the safety of our billions of users therefore requires us to go further: we're also retrofitting secure-by-design principles to our existing C++ codebase wherever possible.

To that end, we're working towards bringing spatial memory safety into as many of our C++ codebases as possible, including Chrome and the monolithic codebase powering our services.

We’ve begun by enabling hardened libc++, which adds bounds checking to standard C++ data structures, eliminating a significant class of spatial safety bugs. While C++ will not become fully memory-safe, these improvements reduce risk as discussed in more detail in our perspective on memory safety, leading to more reliable and secure software.

This post explains how we're retrofitting hardened libc++ across our codebases and  showcases the positive impact it's already having, including preventing exploits, reducing crashes, and improving code correctness.

Bounds-checked data structures: The foundation for spatial safety

One of our primary strategies for improving spatial safety in C++ is to implement bounds checking for common data structures, starting with hardening the C++ standard library (in our case, LLVM’s libc++). Hardened libc++, recently added by open source contributors, introduces a set of security checks designed to catch vulnerabilities such as out-of-bounds accesses in production.

For example, hardened libc++ ensures that every access to an element of a std::vector stays within its allocated bounds, preventing attempts to read or write beyond the valid memory region. Similarly, hardened libc++ checks that a std::optional isn't empty before allowing access, preventing access to uninitialized memory.

This approach mirrors what's already standard practice in many modern programming languages like Java, Python, Go, and Rust. They all incorporate bounds checking by default, recognizing its crucial role in preventing memory errors. C++ has been a notable exception, but efforts like hardened libc++ aim to close this gap in our infrastructure. It’s also worth noting that similar hardening is available in other C++ standard libraries, such as libstdc++.

Raising the security baseline across the board

Building on the successful deployment of hardened libc++ in Chrome in 2022, we've now made it default across our server-side production systems. This improves spatial memory safety across our services, including key performance-critical components of products like Search, Gmail, Drive, YouTube, and Maps. While a very small number of components remain opted out, we're actively working to reduce this and raise the bar for security across the board, even in applications with lower exploitation risk.

The performance impact of these changes was surprisingly low, despite Google's modern C++ codebase making heavy use of libc++. Hardening libc++ resulted in an average 0.30% performance impact across our services (yes, only a third of a percent).

This is due to both the compiler's ability to eliminate redundant checks during optimization, and the efficient design of hardened libc++. While a handful of performance-critical code paths still require targeted use of explicitly unsafe accesses, these instances are carefully reviewed for safety. Techniques like profile-guided optimizations further improved performance, but even without those advanced techniques, the overhead of bounds checking remains minimal.

We actively monitor the performance impact of these checks and work to minimize any unnecessary overhead. For instance, we identified and fixed an unnecessary check, which led to a 15% reduction in overhead (reduced from 0.35% to 0.3%), and contributed the fix back to the LLVM project to share the benefits with the broader C++ community.

While hardened libc++'s overhead is minimal for individual applications in most cases, deploying it at Google's scale required a substantial commitment of computing resources. This investment underscores our dedication to enhancing the safety and security of our products.

From tests to production

Enabling libc++ hardening wasn't a simple flip of a switch. Rather, it required a multi-stage rollout to avoid accidentally disrupting users or creating an outage:

  1. Testing: We first enabled hardened libc++ in our tests over a year ago. This allowed us to identify and fix hundreds of previously undetected bugs in our code and tests.
  2. Baking: We let the hardened runtime "bake" in our testing and pre-production environments, giving developers time to adapt and address any new issues that surfaced. We also conducted extensive performance evaluations, ensuring minimal impact to our users' experience.
  3. Gradual Production Rollout: We then rolled out hardened libc++ to production over several months, starting with a small set of services and gradually expanding to our entire infrastructure. We closely monitored the rollout, promptly addressing any crashes or performance regressions.

Quantifiable impact

In just a few months since enabling hardened libc++ by default, we've already seen benefits.

Preventing exploits: Hardened libc++ has already disrupted an internal red team exercise and would have prevented another one that happened before we enabled hardening, demonstrating its effectiveness in thwarting exploits. The safety checks have uncovered over 1,000 bugs, and would prevent 1,000 to 2,000 new bugs yearly at our current rate of C++ development.

Improved reliability and correctness: The process of identifying and fixing bugs uncovered by hardened libc++ led to a 30% reduction in our baseline segmentation fault rate across production, indicating improved code reliability and quality. Beyond crashes, the checks also caught errors that would have otherwise manifested as unpredictable behavior or data corruption.

Moving average of segfaults across our fleet over time, before and after enablement.

Easier debugging: Hardened libc++ enabled us to identify and fix multiple bugs that had been lurking in our code for more than a decade. The checks transform many difficult-to-diagnose memory corruptions into immediate and easily debuggable errors, saving developers valuable time and effort.

Bridging the gap with memory-safe languages

While libc++ hardening provides immediate benefits by adding bounds checking to standard data structures, it's only one piece of the puzzle when it comes to spatial safety.

We're expanding bounds checking to other libraries and working to migrate our code to Safe Buffers, requiring all accesses to be bounds checked. For spatial safety, both hardened data structures, including their iterators, and Safe Buffers are necessary.

Beyond improving the safety of our C++, we're also focused on making it easier to interoperate with memory-safe languages. Migrating our C++ to Safe Buffers shrinks the gap between the languages, which simplifies interoperability and potentially even an eventual automated translation.

Building a safer C++ ecosystem

Hardened libc++ is a practical and effective way to enhance the safety, reliability, and debuggability of C++ code with minimal overhead. Given this, we strongly encourage organizations using C++ to enable their standard library's hardened mode universally by default.

At Google, enabling hardened libc++ is only the first step in our journey towards a spatially safe C++ codebase. By expanding bounds checking, migrating to Safe Buffers, and actively collaborating with the broader C++ community, we aim to create a future where spatial safety is the norm.

Acknowledgements

We’d like to thank Emilia Kasper, Chandler Carruth, Duygu Isler, Matthew Riley, and Jeff Vander Stoep for their helpful feedback.

  1. Based on manual analysis of CVEs from July 15, 2014 to Dec 14, 2023. Note that we could not classify 11% of CVEs.. ↩

Safer with Google: Advancing Memory Safety

Posted by Alex Rebert, Security Foundations, and Chandler Carruth, Jen Engel, Andy Qin, Core Developers

Error-prone interactions between software and memory1 are widely understood to create safety issues in software. It is estimated that about 70% of severe vulnerabilities2 in memory-unsafe codebases are due to memory safety bugs. Malicious actors exploit these vulnerabilities and continue to create real-world harm. In 2023, Google’s threat intelligence teams conducted an industry-wide study and observed a close to all-time high number of vulnerabilities exploited in the wild. Our internal analysis estimates that 75% of CVEs used in zero-day exploits are memory safety vulnerabilities.

At Google, we have been mindful of these issues for over two decades, and are on a journey to continue advancing the state of memory safety in the software we consume and produce. Our Secure by Design commitment emphasizes integrating security considerations, including robust memory safety practices, throughout the entire software development lifecycle. This proactive approach fosters a safer and more trustworthy digital environment for everyone.

This post builds upon our previously reported Perspective on Memory Safety, and introduces our strategic approach to memory safety.

Our journey so far

Google's journey with memory safety is deeply intertwined with the evolution of the software industry itself. In our early days, we recognized the importance of balancing performance with safety. This led to the early adoption of memory-safe languages like Java and Python, and the creation of Go. Today these languages comprise a large portion of our code, providing memory safety among other benefits. Meanwhile, the rest of our code is predominantly written in C++, previously the optimal choice for high-performance demands.

We recognized the inherent risks associated with memory-unsafe languages and developed tools like sanitizers, which detect memory safety bugs dynamically, and fuzzers like AFL and libfuzzer, which proactively test the robustness and security of a software application by repeatedly feeding unexpected inputs. By open-sourcing these tools, we've empowered developers worldwide to reduce the likelihood of memory safety vulnerabilities in C and C++ codebases. Taking this commitment a step further, we provide continuous fuzzing to open-source projects through OSS-Fuzz, which helped get over 8800 vulnerabilities identified and subsequently fixed across 850 projects.

Today, with the emergence of high-performance memory-safe languages like Rust, coupled with a deeper understanding of the limitations of purely detection-based approaches, we are focused primarily on preventing the introduction of security vulnerabilities at scale.

Going forward: Google's two-pronged approach

Google's long-term strategy for tackling memory safety challenges is multifaceted, recognizing the need to address both existing codebases and future development, while maintaining the pace of business.

Our long-term objective is to progressively and consistently integrate memory-safe languages into Google's codebases while phasing out memory-unsafe code in new development. Given the amount of C++ code we use, we anticipate a residual amount of mature and stable memory-unsafe code will remain for the foreseeable future.

Graphic of memory-safe language growth as memory-unsafe code is hardened and gradually decreased over time.

Migration to Memory-Safe Languages (MSLs)

The first pillar of our strategy is centered on further increasing the adoption of memory-safe languages. These languages drastically drive down the risk of memory-related errors through features like garbage collection and borrow checking, embodying the same Safe Coding3 principles that successfully eliminated other vulnerability classes like cross-site scripting (XSS) at scale. Google has already embraced MSLs like Java, Kotlin, Go, and Python for a large portion of our code.

Our next target is to ramp up memory-safe languages with the necessary capabilities to address the needs of even more of our low-level environments where C++ has remained dominant. For example, we are investing to expand Rust usage at Google beyond Android and other mobile use cases and into our server, application, and embedded ecosystems. This will unlock the use of MSLs in low-level code environments where C and C++ have typically been the language of choice. In addition, we are exploring more seamless interoperability with C++ through Carbon, as a means to accelerate even more of our transition to MSLs.

In Android, which runs on billions of devices and is one of our most critical platforms, we've already made strides in adopting MSLs, including Rust, in sections of our network, firmware and graphics stacks. We specifically focused on adopting memory safety in new code instead of rewriting mature and stable memory-unsafe C or C++ codebases. As we've previously discussed, this strategy is driven by vulnerability trends as memory safety vulnerabilities were typically introduced shortly before being discovered.

As a result, the number of memory safety vulnerabilities reported in Android has decreased dramatically and quickly, dropping from more than 220 in 2019 to a projected 36 by the end of this year, demonstrating the effectiveness of this strategic shift. Given that memory-safety vulnerabilities are particularly severe, the reduction in memory safety vulnerabilities is leading to a corresponding drop in vulnerability severity, representing a reduction in security risk.

Risk Reduction for Memory-Unsafe Code

While transitioning to memory-safe languages is the long-term strategy, and one that requires investment now, we recognize the immediate responsibility we have to protect the safety of our billions of users during this process. This means we cannot ignore the reality of a large codebase written in memory-unsafe languages (MULs) like C and C++.

Therefore the second pillar of our strategy focuses on risk reduction & containment of this portion of our codebase. This incorporates:

  • C++ Hardening: We are retrofitting safety at scale in our memory-unsafe code, based on our experience eliminating web vulnerabilities. While we won't make C and C++ memory safe, we are eliminating sub-classes of vulnerabilities in the code we own, as well as reducing the risks of the remaining vulnerabilities through exploit mitigations.

    We have allocated a portion of our computing resources specifically to bounds-checking the C++ standard library across our workloads. While bounds-checking overhead is small for individual applications, deploying it at Google's scale requires significant computing resources. This underscores our deep commitment to enhancing the safety and security of our products and services. Early results are promising, and we'll share more details in a future post.

    In Chrome, we have also been rolling out MiraclePtr over the past few years, which effectively mitigated 57% of use-after-free vulnerabilities in privileged processes, and has been linked to a decrease of in-the-wild exploits.

  • Security Boundaries: We are continuing4 to strengthen critical components of our software infrastructure through expanded use of isolation techniques like sandboxing and privilege reduction, limiting the potential impact of vulnerabilities. For example, earlier this year, we shipped the beta release of our V8 heap sandbox and included it in Chrome's Vulnerability Reward Program.
  • Bug Detection: We are investing in bug detection tooling and innovative research such as Naptime and making ML-guided fuzzing as effortless and wide-spread as testing. While we are increasingly shifting towards memory safety by design, these tools and techniques remain a critical component of proactively identifying and reducing risks, especially against vulnerability classes currently lacking strong preventative controls.

    In addition, we are actively working with the semiconductor and research communities on emerging hardware-based approaches to improve memory safety. This includes our work to support and validate the efficacy of Memory Tagging Extension (MTE). Device implementations are starting to roll out, including within Google’s corporate environment. We are also conducting ongoing research into Capability Hardware Enhanced RISC Instructions (CHERI) architecture which can provide finer grained memory protections and safety controls, particularly appealing in security-critical environments like embedded systems.

    Looking ahead

    We believe it’s important to embrace the opportunity to achieve memory safety at scale, and that it will have a positive impact on the safety of the broader digital ecosystem. This path forward requires continuous investment and innovation to drive safety and velocity, and we remain committed to the broader community to walk this path together.

    We will provide future publications on memory safety that will go deeper into specific aspects of our strategy.

    Notes

    1. Anderson, J. Computer Security Technology Planning Study Vol II. ESD-TR-73-51, Vol. II, Electronic Systems Division, Air Force Systems Command, Hanscom Field, Bedford, MA 01730 (Oct. 1972).

      https://seclab.cs.ucdavis.edu/projects/history/papers/ande72.pdf  ↩

    2. https://www.memorysafety.org/docs/memory-safety/#how-common-are-memory-safety-vulnerabilities  ↩

    3. Kern, C. 2024. Developer Ecosystems for Software Safety. Commun. ACM 67, 6 (June 2024), 52–60. https://doi.org/10.1145/3651621 ↩

    4. Barth, Adam, et al. “The security architecture of the chromium browser." Technical report. Stanford University, 2008.

      https://seclab.stanford.edu/websec/chromium/chromium-security-architecture.pdf ↩

Virtual Escape; Real Reward: Introducing Google’s kvmCTF

Marios Pomonis, Software Engineer

Google is committed to enhancing the security of open-source technologies, especially those that make up the foundation for many of our products, like Linux and KVM. To this end we are excited to announce the launch of kvmCTF, a vulnerability reward program (VRP) for the Kernel-based Virtual Machine (KVM) hypervisor first announced in October 2023.




KVM is a robust hypervisor with over 15 years of open-source development and is widely used throughout the consumer and enterprise landscape, including platforms such as Android and Google Cloud. Google is an active contributor to the project and we designed kvmCTF as a collaborative way to help identify & remediate vulnerabilities and further harden this fundamental security boundary. 




Similar to kernelCTF, kvmCTF is a vulnerability reward program designed to help identify and address vulnerabilities in the Kernel-based Virtual Machine (KVM) hypervisor. It offers a lab environment where participants can log in and utilize their exploits to obtain flags. Significantly, in kvmCTF the focus is on zero day vulnerabilities and as a result, we will not be rewarding exploits that use n-days vulnerabilities. Details regarding the  zero day vulnerability will be shared with Google after an upstream patch is released to ensure that Google obtains them at the same time as the rest of the open-source community.  Additionally, kvmCTF uses the Google Bare Metal Solution (BMS) environment to host its infrastructure. Finally, given how critical a hypervisor is to overall system security, kvmCTF will reward various levels of vulnerabilities up to and including code execution and VM escape.




How it works


The environment consists of a bare metal host running a single guest VM. Participants will be able to reserve time slots to access the guest VM and attempt to perform a guest-to-host attack. The goal of the attack must be to exploit a zero day vulnerability in the KVM subsystem of the host kernel. If successful, the attacker will obtain a flag that proves their accomplishment in exploiting the vulnerability. The severity of the attack will determine the reward amount, which will be based on the reward tier system explained below. All reports will be thoroughly evaluated on a case-by-case basis.




The rewards tiers are the following:

  • Full VM escape: $250,000

  • Arbitrary memory write: $100,000

  • Arbitrary memory read: $50,000

  • Relative memory write: $50,000

  • Denial of service: $20,000

  • Relative memory read: $10,000




To facilitate the relative memory write/read tiers and partly the denial of service, kvmCTF offers the option of using a host with KASAN enabled. In that case, triggering a KASAN violation will allow the participant to obtain a flag as proof.




How to participate


To begin, start by reading the rules of the program. There you will find information on how to reserve a time slot, connect to the guest and obtain the flags, the mapping of the various KASAN violations with the reward tiers and instructions on how to report a vulnerability, send us your submission, or contact us on Discord.

Hacking for Defenders: approaches to DARPA’s AI Cyber Challenge

Oliver Chang, Jonathan Metzman, OSS-Fuzz and Alex Rebert, Security Engineering



The US Defense Advanced Research Projects Agency, DARPA, recently kicked off a two-year AI Cyber Challenge (AIxCC), inviting top AI and cybersecurity experts to design new AI systems to help secure major open source projects which our critical infrastructure relies upon. As AI continues to grow, it’s crucial to invest in AI tools for Defenders, and this competition will help advance technology to do so. 




Google’s OSS-Fuzz and Security Engineering teams have been excited to assist AIxCC organizers in designing their challenges and competition framework. We also playtested the competition by building a Cyber Reasoning System (CRS) tackling DARPA’s exemplar challenge. 




This blog post will share our approach to the exemplar challenge using open source technology found in Google’s OSS-Fuzz,  highlighting opportunities where AI can supercharge the platform’s ability to find and patch vulnerabilities, which we hope will inspire innovative solutions from competitors.



Leveraging OSS-Fuzz

AIxCC challenges focus on finding and fixing vulnerabilities in open source projects. OSS-Fuzz, our fuzz testing platform, has been finding vulnerabilities in open source projects as a public service for years, resulting in over 11,000 vulnerabilities found and fixed across 1200+ projects. OSS-Fuzz is free, open source, and its projects and infrastructure are shaped very similarly to AIxCC challenges. Competitors can easily reuse its existing toolchains, fuzzing engines, and sanitizers on AIxCC projects. Our baseline Cyber Reasoning System (CRS) mainly leverages non-AI techniques and has some limitations. We highlight these as opportunities for competitors to explore how AI can advance the state of the art in fuzz testing.



Fuzzing the AIxCC challenges

For userspace Java and C/C++ challenges, fuzzing with engines such as libFuzzer, AFL(++), and Jazzer is straightforward because they use the same interface as OSS-Fuzz.




Fuzzing the kernel is trickier, so we considered two options:



  • Syzkaller, an unsupervised coverage guided kernel fuzzer

  • A general purpose coverage guided fuzzer, such as AFL




Syzkaller has been effective at finding Linux kernel vulnerabilities, but is not suitable for AIxCC because Syzkaller generates sequences of syscalls to fuzz the whole Linux kernel, while AIxCC kernel challenges (exemplar) come with a userspace harness to exercise specific parts of the kernel. 




Instead, we chose to use AFL, which is typically used to fuzz userspace programs. To enable kernel fuzzing, we followed a similar approach to an older blog post from Cloudflare. We compiled the kernel with KCOV and KSAN instrumentation and ran it virtualized under QEMU. Then, a userspace harness acts as a fake AFL forkserver, which executes the inputs by executing the sequence of syscalls to be fuzzed. 




After every input execution, the harness read the KCOV coverage and stored it in AFL’s coverage counters via shared memory to enable coverage-guided fuzzing. The harness also checked the kernel dmesg log after every run to discover whether or not the input caused a KASAN sanitizer to trigger.




Some changes to Cloudflare’s harness were required in order for this to be pluggable with the provided kernel challenges. We needed to turn the harness into a library/wrapper that could be linked against arbitrary AIxCC kernel harnesses.




AIxCC challenges come with their own main() which takes in a file path. The main() function opens and reads this file, and passes it to the harness() function, which takes in a buffer and size representing the input. We made our wrapper work by wrapping the main() during compilation via $CC -Wl,--wrap=main harness.c harness_wrapper.a  




The wrapper starts by setting up KCOV, the AFL forkserver, and shared memory. The wrapper also reads the input from stdin (which is what AFL expects by default) and passes it to the harness() function in the challenge harness. 




Because AIxCC's harnesses aren't within our control and may misbehave, we had to be careful with memory or FD leaks within the challenge harness. Indeed, the provided harness has various FD leaks, which means that fuzzing it will very quickly become useless as the FD limit is reached.




To address this, we could either:


  • Forcibly close FDs created during the running of harness by checking for newly created FDs via /proc/self/fd before and after the execution of the harness, or

  • Just fork the userspace harness by actually forking in the forkserver. 




The first approach worked for us. The latter is likely most reliable, but may worsen performance.




All of these efforts enabled afl-fuzz to fuzz the Linux exemplar, but the vulnerability cannot be easily found even after hours of fuzzing, unless provided with seed inputs close to the solution.


Improving fuzzing with AI

This limitation of fuzzing highlights a potential area for competitors to explore AI’s capabilities. The input format being complicated, combined with slow execution speeds make the exact reproducer hard to discover. Using AI could unlock the ability for fuzzing to find this vulnerability quickly—for example, by asking an LLM to generate seed inputs (or a script to generate them) close to expected input format based on the harness source code. Competitors might find inspiration in some interesting experiments done by Brendan Dolan-Gavitt from NYU, which show promise for this idea.


Another approach: static analysis

One alternative to fuzzing to find vulnerabilities is to use static analysis. Static analysis traditionally has challenges with generating high amounts of false positives, as well as difficulties in proving exploitability and reachability of issues it points out. LLMs could help dramatically improve bug finding capabilities by augmenting traditional static analysis techniques with increased accuracy and analysis capabilities.


Proof of understanding (PoU)

Once fuzzing finds a reproducer, we can produce key evidence required for the PoU:

  1. The culprit commit, which can be found from git history bisection.

  2. The expected sanitizer, which can be found by running the reproducer to get the crash and parsing the resulting stacktrace.


Next step: “patching” via delta debugging

Once the culprit commit has been identified, one obvious way to “patch” the vulnerability is to just revert this commit. However, the commit may include legitimate changes that are necessary for functionality tests to pass. To ensure functionality doesn’t break, we could apply delta debugging: we progressively try to include/exclude different parts of the culprit commit until both the vulnerability no longer triggers, yet all functionality tests still pass.




This is a rather brute force approach to “patching.” There is no comprehension of the code being patched and it will likely not work for more complicated patches that include subtle changes required to fix the vulnerability without breaking functionality. 



Improving patching with AI

These limitations highlight a second area for competitors to apply AI’s capabilities. One approach might be to use an LLM to suggest patches. A 2024 whitepaper from Google walks through one way to build an LLM-based automated patching pipeline.




Competitors will need to address the following challenges:


  • Validating the patches by running crashes and tests to ensure the crash was prevented and the functionality was not impacted

  • Narrowing prompts to include only the functions present in the crashing stack trace, to fit prompt limitations

  • Building a validation step to filter out invalid patches




Using an LLM agent is likely another promising approach, where competitors could combine an LLM’s generation capabilities with the ability to compile and receive debug test failures or stacktraces iteratively.




Advancing security for everyone

Collaboration is essential to harness the power of AI as a widespread tool for defenders. As advancements emerge, we’ll integrate them into OSS-Fuzz, meaning that the outcomes from AIxCC will directly improve security for the open source ecosystem. We’re looking forward to the innovative solutions that result from this competition!

Time to challenge yourself in the 2024 Google CTF

Hlynur Gudmundsson, Software Engineer

It’s Google CTF time! Install your tools, commit your scripts, and clear your schedule. The competition kicks off on June 21 2024 6:00 PM UTC and runs through June 23 2024 6:00 PM UTC. Registration is now open at goo.gle/ctf.


Join the Google CTF (at goo.gle/ctf), a thrilling arena to showcase your technical prowess. The Google CTF consists of a set of computer security puzzles (or challenges) involving reverse-engineering, memory corruption, cryptography, web technologies, and more. Participants can use obscure security knowledge to find exploits through bugs and creative misuse, and with each completed challenge your team will earn points and move up through the ranks. 




The top 8 teams of the Google CTF will qualify for our Hackceler8 competition taking place in Málaga, Spain later this year as a part of our larger Escal8 event. Hackceler8 is our experimental esport-style hacking game competition, custom-made to mix CTF and speedrunning. 


Screenshot from last year’s Hackceler8 game




In the competition, teams need to find clever ways to abuse the game features to capture flags as quickly as possible.




Last year, teams assumed the role of Bartholomew (Mew for short), the fuzzy and adorable protagonist of Hackceler8 2023, set to defeat and overcome the evil rA.Ibbit taking over Silicon Valley! What adventures will Mew encounter this year? See the 2023 grand final to get a sense of the story and gameplay. The prize pool for this year’s Google CTF and Hackceler8 stands at more than $32,000.



Itching to get started early? Want to learn more, or get a leg up on the competition? Review challenges from previous years, including previous Hackceler8 matches, all open-sourced here. Or gain inspiration by binge watching hours of Hackceler8 2023 videos!




If you are just starting out in this space, check out our documentary H4CK1NG GOOGLE, it’s a great way to get acquainted with security. We also recommend checking out this year’s Beginner’s Quest that’ll be launching later this summer which will teach you some of the tools and tricks with simpler gamified challenges. For example, last year we explored hacking through time – you can use this to prepare for what’s yet to come.




Whether you’re a seasoned CTF player or just curious about cybersecurity and ethical hacking, we want to invite you to join us. Sign up for the Google CTF to expand your skill set, meet new friends in the security community, and even watch the pros in action. For the latest announcements, see goo.gle/ctf, subscribe to our mailing list, or follow us on Twitter @GoogleVRP. Interested in bug hunting for Google? Check out bughunters.google.com. See you there!


On Fire Drills and Phishing Tests

Matt Linton, Chaos Specialist


In the late 19th and early 20th century, a series of catastrophic fires in short succession led an outraged public to demand action from the budding fire protection industry. Among the experts, one initial focus was on “Fire Evacuation Tests”. The earliest of these tests focused on individual performance and tested occupants on their evacuation speed, sometimes performing the tests “by surprise” as though the fire drill were a real fire. These early tests were more likely to result in injuries to the test-takers than any improvement in survivability. It wasn’t until introducing better protective engineering - wider doors, push bars at exits, firebreaks in construction, lighted exit signs, and so on - that survival rates from building fires began to improve. As protections evolved over the years and improvements like mandatory fire sprinklers became required in building code, survival rates have continued to improve steadily, and “tests” have evolved into announced, advanced training and posted evacuation plans.




In this blog, we will analyze the modern practice of Phishing “Tests” as a cybersecurity control as it relates to industry-standard fire protection practices.


Modern “Phishing tests” strongly resemble the early “Fire tests”


Google currently operates under regulations (for example, FedRAMP in the USA) that require us to perform annual “Phishing Tests.” In these mandatory tests, the Security team creates and sends phishing emails to Googlers, counts how many interact with the email, and educates them on how to “not be fooled” by phishing. These exercises typically collect reporting metrics on sent emails and how many employees “failed” by clicking the decoy link. Usually, further education is required for employees who fail the exercise. Per the FedRAMP pen-testing guidance doc: “Users are the last line of defense and should be tested.”


These tests resemble the first “evacuation tests” that building occupants were once subjected to. They require individuals to recognize the danger, react individually in an ‘appropriate’ way, and are told that any failure is an individual failure on their part rather than a systemic issue. Worse, FedRAMP guidance requires companies to bypass or eliminate all systematic controls during the tests to ensure the likelihood of a person clicking on a phishing link is artificially maximized.


Among the harmful side effects of these tests:


  • There is no evidence that the tests result in fewer incidences of successful phishing campaigns;

    • Phishing (or more generically social engineering) remains a top vector for attackers establishing footholds at companies.

    • Research shows that these tests do not effectively prevent people from being fooled. This study with 14,000 participants showed a counterproductive effect of phishing tests, showing that “repeat clickers” will consistently fail tests despite recent interventions.

  • Some (e.g, FedRAMP) phishing tests require bypassing existing anti-phishing defenses. This creates an inaccurate perception of actual risks, allows penetration testing teams to avoid having to mimic actual modern attacker tactics, and creates a risk that the allowlists put in place to facilitate the test could be accidentally left in place and reused by attackers.

  • There has been a significantly increased load on Detection and Incident Response (D&R) teams during these tests, as users saturate them with thousands of needless reports. 

  • Employees are upset by them and feel security is “tricking them”, which degrades the trust with our users that is necessary for security teams to make meaningful systemic improvements and when we need employees to take timely actions related to actual security events.

  • At larger enterprises with multiple independent products, people can end up with numerous overlapping required phishing tests, causing repeated burdens.


But are users the last line of defense?

Training humans to avoid phishing or social engineering with a 100% success rate is a likely impossible task. There is value in teaching people how to spot phishing and social engineering so they can alert security to perform incident response. By ensuring that even a single user reports attacks in progress, companies can activate full-scope responses which are a worthwhile defensive control that can quickly mitigate even advanced attacks. But, much like the Fire Safety professional world has moved to regular pre-announced evacuation training instead of surprise drills, the information security industry should move toward training that de-emphasizes surprises and tricks and instead prioritizes accurate training of what we want staff to do the moment they spot a phishing email - with a particular focus on recognizing and reporting the phishing threat.



In short - we need to stop doing phishing tests and start doing phishing fire drills.


A “phishing fire drill” would aim to accomplish the following:

  • Educate our users about how to spot phishing emails

  • Inform the users on how to report phishing emails

  • Allow employees to practice reporting a phishing email in the manner that we would prefer, and

  • Collect useful metrics for auditors, such as:

    • The number of users who completed the practice exercise of reporting the email as a phishing email

    • The time between the email opening and the first report of phishing

    • Time of first escalation to the security team (and time delta)

    • Number of reports at 1 hour, 4 hours, 8 hours, and 24 hours post-delivery

Example 

When performing a phishing drill, someone would send an email announcing itself as a phishing email and with relevant instructions or specific tasks to perform. An example text is provided below.

Hello!  I am a Phishing Email. 


This is a drill - this is only a drill!


If I were an actual phishing email, I might ask you to log into a malicious site with your actual username or password, or I might ask you to run a suspicious command like <example command>. I might try any number of tricks to get access to your Google Account or workstation.


You can learn more about recognizing phishing emails at <LINK TO RESOURCE> and even test yourself to see how good you are at spotting them. Regardless of the form a phishing email takes, you can quickly report them to the security team when you notice they’re not what they seem.


To complete the annual phishing drill, please report me. To do that, <company-specific instructions on where to report phishing>.


Thanks for doing your part to keep <company> safe!


  1. Tricky. Phish, Ph.D





You can’t “fix” people, but you can fix the tools.

Phishing and Social Engineering aren’t going away as attack techniques. As long as humans are fallible and social creatures, attackers will have ways to manipulate the human factor. The more effective approach to both risks is a focused pursuit of secure-by-default systems in the long term, and a focus on investment in engineering defenses such as unphishable credentials (like passkeys) and implementing multi-party approval for sensitive security contexts throughout production systems. It’s because of investments in architectural defenses like these that Google hasn’t had to seriously worry about password phishing in nearly a decade.




Educating employees about alerting security teams of attacks in progress remains a valuable and essential addition to a holistic security posture. However, there’s no need to make this adversarial, and we don’t gain anything by “catching” people “failing” at the task. Let's stop engaging in the same old failed protections and follow the lead of more mature industries, such as Fire Protection, which has faced these problems before and already settled on a balanced approach. 

Your Google Account allows you to create passkeys on your phone, computer and security keys

Sriram Karra and Christiaan Brand, Google product managers

Last year, Google launched passkey support for Google Accounts. Passkeys are a new industry standard that give users an easy, highly secure way to sign-in to apps and websites. Today, we announced that passkeys have been used to authenticate users more than 1 billion times across over 400 million Google Accounts.




As more users encounter passkeys, we’re often asked questions about how they relate to security keys, how Google Workspace administrators can configure passkeys for the user accounts that they manage, and how they relate to the Advanced Protection Program (APP). This post will seek to clarify these topics.




Passkeys and security keys

Passkeys are an evolution of security keys, meaning users get the same security benefits, but with a much simplified experience. Passkeys can be used in the Google Account sign-in process in many of the same ways that security keys have been used in the past — in fact, you can now choose to store your passkey on your security key. This provides users with three key benefits:




  • Stronger security. Users typically authenticate with passkeys by entering their device’s screen lock PIN, or using a biometric authentication method, like a fingerprint or a face scan. By storing the passkey on a security key, users can ensure that passkeys are only available when the security key is plugged into their device, creating a stronger security posture.


  • Flexible portability. Today, users rely on password managers to make passkeys available across all of their devices. Security keys provide an alternate way to use your passkeys across your devices: by bringing your security keys with you.


  • Simpler sign-in. Passkeys can act as a first- and second-factor, simultaneously. By creating a passkey on your security key, you can skip entering your password. This replaces your remotely stored password with the PIN you used to unlock your security key, which improves user security. (If you prefer to continue using your password in addition to using a passkey, you can turn off “Skip password when possible” in your Google Account security settings.)




Passkeys bring strong and phishing-resistant authentication technology to a wider user base, and we’re excited to offer this new way for passkeys to meet more user needs.




Google Workspace admins have additional controls and choice

Google Workspace accounts have a domain level “Allow users to skip passwords at sign-in by using passkeys” setting which is off by default, and overrides the corresponding user-level configuration. This retains the need for a user’s password in addition to presenting a passkey. Admins can also change that setting and allow users to sign-in with just a passkey.




When the domain-level setting is off, end users will still see a “use a security key” button on their “passkeys and security keys” page, which will attempt to enroll any security key for use as a second factor only. This action will not require the user to set up a PIN for their security key during registration. This is designed to give enterprise customers who have deployed legacy security keys additional time to make the change to passkeys, with or without a password.




Passkeys for Advanced Protection Program (APP) users

Since the introduction of passkeys in 2023, users enrolled in APP have been able to add any passkey to their account and use it to sign in. However users are still required to present two security keys when enrolling into the program. We will be updating the enrollment process soon to enable a user with any passkey to enroll in APP. By allowing any passkey to be used (rather than only hardware security keys) we expect to reach more high risk users who need advanced protection, while maintaining phishing-resistant authentication.

Accelerating incident response using generative AI

Lambert Rosique and Jan Keller, Security Workflow Automation, and Diana Kramer, Alexandra Bowen and Andrew Cho, Privacy and Security Incident Response

Introduction

As security professionals, we're constantly looking for ways to reduce risk and improve our workflow's efficiency. We've made great strides in using AI to identify malicious content, block threats, and discover and fix vulnerabilities. We also published the Secure AI Framework (SAIF), a conceptual framework for secure AI systems to ensure we are deploying AI in a responsible manner. 




Today we are highlighting another way we use generative AI to help the defenders gain the advantage: Leveraging LLMs (Large Language Model) to speed-up our security and privacy incidents workflows.




Incident management is a team sport. We have to summarize security and privacy incidents for different audiences including executives, leads, and partner teams. This can be a tedious and time-consuming process that heavily depends on the target group and the complexity of the incident. We estimate that writing a thorough summary can take nearly an hour and more complex communications can take multiple hours. But we hypothesized that we could use generative AI to digest information much faster, freeing up our incident responders to focus on other more critical tasks - and it proved true. Using generative AI we could write summaries 51% faster while also improving the quality of them. 



Our incident response approach

When suspecting a potential data incident, for example,we follow a rigorous process to manage it. From the identification of the problem, the coordination of experts and tools, to its resolution and then closure. At Google, when an incident is reported, our Detection & Response teams work to restore normal service as quickly as possible, while meeting both regulatory and contractual compliance requirements. They do this by following the five main steps in the Google incident response program:



  1. Identification: Monitoring security events to detect and report on potential data incidents using advanced detection tools, signals, and alert mechanisms to provide early indication of potential incidents.

  2. Coordination: Triaging the reports by gathering facts and assessing the severity of the incident based on factors such as potential harm to customers, nature of the incident, type of data that might be affected, and the impact of the incident on customers. A communication plan with appropriate leads is then determined.

  3. Resolution: Gathering key facts about the incident such as root cause and impact, and integrating additional resources as needed to implement necessary fixes as part of remediation.

  4. Closure: After the remediation efforts conclude, and after a data incident is resolved, reviewing the incident and response to identify key areas for improvement.

  5. Continuous improvement: Is crucial for the development and maintenance of incident response programs. Teams work to improve the program based on lessons learned, ensuring that necessary teams, training, processes, resources, and tools are maintained.




Google’s Incident Response Process diagram flow



Leveraging generative AI 

Our detection and response processes are critical in protecting our billions of global users from the growing threat landscape, which is why we’re continuously looking for ways to improve them with the latest technologies and techniques. The growth of generative AI has brought with it incredible potential in this area, and we were eager to explore how it could help us improve parts of the incident response process. We started by leveraging LLMs to not only pioneer modern approaches to incident response, but also to ensure that our processes are efficient and effective at scale. 




Managing incidents can be a complex process and an additional factor is effective internal communication to leads, executives and stakeholders on the threats and status of incidents. Effective communication is critical as it properly informs executives so that they can take any necessary actions, as well as to meet regulatory requirements. Leveraging LLMs for this type of communication can save significant time for the incident commanders while improving quality at the same time.



Humans vs. LLMs

Given that LLMs have summarization capabilities, we wanted to explore if they are able to generate summaries on par, or as well as humans can. We ran an experiment that took 50 human-written summaries from native and non-native English speakers, and 50 LLM-written ones with our finest (and final) prompt, and presented them to security teams without revealing the author.




We learned that the LLM-written summaries covered all of the key points, they were rated 10% higher than their human-written equivalents, and cut the time necessary to draft a summary in half. 




Comparison of human vs LLM content completeness




Comparison of human vs LLM writing styles

Managing risks and protecting privacy

Leveraging generative AI is not without risks. In order to mitigate the risks around potential hallucinations and errors, any LLM generated draft must be reviewed by a human. But not all risks are from the LLM -  human misinterpretation of a fact or statement generated by the LLM can also happen. That is why it’s important to ensure there is human accountability, as well as to monitor quality and feedback over time. 




Given that our incidents can contain a mixture of confidential, sensitive, and privileged data, we had to ensure we built an infrastructure that does not store any data. Every component of this pipeline - from the user interface to the LLM to output processing - has logging turned off. And, the LLM itself does not use any input or output for re-training. Instead, we use metrics and indicators to ensure it is working properly. 



Input processing

The type of data we process during incidents can be messy and often unstructured: Free-form text, logs, images, links, impact stats, timelines, and code snippets. We needed to structure all of that data so the LLM “knew” which part of the information serves what purpose. For that, we first replaced long and noisy sections of codes/logs by self-closing tags (<Code Section/> and <Logs/>) both to keep the structure while saving tokens for more important facts and to reduce risk of hallucinations.




During prompt engineering, we refined this approach and added additional tags such as <Title>, <Actions Taken>, <Impact>, <Mitigation History>, <Comment> so the input’s structure becomes closely mirrored to our incident communication templates. The use of self-explanatory tags allowed us to convey implicit information to the model and provide us with aliases in the prompt for the guidelines or tasks, for example by stating “Summarize the <Security Incident>”.



Sample {incident} input

Prompt engineering

Once we added structure to the input, it was time to engineer the prompt. We started simple by exploring how LLMs can view and summarize all of the current incident facts with a short task:


Caption: First prompt version




Limits of this prompt:

  • The summary was too long, especially for executives trying to understand the risk and impact of the incident

  • Some important facts were not covered, such as the incident’s impact and its mitigation

  • The writing was inconsistent and not following our best practices such as “passive voice”, “tense”, “terminology” or “format”

  • Some irrelevant incident data was being integrated into the summary from email threads

  • The model struggled to understand what the most relevant and up-to-date information was




For version 2, we tried a more elaborate prompt that would address the problems above: We told the model to be concise and we explained what a well-written summary should be: About the main incident response steps (coordination and resolution).


Second prompt version




Limits of this prompt:

  • The summaries still did not always succinctly and accurately address the incident in the format we were expecting

  • At times, the model lost sight of the task or did not take all the guidelines into account

  • The model still struggled to stick to the latest updates

  • We noticed a tendency to draw conclusions on hypotheses with some minor hallucinations




For the final prompt, we inserted 2 human-crafted summary examples and introduced a <Good Summary> tag to highlight high quality summaries but also to tell the model to immediately start with the summary without first repeating the task at hand (as LLMs usually do).



Final prompt


This produced outstanding summaries, in the structure we wanted, with all key points covered, and almost without any hallucinations.



Workflow integration

In integrating the prompt into our workflow, we wanted to ensure it was complementing the work of our teams, vs. solely writing communications. We designed the tooling in a way that the UI had a ‘Generate Summary’ button, which would pre-populate a text field with the summary that the LLM proposed. A human user can then either accept the summary and have it added to the incident, do manual changes to the summary and accept it, or discard the draft and start again. 


UI showing the ‘generate draft’ button and LLM proposed summary around a fake incident 



Quantitative wins

Our newly-built tool produced well-written and accurate summaries, resulting in 51% time saved, per incident summary drafted by an LLM, versus a human.




Time savings using LLM-generated summaries (sample size: 300)



The only edge cases we have seen were around hallucinations when the input size was small in relation to the prompt size. In these cases, the LLM made up most of the summary and key points were incorrect. We fixed this programmatically: If the input size is smaller than 200 tokens, we won’t call the LLM for a summary and let the humans write it. 



Evolving to more complex use cases: Executive updates

Given these results, we explored other ways to apply and build upon the summarization success and apply it to more complex communications. We improved upon the initial summary prompt and ran an experiment to draft executive communications on behalf of the Incident Commander (IC). The goal of this experiment was to ensure executives and stakeholders quickly understand the incident facts, as well as allow ICs to relay important information around incidents. These communications are complex because they go beyond just a summary - they include different sections (such as summary, root cause, impact, and mitigation), follow a specific structure and format, as well as adhere to writing best practices (such as neutral tone, active voice instead of passive voice, minimize acronyms).




This experiment showed that generative AI can evolve beyond high level summarization and help draft complex communications. Moreover, LLM-generated drafts, reduced time ICs spent writing executive summaries by 53% of time, while delivering at least on-par content quality in terms of factual accuracy and adherence to writing best practices. 



What’s next

We're constantly exploring new ways to use generative AI to protect our users more efficiently and look forward to tapping into its potential as cyber defenders. For example, we are exploring using generative AI as an enabler of ambitious memory safety projects like teaching an LLM to rewrite C++ code to memory-safe Rust, as well as more incremental improvements to everyday security workflows, such as getting generative AI to read design documents and issue security recommendations based on their content.

Google Public DNS’s approach to fight against cache poisoning attacks

Tianhao Chi and Puneet Sood, Google Public DNS


The Domain Name System (DNS) is a fundamental protocol used on the Internet to translate human-readable domain names (e.g., www.example.com) into numeric IP addresses (e.g., 192.0.2.1) so that devices and servers can find and communicate with each other. When a user enters a domain name in their browser, the DNS resolver (e.g. Google Public DNS) locates the authoritative DNS nameservers for the requested name, and queries one or more of them to obtain the IP address(es) to return to the browser.

When DNS was launched in the early 1980s as a trusted, content-neutral infrastructure, security was not yet a pressing concern, however, as the Internet grew DNS became vulnerable to various attacks. In this post, we will look at DNS cache poisoning attacks and how Google Public DNS addresses the risks associated with them.

DNS Cache Poisoning Attacks

DNS lookups in most applications are forwarded to a caching resolver (which could be local or an open resolver like. Google Public DNS). The path from a client to the resolver is usually on a local network or can be protected using encrypted transports like DoH, DoT. The resolver queries authoritative DNS servers to obtain answers for user queries. This communication primarily occurs over UDP, an insecure connectionless protocol, in which messages can be easily spoofed including the source IP address. The content of DNS queries may be sufficiently predictable that even an off-path attacker can, with enough effort, forge responses that appear to be from the queried authoritative server. This response will be cached if it matches the necessary fields and arrives before the authentic response. This type of attack is called a cache poisoning attack, which can cause great harm once successful. According to RFC 5452, the probability of success is very high without protection. Forged DNS responses can lead to denial of service, or may even compromise application security. For an excellent introduction to cache poisoning attacks, please see “An Illustrated Guide to the Kaminsky DNS Vulnerability”.

Cache poisoning mitigations in Google Public DNS

Improving DNS security has been a goal of Google Public DNS since our launch in 2009. We take a multi-pronged approach to protect users against DNS cache-poisoning attacks. There is no silver bullet or countermeasure that entirely solves the problem, but in combination they make successful attacks substantially more difficult.


RFC 5452 And DNS Cookies

We have implemented the basic countermeasures outlined in RFC 5452 namely randomizing query source ports and query IDs. But these measures alone are not sufficient (see page 8 of our OARC 38 presentation).

We have therefore also implemented support for RFC 7873 (DNS Cookies) which can make spoofing impractical if it’s supported by the authoritative server. Measurements indicate that the DNS Cookies do not provide sufficient coverage, even though around 40% of nameservers by IP support DNS Cookies, these account for less than 10% of overall query volume. In addition, many non-compliant nameservers return incorrect or ambiguous responses for queries with DNS Cookies, which creates further deployment obstacles. For now, we’ve enabled DNS Cookies through manual configuration, primarily for selected TLD zones.

Case Randomization (0x20)

The query name case randomization mechanism, originally proposed in a March 2008 draft “Use of Bit 0x20 in DNS Labels to Improve Transaction Identity”, however, is highly effective, because all but a small minority of nameservers are compatible with query name case randomization. We have been performing case randomization of query names since 2009 to a small set of chosen nameservers that handle only a minority of our query volume. 

In 2022 we started work on enabling case randomization by default, which when used, the query name in the question section is randomized and the DNS server’s response is expected to match the case-randomized query name exactly in the request. For example, if “ExaMplE.CoM” is the name sent in the request, the name in the question section of the response must also be “ExaMplE.CoM” rather than, e.g., “example.com.” Responses that fail to preserve the case of the query name may be dropped as potential cache poisoning attacks (and retried over TCP).

We are happy to announce that we’ve already enabled and deployed this feature globally by default. It covers over 90% of our UDP traffic to nameservers, significantly reducing the risk of cache poisoning attacks.

Meanwhile, we maintain an exception list and implement fallback mechanisms to prevent potential issues with non-conformant nameservers. However we strongly recommend that nameserver implementations preserve the query case in the response.

DNS-over-TLS

In addition to case randomization, we’ve deployed DNS-over-TLS to authoritative nameservers (ADoT), following procedures described in RFC 9539 (Unilateral Opportunistic Deployment of Encrypted Recursive-to-Authoritative DNS). Real world measurements show that ADoT has a higher success rate and comparable latency to UDP. And ADoT is in use for around 6% of egress traffic. At the cost of some CPU and memory, we get both security and privacy for nameserver queries without DNS compliance issues.

Summary

Google Public DNS takes security of our users seriously. Through multiple countermeasures to cache poisoning attacks, we aim to provide a more secure and reliable DNS resolution service, enhancing the overall Internet experience for users worldwide. With the measures described above we are able to provide protection against passive attacks for over 90% of authoritative queries.


To enhance DNS security, we recommend that DNS server operators support one or more of the  security mechanisms described here. We are also working with the DNS community to improve DNS security. Please see our presentations at DNS-OARC 38 and 40 for more technical details.