Fuzzing is an automated methodology that attempts to identify bugs within software, especially security vulnerabilities. We on the Security Research Team at ServiceNow utilize the fuzzer Jazzer internally for specific use cases.

Jazzer is a coverage-guided, in-process fuzzer for the JVM platform developed by Code Intelligence. It is based on libFuzzer and brings many of its instrumentation-powered mutation features to the JVM.1

While Code Intelligence no longer maintains Jazzer, the fuzzer remains functional and available for use from GitHub.

When it comes to large-scale enterprise Java applications, throwing a fuzzer at some code is unlikely to identify interesting vulnerabilities. Instead, you need to think about application state and what sort of state indicates a potential security issue. After that, it’s a matter of wrangling a fuzzer to effectively identify that state.

Unfortunately, like many technical tools, the documentation for Jazzer is a little lighter than we’d like, and examples across the Internet leave a bit to be desired complexity-wise. In this post, we will share and compile some lessons learned and advanced tips for effectively making use of Jazzer with an enterprise Java application.

Extra Fuzz Target Methods

If you stick to the documentation on GitHub or official blogs from Code Insight, you might easily overlook some helpful methods available to you when creating fuzz targets.

The optional fuzzerInitialize method runs before iterating through calls of fuzzerTestOneInput. This initialization is especially helpful when fuzzing larger applications. You might need to prepare some of the normal application, but you don’t need to actually run the entire thing. Early initialization steps might also avoid unnecessary overhead on each call to fuzzerTestOneInput. A basic example is included below.

public class ExampleApp{
    
  public static void fuzzerInitialize() {
    System.setProperty("test.property.1", "false");
    System.setProperty("test.db.name", "db_master");
    TestProperties.loadSystemProperties();
    TestProperties.reload();

    new TestEnvironment();
  }
  
  public static void fuzzerTestOneInput(FuzzedDataProvider data) {
    String c = data.consumeString(100);
    
    //Do something....
  }
}

In this example, we set some system properties, load those properties within the application context, and then prepare a mock application state with new TestEnvironment(). All of this will wildly differ depending on the minimum components your application require to run for fuzzing.

In a similar vein, optional method fuzzerTearDown can be used to clean up resources after the completion of a fuzzing run.

Method Hooks & Classpath

With a memory safe language like Java, you’re unlikely to be interested in fuzzing for memory corruption vulnerabilities. Additionally, an enterprise Java application is less likely to be as affected by denial of service conditions. So you’re generally going to be most interested in fuzzing for particular application states that indicate specific types of vulnerabilities only relevant in the context of your application.

Code Intelligence provides a decent resource on using method hooks to detect these sort of application states. Still, the devil is in the details.

Method hooks themselves must be built and provided in a separate JAR file from the fuzzing target. It’s an easy to miss detail, but it prevents all sorts of access and classpath issues down the road. So a basic run example might look like this:

java --cp=jazzer_standalone.jar:target.jar:hook.jar \
com.code_intelligence.jazzer.Jazzer \
--target_class=sn.securityresearch.fuzzing.target.ExampleApp \
--custom_hooks=sn.securityresearch.fuzzing.hooks.ExampleHook

Second, during runtime, method hooks themselves appear to be limited in scope to system level JARs. This occurs whether or not the method hook itself was compiled with other JARs in its classpath and those same JARs are available on the classpath provided to Jazzer.

So, when you’re attempting to peek at your application state from a method hook, you might find yourself limited in the classes you would normally use to do so within the application itself. What’s the alternative then? Reflection!

In a slightly contrived example, imagine we have an application that allows users to run certain types of script. Low privilege users get access to a restricted script environment, normally the UserContext class, which contains a limited set of APIs. Admin level users get access to the entire script environment, normally the AdminContext class, which contains the full set of APIs. We’ve created a before-type method hook at the actual lower level script API invoker,APIRunner.handleInvocation to attempt to detect scenarios where certain script has escaped the low privilege context UserContext and entered the privileged AdminContext.

The normal method for APIRunner.handleInvocation looks like the following.

public class APIRunner {
    
    protected static Object handleInvocation(ScriptAPI api, Runtime rt, Context cx) {
        // Invoke API...
    }
}

Within the application itself, it would be trivial to poke at the classes Runtime and Context and their respective fields to identify an escalation from a restricted to privileged environment. We simply could grab the type of cx and then compare with the callContext field from rt. Instead, since our method hook is unaware of Runtime and Context, we must use reflection to tease out the same information, resulting in a slightly less readable method hook like below.

public class ExampleHook {

    @MethodHook(
            type = HookType.BEFORE,
            targetClassName = "com.securityresearch.script.APIRunner",
            targetMethod = "handleInvocation"
    )
    public static void checkContextEscape(MethodHandle method, Object thisObject, Object[] arguments, int hookId) {
        detectContextEscape(arguments);
    }

    private static void detectContextEscape(Object[] arguments) {

        String currentContext = null;
        String callContext = null;

        try {
            // Current Context
            currentContext = arguments[2].getClass().getSimpleName();

            // Calling Context
            Class c = arguments[1].getClass();
            Field f = c.getDeclaredField("callContext");
            f.setAccessible(true);
            callContext = f.get(arguments[1]).getClass().getSimpleName();

        } catch (Exception ignore) {}

        if (currentContext.equals("AdminContext") && callContext.equals("UserContext")) {
            Jazzer.reportFindingFromHook(new FuzzerSecurityIssueCritical("Possible Context Escape"));
        }
    }
}

The @MethodHook annotation tells Jazzer the class and method to target and first execute the custom method checkContextEscape. Since we do not directly have access to the Context object, we use reflection, arguments[2].getClass().getSimpleName(), to identify the class of the cx argument. We similarly parse out the call context by inspecting the callContext field of the Runtime argument rt. We then compare the current and call context values we identified through reflection to see if a privilege escalation occurs. Jazzer.reportFindingFromHook is used to report the state as a finding.

Timeouts

LibFuzzer itself reports timeouts, stored on disk alongside findings as “timeout-<sha1>”. Timeouts are defined with a default value of 1200 seconds (20 minutes), though this can be tweaked when calling Jazzer and passing -timeout=120 to libFuzzer. Unfortunately, timeouts are terminal and may kill a fuzzing run if encountered. This may be desirable for some folks, but we’ve often found the need to gracefully handle timeouts ourselves in the context of a larger applications and extended fuzzing runs.

There are a couple options in this scenario. LibFuzzer has a few experimental settings you may configure when calling Jazzer. In particular, fork mode allows you to spin up parallel processes for fuzzing, with the side effect of crash resistance. By default, timeouts are ignored, but this can be configured with -ignore_timeouts flag. So a run might look like this:

java --cp=jazzer_standalone.jar:target.jar:hook.jar \
com.code_intelligence.jazzer.Jazzer \
-fork=4 \
--target_class=sn.securityresearch.fuzzing.target.ExampleApp \
--custom_hooks=sn.securityresearch.fuzzing.hooks.ExampleHook

Alternatively, it’s possible to detect and handle timeouts yourself within the context of the fuzz target. This is tricky and not typically going to be necessary outside niche use cases. We can utilize the standard Java Future API to detect and throw our own Jazzer security findings for timeouts. In the example below, our fuzzerTestOneInput method merely serves as a wrapper to pass off our fuzzer input to a separate worker, ExampleApplicationWorker.

public class ExampleApp{
    
    public static void fuzzerTestOneInput(FuzzedDataProvider data) {
        String c = data.consumeString(100);

        ExecutorService executor = Executors.newSingleThreadExecutor();
        Future future = executor.submit(new ExampleApplicationWorker(c));

        try {
            future.get(30, TimeUnit.SECONDS);
        } catch (TimeoutException e) {
            future.cancel(true);
            throw new FuzzerSecurityIssueCritical("Potential Timeout");
        } catch (Exception ignore) {
        } finally {
            executor.shutdownNow();
        }
    }
}

The standard fuzzerTestOneInput method takes in a string from the fuzzer itself, and passes it to a separate thread, ExampleApplicationWorker. The worker thread is monitored to ensure it finishes in 30 seconds, or the thread is terminated and a finding thrown.

Within ExampleApplicationWorker, we place the functionality that resides in a typical fuzz target.

public class ExampleApplicationWorker implements Runnable{

    private String input;

    public ExampleApplicationWorker(String input) {
        this.input = input;
    }

    @Override
    public void run() {
        // Do Something....
    }
}

Jazzer continues to functional normally, method hooks included, and we can handle timeouts how we wish, though that might mean more logging or cleaning up application resources in true timeout scenarios.

Advanced Run Options

Both Jazzer itself and libFuzzer contain even more options useful for customizing your fuzzing runs. We’ve included a few we found particularly useful.

Disable Existing Hooks

As we mentioned earlier, fuzzing becomes particularly useful in the context of your own application and detecting vulnerable states, which means working with your own custom method hooks. Jazzer, by default, comes with several existing method hooks it will run with by default. You may find these useful; you may not. If you’re looking to optimize your fuzzing run and hone in on your own findings, you can disable these existing method hooks with the --disabled_hooks flag. In particular, we found the RegexInjection and SqlInjection method hooks to be troublesome for our use cases and application, resulting in false positives or unnecessarily consuming fuzzing resources.

java --cp=jazzer_standalone.jar:target.jar:hook.jar \
com.code_intelligence.jazzer.Jazzer \
--target_class=sn.securityresearch.fuzzing.target.ExampleApp \
--custom_hooks=sn.securityresearch.fuzzing.hooks.ExampleHook \
--disabled_hooks=com.code_intelligence.jazzer.sanitizers.RegexInjection:com.code_intelligence.jazzer.sanitizers.SqlInjection

You can see some of the other method hooks here.

Instrumentation Includes

To hone in on your own application, or even better, specific focus areas of your own application, you may find it useful to limit instrumentation using the --instrumentation_includes flag. If you have a large enterprise Java application, you really don’t want or need Jazzer instrumenting your entire codebase, including potentially hundreds of dependencies.

java --cp=jazzer_standalone.jar:target.jar:hook.jar \
com.code_intelligence.jazzer.Jazzer \
--target_class=sn.securityresearch.fuzzing.target.ExampleApp \
--custom_hooks=sn.securityresearch.fuzzing.hooks.ExampleHook \
--disabled_hooks=com.code_intelligence.jazzer.sanitizers.RegexInjection:com.code_intelligence.jazzer.sanitizers.SqlInjection \
--instrumentation_includes=sn.securityresearch.**:com.snc.**

Keep Going

If you’re going through the work of creating a fairly complicated fuzzing target and method hook, you’re probably not going to want to stop fuzzing runs after a single crash/finding. We’d recommend running Jazzer with the --keep_going flag. Jazzer is more than capable enough to not report duplicate crashes, assuming the stack traces match.

java --cp=jazzer_standalone.jar:target.jar:hook.jar \
com.code_intelligence.jazzer.Jazzer \
--target_class=sn.securityresearch.fuzzing.target.ExampleApp \
--custom_hooks=sn.securityresearch.fuzzing.hooks.ExampleHook \
--disabled_hooks=com.code_intelligence.jazzer.sanitizers.RegexInjection:com.code_intelligence.jazzer.sanitizers.SqlInjection \
--instrumentation_includes=sn.securityresearch.**:com.snc.** \
--keep_going=10

Final Thoughts

We hope this post will save at least a few folks some time and improve their fuzzing performance with Jazzer going forward. If you take anything from this post, it’s that running out of the box Jazzer with the default configurations may not be the best to fuzz sizable Java applications for meaningful vulnerabilities. However, by considering the specific application state that indicates a potential vulnerability and using the techniques outlined here, you can use Jazzer to more effectively identify actionable security issues.

  1. https://github.com/CodeIntelligenceTesting/jazzer ↩