Cracking Java byte-code encryption

Doing this interception is not hard at all. In fact, breaking my own protection scheme takes less time that it took to implement it! First, I get the source for java.lang.ClassLoader for my Java 2 Platform, Standard Development Kit (J2SDK) and modify defineClass(String, byte[], int, int, ProtectionDomain) to have some additional class logging:

    ...
        c = defineClass0(name, b, off, len, protectionDomain);
        
        // Intercept classes defined by the system loader and its children:
        if (isAncestor (getSystemClassLoader ().getParent ()))
        {
            // Choose your own dump location here [use an absolute pathname]:
            final File parentDir = new File ("c:/TEMP/classes/");
            File dump = new File (parentDir,
                name.replace ('.', File.separatorChar) + "[" +
                getClass ().getName () + "@" + 
                Long.toHexString (System.identityHashCode (this)) + "].class");
                            
            dump.getParentFile ().mkdirs ();
            
            FileOutputStream out = null;
            try
            {
                out = new FileOutputStream (dump);                
                out.write (b, off, len);
            }
            catch (IOException ioe)
            {
                ioe.printStackTrace (System.out);
            }
            finally
            {
                if (out != null) try { out.close (); } catch (Exception ignore) {}
            }
        }
    ...

Note that the added lines are guarded by an if statement that filters for classes loaded by the system (-classpath) and its descendant classloaders. Also, the logging occurs only if defineClass() does not fail. Finally, because it is not inconceivable that more than one ClassLoader instance might load a class, I disambiguate the results by embedding the classloader identity in the dumped filename.

The final step is to temporarily replace rt.jar used by my Java Runtime Environment (JRE) (note that it could be different from the one used by J2SDK) with one that contains my doctored java.lang.ClassLoader implementation. Or, you could use the -Xbootclasspath/p option.

I run the encrypted application again and voila, I have recovered all my unencrypted and thus easily decompilable .class definitions. And note I have not used any knowledge of EncryptedClassLoader inner workings to accomplish this.

Observe that if I did not want to instrument a system class, I could have used other options such as a custom JVMPI agent that handles JVMPI_EVENT_CLASS_LOAD_HOOK events.

Lessons learned

I hope you found this quick excursion into details of Java classloading interesting. An important point to realize is that some tools on the market promise solutions to Java's easy reverse engineering problem through class encryption, and you should think twice before buying one. Until JVM architecture changes to, say, support class decoding inside native code, you will be better off with traditional obfuscators that perform byte-code transformations.

There is another, more useful side to such tricks as well: debugging Java classloading. Being able to get a load trace for a custom classloader could be invaluable, especially if you are trying to track down the cause of a classloader constraint violation (more on this in future Java Q&A posts). So, maybe Java was born to be a language for pure open source development after all? Of course, other architectures based on platform-neutral byte code (such as .Net) are equally prone to reverse engineering. I will leave you with this thought for now.

This story, "Cracking Java byte-code encryption" was originally published by JavaWorld.