Jonathan Lalou's blog

Posts Tagged ‘OutOfMemoryError’

If you’re working with the AWS SDK for Java and you’ve seen the following log message:

WARN --- AmazonS3Client : No content length specified for stream data. Stream contents will be buffered in memory and could result in out of memory errors.

…you’re not alone. This warning might seem harmless at first, but it can lead to serious issues, especially in production environments.

What’s Really Happening?

This message appears when you upload a stream to Amazon S3 without explicitly setting the content length in the request metadata.

When that happens, the SDK doesn’t know how much data it’s about to upload, so it buffers the entire stream into memory before sending it to S3. If the stream is large, this could lead to:

• Excessive memory usage
• Slow performance
• OutOfMemoryError crashes

✅ How to Fix It

Whenever you upload a stream, make sure you calculate and set the content length using ObjectMetadata.

Example with Byte Array:

byte[] bytes = ...; // your content
ByteArrayInputStream inputStream = new ByteArrayInputStream(bytes);

ObjectMetadata metadata = new ObjectMetadata();
metadata.setContentLength(bytes.length);

PutObjectRequest request = new PutObjectRequest(bucketName, key, inputStream, metadata);
s3Client.putObject(request);

Example with File:

File file = new File("somefile.txt");
FileInputStream fileStream = new FileInputStream(file);

ObjectMetadata metadata = new ObjectMetadata();
metadata.setContentLength(file.length());

PutObjectRequest request = new PutObjectRequest(bucketName, key, fileStream, metadata);
s3Client.putObject(request);

What If You Don’t Know the Length?

Sometimes, you can’t know the content length ahead of time (e.g., you’re piping data from another service). In that case:

• Write the stream to a ByteArrayOutputStream first (good for small data)
• Use the S3 Multipart Upload API to stream large files without specifying the total size

Conclusion

Always set the content length when uploading to S3 via streams. It’s a small change that prevents large-scale problems down the road.

By taking care of this up front, you make your service safer, more memory-efficient, and more scalable.

Got questions or dealing with tricky S3 upload scenarios? Drop them in the comments!

Case:

I have to redeploy many times a day a WAR on a WebLogic 10 server, using Hotspot (Sun JVM) with Java 6. After some cycles deploy/undeploy, Hotspot crashes with the following error:

java.lang.OutOfMemoryError: PermGen space

Complete stacktrace:

2010-08-17 11:26:56,718 ERROR context.ContextLoader               - Context initialization failed
java.lang.IllegalStateException: Unable to load Java 1.5 dependent class [org.springframework.beans.factory.annotation.QualifierAnnotationAutowireCandidateResolver]
        at org.springframework.beans.factory.support.AutowireUtils.createAutowireCandidateResolver(AutowireUtils.java:125)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.<init>(DefaultListableBeanFactory.java:103)
        at org.springframework.context.support.AbstractRefreshableApplicationContext.createBeanFactory(AbstractRefreshableApplicationContext.java:176)
        at org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:121)
        at org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:423)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:353)
        at org.springframework.web.context.ContextLoader.createWebApplicationContext(ContextLoader.java:255)
        at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:199)
        at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:45)
        at weblogic.servlet.internal.EventsManager$FireContextListenerAction.run(EventsManager.java:481)
        at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
        at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
        at weblogic.servlet.internal.EventsManager.notifyContextCreatedEvent(EventsManager.java:181)
        at weblogic.servlet.internal.WebAppServletContext.preloadResources(WebAppServletContext.java:1863)
        at weblogic.servlet.internal.WebAppServletContext.start(WebAppServletContext.java:3126)
        at weblogic.servlet.internal.WebAppModule.startContexts(WebAppModule.java:1512)
        at weblogic.servlet.internal.WebAppModule.start(WebAppModule.java:486)
        at weblogic.application.internal.flow.ModuleStateDriver$3.next(ModuleStateDriver.java:425)
        at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:41)
        at weblogic.application.internal.flow.ModuleStateDriver.start(ModuleStateDriver.java:119)
        at weblogic.application.internal.flow.ScopedModuleDriver.start(ScopedModuleDriver.java:200)
        at weblogic.application.internal.flow.ModuleListenerInvoker.start(ModuleListenerInvoker.java:247)
        at weblogic.application.internal.flow.ModuleStateDriver$3.next(ModuleStateDriver.java:425)
        at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:41)
        at weblogic.application.internal.flow.ModuleStateDriver.start(ModuleStateDriver.java:119)
        at weblogic.application.internal.flow.StartModulesFlow.activate(StartModulesFlow.java:27)
        at weblogic.application.internal.BaseDeployment$2.next(BaseDeployment.java:1267)
        at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:41)
        at weblogic.application.internal.BaseDeployment.activate(BaseDeployment.java:409)
        at weblogic.application.internal.SingleModuleDeployment.activate(SingleModuleDeployment.java:43)
        at weblogic.application.internal.DeploymentStateChecker.activate(DeploymentStateChecker.java:161)
        at weblogic.deploy.internal.targetserver.AppContainerInvoker.activate(AppContainerInvoker.java:79)
        at weblogic.deploy.internal.targetserver.operations.AbstractOperation.activate(AbstractOperation.java:569)
        at weblogic.deploy.internal.targetserver.operations.ActivateOperation.activateDeployment(ActivateOperation.java:150)
        at weblogic.deploy.internal.targetserver.operations.ActivateOperation.doCommit(ActivateOperation.java:116)
        at weblogic.deploy.internal.targetserver.operations.AbstractOperation.commit(AbstractOperation.java:323)
        at weblogic.deploy.internal.targetserver.DeploymentManager.handleDeploymentCommit(DeploymentManager.java:844)
        at weblogic.deploy.internal.targetserver.DeploymentManager.activateDeploymentList(DeploymentManager.java:1253)
        at weblogic.deploy.internal.targetserver.DeploymentManager.handleCommit(DeploymentManager.java:440)
        at weblogic.deploy.internal.targetserver.DeploymentServiceDispatcher.commit(DeploymentServiceDispatcher.java:163)
        at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer.doCommitCallback(DeploymentReceiverCallbackDeliverer.java:195)
        at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer.access$100(DeploymentReceiverCallbackDeliverer.java:13)
        at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer$2.run(DeploymentReceiverCallbackDeliverer.java:68)
        at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:528)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
Caused by: java.lang.OutOfMemoryError: PermGen space
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
        at weblogic.utils.classloaders.GenericClassLoader.defineClass(GenericClassLoader.java:328)
        at weblogic.utils.classloaders.GenericClassLoader.findLocalClass(GenericClassLoader.java:285)
        at weblogic.utils.classloaders.GenericClassLoader.findClass(GenericClassLoader.java:253)
        at weblogic.utils.classloaders.ChangeAwareClassLoader.findClass(ChangeAwareClassLoader.java:56)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
        at weblogic.utils.classloaders.GenericClassLoader.loadClass(GenericClassLoader.java:177)
        at weblogic.utils.classloaders.ChangeAwareClassLoader.loadClass(ChangeAwareClassLoader.java:37)
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
        at weblogic.utils.classloaders.GenericClassLoader.defineClass(GenericClassLoader.java:328)
        at weblogic.utils.classloaders.GenericClassLoader.findLocalClass(GenericClassLoader.java:285)
        at weblogic.utils.classloaders.GenericClassLoader.findClass(GenericClassLoader.java:253)
        at weblogic.utils.classloaders.ChangeAwareClassLoader.findClass(ChangeAwareClassLoader.java:56)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
        at weblogic.utils.classloaders.GenericClassLoader.loadClass(GenericClassLoader.java:177)
        at weblogic.utils.classloaders.ChangeAwareClassLoader.loadClass(ChangeAwareClassLoader.java:37)
        at org.springframework.util.ClassUtils.forName(ClassUtils.java:242)
        at org.springframework.beans.factory.support.AutowireUtils.createAutowireCandidateResolver(AutowireUtils.java:120)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.<init>(DefaultListableBeanFactory.java:103)
        at org.springframework.context.support.AbstractRefreshableApplicationContext.createBeanFactory(AbstractRefreshableApplicationContext.java:176)
        at org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:121)
        at org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:423)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:353)
        at org.springframework.web.context.ContextLoader.createWebApplicationContext(ContextLoader.java:255)
<Aug 17, 2010 11:26:56 AM CEST> <Warning> <HTTP> <BEA-101162> <User defined listener org.springframework.web.context.ContextLoaderListener failed: java.lang.IllegalStateException: Unable to load Java
1.5 dependent class [org.springframework.beans.factory.annotation.QualifierAnnotationAutowireCandidateResolver].
java.lang.IllegalStateException: Unable to load Java 1.5 dependent class [org.springframework.beans.factory.annotation.QualifierAnnotationAutowireCandidateResolver]
        at org.springframework.beans.factory.support.AutowireUtils.createAutowireCandidateResolver(AutowireUtils.java:125)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.<init>(DefaultListableBeanFactory.java:103)
        at org.springframework.context.support.AbstractRefreshableApplicationContext.createBeanFactory(AbstractRefreshableApplicationContext.java:176)
        at org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:121)
        at org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:423)
        Truncated. see log file for complete stacktrace

Caused By: java.lang.OutOfMemoryError: PermGen space
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
        at weblogic.utils.classloaders.GenericClassLoader.defineClass(GenericClassLoader.java:328)
        Truncated. see log file for complete stacktrace
>
<Aug 17, 2010 11:26:56 AM CEST> <Error> <Deployer> <BEA-149265> <Failure occurred in the execution of deployment request with ID '1282037204562' for task '9'. Error is: 'weblogic.application.ModuleException: '
weblogic.application.ModuleException:
        at weblogic.servlet.internal.WebAppModule.startContexts(WebAppModule.java:1514)
        at weblogic.servlet.internal.WebAppModule.start(WebAppModule.java:486)
        at weblogic.application.internal.flow.ModuleStateDriver$3.next(ModuleStateDriver.java:425)
        at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:41)
        at weblogic.application.internal.flow.ModuleStateDriver.start(ModuleStateDriver.java:119)
        Truncated. see log file for complete stacktrace

Caused By: java.lang.OutOfMemoryError: PermGen space
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
        at weblogic.utils.classloaders.GenericClassLoader.defineClass(GenericClassLoader.java:328)
        Truncated. see log file for complete stacktrace
>

A short explanation

The error seems to be linked to the garbage collection. From what I could read on the web, static objects used in CGlib (and/or jars depending on the library, such as the wide spread Hibernate, Spring, etc.) are the source of the crash.

Fix

Switching HotSpot to Oracle/BEA’s JVM, aka JRockit, makes the error (almost) disappear.
Alternatively, if you can’t change the JVM, a workaround consists in upgrading the following parameter on JVM launching:
-XX:MaxPermSize=256m