Memory Error When Processing Large Files In TotalAgility

Issue

When processing very large files in TotalAgility that are 1-2 GB in size the following error is encountered and the process fails.

"Unable to read data from the transport connection. An existing connection was forcibly closed by the remote host.
The underlying connection was closed. An unexpected error occurred on a receive."

Cause

This error occurs because the HTTP response max size limit has been exceeded. This limit is 2 GB, which is also the limit of the file size stored in SQL Server.

This issue can also occur when processing a file that is less than 2 GB in size because this limit gets exceeded based on how WCF manages its memory.

Here is the exception text:

<ExceptionType>System.InsufficientMemoryException, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</ExceptionType>
<Message>Failed to allocate a managed memory buffer of 2147483647 bytes. The amount of available memory may be low.</Message>

Solution

The TotalAgility documentation has been updated since KTA 7.8 to include this information explaining these limitations:

NOTE: If the file you are importing is too large in size, an error can occur while processing it. The maximum size limit for HTTP response is 2GB, which is also the size limit for files stored in SQL server. However, the processing of the file depends on system load and memory usage, and an error can still occur for sizes smaller than 2GB due to the way WCF manages its memory.

Level of Complexity 

Easy

Applies to