Quantcast
Channel: Active questions tagged rest - Stack Overflow
Viewing all articles
Browse latest Browse all 3637

OutOfMemoryException when reading a string

$
0
0

I have a C# .NET system that takes a JSON data feed and converts it to an object using the Newtonsoft.Json.JsonConvert.DeserializeObject converter.

This process works perfect as long as the JSON string is below a certain size (a few Mb) but as soon as the returned data is large (almost 100Mb) I get the error OutOfMemoryException

This code works great for small data:

// WebClient ------------------------------------------------------------------var _client = new System.Net.WebClient();var _content = _client.DownloadString(_url);

but blows up on the last line (DownloadString)

I tried changing to this which also works for small data but it still blew up on the ReadToEnd line when the data grew in size.

using (var _response = (System.Net.HttpWebResponse)_request.GetResponse()){    using (System.IO.Stream _dataStream = _response.GetResponseStream())    {        using (System.IO.StreamReader _streamReader = new System.IO.StreamReader(_dataStream))        {            string _responseFromServer = _streamReader.ReadToEnd();        }    }}

Finally I tried this which worked:

StringBuilder _stringBuilder = new StringBuilder();using (var _response = (System.Net.HttpWebResponse)_request.GetResponse()){    using (System.IO.Stream _dataStream = _response.GetResponseStream())    {        using (System.IO.StreamReader _streamReader = new System.IO.StreamReader(_dataStream))        {            while (!streamReader.EndOfStream)            {                char[] _buffer = new char[4096];                _streamReader.ReadBlock(_buffer, 0, _buffer.Length);                var _bufferString = new String(_buffer);                _stringBuilder.Append(_bufferString);            }        }    }}

But it blew up with an OutOfMemoryException error when it got to the next line here:

var _results = Newtonsoft.Json.JsonConvert.DeserializeObject<List<MyObject>>(_stringBuilder.ToString());

It didn't like the ToString() method.

It also crashed with a simple line like

string _convertedString = _stringBuilder.ToString();

The full error is:

An exception of type 'System.OutOfMemoryException' occurred in mscorlib.dll but was not handled in user code

The machine is running 64bit windows with 16Gb of memory.

So, what are my options?

All I want is an IQueryable<MyObject> from a (very large) JSON string.


Viewing all articles
Browse latest Browse all 3637

Trending Articles