What's new

Reading From One File and Writing To Another

W

Wheen

Bytesexual
Retired
Messages
6,154
Reaction score
2,342
I'm having trouble reading from one file, encrypting the text, and then writing it again. Below is a stripped down version of the algorithm I'm using. It works perfectly, but it doesn't clear the memory after it is done. For example, if it takes 100,000kb to do it the first time, if I run it again without restarting the program, it goes up to 200,000kb, and once the method is finished, it doesn't return back to normal. Here's the algorithm:

Code:
            StreamReader file = new StreamReader(origPath);
StreamWriter newFile = new StreamWriter(newPath);

try
{
string line;

while ((line = file.ReadLine()) != null)
{
newFile.WriteLine(line);
}
newFile.Close();
file.Close();
}

Any helps is appreciated.
 
C

Calamity

Retired
Retired
Messages
6,716
Reaction score
1,156
I'm having trouble reading from one file, encrypting the text, and then writing it again. Below is a stripped down version of the algorithm I'm using. It works perfectly, but it doesn't clear the memory after it is done. For example, if it takes 100,000kb to do it the first time, if I run it again without restarting the program, it goes up to 200,000kb, and once the method is finished, it doesn't return back to normal. Here's the algorithm:

Code:
            StreamReader file = new StreamReader(origPath);
StreamWriter newFile = new StreamWriter(newPath);

try
{
string line;

while ((line = file.ReadLine()) != null)
{
newFile.WriteLine(line);
}
newFile.Close();
file.Close();
}

Any helps is appreciated.

Sounds like they aren't being disposed of properly?

Try either calling Dispose() method (newFile.Dispose() I believe), or use the using statement (I would still call the Close method too).

E.g.

Code:
using (StreamReader file = new StreamReader(origPath),
StreamWriter newFile = new StreamWriter(newPath))
{
// Code here
}
// the file object doesn't exist here, automatically calls the Dispose method.
 
W

Wheen

Bytesexual
Retired
Messages
6,154
Reaction score
2,342
Sounds like they aren't being disposed of properly?

Try either calling Dispose() method (newFile.Dispose() I believe), or use the using statement (I would still call the Close method too).

E.g.

Code:
using (StreamReader file = new StreamReader(origPath),
StreamWriter newFile = new StreamWriter(newPath))
{
// Code here
}
// the file object doesn't exist here, automatically calls the Dispose method.
I'll try the using thing when I get home, I already tried .Dispose() and it didn't work.
 
C

Calamity

Retired
Retired
Messages
6,716
Reaction score
1,156
I'll try the using thing when I get home, I already tried .Dispose() and it didn't work.

Look at the variables like the variable 'line' for example, that gets the 100,000kb content if I'm seeing that right, and I don't see it being disposed anywhere, therefore it would hold the entire contents until you emptied it I believe.
 
W

Wheen

Bytesexual
Retired
Messages
6,154
Reaction score
2,342
Look at the variables like the variable 'line' for example, that gets the 100,000kb content if I'm seeing that right, and I don't see it being disposed anywhere, therefore it would hold the entire contents until you emptied it I believe.
For the using statement, it delivers StreamWriter as an error:
"You must provide an initializer in a fixed or using statement decleration".

As for Line, I don't think so, since every time it passes through the loop it writes over it with new data. For example:

Code:
line = "This is line";
line = "This is also line";

When printed out, line would just be "This is also line".

On the otherhand, if it were like this:

Code:
line = "This is line";
line += "This is also line";

It would be like you said, and it would print out "This is lineThis is also line";

As for the other, I disposed of both the streamreader and the streamwriter, and set line to null, and it still shows the same problem.
 
C

Calamity

Retired
Retired
Messages
6,716
Reaction score
1,156
For the using statement, it delivers StreamWriter as an error:
"You must provide an initializer in a fixed or using statement decleration".

As for Line, I don't think so, since every time it passes through the loop it writes over it with new data. For example:

Code:
line = "This is line";
line = "This is also line";

When printed out, line would just be "This is also line".

On the otherhand, if it were like this:

Code:
line = "This is line";
line += "This is also line";

It would be like you said, and it would print out "This is lineThis is also line";

As for the other, I disposed of both the streamreader and the streamwriter, and set line to null, and it still shows the same problem.

I was just saying for example if that variable or any other variable did hold all of the values.

Also, I just did a quick search on it, read the first response here.

Are you just worried about it, or is it running out of memory by chance?
 
W

Wheen

Bytesexual
Retired
Messages
6,154
Reaction score
2,342
I was just saying for example if that variable or any other variable did hold all of the values.

Also, I just did a quick search on it, read the first response here.

Are you just worried about it, or is it running out of memory by chance?
A little bit of both. I'm personally not worried, as I have a good 8 gigs. On a computer without such RAM generosity, it may not run well.

I'll try what is discussed in that link, thanks.
 
Z

Zer0-One

Experienced Member
Messages
2,180
Reaction score
256
I'm having trouble reading from one file, encrypting the text, and then writing it again. Below is a stripped down version of the algorithm I'm using. It works perfectly, but it doesn't clear the memory after it is done. For example, if it takes 100,000kb to do it the first time, if I run it again without restarting the program, it goes up to 200,000kb, and once the method is finished, it doesn't return back to normal. Here's the algorithm:

Code:
            StreamReader file = new StreamReader(origPath);
StreamWriter newFile = new StreamWriter(newPath);

try
{
string line;

while ((line = file.ReadLine()) != null)
{
newFile.WriteLine(line);
}
newFile.Close();
file.Close();
}

Any helps is appreciated.

if this is inside a method, then your problem is that the garbage collector isn't picking up the objects you're not using anymore. one of the things I don't like about a managed framework is that you can't just delete objects, since there might still be references to it. The best you can do is to set "file" and "newfile" = null after you're done using them. They won't be deleted immediately, since the garbage collector works automatically, but make sure that you close them, and that no other object/variable references them, and when the GC sees that myObject = null, it'll free up the memory.
 
A

AsparagusMan

Enthusiast
Messages
493
Reaction score
115
Code:
FileStream reader = new FileStream("C:/ReadFile", /* Parameters */);
FileStream writer = new FileStream("C:/WriteFile", /* Parameters */);

long total = 0;
uint cur;
byte[] buff = new byte[0x200];

reader.Position = 0;
writer.Position = 0;

// Copies the file in 512 byte blocks so shouldn't run out of memory
while (total < reader.Length) {
cur = reader.Read(buff, 0, 0x200);
// Encrypt data
writer.Write(buff, 0, buff.Length);
total += cur;
}

writer.Close();
Reader.Close();
Do note, depending on the encryption method, you won't always get a data return of the encrypted data as the same length, only most symmetrical encryption's have same size output.
 
Z

Zer0-One

Experienced Member
Messages
2,180
Reaction score
256
Code:
FileStream reader = new FileStream("C:/ReadFile", /* Parameters */);
FileStream writer = new FileStream("C:/WriteFile", /* Parameters */);

long total = 0;
uint cur;
byte[] buff = new byte[0x200];

reader.Position = 0;
writer.Position = 0;

// Copies the file in 512 byte blocks so shouldn't run out of memory
while (total < reader.Length) {
cur = reader.Read(buff, 0, 0x200);
// Encrypt data
writer.Write(buff, 0, buff.Length);
total += cur;
}

writer.Close();
Reader.Close();
Do note, depending on the encryption method, you won't always get a data return of the encrypted data as the same length, only most symmetrical encryption's have same size output.


he's having a problem with excessive memory usage in terms of the entire program segment, not the buffer he's trying to write to.

My guess was that since he's creating a new reader/writer on the heap for every time that the method is called, that those objects were all taking up memory. And since those objects were never destroyed or nulled, they remain in memory until the program ends.
 
A

AsparagusMan

Enthusiast
Messages
493
Reaction score
115
he's having a problem with excessive memory usage in terms of the entire program segment, not the buffer he's trying to write to.

My guess was that since he's creating a new reader/writer on the heap for every time that the method is called, that those objects were all taking up memory. And since those objects were never destroyed or nulled, they remain in memory until the program ends.
The .NET garbage collector disposes everything when it is not referenced anymore, so once it leaves the context, it *should* dispose of the object.
 
Z

Zer0-One

Experienced Member
Messages
2,180
Reaction score
256
oh, wait. right. stupid mistake on my part. Once the method is over, the reference to the reader/writer objects are gone.

so wait... the only bit of that code that causes any big memory allocation are his creation of those 2 objects. no? if not, what else could be causing this jump in memory use?
 
A

AsparagusMan

Enthusiast
Messages
493
Reaction score
115
oh, wait. right. stupid mistake on my part. Once the method is over, the reference to the reader/writer objects are gone.

so wait... the only bit of that code that causes any big memory allocation are his creation of those 2 objects. no? if not, what else could be causing this jump in memory use?
No, he is using some sort of encryption method, I assume the algorithm is either A ) created by his own hands and he wrote it very inefficiently or B ) He is using a whole bunch of unneeded instances of cryptography objects.
 
Z

Zer0-One

Experienced Member
Messages
2,180
Reaction score
256
arrghh. massive fail on my part. he said "stripped down version".

Wheen, do post the complete source for us, as your problem probably lies in whatever you left out.

maybe after some sleep I'll regain my basic reading skills *grumble*
 
W

Wheen

Bytesexual
Retired
Messages
6,154
Reaction score
2,342
arrghh. massive fail on my part. he said "stripped down version".

Wheen, do post the complete source for us, as your problem probably lies in whatever you left out.

maybe after some sleep I'll regain my basic reading skills *grumble*

Code:
public static void createList(string filePath, string wordList, string method)
{
TextReader reader = new StreamReader(wordList);
using (TextWriter writer = File.CreateText(filePath))
{
WriteText(reader, writer, method);
}
}

private static void WriteText(TextReader file, TextWriter newFile, string method)
{
try
{
status = "Running";
string line;

while ((line = file.ReadLine()) != null)
{
if (method == "MD5")
{
newFile.WriteLine(CalcHash.GetMD5(line));
}
else if (method == "RIPEMD-160")
{
newFile.WriteLine(CalcHash.GetRIPEMD160(line));
}
else if (method == "SHA-1")
{
newFile.WriteLine(CalcHash.GetSHA1(line));
}
else if (method == "SHA-256")
{
newFile.WriteLine(CalcHash.GetSHA256(line));
}
else if (method == "SHA-384")
{
newFile.WriteLine(CalcHash.GetSHA384(line));
}
else if (method == "SHA-512")
{
newFile.WriteLine(CalcHash.GetSHA512(line));
}
}
}

catch
{
status = "Error";
MessageBox.Show("Check read or write file destinations", "Error Creating List");
}

status = "Done";
MessageBox.Show("Hash List Created", "List Created");
}

^That's the current iteration that uses some of what Calamity said.
 
A

AsparagusMan

Enthusiast
Messages
493
Reaction score
115
A ) have you ever heard of booleans?
Code:
bool running = true;
// do stuff
running = false;
B ) have you ever heard of enums?
Code:
enum hashType {
sha1 = 0,
md5, // = 1
sha256, // = 2
}

// in function
hashType x = input;
switch (x) {
case sha1:
// calc sha1
break;

case md5:
// calc md5
break;

case sha256:
// calc sha256
break;

default:
throw "invalid selection";
break
}

C ) you do know that hases aren't encryption algos right? Meaning you wont be able to get the original data back, a hash is just a digest, aka a signature, not encrypted data.

D ) im assuming your static class of hashes just keeps making new instances of all the hash objects, dont ask me why they dont die after the context, but i recommend making a static global variable and only initializing it once
Code:
static class myHashClass {
static ParentHashClass hasher; // I cant remember the name of
// the hash objects parent class

static void initialize(hashType t) {
switch (t){
case sha1:
hasher = new Sha1();
break;

//etc etc
} 
}

static byte[] calcHash(byte[] data) { return hasher.ComputeHash(data); }

}

// in function
myHashClass.initialize(hashSelection);

/* for each line */ {
Write(hasher.calcHash(Read(line)));
}

Ps this was all typed out on my ipod touch x.x
 
B

BuC-ShoTz

Enthusiast
Messages
181
Reaction score
72
try that and see what happens

StreamReader file = new StreamReader(origPath);
StreamWriter newFile = new StreamWriter(newPath);

try
{
string line;

while ((line = file.ReadLine()) != null)
{
newFile.WriteLine(line);
}
newFile.Flush();
newFile.Close();

file.Close();
}
 
E

etownlax

Getting There
Messages
1,605
Reaction score
192
I don't know if you've fixed this yet but, you can probably simply re-initiate the streams, like so...

Code:
StreamReader file = new StreamReader(origPath);
StreamWriter newFile = new StreamWriter(newPath);

try
{
string line;

while ((line = file.ReadLine()) != null)
{
newFile.WriteLine(line);
}

newFile.Close();
file.Close();

file = new StreamReader(origPath);
newFile = new StreamWriter(newPath);

}
 
Top Bottom