Why so slow/no images

[Tony Collins]

I'm in the process of creating a backup (of data files, not images - so I've turned off image processing)

There are about 18,000 images in the archive on 4 CD's.

Processing has now been running for 1 hour 45 minutes, and no indication of how much longer is left. The file indicator at the bottom of the screen seems to scroll through a few files, and then pause for up to a minute. These pauses are getting longer as time goes by.

Pentium 4 2.4Gig, 512Mb Ram Win xp home

What is it doing with all this time, if it has no thumbnails to produce?

Thanks

Tony

[MichaelT]

Tony,

A few questions...

Are you making an HTML Index? (but with no thumbs or viewing images), or is the Index turned off?

What is the icon on the progress section...are you in the processing section or the HTML section (if you are making an Index).

How are your files distributed? Are there many files within each folder? If so how many? This could be the issue if there are 1000s of files within specific folders.

Thanks..

[Tony Collins]

Yes, I am creating an html index.

There are some folders with large numbers of files. I've just found 1 with about 450 files in.

Is this causeing the problem?

What is strange is that the first pass index was quite quick - only a few minutes. Now it is in the "optimising index" phase, which has now run for about 2:25

Thanks for your quick answer.

[Tony Collins]

Gave up after 3.5 hours and still only processing the second disk.

Trying again with HTML index turned off.

[MichaelT]

I do not think that the 450 files should cause a problem.

I just did 12,000 files with an Index in a few minutes. I am trying more now and will alsomake some folders with more files in them. My set did not need optimizing, however. We will experiment and see if we can se what might be the cause.

If you can see the type of files where it waits for a minue or so, and which folder they come from, there ight be a hint in that.

Thanks for your help.

I just finished 21,000 files as I wrote this, but onto 12 disks, so my files are bigger than yours, which will have an effect.

[Tony Collins]

I'm just about to try again, with the same file selections. I will try to time the different sections of the processing......

Processed

[PeterM]

Tony,

I have seen some issues like this having to do with large numbers of files in the same folder. My guess is that if you turn off indexing all together, it will process very fast. Also, if you take your large folder and split it into two folders, half the files in one and half in the other my guess is that it would be much much faster even with indexing on.

Please let me know if either of these are true. That will help narrow this down.

Also, are all your file similarly named...ie same first x characters?

Thanks again

[Tony Collins]

Sorry, dont seem to be able to edit previous message after accidentally pressing enter.

To continue.....

processed 17726 files and esimated html indes size in less than a minute......

flagged up four files with the same garbled name in 4 folders as being "larger than the collection capacity". As far as I can see the file does not exist (it is not a hidden file). I could email screen shots if you wish.

Estimated archive size 4 disks

Now processing final archive layout - askes me to pass on the 4 garbled filenames again.

Now generating HTML Index. Index size has been estimated at 13.3 MB. After about 1 to 2 minutes of continuous processing, there is now a burst of files followed by about 1/2 second gap, It is on the second disk.....It has now been 7 minutes, the gap is getting longer, between 2 and 10 seconds. The PC has become very sluggish. NEARLY ALL THE FILES IT PAUSES ON ARE IMAGE FILES - GIF'S, JPG'S, PSD', PDF'S, etc. Strangely it has just paused on a WP5. The gaps are getting longer - it is almost as though it is building a list, and as the list gets longer the processing gets slower.

After about 12 minutes, it has finished the layout, asked me to skip the 4 garbled file names, and gone into an "optimising layout index" mode. The index size is now shown as 19.1Mb. The gap between bursts of files is now between 12 and 30 seconds (longer is more frequent) The PC Is very sluggish and there is a lot of disk activity. Task manager reports archive creator memory usage as sitting at around 60 or 70Mb with longish peaks up to 350Mb. Now 20 minutes into the process, and the gap is sitting at around 10 to 20 seconds - slightly shorter than before.

Memory usage is rarely dropping below 200Mb and peaking to well over 400Mb. Having said that, I've just looked and it is now at 40Mb. Oh, back at 400+

I've just seen it pause on an HTML File. But mostly it is gifs (I have a lot of saved web pages, and also many self created web pages)

OK, it has now been half an hour and it is still optimising the first disk. I have more productive stuff I must do tonight. I hope this helps.

Cheers

Tony

PS Task manager performance tab reports page file usage of around 570Mb.

[Tony Collins]

When I closed archive creator, page file usage dropped to 200Mb

[PeterM]

Tony, if you are a photoshop user, you should be aware of a feature in photoshop that can drastically slow down other applications. Dont know if this has anything to do with it but if you do use photoshop please see..

http://www.pictureflow.com/forum/showthread.php?s=&threadid=873

[Tony Collins]

The photoshop issue sounds a likely candidate - I'll give it a go tonight.

[Tony Collins]

Got home, and checked - photoshop's display thumbnail option was already turned off.

Tried again anyway, and behaviour was exactly as last night.

To answer an earlier question, if I turn off the indes completely, then there is no problem.

Cheers....

[PeterM]

Tony,

I have been looking into this issue and am finding it hard to reproduce. The delays that I am aware of, I feel, are not relavant to your situation.

There is one test I would like for you to do if you could. Before starting the long process, bring up the task manager , go to the Processes tab and sort by CPU column (click on the CPU column so that the process using the highest CPU percentage is always on top).

Now run AC and when the really slow processing is taking place, look at the task manager and let me know if AC is allway using all the processor or if some other task are getting time.

I am trying to verify that no other appliation on your system is getting tickled while AC is doing this processing. I have never seen AC perform this way so please let me know what you find.

Thanks for your time on this. Hopefully we can get to the bottom of this.

[Tony Collins]

Gave your suggestion a try. There are two screen shots. The first, here shows when AC is running through the file list quickly - note there appear to be two instances of AC, and that AC is taking most of the CPU time (sometimes it is a bit less - say 70%, with system idle making up the balance.

http://www.tonycollins.dsl.pipex.com/photos/acss5.jpg

As time goes by, the file scan gets slower and slower - with the pauses as described before. Then the screen shot taken during one of the pauses (at this time about 20s) becomes

http://www.tonycollins.dsl.pipex.com/photos/acss4.jpg

Note now that system idle is taking all the time. The disk drive is going nuts (page filing?) and system response is *very* sluggish.

One other thing that is interesting. When I cancelled the file scan, and restarted from the beginning, the process was immediately slow (as slow as just before the cancel) rather than starting fast and getting gradually slower. If I exited and restarted AC, then the next scan started fast and then got slower as previously described.

I am now going to try again, with all the system tray programs removed.

Note also that in the screenshots, photoshop is running, This is only to recieve the screen shots - I got the same results when AC was the only program running.

Cheers.

[Tony Collins]

I just shut down all the task bar programs, including antivirus, and firewall.

Ran the test again, with exactly the same results.


One other unrelated point - when I get the dialog asking me to skip the garbled filenames, the mouse pointer stays as an hour glass.

Thanks

Tony

[PeterM]

Thanks for the info tony. The screen shots are very helpful.


Archive Creator keeps some .xml settings files around. Can you zip and email these files to me. It would be most helpful. These files can be found in the install folder, usually c:\program files\archive creator. Just find all the .xml files and email them to peterm@fnol.net.

There is also another folder in the install folder called CollectionFileSets. Can you email me the contents of this folder as well.

You might want to check one other setting...in the preferences page there is a setting called "Use Lowest Common Folder" Please make sure this is set and the setting below it is not set (Use Manual selection).


The bottom line is that something about your system, file structure etc is causing the program to use large amounts of memory, causing page faults and other system problems. We have not seen this before.

Hopefully we will get to the bottom of this soon. Thanks for your help here.

[Tony Collins]

I'm at work at the moment - I'll send you all the stuff when I get home.

Is it worth trying the older verion (1.?) to see if it behaves in the same way?

I'll also try with one of each of the 3 folder sets I am including in the scan removed to see if this makes a difference.

PS - this is why I love shareware/trialware. Generally the authors *care* about their product.

Thanks for spending the time on this.


Tony

[PeterM]

Yes, you could try with an older version. The more info the better.

Also, just selecting/unselecting different folders can give more info as well. Perhaps there are some file types that are the root cause and if those are not selected things might work fine.

[Tony Collins]

Ok, here is what I've found.

First, I have managed to throw another error... more later.

Removing one folder from the 4 I was archiving significantly improved the speed. Adding it back slowed down again. Having that folder only didn't seem to have enough files to create the memory paging problem but I could see the pauses getting longer and longer until the scan finished. Looking at the files that were pausing (gif's and jpgs mostly) I identified one subfolder of the slow one which was taking most of the time. This is my "saved web pages" folder where I save pages for reference and offline reading. This folder contains a lot of gifs and jpgs, plus other graphics files. A total of 1928 out of 3171 files in fact. Archiving just this folder showed the slowing down, but again, not enough files to truly cripple it.

So I duplicated the folder 6 times by copying into subfolders, resulting in a total of around 19000 files of which around 2/3 are graphics files.

Bingo - after 10 to 15 minutes the system is thrashing the disk, and AC is using about 200Mb of RAM. 2 hours later it is still trying, when I cancel. Is it possible that AC is still trying to create thumnails and store them somewhere in RAM, even though I've told it not to?

The beauty of this is I can email you a copy of the original folder in a single 228K zip file if you want to try to duplicate my results. (You would have to create the 6 copies locally). Just let me know if you want it. I guess the time it takes to start creating memory problems may depend on the physical ram size on the machine - mine is 512Mb.


Back to the other error - I tried to archive the program files folder (62000 files) and got an unexpected error message.

See screenshot below.

http://www.tonycollins.dsl.pipex.com/photos/acss6.jpg

I wasn't able to duplicate this the next time I tried.

Cheers

Tony

[Tony Collins]

V1.0 PRA also behaves the same on my system.

[PeterM]

Absolutely email me the files...

peterm@fnol.net

If I can reproduce it, then we can fix it.

[PeterM]

Tony, I got the file and just want to be sure it is the proper thing...the zip file has only 38 file in it. Is this what you meant to send?

[PeterM]

Tony, can you do the following.

1-Get the app into the funny state where it is taking soo long.
2-Kill the app
3-zip and email me the contents of

c:\program files\SpanDisk\TempIndex\ACSys\index\images\

Thanks...

By the way, as a test, I selected several folder in my system, totaling over 150000 files and processed as you are with no problems.

[Tony Collins]

Sorry, been away for the weekend - I'll do that stuff now.

I failed to zip up the subfolders, so I'll do that now as well.

[Tony Collins]

Just sent the images folder (I assume you meant in the ArchiveCreator folder, rather than the SpanDisk folder.

I am thinking there may still be somthing going on with the Adobe plugin which displays thumnails.

Even though I've switched thumnail display off, it must still intercept the file system in some way. I'm going to try and find some way to turn this off altogether - so there is no "photoshop image" tab in the properties dialogue. Any idea's on how to do this from anyone would be appreciated.

[Tony Collins]

OK, After browsing on the internet for most of the evening, I've renamed psicon.dll. to psicon.old

Now there is no tab specific to photoshop on the image properties, and Archive Creator is as fast as it should be.

Sorry for wasting your time on this - but at least we've found the cause.

Now there is just the problem of garbled filenames throwing errors to solve. I'll post a screen shot on a new thread (so as not to confuse the issue)

Cheers, and thanks.

[PeterM]

Great, you beat me to it...after looking at your html index images folder you sent me it was obvious that photo shop is still effecting your images because the image icons were all photo shop icons.

AC goes to the file system to get the Icon for all the files it archives (when thumbs are turned off). What the OS does with this request is out of our hand. In the case of photoshop, it seems that photoshop gets called and does some work. It might even have a memory problem which is why your memory usage was getting so high.

We will look at this to see if we can find a better way to resolve this without having to get rid of the photoshop dll.