Post Reply 
 
Thread Rating:
  • 0 Votes - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Running out of memory on Mac
08-18-2007, 22:42 (This post was last modified: 08-19-2007 02:09 by JTK.)
Post: #1
Running out of memory on Mac
Hello,

Swarp 2.6.12 is running out memory on Mac OS X 10.4, using just about any setting of the parameters VMEM_MAX, MEM_MAX, and COMBINE_BUFSIZE (VMEM_DIR is big enough). I am combining a large number of 18000x18000 1.3 GB single float images, but it fails when resampling the first one.

It builds the background map, and then fails with a message (at line 91 of data.c) that it does have enough memory, "either in memory or mapped to disk". Looking at virtual memory usage using 'top', usage is 100MB, then it jumps to 1.3 GB, then swarp quits. It also creates two 1.3 GB temporary (virtual memory?) files before failing; the 2nd file might not be a full 1.3 GB.

Do I understand correctly that swarp requires enough virtual memory to map both the input image, and the background map, both as single floats, so I am running into a fundamental 32 bit virtual memory limit? This might make sense, because it seems to fail at a 2.6 GB virtual memory allocation, close to the 4 GB size of a 32 bit address space. The machine is a G5, nominally 64 bit, but swarp is probably compiled as 32 bit.

Jan
Find all posts by this user
Quote this message in a reply
08-20-2007, 18:24
Post: #2
RE: Running out of memory on Mac
Hi Jan,

Quote:Do I understand correctly that swarp requires enough virtual memory to map both the input image, and the background map, both as single floats, so I am running into a fundamental 32 bit virtual memory limit? This might make sense, because it seems to fail at a 2.6 GB virtual memory allocation, close to the 4 GB size of a 32 bit address space.
Actually SWarp requires memory only for the input image (and the weight-map, if provided, is it the case?). 1.3GB should not create problems afaik. The amount of memory taken by the background map is negligible.
Quote:The machine is a G5, nominally 64 bit, but swarp is probably compiled as 32 bit.
Could you try with a 64bit version?
Thanks.
- Emmanuel.
Visit this user's website Find all posts by this user
Quote this message in a reply
08-22-2007, 08:33 (This post was last modified: 08-23-2007 02:10 by JTK.)
Post: #3
RE: Running out of memory on Mac
Hello Emmanuel,

First, I mis-stated a problem - yes, it is a weight map, not a background map. Hence the requirement including the weight map is 2.6 GB. I've had trouble allocating this much memory with other applications, probably because various regions of the address space are reserved by the system, so nothing near the full 4 GB is available for user data.

I don't think I can try a 64 bit version, because my only machine is a G5, not the 64 bit Linux variants supported in 64 bit mode.

But the fact that you say that the full 2.6 GB actually needs to be be mapped convinces me that the small address space is indeed the problem. I've solved it by throwing away regions of the image I don't need. Thanks for clarifying things. Time to get a 64 bit machine ...

Jan
Find all posts by this user
Quote this message in a reply
03-26-2015, 14:04 (This post was last modified: 04-09-2015 10:53 by dravidmilo.)
Post: #4
RE: Running out of memory on Mac
I confirm your findings. There are many possible combinations for defining the SWarp output frame, and indeed some of them might not work as expected; these options are there only for quick-and-dirty jobs. I will try to find a fix when I find the time (probably not before November). In any case, for perfect control of the output frame coordinates, my advice is always to setup a coadd.head (pseudo)header file with the exact WCS parameters you wish to see in the result!











_______________________________________________

We are the pioneers in providing http://www.test-king.com/ dumps and with exam pass guarantee. Download our latest testkingccie sckans questions to pass real exam of mcitp training in www.muih.edu
Find all posts by this user
Quote this message in a reply
Post Reply 


Forum Jump:


User(s) browsing this thread: 1 Guest(s)