Post Reply 
 
Thread Rating:
  • 0 Votes - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Bug in coadd.c when COMBINE_BUFSIZE > 2 GB ?
01-04-2012, 03:13
Post: #1
Bug in coadd.c when COMBINE_BUFSIZE > 2 GB ?
[ Using the trunk version of SWarp from the public svn, labeled v2.30.0 ]

In coadd.c, line ~323:
Code:
nbuflinesmax = (prefs.coaddbuf_size * 1024 * 1024)
    / (2*(multiwidth+outwidth)*sizeof(PIXTYPE));

If prefs.coaddbuf_size is > 2048, this statement overflows 4 byte integers, leading to unexpected results for nbuflinesmax. This bug can then manifest itself in the call a few lines later:

Code:
QMALLOC(multibuf, PIXTYPE, nbuflinesmax*multiwidth);

which raises the error:
> Not enough memory for multibuf (nbuflinesmax * multiwidth elements) !

Two side notes:

First, it would be nice if the above error raised by QMALLOC (in define.h) reported the actual number of elements (and/or bytes) it is trying to allocate, in addition to the names of the variables.

Second (and I may be misunderstanding exactly what the code is doing here) if the user sets prefs.coaddbuf_size way higher than it needs to be (how I ran into this problem), would it make sense to only allocate the space needed (e.g., total size of all images) rather than the entire value of prefs.coaddbuf_size?
Find all posts by this user
Quote this message in a reply
01-05-2012, 19:06
Post: #2
RE: Bug in coadd.c when COMBINE_BUFSIZE > 2 GB ?
Hi,

(01-04-2012 03:13)kbarbary Wrote:  If prefs.coaddbuf_size is > 2048, this statement overflows 4 byte integers,

You are right! 64bit computing is now everywhere, there is no reason to stay with such a limitation.

Quote:First, it would be nice if the above error raised by QMALLOC (in define.h) reported the actual number of elements (and/or bytes) it is trying to allocate, in addition to the names of the variables.

Agreed.

Quote:Second (and I may be misunderstanding exactly what the code is doing here) if the user sets prefs.coaddbuf_size way higher than it needs to be (how I ran into this problem), would it make sense to only allocate the space needed (e.g., total size of all images) rather than the entire value of prefs.coaddbuf_size?

Yes, good point too. Let me work on that and come back with a corrected version in the coming days.
Thanks!
- Emmanuel.
Visit this user's website Find all posts by this user
Quote this message in a reply
01-06-2012, 18:07
Post: #3
RE: Bug in coadd.c when COMBINE_BUFSIZE > 2 GB ?
Done! Please check out the latest unstable version on the SVN repository and tell me if it works as expected.
- Emmanuel.
Visit this user's website Find all posts by this user
Quote this message in a reply
03-21-2012, 13:39
Post: #4
RE: Bug in coadd.c when COMBINE_BUFSIZE > 2 GB ?
I too was having problems with multibuf errors.

I've installed the svn version and running with an 8Gb COMBINE_BUFSIZE
I get

"Maximum overlap density: 398 frames

> Could not allocate memory for multibuf (nbuflinesmax*multiwidth=-7310258816 elements) at line 338 in module coadd.c ! "

I'm not sure whether I'm really hitting memory limits or not.

I've produced similar size mosaics with a similar number of inputs without issue previously.

Mike
Find all posts by this user
Quote this message in a reply
03-22-2012, 06:58
Post: #5
RE: Bug in coadd.c when COMBINE_BUFSIZE > 2 GB ?
Hi Mike,

Strange. Just to be sure about this issue, would you mind checking the following points: Thanks!
- Emmanuel.
Visit this user's website Find all posts by this user
Quote this message in a reply
03-22-2012, 09:42
Post: #6
RE: Bug in coadd.c when COMBINE_BUFSIZE > 2 GB ?
(03-22-2012 06:58)Emmanuel Bertin Wrote:  Strange. Just to be sure about this issue, would you mind checking the following points: Thanks!
- Emmanuel.

Hi Emmanuel

Yes SWarp version 2.30.1 (2012-03-21) (I think you added the
extra reporting of the elements in, so that I can tell it's picking up
the latest version)

Yes 32Gb

Yes (uname -m = > x86_64 & Debian GNU/Linux 7.0 (“wheezy”)

I'm running the mosaic in chunks to see if that gets round my problem.

Thanks
Mike
Find all posts by this user
Quote this message in a reply
04-25-2012, 22:41
Post: #7
RE: Bug in coadd.c when COMBINE_BUFSIZE > 2 GB ?
Thanks Mike.

At last hopefully the remaining issues should be fixed in the latest development version.
Please, if you happen to run SWarp once more, would you mind telling if it works now?
Thanks!
- Emmanuel.
Visit this user's website Find all posts by this user
Quote this message in a reply
Post Reply 


Forum Jump:


User(s) browsing this thread: 1 Guest(s)