AstrOmatic forum
file positioning failed - Printable Version

+- AstrOmatic forum (https://www.astromatic.net/forum)
+-- Forum: AstrOmatic software (/forumdisplay.php?fid=3)
+--- Forum: SWarp (/forumdisplay.php?fid=5)
+--- Thread: file positioning failed (/showthread.php?tid=303)

Pages: 1 2


file positioning failed - acbecker - 12-04-2007 20:44

Hi - I recently ran Scamp+Swarp on a set of CFHT data to create the file coadd.fits. The Scamp-ing looked successful from the output diagnostics. The command was
Code:
scamp -MOSAIC_TYPE SAME_CRVAL -ASTREF_CATALOG UCAC-2 -AHEADER_GLOBAL megacam.ahead *.ldac

I then ran Swarp with the output .head files
Code:
swarp *.fits
. The image coadd.fits is about the correct size (60k x 80k pixels; 19GB at -32 bit per pix).

I next wanted to run Swarp to chop out that part of the template image corresponding to one of the science images :
Code:
imhead science.fits > template.head
;
Code:
swarp coadd.fits -IMAGEOUT_NAME template.fits
. This fails with

Code:
> WARNING: default.swarp not found, using internal defaults

----- SWarp 2.16.4-MP started on 2007-12-04 at 12:45:49 with 8 threads

> Creating NEW output image...
> WARNING: FITS header data read in 870505p_01_tmpl_img.head

------- Output File 870505p_01_tmpl_img.fits:
    "no ident"  WEIGHTED  EXT. HEADER  2112x4644  32 bits (floats)
    Center: 01:18:03.08 +08:50:33.3   6.48'x14.3'  Scale: 0.1842 ''/pixel
    Gain: 0 e-/ADU   Flux scaling (astrom/photom): 1 X / 1 X
-------------- File coadd.fits:
    "Main11"  unweighted  no ext. header  80647x61092  32 bits (floats)
    Center: 01:10:16.64 +07:27:42.3   250'x189'  Scale: 0.1857 ''/pixel
    Gain: 3.54e+12 e-/ADU   Flux scaling (astrom/photom): 0.4073 X / 1 X
> Setting up background map at line:   6784 / 61092
> *Error*: file positioning failed in coadd.fits

I have also tried this with
Code:
-NTHREADS 1
and it still fails. Interestingly, running SExtractor on this seems to at least get farther.

Code:
sex coadd.fits

> WARNING: default.sex not found, using internal defaults

----- SExtractor 2.5.0 started on 2007-12-04 at 13:21:17 with 1 thread

Measuring from: "Main11"  / 80647 x 61092 / 32 bits FLOATING POINT data
(M+D) Background: 2.72008e-14 RMS: 6.42918e-12 / Threshold: 9.64377e-12
> Line: 1616  Objects:     1416 detected /        0 sextracted

> WARNING: Pixel stack overflow at position 25305,1625
Ad infinitum the above.

Any idea why this may be happening, or how to get around it?


RE: file positioning failed - acbecker - 12-04-2007 20:49

As additional information, other .fits file accessing programs (ds9, pyfits, cfitsio) have problems with the coadd.fits, so I think the file is corrupted somehow. Or its a file-size issue, but 19GB is pretty benign these days.

Is it possible for Swarp to write an image it can't then read!? Wink


RE: file positioning failed - acbecker - 12-04-2007 22:53

And finally, SExtractor seems to successfully reduce the entire image, after all the error messages have passed.


RE: file positioning failed - Henry Joy McCracken - 12-05-2007 09:38

Hi,

Well, to cut up the 19gb image with SWARP (which means reading the image into memory) requires at least that amount of physical ram installed in your system (and double that amount if you have a weight-map), otherwise you will have memory problems. Simply coadding the images, on the other hand, only requires memory mapping each individual CCD which is much smaller.

You are probably better off dealing with smaller output images (say up to around 4gb or so).

Incidentally, you do not include any weight images in your swarp? The results won't be very nice (around the borders and at the edges) without the weight-maps.

cheers
henry


RE: file positioning failed - acbecker - 12-05-2007 19:22

Hi -

I have considered this, but I'm not sure the various failure modes when accessing this image indicate that this is the case. You are suggesting there are subtleties in the source code that allow Swarp to write out an image that it can not easily read (but that is able to be read and analyzed by SExtractor)? If its a running out of memory issue, I would expect the failure mode to be somewhat different - using up all RAM, going to swap, my computer bogging down, and my desperate attempts to get focus in a window so I can kill -9 the process. However, this does not happen. The failure modes instead seem to indicate that this is not a vaild fits file in some sense.

Andy


RE: file positioning failed - Henry Joy McCracken - 12-06-2007 08:45

Hi Andy,

Yes, that's what I am suggesting. Swarp can write an arbitrary large fits file, no problem. That arbitrary large fits file can also be read by sextractor. But to resample that fits file requires that it fits into physical memory. What I mean is, to resample a fits file of 2gb requires a contiguous block in RAM of 2gb. More if there are weight maps. although I agree that the point at which the code is crashing is not the point where it should crash (there would be a memory error)! How much ram do you have on your system? You are using the 64 bit version of swarp?

cheers
henry


RE: file positioning failed - Emmanuel Bertin - 12-06-2007 09:43

Apparently the problem arises when SWarp crosses the 2GB barrier. I second Henry's comment. Unfortunately you need a 64-bit system to handle properly input images larger than ~2GB with SWarp (irrespective of the amount of RAM in your machine, this is a limitation of addressable space). If you are running a 64-bit system, check that the config parameter VMEM_MAX is at least as large as the input file. If this is the case, you might have found a bug. Please tell us!
- Emmanuel.


RE: file positioning failed - acbecker - 12-06-2007 19:17

OK, I *think* my system is entirely 64-bit

Code:
Linux 2.6.9-55.0.2.ELsmp #1 SMP Tue Jun 12 17:59:08 EDT 2007 i686 i686 i386 GNU/Linux

although the last i386 might indicate some sort of 32-bit emulation mode. I run SWarp with the following flags

Code:
swarp ../templates/TALCS.r.fits -IMAGEOUT_NAME output/870505p_01_tmpl_img.fits -VMEM_MAX 32768 -MEM_MAX 1024 -COMBINE_BUFSIZE 512


> WARNING: default.swarp not found, using internal defaults

----- SWarp 2.16.4-MP started on 2007-12-06 at 12:13:17 with 8 threads

> Creating NEW output image...
> WARNING: FITS header data read in output/870505p_01_tmpl_img.head

------- Output File 870505p_01_tmpl_img.fits:
    "no ident"  WEIGHTED  EXT. HEADER  2112x4644  32 bits (floats)
    Center: 01:18:03.08 +08:50:33.3   6.48'x14.3'  Scale: 0.1842 ''/pixel
    Gain: 0 e-/ADU   Flux scaling (astrom/photom): 1 X / 1 X
-------------- File TALCS.r.fits:
    "Main1"  unweighted  no ext. header  80692x61124  32 bits (floats)
    Center: 01:10:16.96 +07:27:44.5   250'x189'  Scale: 0.1858 ''/pixel
    Gain: 1.16e+13 e-/ADU   Flux scaling (astrom/photom): 0.4084 X / 1 X
> Setting up background map at line:   6656 / 61124
> *Error*: file positioning failed in ../templates/TALCS.r.fits

I have checked the executable to see how it was compiled

Code:
file ~/src/swarp-2.16.4/src/swarp

/home/becker/src/swarp-2.16.4/src/swarp: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.2.5, dynamically linked (uses shared libs), not stripped

and it looks like it was compiled 32-bit. I will try and force it to compile as 64-bit and retry. Thanks for the ideas!


RE: file positioning failed - Emmanuel Bertin - 12-06-2007 19:20

Nope, unfortunately your OS is 32bit (otherwise for a PC the architecture would be called x86_64 or something like that). So if you want to use the 64-bit version of SWarp you will need the 64-bit flavour of your preferred OS.
- Emmanuel.


RE: file positioning failed - acbecker - 12-07-2007 20:58

Indeed, this seems to work A-OK on an x86_64 computer. Thanks for the advice! If possible, I'd suggest a clearer failure mode for this circumstance.

Thanks!
Andy