RE: FLUKA crash from Chris Theis on 2008-12-03 (fluka discuss archive)

From: Chris Theis <Christian.Theis_at_cern.ch>
Date: Wed, 3 Dec 2008 12:05:23 +0100

Hi Noemi,

the symptoms you describe are identical to a case I was dealing with
just a couple of weeks ago. It seems that these mysterious FLUKA crashes
occur on AMD chips with K8/K10 core architectures running under a 32 bit
linux. This is independent of the distribution as I tried several
different ones, with different math libs etc. Currently, the reason is
not yet fully clear so I would suggest to follow the steps that Paola
described in her mail from yesterday as it would be helpful to pinpoint
the problem to a specific scenario.

In the meantime I can only offer a remedy which solved this problem in
our case. If you have the possibility you should try to switch to a 64
bit version of linux. I ran your input on such a machine we previously
had problems with and which is now running the 64 bit version of linux.
Your input finished gracefully following exactly the same random number
sequence like on your machine, just as expected.

All 32 bit programs will be running under the 64 bit OS as well but you
might need to install a couple of additional libraries for FLUKA to
work. Some hints can be found in the following presentation of a
colleague which could be helpful:

http://info-fluka-discussion.web.cern.ch/info-fluka-discussion/talks/Ludovic_From32To64_190407.ppt

I saw that you're sending your jobs to a LSF batch system and probably
cannot change the OS there. In that case you might try to investigate
the possibilities to exclude certain architectures (I'd suggest AMD
Opteron and Phenom CPUs) from the scheduling process.
Cheers

Chris

From: Finetti Noemi [mailto:noemi.finetti_at_aquila.infn.it]=20
Sent: 03 December 2008 10:34
To: Chris Theis
Subject: Re: FLUKA crash

=20

Hi Christian,
1) I am using a 32 bit version of SLC3;
2) when I run my FLUKA job the crash on the batch machines occurs at
differnt points (sometimes it occurs earlier and sometimes later).
3) -bash-3.1$ more /proc/cpuinfo
processor : 0
vendor_id : AuthenticAMD
cpu family : 15
model : 5
model name : AMD Opteron(tm) Processor 244
stepping : 10
cpu MHz : 1800.000
cache size : 1024 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca
cmov pat
pse36 clflush mmx fxsr sse sse2 syscall nx mmxext lm 3dnowext 3dnow ts
fid vid ttp
bogomips : 3617.79
=20
processor : 1
vendor_id : AuthenticAMD
cpu family : 15
model : 5
model name : AMD Opteron(tm) Processor 244
stepping : 10
cpu MHz : 1800.000
cache size : 1024 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca
cmov pat
pse36 clflush mmx fxsr sse sse2 syscall nx mmxext lm 3dnowext 3dnow ts
fid vid ttp
bogomips : 3616.16

Thanks in advance,
noemi

Chris Theis wrote:

Hi Noemi,
=20
I'm currently looking into your problem and I would need some more
information. Could you please answer the following questions which would
help me to check whether this problem is similar to one that I have seen
just recently:
=20
- Are you using the 32 bit or the 64 bit version of SLC3?
=20
- When you run your FLUKA job does the crash on the batch machines
always occur at the same
point or does it sometimes occur earlier and sometimes later?
=20
- Could you please run the following command on the execution PC and
send me the output
"cat /proc/cpuinfo"
=20
Ciao
Chris
=20
------------------------------------------------------------------------
Chris Theis
CERN/SC-RP - European Organization for Nuclear Research
1211 Geneva 23, Switzerland
Phone: +41 22 767 8069 Office: 892-2A-015
e-mail: Christian.Theis@cern.ch www: http://www.cern.ch/theis
------------------------------------------------------------------------
=20
=20
=20

        -----Original Message-----
        From: owner-fluka-discuss_at_mi.infn.it [mailto:owner-fluka-
        discuss_at_mi.infn.it] On Behalf Of Finetti Noemi
        Sent: 27 November 2008 15:31
        To: fluka-discuss_at_fluka.org
        Subject: FLUKA crash
        =20
        Hi all,
        I have installed FLUKA 2008.3 on a linux machine (Model: 2 AMD
Opteron
        (tm) Processor 244 - 1.8GHz; Operating system: Linux -
Scientific
           =20

Linux
=20

        3.0.5; with g77) where I have compiled my executable file
(myfluka)
        which calls the user routines USRINI, HISTIN,SOURCE, USRMED,
USROUT
        (see user_routines.tar.gz). Executing the job in batch (for
399528
        primaries) the run crashed while running the same job
interactively
        (for 100
        primaries) every thing was fine. What could be the reasons?
        =20
        I point out that the batch job was executed on a 2 Dual Core AMD
        Opteron(tm) Processor 280 - 2.4 GHz with Linux - Scientific
Linux
        3.0.8.
        =20
        In attachment the LSF message, the gdb output (file
fluka_gdb.out),
           =20

the
=20

        files .err, .log, .out, fort.1 and fort.2.
        =20
        Thanks in advance,
        noemi
        =20
        --
=09
---------------------------------------------------------------------
        * Address: Dott.ssa Noemi Finetti
           c/o Dipartimento di Fisica dell'Universita' degli Studi
dell'Aquila
           Via Vetoio - 67010 Coppito - L'Aquila - Italy
        * Phone: +39-0862-433051 (Office); +39-0862-433043 (Laboratory)
        * Fax: +39-0862-433033 (Department).
=09
---------------------------------------------------------------------
        =20
        =20
        =20
           =20

=20
=20
=20

--=20
---------------------------------------------------------------------
* Address: Dott.ssa Noemi Finetti
c/o Dipartimento di Fisica dell'Universita' degli Studi dell'Aquila
Via Vetoio - 67010 Coppito - L'Aquila - Italy
* Phone: +39-0862-433051 (Office); +39-0862-433043 (Laboratory)
* Fax: +39-0862-433033 (Department).
---------------------------------------------------------------------
=20
=20

------_=_NextPart_001_01C95537.0C95CDE5
Content-Type: text/html; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Hi Noemi,<o:p></o:p>

<o:p> </o:p>

the symptoms you describe are identical to a case I was =
dealing
with just a couple of weeks ago. It seems that these mysterious FLUKA =
crashes
occur on AMD chips with K8/K10 core architectures running under a 32 bit =
linux.
This is independent of the distribution as I tried several different =
ones, with
different math libs etc. Currently, the reason is not yet fully clear so =
I
would suggest to follow the steps that Paola described in her mail from
yesterday as it would be helpful to pinpoint the problem to a specific
scenario.<o:p></o:p>

<o:p> </o:p>

In the meantime I can only offer a remedy which solved =
this
problem in our case. If you have the possibility you should try to =
switch to a
64 bit version of linux. I ran your input on such a machine we =
previously had
problems with and which is now running the 64 bit version of linux. Your =
input finished
gracefully following exactly the same random number sequence like on =
your
machine, just as expected.<o:p></o:p>

<o:p> </o:p>

All 32 bit programs will be running under the 64 bit OS =
as well
but you might need to install a couple of additional libraries for FLUKA =
to
work. Some hints can be found in the following presentation of a =
colleague which
could be helpful:<o:p></o:p>

<o:p> </o:p>

<a
href=3D"http://info-fluka-discussion.web.cern.ch/info-fluka-discussion/ta=
lks/Ludovic_From32To64_190407.ppt">http://info-fluka-discussion.web.cern.=
ch/info-fluka-discussion/talks/Ludovic_From32To64_190407.ppt</a><o:p></o:=
p>

<o:p> </o:p>

I saw that you’re sending your jobs to a LSF batch =
system and
probably cannot change the OS there. In that case you might try to =
investigate
the possibilities to exclude certain architectures (I’d suggest =
AMD
Opteron and Phenom CPUs) from the scheduling process. =
<o:p></o:p>

<o:p> </o:p>

Cheers<o:p></o:p>

Chris<o:p></o:p>

<o:p> </o:p>

<div>

From: Finetti
Noemi [mailto:noemi.finetti_at_aquila.infn.it] 
Sent: 03 December 2008 10:34 
To: Chris Theis 
Subject: Re: FLUKA crash<o:p></o:p>

</div>

<o:p> </o:p>

Hi Christian, 
1) I am using a 32 bit version of SLC3; 
2) when I run my FLUKA job the crash on the batch machines occurs at =
differnt points 
(sometimes it occurs earlier and sometimes later). 
3)  -bash-3.1$ more /proc/cpuinfo 
processor       : 0 
vendor_id       : AuthenticAMD 
cpu family      : 15 
model           : =
5 
model name      : AMD Opteron(tm) Processor =
244 
stepping        : 10 
cpu MHz         : 1800.000 
cache size      : 1024 KB 
fdiv_bug        : no 
hlt_bug         : no 
f00f_bug        : no 
coma_bug        : no 
fpu           &nbs=
p; :
yes 
fpu_exception   : yes 
cpuid level     : 1 
wp           &nbsp=
; 
: yes 
flags           : fpu =
vme de
pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat 
pse36 clflush mmx fxsr sse sse2 syscall nx mmxext lm 3dnowext 3dnow ts =
fid vid
ttp 
bogomips        : 3617.79 
  
processor       : 1 
vendor_id       : AuthenticAMD 
cpu family      : 15 
model           : =
5 
model name      : AMD Opteron(tm) Processor =
244 
stepping        : 10 
cpu MHz         : 1800.000 
cache size      : 1024 KB 
fdiv_bug        : no 
hlt_bug         : no 
f00f_bug        : no 
coma_bug        : no 
fpu           &nbs=
p; :
yes 
fpu_exception   : yes 
cpuid level     : 1 
wp           &nbsp=
; 
: yes 
flags           : fpu =
vme de
pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat 
pse36 clflush mmx fxsr sse sse2 syscall nx mmxext lm 3dnowext 3dnow ts =
fid vid
ttp 
bogomips        : 3616.16 
 
Thanks in advance, 
noemi 
 
 
Chris Theis wrote: 
 
<o:p></o:p>

<pre>Hi Noemi,<o:p></o:p></pre><pre><o:p> </o:p></pre><pre>I'm =
currently looking into your problem and I would need some =
more<o:p></o:p></pre><pre>information. Could you please answer the =
following questions which would<o:p></o:p></pre><pre>help me to check =
whether this problem is similar to one that I have =
seen<o:p></o:p></pre><pre>just =
recently:<o:p></o:p></pre><pre><o:p> </o:p></pre><pre>- Are you =
using the 32 bit or the 64 bit version of =
SLC3?<o:p></o:p></pre><pre><o:p> </o:p></pre><pre>- When you run =
your FLUKA job does the crash on the batch =
machines<o:p></o:p></pre><pre>always occur at the =
same<o:p></o:p></pre><pre>  point or does it sometimes occur =
earlier and sometimes =
later?<o:p></o:p></pre><pre><o:p> </o:p></pre><pre>- Could you =
please run the following command on the execution PC =
and<o:p></o:p></pre><pre>send me the output<o:p></o:p></pre><pre>  =
"cat =
/proc/cpuinfo"<o:p></o:p></pre><pre><o:p> </o:p></pre><pre>Ciao=
<o:p></o:p></pre><pre>Chris<o:p></o:p></pre><pre><o:p> </o:p></pre><=
pre>---------------------------------------------------------------------=
---<o:p></o:p></pre><pre>Chris Theis<o:p></o:p></pre><pre>CERN/SC-RP - =
European Organization for Nuclear Research<o:p></o:p></pre><pre>1211 =
Geneva 23, Switzerland<o:p></o:p></pre><pre>Phone: +41 22 767 =
8069           &nb=
sp;        Office: =
892-2A-015<o:p></o:p></pre><pre>e-mail: <a
href=3D"mailto:Christian.Theis_at_cern.ch">Christian.Theis_at_cern.ch</a> =
          www: <a
href=3D"http://www.cern.ch/theis">http://www.cern.ch/theis</a><o:p></o:p>=
</pre><pre>--------------------------------------------------------------=
----------<o:p></o:p></pre><pre><o:p> </o:p></pre><pre><o:p> </=
o:p></pre><pre>  <o:p></o:p></pre>

<blockquote =
style=3D'margin-top:5.0pt;margin-bottom:5.0pt'><pre>-----Original =
Message-----<o:p></o:p></pre><pre>From: <a
href=3D"mailto:owner-fluka-discuss_at_mi.infn.it">owner-fluka-discuss_at_mi.inf=
n.it</a> [<a
href=3D"mailto:owner-fluka">mailto:owner-fluka</a>-<o:p></o:p></pre><pre>=
<a
href=3D"mailto:discuss_at_mi.infn.it">discuss_at_mi.infn.it</a>] On Behalf Of =
Finetti Noemi<o:p></o:p></pre><pre>Sent: 27 November 2008 =
15:31<o:p></o:p></pre><pre>To: <a
href=3D"mailto:fluka-discuss_at_fluka.org">fluka-discuss_at_fluka.org</a><o:p><=
/o:p></pre><pre>Subject: FLUKA =
crash<o:p></o:p></pre><pre><o:p> </o:p></pre><pre>Hi =
all,<o:p></o:p></pre><pre>I have installed FLUKA 2008.3 on a linux =
machine (Model: 2 AMD Opteron<o:p></o:p></pre><pre>(tm) Processor 244 - =
1.8GHz; Operating system: Linux - =
Scientific<o:p></o:p></pre><pre>    =
<o:p></o:p></pre></blockquote>

<pre>Linux<o:p></o:p></pre><pre>  <o:p></o:p></pre>

<blockquote style=3D'margin-top:5.0pt;margin-bottom:5.0pt'><pre>3.0.5; =
with g77) where I have compiled my executable file =
(myfluka)<o:p></o:p></pre><pre>which calls the user routines USRINI, =
HISTIN,SOURCE, USRMED, USROUT<o:p></o:p></pre><pre>(see =
user_routines.tar.gz). Executing the job in batch (for =
399528<o:p></o:p></pre><pre>primaries) the run crashed while running the =
same job interactively<o:p></o:p></pre><pre>(for =
100<o:p></o:p></pre><pre>primaries) every thing was fine. What could be =
the reasons?<o:p></o:p></pre><pre><o:p> </o:p></pre><pre>I point =
out that the batch job was executed on a 2 Dual Core =
AMD<o:p></o:p></pre><pre>Opteron(tm) Processor 280 - 2.4 GHz with Linux =
- Scientific =
Linux<o:p></o:p></pre><pre>3.0.8.<o:p></o:p></pre><pre><o:p> </o:p><=
/pre><pre>In attachment the LSF message, the gdb output (file =
fluka_gdb.out),<o:p></o:p></pre><pre>    =
<o:p></o:p></pre></blockquote>

<blockquote style=3D'margin-top:5.0pt;margin-bottom:5.0pt'><pre>files =
.err, .log, .out, fort.1 and =
fort.2.<o:p></o:p></pre><pre><o:p> </o:p></pre><pre>Thanks in =
advance,<o:p></o:p></pre><pre>noemi<o:p></o:p></pre><pre><o:p> </o:p=
></pre><pre>--<o:p></o:p></pre><pre>-------------------------------------=
--------------------------------<o:p></o:p></pre><pre>* Address: =
Dott.ssa Noemi Finetti<o:p></o:p></pre><pre>   c/o =
Dipartimento di Fisica dell'Universita' degli Studi =
dell'Aquila<o:p></o:p></pre><pre>   Via Vetoio - 67010 Coppito =
- L'Aquila - Italy<o:p></o:p></pre><pre>* Phone: +39-0862-433051 =
(Office); +39-0862-433043 (Laboratory)<o:p></o:p></pre><pre>* Fax: =
+39-0862-433033 =
(Department).<o:p></o:p></pre><pre>--------------------------------------=
-------------------------------<o:p></o:p></pre><pre><o:p> </o:p></p=
re><pre><o:p> </o:p></pre><pre><o:p> </o:p></pre><pre> &nb=
sp;  <o:p></o:p></pre></blockquote>

<o:p></o:p>

<pre>-- =
<o:p></o:p></pre><pre>---------------------------------------------------=
------------------<o:p></o:p></pre><pre>* Address: Dott.ssa Noemi =
Finetti<o:p></o:p></pre><pre>  c/o Dipartimento di Fisica =
dell'Universita' degli Studi dell'Aquila<o:p></o:p></pre><pre>  Via =
Vetoio - 67010 Coppito - L'Aquila - Italy<o:p></o:p></pre><pre>* Phone: =
+39-0862-433051 (Office); +39-0862-433043 =
(Laboratory)<o:p></o:p></pre><pre>* Fax: +39-0862-433033 =
(Department).<o:p></o:p></pre><pre>--------------------------------------=
-------------------------------<o:p></o:p></pre><pre><o:p> </o:p></p=
re><pre><o:p> </o:p></pre></div>

</div>

</body>

</html>

------_=_NextPart_001_01C95537.0C95CDE5--
Received on Wed Dec 03 2008 - 15:49:09 CET

This archive was generated by hypermail 2.2.0 : Wed Dec 03 2008 - 15:49:39 CET