<html><head></head><body><div class="ydp6a627375yahoo-style-wrap" style="font-family: Helvetica Neue, Helvetica, Arial, sans-serif; font-size: 13px;"><div><div dir="ltr" data-setdir="false">Thank you Michael for pitching in to trouble shoot the config file.</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">Now my configfile looks like:</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false"><div>ClusterName=linux<br>ControlMachine=abhi-Latitude-E6430<br>SlurmctldPort=6817<br>SlurmdPort=6818<br>AuthType=auth/munge<br>SwitchType=switch/none<br>MpiDefault=none<br>ProctrackType=proctrack/pgid<br>Epilog=/usr/local/slurm/sbin/epilog<br>Prolog=/usr/local/slurm/sbin/prolog<br>SlurmdSpoolDir=/var/tmp/slurmd.spool<br>StateSaveLocation=/usr/local/slurm/slurm.state<br>TmpFS=/tmp<br>NodeName=abhi-Lenovo-ideapad-330-15IKB CPUS=4<br>NodeName=abhi-HP-EliteBook-840-G2 CPUS=4<br>PartitionName=debug Nodes=ALL Default=YES MaxTime=INFINITE State=UP<br><br></div><div dir="ltr" data-setdir="false"><div>abhi@abhi-Lenovo-ideapad-330-15IKB:~$ service slurmd status<br>● slurmd.service - Slurm node daemon<br> Loaded: loaded (/lib/systemd/system/slurmd.service; enabled; vendor preset: enabled)<br> Active: active (running) since Thu 2020-05-14 04:11:32 IST; 2h 28min ago<br> Docs: man:slurmd(8)<br> Process: 977 ExecStart=/usr/sbin/slurmd $SLURMD_OPTIONS (code=exited, status=0/SUCCESS)<br> Main PID: 1028 (slurmd)<br> Tasks: 2<br> Memory: 3.9M<br> CGroup: /system.slice/slurmd.service<br> └─1028 /usr/sbin/slurmd<br><br></div><div dir="ltr" data-setdir="false"><div>abhi@abhi-HP-EliteBook-840-G2:~$ service slurmd status<br>● slurmd.service - Slurm node daemon<br> Loaded: loaded (/lib/systemd/system/slurmd.service; enabled; vendor preset: enabled)<br> Active: active (running) since Thu 2020-05-14 04:18:51 IST; 2h 24min ago<br> Docs: man:slurmd(8)<br> Process: 1313 ExecStart=/usr/sbin/slurmd $SLURMD_OPTIONS (code=exited, status=0/SUCCESS)<br> Main PID: 1372 (slurmd)<br> Tasks: 2<br> Memory: 3.8M<br> CGroup: /system.slice/slurmd.service<br> └─1372 /usr/sbin/slurmd<br><br></div><div dir="ltr" data-setdir="false"><div>abhi@abhi-Latitude-E6430:~$ service slurmctld status<br>● slurmctld.service - Slurm controller daemon<br> Loaded: loaded (/lib/systemd/system/slurmctld.service; enabled; vendor preset: enabled)<br> Active: active (running) since Thu 2020-05-14 04:11:21 IST; 2h 32min ago<br> Docs: man:slurmctld(8)<br> Process: 1208 ExecStart=/usr/sbin/slurmctld $SLURMCTLD_OPTIONS (code=exited, status=0/SUCCESS)<br> Main PID: 1306 (slurmctld)<br> Tasks: 7<br> Memory: 6.7M<br> CGroup: /system.slice/slurmctld.service<br><div> └─1306 /usr/sbin/slurmctld</div><div><br></div><div dir="ltr" data-setdir="false">However still:</div><div dir="ltr" data-setdir="false"><div> sinfo<br>PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br>debug* up infinite 1 down* abhi-Lenovo-ideapad-330-15IKB<br><b><br></b></div><div dir="ltr" data-setdir="false"><b>My Study is inconclusive<br></b></div></div><br></div></div></div></div></div><div class="ydp6a627375signature"><div style="font-family:Helvetica, Arial, sans-serif;font-size:13px;"><div dir="ltr">Best Regards,</div><div dir="ltr"><b><div class="ydp9cad327apasted-link"><div dir="ltr"><div><div dir="ltr" style="font-family:Helvetica, Arial, sans-serif;">Abhinandan H. Patil,<b> +919886406214</b></div><div class="ydp9cad327ayiv5893919466ydp86a042edyiv5783497666ydp139ac2easignature" style="font-family:Helvetica, Arial, sans-serif;"><div style="font-family:Helvetica, Arial, sans-serif;"><div dir="ltr"><a href="http://Abhinandan's home page" rel="nofollow" target="_blank"><b>https://www.AbhinandanHPatil.info</b></a></div></div></div></div><br></div><div><br></div></div></b></div></div></div></div>
<div><br></div><div><br></div>
</div><div id="ydpbc85793fyahoo_quoted_9714210591" class="ydpbc85793fyahoo_quoted">
<div style="font-family:'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:13px;color:#26282a;">
<div>----- Forwarded message -----</div>
<div><b>From:</b> "slurm-users-request@lists.schedmd.com" <slurm-users-request@lists.schedmd.com></div><div><b>To:</b> "slurm-users@lists.schedmd.com" <slurm-users@lists.schedmd.com></div><div><b>Sent:</b> Thursday, 14 May 2020, 2:39:40 am GMT+5:30</div><div><b>Subject:</b> slurm-users Digest, Vol 31, Issue 50</div><div><br></div>
<div><div dir="ltr">Send slurm-users mailing list submissions to<br></div><div dir="ltr"> <a href="mailto:slurm-users@lists.schedmd.com" rel="nofollow" target="_blank">slurm-users@lists.schedmd.com</a><br></div><div dir="ltr"><br></div><div dir="ltr">To subscribe or unsubscribe via the World Wide Web, visit<br></div><div dir="ltr"> <a href="https://lists.schedmd.com/cgi-bin/mailman/listinfo/slurm-users" rel="nofollow" target="_blank">https://lists.schedmd.com/cgi-bin/mailman/listinfo/slurm-users</a><br></div><div dir="ltr">or, via email, send a message with subject or body 'help' to<br></div><div dir="ltr"> <a href="mailto:slurm-users-request@lists.schedmd.com" rel="nofollow" target="_blank">slurm-users-request@lists.schedmd.com</a><br></div><div dir="ltr"><br></div><div dir="ltr">You can reach the person managing the list at<br></div><div dir="ltr"> <a href="mailto:slurm-users-owner@lists.schedmd.com" rel="nofollow" target="_blank">slurm-users-owner@lists.schedmd.com</a><br></div><div dir="ltr"><br></div><div dir="ltr">When replying, please edit your Subject line so it is more specific<br></div><div dir="ltr">than "Re: Contents of slurm-users digest..."<br></div><div dir="ltr"><br></div><div dir="ltr"><br></div><div dir="ltr">Today's Topics:<br></div><div dir="ltr"><br></div><div dir="ltr"> 1. Re: Ubuntu Cluster with Slurm (Renfro, Michael)<br></div><div dir="ltr"> 2. Re: sacct returns nothing after reboot (Roger Mason)<br></div><div dir="ltr"> 3. Re: Reset TMPDIR for All Jobs (Ellestad, Erik)<br></div><div dir="ltr"> 4. Re: additional jobs killed by scancel. (Alastair Neil)<br></div><div dir="ltr"><br></div><div dir="ltr"><br></div><div dir="ltr">----------------------------------------------------------------------<br></div><div dir="ltr"><br></div><div dir="ltr">Message: 1<br></div><div dir="ltr">Date: Wed, 13 May 2020 14:05:21 +0000<br></div><div dir="ltr">From: "Renfro, Michael" <<a href="mailto:Renfro@tntech.edu" rel="nofollow" target="_blank">Renfro@tntech.edu</a>><br></div><div dir="ltr">To: Slurm User Community List <<a href="mailto:slurm-users@lists.schedmd.com" rel="nofollow" target="_blank">slurm-users@lists.schedmd.com</a>><br></div><div dir="ltr">Subject: Re: [slurm-users] Ubuntu Cluster with Slurm<br></div><div dir="ltr">Message-ID: <<a href="mailto:B4E26014-E420-4506-8A7F-DCDF01E4AAD3@tntech.edu" rel="nofollow" target="_blank">B4E26014-E420-4506-8A7F-DCDF01E4AAD3@tntech.edu</a>><br></div><div dir="ltr">Content-Type: text/plain; charset="utf-8"<br></div><div dir="ltr"><br></div><div dir="ltr">I?d compare the RealMemory part of ?scontrol show node abhi-HP-EliteBook-840-G2? to the RealMemory part of your slurm.conf:<br></div><div dir="ltr"><br></div><div dir="ltr">> Nodes which register to the system with less than the configured resources (e.g. too little memory), will be placed in the "DOWN" state to avoid scheduling jobs on them.<br></div><div dir="ltr"><br></div><div dir="ltr">? <a href="https://slurm.schedmd.com/slurm.conf.html" rel="nofollow" target="_blank">https://slurm.schedmd.com/slurm.conf.html</a><br></div><div dir="ltr"><br></div><div dir="ltr">As far as GPUs go, it looks like you have Intel graphics on the Lenovo and a Radeon R7 on the HP? If so, then nothing is CUDA-compatible, but you might be able to make something work with OpenCL. No idea if that would give performance improvements over the CPUs, though.<br></div><div dir="ltr"><br></div><div dir="ltr">-- <br></div><div dir="ltr">Mike Renfro, PhD / HPC Systems Administrator, Information Technology Services<br></div><div dir="ltr">931 372-3601 / Tennessee Tech University<br></div><div dir="ltr"><br></div><div dir="ltr">> On May 13, 2020, at 8:42 AM, Abhinandan Patil <<a href="mailto:abhinandan_patil_1414@yahoo.com" rel="nofollow" target="_blank">abhinandan_patil_1414@yahoo.com</a>> wrote:<br></div><div dir="ltr">> <br></div><div dir="ltr">> Dear All,<br></div><div dir="ltr">> <br></div><div dir="ltr">> Preamble<br></div><div dir="ltr">> ----------<br></div><div dir="ltr">> I want to form simple cluster with three laptops:<br></div><div dir="ltr">> abhi-Latitude-E6430 //This serves as the controller<br></div><div dir="ltr">> abhi-Lenovo-ideapad-330-15IKB //Compute Node<br></div><div dir="ltr">> abhi-HP-EliteBook-840-G2 //Compute Node<br></div><div dir="ltr">> <br></div><div dir="ltr">> <br></div><div dir="ltr">> Aim<br></div><div dir="ltr">> -------------<br></div><div dir="ltr">> I want to make use of CPU+GPU+RAM on all the machines when I execute JAVA programs or Python programs.<br></div><div dir="ltr">> <br></div><div dir="ltr">> <br></div><div dir="ltr">> Implementation<br></div><div dir="ltr">> ------------------------<br></div><div dir="ltr">> Now let us look at the slurm.conf<br></div><div dir="ltr">> <br></div><div dir="ltr">> On Machine abhi-Latitude-E6430<br></div><div dir="ltr">> <br></div><div dir="ltr">> ClusterName=linux<br></div><div dir="ltr">> ControlMachine=abhi-Latitude-E6430<br></div><div dir="ltr">> SlurmUser=abhi<br></div><div dir="ltr">> SlurmctldPort=6817<br></div><div dir="ltr">> SlurmdPort=6818<br></div><div dir="ltr">> AuthType=auth/munge<br></div><div dir="ltr">> SwitchType=switch/none<br></div><div dir="ltr">> StateSaveLocation=/tmp<br></div><div dir="ltr">> MpiDefault=none<br></div><div dir="ltr">> ProctrackType=proctrack/pgid<br></div><div dir="ltr">> NodeName=abhi-Lenovo-ideapad-330-15IKB RealMemory=12000 CPUs=2<br></div><div dir="ltr">> NodeName=abhi-HP-EliteBook-840-G2 RealMemory=14000 CPUs=2<br></div><div dir="ltr">> PartitionName=debug Nodes=ALL Default=YES MaxTime=INFINITE State=UP<br></div><div dir="ltr">> <br></div><div dir="ltr">> Same slurm.conf is copied to all the Machines.<br></div><div dir="ltr">> <br></div><div dir="ltr">> <br></div><div dir="ltr">> Observations<br></div><div dir="ltr">> --------------------------------------<br></div><div dir="ltr">> Now when I do<br></div><div dir="ltr">> <a href="mailto:abhi@abhi-HP-EliteBook-840-G2" rel="nofollow" target="_blank">abhi@abhi-HP-EliteBook-840-G2</a>:~$ service slurmd status<br></div><div dir="ltr">> ? slurmd.service - Slurm node daemon<br></div><div dir="ltr">> Loaded: loaded (/lib/systemd/system/slurmd.service; enabled; vendor preset: enabled)<br></div><div dir="ltr">> Active: active (running) since Wed 2020-05-13 18:50:01 IST; 1min 49s ago<br></div><div dir="ltr">> Docs: man:slurmd(8)<br></div><div dir="ltr">> Process: 98235 ExecStart=/usr/sbin/slurmd $SLURMD_OPTIONS (code=exited, status=0/SUCCESS)<br></div><div dir="ltr">> Main PID: 98253 (slurmd)<br></div><div dir="ltr">> Tasks: 2<br></div><div dir="ltr">> Memory: 2.2M<br></div><div dir="ltr">> CGroup: /system.slice/slurmd.service<br></div><div dir="ltr">> ??98253 /usr/sbin/slurmd<br></div><div dir="ltr">> <br></div><div dir="ltr">> <a href="mailto:abhi@abhi-Lenovo-ideapad-330-15IKB" rel="nofollow" target="_blank">abhi@abhi-Lenovo-ideapad-330-15IKB</a>:~$ service slurmd status<br></div><div dir="ltr">> ? slurmd.service - Slurm node daemon<br></div><div dir="ltr">> Loaded: loaded (/lib/systemd/system/slurmd.service; enabled; vendor preset: enabled)<br></div><div dir="ltr">> Active: active (running) since Wed 2020-05-13 18:50:20 IST; 8s ago<br></div><div dir="ltr">> Docs: man:slurmd(8)<br></div><div dir="ltr">> Process: 71709 ExecStart=/usr/sbin/slurmd $SLURMD_OPTIONS (code=exited, status=0/SUCCESS)<br></div><div dir="ltr">> Main PID: 71734 (slurmd)<br></div><div dir="ltr">> Tasks: 2<br></div><div dir="ltr">> Memory: 2.0M<br></div><div dir="ltr">> CGroup: /system.slice/slurmd.service<br></div><div dir="ltr">> ??71734 /usr/sbin/slurmd<br></div><div dir="ltr">> <br></div><div dir="ltr">> <a href="mailto:abhi@abhi-Latitude-E6430" rel="nofollow" target="_blank">abhi@abhi-Latitude-E6430</a>:~$ service slurmctld status <br></div><div dir="ltr">> ? slurmctld.service - Slurm controller daemon<br></div><div dir="ltr">> Loaded: loaded (/lib/systemd/system/slurmctld.service; enabled; vendor preset: enabled)<br></div><div dir="ltr">> Active: active (running) since Wed 2020-05-13 18:48:58 IST; 4min 56s ago<br></div><div dir="ltr">> Docs: man:slurmctld(8)<br></div><div dir="ltr">> Process: 97114 ExecStart=/usr/sbin/slurmctld $SLURMCTLD_OPTIONS (code=exited, status=0/SUCCESS)<br></div><div dir="ltr">> Main PID: 97116 (slurmctld)<br></div><div dir="ltr">> Tasks: 7<br></div><div dir="ltr">> Memory: 2.6M<br></div><div dir="ltr">> CGroup: /system.slice/slurmctld.service<br></div><div dir="ltr">> ??97116 /usr/sbin/slurmctld<br></div><div dir="ltr">> <br></div><div dir="ltr">> <br></div><div dir="ltr">> However <a href="mailto:abhi@abhi-Latitude-E6430" rel="nofollow" target="_blank">abhi@abhi-Latitude-E6430</a>:~$ sinfo<br></div><div dir="ltr">> PARTITION AVAIL TIMELIMIT NODES STATE NODELIST<br></div><div dir="ltr">> debug* up infinite 1 down* abhi-Lenovo-ideapad-330-15IKB<br></div><div dir="ltr">> <br></div><div dir="ltr">> <br></div><div dir="ltr">> Advice needed<br></div><div dir="ltr">> ------------------------<br></div><div dir="ltr">> Please let me know Why I am seeing only one node. <br></div><div dir="ltr">> Further how the total memory is calculated? Can Slurm make use of GPU processing power as well<br></div><div dir="ltr">> Please let me know if I have missed something in configuration or explanation.<br></div><div dir="ltr">> <br></div><div dir="ltr">> Thank you all<br></div><div dir="ltr">> <br></div><div dir="ltr">> Best Regards,<br></div><div dir="ltr">> Abhinandan H. Patil, +919886406214<br></div><div dir="ltr">> <a href="https://www.AbhinandanHPatil.info" rel="nofollow" target="_blank">https://www.AbhinandanHPatil.info</a><br></div><div dir="ltr">> <br></div><div dir="ltr">> <br></div><div dir="ltr"><br></div><div dir="ltr"><br></div><div dir="ltr">------------------------------<br></div><div dir="ltr"><br></div><div dir="ltr">Message: 2<br></div><div dir="ltr">Date: Wed, 13 May 2020 12:20:11 -0230<br></div><div dir="ltr">From: Roger Mason <<a href="mailto:rmason@mun.ca" rel="nofollow" target="_blank">rmason@mun.ca</a>><br></div><div dir="ltr">To: Slurm User Community List <<a href="mailto:slurm-users@lists.schedmd.com" rel="nofollow" target="_blank">slurm-users@lists.schedmd.com</a>><br></div><div dir="ltr">Subject: Re: [slurm-users] sacct returns nothing after reboot<br></div><div dir="ltr">Message-ID: <<a href="mailto:y65sgg399ek.fsf@mun.ca" rel="nofollow" target="_blank">y65sgg399ek.fsf@mun.ca</a>><br></div><div dir="ltr">Content-Type: text/plain<br></div><div dir="ltr"><br></div><div dir="ltr">Hello,<br></div><div dir="ltr"><br></div><div dir="ltr">Marcus Boden <<a href="mailto:mboden@gwdg.de" rel="nofollow" target="_blank">mboden@gwdg.de</a>> writes:<br></div><div dir="ltr"><br></div><div dir="ltr">> the default time window starts at 00:00:00 of the current day:<br></div><div dir="ltr">> -S, --starttime<br></div><div dir="ltr">> Select jobs in any state after the specified time. Default<br></div><div dir="ltr">> is 00:00:00 of the current day, unless the '-s' or '-j'<br></div><div dir="ltr">> options are used. If the '-s' option is used, then the<br></div><div dir="ltr">> default is 'now'. If states are given with the '-s' option<br></div><div dir="ltr">> then only jobs in this state at this time will be returned.<br></div><div dir="ltr">> If the '-j' option is used, then the default time is Unix<br></div><div dir="ltr">> Epoch 0. See the DEFAULT TIME WINDOW for more details.<br></div><div dir="ltr"><br></div><div dir="ltr">Thank you! Obviously I did not read far enough down the man page.<br></div><div dir="ltr"><br></div><div dir="ltr">Roger<br></div><div dir="ltr"><br></div><div dir="ltr"><br></div><div dir="ltr"><br></div><div dir="ltr">------------------------------<br></div><div dir="ltr"><br></div><div dir="ltr">Message: 3<br></div><div dir="ltr">Date: Wed, 13 May 2020 15:18:09 +0000<br></div><div dir="ltr">From: "Ellestad, Erik" <<a href="mailto:Erik.Ellestad@ucsf.edu" rel="nofollow" target="_blank">Erik.Ellestad@ucsf.edu</a>><br></div><div dir="ltr">To: Slurm User Community List <<a href="mailto:slurm-users@lists.schedmd.com" rel="nofollow" target="_blank">slurm-users@lists.schedmd.com</a>><br></div><div dir="ltr">Subject: Re: [slurm-users] Reset TMPDIR for All Jobs<br></div><div dir="ltr">Message-ID:<br></div><div dir="ltr"> <<a href="mailto:BY5PR05MB690060B056D48D6B031A2DA99ABF0@BY5PR05MB6900.namprd05.prod.outlook.com" rel="nofollow" target="_blank">BY5PR05MB690060B056D48D6B031A2DA99ABF0@BY5PR05MB6900.namprd05.prod.outlook.com</a>><br></div><div dir="ltr"> <br></div><div dir="ltr">Content-Type: text/plain; charset="utf-8"<br></div><div dir="ltr"><br></div><div dir="ltr">Woo!<br></div><div dir="ltr"><br></div><div dir="ltr">Thanks Marcus, that works.<br></div><div dir="ltr"><br></div><div dir="ltr">Though, ahem, SLURM/SchedMD, if you're listening, would it hurt to cover this in the documentation regarding prolog/epilog, and maybe give an example?<br></div><div dir="ltr"><br></div><div dir="ltr"><a href="https://slurm.schedmd.com/prolog_epilog.html" rel="nofollow" target="_blank">https://slurm.schedmd.com/prolog_epilog.html</a><br></div><div dir="ltr"><br></div><div dir="ltr">Just a thought,<br></div><div dir="ltr"><br></div><div dir="ltr">Erik<br></div><div dir="ltr"><br></div><div dir="ltr">--<br></div><div dir="ltr">Erik Ellestad<br></div><div dir="ltr">Wynton Cluster SysAdmin<br></div><div dir="ltr">UCSF<br></div><div dir="ltr"><br></div><div dir="ltr"><br></div><div dir="ltr">-----Original Message-----<br></div><div dir="ltr">From: slurm-users <<a href="mailto:slurm-users-bounces@lists.schedmd.com" rel="nofollow" target="_blank">slurm-users-bounces@lists.schedmd.com</a>> On Behalf Of Marcus Wagner<br></div><div dir="ltr">Sent: Tuesday, May 12, 2020 10:08 PM<br></div><div dir="ltr">To: <a href="mailto:slurm-users@lists.schedmd.com" rel="nofollow" target="_blank">slurm-users@lists.schedmd.com</a><br></div><div dir="ltr">Subject: Re: [slurm-users] Reset TMPDIR for All Jobs<br></div><div dir="ltr"><br></div><div dir="ltr">Hi Erik,<br></div><div dir="ltr"><br></div><div dir="ltr">the output of task-prolog is sourced/evaluated (not really sure, how) in <br></div><div dir="ltr">the job environment.<br></div><div dir="ltr"><br></div><div dir="ltr">Thus you don't have to export a variable in task-prolog, but echo the <br></div><div dir="ltr">export, e.g.<br></div><div dir="ltr"><br></div><div dir="ltr">echo export TMPDIR=/scratch/$SLURM_JOB_ID<br></div><div dir="ltr"><br></div><div dir="ltr">The variable will then be set in job environment.<br></div><div dir="ltr"><br></div><div dir="ltr"><br></div><div dir="ltr">Best<br></div><div dir="ltr">Marcus<br></div><div dir="ltr"><br></div><div dir="ltr">Am 12.05.2020 um 17:40 schrieb Ellestad, Erik:<br></div><div dir="ltr">> I was wanted to set TMPDIR from /tmp to a per job directory I create in <br></div><div dir="ltr">> local /scratch/$SLURM_JOB_ID (for example)<br></div><div dir="ltr">> <br></div><div dir="ltr">> This bug suggests I should be able to do this in a task-prolog.<br></div><div dir="ltr">> <br></div><div dir="ltr">> <a href="https://bugs.schedmd.com/show_bug.cgi?id=2664" rel="nofollow" target="_blank">https://bugs.schedmd.com/show_bug.cgi?id=2664</a><br></div><div dir="ltr">> <br></div><div dir="ltr">> However adding the following to task-prolog doesn?t seem to affect the <br></div><div dir="ltr">> variables the job script is running with.<br></div><div dir="ltr">> <br></div><div dir="ltr">> unset TMPDIR<br></div><div dir="ltr">> <br></div><div dir="ltr">> export TMPDIR=/scratch/$SLURM_JOB_ID<br></div><div dir="ltr">> <br></div><div dir="ltr">> It does work if it is done in the job script, rather than the task-prolog.<br></div><div dir="ltr">> <br></div><div dir="ltr">> Am I missing something?<br></div><div dir="ltr">> <br></div><div dir="ltr">> Erik<br></div><div dir="ltr">> <br></div><div dir="ltr">> --<br></div><div dir="ltr">> <br></div><div dir="ltr">> Erik Ellestad<br></div><div dir="ltr">> <br></div><div dir="ltr">> Wynton Cluster SysAdmin<br></div><div dir="ltr">> <br></div><div dir="ltr">> UCSF<br></div><div dir="ltr">> <br></div><div dir="ltr"><br></div><div dir="ltr"><br></div><div dir="ltr">------------------------------<br></div><div dir="ltr"><br></div><div dir="ltr">Message: 4<br></div><div dir="ltr">Date: Wed, 13 May 2020 17:08:55 -0400<br></div><div dir="ltr">From: Alastair Neil <<a href="mailto:ajneil.tech@gmail.com" rel="nofollow" target="_blank">ajneil.tech@gmail.com</a>><br></div><div dir="ltr">To: Slurm User Community List <<a href="mailto:slurm-users@lists.schedmd.com" rel="nofollow" target="_blank">slurm-users@lists.schedmd.com</a>><br></div><div dir="ltr">Subject: Re: [slurm-users] additional jobs killed by scancel.<br></div><div dir="ltr">Message-ID:<br></div><div dir="ltr"> <CA+<a href="mailto:SarwpQMepkhWLC_RUqSi1SzaNb8MHk77wCSFQAFFyTB7fx2Q@mail.gmail.com" rel="nofollow" target="_blank">SarwpQMepkhWLC_RUqSi1SzaNb8MHk77wCSFQAFFyTB7fx2Q@mail.gmail.com</a>><br></div><div dir="ltr">Content-Type: text/plain; charset="utf-8"<br></div><div dir="ltr"><br></div><div dir="ltr">invalid field requested: "reason"<br></div><div dir="ltr"><br></div><div dir="ltr">On Tue, 12 May 2020 at 16:47, Steven Dick <<a href="mailto:kg4ydw@gmail.com" rel="nofollow" target="_blank">kg4ydw@gmail.com</a>> wrote:<br></div><div dir="ltr"><br></div><div dir="ltr">> What do you get from<br></div><div dir="ltr">><br></div><div dir="ltr">> sacct -o jobid,elapsed,reason,exit -j 533900,533902<br></div><div dir="ltr">><br></div><div dir="ltr">> On Tue, May 12, 2020 at 4:12 PM Alastair Neil <<a href="mailto:ajneil.tech@gmail.com" rel="nofollow" target="_blank">ajneil.tech@gmail.com</a>><br></div><div dir="ltr">> wrote:<br></div><div dir="ltr">> ><br></div><div dir="ltr">> > The log is continuous and has all the messages logged by slurmd on the<br></div><div dir="ltr">> node for all the jobs mentioned, below are the entries from the slurmctld<br></div><div dir="ltr">> log:<br></div><div dir="ltr">> ><br></div><div dir="ltr">> >> [2020-05-10T00:26:03.097] _slurm_rpc_kill_job: REQUEST_KILL_JOB<br></div><div dir="ltr">> JobId=533898 uid 1224431221<br></div><div dir="ltr">> >><br></div><div dir="ltr">> >> [2020-05-10T00:26:03.098] email msg to <a href="mailto:sshres2@masonlive.gmu.edu" rel="nofollow" target="_blank">sshres2@masonlive.gmu.edu</a>:<br></div><div dir="ltr">> Slurm Job_id=533898 Name=r18-relu-ent Ended, Run time 04:36:17, CANCELLED,<br></div><div dir="ltr">> ExitCode 0<br></div><div dir="ltr">> >> [2020-05-10T00:26:03.098] job_signal: 9 of running JobId=533898<br></div><div dir="ltr">> successful 0x8004<br></div><div dir="ltr">> >> [2020-05-10T00:26:05.204] _job_complete: JobId=533902 WTERMSIG 9<br></div><div dir="ltr">> >> [2020-05-10T00:26:05.204] email msg to <a href="mailto:sshres2@masonlive.gmu.edu" rel="nofollow" target="_blank">sshres2@masonlive.gmu.edu</a>:<br></div><div dir="ltr">> Slurm Job_id=533902 Name=r18-soft-ent Failed, Run time 04:30:39, FAILED<br></div><div dir="ltr">> >> [2020-05-10T00:26:05.205] _job_complete: JobId=533902 done<br></div><div dir="ltr">> >> [2020-05-10T00:26:05.210] _job_complete: JobId=533900 WTERMSIG 9<br></div><div dir="ltr">> >> [2020-05-10T00:26:05.210] email msg to <a href="mailto:sshres2@masonlive.gmu.edu" rel="nofollow" target="_blank">sshres2@masonlive.gmu.edu</a>:<br></div><div dir="ltr">> Slurm Job_id=533900 Name=r18-soft Failed, Run time 04:32:51, FAILED<br></div><div dir="ltr">> >> [2020-05-10T00:26:05.215] _job_complete: JobId=533900 done<br></div><div dir="ltr">> ><br></div><div dir="ltr">> ><br></div><div dir="ltr">> > it is curious, that all the jobs were running on the same processor,<br></div><div dir="ltr">> perhaps this is a cgroup related failure?<br></div><div dir="ltr">> ><br></div><div dir="ltr">> > On Tue, 12 May 2020 at 10:10, Steven Dick <<a href="mailto:kg4ydw@gmail.com" rel="nofollow" target="_blank">kg4ydw@gmail.com</a>> wrote:<br></div><div dir="ltr">> >><br></div><div dir="ltr">> >> I see one job cancelled and two jobs failed.<br></div><div dir="ltr">> >> Your slurmd log is incomplete -- it doesn't show the two failed jobs<br></div><div dir="ltr">> >> exiting/failing, so the real error is not here.<br></div><div dir="ltr">> >><br></div><div dir="ltr">> >> It might also be helpful to look through slurmctld's log starting from<br></div><div dir="ltr">> >> when the first job was canceled, looking at any messages mentioning<br></div><div dir="ltr">> >> the node or the two failed jobs.<br></div><div dir="ltr">> >><br></div><div dir="ltr">> >> I've had nodes do strange things on job cancel. Last one I tracked<br></div><div dir="ltr">> >> down to the job epilog failing because it was NFS mounted and nfs was<br></div><div dir="ltr">> >> being slower than slurm liked, so it took the node offline and killed<br></div><div dir="ltr">> >> everything on it.<br></div><div dir="ltr">> >><br></div><div dir="ltr">> >> On Mon, May 11, 2020 at 12:55 PM Alastair Neil <<a href="mailto:ajneil.tech@gmail.com" rel="nofollow" target="_blank">ajneil.tech@gmail.com</a>><br></div><div dir="ltr">> wrote:<br></div><div dir="ltr">> >> ><br></div><div dir="ltr">> >> > Hi there,<br></div><div dir="ltr">> >> ><br></div><div dir="ltr">> >> > We are using slurm 18.08 and had a weird occurrence over the<br></div><div dir="ltr">> weekend. A user canceled one of his jobs using scancel, and two additional<br></div><div dir="ltr">> jobs of the user running on the same node were killed concurrently. The<br></div><div dir="ltr">> jobs had no dependency, but they were all allocated 1 gpu. I am curious to<br></div><div dir="ltr">> know why this happened, and if this is a known bug is there a workaround<br></div><div dir="ltr">> to prevent it happening? Any suggestions gratefully received.<br></div><div dir="ltr">> >> ><br></div><div dir="ltr">> >> > -Alastair<br></div><div dir="ltr">> >> ><br></div><div dir="ltr">> >> > FYI<br></div><div dir="ltr">> >> > The cancelled job (533898) has this at the end of the .err file:<br></div><div dir="ltr">> >> ><br></div><div dir="ltr">> >> >> slurmstepd: error: *** JOB 533898 ON NODE056 CANCELLED AT<br></div><div dir="ltr">> 2020-05-10T00:26:03 ***<br></div><div dir="ltr">> >> ><br></div><div dir="ltr">> >> ><br></div><div dir="ltr">> >> > both of the killed jobs (533900 and 533902) have this:<br></div><div dir="ltr">> >> ><br></div><div dir="ltr">> >> >> slurmstepd: error: get_exit_code task 0 died by signal<br></div><div dir="ltr">> >> ><br></div><div dir="ltr">> >> ><br></div><div dir="ltr">> >> > here is the slurmd log from the node and ths how-job output for each<br></div><div dir="ltr">> job:<br></div><div dir="ltr">> >> ><br></div><div dir="ltr">> >> >> [2020-05-09T19:49:46.735] _run_prolog: run job script took usec=4<br></div><div dir="ltr">> >> >> [2020-05-09T19:49:46.735] _run_prolog: prolog with lock for job<br></div><div dir="ltr">> 533898 ran for 0 seconds<br></div><div dir="ltr">> >> >> [2020-05-09T19:49:46.754] ====================<br></div><div dir="ltr">> >> >> [2020-05-09T19:49:46.754] batch_job:533898 job_mem:10240MB<br></div><div dir="ltr">> >> >> [2020-05-09T19:49:46.754] JobNode[0] CPU[0] Job alloc<br></div><div dir="ltr">> >> >> [2020-05-09T19:49:46.755] JobNode[0] CPU[1] Job alloc<br></div><div dir="ltr">> >> >> [2020-05-09T19:49:46.756] JobNode[0] CPU[2] Job alloc<br></div><div dir="ltr">> >> >> [2020-05-09T19:49:46.757] JobNode[0] CPU[3] Job alloc<br></div><div dir="ltr">> >> >> [2020-05-09T19:49:46.758] ====================<br></div><div dir="ltr">> >> >> [2020-05-09T19:49:46.758] Launching batch job 533898 for UID<br></div><div dir="ltr">> 1224431221<br></div><div dir="ltr">> >> >> [2020-05-09T19:53:14.060] _run_prolog: run job script took usec=3<br></div><div dir="ltr">> >> >> [2020-05-09T19:53:14.060] _run_prolog: prolog with lock for job<br></div><div dir="ltr">> 533900 ran for 0 seconds<br></div><div dir="ltr">> >> >> [2020-05-09T19:53:14.080] ====================<br></div><div dir="ltr">> >> >> [2020-05-09T19:53:14.080] batch_job:533900 job_mem:10240MB<br></div><div dir="ltr">> >> >> [2020-05-09T19:53:14.081] JobNode[0] CPU[4] Job alloc<br></div><div dir="ltr">> >> >> [2020-05-09T19:53:14.082] JobNode[0] CPU[5] Job alloc<br></div><div dir="ltr">> >> >> [2020-05-09T19:53:14.083] JobNode[0] CPU[6] Job alloc<br></div><div dir="ltr">> >> >> [2020-05-09T19:53:14.083] JobNode[0] CPU[7] Job alloc<br></div><div dir="ltr">> >> >> [2020-05-09T19:53:14.084] ====================<br></div><div dir="ltr">> >> >> [2020-05-09T19:53:14.085] Launching batch job 533900 for UID<br></div><div dir="ltr">> 1224431221<br></div><div dir="ltr">> >> >> [2020-05-09T19:55:26.283] _run_prolog: run job script took usec=21<br></div><div dir="ltr">> >> >> [2020-05-09T19:55:26.284] _run_prolog: prolog with lock for job<br></div><div dir="ltr">> 533902 ran for 0 seconds<br></div><div dir="ltr">> >> >> [2020-05-09T19:55:26.304] ====================<br></div><div dir="ltr">> >> >> [2020-05-09T19:55:26.304] batch_job:533902 job_mem:10240MB<br></div><div dir="ltr">> >> >> [2020-05-09T19:55:26.304] JobNode[0] CPU[8] Job alloc<br></div><div dir="ltr">> >> >> [2020-05-09T19:55:26.305] JobNode[0] CPU[9] Job alloc<br></div><div dir="ltr">> >> >> [2020-05-09T19:55:26.306] JobNode[0] CPU[10] Job alloc<br></div><div dir="ltr">> >> >> [2020-05-09T19:55:26.306] JobNode[0] CPU[11] Job alloc<br></div><div dir="ltr">> >> >> [2020-05-09T19:55:26.307] ====================<br></div><div dir="ltr">> >> >> [2020-05-09T19:55:26.307] Launching batch job 533902 for UID<br></div><div dir="ltr">> 1224431221<br></div><div dir="ltr">> >> >> [2020-05-10T00:26:03.127] [533898.extern] done with job<br></div><div dir="ltr">> >> >> [2020-05-10T00:26:03.975] [533898.batch] error: *** JOB 533898 ON<br></div><div dir="ltr">> NODE056 CANCELLED AT 2020-05-10T00:26:03 ***<br></div><div dir="ltr">> >> >> [2020-05-10T00:26:04.425] [533898.batch] sending<br></div><div dir="ltr">> REQUEST_COMPLETE_BATCH_SCRIPT, error:0 status 15<br></div><div dir="ltr">> >> >> [2020-05-10T00:26:04.428] [533898.batch] done with job<br></div><div dir="ltr">> >> >> [2020-05-10T00:26:05.202] [533900.batch] error: get_exit_code task 0<br></div><div dir="ltr">> died by signal<br></div><div dir="ltr">> >> >> [2020-05-10T00:26:05.202] [533902.batch] error: get_exit_code task 0<br></div><div dir="ltr">> died by signal<br></div><div dir="ltr">> >> >> [2020-05-10T00:26:05.202] [533900.batch] sending<br></div><div dir="ltr">> REQUEST_COMPLETE_BATCH_SCRIPT, error:0 status 9<br></div><div dir="ltr">> >> >> [2020-05-10T00:26:05.202] [533902.batch] sending<br></div><div dir="ltr">> REQUEST_COMPLETE_BATCH_SCRIPT, error:0 status 9<br></div><div dir="ltr">> >> >> [2020-05-10T00:26:05.211] [533902.batch] done with job<br></div><div dir="ltr">> >> >> [2020-05-10T00:26:05.216] [533900.batch] done with job<br></div><div dir="ltr">> >> >> [2020-05-10T00:26:05.234] [533902.extern] done with job<br></div><div dir="ltr">> >> >> [2020-05-10T00:26:05.235] [533900.extern] done with job<br></div><div dir="ltr">> >> ><br></div><div dir="ltr">> >> ><br></div><div dir="ltr">> >> >> [<a href="mailto:root@node056" rel="nofollow" target="_blank">root@node056</a> 2020-05-10]# cat 533{898,900,902}/show-job.txt<br></div><div dir="ltr">> >> >> JobId=533898 JobName=r18-relu-ent<br></div><div dir="ltr">> >> >> UserId=sshres2(1224431221) GroupId=users(100) MCS_label=N/A<br></div><div dir="ltr">> >> >> Priority=19375 Nice=0 Account=csjkosecka QOS=csqos<br></div><div dir="ltr">> >> >> JobState=CANCELLED Reason=None Dependency=(null)<br></div><div dir="ltr">> >> >> Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:15<br></div><div dir="ltr">> >> >> RunTime=04:36:17 TimeLimit=5-00:00:00 TimeMin=N/A<br></div><div dir="ltr">> >> >> SubmitTime=2020-05-09T19:49:45 EligibleTime=2020-05-09T19:49:45<br></div><div dir="ltr">> >> >> AccrueTime=2020-05-09T19:49:45<br></div><div dir="ltr">> >> >> StartTime=2020-05-09T19:49:46 EndTime=2020-05-10T00:26:03<br></div><div dir="ltr">> Deadline=N/A<br></div><div dir="ltr">> >> >> PreemptTime=None SuspendTime=None SecsPreSuspend=0<br></div><div dir="ltr">> >> >> LastSchedEval=2020-05-09T19:49:46<br></div><div dir="ltr">> >> >> Partition=gpuq AllocNode:Sid=ARGO-2:7221<br></div><div dir="ltr">> >> >> ReqNodeList=(null) ExcNodeList=(null)<br></div><div dir="ltr">> >> >> NodeList=NODE056<br></div><div dir="ltr">> >> >> BatchHost=NODE056<br></div><div dir="ltr">> >> >> NumNodes=1 NumCPUs=4 NumTasks=0 CPUs/Task=4 ReqB:S:C:T=0:0:*:*<br></div><div dir="ltr">> >> >> TRES=cpu=4,mem=10G,node=1,billing=4,gres/gpu=1<br></div><div dir="ltr">> >> >> Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*<br></div><div dir="ltr">> >> >> MinCPUsNode=4 MinMemoryNode=10G MinTmpDiskNode=0<br></div><div dir="ltr">> >> >> Features=(null) DelayBoot=00:00:00<br></div><div dir="ltr">> >> >> OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null)<br></div><div dir="ltr">> >> >><br></div><div dir="ltr">> Command=/scratch/sshres2/workspace-scratch/cs747-project/command_resnet18_relu_ent.slurm<br></div><div dir="ltr">> >> >> WorkDir=/scratch/sshres2/workspace-scratch/cs747-project<br></div><div dir="ltr">> >> >><br></div><div dir="ltr">> StdErr=/scratch/sshres2/workspace-scratch/cs747-project/logs_slurm/r18-relu-ent-533898.err<br></div><div dir="ltr">> >> >> StdIn=/dev/null<br></div><div dir="ltr">> >> >><br></div><div dir="ltr">> StdOut=/scratch/sshres2/workspace-scratch/cs747-project/logs_slurm/r18-relu-ent-533898.out<br></div><div dir="ltr">> >> >> Power=<br></div><div dir="ltr">> >> >> TresPerNode=gpu:1<br></div><div dir="ltr">> >> >><br></div><div dir="ltr">> >> >> JobId=533900 JobName=r18-soft<br></div><div dir="ltr">> >> >> UserId=sshres2(1224431221) GroupId=users(100) MCS_label=N/A<br></div><div dir="ltr">> >> >> Priority=19375 Nice=0 Account=csjkosecka QOS=csqos<br></div><div dir="ltr">> >> >> JobState=FAILED Reason=JobLaunchFailure Dependency=(null)<br></div><div dir="ltr">> >> >> Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:9<br></div><div dir="ltr">> >> >> RunTime=04:32:51 TimeLimit=5-00:00:00 TimeMin=N/A<br></div><div dir="ltr">> >> >> SubmitTime=2020-05-09T19:53:13 EligibleTime=2020-05-09T19:53:13<br></div><div dir="ltr">> >> >> AccrueTime=2020-05-09T19:53:13<br></div><div dir="ltr">> >> >> StartTime=2020-05-09T19:53:14 EndTime=2020-05-10T00:26:05<br></div><div dir="ltr">> Deadline=N/A<br></div><div dir="ltr">> >> >> PreemptTime=None SuspendTime=None SecsPreSuspend=0<br></div><div dir="ltr">> >> >> LastSchedEval=2020-05-09T19:53:14<br></div><div dir="ltr">> >> >> Partition=gpuq AllocNode:Sid=ARGO-2:7221<br></div><div dir="ltr">> >> >> ReqNodeList=(null) ExcNodeList=(null)<br></div><div dir="ltr">> >> >> NodeList=NODE056<br></div><div dir="ltr">> >> >> BatchHost=NODE056<br></div><div dir="ltr">> >> >> NumNodes=1 NumCPUs=4 NumTasks=0 CPUs/Task=4 ReqB:S:C:T=0:0:*:*<br></div><div dir="ltr">> >> >> TRES=cpu=4,mem=10G,node=1,billing=4,gres/gpu=1<br></div><div dir="ltr">> >> >> Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*<br></div><div dir="ltr">> >> >> MinCPUsNode=4 MinMemoryNode=10G MinTmpDiskNode=0<br></div><div dir="ltr">> >> >> Features=(null) DelayBoot=00:00:00<br></div><div dir="ltr">> >> >> OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null)<br></div><div dir="ltr">> >> >><br></div><div dir="ltr">> Command=/scratch/sshres2/workspace-scratch/cs747-project/command_resnet18_soft.slurm<br></div><div dir="ltr">> >> >> WorkDir=/scratch/sshres2/workspace-scratch/cs747-project<br></div><div dir="ltr">> >> >><br></div><div dir="ltr">> StdErr=/scratch/sshres2/workspace-scratch/cs747-project/logs_slurm/r18-soft-533900.err<br></div><div dir="ltr">> >> >> StdIn=/dev/null<br></div><div dir="ltr">> >> >><br></div><div dir="ltr">> StdOut=/scratch/sshres2/workspace-scratch/cs747-project/logs_slurm/r18-soft-533900.out<br></div><div dir="ltr">> >> >> Power=<br></div><div dir="ltr">> >> >> TresPerNode=gpu:1<br></div><div dir="ltr">> >> >><br></div><div dir="ltr">> >> >> JobId=533902 JobName=r18-soft-ent<br></div><div dir="ltr">> >> >> UserId=sshres2(1224431221) GroupId=users(100) MCS_label=N/A<br></div><div dir="ltr">> >> >> Priority=19375 Nice=0 Account=csjkosecka QOS=csqos<br></div><div dir="ltr">> >> >> JobState=FAILED Reason=JobLaunchFailure Dependency=(null)<br></div><div dir="ltr">> >> >> Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:9<br></div><div dir="ltr">> >> >> RunTime=04:30:39 TimeLimit=5-00:00:00 TimeMin=N/A<br></div><div dir="ltr">> >> >> SubmitTime=2020-05-09T19:55:26 EligibleTime=2020-05-09T19:55:26<br></div><div dir="ltr">> >> >> AccrueTime=2020-05-09T19:55:26<br></div><div dir="ltr">> >> >> StartTime=2020-05-09T19:55:26 EndTime=2020-05-10T00:26:05<br></div><div dir="ltr">> Deadline=N/A<br></div><div dir="ltr">> >> >> PreemptTime=None SuspendTime=None SecsPreSuspend=0<br></div><div dir="ltr">> >> >> LastSchedEval=2020-05-09T19:55:26<br></div><div dir="ltr">> >> >> Partition=gpuq AllocNode:Sid=ARGO-2:7221<br></div><div dir="ltr">> >> >> ReqNodeList=(null) ExcNodeList=(null)<br></div><div dir="ltr">> >> >> NodeList=NODE056<br></div><div dir="ltr">> >> >> BatchHost=NODE056<br></div><div dir="ltr">> >> >> NumNodes=1 NumCPUs=4 NumTasks=0 CPUs/Task=4 ReqB:S:C:T=0:0:*:*<br></div><div dir="ltr">> >> >> TRES=cpu=4,mem=10G,node=1,billing=4,gres/gpu=1<br></div><div dir="ltr">> >> >> Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*<br></div><div dir="ltr">> >> >> MinCPUsNode=4 MinMemoryNode=10G MinTmpDiskNode=0<br></div><div dir="ltr">> >> >> Features=(null) DelayBoot=00:00:00<br></div><div dir="ltr">> >> >> OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null)<br></div><div dir="ltr">> >> >><br></div><div dir="ltr">> Command=/scratch/sshres2/workspace-scratch/cs747-project/command_resnet18_soft_ent.slurm<br></div><div dir="ltr">> >> >> WorkDir=/scratch/sshres2/workspace-scratch/cs747-project<br></div><div dir="ltr">> >> >><br></div><div dir="ltr">> StdErr=/scratch/sshres2/workspace-scratch/cs747-project/logs_slurm/r18-soft-ent-533902.err<br></div><div dir="ltr">> >> >> StdIn=/dev/null<br></div><div dir="ltr">> >> >><br></div><div dir="ltr">> StdOut=/scratch/sshres2/workspace-scratch/cs747-project/logs_slurm/r18-soft-ent-533902.out<br></div><div dir="ltr">> >> >> Power=<br></div><div dir="ltr">> >> >> TresPerNode=gpu:1<br></div><div dir="ltr">> >> ><br></div><div dir="ltr">> >> ><br></div><div dir="ltr">> >> ><br></div><div dir="ltr">> >><br></div><div dir="ltr">><br></div><div dir="ltr">><br></div><div dir="ltr">-------------- next part --------------<br></div><div dir="ltr">An HTML attachment was scrubbed...<br></div><div dir="ltr">URL: <<a href="http://lists.schedmd.com/pipermail/slurm-users/attachments/20200513/8ff7b80b/attachment.htm" rel="nofollow" target="_blank">http://lists.schedmd.com/pipermail/slurm-users/attachments/20200513/8ff7b80b/attachment.htm</a>><br></div><div dir="ltr"><br></div><div dir="ltr">End of slurm-users Digest, Vol 31, Issue 50<br></div><div dir="ltr">*******************************************<br></div></div>
</div>
</div></body></html>