Lumerical FDTD

Introduction

Employing the industry-proven finite-difference time-domain (FDTD) method, FDTD Solutions empowers device and components designers to confront challenging optical design problems. Rapid prototyping and highly-accurate simulations reduce reliance upon costly experimental prototypes, leading to quicker assessment of design concepts and reduced product development costs. Software documentation, including Getting Started, User Guide and Application Help sections can be found at http://www.lumerical.com/fdtd_online_help/fdtd_online_help_summary.php

Benefits

• Reduced time-to-market owing to a highly-optimized simulation engine engineered for high-throughput design assessment on leading-edge computational systems
• Decreased product development costs via highly-accurate algorithms and design methods that allow for virtual prototyping to replace costly physical prototype building and testing
• Increased productivity via design tools engineered with ease-of-use in mind to facilitate fast learning and rapid deployment

Overview

• 2D and 3D simulation software capabilities
• Design parameterization and hierarchical layout via structure groups and analysis groups
• Nonuniform mesh and automesh algorithms
• Simulation convergence autoshutoff
• Parallel computation on multi-core, multiprocessor, and high-performance computing clusters
• Movie (.mpg) generation of simulation dynamics
• Scripting software language to customize simulation and analysis and perform powerful post-processing of simulation results
• Lorentz, Drude, Debye and anisotropic materials
• Data import/export with BRO's ASAP ray-tracing software package
• Data export to Matlab software, or ASCII file formats
• Structure import from GDSII files

Features

• Boundary conditions:absorbing (PML), periodic, Bloch, symmetric, asymmetric, and metal boundaries
• Simulation objects:primitives that can be rotated and placed in three dimensions, and structure definition from imported SEM/image files; primitives include triangles, rectangular blocks, cylinders, conic surfaces, polyons, rings, user-defined (parametric) surfaces, spheres and pyramids
• Radiation sources:waveguide sources, dipoles, plane waves, focused beams and diffraction-limited spots, total-field scattered-field (TFSF) sources, and source import/export from/to BRO's ASAP ray-tracing program
• Measurement monitors:refractive index monitors, time- and frequency-domain monitors to measure pulsed or continuous-wave (CW) field profiles and power flow, and movie monitors to generate .mpg movies of field dynamics

Licensing Issues

Lumerical has freely provided 10 FDTD engine licenses for running simulation jobs on the cluster by academic users. Each simulation job can use as many nodes as desired. In order to setup the simulation files and analyse the results, it is necessary to have access to the CAD graphical layout editor which comes with the full software license. New users who are interested in testing simulations on the cluster can download a free, 30 day trial of the software from www.lumerical.com. Current users of FDTD Solutions who have access to the cluster can simply submit their simulation files at any time. This provides a convenient way to run large simulation files or parameter sweeps quickly at no additional charge.

How to use

The configured parallel implementation is the MPICH1 based. The FDTD license server, with maximum of running jobs 10 was also installed in the cluster. Software is advertised in the na4.vo.eu-egee.org VO via the tag : VO-na4.vo.eu-egee.org-lumerical-FDTD-6.5.9.

To get access to the FTDT code please first apply for membership in the na4.vo.eu-egee.org VO of the EGEE grid. Once your membership is approved, you should first create a VOMS proxy on the User Interface machine using :

voms-proxy-init --voms na4.vo.eu-egee.org

NOTE: For this example, we are using the FQDNs of the Cream Computing Element (cream-ce01.marie.hellasgrid.gr) and the Storage Element (se01.marie.hellasgrid.gr) hosted at HG-02-IASA. You can change the CreamCE /SE FQDNs to point to another cluster where FDTD is installed (See Note after jdl description bellow).

In order to run a job, one needs a JDL file (fdtd.jdl), an SH shell script (fdtd.sh)and the input of the run (fdtd_test.fsp). Since fsp files are usually large, first they must be uploaded to a storage element using lcg-cr (or other ways) from the UI : for example :

lcg-cr -d se01.marie.hellasgrid.gr -l lfn:/grid/SOMEPATH/fdtd_test.fsp  file:`pwd`/fdtd_test.fsp

where SOMEPATH is the path in SE where you copy the file.

The shell script looks like :

###########
#fdtd.sh
###########
#!/bin/sh
FDTDDIR=/opt/exp_soft/common/FDTD
lcg-cp lfn:/grid/SOMEPATH/fdtd_test.fsp file:`pwd`/fdtd_test.fsp
$MPI_MPICH_MPIEXEC $FDTDDIR/bin/FDTD-par -nw -fullinfo -logall fdtd_test.fsp

and the corresponding jdl for submission via a wms for run using 2 processes,

###########
#fdtd.jdl
###########
Type = "job";
JobType = "Mpich";
CPUNumber = 2;
Executable = "fdtd.sh";
StdOutput = "fdtd.out";
StdError = "fdtd.err";

InputSandbox = {"fdtd.sh"};
OutputSandbox = {"fdtd.out","fdtd.err"};


VirtualOrganisation = "na4.vo.eu-egee.org";

requirements =   ( RegExp("cream-ce01.marie.hellasgrid.gr:",other.GlueCEUniqueID)  ) 
                     && ( other.GlueCEStateStatus == "Production" ) 
                     && (Member("MPICH",other.GlueHostApplicationSoftwareRunTimeEnvironment) )
                     && (Member("VO-na4.vo.eu-egee.org-lumerical-FDTD-6.5.9",other.GlueHostApplicationSoftwareRunTimeEnvironment) );

cerequirements = "GlueHostApplicationSoftwareRunTimeEnvironment==\"FDTD\"";

Then issue

glite-wms-job-list-match -a fddt.jdl

This should return :

==========================================================================

           COMPUTING ELEMENT IDs LIST 
 The following CE(s) matching your job requirements have been found:

   *CEId*
 - cream-ce01.marie.hellasgrid.gr:8443/cream-pbs-see
 - cream-ce01.marie.hellasgrid.gr:8443/cream-pbs-see

==========================================================================

and then

glite-wms-job-submit -o jIDs -a fdtd.jdl

NOTE: In the above jdl file there 2 requirements that overlap (RegExp ("cream-ce01.marie.hellasgrid.gr:",other.GlueCEUniqueID) and Member("VO-na4.vo.eu-egee.org-lumerical-FDTD-6.5.9",other.GlueHostApplicationSoftwareRunTimeEnvironment). One has to use only one of these for normal jobs. In this jdl, we use both in order to force a certain CE to be used.

If for any reason the submission via a WMS fails or job never runs, one may use the direct submission to a Cream CE.

glite-ce-job-submit -o jIDs -a fdtd.jdl  -r cream-ce01.marie.hellasgrid.gr:8443/cream-pbs-na4vo

In this case, the shell script is the same as above, but the jdl should be modified for input and output Sandbox :

#################
#fdtd.CE.jdl
#################
Type = "job";
JobType = "Mpich";
CPUNumber = 2;
Executable = "fdtd.sh";
StdOutput = "fdtd.out";
StdError = "fdtd.err";

InputSandbox={
      "gsiftp://se01.marie.hellasgrid.gr/SOMEPATH/fdtd.sh"
             };
OutputSandbox={"fdtd.out","fdtd.err"};
OutputSandboxBaseDestURI="gsiftp://se01.marie.hellasgrid.gr/SOMEPATH";


VirtualOrganisation = "na4.vo.eu-egee.org";

requirements =   ( RegExp("cream-ce01.marie.hellasgrid.gr:",other.GlueCEUniqueID)  ) 
                     && ( other.GlueCEStateStatus == "Production" ) 
                     && (Member("MPICH",other.GlueHostApplicationSoftwareRunTimeEnvironment) )
                     && (Member("VO-na4.vo.eu-egee.org-lumerical-FDTD-6.5.9",other.GlueHostApplicationSoftwareRunTimeEnvironment) );

cerequirements = "GlueHostApplicationSoftwareRunTimeEnvironment==\"FDTD\"";