Proposal of Benchmark Tests

Question of how to approach the benchmarking activity is definitely not the easy one. We are following here a proposal of Yahya Rahmat-Samii from his talk "Benchmarking Full Wave Analysis of Periodic Structures: Non Perpendicularity at Periodic Boundaries" in Special session "Benchmarking at the Frontiers of Computational EM" at IEEE APS/URSI 2017 in San Diego:

  1. Select a real life example. (Characteristic modes.)
  2. Create a simple and canonical version of the example. (Characteristic modes of sphere.)
  3. Try to identify an analytical solution at the reference. (Properly normalized spherical harmonics.)
  4. Compare the software results against the reference. (We do that for all commercial and many in-house tools.)
  5. Identify potential shortcomings. (They are clearly communicated - tracking, numerical dynamics, matrix symmetry, etc.)
  6. Bring this to the attention of community and software vendors. (A paper has been published in IEEE Trans. Antennas and Propagation.)
  7. Make this as an example to be archived for others to test their codes. (This web page serves as proposal and database of all tests, results are stored in FTP.)
  8. Be clear that this is your best effort and take no responsibility. (Done.)

These points nicely summarized overall consensus done there and support other recommendation like

  • Estimate algorithm complexity without notion of accuracy is incorrect.
  • To specify size of problem in terms of unknowns is incorrect.
  • Do not compare data with simulators, but with measurements.

 

Test #1:

Motivation: The fist task aims at quality of the impedance matrix. Impedance matrix is the only one input into the Characteristic mode decomposition, therefore its quality is of great interest. Calculation of impedance matrix is dependent on used numerical precision, field equation (EFIE/CFIE), quadrature rules, singularity treatment and other aspects like testing procedure (Galerkin scheme). There is obvious dependence on quality of mesh grid, used density function and number of discretization elements. Real part of impedance matrix has typically many negative eigenvalues which complicates solution of CM decomposition.

For specification of inputs, ouputs, results and discussion, see here.

Input(s): Calculate impedance matrices for two frequencies specified in FTP:Data/Inputs/frequencies.txt for discretized structure (triangular grid) saved in file FTP:Data/Inputs/Sphere_500tria. The mesh grid is available in NASTRAN format (.nas) or as [p] and [t] matrices in Matlab .mat file.

Output(s): Upload one folder with the impedance matrix (matrices) from your package to the FTP:Upload folder. Name the matrix (matrices) as [P]_zMat_[E]edges_f[X]_q[Y], in which

[P]: prefix denoting your package (e.g.,[P]=AToM),
[E]: number of unknowns, i.e., number of columns or rows (e.g., [E]=750),
[X]: number of frequency sample, i.e., X=1 or X=2,
[Y]: order of quadrature rule.

Example: AToM_zMat_750edges_f1_q4.mat

Ideally, upload Matlab .mat file containing impedance matrix as variable zMat. ASCII files with explicitly specified structure are also acceptable (use same naming convention). Please, do not forget to provide contact informations (just in case).

Optionally, provide us in a text file the information about used field equation (EFIE/CFIE), used basis functions (RWG/nRWG/...), testing procedure (Galerkin?), order of quadrature rules for singular and non-singular terms (1/4/8/n). If you like, provide us also computing time and RAM requirements (it is only fair to provide true values).

 


 

Test #2:

Motivation: The second task is devoted to tracking of the modal data. Notwithstanding the recent advances, the tracking of Characteristic modes is still not completely solved problem. This is true mainly because the frequency behavior of the modes is complicated and one need to take into account various aspects like symmetries of the object, number of analyzed modes, complexity of the body, etc. The procedure is influenced by the limited numerical dynamics, used numerical precision and type of eigendecomposition (generalized Schur decomposition, implicitly restarted Arnoldi method). There are nowadays several (powerful) tracking schemes whose performance should be studied.

For specification of inputs, ouputs, results and discussion, see here.

Input(s): Calculate characteristic modes for the same structure as in Task #1 (FTP:Data/Inputs/Sphere_500tria) at frequencies specified in FTP:Data/Inputs/tracking.txt and track them in your package.

Output(s): Upload two files into your folder in FTP:Upload. The files should be named as [P]_tracking_[E]edges_ka_0p5_5_100m_226f_unsorted and [P]_tracking_[E]edges_ka_0p5_5_100m_226f_sorted, in which

[P]: prefix denoting your package,
[E]: number of unknowns, i.e., number of columns or rows,
_unsorted: indicates that data are unsorted (raw data from decomposition without any tracking applied)
_sorted: indicates that data are tracked

Example: AToM_tracking_750edges_ka_0p5_100m_226f_sorted.mat

Both files should contain exactly two variables of fixed name and size:

ka: list of normalized frequencies (in ka) of size (M×1), M denotes number of samples, $ka = 2 \pi f a / c_0$ in which f is the vector of frequencies, radius a=1 and speed of light c0 = 299792458
eigNum: matrix of size (M×N) containing eigennumbers, M denotes number of samples, N denotes number of founds modes (use maximally)

ASCII files with explicitly specified structure are also acceptable (use same naming convention). Specify type of tracking you use. It Please, do not forget to provide contact informations (just in case).

Optionally, explain special techniques you implemented.

 

Test #3:

(not performed here)

Motivation: The third task measures resemblance between calculated modal currents and (known) currents generating the spherical harmonics. This infromation is extremely important as we would like to know up to which mode the data are well-represented.

For specification of inputs, ouputs, results and discussion, see here.

Input(s): Modal current densities evaluated in a given set of points are needed.

Outputs(s): Projection graph can be depicted for both TE and TM modes.

Note: Since this test requires evaluation of current densities and this is implemented in different packages differently, this test is not performed now and the results are available only for AToM package.

 

Test #4:

(not performed here)

Motivation: The fourth test measures conformity between analytically known characteristic far fields and numerically calculated far fields from characteristic modes. This information is useful check of fundamental CM property of far field orthogonality.

For specification of inputs, ouputs, results and discussion, see here.

Input(s): Modal current densities evaluated in a given set of points are needed.

Outputs(s): Projection graph can be depicted for both TE and TM modes.

 


/Edited 1. 8. 2017, MC/