Question of how to approach the benchmarking activity is definitely not the easy one. We are following here a proposal of Yahya Rahmat-Samii from his talk “Benchmarking Full Wave Analysis of Periodic Structures: Non Perpendicularity at Periodic Boundaries” in Special session “Benchmarking at the Frontiers of Computational EM” at IEEE APS/URSI 2017 in San Diego:
- Select a real life example. (Characteristic modes.)
- Create a simple and canonical version of the example. (Characteristic modes of sphere.)
- Try to identify an analytical solution at the reference. (Properly normalized spherical harmonics.)
- Compare the software results against the reference. (We do that for all commercial and many in-house tools.)
- Identify potential shortcomings. (They are clearly communicated – tracking, numerical dynamics, matrix symmetry, etc.)
- Bring this to the attention of community and software vendors. (A paper has been published in IEEE Trans. Antennas and Propagation.)
- Make this as an example to be archived for others to test their codes. (This web page serves as proposal and database of all tests, results are stored in FTP.)
- Be clear that this is your best effort and take no responsibility. (Done.)
These points nicely summarized overall consensus done there and support other recommendation like
- Estimate algorithm complexity without notion of accuracy is incorrect.
- To specify size of problem in terms of unknowns is incorrect.
- Do not compare data with simulators, but with measurements.
Test #1:
Motivation: The fist task aims at quality of the impedance matrix. Impedance matrix is the only one input into the Characteristic mode decomposition, therefore its quality is of great interest. Calculation of impedance matrix is dependent on used numerical precision, field equation (EFIE/CFIE), quadrature rules, singularity treatment and other aspects like testing procedure (Galerkin scheme). There is obvious dependence on quality of mesh grid, used density function and number of discretization elements. Real part of impedance matrix has typically many negative eigenvalues which complicates solution of CM decomposition.
For specification of inputs, ouputs, results and discussion, see here.
Input(s): Calculate impedance matrices for two frequencies specified in FTP:Data/Inputs/frequencies.txt for discretized structure (triangular grid) saved in file FTP:Data/Inputs/Sphere_500tria. The mesh grid is available in NASTRAN format (.nas) or as [p] and [t] matrices in Matlab .mat file.
Output(s): Upload one folder with the impedance matrix (matrices) from your package to the FTP:Upload folder. Name the matrix (matrices) as [P]_zMat_[E]edges_f[X]_q[Y], in which
Example: AToM_zMat_750edges_f1_q4.mat
Ideally, upload Matlab .mat file containing impedance matrix as variable zMat. ASCII files with explicitly specified structure are also acceptable (use same naming convention). Please, do not forget to provide contact informations (just in case).
Optionally, provide us in a text file the information about used field equation (EFIE/CFIE), used basis functions (RWG/nRWG/…), testing procedure (Galerkin?), order of quadrature rules for singular and non-singular terms (1/4/8/n). If you like, provide us also computing time and RAM requirements (it is only fair to provide true values).
Test #2:
Motivation: The second task is devoted to tracking of the modal data. Notwithstanding the recent advances, the tracking of Characteristic modes is still not completely solved problem. This is true mainly because the frequency behavior of the modes is complicated and one need to take into account various aspects like symmetries of the object, number of analyzed modes, complexity of the body, etc. The procedure is influenced by the limited numerical dynamics, used numerical precision and type of eigendecomposition (generalized Schur decomposition, implicitly restarted Arnoldi method). There are nowadays several (powerful) tracking schemes whose performance should be studied.
For specification of inputs, ouputs, results and discussion, see here.
Input(s): Calculate characteristic modes for the same structure as in Task #1 (FTP:Data/Inputs/Sphere_500tria) at frequencies specified in FTP:Data/Inputs/tracking.txt and track them in your package.
Output(s): Upload two files into your folder in FTP:Upload. The files should be named as [P]_tracking_[E]edges_ka_0p5_5_100m_226f_unsorted and [P]_tracking_[E]edges_ka_0p5_5_100m_226f_sorted, in which
Example: AToM_tracking_750edges_ka_0p5_100m_226f_sorted.mat
Both files should contain exactly two variables of fixed name and size:
ASCII files with explicitly specified structure are also acceptable (use same naming convention). Specify type of tracking you use. It Please, do not forget to provide contact informations (just in case).
Optionally, explain special techniques you implemented.
Test #3:
(not performed here)
Motivation: The third task measures resemblance between calculated modal currents and (known) currents generating the spherical harmonics. This infromation is extremely important as we would like to know up to which mode the data are well-represented.
For specification of inputs, ouputs, results and discussion, see here.
Input(s): Modal current densities evaluated in a given set of points are needed.
Outputs(s): Projection graph can be depicted for both TE and TM modes.
Note: Since this test requires evaluation of current densities and this is implemented in different packages differently, this test is not performed now and the results are available only for AToM package.
Test #4:
(not performed here)
Motivation: The fourth test measures conformity between analytically known characteristic far fields and numerically calculated far fields from characteristic modes. This information is useful check of fundamental CM property of far field orthogonality.
For specification of inputs, ouputs, results and discussion, see here.
Input(s): Modal current densities evaluated in a given set of points are needed.
Outputs(s): Projection graph can be depicted for both TE and TM modes.
/Edited 1. 8. 2017, MC/
