石山 智明

As an entry for the 2012 Gordon-Bell performance prize, we report performance results of astrophysical N-body simulations of one trillion particles performed on the full system of K computer. This is the first gravitational trillion-body simulation in the world. We describe the scientific motivation, the numerical algorithm, the parallelization strategy, and the performance analysis. Unlike many previous Gordon-Bell prize winners that used the tree algorithm for astrophysical N-body simulations, we used the hybrid TreePM method, for similar level of accuracy in which the short-range force is calculated by the tree algorithm, and the long-range force is solved by the particle-mesh algorithm. We developed a highly-tuned gravity kernel for short-range forces, and a novel communication algorithm for long-range forces. The average performance on 24576 and 82944 nodes of K computer are 1.53 and 4.45 Pflops, which correspond to 49% and 42% of the peak speed.

We report on the performance of our cold dark matter cosmological N-body simulation that was carried out concurrently using supercomputers across the globe. We ran simulations on 60-750 cores distributed over a variety of supercomputers in Amsterdam (The Netherlands, Europe), in Tokyo (Japan, Asia), Edinburgh (UK, Europe) and Espoo (Finland, Europe). Regardless of the network latency of 0.32 s and the communication over 30 000 km of optical network cable, we are able to achieve ∼87% of the performance compared to an equal number of cores on a single supercomputer. We argue that using widely distributed supercomputers in order to acquire more compute power is technically feasible and that the largest obstacle is introduced by local scheduling and reservation policies. © 2011 IOP Publishing Ltd.

The computational requirements of simulating a sector of the universe led an international team of researchers to try concurrent processing on two supercomputers half a world apart. Data traveled nearly 27,000 km in 0.277 second, crisscrossing two oceans to go from Amsterdam to Tokyo and back.

In this paper, we describe the implementation and performance of GreeM, a massively parallel TreePM code for large-scale cosmological N-body simulations. GreeM uses a recursive multi-section algorithm for domain decomposition. The size of the domains are adjusted so that the total calculation time of the force becomes the same for all processes. The loss of performance due to non-optimal load balancing is around 4%, even for more than 10(3) CPU cores. GreeM runs efficiently on PC clusters and massively-parallel computers, such as a Cray XT4. The measured calculation speed on Cray XT4 is 5 x 10(4) particles per second per CPU core, for the case of an opening angle of theta = 0.5, if the number of particles per CPU core is larger than 10(6).

We analyzed the statistics of subhalo abundance of galaxy-sized and giant-galaxy-sized halos formed in a high-resolution cosmological simulation of a 46.5 Mpc cube with the uniform mass resolution of 10(6) M(circle dot). We analyzed all halos with mass more than 1.5 x 10(12) M(circle dot) formed in this simulation box. The total number of halos was 125. We found that the subhalo abundance, measured by the number of subhalos with maximum rotation velocity larger than 10% of that of the parent halo, shows large halo-to-halo variations. The results of recent ultra-high-resolution runs fall within the variation of our samples. We found that the concentration parameter and the radius at the moment of the maximum expansion show fairly tight correlation with the subhalo abundance. This correlation suggests that the variation of the subhalo abundance is at least partly due to the difference in the formation history. Halos formed earlier have a smaller number of subhalos at present.

Recent high-resolution simulations of the formation of dark-matter halos have shown that the distribution of subhalos is scale-free, in the sense that if scaled by the velocity dispersion of the parent halo, the subhalo velocity distribution functions of galaxy-sized and cluster-sized halos are identical. For cluster-sized halos, the simulation results agreed well with the observations. The simulations, however, predicted far too many subhalos for galaxy-sized halos. Our galaxy has several tens of known dwarf galaxies. On the other hand, simulated dark-matter halos contain thousands of subhalos. We performed a simulation of a single large volume, and measured the abundance of subhalos in all massive halos. We found that the variation of the subhalo abundance is very large, and those with the largest number of subhalos correspond to the simulated halos in previous studies. The subhalo abundance depends strongly on the local density of the background. Halos in high-density regions contain a large number of subhalos. Our Galaxy is in the low-density region. For our simulated halos in low-density regions, the number of subhalos is within a factor of four to that of our Galaxy. We argue that the "missing dwarf problem" is not a real problem, but is caused by biased selections of the initial conditions in previous studies, which were not appropriate for field galaxies.