High concentration photovoltaic devices require effective heat rejection to keep the solar cells within a suitable temperature range and to achieve acceptable system efficiencies. Various techniques have been developed to achieve these goals. For example, nanofluids as coolants have remarkable heat transfer characteristics with broad applications; but, little is known of its performance for concentration photovoltaic cooling. Generally, a cooling system should be designed to keep the system within a tolerable temperature range, to minimize energy waste, and to maximize system efficiency. In this paper, the thermal performance of an Al2O3-water cooling system for densely packed photovoltaic cells under high concentration has been computationally investigated. The model features a representative 2D cooling channel with photovoltaic cells, subject to heat conduction and turbulent nanofluid convection. Considering a semi-empirical nanofluid model for the thermal conductivity, the influence of different system design and operational parameters, including required pumping power, on cooling performance and improved system efficiency has been evaluated. Specifically, the varied system parameters include the nanoparticle volume fraction, the inlet Reynolds number, the inlet nanofluid temperature, and different channel heights. Optimal parameter values were found based on minimizing the system's entropy generation. Considering a typical 200-sun concentration, the best performance can be achieved with a channel of 10 mm height and an inlet Reynolds number of around 30,000, yielding a modest system efficiency of 20%. However, higher nanoparticle volume fractions and lower nanofluid inlet temperatures further improve the cell efficiency. For a more complete solar energy use, a combined concentration photovoltaic and thermal heating system are suggested.