In this study we investigate the quality of the selection process of an open access (OA) journal, taking as an example the journal Atmospheric Chemistry and Physics (ACP). ACP is working with a new system of public peer review. We examined the predictive validity of the ACP peer review system - namely, whether the process selects the best of the manuscripts submitted. We have data for 1111 manuscripts that went through the complete ACP selection process in the years 2001 to 2006. The predictive validity was investigated on the basis of citation counts for the later published manuscripts. The results of the citation analysis confirm the predictive validity of the editorial decisions at ACP: They covary with citation counts for the published manuscripts.