Calibrated Peer Review (CPR), an online program that purportedly helps students develop as writers and critical thinkers, is being increasingly used by science educators. CPR is an enticing tool since it does not require instructors to grade student writing, and instructors can adopt assignments directly from a library. To determine the extent to which CPR is being used to foster the development of critical thinking and writing skills, the authors analyzed a sample from the CPR assignment library. They found that approximately two-thirds of assignments were designed to engage students in some type of critical thinking, but less than one-third were designed to promote higher-order writing skills. They recommend that the current library be used with caution, a CPR user manual be written (with detailed instructions for creating high-quality writing assignments), and that a new CPR library be created to include only assignments that have been reviewed by experts in critical thinking or writing pedagogy.