We adhere to values of people, technology, collaboration and open science in both proposal selection and evaluation of progress.
Applications will be evaluated for their existing impact, the quality of the open source software project(s) involved, the feasibility of the proposal, the expected value of the funded work to the biomedical community, and their diversity, equity, and inclusion statement. Each of these categories will be assessed through quantitative and qualitative factors. Relevant materials will be provided by the applicants and obtained by The Funders from publicly available sources where possible (e.g., GitHub or other public code repositories).
For those applicants asked to submit a Full Application, additional quantitative metrics on software projects will be requested. Independent expert review will be solicited, and final decisions will be made by CZI, Wellcome and Kavli staff in consultation with our expert advisors.
Impact will assess the importance of the open source software project(s) involved in the proposal to science and the open source ecosystem, in alignment with CZI’s mission to support the science and technology that will make it possible to cure, prevent, or manage all diseases by the end of the century. Reviewers will evaluate:
- Demonstrated scientific impact and adoption of the software project, with a particular focus on its use in biomedicine
- The role of the software project in the scientific open source ecosystem
Alongside qualitative materials, expert evaluation may use metrics such as:
- Number of users and recent growth
- Adoption within relevant communities
- Number of citations or mentions of the software project in scientific literature
- Number of potential contributors and diversity of the organizations they represent
- Number of past contributions to related software projects in the relevant software stack (pushing changes upstream to dependencies, receiving changes from other nearby software projects).
Quality will assess the maturity of the software project(s), its compliance with best practices in open source development, and the existence of a healthy and diverse contributor community. It will again be assessed qualitatively and quantitatively. Reviewers will evaluate:
- Composition and leadership of team
- Governance structure of the software project
- Software project communications and community engagement
- Existence, clarity, and recency of software project roadmap
- Clarity of process for external contributions
- Evidence of external contributions from outside of the core developer team (in the form of code, bugs/issue reports, documentation, etc.)
- Availability of tutorials and examples
- Quality and comprehensiveness of documentation.
Alongside qualitative materials, evaluation may leverage metrics such as:
- Frequency and growth trajectory of commits over time
- Size and make-up of current developer team
- Frequency of external contributions
- Number of open issues, and rate of issues both opened and closed
- Time between opening and closing of pull requests.
Feasibility will assess the plan of work described in the proposal and whether it can be accomplished given the requested budget and key personnel involved. Reviewers will evaluate the following based on qualitative materials:
- Specificity and clarity of plan of work to be accomplished
- Proposed use of funds (relative to plan of work)
- Likelihood of the work being accomplished
- Plan for tracking and validating progress against goals
- Degree of unmet need given existing resources
- Future plans for sustaining or maintaining the work funded by the grant.
In addition to the above criteria, reviewers will evaluate the value to the biomedical community of the proposed work, in particular:
- How the output of the proposal will advance project(s) adoption among biomedical researchers or produce value to their work
- Biomedical user needs that are unmet by the software project(s) in their current state and that will be addressed through the proposal
- Any improvement or integration with other tools that will improve the adoption, usability, functionality, extensibility, ease of use, or performance of the project in the context of the biomedical research community.
Lastly, all applicants invited to submit a Full Application must include in their proposal a Diversity, Equity, and Inclusion (DEI) statement, describing (1) any efforts the software project(s) named in this proposal have undertaken to increase diversity, equity, and inclusion with respect to their contributors and audience; and (2) the results of such efforts, if applicable. This statement will be reviewed alongside the above criteria. Please see examples of DEI statements from successful proposals funded in previous cycles.
There is no expectation of any specific number of awards for this RFA program, however this grant program operates within a budget which will inform the overall number of awards that are recommended for funding. The Funders reserve the right to not recommend the funding of any applications.
REPORTING & PROGRESS
Annual reports, including a summary of the project progress that may be made publicly available, will be required from successful grantees to ensure that progress is on track toward the deliverables described in the proposal. Measures of progress will also include additional indicators on the growth and uptake of the software project, as obtained from code repositories, issue trackers, package distribution systems, community forums, mailing lists, etc.
