To ensure a fair and efficient review of the code, the following guidelines must be followed when you prepare your submission. If there are technical problems that prevent you from complying with the requirements, please contact the Chair of the Replicability Committee.
The software must compile and run on at least one of the following operating systems:
Note that you can pick whichever operating system you developed your code on.
Your code does not need to compile on all of them to obtain the stamp.
The code must compile and run on a vanilla installation, without any
software installed. If you need specific software to be installed, you should
provide a script that automatically downloads and installs all the required
dependencies. In the case of Windows, a text file with precise installation
instructions will also be accepted.
The software can be written in any programming language, as long as the
compiler/interpreter is free to use for academic purposes. One exception is
MATLAB, which will be accepted (version 2016b or newer) since it is popular in
the graphics community. If you choose to use MATLAB, you may use any standard or custom toolbox and you may
have mex functions, as long as they satisfy the license requirements described
Detailed instructions to install the compilers/interpreter and the
dependencies should be available in the readme file.
- Windows 10 or newer
- Ubuntu 16.04.1 LTS or newer
- MacOSX 10.12 or newer
2. Replicating Results
A script should be provided to reproduce one representative figure or table in the paper
that contains a result generated with the proposed algorithm. The script should run
without parameters and generate the data that is shown in the
figure or listed in the table. If the produced data is not in a standard
format, please describe the format in a text file attached to the submission.
The script may either directly render the image or generate the data that has
been used to produce the image. The review process will only ensure that the
code runs and is able to reproduce the material for the figure. The quality
and robustness of the code will not be evaluated and they will have no impact
on the review process.
3. Interactive Applications
If your application is interactive and
it is not possible to script it, you must provide a short movie clip that captures the screen while you use your software to reproduce the representative figure. The reviewers will then follow step by step the video to reproduce the same result. If you want to use this option, you will have to provide a short description to justify why it is not possible to script the generation of the results.
4. Data Restrictions
There might be cases where the data used in a paper cannot be publicly released with the same license as the code. For example, it is common to have additional clauses that a user needs to accept before being allowed to download the data. In these cases, it is sufficient to provide a url that points to a website that allows users to apply for (or buy) the license needed to obtain the data. A confidential copy of the data (which will be deleted after the review) should be sent to the reviewer.
5. Dataset Papers
The main contribution of a dataset paper is the acquisition or creation of a large dataset, which is useful for the community. In these cases, the stamp will be awarded if the authors released their dataset freely for non-commercial usage — it is not mandatory (but still encouraged) for the authors to release the code to replicate (and/or regenerate if possible and appropriate) the entire dataset or the figures in the accompanying paper.
6. User Studies
For some papers the main contribution is reporting about data/measure collections created out of user studies. These papers typically reach useful conclusions based on a statistical analysis over such data. In these cases, the stamp will be awarded if the authors released the data collected freely for non-commercial usage, along with a script to process this data and replicate one of the figures/tables shown in the paper. Note that if the paper also includes an algorithmic contribution, then the code of such an algorithm is required too.
7. Long Running Times
The authors should provide precomputed data for each result/algorithmic step requiring more than 24 hours. The code and training data required to reproduce the result/algorithmic step should still be in the repository, but the reviewer will not rerun the computation to save time and resources. This applies in particular to training of deep networks or expensive parameter searches: for these cases, providing the weights of the network or the parameters will allow the reviewer to validate the results, without having to recompute them. While highly discouraged, it is possible to opt out of releasing the data if there are copyright problems as explained in (4).
8. Hardware Requirements
We encourage the authors to provide code
that runs without requiring specific hardware, to favor portability and
simplify the replicability of the results. However, if this is not possible, we
will accept submissions that run on specific hardware as long as this is
justified in the submission. The following non-standard hardware is currently
allowed: NVIDIA Tesla, NVIDIA GTX 1080, NVIDIA TITAN BLACK, and AMD Firepro
D700. If the necessary hardware is not included on this list, please contact the Replicability Committee Chair.
9. Custom Hardware
Submissions containing a custom hardware component (such as a new custom display or VR technology) can also apply for the stamp, as long as the submission also contains a software contribution. The replicability stamp review process will validate only the software part. In these cases, the authors should provide two additional archives in the code repository: (1) the complete hardware blueprints (which must contain sufficient information to replicate the hardware), and (2) the raw data acquired with the system and that it is used to create the figures/tables in the submission.
The code must be distributed with a license that allows free non-commercial usage. Its
dependencies must have a license that allows free non-commercial or academic usage. Note that
this does not preclude charging your users for commercial usage (the CGAL
project is a successful example of this code distribution model).
For example, you can have Mosek or Gurobi as a dependency, since they are free for academic
purposes, while KNitro cannot be used since they do not provide a free non-commercial or academic license. The only exception to this rule is the use of MATLAB, due to its popularity
in our community. If your project cannot satisfy this rule, please contact
the chair to discuss the special case.
The previous rules are general guidelines: If your
software cannot satisfy one of the previous requirements, please write as soon
as possible to the Replicability Committee Chair to explain the reasons. In many cases, it will be
possible to lift any of the previous requirements if there is a good reason for
12. Submission Procedure
The submission for the stamp takes place only after the paper is accepted at a journal or conference participating in the GRSI. The review process is not blind: the authors and reviewers are aware of each other's identities. A single reviewer will be assigned to
each submission and will work with the authors to ensure that the submission is improved until it satisfies all the guidelines.
The authors must follow the specific instructions published on the website of the corresponding journal.
The repository should contain:
- A txt file containing the title of the submission, the authors, and the
- A script that automatically downloads all required dependencies and
compiles the code. If this is not possible (for example if MATLAB needs
to be installed), then a text file with precise instructions
should be provided. The instructions should describe all the
software dependencies that need to be installed, together with
detailed compilation instructions. Please always specify
entire paths: for example, do not write "change the cmake
variable sx to point to the directory where solver x is
installed" but instead tell the user to install the solver x in
"c:/solverx" and always use absolute paths (assuming a vanilla
installation of the operating system).
- The source code.
- A script for one representative figure or table created with the proposed algorithm,
which runs the provided implementation and generates the data used to create
the figure. In case of a comparison figure, the script only needs to reproduce
the parts created with the proposed algorithm. Note that the script does not
necessarily need to create a figure: if the figure shows a triangle mesh, it is
sufficient to export the triangle mesh into a text file.
- A liability form that gives permission to the reproducibility committee
and reviewers to review the code and advertise the review publicly after the
stamp is approved.