Creating Dandisets and Uploading Data¶
To create a new Dandiset and upload your data, you need to have a DANDI account.
Create a Dandiset and Add Data¶
You can create a new Dandiset at https://dandiarchive.org. This Dandiset can be fully public or embargoed according to NIH policy. When you create a Dandiset, a permanent ID is automatically assigned to it. To prevent the production server from being inundated with test Dandisets, we encourage developers to develop against the development server (https://gui-staging.dandiarchive.org/). Note that the development server should not be used to stage your data. All data are uploaded as draft and can be adjusted before publishing on the production server. The development server is primarily used by users learning to use DANDI or by developers.
The below instructions will alert you to where the commands for interacting with these two different servers differ slightly.
Setup¶
- To create a new Dandiset and upload your data, you need to have a DANDI account. See the Create a DANDI Account page.
- Log in to DANDI and copy your API key. Click on your user initials in the top-right corner after logging in. Production (dandiarchive.org) and staging (gui-staging.dandiarchive.org) servers have different API keys and different logins.
-
Locally:
- Create a Python environment. This is not required, but strongly recommended; e.g. miniconda, virtualenv.
-
Install the DANDI CLI into your Python environment:
pip install -U dandi
-
Store your API key somewhere that the CLI can find it; see "Storing Access Credentials" below.
Data upload/management workflow¶
- Register a Dandiset to generate an identifier. You will be asked to enter
basic metadata: a name (title) and description (abstract) for your dataset.
Click
NEW DANDISET
in the Web application (top right corner) after logging in. After you provide a name and description, the dataset identifier will be created; we will call this<dataset_id>
. -
NWB format:
- Convert your data to NWB 2.1+ in a local folder. Let's call this
<source_folder>
. We suggest beginning the conversion process using only a small amount of data so that common issues may be spotted earlier in the process. This step can be complex depending on your data. NeuroConv automates conversion to NWB from a variety of popular formats. nwb-overview.readthedocs.io points to more tools helpful for working with NWB files, and BIDS converters if you are preparing a BIDS dataset containing NWB files. Feel free to reach out to us for help. -
Check your files for NWB Best Practices by installing the NWBInspector (
pip install -U nwbinspector
) and runningnwbinspector <source_folder> --config dandi
-
Thoroughly read the NWBInspector report and try to address as many issues as possible. DANDI will prevent validation and upload of any issues labeled as level 'CRITICAL' or above when using the
--config dandi
option. See "Validation Levels for NWB Files" for more information about validation criteria for uploading NWB files and which are deemed critical. We recommend regularly running the inspector early in the process to generate the best NWB files possible. Note that some autodetected violations, such ascheck_data_orientation
, may be safely ignored in the event that the data is confirmed to be in the correct form; this can be done using either the--ignore <name_of_check_to_suppress>
flag or a config file. See the NWBInspector CLI documentation for more details and other options, or typenwbinspector --help
. If the report is too large to efficiently navigate in your console, you can save a report usingnwbinspector <source_folder> --config dandi --report-file-path <report_location>.txt
-
Once your files are confirmed to adhere to the Best Practices, perform an official validation of the NWB files by running:
dandi validate --ignore DANDI.NO_DANDISET_FOUND <source_folder>
. If you are having trouble with validation, make sure the conversions were run with the most recent version ofdandi
,PyNWB
andMatNWB
. - Now, prepare and fully validate again within the dandiset folder used for upload:
dandi download https://dandiarchive.org/dandiset/<dataset_id>/draft cd <dataset_id> dandi organize <source_folder> -f dry dandi organize <source_folder> dandi validate . dandi upload
Note that the
organize
steps should not be used if you are preparing a BIDS dataset with the NWB files. Uploading to the development server is controlled via-i
option, e.g.dandi upload -i dandi-staging
. Note that validation is also done duringupload
, but ensuring compliance usingvalidate
prior upload helps avoid interruptions of the lengthier upload process due to validation failures. 6. Add metadata by visiting your Dandiset landing page:https://dandiarchive.org/dandiset/<dataset_id>/draft
and clicking on theMETADATA
link. - Convert your data to NWB 2.1+ in a local folder. Let's call this
If you have an issue using the Python CLI, see the Dandi Debugging section.
Storing Access Credentials¶
There are two options for storing your DANDI access credentials.
-
DANDI_API_KEY
Environment Variable-
By default, the DANDI CLI looks for an API key in the
DANDI_API_KEY
environment variable. To set this on Linux or macOS, run:export DANDI_API_KEY=personal-key-value
-
Note that there are no spaces around the "=".
-
-
keyring
Library-
If the
DANDI_API_KEY
environment variable is not set, the CLI will look up the API key using the keyring library, which supports numerous backends, including the system keyring, an encrypted keyfile, and a plaintext (unencrypted) keyfile. -
Specifying the
keyring
backend- You can set the backend the
keyring
library uses either by setting thePYTHON_KEYRING_BACKEND
environment variable or by filling in thekeyring
library's configuration file. - IDs for the available backends can be listed by running
keyring --list
. - If no backend is specified in this way, the library will use the available backend with the highest priority.
- If the DANDI CLI encounters an error while attempting to fetch the API key
from the default keyring backend, it will fall back to using an encrypted
keyfile (the
keyrings.alt.file.EncryptedKeyring
backend). If the keyfile does not already exist, the CLI will ask you for confirmation; if you answer "yes," thekeyring
configuration file (if it does not already exist; see above) will be configured to useEncryptedKeyring
as the default backend. If you answer "no," the CLI will exit with an error, and you must store the API key somewhere accessible to the CLI on your own.
- You can set the backend the
-
Storing the API key with
keyring
-
You can store your API key where the
keyring
library can find it by using thekeyring
program: Runkeyring set dandi-api-dandi key
and enter the API key when asked for the password forkey
indandi-api-dandi
. -
If the API key isn't stored in either the
DANDI_API_KEY
environment variable or in the keyring, the CLI will prompt you to enter the API key, and then it will store it in the keyring. This may cause you to be prompted further; you may be asked to enter a password to encrypt/decrypt the keyring, or you may be asked by your operating system to confirm whether to give the DANDI CLI access to the keyring.
-
-