--- title: "Read 'Blackrock' Neural Event & Signals" output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{Read 'Blackrock' Neural Event & Signals} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r, include = FALSE} knitr::opts_chunk$set( collapse = TRUE, eval = FALSE, comment = "#>" ) ``` ## Introduction to `readNSx` `readNSx` is an R package that converts `Blackrock-Microsystems` files (`.nev`, `.nsx`) into commonly used formats that are compatible with `R`, `Python`, `Matlab`. The package is designed for neuroscientist who study intracranial Electroencephalography and use systems like `Blackrock`, `Ripple`, etc. Support versions are `2.2`, `2.3`, `3.0`. To install `readNSx` from `CRAN`, ```r install.packages("readNSx") ``` To install development version ```r install.packages("readNSx", repos = c( dipterix = "https://dipterix.r-universe.dev", CRAN = "https://cloud.r-project.org")) ``` ## Basic Usage *(Well, this package has no advanced usage.)* ### One-time import for the first time If you have never converted data, use the following one-time function call to import `.nev` and `.nsx` data: ```r readNSx::import_nsp( path = "YAB_Datafile_001.nev", prefix = "YAB_Converted_001", exclude_events = "spike", partition_prefix = "/part" ) ``` * `path`: path to `.nev` or any `nsx` file (such as `ns3, ns5`) * `prefix`: path prefix of imported files (where the data will be saved) * `exclude_events`: events to exclude from importing; default is "spike" (spike-sorting waveform and classes) * `partition_prefix`: see Section "Anatomy of imported files". Please check help function `?readNSx::import_nsp` ### Load imported data from R *(If you have never imported data before, please check previous sub-section first)* #### Example 1: load epoch from `NEV` comment data packet events The following example loads the trial epoch table from `NEV` comment events. The `time_in_seconds` is the second of stimuli relative to `time_origin`. Column `comment` is whatever comments sent by the task script (sent from `psychtoolbox` or related software that you use when collecting data). ```r # prefix <- "YAB_Converted_001" nev <- readNSx::get_nev(prefix) # `time_in_seconds` is relative to time-origin nev$header_basic$time_origin #> Year Month DayofWeek Day Hour #> 2022 8 5 26 15 #> Minute Second Millisecond #> 4 10 156 readNSx::get_event(nev, "comment") #> timestamp packet_id char_set flag data comment event time_in_seconds #> 1 683033 65535 0 0 255 audio-ba comment 22.76777 #> 2 753242 65535 0 0 255 video-ba comment 25.10807 #> ... ``` #### Example 2: get channel information and data ```r # prefix <- "YAB_Converted_001" # Gather information of channel 10 loaded <- readNSx::get_channel(prefix, channel_id = 10) # Get NSx configurations, loaded$nsx #> Basic header information (NSx): #> Internal type: NEURALCD #> Channel count: 152 #> Sample rate: 2000 Hz #> Time origin: 2022-08-26 15:04:10 158ms #> Extended header information (NSx): #> - CC (152 x 16, channels: 1-152): type, electrode_id, electrode_label, ... #> Cache status: #> Prefix: ... #> Number of partitions: 1 # E.g. number of partitions (i.e. unpaused continuous recordings) loaded$nsx$nparts #> 1 # Get channel information loaded$channel_info #> type electrode_id electrode_label physical_connector connector_pin #> 10 CC 10 RA10-010 1 10 #> min_digital_value max_digital_value min_analog_value #> 10 -32764 32764 -8191 #> max_analog_value units high_freq_corner high_freq_order #> 10 8191 uV 300 1 #> high_freq_type low_freq_corner low_freq_order low_freq_type #> 10 1 7500000 3 1 #> sample_rate_signal sample_rate_timestamp which_nsp filename #> 10 2000 30000 3 RA10-010.h5 # Get channel data channel_signal <- loaded$channel_detail$part1$data; channel_signal #> Class: H5D #> Dataset: /data #> Filename: ... #> Access type: H5F_ACC_RDONLY #> Datatype: H5T_IEEE_F64LE #> Space: Type=Simple Dims=798264 Maxdims=Inf #> Chunk: 16384 # Get the actual numbers channel_signal[] ``` Please use square bracket `channel_signal[]` to load the data into the memory. ## Anatomy of imported files The imported file paths will start with `prefix`, which is specified by you, my dear users. In the following demonstration, I'll use a placeholder `` to represent your inputs. ``` partition_info - Name of continuous recording within the block, sample rates, starting time per partition per NSx _scans - Basic information for current block _channels - Electrode channel information ( ID, Label, ... ) _events/ - NEV setting headers and data packets (events) - DIGLABEL - Digital input setup - NEUEVLBL - Channel labels - NEUEVWAVE - Spike waveform settings - ... - Other settings - event-*** - Data packets (digital inputs, comments...) - waveforms.h5 - Spike waveforms & cluster _ieeg - NSx data folder - configurations.rds - NSx basic headers (versions, number of partitions, ...) - partition_info - Continuous recording duration, start time, sample rates - nsx_summary.rds - Internally used - part1/, part2/, ...- Channel folder - XXXX-001.h5 - Channel data file, each file correspond to a channel. - XXXX-002.h5 The file name ALWAYS ends with channel ID. - ... Each HDF5 file contains a "meta" and a "data" part, "meta": JSON string of channel information "data": numerical signal voltage data (in `uV`) ``` The signal data is stored at `_ieeg/part*` by default, where `*` are positive integers representing the partition number. The `Blackrock` system allows users to pause their recordings and resume later without having to start a new block. A partition in `readNSx` represents one continuous recording within a "block". In most of cases, when there is no pause within a block, you will see only one partition. In some experimental settings, there could be one or more pauses, `readNSx` will store each continuous recordings in separate folders to make sure each partition is always continuous. The start time of the partitions will be stored in `partition_info.tsv`. Please be aware that partition pattern `_ieeg/part*` is not fixed. You can change the pattern via parameter `partition_prefix` when importing the data. For example, `partition_prefix="_part` will create partition files within directory `_ieeg_part*`. When `readNSx` stores the data, the channels are saved individually. For example, channel `1 (LA-2)` and `2 (LA-2)` are stored in separate `HDF5` files. This arrangement is out of the file-size and computational considerations: * When storing all channels together, the file size will become super big. Reading one channel might resulting in reading the whole file (if handled poorly, or if using network drive) * In some cases, `HDF5` files can only have one file pointers (file locked by some software). This could limit batch computing algorithms that can be paralleled at channel-level * My personal research requires fast approach to copy or share data for quick inspection/analysis/visualizations (don't want to move large files around) ### File formats The following file types will be generated: * `.h5` (`HDF5` file): common file format that can include one or more data within a single file. - `Matlab`: use `h5disp` to get enclosing data names; use `h5read` to read specific data - `Python`: use `h5py` package to load the files - `R`: if you want to load them individually, use `raveio::h5_names` to get enclosing data names; use `raveio::load_h5` to read specific data. `readNSx` also provides high-level functions to load them (see `?readNSx::get_channel`). * `.tsv` (tab-separated values file): plain text files that can be easily read by many languages. You can open them in `Microsoft Office: Excel`. * `.rds` (R object file): can only be read from R, internally used by `readNSx` to store data objects. Users do not need to read from these files (but also do NOT delete them, or `readNSx` will break). They serve as redundant files in case the `tsv` files are altered accidentally (for example, `Excel` might alter data formats automatically). If you see an `.rds` file sharing the same name as `.tsv`, they share the same information.