Startup
Edit this to point at the path to where you installed the neuropixel-utils distribution or repo.
% Load the Neuropixel Utils library
homeDir = string(java.lang.System.getProperty('user.home'));
npixUtilsDistRoot = fullfile(homeDir, 'local', 'repos', 'npxutils-apj-wip-01');
addpath(fullfile(npixUtilsDistRoot, 'Mcode'));
fprintf('Loaded Neuropixel Utils %s\n', npxutils.globals.version)
Loaded Neuropixel Utils 0.5.0-SNAPSHOT
Set the default channel map file you want to use with your data. Some standard channel maps are included in the map_files directory in the neuropixel-utils distribution.
setenv('NEUROPIXEL_MAP_FILE', fullfile(npxutils.globals.distroot, 'map_files', 'neuropixPhase3A_kilosortChanMap.mat'))
Optional: Configure Neuropixel Utils to generate paths beneath your data root directory, based on its naming conventions.
This is entirely optional; you can arrange your data however you'd like.
myExampleDataDir = fullfile(homeDir, 'work', 'npxutils', 'example-data');
setenv('NEUROPIXEL_DATAROOT', myExampleDataDir)
Load Raw Imec dataset
Here we construct the path to the original, raw dataset as it was recorded by SpikeGLX. You don't need to use this folder nesting structure.
rawBinFile = 'Vinnie_20180817_All.imec.ap.bin';
imecFile = npxutils.generatePath(subject, 'raw_datasets', dateStr, rawBinFile)
imecFile = '/home/janke/work/npxutils/example-data/Vinnie/raw_datasets/2018-08-17/Vinnie_20180817_All.imec.ap.bin'
We then use this path to construct an ImecDataset object which will facilitate access to the raw data and load the metadata from disk. You can pass the path of the Imec dataset in several ways:
- Path to raw .imec.ap.bin file or imec.lf.bin file.
- Path to the containing directory, if there is only one set of .bin files within.
To construct the ImecDataset object, pass the identifying path directly to the constructor.
imec = npxutils.ImecDataset(imecFile)
Specifying the channel map manually
By default, the channel map will be loaded from whatever the environment variable NEUROPIXEL_MAP_FILE is set to, which we set above. If you want to manaully specify a channel map, pass it in to the ImecDataset constructor using the channelMap option:
imec = npxutils.ImecDataset(imecFile, 'channelMap', '/path/to/channelMap.mat');
Explore the loaded metadata
Here we see that this option 3A probe has 384 total channels (saved in the .imec.ap.bin file), and that 374 of these channels are connected, namely those listed in connectedChannels.
imec.channelMap
ans =
ChannelMap with properties:
file: "/home/janke/local/repos/npxutils-apj-wip-01/map_files/neuropixPhase3A_kilosortChanMap.mat"
name: "neuropixPhase3A_kilosortChanMap"
channelIdsMapped: [384×1 uint32]
connected: [384×1 logical]
shankInd: [384×1 double]
nSpatialDims: 2
xcoords: [384×1 double]
ycoords: [384×1 double]
zcoords: [384×1 double]
syncChannelIndex: 385
syncChannelId: 385
coords: [384×2 double]
syncInAPFile: 1
syncInLFFile: 1
channelIds: [385×1 uint32]
nChannels: 385
nChannelsMapped: 384
connectedChannels: [374×1 uint32]
referenceChannels: [10×1 uint32]
invertChannelsY: 1
yspacing: 20
xspacing: 16
zspacing: [0×1 double]
xlim: [-9 79]
ylim: [0 3860]
We can determine the duration of the recording using the number of samples and the frequency in the Imec data:
durationMinutes = imec.nSamplesAP / imec.fsAP / 60;
fprintf('Duration of recording %s is %.1f minutes.\n', imec.fileStem, durationMinutes)
Duration of recording Vinnie_20180817_All is 20.1 minutes.
Accessing the raw AP data
You can access the raw AP data in several ways. (See the full list of methods with methods(imec) or doc npxutils.ImecDataset). The most straightforward is to memory-map the full binary data:
This bit is broken because it references code that doesn't exist in this library. Probably left over from an older version of the library and never updated? Or maybe it has an external dependency?
% This is broken because NeuropixelExpt doesn't exist?
% NeuropixelExpt.DataLoad.setImecSyncBitNames(imec);
% What does "p" refer to here?
% if ~isempty(p.Results.badChannels)
% imec.markBadChannels(p.Results.badChannels);
This stuff works:
rmsBadChannels = imec.markBadChannelsByRMS('rmsRange', [3 100]);
fprintf('Marked %d channels bad based on RMS\n', numel(rmsBadChannels));
Marked 0 channels bad based on RMS
% Save the bad channels and sync bit names back to the meta file
imec.writeModifiedAPMeta();
% CAR the file and flush the unused sync bits to the cleaned_datasets folder
cleanedBinFile = 'Vinnie_20180817_All_cleaned.imec.ap.bin'
cleanedBinFile = 'Vinnie_20180817_All_cleaned.imec.ap.bin'
cleanedPath = npxutils.generatePath(subject, 'cleaned_datasets', dateStr, cleanedBinFile);
fprintf('Writing CAR version at %s\n', cleanedPath);
Writing CAR version at /home/janke/work/npxutils/example-data/Vinnie/cleaned_datasets/2018-08-17/Vinnie_20180817_All_cleaned.imec.ap.bin
extraMeta.run_clearUnusedSyncBits = true;
extraMeta.run_detectAndMarkStimArtifactWindows = true;
% This is broken because NeuropixelExpt doesn't exist?
% @NeuropixelExpt.DataClean.clearUnusedSyncBits
@npxutils.dataprocess.commonAverageReference
cleanedDataDir = fullfile(myExampleDataDir, 'Vinnie', 'cleaned_datasets');
npxutils.io.rmrf(cleanedDataDir);
imec = imec.saveTransformedDataset(cleanedPath, ...
'goodChannelsOnly', false, 'writeSyncSeparate', false, ...
'transformAP', fnList, 'extraMeta', extraMeta);
Writing AP meta file Vinnie_20180817_All_cleaned.imec.ap.meta
Writing AP bin file /home/janke/work/npxutils/example-data/Vinnie/cleaned_datasets/2018-08-17/Vinnie_20180817_All_cleaned.imec.ap.bin
Writing contents of Vinnie_20180817_All
Copying ap file 1 / 1: Vinnie_20180817_All : 100% [_______________]
Warning: AP bin file size is not an integral number of samples, file data may not be fully copied, truncating nSamplesAP
Error using npxutils.ConcatenationInfo (line 56)
Mismatch between time shifts and total number of samples
Error in npxutils.ImecDataset/readInfo (line 313)
this.concatenationInfoAP = npxutils.ConcatenationInfo(this, 'ap', metaAP);
Error in npxutils.ImecDataset (line 193)
this.readInfo();
Error in npxutils.ImecDataset.writeConcatenatedFileMatchGains (line 2763)
imecOut = npxutils.ImecDataset(outFile, 'channelMap', imecList{1}.channelMapFile, 'sourceDatasets', cat(1, imecList{:}));
Error in npxutils.ImecDataset/saveTransformedDataset (line 1943)
imecOut = npxutils.ImecDataset.writeConcatenatedFileMatchGains({this}, outPath, p.Results);
% Symlink into ks directory for Kilosort's use
ksPath = npxutils.generatePath(subject, 'ks', dateStr, cleanedBinFile);
fprintf('Symlinking to %s\n', ksPath);
% And run Kilosort on the symlinked data set
imec = imec.symLinkAPIntoDirectory(ksPath);
fprintf('Running Kilosort2\n');
npxutils.runKilosort2(imec);
% Here's a little summary output
info.cleanedPath = cleanedPath;