Skip to content

Archiving ieeg Files

What this page tells you

Archive a finished HUPXXX: in cnt1 run bash zip_dataset.sh on the cnt-fs folder inside a screen, verify checksums, delete the folder from cnt-fs, move the tarball to ready_for_archiving, upload with upload_azcli.sh (via cntgpu1 or cnt2), confirm it in the cntlitt Azure container, then delete it.

  1. In cnt1, navigate to scripts in /project/eeg_process/azure_archive folder:
    cd /project/eeg_process/azure_archive/scripts

  2. open a new screen in cnt1: screen -S HUPXXX_archive

  3. bash zip_dataset.sh /mnt/cnt-fs/eeg_raw/ieeg_raw/HUPXXX

  4. if you get an error that says "permission denied", run kinit

  5. this step zips up the HUPXXX file into a tarball (zip file)

  6. when the script has finished running (this takes a long time):

  7. check that the 2 top numbers are =

  8. check that 2 bottom numbers are =

  9. delete the HUPXXX folder from cnt-fs

  10. move the HUPXXX tarball HUPXXX.tar.gz file into the logs_archive folder /project/eeg_process/azure_archive/ready_for_archiving

  11. The HUPXXXtarball will be in the /project/eeg_process/azure_archive/scripts, so make sure you are in that file path first before running the below script:

    • mv HUPXXX.tar.gz /project/eeg_process/azure_archive/ready_for_archiving

  12. In a new screen / tab, ssh pmacsusername@scisub9

  13. Enter your pmacs password to log in

  14. bsub -Is -q cntgpu -m cntgpu1 'bash'

  15. You should see this pop up and your user change from [pmacsuser@scisub9 ~]$ to [pmacsuser@cntgpu1 ~]$
    Job <47179607> is submitted to queue .
    <>
    <>

  16. cd /project/eeg_process/azure_archive/scripts
    nohup ./upload_azcli.sh HUPXXX.tar.gz > /project/eeg_process/azure_archive/logs_azure/HUPXXX_azure_archive.log 2>&1 &

  17. You can check the process using ps -ef|grep az

10. Login to http://portal.azure.com to check that the tarball has been uploaded to Brian Litt's account

1. Login using Penn Medicine (UPHS) account
2. Search for the "**cntlitt**" storage account
3. In the dropdown on the left-hand side, go to **"Data Storage" → "Containers"**
4. Open "**cntlitt**" container
5. Look for **HUPXXX.tar.gz**

11. Once you make sure the HUPXXX.tar.gz is in Brian Litt's Azure account, delete it from cnt1

    • Go to the ready_for_archiving folder:
      cd /project/eeg_process/azure_archive/ready_for_archiving
    • rm -r HUPXXX.tar.gz

05/19/2026 - cnt1 crashed and cannot support azure uploads so this step is unusable. If cnt2 is running, try replacing Steps 7-9 above with the below step.

  1. Make sure you are in the /project/eeg_process/azure_archive/scripts folder, then run the code to upload the tarball to Azure:

nohup bash upload_azcli.sh HUPXXX.tar.gz &> /project/eeg_process/azure_archive/logs_azure/HUPXXX_azure_archive.log

  1. Check the log file /project/eeg_process/azure_archive/logs_azure/HUPXXX_azure_archive.log for process on uploading the tarball to Azure. Check that the upload is finished and check that the size of the remote tarball file = size of the local tarball file
    Josh's SOP
    KNOWLEDGE BASE (a page (retired page))