Use %%time at the start of cell blocks. Particularly useful when working with large datasets, can extrapolate the time for one iteration/batch to calculate the time for entire dataset.
Sets up a progress bar for an iterator, to show progress and estimate time remaining for iterations. Handy to get the feel of progress in a cell block, especially for iterating large datasets. Some good instructions can be found here.
Consider running Jupyter server continuously in the background, either locally or on a virtual machine. Make sure you start the server in the appropriate root folder for your needs. This will mean you won't need to keep a terminal with the server running, and this way it keeps your workspace clean. Also it's worthwhile creating a bash script to quickly launch the server, so you don't need to remember the nohup command.
Here are some steps to enable your virtual environments in Jupyter notebook. You may be able to skip step 1, and go directly to step 2.
STEP 1: in conda env terminal, ipython kernel install —user —name=.venv (or name of your
environment). STEP 2: in conda env, conda install nb_conda_kernels
There are two options to view a jupyter notebook on a remote server. One is while sshing into the server to tunnel as follows:
For further reference about this refer to this link
local_user@local_host$ ssh -N -f -L localhost:8888:localhost:8889 remote_user@remote_host
Alternatively you can use Jupyter Lab to do this and run this command on the server:
jupyter lab --ip 0.0.0.0 --port 45888
Jupyter NoteBook (JNB) is an amazing tool and in particular when it is used for the right purposes in the right manner including for data analysis and debugging. As a loose guideline, JNB is particularly useful when you are prone to errors at an early stage when writing a script or short program. When you are freely experimenting, exploring or don't know what you are doing use JNB. Additionally, JNB's strength is it's documentation capabilities that allow you to write cells with Markdown.
To keep notes on experimentation within the one notebook, it can be useful to mark cells or groups of cells as Experiment 1, Experiment 2 etc. JNB's can be exported as html to be served on websites, or as python files to be deployed elsewhere. It can also be useful when developing a script, with the knowledge that you will be converting it to a python file, you can begin writing it in JNB as if you were preparing it in Pycharm for example, after converting you can keep a copy of the NB in the repo, for future development.