Use %%time at the start of cell blocks. Particularly useful when working with large datasets, can extrapolate the time for one iteration/batch to calculate the time for entire dataset.
Sets up a progress bar for an iterator, to show progress and estimate time remaining for iterations. Handy to get the feel of progress in a cell block, especially for iterating large datasets. Some good instructions can be found here.
Consider running Jupyter server continuously in the background, either locally or on a virtual machine. Make sure you start the server in the appropriate root folder for your needs. This will mean you won't need to keep a terminal with the server running, and this way it keeps your workspace clean. Also it's worthwhile creating a bash script to quickly launch the server, so you don't need to remember the nohup command.
Here are some steps to enable your virtual environments in Jupyter notebook. You may be able to skip step 1, and go directly to step 2.
STEP 1: in conda env terminal, ipython kernel install —user —name=.venv (or name of your
environment). STEP 2: in conda env, conda install nb_conda_kernels
There are two options to view a jupyter notebook on a remote server. One is while sshing into the server to tunnel as follows:
For further reference about this refer to this link
local_user@local_host$ ssh -N -f -L localhost:8888:localhost:8889 remote_user@remote_host
Alternatively you can use Jupyter Lab to do this and run this command on the server:
jupyter lab --ip 0.0.0.0 --port 45888
Jupyter NoteBook (JNB) is an amazing tool and in particular when it is used for the right purposes in the right manner including for data analysis and debugging. As a loose guideline, JNB is particularly useful when you are prone to errors at an early stage when writing a script or short program. When you are freely experimenting, exploring or don't know what you are doing use JNB. Additionally, JNB's strength is it's documentation capabilities that allow you to write cells with Markdown.
To keep notes on experimentation within the one notebook, it can be useful to mark cells or groups of cells as Experiment 1, Experiment 2 etc. JNB's can be exported as html to be served on websites, or as python files to be deployed elsewhere. It can also be useful when developing a script, with the knowledge that you will be converting it to a python file, you can begin writing it in JNB as if you were preparing it in Pycharm for example, after converting you can keep a copy of the NB in the repo, for future development.
From the command line the basic command to convert using nbconvert is:
jupyter nbconvert --to script my_notebook.ipynbUsing the above method preserves, comments, new lines, and converts markdown into comments, aswell.
Here is a short program that cleanly converts a notebook. You can store it in a file (e.g. jnb_to_python.py) and run it from the command line. Passing the jupyter notebook file as the first argument. Consider adding it to your PATH, for convenient access.
#!/usr/bin/env python3
import sys
import json
import argparse
def main(files):
for file in files:
print('#!/usr/bin/env python')
print('')
code = json.load(open(file))
for cell in code['cells']:
if cell['cell_type'] == 'code':
for line in cell['source']:
if not line.strip().startswith("#") and not line.isspace():
print(line, end='')
print('\n')
if __name__ == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument('file',nargs='+',help='Path to the file')
args_namespace = parser.parse_args()
args = vars(args_namespace)['file']
main(args)After activating a new environment, you need to register the environment with Jupyter. For example with HOML, we do the following.
You just need to register the tf2 conda environment to Jupyter. The notebooks in this project will default to the environment named python3, so it's best to register this environment using the name python3 (if you prefer to use another name, you will have to select it in the "Kernel > Change kernel..." menu in Jupyter every time you open a notebook):
$ python3 -m ipykernel install --user --name=python3