Using pipenv in Jupyter Notebook

Ipykernel is the Python execution backend for Jupyter. Jupyter Notebook automatically ensures that Ipykernel is available but if I want to use a kernel in a virtual environment, I need to install it manually. (Read)

1. Install ipykernel in the project folder where my pipfile is located.

pipenv install ipykernel

2. Start the virtual environment.

pipenv shell

This will launch the virtual environment of the project.

(my-virtualenv-name) $

3. Install Python kernel with a name

python3 -m ipykernel install --user --name=my-virtualenv-name

In my case, I used ml-basics for my virtual env name.

You can now see the kernel name ml-basics in the kernel option.

Source: SlackOverflow

 

conda command not found after installing oh-my-zsh

conda command not found

I recently installed oh-my-zsh to avoid my mistakes I have been constantly making while using git and improve my command line interface experience.

oh-my-zsh was great but It suddenly started not recognizing conda command.

Naturally, I got a problem with using conda virtual environment.

Add Path to ~/.zshrc file

It seems like this new theme in zsh does not have path to conda so does not know where to look when conda command comes. As suggested here, I found the right path from .bash_profile and pasted it to .zshrc file.

So first, open your .bash_profile file by running following command.

open ~/.bash_profile

You will see the PATH automatically added by Miniconda in my case. If you installed Anaconda, you will see Anaconda instead. Copy this two lines.

# added by Miniconda3 installer
export PATH="/Users/mkang/miniconda3/bin:$PATH"

Go back to the terminal and open .zshrc file via vim and paste the above lines somewhere.

vim ~/.zshrc

source the file or open a new terminal to make the change in effect.

source ~/.zshrc

Now zsh understands conda command!

Using different Jupyter kernels for different Conda environments

While using new Python 2 Kernel for Jupyter Notebook installed last time (Previous post), I found I cannot use Python libraries I had in my Python 2 in my local. Apparently, Python 2 environment used in the new Python 2 Kernel was not the same Python 2 environment that I have been using so far.

To resolve this issue and avoid confusion, I separated python environments using Conda and tie a specific environment to each Jupyter Kernel.

The first step is obviously creating a python environment (Later I will explain how I did it but here is the basic tutorial for this). But unfortunately, running Jupyter notebook from an environment does not mean the notebook will run in the same environment. It uses the default Conda environment instead of specific environment you are running the notebook from.

Installing nb_conda

I had to install nb_conda, which is an extension to provide Conda environment and an access to the associated packages from within Jupyter (Ref). Detailed info can be found here on GitHub.

This is how you install nb_conda.

$ source activate py2
(py2) $ conda install nb_conda

Checking Kernel from Jupyter

Run Jupyter Notebook after activating your environment. You can see now your Conda environment can be found in the kernel list. You can also check which kernel you are running the current notebook with from the top right.

(py2) $ Jupyter notebook

Jupyter notebook running a wrong Python kernel (Python 2 vs. Python 3)

I don’t know exactly what caused this and when, but I started to get an error while using Jupyter Notebook. The error was basically saying I can’t use Python 2 syntax even though I was in Python 2 Kernel.  I also couldn’t use packages installed in Python 2 environment such as Pandas.

print syntax error. It requires me to use Python 3 syntax while I am in Python 2 kernel.

The printed system version tells me I am using Python 3.6 instead of Python 2.7.

Why Python 3 in Python 2 kernel?

After a quick research, I figured that I did not have right kernelspec for Python 2 and Python 3 and Jupyter Notebook automatically found and used Python 3 as a default (Source: GitHub discussion). As suggested here, I set up a new kernel for Python 2.

Minkyungs-MacBook-Pro:~ mkang$ sudo python2 -m ipykernel install
Installed kernelspec python2 in /usr/local/share/jupyter/kernels/python2

Now I have two kernels both for Python 2 and Python 3.

 In the file, I have language python for Python 2.

{
"display_name": "Python 2",
"language": "python",
"argv": [
"/usr/local/opt/python/bin/python2.7",
"-m",
"ipykernel_launcher",
"-f",
"{connection_file}"
]
}

My guess is that something went wrong while in the automatic migration from Ipython Notebook to Jupyter Notebook as explained here (http://jupyter.readthedocs.io/en/latest/migrating.html?highlight=kernelspec) but I am not 100% sure.

Add module paths in Python

https://stackoverflow.com/questions/34478398/import-local-function-from-a-module-housed-in-another-directory-with-relative-im

ImportError: No module named

Even though the package is successfully installed, it is possible that I get this ImportError: No module named <package>. This is because I have several different Python interpreters and the specific Python interpreter that I am using does not have the path that the package. So, print out this in the very first line of the code:

import sys; print(sys.path)

And see if any of the paths has the specific package I am looking for. If not, 1) add the path that the package is installed or 2) install the package in the interpreter that I want to use.

 

reference: https://stackoverflow.com/questions/40958820/importerror-no-module-named-cryptography-fernet

Three ways of creating dictionaries in Python

https://developmentality.wordpress.com/2012/03/30/three-ways-of-creating-dictionaries-in-python/

Flatten 2d list to 1d list

import itertools
a=[range(i, i+3) for i in xrange(3)]
list(itertools.chain(*a))
>> [0, 1, 2, 1, 2, 3, 2, 3, 4]

Open CSV and Save CSV

 

import csv 

def openCSVFile(fileName, r): 
 lines = [] 

 with open(fileName, r) as f: 
 for line in f.readlines():
 lines.append(line.strip().split(','))
 
 return lines

def saveCSV(fileName, data):
 out = csv.writer(open(fileName, 'w'), delimiter = ',')
 for line in data:
 out.writerow(line)

Sort list X by Y

[x for (y, x) in sorted(zip(Y, X))]