CHEM455/CHEM555: Chemometrics
Table of Contents
- Class information
- First day lecture and some motivational words
- What is Chemometrics?
- Examples of Chemometrics: Concentration Determination.
- How do you deal with mixtures and Beer's law?
- Examples of Chemometrics: What is it?
- Identification of unknown material: IR spectral classification
- What is it?
- Example questions from Forensic Science
- Chemometrics is
- Industry standard options
- Why not MS Excel? So you noticed that MS Excel was not listed above as a tool for doing chemometrics?
- Octave is a free choice
- R, S, SAS, SPSS
- Python is a modern choice
- Getting the software and assignments
- Introduction to Python in Jupyter
- Introduction to Jupyter
- A computer is a calculator
- Functions and a few commands
- Some more Jupyter knowledge (Markdown)
- User input
- Control Structures: if
- Strings, Lists
- Strings
- Concatenation
- indexing and strings
- slicing stings
- Learning Review
- the immutable string
- determine the length of a string
- Learning Review A:
- Learning Review B: More Advanced (to do at your own pace)
- lists
- Defining lists
- indexing and slicing lists
- Concatenation of lists
- Lists are Mutable
- Adding items to the end of a list
- Getting the length of a list
- Lists of Lists
- learning Review
- Assignment 6 (ass6) Part 1: create a function that
- Assignment 6 (ass6): Part 2 List slicing (10 points)
- control structures: for loop
- Arrays and Matrices for data storage
- creating arrays
- Element-wise Operations on an array
- indexing and slicing
- modification of an array
- Special arrays
- Whole array manipulations (continued)
- array methods
- the Matrix
- Arrays review: math works element-wise
- Matrix: math works using the rules of Linear Algebra
- Assignment 8 Arrays (ass8)
- Assignment 9 Matrix math (ass9)
- Dataframes (the named arrays)
- Making Graphics and Figures
- General Philosophy for what type of graph to use
- Generic Example
- plotting useless data
- First some data
- Loading Multiple Spectra and their Graphs
- Multiple spectra: spectral overlays with just a few files loaded manually
- Multiple spectra: loaded with a for loop (Just load and plot the data)
- Multiple spectra: loaded with a for loop (Load, store and plot the data)
- for loop: array data storage
- for loop: dictionary / dataframe data storage load file
- for loop: dictionary / dataframe data storage load file information from Excel
- Multiple Spectra: from a pre-made dataset file (HDF or mat)
- Spectra plotting (overlayed)
- spectra plotting (stacked)
- scatter plot of linear data
- histogram of data
- Tricks that can be applied to most or all of the Former Graphs
- Computer Screen Inspection of Data
- Publication Quality Printout of Data
- Thumbnail or Lab Notebook Printout of Data
- Beamer Presentation of Data
- Reversing the x-axis for Vibrational Spectroscopic Data
- Change the Line Width of the Lines so the Show Up Better
- Change or Control the Color of the Line
- Adding Axis Labels
- control the font size
- insert a chemical structure into the graphic
- Multiple Graphs on one Figure
- Resize the Displayed Region (zoom)
- Adding a Vertical Line to a Graph and Text annotations
- adding grid lines
- adding a band label
- adding a scale bar to the graph
- ass11
- Library Searching
- Pure components
- Hit Quality Index (HQI)
- Some Math Definitions
- scalar
- Vector
- Transpose
- Matrix
- Vector and Matrix Multiplication
- dot product
- What is the HQI
- Code for calculating the HQI
- Assignment 12 (ass12)
- Other similarity Measurements (metrics of distance)
- Euclidean Distance
- Cosine metric
- Pearson's correlation coefficient
- Similarity
- ass13 (assignment 13)
- Mixture Identity
- Pure components
- Pre-Processing data
- Introduction to Linear Algebra
- K-matrix Method
- The Basic Procedure
- Prepare calibration standards
- Prepare unknown samples
- measure standards and samples
- Store data from calibration standards in to matrix
- Store data from samples in to matrix
- Store known concentrations into matrix
- Fit the data
- Spreadsheet (Excel, LibreOffice Calc, Gnumeric)
- univaritate analysis, used in samples with a single unknown or multiple unknowns with non-overlapping data (e.g. ICP-OES, etc.)
- multivariate analysis, overlapping bands or using the whole spectrum
- Use the model (K) or fitting parameters (slope and intercept) to calculate the concentration of the unknown spectra
- The math for K-matrix method
- Code for K-matrix method
- The Basic Procedure
- Model evaluation
- Principal Component Analysis and Regression
- Cluster Analysis
- Classification
- Other resources
Class information
First day lecture and some motivational words
What is Chemometrics?
My Definition of Chemometrics
The application of numerical methods to extract chemical information from chemical data.
What are numerical methods?
Math
What is chemical Data?
numbers that you get from measurements from instrumentation in a chemical lab
What is chemical information
- results
- concentrations
- constants
- equilibrium constants
- rate constants
- bond lengths
- knowledge
- models
- conclusions??
- etc.
Another resource
Examples of Chemometrics: Concentration Determination.
All analytical chemistry deals with mixture analysis. From this perspective there are two issues that are coupled.
- What are the compounds in the sample?
- How much of each are there?
- This first part deals with "How much is there?"
Determining concentration of an unknown sample.
Let us pretend you are a chemist and need to know the concentration of phosphate in an aqueous solution. What do you do?
- First is the calibration standards (external calibration)
- prepare a series of phosphate solutions of known concentrations
- measure the UV absorbance spectrum of each solution.
- WAIT A MINUTE!!!!
- you don't measure absorbance, what do you measure?
- if you are using a photocurrent detector for your measurement,
- then you measure current as a function of wavelength (sometimes called a single beam spectrum) of each solution including from a blank sample
- typically the computer software of you spectrometer converts single beam spectra of each solution and the blank into an absorbance spectrum using the following equation
- Absorbance Spectra of Phosphate
Figure 1: Absorbance spectra of sodium phosphate solutions
- Create table of concentrations of phosphate and band absorbance
Table 1: Phosphate data Concentration of Phospate /mM Absorbance at 210 nm 0.100000 0.097067 0.300000 0.296739 0.500000 0.468451 0.700000 0.669580 0.900000 0.861085 Note: 210 nm is the top of the spectral band for phosphate. This is the analytical wavelength.
- Make a Beer's Law Graph
Figure 2: Beer's Law graph of phosphate absorbance at 210 nm vs. known concentration of phosphate
- Fit the data with a linear function
Because of the linear relationship between concentration of photon absorbing species and the number of photons absorbed. (This relationship is called Beer–Bouguer–Lambert extinction law or simply Beer's Law.)
\begin{equation} A = \epsilon l C \end{equation}\noindent where the table below shows the meaning of the variables
Table 2: Beer's law variable definitions variable designation typical units meaning property of A absorbance logarithms do not have units so unitless (all below) ε absorptivity depends upon other units probability (non-normalized) of photo absorption species specific l pathlength cm how far light travels through sample measurement specific C concentration M count of specific species per sample sample specific We can fit the data with a linear function.
\begin{equation} A = m C + b \end{equation}This equation (specifically the slope and intercept) is a model for the behavior of this chemical system (chemical information).
- m is the slope (sensitivity of the photon absorbing analytes to absorb light)
- b is the y-intercept (empirical parameter that encapsulates the optical differences between the blank and the calibration standards, ideally this is zero.)
- these (m and b) are also called the fitting parameters
"Measure" the aborbance spectrum of the solution of unknown concentration
- and get its absorbance at 210 nm. Remember this wavelength (210 nm) is the analytical wavelength, because we are using the absorbance at this wavelength to determine concentration.
- in this case it is also the λmax, which is the wavelength of maximum absorbance
- solve the linear function for C
- and plug the unknown solution's absorbance at 210 nm into the rearranged linear function
Cunk is the concentration of phosphate in the solution, and is also chemical information.
What assumptions were made during this example?
- unknown solution has the same chemical environment as the calibration standards
- e.g. pH affects the λmax of
- interactions between absorbing species do not change with concentration
- usually means: relatively low concentrations (therefore we can ignore Activity)
- no interaction between absorbing species and matrix
- usually means: don't have to use method of standard addition ( ex. FAAS)
- the analyte (phosphate in the example above) is the only absorbing species at the analytical wavelength
- Very simple system,
- requires lots of unknown sample preparation to get the solution to only contain phosphate
- lots of human labor for highly variable situations (blood, river water, detergent factory run-off, drainage near a mining operation, etc.)
- and therefore of only limited use
- or involves chromatography to cleanup the sample and takes a "long" time per sample
In the example above, the sample and calibration standards contain phosphate PO43+ + Na+ in water. It is a mixture, but only the phosphate anion absorbs UV light. So from a spectroscopic/chemometric perspective this chemical system is a single component system.
How do you deal with mixtures and Beer's law?
Another word for mixture analysis is multicomponant analysis. There are 2 situations for calibration curve creation and unknown concentration determination.
1: You know every component's pure spectrum (or ε) and concentrations
| A | B | |
|---|---|---|
| situation → | spectra don't overlap | spectra do overlap |
| [C] determination → | can use 2 curves | ??? |
2: You do not know every component's pure spectrum nor do you know all of the concentrations of all of the species that contribute to the spectra
In other words, you know the concentration of your analyte in the calibration standards but not the concentrations of other spectral contributors in you unknown samples, and maybe even your calibration standards.
This is a bad situation that is all too common in the real world. You might be asking yourself, why does this situation occur? Here are some reasons:
- the sample that you need to analyze is precious (think Gollum from the Hobbit and Lord of the Rings) and can not be replaced or damaged in the measurement. Examples
- forensic science evidence
- art that is not replaceable
- the sample is expensive
- purifying the sample renders the sample useless
- Here are two examples from the literature
- the sample as a mixture has important value that is greater than the purified parts (think BBQ sauce, or steak)
- at the current technological state of the world, we can not simply mix the chemical components in wine or beer, and call the mixture good wine or beer
- instead we have to grow these products
- so identifying all of the components in the beer may not allow you to predict its quality better than measuring the beer as a whole
- https://doi.org/10.1080/10408398.2012.726659
- https://doi.org/10.1016/j.aca.2006.04.070
Some examples that are real but don't fit into the categories above
- NIR spectroscopic determination of glucose concentration in blood (everything in the blood contributes to the NIR spectra of blood)
- but for a diabetic knowing the glucose concentration in their blood is critically important for maintaining their health
- UV spectroscopic determination of nitrate in natural waters (nitrate, phospate, and chloride all "look" similar to each other in the UV spectra)
Figure 6: Toluene and p-xylene UV absorbance spectrum in a series of 2 component mixtures
You will learn in this course, how to predict the concentration of multiple components in mixtures of both situations 1 and 2.
Examples of Chemometrics: What is it?
All analytical chemistry deals with mixture analysis. From this perspective there are two issues that are coupled. What are the compounds in the sample, and how much of each are there. We saw some examples of the second part of this sentence earlier. These types of tasks are sometimes called quantitative analysis. This first part of the sentence deals with "What is this unknown material, and how does it relate to other materials?" In other courses this task is sometimes called qualitative analysis.
Identification of unknown material: IR spectral classification
In organic chemistry, you probably did some IR spectral analysis in which you identified parts of a compound by its IR spectral features. For example, you might have been given the following IR spectrum and asked to determine if it contains a carbonyl functional group or not, and if so what kind of carbonyl (i.e. amide, ketone, carboxylic acid, aldehyde, etc)
Figure 7: What kind of molecule is it?
What is it?
Lots of analytical tools will give us data that can be used to determine chemical identity.
Example questions from Forensic Science
- What polymer was used in this textile?
- Is this powder cocaine?
- Is this powder an explosive residue?
- Is this powder anthrax spores?
These things can be done by visual inspection of the data. An example is spectral interpretation, and manual comparison of bands/peaks in a spectrum to a set of reference tables. These both take lots of time.
- Washington, D.C. anthrax spores were sent in the mail.
- Ideally: need to scan every letter that passes through the mail system (millions per day)
- Whenever we need to repeat a task many, many times, often it is better to let a computer do it.
- The trick is figuring out how to train the computer to do what a human can do.
- This is what chemometrics is all about.
Chemometrics is
- getting the computer to process your data for you, rather than processing it by hand.
Therefore we will need a computer and software to do this.
Industry standard options
Why not MS Excel? So you noticed that MS Excel was not listed above as a tool for doing chemometrics?
- you are a scientist
- MS Excel is a tool for buisness people who are afraid of doing math
- I am a MS bigot
- spreadsheets are not good for massive amounts of data
- So this is a choice. You can make your own, but I am going to use another choice for this course.
Octave is a free choice
- similar to MATLAB (mostly compatable with Matlab)
- free
- open source
- if your really like it you can contribute to it or help maintain its awesomeness
- I use it all the them for some jobs
R, S, SAS, SPSS
- relatively easy to use
- these are called area specific languages (they do stats well, but only stats)
- little difficult to understand what is happening underneath the hood
- I prefer something else, and I am the teacher
Python is a modern choice
- One limitation of all of these other choices is that they are all area specific tools
- this can be a limitation, if you ever want to do something else besides process data
- Python is a general purpose computer language
- it can be used for most computer programming tasks
- it is becoming, if not already is, the language for Data Science
- Chemometrics is a subset of Data Science (and pre-dates it by more than 40 years)
- this is what we will use for this class
Getting the software and assignments
Anaconda distribution
We will use the Anaconda distribution for getting python in this course. Some benefits are
- all self-contained
- easy to install all of the parts
- does not interfere with the rest of your computer
You have 2 choices:
- use the http://virtual.wcu.edu (requires internet/network access to use, but very little computer resources)
- install anaconda
- goto https://www.anaconda.com/download/
- download the Anaconda installer for your computer (Linux, MacOS, Windoze) Python version should be > 3.9
- install it on your computer
- run it to make sure that it is working
Here is a pdf with screen captures of me installing the Anaconda distribution on a Windoze computer: ./figures/anaconda_installation_guide.pdf
NOTE: we will trying ONEDRIVE this semester instead of Dropbox, so please ignore the Dropbox section in the above pdf.
ass01
Prove to me that you able to run python in a Jupyter Notebook or Jupter lab. Go to your menu system and start Jupyter notebook. This will open a tab in your default browser. In Windoze, it should look something like:
./figures/anaconda_installation_guide.pdf
- For now, select Python 3
- This should open a new tab in you browser, running jupyter/python
- Do the following in Jupyter Notebook
Assignment 1: 1 point
In the Cell beside
In [ ]
Type the following things:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
and press Enter or Return on your keyboard while holding down the SHIFT key
in the next cell type:
print('Your Name')
and press Enter on your keyboard while holding down the SHIFT key
Screen capture the Juypter tab.
Upload this image file in the Learning Management System DropHole under ass01 or "Proof of Installation"
- You can delete this Jupyter notebook file after you email the image.
OneDrive
In this course, we will use MS OneDrive to share files. Here is a link to some instructions for OneDrive Access. ./figures/onedrive_instructions.pdf
So your second assignment is to figure out how to use this service that you already paid for with your university fees. This service uses your catamount email account as a login. Your second assignment will be to show me via a screen should emailed to shuffman@wcu.edu of your computer showing the directory structure shared by me to you on OneDrive Put in the Subject line:
Assignment 2: OneDrive
Once my account and your account are linked in this directory, you should see in that directory/folder the course files that we are sharing. Note: I will not be able to see any other OneDrive stuff of yours but only this one directory. We will refer to directory as the course home directory. Make sure that when you run Jupyter that you can access this course home directory. (One way to do this is to run Jupyter notebook/lab from this directory.) Another way is to put the OneDrive directory within your course directory on your computer. See the following structure.
chemoclass
├── OneDrive
├── learn
├──resources
├──ps1
Here is a link to a video one how to inter-operate between OneDrive and Jupyter. https://wcu.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=7918593b-9345-433a-b8e3-b2650139cbff
In this course home directory will be several subdirectories:
- resources - this contains the textbooks and papers to read
- learn - this is where you can practice your stuff
- assignments - there will be lots of these that correspond to the assignments. Each assignment will have a unique directory. They will begin with:
- ass## - assignment directory the ## is 01, 02, etc that corresponds to the assignment number
An example homework directory may already be in the OneDrive course home directory, called ps1. You can poke around there if you want. This directory will never be accessed by your instructor.
Here is what the course home directory will look like.
├── learn <-- practice stuff here, and take lecture notes here ├── ps1 <-- example assignment └── resources <-- reading material (the text book in pdf form)
The learn directory
The learn directory will have the following structure at the start. This is were you can do most of your learning/practicing for this course.
learn ├── data ├── figures ├── lib └── notebook_template.ipynb
The whole enchilada
Here is what your OneDRive course home should look like at the beginning of this course.
├── learn
│ ├── data <-- this is where learning/practicing data that you will process is located
│ ├── figures <-- this is where you save your figures generated in python
│ ├── lib <-- in this directory are library files that we will use later
│ │ ├── analtool3.py
│ │ └── chemo.py
│ └── notebook_template.ipynb
├── ps1 <-- practice assignment
│ ├── jupyter.png
│ ├── problem1.ipynb
│ └── problem2.ipynb
└── resources
└── textbook.pdf
Doing the Asssignments
Whenever you work on an assignment, make sure that you do not change the name of the files, and make sure that the files are in the same directory from which they came. ALSO, type your name in the Name Cell.
Once the assignment has beed graded, you will receive feedback in the same directory as the assignment and/or via email.
ass02
Assignment 02 will be proof that you have installed OneDrive on your computer and can receive an assignment and submit it back to me. In the ass02 directory open the jupyter notebook and type your name in the cell below the instructions.
Introduction to Python in Jupyter
Here is a tutorial for python. Much of the ideas in this learning module can be found at this tutorial.
Introduction to Jupyter
Whenever you run Jupyter on your computer, you will see in your browser the following:
From this screen you can either start a new python notebook or open an existing one. In your home/learn directory you will find at least one notebook template. Copy that template to you working Jupyter directory. From the home screen you can check the notebook template checkbox and an option at the top to duplicate the notebook will appear. Do this. The reasoning is that you preserve the template for future use and only work on copies of it. You can rename the copy to anything that you like.
Figure 9: Jupyter notebook page running
To run python code, you type the code in to an active cell in Jupyter (as indicated by the blue bar on the left side of the active cell). Once you have finished typing the code to execute (or run) the code you simultaneously depress the Shift-Enter keys. (hold SHIFT and press ENTER)
Use the template
A lot of our code in this course will rely upon pieces of code that other people have written, these pieces of code are stored in modules that come with the Anaconda python distribution that you have hopefully already installed. A template has been provided in your learn directory. You can start with this template and make a copy of it directly from Jupyter. See if you can figure out how. I would suggest that you keep a pristine copy of this template. That way if you break what you are working on, you can go back.
learn ├── data ├── figures ├── lib └── notebook_template.ipynb <------HERE
A computer is a calculator
For the purposes of this course and in my opinion, a computer is nothing more than a calculator (although a big, powerful, awesome one). (parts 1-5)
What do you do with a calculator?
- read/write data
- process data
- plot data
- evaluate data
- extract information from data (math)
What are the parts of the calculator
- user interface for input (for us: jupyter)
- user interface for output (for us: jupyter)
- a computational engine for doing the math (for us: python)
- storage space for saving things for later usage (for us: jupyter for temporary and your hardrive for longer term)
comments in python
- Use the # symbol to add comments (extra info that might be useful latter but is not used in the calculations) to your math or code
- definition: code are the words and other groups of letters and numbers that you type to get the computer to do work
Some Rules of python
- the code in python is case sensitive, so a variable named tup is different than Tup, and TUp and TUP, etc.
- some symbols have specific predefined meaning in python:
- all variables must begin with either a letter or an underscore
- after the first character the remainder of your variables may consist of any number of letters, numbers or underscores
- Don't use the following words in variable names that you create
| and | assert | break | class | continue |
| def | del | elif | else | except |
| exec | finally | for | from | global |
| if | import | in | is | lambda |
| not | or | pass | raise | |
| return | try | while | ||
| False | class | finally | is | return |
| None | continue | for | lambda | try |
| True | def | from | nonlocal | while |
| and | del | global | not | with |
| as | elif | if | or | yield |
| assert | else | import | pass | |
| break | except | in | raise |
or these
| numpy | Float | Int | Numeric | Oxphys |
| array | close | float | int | input |
| open | range | type | write | zeros |
or these
| acos | asin | atan | cos | e |
| exp | fabs | floor | log | log10 |
| pi | sin | sqrt | tan |
- white space matters: don't use spaces in filenames or functions or variables
- don't use hyphens - or parentheses or brackets ( or ), [, ], {, or ] in variable names
- don't use any special character other than underscores invariables names
All of these special characters have specific meaning in python and will be mis-interpreted
- Review:
- type some words from the tables above in the cells of jupyter to see that the change color (this is called syntax highlighting), and it means that Jupyter recognized those words as predefined Python words
- type some variables in a different cell disobeying the rules (just do one at a time) and execute the cell (Shift-ENTER or Shift-Return). Look at the error messages that are created.
- Use this as a template for defining a variable (x is the variable).
x = 1.0
displaying stuff
- use print( whatever you want to print, separated by commas) to print stuff to the screen
arithmetic
Computers can do arithmetic. Here are the commands for various arithmetic operations.
Type the following things in a jupyter cell and press shift-enter after each one:
print(4 + 4) # this command prints the results of 4+4 to the screen 4+4 #this one does not (unless it is the last command in the cell) print(4 - 4) #What does this do? print(4 * 4 ) #and this print(4 / 4) print(4**2)
in these examples the +, -, , /, and * are called operators.
Note: In Python ** is the operator symbol for power.
algebra
Lets say we have this simple algebra equation
\begin{equation} 4=x-1 \end{equation}and your task is to solve for x. What do you do? You add 1 to both sides to give you
\begin{equation} x = 4+1 \end{equation}in this equation x is called a variable. In a cell, type:
x =5 #this command assigns the value 5 to the variable x # but will not print the contents of x print(x) #this command prints the contents of x
Now type in a cell
x = 4+1 # this command assigns the results of the operation 4+1 to x # but will not print the contents of x print(x) # ths command prints the value contained in x
Notice how you get the same results. This means that the computational engine (python interpreter) is computing 4+1 and storing it into the variable x.
Note for later: 5 is called a scaler number.
Once you have this variable filled you can perform operations on it. Type
x = 5 print(x*5) ##this command prints x times 5 to the screen
or we store this result in a new variable
x = 5 u = x*5 print(x,u) #this command prints x and u to the screen
Fancy printing part 1
x = 5 u = x*5 print(x,u) #this command prints x and u to the screen print('x*5 = ',x) #this command prints x to the screen with some text showing how it was calculated
Fancy printing part 2
The previous output may not necessarily be what you want. Take this example:
# pi = 3.141592653589793238462643383279502884197169399375105820 print('pi = ',pi) # this prints an undefined by you amount of digits print('pi = {:.2f}'.format(pi)) # in this case you choose the significant digits to print
This example uses the format function. It does 2 things.
- it converts the number stored in the pi variable to a string (letters and numbers and symbols) for display/printing purposes
- it allows you to control what that string looks like
Why do this? Humans and computers need different formatting to read information. This format command converts computer looking information to human useful formatting.
In the second print command the f tells the computer that the number (pi) is a f loating point number (has a decimal point). The .2 tells the computer that you want 2 digits to the right of the decimal.
Here is a link for how to control this formatting. You can explore this at your leisure.
- Review:
- Store you name as a string variable
- Figure out how old you are in Days (http://jalu.ch/coding/days/en)
- Store this number in a variable
- Calculate the percentage of your living days in a century of days ( you can use 36525 as an approximation) and store it as a variable.
- print a sentence (string using single quotations) that says something like: My name is X, and I have lived Y percent of a century.
- Use the format method (command) to substitute your name and century percentage in the printed sentence
geometry
Given the figure below what is the distance r?
Figure 10: Distance in 2D space
You may or may not remember the Pythagorean theorem
\begin{equation} r^2 = a^2 + b^2 \end{equation}where
\begin{equation} a = 3 -1 \end{equation}and
\begin{equation} b = 2-4 \end{equation}To solve for r, we need to take the square root of each side of the equation above.
\begin{equation} r = \sqrt{a^2 + b^2} \end{equation}Let us calculate r using python. Type
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np a = 3-1 b = 2-4 r = np.sqrt( a**2 + b**2 ) ## note the np.sqrt, note the ** for power print('np.sqrt( a**2 + b**2 ) = ',r) print('np.sqrt( a**2 + b**2)= {:2.2f}'.format(r))
What is the np.sqrt word in python? It is a built-in function; meaning that it comes with python as part of the numpy module that we loaded as np. Functions have the format
output = function( input ) Note the type of parentheses
where output (called returns in the python lingo) is the result of the function doing its thing, and input is the directions, data, and numbers that the function uses. This function np.sqrt computes the square root of a number.
Probably every mathematical function that you might need is available in the numpy module.
- General ideas about functions
- they can have more than one input (called parameters in the python lingo)
- the inputs can be any kind of data structure, i.e. numbers, letters, matrices, vectors, etc.
- to assign output to a variable simply used an equal sign =
- function usage: outputvariable = function(input)
- help about a function can be found by typing a ? in front of or after the function
type ?np.sqrt in a cell and execute the cell.
Review: Look at the numpy mathematical functions reference page
https://numpy.org/doc/stable/reference/routines.math.html
- lookup some functions (mathematical) that might be useful and figure out how to use them. Here is a list that I use all the time
- cos
- sin
- tan
- round
- exp
- log
- log10
- sqrt
- max
- min
- Check you answers in python with a calculator to ensure that you are using them correctly.
- Here is an example (https://numpy.org/doc/stable/reference/generated/numpy.round_.html#numpy.round_)
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np pi = np.pi ## notice that we also get pi from the numpy (np) module xrounded = np.round(pi, 5) ### round pi to 5 decimal places print(xrounded)
Functions and a few commands
Showing the cell line numbers
If you want to show the cell linenumbers, click to the left of the [ ]: of any cell. You should see a thick vertical blue bar. Then depress Shift-L. This key-stroke sequence will toggle the cell line numbers on and off.
Some more python/jupyter info
First some more ideas about how jupyter/python
- Each time you start jupyter is called a session. By default all of the memory is wiped for each session.
- Each time you execute a command block or cell (type the command and depress the shift-enter keys) the results are stored in the memory for that session.
type each in a cell and execute:
x = 1 who ##this command prints the variables currently in memory
y = 100 whos # this is a "magic" command and prints variables and their types
Building your own functions (ideal setup)
The numpy module for python contains probably every math function that you might want. Here is the website. https://numpy.org/, so you can check it out.
What if numpy or Python does not have a function that you want? What do you do?
Build you own!
For example, if we wanted to find the angle Q in the geometry problem from 10. You would need to use the following trigonometric relationship
\begin{equation} Q = arctan\left( \frac{a}{b} \right) \end{equation}How do you go about creating a function to find Q?
First solve the problem (for now, not a function)
Before we create a function lets programmatically solve the problem, then wrap the solution into a function.
Let us solve the angle problem that we saw before and find Q. Type the following in a Jupyter cell and execute
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np a = 3-1 b = 2-4 Q = np.arctan(a/b) print('the angle Q is equal to {:2.3f}'.format(Q))
OK, so this gives you a benchmark in which to compare, i.e. the correct value for Q. Our function should give us the same answer. Note that the angle is in radians not degrees. This is the default.
OK now a function
Defining a new function in python involves wrapping the code with
def function_name( inputvariable1, inputvariable2): ## end the first line of a code block with a colon ''' this is a comment section or help section that prints when you type ? before a function in jupyter notice the indentation. this is important in python ''' import numpy as np ### this is where you type the code for your calculations return outputvariable ##return stuff back to the caller
Note: In python, blocks of code are delineated by indentation
Now wrap the findq code above into a function.
def findq(): ## end the first line of a code block with a colon ''' for now let's just have the function make the calculation and print it out. Later we will make it more flexable ''' import numpy as np #this command loads all of the functions in numpy and labels them np a = 3-1 #Coordinates hard coded into the function b = 2-4 #Coordinates hard coded into the function Q = np.arctan(a/b) print('the angle Q is equal to {:.3f}'.format(Q)) return #notice the indentation is back to the first column. So the above code block is ended findq() # you need the () they are part of the function.
Note: when you define a function you have not run or executed or called the function. To do that you have to use the function's name or call the function. (See the last line in the code block above.)
What does this function do? It calculates the angle Q from a set of fixed coordinates that are coded into the function and prints the value of Q to the screen. This approach has several limitations:
- you need to redefine the function for a different a, b, and Q. (type in the triangle's coordinates for a and b and execute it (Shift-ENTER)
- To use the value of Q in this form you would need to copy the value by hand and type it again (or copy->paste)
- The value has been truncated. That could give you rounding errors further down the line of calculations. (Remember you don't round until the end of you calculations.)
Now a more useful function
Great. Now let us make this function a little more readable and useful. In the function findq
- add some comments to the header portion of the function
- some commented definitions to explain what this function does.
- return the value of Q to the caller
def findq(): ## end the first line of a code block with a colon ''' this function calculates the angle between to line segments. Usage: angle = findq() prints the answer to the screen and returns the angle The angle has units of radians ''' import numpy as np #this command loads all of the functions in numpy and labels them np a = 3-1 b = 2-4 Q = np.arctan(a/b) print('the angle Q is equal to {:2.3f}'.format(Q)) return Q #return angle Q angle = findq() ##store q in the variable angle print('The square of the angle is ',angle**2, 'and the angle is',angle) #now we can use this angle for something else
What does this function do?
- It calculates the angle Q from a set of fixed coordinates that are coded into the function
- prints the value of Q to the screen.
- it returns the value of Q with all of its digits (that can be utilized in future calculations)
What are its limitations:
- it only calculates the angle of a very specific triangle
and now an even more useful function
This function findq is not that useful; it only calculates the angle of a very specific triangle, prints the value to the screen, and outputs the angle.
You could do the same thing a lot easier without a calculation:
def sameasfindq(): ''' same behavior as findq ''' print('the angle Q is equal to -0.785') return -0.785398163397 angle = sameasfindq() print('The square of the angle is ',angle**2, 'and the angle is',angle)
It is not quite the behavior that we want. A better function would be one that could be used for multiple problems i.e. a more flexible function.
Let's alter the function to make it one that can calculate the angle given by a user's set of line segments. NOTE: Who is this user? It could be you. Or a future you. Or a collaborator to whom you share the function?
Users of the function tell function what to use for input by specifying the variable inside the () in the definition.
def findq(a,b): ## end the first line of a code block with a colon ''' this function calculates the angle between to sides of a triangle. Usage: Q = findq(a,b) prints the answer to the screen and returns the angle The angle given has units of radians a is length of the opposite side b is the length of the adjacent side ''' import numpy as np #this command loads all of the functions in numpy and labels them np #a = 3-1 #commmented out so you can see the change #b = 2-4 #commendted out so you can see the change Q = np.arctan(a/b) print('the angle Q is equal to {:2.3f}'.format(Q)) return Q #return angle Q angle = findq(3-1,2-4) ##store q in the variable angle a = 3-1 b = 2-4 angle=findq(a,b) print('The square of the angle is ',angle**2, 'and the angle is',angle) #now we can use this angle for something else
This is a more flexible function.
Review:
write a function that will use the general solution to the quadratic equation
\begin{equation} 0 = ax^2+bx+c \end{equation}Your function should do the following:
- take 3 numbers (a, b, c from the equation above) as inputs
- solve for the roots of the equation
- print the roots to the screen
- return the roots
Do this now.
Assignment 3: ass03 toy code
Do the ass03 assignment in your dropbox home.
- There is 1 notebook.
- There is 1 part in the notebook.
- Due date is written in the txt file in the ass03 directory in your dropbox home
The purpose of this assignment is to practice a little more using Jupyter and to learn (both for your instructor and for you) about how to use notebooks with assignments embedded in them, and to write a function.
Some more Jupyter knowledge (Markdown)
- Take this tutorial
https://www.markdowntutorial.com/
- jupyter's help on markdown cells
- book on LATEX math formatting
practice making headers, lists, and math equations in juytper using the markdown cell type
- Make the is list in Jupyter
- typeset the following equation in Jupyter
User input
Sometimes when you need input from the user the easiest is to use input.
Here is an example.
ui = input('type something ')
print(ui)
ui+1
ui = float(ui)
print(ui+2)
Control Structures: if
Now to Controlling things
Computers are not useful unless they make our lives better or easier. Consider the following situations:
- adding up all of numbers from 1:100
- adding up all of the even numbers from 1:100
- determining the square root of a list of numbers
- reading a file line by line until you reach the end of the file
Here is the first one in code form:
x = 1+2+3+4+5+6+7+8+9+10+11+12+13+14+15+16+17+18+19+20+21+22+23+24+25+26+27+28+29+30+31+32+33+34+35+36+37+38+39+40+41+42+43+44+45+46+47+48+49+50+51+52+53+54+55+56+57+58+59+60+61+62+63+64+65+66+67+68+69+70+71+72+73+74+75+76+77+78+79+80+81+82+83+84+85+86+87+88+89+90+91+92+93+94+95+96+97+98+99+100 print(x)
Typing all of this is annoying, and if we have to type everything then the computer has not made our lives easier, nor have we gained any time if we want to repeat this computation, or alter it (only the odd numbers), or expand it 1 to 1000.
So most computer languages contain structures that are time saving devices or allow for the program to alter its computational trajectory if a situation changes.
In python there are 3 that are important for this course.
- if then else
- for loops
- while loops
To help you understand how theses structures work, we will use children's games.
If then else
This control structure can be understood under the context of the children's game Simon Says
Rules of the game:
if (the leader says "Simon say's" before they tell you to do something): [condition] then you do it [conditional action] else: don't do it [default action] end of turn
Translation of this previous block into code,
the text in the ( ) after the if statement is a testable condition (usually a mathematical comparison).
If this condition is True (sometimes designated as 1) then the [conditional action] code block is performed.
The else tells the computer that the default code block is coming up next
If the condition is False (sometimes designated as 0) then the default code block is performed.
The end lets the computer know that the control structure is over
Here is a toy example of using the if statement to see if two numbers are equal
x = 4 if x == 3: #in python the conditional if line ends in a colon (:) print('x was 3') x = x**2 print('now it is nine') else: # there is also a colon after else: print('x was not 3') # in python a code block (the whole if block) is "ended" by returning back to the # same column number as the i in if print('Thanks for playing!')
NOTE:
- the = symbol is for setting something equal
- the == symbol is for comparison
There are many kinds of testable conditions that can go after the if statement, but some common ones are
- == (is equal to) Note that there are 2 equal signs next to each other. This is for comparison.
- < (is less than)
- <= (is less than or equal to)
- >= ( is greater than or equal to)
- > (is greater than)
- != not equal to
Here is another kind of if testing:
refletters = 'abcdefg' #this is a string questionletter = 'g' #here is another one if questionletter in refletters: #remember to end with a : print(questionletter, 'was in the set',refletters) else: #a new code block begins so remember to end with a : print(questionletter, 'was NOT in the set',refletters) print('Thanks for playing')
NOTE:
- the in conditional test means just what it sounds like "in"
- Also try if questionletter not in refletters and see what that does.
- Also try if questionletter == refletters and see what that does.
- <
- >
- ~=
These logical notations work as well but for strings may not be as easy to understand. The in and not use in these conditional tests are examples of how Python was designed to be easy to learn: Python was born to be readable.
else if or elif
There is also the possibility of performing multiple comparisons within one if then else structure. To do this use else if or elif in python. Here is an example:
#this program tests if the number stored in x to be greater than 3 # and if it is then set it to a maximun value of 3 x = 3.9 if float(x) == 3: # the colon tells python that a code block has begun print('x was 3') elif x > 3: # another code block so you need another colon at the end of the line print('x was greater than 3') x = 3.0 ## redefine x as 3 elif x < 3: # another code block so you need another colon at the end of the line print('x was less than 3. So I letter it pass.') else: # another code block so you need another colon at the end of the line pass #this command is only requred if this else or elif choise is empty msg = 'x is now {:.1f}'.format(x) ## some pretty printing of the result print(msg)
- what happens when you change what x = equal to? Try the values 4.5, -12 and 0
- what happens when you set x = 'v' (a string)?
- this raises an error and crashes the program
- we will learn to deal with this later
- Why? you can not compare a number (floating point or integer) and a string
- Comparisons need to be made between the same types of python objects (type whos in a cell to get a list of variables and their types)
Boolean logical combinations
Another way to combine multiple conditional comparisons is to combine two comparisons with Boolean expressions or operators
Extra reading on Boole and his Logic
Some example ideas:
- The 3 basic Boolean operators are
- and
- or
- not
- If you speak English then you know what these mean
- and both sides of the and needs to be true for the whole statement is true
- or either side of the or needs to be true for the whole statement is true
- not is often called an inverter. Its presence in a statement inverts the condition
- True becomes False
- False becomes True
- The same ideas apply to programming languages
- Here is an example:
#this program checks to see if x to be greater than 3 and less than 1 x = 5 if x <= 3 & x >= 1: #both must be true to for the conditional action to be executed print('x was between 1 and 3') #conditional action else: print('x was outside of the range 1 and 3') msg = 'x is {:.1f}'.format(x) print(msg)
NOTE: the syntax (symbols and commands) used in Python for Boolean operators are either the simple English words (and, or , not) or 3 symbols:
| Boolean operator | word for symbol | symbol |
|---|---|---|
| and | ampersand | & |
| not | bang | ! |
| or | pipe or vertical bar | https://en.wikipedia.org/wiki/Vertical_bar |
General form
Note that the elif and else parts are optional, but make your code cooler looking.
if condition: code block elif other condition: #optonal code block #optional else: #optional defaultcode block # optional
Learning Review:
Write a function that
- takes one input
- determines if the number is greater than 0, less than zero, or zero.
- and prints the string '+', 'nil', or '-' depending upon the conditional testing results
Learning Review: Answer
def numtest(x): if x < 0: msg = '-' elif x > 0: msg = '+' elif x == 0: msg = 'nil' print(msg) return ## test the function numtest(75)
Strings, Lists
Strings
Before we jump into lists and arrays let us start with strings. We have already seen some string usage in this tutorial, but some ideas about lists can be learned from using stings.
First a string is surrounded by either single or double quotes. They both work, but you must used either single or double to define one string. Don't mix them! A string contains zero, one or more characters. For example:
word = 'python'
The variable word contains a string with 6 characters in it.
There are some special characters that can be in strings.
- \n results in a newline
- \t results in a tab
- \' puts a single quote mark in the string
Here is some usage
words = 'This is a string in \t \t python. \n\t Isn\'t it awesome!' print(words)
Sometimes you want to put special characters in the string as normal characters. Here is an example and a solution
- use the r before the string definition to specify that it is raw and not interpreted.
path = 'C:\some\name_of_directory' print(path) betterpath = r'C:\some\name_of_directory' print(betterpath)
Concatenation
Strings can be concatenated.
What is Concatenation?
- The linking of things together in a chain or series
- Strings are concatenated with the + between strings
Here are some examples:
## un un un + ium name = 3 * 'un' + 'ium' # NOTICE the + between the un and ium + means concatenate in strings print(name)
NOTE: notice the 3* 'un' in the code above. Some operators (multiply) can be used on strings. These operations mean to take 3 of the strings in a row, or concatenate 3 of the strings. Try using other operators on strings (divide, addition, subtraction), you will notice an error.
indexing and strings
From the python tutorial: https://docs.python.org/3/tutorial/introduction.html#strings
Strings can be indexed (subscripted), with the first character having index 0. (Some programming languages have a separate character type of variable. In Python there is no separate character type; a character is simply a string of size one. If you don't understand this set of statements, you can ignore it.)
Here are a bunch of examples of indexing strings:
word = 'Python' print(word[0]) # print the character in position 0 print(word[5]) # print character in position 5 ##extraction of subsets of the string x = word[1] #extract the second letter in the string or charactor as index = 1 print('the string\'s second letter or index 1 is',x) x = word[-1] # extract the last character in the string print('last is ',x) x = word[-2] # extract the second-last character in the string print('second to last is',x)
NOTE: Since -0 is the same as 0, negative indices start from -1. NOTE: Indices can have positive and negative values (see the last to examples in the code block above.)
slicing stings
In addition to indexing, slicing is also supported. While indexing is used to obtain individual characters, slicing allows you to obtain a substring:
word = 'Python' ### substrings x = word[0:2] # characters from position 0 (included) up to 2 (excluded) print('substring',x) x = word[2:5] # characters from position 2 (included) up to 5 (excluded) print('substring',x) print(word[:3] + word[3:]) ## see note below
Note how the start is always included, and the end always excluded. This makes sure that s[:i] + s[i:] is always equal to s. This behavior is the main reason why python starts its indexing at zero. Here are some more demonstrations:
word = 'Python' ### substrings concatenation x = word[:2] + word[2:] print('notice that ',x,' is the same as ',word) x = word[:4] + word[4:] print('Also, notice that ',x,' is the same as ',word)
Slice indices have useful defaults;
- an omitted first index defaults to zero,
- an omitted second index defaults to the size of the string being sliced or the end of the string.
word = 'Python' ### substring slicing x = word[:2] # characters from the beginning to position 2 (excluded) print('characters from the beginning to position 2 (excluded) are => ',x) x = word[4:] # characters from position 4 (included) to the end print('characters from position 4 (included) to the end are => ',x) x = word[-2:] # characters from the second-to-last (included) to the end print('characters from the second to last (included) to the end are => ',x)
One way to remember how slices work is to think of the indices as pointing between characters, with the left edge of the first character numbered 0. Then the right edge of the last character of a string of n characters has index n, for example:
+---+---+---+---+---+---+ | P | y | t | h | o | n | +---+---+---+---+---+---+ i 0 1 2 3 4 5 6 j -6 -5 -4 -3 -2 -1
- The first row of numbers gives the position of the indices 0…6 in the string;
- The second row gives the corresponding negative indices.
The slice from i to j consists of all characters between the edges labeled i and j, respectively.
Note: For non-negative indices, the length of a slice is the difference of the indices, if both are within bounds. For example, the length of word[1:3] is 2.
Attempting to use an index that is too large will result in an error. Type and see what happens:
word = 'Python' word[42]
In contrast: out of range slice indexes are handled gracefully when used for slicing:
word = 'Python' x = word[4:42] print(x) print(word[42:]) ##what is this doing prints an empty string
Learning Review
Using this template above: Figure out how to extract some of these examples from the word variable (word = 'Python'):
- hon
- Pyth
- Pytho
- n
- tho
- ytho
the immutable string
Python strings cannot be changed — the computer science word for this is immutable. Therefore, assigning to an indexed position in the string results in an error: Here is a definition: https://docs.python.org/3/glossary.html#term-immutable
Here is an example in which we need to lower case the first letter of Python.
word = 'Python' #define word the first time no problem word[0] = 'p' # try to change it you will get an error message word = 'python' # redining the whole variable is no problem
Here is another way to fix this particular problem (see above).
word = 'Python' print('p' + word[1:])
determine the length of a string
use the len() function to get the length of things:
word = 'supercalifragilisticexpialidocious' length = len(word) print('the word ', word,' has ',length,' characters.')
Learning Review A:
Write a function that compares the lengths of two strings and returns the longest word with its length.
- function
- 2 inputs
- compute length of each string
- compare lengths
- output string and its length (HINT: last line of the function should include return string, length )
Learning Review B: More Advanced (to do at your own pace)
Write a function that compares the lengths of two strings and returns the longest word with its length.
- function
- 3 inputs
- compute length of each string
- compare lengths
- output string and its length (HINT: last line should include return string, length )
lists
From Python tutorial https://docs.python.org/3/tutorial/introduction.html#lists
Python knows a number of compound data types, used to group together items. The most versatile is the list, which can be written as a list of comma-separated items (numbers, strings, other lists, whatever) between square brackets [ ] . Lists might contain items of different types, but usually for this tutorial the items all have the same type.
Defining lists
squares = [1, 4, 9, 16, 25] ## define a list by enclosing csv in [] print(squares) letters = ['A','B','C','D'] print(letters)
indexing and slicing lists
Just like strings in a previous section (and all other built-in sequence types), lists can be indexed and sliced:
squares = [1, 4, 9, 16, 25] ## define a list by enclosing csv in [] x = squares[0] # indexing returns the item print('x =', x) y = squares[-1] print('y =', y) z = squares[-3:] # slicing returns a new list print('z = ',z)
All slice operations return a new list containing the requested elements. This means that the following slice returns a new copy of the list:
squares = [1, 4, 9, 16, 25] ## define a list by enclosing csv in [] x = squares[:] print('x = ',x)
Concatenation of lists
Lists also support operations like concatenation:
squares = [1, 4, 9, 16, 25] ## define a list by enclosing csv in [] x = squares + [36, 49, 64, 81, 100] # concatentation with + print('now x = ',x)
The multiply operator on a list behaves like strings. Here is a demonstration:
squares = [1, 4, 9, 16, 25] ## define a list by enclosing csv in [] x = 3* squares # repeated concatentation with * print('now x = ',x)
NOTE: this behavior of lists always seems weird to me. It is not what I would want to happen. We will learn later several ways to operate on all of the elements in a sequence (List, arrays, dataframes).
Lists are Mutable
Unlike strings, which are immutable, lists are a mutable type, i.e. it is possible to change their content:
squares = [1, 4, 9, 16, 25] ## define a list by enclosing csv in [] cubes = [1, 8, 27, 65, 125] # something's wrong here print('something is wrong here: ',cubes) print('but ',4 ** 3, ' is the cube of 4') # the cube of 4 is 64, not 65! cubes[3] = 64 # replace the wrong value print('now cubes = ',cubes)
Adding items to the end of a list
You can also add new items at the end of the list, by using the append() method (we will see more about methods later)
squares = [1, 4, 9, 16, 25] ## define a list by enclosing csv in [] cubes = [1, 8, 27, 65, 125] # something's wrong here print('cubes was initially defined as ',cubes) cubes.append(216) # add the cube of 6 cubes.append(7 ** 3) # and the cube of 7 print('appending 2 new values to cubes. now cubes is ',cubes)
Assignment to slices is also possible, and this can even change the size of the list or clear it entirely:
letters = ['a', 'b', 'c', 'd', 'e', 'f', 'g'] print('letters is ',letters) # replace some values letters[2:5] = ['C', 'D', 'E'] print('replacing indices 2-5 yields',letters) # now remove them letters[2:5] = [] print('droping the indeces 2-5 yeilds',letters) # clear the list by replacing all the elements with an empty list letters[:] = [] print('clearing the whole list yeilds and empty list',letters)
Getting the length of a list
The built-in function len() also applies to lists:
letters = ['a', 'b', 'c', 'd'] length = len(letters) print('The length of this list is ',length)
Lists of Lists
It is possible to nest lists (create lists containing other lists), for example:
### hand coding a list of lists LL = [['I','think', 'we'],['are','here']] print('hand coded list of lists ',LL) ## or more programatically a = ['a', 'b', 'c'] #make 1 list n = [1, 2, 3] #make another list x = [a, n] # now make a list of lists or nested lists print('look at what x is ',x) ##indexing a nested list b = x[0] print('the 0th index in x is ',b, ' and is the sublist a =>',a) ## indexing a nested list deepinside = x[0][1] ## note the 2 sets of square brackets print('here is an item within the 0 index nested list = >',deepinside) print('here is an item within the 1 index nested list = >',x[1][2])
learning Review
Create a funny sentence function. This function will pick 3 random indices and pull words from 3 random lists, then print to the screen the concatenated funny sentence.
Example: list of subjects: I, you, donkey list of verbs: eat, like, poop list of objects: bread, corn, hippos
randomly choose three indices 0,1,2 then your funny sentence that should be printed to the screen is:
I like hippos
Use the numpy function randint to get an integer that can be used as an index.
import numpy as np #this command loads all of the functions in numpy and labels them np from numpy.random import randint ## this command loads the function randint in to memory for later use subjects = ['I', 'You'] idx = randint(0,high=2) # the 0 is the lowest integer (0th index) # ths high=3 is one count above the highest integer that you need # to make this flexible maybe don't hard code the high = 2 # but get it from the list print('idx is ',idx)
Assignment 6 (ass6) Part 1: create a function that
- takes 2 inputs
- 1\(^{st}\) is a list of floating point numbers (1.0, 3.12, etc.)
- 2\(^{nd}\) is an index that is less than the length of the list
- takes the number in the list at the index position and replaces it with 13
- appends the number 114 to the end of the list
- uses the built-in function sum() to sum all of the items in the list
- returns the sum of the newlist
Due Date is in the directory
Assignment 6 (ass6): Part 2 List slicing (10 points)
create a function that
- takes 1 input
- the input is a list of items 'dog',1,'R',1.0, 3.12, or whatever with length > 5
- checks to see if the list is > 5 items long
- prints an error message if the input list is too short, and returns a 1
- otherwise takes a slice of the 1,2,4 index of the list
- returns the returns the slice of the list
Due Date is in the directory
control structures: for loop
This control structure is a little difficult to compare to a children's game. The most similar game is a relay race. Your goal in this race is to carry a stick around the track a specific number of times.
for each time around the track carry the baton when you have looped around the track the required amount of times the race is at an end
Use a for loop to do a task a specific number to times. Here is an example
for i in [1,2,3,4,5,6,7,8,9,10]: print(i)
In the previous example, [1,2,3,4,5,6,7,8,9,10] is a list. I contains a list of integer numbers.
You would read this code block as "for i equals items in the list", [1,2,3,4,5,6,7,8,9,10], then display i then end the code block. In a for loop, the iterator variable i in the case above takes on the values from the list 1, 2, 3, 4, 5, ….
range()
What would be really nice is if we had a way to generating this list [1,2,3,4,5,6,7,8,9,10] rather than typing it. Use the command range()
This example should give the same output.
for i in range(10): print(i)
The range() function can take up to 3 inputs.
- range(begin,end,stepsize)
Let's use this range() function to add up some numbers
start = 1 stop = 101 step = 1 count = 0 ## this is the seed of the count for i in range(start,stop,step): count = count + i print(i, count)
Here is another way to do the same thing, but it stores the values of i in a list:
start = 1 stop = 101 step = 1 L = [] ## this starts a list to put the numbers in for i in range(start,stop,step): L.append(i) print(i, L) Ls = sum(L) print('The sum is ', Ls)
NOTE: the command sum(). It does what you think it should. It sums the list L.
List processing with for loops with range()
This list generator (range()) is good for generating indices for accessing components in a list or other sequence data structure.
L = ['The','cat', 'has','a','scary','smell'] for i in range(len(L)): print(L[i])
In Python, this iterating over a list is very powerful, especially because the list can contain any python structure.
L = ['The','cat', 'has','a','scary','smell'] for word in L: print(word) #Notice how there is no math here at all
We can also access more than one list by using an indices generator.
L = ['The','cat', 'has','a','scary','smell'] P = ['THE','CAT', 'HAS','A','SCARY','SMELL'] for i,word in enumerate(L): ##use the enumerate around a list to create an index generator print(i,word,P[i]) # i is the index of the list and word is the item in the list
Learning Review: add up some stuff
Create a function to add up all of the numbers in a range
- use a for loop to step through the range of numbers
- use range: 12 to 115
- also figure out a quick change to your range of every 3rd number
- print and returns the sum
Learning Review: Answer
# learning review check add up some stuff #first do it not in a function start = 12 stop = 115 step = 1 count = 0 for i in range(start, stop + 1, step): count = count + i print(i,count) ## then make a function def addmup(start,stop, step): #put the parts defined by the user as inputs ''' Adds up all the numbers from start to stop step is the step size between the start and stop, 1,2,3, etc. ''' count = 0 ## you start adding at zero for i in range(start, stop + 1, step): ## notice stop + 1, range before the value in stop count = count + i print('The sum of all values in the range (',start,' to ',stop,') is',count) return count ## test out your function addmup(12,115,1)
Learning Review: Fibonacci list
write a function that
- takes 1 input as an integer (n)
- creates a list that contains the Fibonacci sequence up to and including the nth
- returns the list
Learning Review: Fibonacci list: Answer
def fibonacci(n): ''' function to generate a list of the first nth numbers in the Fibonacci sequence n is the integer number in the fibonacci sequence n must be larger than 1 ''' F1 = 1 #first number in the Fibonacci sequence F2 = 1 # second number in the Fibonacci sequence fiblist = [F1,F2] # seed the list with the first two fibonacci numbers for i in range(1,n-1): # start loop up to n, note: start with 1 because that is how most people count, n-1 already have fisrt 2 in list, remember in range(stop) the stop number is 1< stop F = F1+ F2 # calculate the next number fiblist.append(F) #store the F in running list F1= F2 # Shift current F1 to F2 F2 = F # shift current F to F1 return fiblist #return the list print(fibonacci(12))
Check if an item is in a list
if 'cat' in ['cat','dog','cow']: print('yes')
ass7: Fibonacci 1
Write a function that will
- take 1 input that is a positive integer (n)
- if the input is not an integer return an error of 1 (HINT: use the isinstance function link)
- and print an error message
- calculate the nth Fibonacci number
- return the Fibonacci number
ass7: Fibonacci 2
write a function that
- takes an integer as an input
- checks if the input is a Fibonacci number
- returns a True if it is a Fibonacci number
- returns a False if it is not a Fibonacci number
ass7: Fibonacci 3
Format in Markdown in the cell provided the Equation:
\begin{equation} F_n = F_{n-1} + F_{n-2} \end{equation}- Due Date Feb 16, 2018 at noon
Arrays and Matrices for data storage
As we learned in the last several modules, lists are powerful Python objects that can be used to store information or data. However there are several (2) limitations with lists that make them less useful for storing data.
Specifically,
- mathematical operations on a list don't work in a useful way.
- ???
Here is an example this limitation: Let's say you needed to calculate the square of a list of numbers (such as in the calculation of the standard deviation.)
Try this in python
data = [15.2, 16.7, 17.34, 14.6, 13.9] data*data # this is the calculation for the square of a number data ** 2 # nor can you do this
Notice how it gives an error. You can not multiply 2 lists together nor can you take the power 2 of a list.
Here are 2 ways of doing this operation using a for loop Note there are more variables used here than are necessary. I am using them to make the example more understandable.
data = [15.2, 16.7, 17.3, 14.6, 13.9] ## the org data sqdata = [ ] ## create an empty list to store the squares for d in data: ## us 'in' to get the items out of the data list one at a time sqd = d**2 ## square the data and store it in a new variable sqdata.append(sqd) ## append the square to the list print(data) print(sqdata)
data = [15.2, 16.7, 17.3, 14.6, 13.9] ## the org data sqdata = [ ] ## create an empty list to store the squares for i in range(len(data)): ## create an increasing index 'i' to access the items in data d = data[i] ## extract the item from list 'data' sqd = d*d ## square the data and store it in a new variable sqdata.append(sqd) ## append the square to the list print(data) print(sqdata)
Using a for loop to process lists can be tedious. If you have a huge list it is also very slow. (This is a fact that is not self-evident from what we have covered so far. You will have to either accept this statement or you can read about it link.
So the Python community have created additional Python object types to help with this kind of problem. They are array, matrix, and dataframe. These are all very similar to List, but they have special properties that are useful for data analysis.
You might envision arrays as a kind of data table or spreadsheet-type of computer object.
Look at this table.
| 1 |
| 2 |
| 3 |
| 4 |
It is basically an array of dimensions 4 by 1 (4x1) or (4,1). [ Notice the dimensions are (row, column). ] And here
| 1 | 2 | 3 | 4 |
is an array of dimensions 1 by 4 (1x4) or (1,4). These are both considered 1D arrays (another word is vector), which means that they only have 1 dimension that is greater than 1.
Now consider the following array
| 1 | 2 |
| 3 | 4 |
What are its dimensions?
- ANSWER: (2,2) two by two
The only limitation with thinking of arrays as a kind of table is that most people don't think of data tables as having more than 2 dimensions. But arrays can have as many dimensions as you can imagine (or whatever the limit of the computer language Creator's imagination was).
Here is some code for understanding arrays.
creating arrays
To create/operate/use arrays, we must use the numpy module.
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np A = np.array([4,5,6,7]) #turn a list into an array print(type(A)) ## print the type of object print('the shape of the array is ',A.shape) # print the shape of the array.
Type %whos in the cell of Jupyter (the % means magic command in Jupyter) and you should see in the list:
Variable Type Data/Info ------------------------------- A ndarray 4: 4 elems, type `int64`, 32 bytes np module <module 'numpy' from '/ho<...>kages/numpy/__init__.py'>
Notice the variable A type is ndarray. You can also access this information with the type(function). ndarray means n-dimensional array.
Use the .shape method to extract the shape of an array. Why does it say (4,) instead of (4,1)? This is just Python's way of saving space/memory.
Element-wise Operations on an array
A good reason for using arrays is to operate on them easily. Let's take the problem of squaring a list above. Here is how to solve this problem with numpy arrays.
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np datalist = [15.2, 16.7, 17.3, 14.6, 13.9] ## the org data data = np.array(datalist) ## first make the list into an array sqdata = data**2 ## square the data print(datalist) print(sqdata) print(type(sqdata))
Whenever you use an operator on an ndarray object it operates on each element and returns a new array. Try a several
- data - sqdata
- data * sqdata
- data + data
These all do element-wise calculations.
indexing and slicing
Array indexing and slicing works just like a list.
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np data = np.array([15.2, 16.7, 17.3, 14.6, 13.9] ) # onestep create of array print('The zeroth index of the array is ', data[0]) #indexing returns a number print('The first 3 items in the array are ', data[:3]) ## slicing returns an array
modification of an array
Arrays are like lists in that they can be modified.
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np data = np.array([15.2, 16.7, 17.3, 14.6, 13.9] ) # onestep create of array print('The zeroth index of the array is ', data[0]) #indexing returns a number data[0] = 55.5 ## replace this zeroth indexed number print('The zeroth index of the array is now', data[0]) #indexing returns a number
Special arrays
There are some special arrays that are easy to create with numpy.
- an array of all ones
- an array of all zeros
- an array of all random numbers
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np ## all ones A = np.ones( (3,3) ) ## the (3,3) is the shape that you want print('A is \n', A) ## the \n prints a new line print(A.shape) ## all zeros B = np.ones( (4,4) ) ## the (4,4) is the shape print('B is \n', B) ## the \n prints a new line print(B.shape) ## normally distributed random numbers center = 3 ##center of random numbers spread = 2 ## standard deivation of the random numbers shape = (2,4) ## shape of the array created. this is called a tuple C = np.random.normal(loc=center,scale=spread,size=shape) print('C is \n',C) ## the \n prints a new line print(C.shape) ## uniformly distributed random numbers D = np.random.rand(3,5) ## random numbers are spread out evenly between 0 and 1 print('D is \n ', D) ## the \n prints a new line print(D.shape)
Whole array manipulations (continued)
Another powerful utility of numpy is its built-in functions. Here are a few useful examples that do what you think they do.
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np data = np.array([15.2, 16.7, 17.3, 14.6, 13.9] ) # onestep create of array print(data) print(np.sin( data )) print(np.sum(data)) print(np.median(data)) print(np.min(data)) ## determines the miniumn value in an array print(np.max(data)) ## determines the maximun value in an array print(np.std(data,ddof=1)) ## determine the standard deviation "ddof = 1" means use degree of freedom = 1, which is the words for n-1 in the bottom of the std. dev. equation.
numpy has about 99% of all of the functions that you might want. Here is a list in their reference page.
array methods
In the numpy module, the array objects have methods associated with them. A method is an operation on an object that can be applied to that object. Jupyter has several nice features that can help use understand some of these.
import numpy as np #this command loads all of the functions in numpy and labels them np data = np.array([15.2, 16.7, 17.3, 14.6, 13.9] ) # onestep create of array print( np.sum(data) ) ## print the np.sum of the data ndarray print( data.sum() ) ## same as np.sum( ) but as a method: the dot .sum() makes it a method
The methods produce the same results as the functions. Note: Not all functions are methods. Try these methods:
- sum()
- mean()
- max()
- sin()
In Jupyter there are some convenience features. Type
import numpy as np #this command loads all of the functions in numpy and labels them np data = np.array([15.2, 16.7, 17.3, 14.6, 13.9] ) # onestep create of array # then type data. and depress the tab button, and wait, you will see a menu pop-up of all the methods
the Matrix
Python provides a special type of array called a matrix. The major differences between a matrix and an array is the manner in which mathematical operations are handled. We will only use the python matrix when necessary.
Arrays review: math works element-wise
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np A = np.array([[ 2, 2.], [-2. , 2]] ) ## create array print(A*A) ## perform element-wise multiplication B = np.array([2,2]) print('The shape of B is', B.shape) ## notice how the shape is (2,) #you can also flip the rows and columns, this process is called transpose print('B is \n', B.T) # .T method for transpose print(B.T.shape) ## notice the shape now, transpose does not mean anything for a (2,) array
Matrix: math works using the rules of Linear Algebra
We will learn some basics of linear algebra later in this tutorial, but for now here is an example of this linear algebra behavior of matrices. Unusual results from mathematical operation can result.
For now, some concepts fro linear algebra are transpose and matrix multiplication.
In python to produce the transpose (or take the transpose) of an array, append the variable with a .T. The definition of transpose is to switch the rows and columns.
Matrix multiplication means a very special procedure for multiplying vectors, and matrices. We will learn the mean of matrix multiplication latter in this tutorial. For now, to matrix multiply 2 matricies use the @ operator.
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np A = np.array([[ 2, 2.], [-2. , 2]] ) ## create an array print(A) ## print the array print(A.T) ## print the transpose of the array print(A*A) ## print the element-wise multiplication of the matrix (array) print(A@A) ## perform matrix multiplication, not element-wise multiplication
Here are some examples of similar results using the np.matrix array type. We will not use this very much this semester.
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np A = np.matrix([[ 2, 2.], [-2. , 2]] ) ## create a matrix print(A*A) ## perform matrix multiplication, not element-wise multiplication B = np.matrix([2,2]) print('The shape of B is', B.shape) ## notice how the shape is (2,1) and not (2,) like an array #you can also flip the rows and columns, this process is called transpose print('B is \n', B.T) # .T method for transpose print(B.T.shape) ## notice the shape now # you can also convert between the array and the matrix data objects Am = np.matrix([[ 2, 2.], [-2. , 2]] ) print('Matrix type', type(Am)) Aa = np.array(Am) print('Now it is an array', type(Aa)) # the reverse is also true Aa = np.array([15.2, 16.7, 17.34, 14.6, 13.9]) print('Array data type', Aa) Am = np.matrix(Aa) print('Now it is a matrix', type(Am))
Some matrix specific operations are diag and trace. Here are some examples:
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np A = np.array([[ 2, 2.], [-2. , 2]] ) ## create an array print(A) print( np.diag(A) ) ## the diagonal of the array (matrix) is an array (vector) D = np.diag(A) print( np.sum(D) ) ## sum of the diagonal is the trace # or T = np.trace(A) print(T)
Assignment 8 Arrays (ass8)
create a function that will
- take as inputs 2 arrays of the same size
- element-wise add the two arrays together
- divide resulting sum-array by 2
- return the resulting array
Assignment 9 Matrix math (ass9)
create a function that will
- take as inputs 2 arrays of the same size
- matrix multiply them together
- then calculates the trace for product of the two arrays
- returns the trace of the matrix multiplication (or the sum of the diagonal)
Dataframes (the named arrays)
Dictionaries adfad
Dictionaries are another kind of information/data storage object in python. We will not use them very much in this tutorial. They are a good segue into the next section dataframes, so that is why we are covering them here.
Python dictionaries are just like dictionaries in the real world. You put in a word and you get out a definition.
A dictionary is defined by either the word dict() or enclosing it in {}. Here are some examples.
Just like a list you can store anything in a dictionary.
import numpy as np #this command loads all of the functions in numpy and labels them np ## generic example: # name = {key:value,key:value} hr = {'Aaron':47, 'Ruth':60, 'Bonds':73, 'McGwire':70, 'Sosa':66 } print( hr['Bonds'] ) ### a simplified spectroscopic example wavelengths = np.array([200, 250, 300, 350, 400]) spec = np.array([0, 0.2, 0.5, 0.3, 0.001]) d = {'x':wavelengths, 'y':spec} y = d['y'] print(y)
Dataframes
Python has another module called Pandas user guide. That was built to make certain tasks in python/numpy easier. In this module is defined a new data type object called a dataframe. You can think of these types as named arrays, or some kind of combination of array and dictionary.
Here is a continuation of the baseball example above:
import numpy as np #this command loads all of the functions in numpy and labels them np import pandas as pd ## this command loads the pandas module hr = {'Aaron':47, 'Ruth':60, 'Bonds':73, 'McGwire':70, 'Sosa':66 } df = pd.DataFrame.from_dict(hr,orient='index',columns=['HR']) print(df) #the dataframe presents itself as a table with the rows labeled (called the index) and the column(s) labeled print('') print('****************** spacer *********************') print('Average Homeruns \n',df.mean()) ## you can operate on the dataframe, NOTE: the output of these operations is another dataframe print('****************** spacer *********************') #to get the values out of the dataframe use the .values method hrdf = df.mean() print(hrdf) print(hrdf.values) ## output as an array
Here is another example that is a little more relevant to chemistry
import numpy as np #this command loads all of the functions in numpy and labels them np import pandas as pd ## this command loads the pandas module wavelengths = np.array([200, 250, 300, 350, 400]) ## wavelength labels for absorbance values spec1 = np.array([0, 0.2, 0.5, 0.3, 0.001]) ## absorbance values spec2 = np.array([0.2,0.5,0.1,0.0,0.0]) ## aborbance values d = {'spectrum 1':spec1, 'spectrum 2':spec2} ## make a dictionary df = pd.DataFrame.from_dict(d,orient='index',columns=wavelengths) ## make a dataframe print('Type for spec',type(spec1)) print('Type for d',type(d)) print('Type for df',type(df)) print('****************** spacer *********************') ## as a dataframe you can take the average spectrum print(df.mean()) print('****************** spacer *********************') ## or you can get the absorbacne values at a specific wavelength print(df[300])
CANCELED Activity Assignment 10 (ass10) (maybe move to loading data section)
Making Graphics and Figures
All discussion of spectroscopic (and most other) data is often accompanied by some kind of graphical representation of the data. This section will describe the fundamentals of graphical representation of data plus the graphical representation of the meaning of the data. \footnote{I don't actually know what this last clause means, but it sounds cool.}
It is my philosophy that all graphical data presentations must stand by themselves, without oral explanation. So when you hand a printout of data to your colleague or boss, you should not have to explain every detail of the graphic. Some obvious parts of the presentation such as axes labels, legends, and captions easily enhance the understanding of the graphic. Herein are a set of examples from which you can learn.
There are several types of data presentations, and the choice of which to use depends upon the type of data and the intended usage of the data presentation.
There are 6 basic types of data presentation graphs that are commonly used by chemists: line graph, scatter plot, histograms, contour plots, surface plots, and images. (I guess there is also a bar graph and pie graph, but I don't really know when to use these types.) There are many perturbations within each of these graph types including combinations of each within one presentation.
There are several intended uses for these data presentations, and these uses in part dictate how the figures are created.
- Computer-based inspection: For me this is the most useful, because I can interact with the data (zoom in on specific parts, inspect the peak max or min, overlay 2 spectra, etc.).
- Paper-based lab notebook: This printed version does not require super high resolution; I will refer to this type of presentation as a thumbnail print.
- Publication Quality: Related to the thumbnail print is the publication quality print, which is a higher resolution file size and may also have additional annotations and figure captions.
- Beamer Presentation: Also related to the printed version but is for inclusion into a beamer presentation or image projection system for giving talks, which usually don't require a high resolution graphic format or interactivity.
All of these can be created with python, numpy, matplotlib, pandas, and scipy. Additionally, these data presentations may need to be annotated to enhance the understanding of the information presented.
In the following subsections are examples of these different types of graphics and their uses.
General Philosophy for what type of graph to use
In the introduction above were listed 6 types of graphs: line graph, scatter plot, histograms, contour plots, surface plots, and images. So a question might arise in your mind, "Which type do I use?" There have been many discussions over the years about this question. Here are a few links:
- https://www.edwardtufte.com/tufte/
- https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1003833
- http://mirrors.ctan.org/graphics/pgf/base/doc/pgfmanual.pdf
This is an unsettled issue that is full of opinions. Herein I will present my opinions as well as some rules that are settled by conventions within the field of chemistry (mostly IUPAC). I will delineate between my opinions and conventions, by stating when something is my opinion. All other statements should be assumed to be based upon best practices (conventions).
These next 6 subsections are all my opinion, but I believe that lots of scientists would agree with them.
When to use a scatter plot
Whenever you need to fit a smallish set of data with a mathematical function, it is better to plot the data with markers, and the fit with a line. My reasons are that this formula produces a clear distinction between the actual data and the mathematical function. You don't want the viewers of you data presentation to think that you have more data than you do.
Some examples are shown in the Figure below.
Figure 11: Example scatter plots with the data plotted with markers and the fit with a line
When to use a line plot
Whenever you have loads of data points that are follow a predictable trajectory, like in a spectrum or chromatorgram, using markers will make the line so thick that you might inadvertently hide the curvature. In these situations, I believe you can use a smoothly curving line to present the spectrum, chromatorgram, etc. Some examples are shown in the figure below.
Figure 12: Example line plots with lots of points
When to use histograms
Whenever you have 1D data that is from a set of repeated measurements, and your goal is to represent the spread (how repeatable) in the data, then a good choice is a histogram. An example, might be if you were monitoring the arsenic concentration level in bunch of samples.
Figure 13: Example of a histogram of repeated measurements
When to use contour and surface plots and images
In chemistry data, these 3 types of plots can be used interchangeably. One exception is when presenting data that is in fact an image (e.g. photograph, SEM image, etc.). The types of data that utilize these types of graphs fall in to 2 categories:
- correlated/anticorrelated data
These types of data are commonly found in 2D correlation spectroscopy. Some of you may have learned about 2D NMR measurements such as COSY (link ) and HETCOR. These data are often presented as contour plots. These NMR measurements are a subset of experiments that are part of the generalized 2D spectroscopy type of measurements. Here is a very good article link.
In all of these cases, the X and Y dimensions are some kind of spectrum (X and Y can be the same or different spectra or even spectral types), and the in the middle are the relative heights of the correlations between the two (X and Y) spectra portrayed as contours.
Here is an example of a contour plot of the correlation of UV spectra during a chemical reaction.
Figure 14: Example of generalized 2D spectroscopy
This same kind of data could also be displayed as a 3D surface. In fact a contour plot is just a flattened surface plot. Here are the same data shown in 3 different graphs. Notice how the contour graph and image really need a color bar to the right so that the reader can know what the colors represent.
Figure 15: Examples of a contour, surface plot and an image of the same data
- data in which multiple two axes are the same kind of data (spatial) and the 3rd is a different kind of data (concentration, spectra band height, etc.)
You have seen this kind of data before in the following figure.
Figure 16: Finger print, DNA, RDX
Generic Example
Each of the examples within this section of the tutorial will contain several parts. Not all of the parts will be used in every example, but the part is listed to keep the examples consistent. Here is a generic list of the parts:
1. import modules (you only need this part once per jupyter notebook) 2 get or make the data 3. process the data 4. plot the data 5. plot the data 6. annotate the plot 7. save the graphic in the required format 8. house keeping (this is part of the tutorial but not for you to copy)
plotting useless data
Let us start by just learning to plot using made-up, simulated data. There are 3 kinds of graphs that are common in chemistry:
- a continuous graph of data without markers
- an xy scatter graph with markers
- a histogram of data from repeated measurements with some spread in the data
simulated spectrum
For spectra, chromatograms, and other continuous data, it is better to plot the data with a line without markers. The main reason is that the markers would be so close together that they would interfere with the viewer's perspective of the shape of the curve. The default for the matplotlib library is to add space to either sided of these types of graphs. Personally, I don't like this. For now we will leave then there, and learn how to eliminate them latter in this tutorial.
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions # you only need to load the modules once per file import numpy as np #this command loads all of the functions in numpy and labels them np import pandas as pd # data organization module import matplotlib.pyplot as plt # plotting module ## get/make the data x = np.arange(200,400) # these will be the x values in the graph for a simulated UV spectrum xc = 250 # this is the center of the band sigma = 10 # %sigma*2*sqrt(2*ln(2)) is the width at half height of the lambda max of the curve spec = np.exp(-1*((x-xc)**2)/(2*sigma**2)) ##this will make a Gaussian curve (idea shape of lots of data) ## process data ## plot data fig, ax = plt.subplots(figsize=(5,4)) ## figsize is the aspect ratio of the figure ## ax is the variable for the axis object where you want to work ax.plot(x,spec) ## .plot(x,y) method to make a plot on the ax object ax.set_xlim((200,400)) ## set x dim edges of the plot ax.set_xlabel('Wavelength /nm') ## LaTeXworks here ax.set_ylabel('Absorbance') ## and here ofile = 'figures/examplespec.png' ## name of the output file fig.savefig(ofile,dpi=100) ## dpi is dots per inch and is a figure resolution thing ### This part is for making this website, so you don't need it return ofile
simulated Beer's law data
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np import pandas as pd # data organization module import matplotlib.pyplot as plt # plotting module ## get data C = [1.0, 2.0, 3.0, 4.0, 5.0] #these will be the x values in the graph they are simulating concentration in ppm A = [0.1, 0.2, 0.3, 0.4, 0.5] #this is the simulated Aborbance values ## process data ## plot data fig, ax = plt.subplots(figsize=(5,4)) ## figsize is the aspect ratio of the figure ax.scatter(C,A, s=55,marker='.') ## ax is the variable for the axis where you want to work ## s is the size of the marker ## marker is the marker some choices: ('.','o', 'v', '^', '<', '>', '8', 's', 'p', '*', 'h', 'H', 'D', 'd', 'P', 'X') ax.set_xlabel('Concentration /ppm') ## latex works here ax.set_ylabel('Absorbance') ## and here ofile = 'figures/examplescatter.png' fig.savefig(ofile,dpi=100) ## dpi is dots per inch and is a figure resotution thing ### This part is for making this website, so you don't need it return ofile
Simulated Mass Data
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np import pandas as pd # data organization module import matplotlib.pyplot as plt # plotting module ## get data mass = 10+np.random.randn(200)*0.29 ## these will be 200 simulated mass measurements normally distributed around 10 g ## process data ## plot data fig, ax = plt.subplots(figsize=(5,4)) ## figsize is the aspect ratio of the figure ax.hist(mass) ## ax is the variable for the axis where you want to work ax.set_xlabel('Mass /g') ## latex works here ax.set_ylabel('Count') ofile = 'figures/examplehist.png' ## the output filename fig.savefig(ofile,dpi=100) ## dpi is dots per inch and is a figure resolution thing ### This part is for making this website, so you don't need it return ofile
First some data
This part of computer programming is arguably the most difficult to learn for new programmers. Some people call it data wrangling. These words mean getting the data from your instrument's output into the memory of python for processing. Because of this difficulty, in this course we will not deal with it. If you need help getting your data from your research into python for processing ask me, and we can do it outside of class. I am happy to assist you. I am making this teaching choice, because every instrument has its own kind of data file format, and to cover them all would take the entire semester.
So for this class, I will issue to you data in 3 kinds of formats:
- One will be single datafiles *.csv or *.dat with or without headers stored as text files. Some examples of types of data:
- chromatograms
- spectra
- concentration data
- A relative of the *.csv file is the MS Excel files.
It is pretty common for data/results to be stored in an Excel file. I believe this is because it is pretty easy to type data into spreadsheet programs such as MS Excel. I also believe that is why they are so popular. So we will learn how to read data from these file.
- The other type will be datasets in either *.mat files or *.hdf5 files. Some examples:
- hyphenated technique data (GC-MS, Raman and Infrared spectroscopic imaging, 2D NMR data)
- lots of similar data (100's of IR or UV/Vis spectra)
- This type of file is a kind of archive or bundle in which the data has been organized for you
- we will see some examples below
I will assume that all data is stored in a directory under your working directory named data. So all data file paths will look like
filename = 'data/somedatafilename.dat' # on mac or *nix filename = 'data\fomedatafilename.dat' # on a windows computer
loading *.csv or *.dat files with no header and plot it
These will be files that contain 2 or more columns of data separated by a comma or tab or space. Traditionally, the first column is the independent values (you plot them on the x axis), and the 2nd, 3rd, etc. column(s) is(are) the dependent values (you plot them on the y-axis).
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np import pandas as pd # data organization module import matplotlib.pyplot as plt # plotting module ## get data ifile = 'data/sample1.csv' ## the name and relative path of the data file rawdata = np.genfromtxt(ifile,delimiter=",") ## read the file output into an array print('the shape of the d is ',rawdata.shape) ## inspect the data size x = rawdata[:,0] ## extract/slice the wavenumbers (x values) spec = rawdata[:,1] ## extract/slice the absorbance values (y-values) ## process data ## plot data fig, ax = plt.subplots(figsize=(5,4)) ## figsize is the aspect ratio of the figure ax.plot(x,spec) ## ax is the variable for the axis where you want to work ax.set_xlim((650,4000)) ## set graph's limits ax.invert_xaxis() ## vibrational spectra need this ax.set_xlabel('Wavenumber /cm$^{-1}$') ## latex works here ax.set_ylabel('Absorbance') ## latex works here fig.tight_layout() ## removes a bunch of white space around figure ofile = 'figures/dataspec.png' fig.savefig(ofile,dpi=100) ## dpi is dots per inch and is a figure resotution thing ### This part is for making this website, so you don't need it return ofile
Loading *.mat files
This file format is matlab's high performance data format. It is based upon the HDF5 format. http://www.hdfgroup.org/HDF5/
This is the faster and easier of the file formats whenever you have lots of data to read into your program. The data (spectra, chromatograms) are already stored in the variable inside of the .mat file. To use them, load this data. Loading Matlab files requires a new module (Scientific Python or *scipy)
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np import pandas as pd # data organization module import matplotlib.pyplot as plt # plotting module import scipy.io as sio #open matlab files module ## get data ifile = 'data/tolpxyl.mat' ## the path and name of the datafile mat_contents = sio.loadmat(ifile) ## read the file output into a dictionary print('The contents of the mat file is ',mat_contents.keys()) ##print the names of the contents of the matfile Aorg = mat_contents['Aorg'] ## retrieve the Spectra from the dictionary xorg = mat_contents['xorg'] ## retrieve the x values from the dictionary print('the shape of the Aorg is ',Aorg.shape) ## inspect the variable's size x = xorg.T ## xorg is stored funny in the mat file so we transpose it A = Aorg print('x and A shapes are ',x.shape,A.shape) ## process data
Reading data out of Excel files
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np import pandas as pd # data organization module import matplotlib.pyplot as plt # plotting module import scipy.io as sio #open matlab files module ## get data ifile = 'data/glass_elemental_conc.xlsx' ## the path and name of the datafile df = pd.read_excel(ifile,index_col=0) print('df \n',df) ## process data
Loading Multiple Spectra and their Graphs
This idea means that you have to load more than one spectrum or chromatogram into memory for plotting. In this section we will first learn several techniques of loading multiple spectra, and then later we will plot them in several techniques. There are 3 basic strategies that can be employed to load and graph multiple spectra.
- manually, if you just have a few spectra
- with a loop if you have lots of spectra or
- just load a *.mat file that I or someone else prepared for you.
The strategy that is used is somewhat subjective; meaning that we get to make the choice. I usually choose based upon my level of laziness at the moment.
For the demonstration of the different data loading strategies we will employee the spectral overlay graphing technique without any discussion of it. We will discuss the graphing a little later.
Multiple spectra: spectral overlays with just a few files loaded manually
There are two ways to do this. One is in x, y pairs.
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np import pandas as pd # data organization module import matplotlib.pyplot as plt # plotting module ## get data ## one datafile ifileA = 'data/sample1.csv' rawdataA = np.genfromtxt(ifileA,delimiter=",") ## read the file output into an array print('the shape of the d is ',rawdataA.shape) xA = rawdataA[:,0] ## extract the wavenumbers specA = rawdataA[:,1] ## extract the absorbance values ## second datafile ifileB = 'data/sample2.csv' rawdataB = np.genfromtxt(ifileB,delimiter=",") ## read the file output into an array print('the shape of the d is ',rawdataB.shape) xB = rawdataB[:,0] ## extract the wavenumbers specB = rawdataB[:,1] ## extract the absorbance values ## process data ## plot data fig, ax = plt.subplots(figsize=(12,9)) ## figsize is the aspect ratio of the figure ax.plot(xA,specA,label='spec 1') ## ax is the variable for the axis where you want to work ax.plot(xB,specB,label='spec 2') # do it again to plot the second spectrum ax.set_xlim((650,4000)) ## set graph's limits ax.invert_xaxis() ## vibrational spectra need this ax.legend() ## call the legond for the axis to show it ax.set_xlabel('Wavenumber /cm$^{-1}$') ## latex works here ax.set_ylabel('Absorbance') ## latex works here ofile = 'figures/dataspecoverlay2.png' fig.savefig(ofile,dpi=100) ## dpi is dots per inch and is a figure resotution thing ### This part is for making this website, so you don't need it return ofile
Multiple spectra: loaded with a for loop (Just load and plot the data)
Notice in the code below that the data is not stored, but simply plotted. This might be useful, but does not allow the user to do anything else with all of these spectra.
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np import pandas as pd # data organization module import matplotlib.pyplot as plt # plotting module ## define some custom functions def dataloader(ifile): rawdata = np.genfromtxt(ifile,delimiter=",") ## read the file output into an array x = rawdata[:,0] ## extract/slice the wavenumbers (x values) spec = rawdata[:,1] return x,spec ## get data file list filelist = ['specfile1.csv', 'specfile2.csv', 'specfile3.csv', 'specfile4.csv', 'specfile5.csv'] ## list of files datahole = 'data/' ## load and plot data fig, ax = plt.subplots(figsize=(12,9)) ## figsize is the aspect ratio of the figure for ifile in filelist: ## loop through filelist name = ifile[:-4] ## create a legion name from file list x,spec = dataloader(datahole+ifile) ## use you new function to load the data;notice the concatantion of filename and path ax.plot(x,spec,label=name) ## plot each spectrum; add the label ax.set_xlim((650,4000)) ## set graph's limits ax.invert_xaxis() ## vibrational spectra need this ax.legend() ## call the legond for the axis to show it ax.set_xlabel('Wavenumber /cm$^{-1}$') ## latex works here ax.set_ylabel('Absorbance') ## latex works here ofile = 'figures/dataspecoverlayforloop.png' fig.savefig(ofile,dpi=100) ## dpi is dots per inch and is a figure resotution thing ### This part is for making this website, so you don't need it return ofile
Multiple spectra: loaded with a for loop (Load, store and plot the data)
There are two ways to load the data, store it in an array-like object and plot. One uses an array, and the other uses either a dictionary or dataframe.
for loop: array data storage
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np import pandas as pd # data organization module import matplotlib.pyplot as plt # plotting module ## define some custom functions def dataloader(ifile): rawdata = np.genfromtxt(ifile,delimiter=",") ## read the file output into an array x = rawdata[:,0] ## extract/slice the wavenumbers (x values) spec = rawdata[:,1] return x,spec ## get data file list filelist = ['specfile1.csv', 'specfile2.csv', 'specfile3.csv', 'specfile4.csv', 'specfile5.csv'] ## list of files datahole = 'data/' ## load and plot data fig, ax = plt.subplots(figsize=(12,9)) ## figsize is the aspect ratio of the figure A = np.zeros((5,3351)) ## pre populate the array with zeros, need to know the size of the spectra first for n,ifile in enumerate(filelist): ## loop through filelist; enumerate to create a counter name = ifile[:-4] ## create a legion name from file list x,spec = dataloader(datahole+ifile) ## use you new function to load the data;notice the concatantion of filename and path ax.plot(x,spec,label=name) ## plot each spectrum; add the label A[n,:] = spec ## store each spectrum in an array as they are loaded in the for loop ax.set_xlim((650,4000)) ## set graph's limits ax.invert_xaxis() ## vibrational spectra need this ax.legend() ## call the legond for the axis to show it ax.set_xlabel('Wavenumber /cm$^{-1}$') ## latex works here ax.set_ylabel('Absorbance') ## latex works here ofile = 'figures/dataspecoverlayforlooparray.png' fig.savefig(ofile,dpi=100) ## dpi is dots per inch and is a figure resotution thing ### This part is for making this website, so you don't need it return ofile
for loop: dictionary / dataframe data storage load file
In this technique the loading data and plotting are separated (2 for loops). They could be done together (1 for loop). This (2 for loop method) was chosen because it is easier to understand what is happening.
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np import pandas as pd # data organization module import matplotlib.pyplot as plt # plotting module ## define some custom functions def dataloader(ifile): rawdata = np.genfromtxt(ifile,delimiter=",") ## read the file output into an array x = rawdata[:,0] ## extract/slice the wavenumbers (x values) spec = rawdata[:,1] return x,spec ## multiple files from an excel file datahole = 'data/' filelist = ['specfile1.csv', 'specfile2.csv', 'specfile3.csv', 'specfile4.csv', 'specfile5.csv'] ## load the data datadict = {} ## create an empty dictionary to store the data for ifile in filelist: ## for loop through the filelist x,spec = dataloader(datahole+ifile) key = ifile[:-4] ## get dictionary key from ifile datadict[key] = spec ## store spectrum in dictiorary df = pd.DataFrame.from_dict(datadict,orient='index',columns=x) ## convert dictionary to dataframe fig, ax = plt.subplots(figsize=(12,9)) ## figsize is the aspect ratio of the figure for key in df.index: ## loop through dataframe one index at a time x = df.columns ## extract the xvalues from the columns of the dataframe sp = df[df.index == key].values.T ## get the spectrum that correponds to the indexed spectrum (key) ax.plot(x,sp,label=key) ##plot that spectrum ### alternatively, you could skip the second for loop and just out of the df #df.T.plot(ax=ax) ## plot the dataframe with index as labels on the axis ax ax.set_xlim((650,4000)) ## set graph's limits ax.invert_xaxis() ## vibrational spectra need this ax.legend() ## call the legond for the axis to show it ax.set_xlabel('Wavenumber /cm$^{-1}$') ## latex works here ax.set_ylabel('Absorbance') ## latex works here ofile = 'figures/dataspecoverlayforloopdict.png' fig.savefig(ofile,dpi=100) ## dpi is dots per inch and is a figure resotution thing ### This part is for making this website, so you don't need it return ofile
for loop: dictionary / dataframe data storage load file information from Excel
In this technique the loading data and plotting are separated (2 for loops). They could be done together (1 for loop). This (2 for loop method) was chosen because it is easier to understand what is happening. We will also get the file information from an MS Excel file. This is very sophisticated and we will not rely on this for our homework.
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np import pandas as pd # data organization module import matplotlib.pyplot as plt # plotting module ## define some custom functions def dataloader(ifile): rawdata = np.genfromtxt(ifile,delimiter=",") ## read the file output into an array x = rawdata[:,0] ## extract/slice the wavenumbers (x values) spec = rawdata[:,1] return x,spec ## multiple files from an excel file datahole = 'data/' mdf = pd.read_excel('data/metadatafile.xlsx',index_col=0) ## read file info from Excel meta data file #mdf ## load the data datadict = {} ## create an empty dictionary to store the data for ifile in mdf.index: ## for loop through the mdf.index x,spec = dataloader(datahole+ifile) ## load data key = ifile[:-4] ## extract key and legend name from the mdf datadict[ifile] = spec ## store the spectrum in the dictionary under the key # the indices of the dataframe are the keys in the dicitonary (or part of the filenames) # the columns of the df are the wavenumbers of the spectra ## all spectra must be the same size for this to work df = pd.DataFrame.from_dict(datadict,orient='index',columns=x) ## convert the dictionary to a dataframe # plot the data fig, ax = plt.subplots(figsize=(12,9)) ## figsize is the aspect ratio of the figure for key in df.index: ## for loop through each index x = df.columns ## if you already have the x-values in memory then you don't need to do this step sp = df.loc[df.index == key].values.T # get the spectra out one index at a time name = mdf.loc[mdf.index == key].values[0][0] ## get the legend name from mdf ax.plot(x,sp,label=name) ## plot each spectrum labeled with its name ax.legend() ####### or you could do it more directly with the dataframe leglist = mdf.key.values.tolist() ## extract the legend labels from the mdf fig,ax = plt.subplots(figsize= (12,4)) ## create an axis and figure df.T.plot(ax=ax) ## plot the dataframe with index as labels ax.legend(leglist) ## call the legond for the axis to show it ax.set_xlim((650,4000)) ## set graph's limits ax.invert_xaxis() ## vibrational spectra need this #ax.legend() ## call the legond for the axis to show it ax.set_xlabel('Wavenumber /cm$^{-1}$') ## latex works here ax.set_ylabel('Absorbance') ## latex works here ofile = 'figures/dataspecoverlayforloopdict_fromexcel.png' fig.savefig(ofile,dpi=100) ## dpi is dots per inch and is a figure resotution thing ### This part is for making this website, so you don't need it return ofile
Multiple Spectra: from a pre-made dataset file (HDF or mat)
When you have an inconvenient number of spectra to load manually, a good choice is to load a dataset. Two good choices of dataset file formats are hdf5 files and Matlab files (which are actually just hdf5 files).
In this case, these datasets need to be made either by the computer software attached to the instrument that acquired the data or separately in another program. In the latter case, this process is often a custom job, so we will not show it here, but involves looping through every spectrum, storing them in an array, and then saving the arrays as a file. All of the spectra must be the same size for this to work.
Often these type of data don't get a legend, because the number of spectra would make the legend size ridiculous. This number of spectra to plot is a pretty good delimiter for these two types of data.
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np import pandas as pd # data organization module import matplotlib.pyplot as plt # plotting module import scipy.io as sio #module for opening matlab files ## get data ifile = 'data/tolpxyl.mat' mat_contents = sio.loadmat(ifile) ## read the file output into a dictionary print(mat_contents.keys()) ##print the names of the contents of the matfile Aorg = mat_contents['Aorg'] ## retrieve the Spectra xorg = mat_contents['xorg'] ## retrieve the x values print('the shape of the Aorg is ',Aorg.shape) x = xorg.T A = Aorg ## process data ## plot data fig, ax = plt.subplots(figsize=(9,6)) ## figsize is the aspect ratio of the figure ax.plot(x,A.T) ## ax is the variable for the axis where you want to work ax.set_xlim((200,340)) ## set graph's limits #ax.invert_xaxis() ## vibrational spectra need this ax.set_xlabel('Wavelength /nm') ## latex works here ax.set_ylabel('Absorbance') ## latex works here ofile = 'figures/multidataspecoverlay.png' fig.savefig(ofile,dpi=100) ## dpi is dots per inch and is a figure resotution thing ### This part is for making this website, so you don't need it return ofile
- ASS10 Assignment 10 loading some data (no plotting of data required)
NOTE: for this assignment all of the files will be in the ass10 folder not in the ass10/data folder to prevent conflict between Microsoft Windows and MacOS
- Part 1
Create a function that will
- load a single data file (xy columns) from a path + filename input
- return the x values and y values as 2 separate 1D arrays
- Assumed that the data files will be txt files of 2 columns of data each separated by comma (a CSV format)
- HINT: we did this in class
- Part 2
Create a function that will use the function in ass10 Part 1 to
- load a set of data files (use the function from part 1 here) from a list of file names as the input
- store the x-values into a 1D np.array
- store the y-values into an 2D np.array
- the number of rows should be the number of files that you load
- the number of columns should be the number of y-values per file
- return the x-values as a 1D array and the y-values a 2D array
- Part 1
Spectra plotting (overlayed)
This method works well if you have spectra that are either the same size as each other or different sizes (number of wavelengths). One limitation is that you overwrite the spectral data as you step through the loop. So as this code stands you can not process the data beyond plotting it.
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np import pandas as pd # data organization module import matplotlib.pyplot as plt # plotting module ## get data def dataloader(ifile): import numpy as np rawdata = np.genfromtxt(ifile, delimiter=',') ## read the file output into an array x = rawdata[:,0] ## extract the wavenumbers spec = rawdata[:,1] ## extract the absorbance values return x, spec ## return the wavenumbers and absorbance values ifile1 = 'sample2.csv' ## datafile named stored in variable ifile2 = 'specfile1.csv' ## datafile named stored in variable datahole = 'data/' #location of datafiles (path) x,spec1 = dataloader(datahole + ifile1) ## load data file notice the concatenation of the path and datfile x,spec2 = dataloader(datahole + ifile2) fig, ax = plt.subplots(figsize=(9,6)) ## create a figure and axis to plot onto ax.plot(x,spec1,label='spec 1') ## plot one file with label ax.plot(x,spec2 ,label='spec 2') ## plot another file with label ax.set_xlim((700,2000)) ## change the view of the graph ax.invert_xaxis() ##important for IR ax.legend() ## show the legend ax.set_ylabel('Absorbance') ## set the y axis label ax.set_xlabel('Wavenumber /cm$^{-1}$') ## set the x axis label ax.grid() ## turn on a basic grid to help see which bands lineup # put a vertical line on the ax axis # linestyles [‘solid’ | ‘dashed’, ‘dashdot’, ‘dotted’ | (offset, on-off-dash-seq) | '-' | '--' | '-.' | ':' | 'None' | ' ' | ''] # alpha is transparency 0 - 1 ax.axvline(x=837,linestyle='--',color='g',alpha=0.5) ofile = 'figures/dataspecoverlay.png' fig.savefig(ofile,dpi=100) ## dpi is dots per inch and is a figure resotution thing ### This part is for making this website, so you don't need it return ofile
spectra plotting (stacked)
Sometimes overlaying spectra can confuse the viewer of the data. So another way to display the data is to stack them.
Whenever you stack spectra, you need to keep somethings in mind. One, the y axis is no longer accurate, so you need to indicate this to the viewer. A customary way to do this is to add a vertical bar on the graph that indicates a specific δ y distance.
Since the data are not overlayed it is difficult for the viewer to determine if spectral features overlap. So to indicate the x position of a particular band or peak, you will need to annotate the graph. There are several ways to do this:
- add grid lines (this works well if you are printing the graph, and there are lots of bands that you want the viewer to compare)
- add a band label with position and/or height
- add a vertical line centered on just a few of the bands
Look in the tricks below to find the details of all of these.
Here is a basic example of a stacked plot. A constant is added to one of the y vector to move it up.
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np import pandas as pd # data organization module import matplotlib.pyplot as plt # plotting module ## get data def dataloader(ifile): import numpy as np rawdata = np.genfromtxt(ifile, delimiter=',') ## read the file output into an array x = rawdata[:,0] ## extract the wavenumbers spec = rawdata[:,1] ## extract the absorbance values return x, spec ## return the wavenumbers and absorbance values ifile1 = 'sample2.csv' ## datafile named stored in variable ifile2 = 'specfile1.csv' ## datafile named stored in variable datahole = 'data/' #location of datafiles (path) x,spec1 = dataloader(datahole + ifile1) ## load data file notice the concatenation of the path and datfile x,spec2 = dataloader(datahole + ifile2) offset = 0.2 fig, ax = plt.subplots(figsize=(9,6)) ## create a figure and axis to plot onto ax.plot(x,spec1,label='spec 1') ## plot one file with label ax.plot(x,spec2 + offset ,label='spec 2') ## plot another file with label notice the offset ax.set_xlim((700,2000)) ## change the view of the graph ax.invert_xaxis() ##important for IR ax.legend() ## show the legend ax.set_ylabel('Absorbance') ## set the y axis label ax.set_xlabel('Wavenumber /cm$^{-1}$') ## set the x axis label ax.grid() ## turn on a basic grid to help see which bands lineup ax.set_yticklabels('') # absorbance is no longer correct so turn it off # add a bar that indicates the absorbance scale for the two spectra ax.annotate('0.1 absorbance',(1800,0.20),(1800,0.3),ha='center',arrowprops={'arrowstyle':'|-|'}) # put a vertical line on the ax axis # linestyles [‘solid’ | ‘dashed’, ‘dashdot’, ‘dotted’ | (offset, on-off-dash-seq) | '-' | '--' | '-.' | ':' | 'None' | ' ' | ''] # alpha is transparency 0 - 1 ax.axvline(x=837,linestyle='--',color='g',alpha=0.5) ofile = 'figures/dataspecstack.png' fig.savefig(ofile,dpi=100) ## dpi is dots per inch and is a figure resotution thing ### This part is for making this website, so you don't need it return ofile
scatter plot of linear data
Whenever you have data with limited length (number of points < 50), it is probably better to graph it as a scatter plot than a line plot. If you have only one data set this is pretty easy.
Only one data set
Just hard code into the program the data.
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np import pandas as pd # data organization module import matplotlib.pyplot as plt # plotting module ## get data C = [1, 2, 3, 4, 5] # these will be the x values in the graph they are simulating concentration in ppm A = [0.1, 0.2, 0.3, 0.4, 0.5] #this is the simulated Aborbance values ## process data ## plot data fig, ax = plt.subplots(figsize=(5,4)) ## figsize is the aspect ratio of the figure ax.scatter(C,A) ## ax is the variable for the axis where you want to work ax.set_xlabel('Concentration /ppm') ## latex works here ax.set_ylabel('Absorbance') ofile = 'figures/examplescatter.png' fig.savefig(ofile,dpi=100) ## dpi is dots per inch and is a figure resotution thing ### This part is for making this website, so you don't need it return ofile
Multiple sets of data
If the number of sets of data are relatively small, the you can also hard code them into the code. If you have lots of sets of data, then probably a for loop might be useful.
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np import pandas as pd # data organization module import matplotlib.pyplot as plt # plotting module ## get data toluene and xylone C1 = [1, 2, 3, 4, 5] # these will be the x values in the graph they are simulating concentration in ppm A1 = [0.1, 0.2, 0.3, 0.4, 0.5] #this is the simulated Aborbance values C2 = [0.98, 2.10, 3.05, 4.09, 5.01] # toluene A2 = [0.11, 0.25, 0.39, 0.51, 0.691] # toluene ## process data ## plot data fig, ax = plt.subplots(figsize=(5,4)) ## figsize is the aspect ratio of the figure ax.scatter(C1,A1,label='p-Xylene') ## ax is the variable for the axis where you want to work ax.scatter(C2,A2,label='toluene') ax.legend() ## turn the legend on ax.set_xlabel('Concentration of analyte in water /ppm') ## latex works here ax.set_ylabel('Absorbance at 252 nm') ofile = 'figures/examplescattermulti.png' fig.savefig(ofile,dpi=100) ## dpi is dots per inch and is a figure resotution thing ### This part is for making this website, so you don't need it return ofile
histogram of data
Here are some final exam grades of mine. Notice that the grades are also hard coded. This is a nice way to do it, because the data and the code travel together, which makes the execution of the code very reproducible.
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np import pandas as pd # data organization module import matplotlib.pyplot as plt # plotting module ## get data # here are some final grades of mine g = [59.87, 65.35, 53.56, 35.73, 46.15, 59.05, 43.27, 67.27, 50.27, 46.70, 43.41, 62.61, 50.27, 66.73, 61.79, 44.78, 43.41, 50.82, 48.07, 49.17, 49.72, 57.13, 44.51, 47.53, 49.17, 70.02, 61.24, 48.35, 18.73, 62.61, 40.94, 56.85, 43.14, 55.75, 62.61, 55.48, 46.15, 43.41, 54.11] ## process data ## plot data fig, ax = plt.subplots(figsize=(5,4)) ## figsize is the aspect ratio of the figure ax.hist(g) ax.set_xlabel('Grades') ## latex works here ax.set_ylabel('Count') ofile = 'figures/examplehistgrades.png' fig.savefig(ofile,dpi=100) ## dpi is dots per inch and is a figure resotution thing ### This part is for making this website, so you don't need it return ofile
Tricks that can be applied to most or all of the Former Graphs
Computer Screen Inspection of Data
This is what you get as the default.
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np import pandas as pd # data organization module import matplotlib.pyplot as plt # plotting module ## get data x = np.arange(200,400) # these will be the x values in the graph they are simulating a UV spectrum xc = 250 # this is the center of the band sigma = 10 # %sigma*2*sqrt(2*ln(2)) is the width at half height of the Y max of the curve spec = np.exp(-1*((x-xc)**2)/(2*sigma**2)) ##this will make a Gaussian curve ## process data ## plot data fig, ax = plt.subplots(figsize=(12,9)) ## figsize is the aspect ratio of the figure ax.plot(x,spec) ## ax is the variable for the axis where you want to work ax.set_xlabel('Wavelength /nm') ## latex works here ax.set_ylabel('Absorbance') ofile = 'figures/examplespecscreeninpection.png' fig.savefig(ofile,dpi=100) ## dpi is dots per inch and is a figure resotution thing ### This part is for making this website, so you don't need it return ofile
Publication Quality Printout of Data
This is basically controlled by the output method that you choose. In the fig.savefig command the keyword dpi stands for dots per inch. The bigger the dpi number the higher resolution. If you pick a raster graphic file (tiff,jpg,png) the bigger the number after "dpi=" the more pixels per inch in the final file. If you pick a vector graphics type (eps,svg, ps) then the resolution parameter is useless. Publication quality figures usually have a dpi of 600 to 1200. The publisher will tell you what they want.
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np import pandas as pd # data organization module import matplotlib.pyplot as plt # plotting module ## get data x = np.arange(200,400) # these will be the x values in the graph they are simulating a UV spectrum xc = 250 # this is the center of the band sigma = 10 # %sigma*2*sqrt(2*ln(2)) is the width at half height of the Y max of the curve spec = np.exp(-1*((x-xc)**2)/(2*sigma**2)) ##this will make a Gaussian curve ## process data ## plot data fig, ax = plt.subplots(figsize=(12,9)) ## figsize is the aspect ratio of the figure ax.plot(x,spec) ## ax is the variable for the axis where you want to work ax.set_xlabel('Wavelength /nm') ## latex works here ax.set_ylabel('Absorbance') ofile = 'figures/examplespecpub.png' fig.savefig(ofile,dpi=300) ## dpi is dots per inch and is a figure resotution thing ### This part is for making this website, so you don't need it return ofile
Thumbnail or Lab Notebook Printout of Data
Since this figure does not need to be as high quality, you can simply change the resolution parameter dpi from 300 to 75.
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np import pandas as pd # data organization module import matplotlib.pyplot as plt # plotting module ## get data x = np.arange(200,400) # these will be the x values in the graph they are simulating a UV spectrum xc = 250 # this is the center of the band sigma = 10 # %sigma*2*sqrt(2*ln(2)) is the width at half height of the Y max of the curve spec = np.exp(-1*((x-xc)**2)/(2*sigma**2)) ##this will make a Gaussian curve ## process data ## plot data fig, ax = plt.subplots(figsize=(12,9)) ## figsize is the aspect ratio of the figure ax.plot(x,spec) ## ax is the variable for the axis where you want to work ax.set_xlabel('Wavelength /nm') ## latex works here ax.set_ylabel('Absorbance') ofile = 'figures/examplespecthumb.png' fig.savefig(ofile,dpi=75) ## dpi is dots per inch and is a figure resotution thing ### This part is for making this website, so you don't need it return ofile
Beamer Presentation of Data
When you are projecting data for a conference or seminar, the projectors are the limiting factor in data display. Since this figure does not need to be as high quality, you can simply change the resolution parameter dpi from 300 to 150-100.
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np import pandas as pd # data organization module import matplotlib.pyplot as plt # plotting module ## get data x = np.arange(200,400) # these will be the x values in the graph they are simulating a UV spectrum xc = 250 # this is the center of the band sigma = 10 # %sigma*2*sqrt(2*ln(2)) is the width at half height of the Y max of the curve spec = np.exp(-1*((x-xc)**2)/(2*sigma**2)) ##this will make a Gaussian curve ## process data ## plot data fig, ax = plt.subplots(figsize=(12,9)) ## figsize is the aspect ratio of the figure ax.plot(x,spec) ## ax is the variable for the axis where you want to work ax.set_xlabel('Wavelength /nm') ## latex works here ax.set_ylabel('Absorbance') ofile = 'figures/examplespecbeamer.png' fig.savefig(ofile,dpi=150) ## dpi is dots per inch and is a figure resotution thing ### This part is for making this website, so you don't need it return ofile
Reversing the x-axis for Vibrational Spectroscopic Data
According the IUPAC standards (PAC, 2007 pg 354) for spectra, higher energy should be plotted on the left side of a spectra. For vibrational spectra with wavenumber axis tick labels running between 400 and 4000 cm-1 (low to high), traditional graphics programs plot the spectra backwards. Python provides a simple means of reversing the x-axis direction (or y-axis if need be). Here is an example:
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np import pandas as pd # data organization module import matplotlib.pyplot as plt # plotting module ## get data x = np.arange(1600,1800) # these will be the x values in the graph they are simulating a UV spectrum xc = 1656 # this is the center of the band sigma = 50 # %sigma*2*sqrt(2*ln(2)) is the width at half height of the Y max of the curve spec = np.exp(-1*((x-xc)**2)/(2*sigma**2)) ##this will make a Gaussian curve ## process data ## plot data fig, ax = plt.subplots(figsize=(12,9)) ## figsize is the aspect ratio of the figure ax.plot(x,spec) ## ax is the variable for the axis where you want to work ax.invert_xaxis() ## vibrational spectra need this ax.set_xlabel('Wavenumber /cm$^{-1}$') ## latex works here ax.set_ylabel('Absorbance') ofile = 'figures/examplespecreversex.png' fig.savefig(ofile,dpi=75) ## dpi is dots per inch and is a figure resotution thing ### This part is for making this website, so you don't need it return ofile
Change the Line Width of the Lines so the Show Up Better
On the computer screen using python, it is relatively easy to see the lines of a graph. When you display them in a beamer presention or even in publication quality graphics, it is not always so clear. You can control the linewidth.
Here is an example controlling the linewidth in the plot command:
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np import pandas as pd # data organization module import matplotlib.pyplot as plt # plotting module lw = 5 # use a variable to control graphs at once ## get data x = np.arange(1600,1800) # these will be the x values in the graph they are simulating a UV spectrum xc = 1656 # this is the center of the band sigma = 50 # %sigma*2*sqrt(2*ln(2)) is the width at half height of the Y max of the curve spec = np.exp(-1*((x-xc)**2)/(2*sigma**2)) ##this will make a Gaussian curve ## process data ## plot data fig, ax = plt.subplots(figsize=(5,4)) ## figsize is the aspect ratio of the figure ax.plot(x,spec, linewidth=lw) ## ax is the variable for the axis where you want to work ax.invert_xaxis() ## vibrational spectra need this ax.set_xlabel('Wavenumber /cm$^{-1}$') ## latex works here ax.set_ylabel('Absorbance') ofile = 'figures/examplespeclinewidth.png' fig.savefig(ofile,dpi=75) ## dpi is dots per inch and is a figure resotution thing ### This part is for making this website, so you don't need it return ofile
Here is an example controlling the linewidth for the whole Python or Jupyter session:
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np import pandas as pd # data organization module import matplotlib.pyplot as plt # plotting module ###these next lines write the parameters lw,lsz to plotting parameters import matplotlib as mpl ## this loads the moduule matplotlib as mpl lw = 5 #this is the linewidth lsz = 12 #this is the font size mpl.rcParams['xtick.labelsize'] = lsz mpl.rcParams['ytick.labelsize'] = lsz mpl.rcParams['font.size'] = lsz mpl.rcParams['lines.linewidth'] = lw ## get data x = np.arange(1600,1800) # these will be the x values in the graph they are simulating a UV spectrum xc = 1656 # this is the center of the band sigma = 50 # %sigma*2*sqrt(2*ln(2)) is the width at half height of the Y max of the curve spec = np.exp(-1*((x-xc)**2)/(2*sigma**2)) ##this will make a Gaussian curve ## process data ## plot data fig, ax = plt.subplots(figsize=(5,4)) ## figsize is the aspect ratio of the figure ax.plot(x,spec) ## ax is the variable for the axis where you want to work ax.invert_xaxis() ## vibrational spectra need this ax.set_xlabel('Wavenumber /cm$^{-1}$') ## latex works here ax.set_ylabel('Absorbance') ofile = 'figures/examplespeclinewidth2.png' fig.savefig(ofile,dpi=75) ## dpi is dots per inch and is a figure resotution thing ### This part is for making this website, so you don't need it return ofile
Change or Control the Color of the Line
You can control the color of a line in a similar method as you control the linewidth.
| RGBValue | Short Name | Long Name |
|---|---|---|
| (1, 1, 0) | y | yellow |
| (1, 0, 1) | m | magenta |
| (0, 1, 1) | c | cyan |
| (1, 0, 0) | r | red |
| (0, 1, 0) | g | green |
| (0, 0, 1) | b | blue |
| (1, 1, 1) | w | white |
| (0, 0, 0) | k | black |
Here is a basic or simple example of controlling the color of a line:
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np import pandas as pd # data organization module import matplotlib.pyplot as plt # plotting module ## get data x = np.arange(200,400) # these will be the x values in the graph they are simulating a UV spectrum xc = 350 # this is the center of the band sigma = 50 # %sigma*2*sqrt(2*ln(2)) is the width at half height of the Y max of the curve spec = np.exp(-1*((x-xc)**2)/(2*sigma**2)) ##this will make a Gaussian curve ## process data ## plot data fig, ax = plt.subplots(figsize=(6,5)) ## figsize is the aspect ratio of the figure ax.plot(x,spec,'r') ## ax is the variable for the axis where you want to work ax.plot(x,spec+0.5,'k') ax.set_xlabel('Wavelength /nm') ## latex works here ax.set_ylabel('Absorbance') ofile = 'figures/examplespeccolor1.png' fig.savefig(ofile,dpi=75) ## dpi is dots per inch and is a figure resotution thing ### This part is for making this website, so you don't need it return ofile
For finer color control, use a color tuple (3 number sequence surrounded by ( ) to get definition of the color.
Usage: (R, G, B) the numbers are between 0 and 1. All ones (1,1,1) is white, all zeros is black (0,0,0), and (1,0,0) is red.
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np import pandas as pd # data organization module import matplotlib.pyplot as plt # plotting module # use a color tuple # this is a dictionary of colors usage cl[0] yields blue (0,0,1) # you don't have to use this dictionary, I just like it. # you could also redefine the keys to something more memorable cl = {0:(0, 0, 1),#; %blue 4:(0, 1, 0),#; % bright green 6:(0, 1, 1),#; % cyan 3:(0, 0, 0),#; %black 1:(1, 0, 0),#; % red 7:(1, 1, 0),# % yellow 2:(1, 0, 1),#; % magenta 5:(1, 0.5, 0.5),#; % pink brown 8:(0.75, 1, 0.75),# % dull light green 9:(0.5, 0.5, 1),# % greyblue 10:(1, 0.5, 0),#; % orange 11:(0, 0.8, 0.5),# % bluegreen 12:(0.5, 0, 0.6),# % purple 13:(1,0.75,0),## orange 14:(0.5,0,1), ##also purple, dark 15:(0.78,1,0), ##ugly green 16:(0,0.5,1), #blue ski 17:(1,0,0.25), #pink red 18:(0.39,0.39,0.49), #grey blue 19:(0.02,0.42,0.1), #forest green 20:(0.75,0.26,0.36) #dark red } ## get data x = np.arange(200,400) # these will be the x values in the graph they are simulating a UV spectrum xc = 350 # this is the center of the band sigma = 50 # %sigma*2*sqrt(2*ln(2)) is the width at half height of the Y max of the curve spec = np.exp(-1*((x-xc)**2)/(2*sigma**2)) ##this will make a Gaussian curve ## process data ## plot data color1 = cl[0] ## get the 0th color tuple color2 = cl[1] ## get the 1st color tuple print(color1, color2) fig, ax = plt.subplots(figsize=(6,4)) ## figsize is the aspect ratio of the figure ax.plot(x,spec,color=color1) ## ax is the variable for the axis where you want to work ax.plot(x,spec+0.5,color=color2) ax.set_xlabel('Wavenumber /cm$^{-1}$') ## latex works here ax.set_ylabel('Absorbance') ofile = 'figures/examplespeccolor1.png' fig.savefig(ofile,dpi=75) ## dpi is dots per inch and is a figure resotution thing ### This part is for making this website, so you don't need it return ofile
In the past there was a lot of expense associated with using color in journal/report figures, so many people avoided using color. Today these ideals are much less important. However, color should not be used just for the sake of using color. Your color choices should enhance understanding of the graphics, not just prettyfy them.
Colors and markers are attributes that can be controlled in figures. Here is a table of markers.
| specifier | Marker Type |
|---|---|
| '+' | Plus sign |
| 'o' | Circle |
| '*' | Asterisk |
| '.' | Point |
| 'x' | Cross |
| 'square' or 's' | Square |
| 'diamond' or 'd' | Diamond |
| '^' | Upward-pointing triangle |
| 'v' | Downward-pointing triangle |
| '>' | Right-pointing triangle |
| '<' | Left-pointing triangle |
| 'pentagram' or 'p' | Five-pointed star (pentagram) |
| 'hexagram' or 'h' | Six-pointed star (hexagram) |
Here is an example of controlling both the color and the marker in a scatter plot.
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np import pandas as pd # data organization module import matplotlib.pyplot as plt # plotting module ## get data C = [1, 2, 3, 4, 5] # these will be the x values in the graph they are simulating concentration in ppm A = np.array([0.1, 0.2, 0.3, 0.4, 0.5]) #this is the simulated Aborbance values ## process data ## plot data fig, ax = plt.subplots(figsize=(6,5)) ## figsize is the aspect ratio of the figure ax.scatter(C,A,marker='o',color='r',label='data 1') ## ax is the variable for the axis where you want to work ax.scatter(C, A+0.5,marker='+',color='r',label='data 2') ## ax is the variable for the axis where you want to work ax.scatter(C, A+0.3,marker='x',color='g',label='data 3') ax.legend() ax.set_xlabel('Concentration /ppm') ## latex works here ax.set_ylabel('Absorbance') ofile = 'figures/examplescattermakers.png' fig.savefig(ofile,dpi=100) ## dpi is dots per inch and is a figure resolution thing ### This part is for making this website, so you don't need it return ofile
Adding Axis Labels
Use the axis methods .setxlabel('your words here') and .setylabel('your label here') to add/change the labels on axes.
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np import pandas as pd # data organization module import matplotlib.pyplot as plt # plotting module ## get data C = [1, 2, 3, 4, 5] # these will be the x values in the graph they are simulating concentration in ppm A = np.array([0.1, 0.2, 0.3, 0.4, 0.5]) #this is the simulated Aborbance values ## process data ## plot data fig, ax = plt.subplots(figsize=(6,4)) ## figsize is the aspect ratio of the figure ax.scatter(C,A,marker='o',color='r') ## ax is the variable for the axis where you want to work ax.set_xlabel('Concentration /M') ## latex works here ax.set_ylabel('Absorbance') ofile = 'figures/exampleaxislabels.png' fig.savefig(ofile,dpi=100) ## dpi is dots per inch and is a figure resotution thing ### This part is for making this website, so you don't need it return ofile
control the font size
To control the font:
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np import pandas as pd # data organization module import matplotlib.pyplot as plt # plotting module ###these next lines write the parameters lw,lsz to plotting parameters for the whole jupyter/python session import matplotlib as mpl ## this loads the module matplotlib as mpl lw = 5 #this is the linewidth lsz = 50 #this is the font size, 50 is probably WAAAAAYYYYY too big. mpl.rcParams['xtick.labelsize'] = lsz mpl.rcParams['ytick.labelsize'] = lsz mpl.rcParams['font.size'] = lsz mpl.rcParams['lines.linewidth'] = lw ## get data x = np.arange(1600,1800) # these will be the x values in the graph they are simulating a UV spectrum xc = 1656 # this is the center of the band sigma = 50 # %sigma*2*sqrt(2*ln(2)) is the width at half height of the Y max of the curve spec = np.exp(-1*((x-xc)**2)/(2*sigma**2)) ##this will make a Gaussian curve ## process data ## plot data fig, ax = plt.subplots(figsize=(12,9)) ## figsize is the aspect ratio of the figure ax.plot(x,spec) ## ax is the variable for the axis where you want to work ax.invert_xaxis() ## vibrational spectra need this ax.set_xlabel('Wavenumber /cm$^{-1}$') ## latex works here ax.set_ylabel('Absorbance') ofile = 'figures/examplesfont1.png' fig.savefig(ofile,dpi=75) ## dpi is dots per inch and is a figure resotution thing ### This part is for making this website, so you don't need it return ofile
insert a chemical structure into the graphic
- State "TODO" from
Not done yet.
Multiple Graphs on one Figure
To add multiple graphs on one figures use the subplot command to get a different axes for each graph.
subplot(numberrows,numbercolumns,plotindex)
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np import pandas as pd # data organization module import matplotlib.pyplot as plt # plotting module ## get data x = np.arange(200,400) # these will be the x values in the graph they are simulating a UV spectrum xc = 250 # this is the center of the band sigma = 30 # %sigma*2*sqrt(2*ln(2)) is the width at half height of the Y max of the curve spec = np.exp(-1*((x-xc)**2)/(2*sigma**2)) ##this will make a Gaussian curve ## process data ## plot data fig = plt.figure(figsize=(12,8)) ax1 = fig.add_subplot(211) ## axis 1 for subplot 1 ax1.plot(x,spec) ax1.set_xlabel('Wavelength /nm') ## latex works here ax1.set_ylabel('Absorbance') ax2 = fig.add_subplot(212) ## axis 2 for subplot 2 ax2.plot(x,np.sin(x)) ax2.set_xlabel('Time /min') ax2.set_ylabel('Aplitude /V') ofile = 'figures/examplesuplots.png' fig.savefig(ofile,dpi=75) ## dpi is dots per inch and is a figure resotution thing ### This part is for making this website, so you don't need it return ofile
Resize the Displayed Region (zoom)
Sometimes the information that you want to display is not easy to see if you show all of your data at once. In these situations it is better zoom in on the regions of interest. If you want to show where the band max is for example.
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np import pandas as pd # data organization module import matplotlib.pyplot as plt # plotting module from matplotlib.patches import Rectangle ## for drawing the rectangle ## get data x = np.arange(200,400,0.1) # these will be the x values in the graph they are simulating a UV spectrum xc = 250 # this is the center of the band sigma = 30 # %sigma*2*sqrt(2*ln(2)) is the width at half height of the Y max of the curve spec = np.exp(-1*((x-xc)**2)/(2*sigma**2)) ##this will make a Gaussian curve ## process data ## plot data fig = plt.figure(figsize=(12,8)) ax0 = fig.add_subplot(211) ## whole specturm ax0.plot(x,np.sin(x)) ax0.set_xlabel('Wavelength /nm') ## latex works here ax0.set_ylabel('Absorbance') ax0.add_patch(Rectangle((225,0.4),width=50, height=0.6,alpha=0.5)) ax1 = fig.add_subplot(212) ax1.plot(x,np.sin(x)) ax1.set_xlabel('Wavelength /nm') ## latex works here ax1.set_ylabel('Absorbance') ax1.set_xlim(225,275) ### control the view of lower graph ax1.set_ylim(0.4, 1) ### control the view ofile = 'figures/examplezoom.png' fig.savefig(ofile,dpi=75) ## dpi is dots per inch and is a figure resotution thing ### This part is for making this website, so you don't need it return ofile
Adding a Vertical Line to a Graph and Text annotations
If you want to indicate where a band is in more than one spectrum, a nice way to do this is to draw a vertical line though the graph.
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np import pandas as pd # data organization module import matplotlib.pyplot as plt # plotting module import scipy.io as sio #open matlab files ## get data ifile = 'data/tolpxyl.mat' mat_contents = sio.loadmat(ifile) ## read the file output into a dictionary print(mat_contents.keys()) ##print the names of the contents of the matfile Aorg = mat_contents['Aorg'] ## retrieve the Spectra xorg = mat_contents['xorg'] ## retrieve the x values print('the shape of the Aorg is ',Aorg.shape) x = xorg.T A = Aorg C = mat_contents['Ctrue'] ## concentration matrix ## process data ## plot data fig = plt.figure(figsize=(12,8)) ## figsize is the aspect ratio of the figure ax1 = fig.add_subplot(211) ## ax1 is the variable for the axis where you want to work ax1.plot(x,A.T) ax1.set_xlim((200,340)) ## set graph's limits # put a vertical line on the ax axis # linestyles [‘solid’ | ‘dashed’, ‘dashdot’, ‘dotted’ | (offset, on-off-dash-seq) | '-' | '--' | '-.' | ':' | 'None' | ' ' | ''] # alpha is transparency 0 - 1 ax1.axvline(x=222.5,linestyle='--',color='k',alpha=0.9) ax1.axvline(x=274,linestyle='--',color='r',alpha=0.9) ax1.text(224, 2.62, ' this band is not linear with concentration') ## text annotations ax1.text(280,1.3, '$\leftarrow$ this band is linear with concentration') ##latex annotations ax1.set_xlabel('Wavelength /nm') ## latex works here ax1.set_ylabel('Absorbance') ax2 = fig.add_subplot(212) ax2.scatter(C[:,0],A[:,304],color='r',label='274 nm') ax2.scatter(C[:,0],A[:,509],color='k',label='222 nm') ax2.set_xlabel('Analyte Concentration v/v /%') ## latex works here ax2.set_ylabel('Absorbance') ax2.legend() ofile = 'figures/exampleverticalline.png' fig.savefig(ofile,dpi=100) ## dpi is dots per inch and is a figure resotution thing ### This part is for making this website, so you don't need it return ofile
adding grid lines
Grid lines also allow your reader to interpret the data in a graph. They are particularly useful for ovelaping data sets.
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np import pandas as pd # data organization module import matplotlib.pyplot as plt # plotting module import scipy.io as sio #open matlab files ## get data ifile = 'data/tolpxyl.mat' mat_contents = sio.loadmat(ifile) ## read the file output into a dictionary print(mat_contents.keys()) ##print the names of the contents of the matfile Aorg = mat_contents['Aorg'] ## retrieve the Spectra xorg = mat_contents['xorg'] ## retrieve the x values print('the shape of the Aorg is ',Aorg.shape) x = xorg.T A = Aorg ## process data ## plot data fig, ax = plt.subplots(figsize=(12,9)) ## figsize is the aspect ratio of the figure ax.plot(x,A.T) ## ax is the variable for the axis where you want to work ax.set_xlim((200,340)) ## set graph's limits # which ---> major, minor, both # Turn on the minor TICKS, which are required for the minor GRID ax.minorticks_on() ax.grid(True, which='major') ##select which to show, 'major' grid lines or 'minor' ax.set_xlabel('Wavelength /nm') ## latex works here ax.set_ylabel('Absorbance') ofile = 'figures/examplegrid.png' fig.savefig(ofile,dpi=300) ## dpi is dots per inch and is a figure resotution thing ### This part is for making this website, so you don't need it return ofile
adding a band label
Another way of indicating areas of the graph that you want to talk about in the text or bring into focus for the user is to add textual annotations such as a label to your graph. In python you can do this by importing your graph into powerpoint and manually add the label. This works fine until your professor makes you recreate every figure using the Greek alphabet. If you are finding that you need to recreate lots of figures and/or don't like using a mouse you can also add annotations programatically. Here is an example.
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np import pandas as pd # data organization module import matplotlib.pyplot as plt # plotting module import scipy.io as sio #open matlab files ## get data ifile = 'data/tolpxyl.mat' mat_contents = sio.loadmat(ifile) ## read the file output into a dictionary print(mat_contents.keys()) ##print the names of the contents of the matfile Aorg = mat_contents['Aorg'] ## retrieve the Spectra xorg = mat_contents['xorg'] ## retrieve the x values print('the shape of the Aorg is ',Aorg.shape) x = xorg.T A = Aorg ## process data ## plot data fig, ax = plt.subplots(figsize=(12,9)) ## figsize is the aspect ratio of the figure ax.plot(x,A.T) ## ax is the variable for the axis where you want to work ax.set_xlim((200,340)) ## set graph's limits ax.text(275,1.2, '(a)',fontsize=25) ## this fontsize is too big ax.set_xlabel('Wavelength /nm') ## latex works here ax.set_ylabel('Absorbance') ofile = 'figures/exampletext.png' fig.savefig(ofile,dpi=100) ## dpi is dots per inch and is a figure resotution thing ### This part is for making this website, so you don't need it return ofile
adding a scale bar to the graph
I don't really have a great way of doing this. I will think about it and add it later.
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np import pandas as pd # data organization module import matplotlib.pyplot as plt # plotting module ## get data def dataloader(ifile): import numpy as np rawdata = np.genfromtxt(ifile, delimiter=',') ## read the file output into an array x = rawdata[:,0] ## extract the wavenumbers spec = rawdata[:,1] ## extract the absorbance values return x, spec ## return the wavenumbers and absorbance values ifile1 = 'sample2.csv' ## datafile named stored in variable datahole = 'data/' #location of datafiles (path) x,spec1 = dataloader(datahole + ifile1) ## load data file notice the concatenation of the path and datfile fig, ax = plt.subplots(figsize=(12,9)) ## create a figure and axis to plot onto ax.plot(x,spec1,label='spec 1') ## plot one file with label ax.set_xlim((700,2000)) ## change the view of the graph ax.invert_xaxis() ##important for IR ax.legend() ## show the legend ax.set_ylabel('Absorbance') ## set the y axis label ax.set_xlabel('Wavenumber /cm$^{-1}$') ## set the x axis label ax.annotate('0.05 abosorbance',(1750,0.0),(1750,0.05), arrowprops={'arrowstyle':'|-|'},ha='center') ofile = 'figures/dataspeccalebar.png' fig.savefig(ofile,dpi=100) ## dpi is dots per inch and is a figure resotution thing ### This part is for making this website, so you don't need it return ofile
ass11
- create a Jupyter notebook (use the blank ass11.ipynb already in the directory) from scratch in the ass11 directory
- load one or more data file(s) from your ass11/data directory
- plot the data properly
- use 2 of the following tricks
- make a stacked plot
- added grid lines
- use subplots to create 2 axes
- add a bandlabel
- add a vertical line to highlight one band
- full credit will only be given to graphs (located in the figures directory) that are perfect, partial credit will be given.
- No credit without showing your work (code) in the ass11.ipynb
- make a 150 dpi graph of the data
- save this file into the ass11/figures directory
Library Searching
It is very common for undergraduate chemistry students to use a spectral library to help identify an unknown sample. This process of library searching in reality, falls into 2 categories: matching purified sample spectrum to a library, and identifying pure components in a mixture spectrum.
In this tutorial, we will only explore the first category.
Pure components
The first, and perhaps more familiar, library searching task is searching a library of pure components for a spectral match with an unknown spectrum that has been chemically purified. (This is what you probably did in Organic Chemistry Laboratory.)
Hit Quality Index (HQI)
So, you may be wondering what do we mean by "matching" or "comparing" spectra? You could plot all known data with your unknown data and visually look at them and determine which known most looks like the unknown. This would probably work, because humans are really good at pattern matching. The problem is the work load. What if your library of known compound spectra had 20 000 spectra in it? How long would it take to do this task.
So what we really mean by spectral "matching" and "comparison" is math.
The most common similarity measure (though not really the best) used in library searching is the Hit Quality Index (HQI). Here is the equation for its calculation.
\begin{equation} HQI = \frac{(u\cdot r)^2}{(u \cdot u)(r \cdot r)} \end{equation}where
- u is the unknown spectrum
- r is a reference spectrum in your library of known materials
- the \(\cdot\) between them is the dot product
Some Math Definitions
Before we get into understanding how to program this HQI into an algorithm let's first cover some definitions and mathematical concepts. The following definitions are mathematical definitions. In programming languages sometimes the names of things aren't as distinct as in the math world (i.e. vector, matrix)
scalar
A scalar is a single number. In python M.shape shows all dimensions = 1 or has no shape because it is a float
Vector
A vector a single column or single row of numbers. In python V.shape yields (N,), (1,N), (N,1), or (N,1,1) etc. In this tutorial vectors are typeset like v with lower case and bold typeface. The vector [2,3,17] is a called a row vector. You can think of it as the coordinates (x,y,z) in a 3D Cartesian coordinates system. It is also called a 3-dimensional row vector. A vector can also be a column vector.
\begin{equation} \left[ \begin{array}{c} 2 \\ 3 \\ 17 \end{array} \right] \end{equation}In python a vector is simply a specially shaped np.matrix whose shape only has one python dimension > 1. There is no special datatype called a vector in python.
import numpy as np v = np.matrix([4,4,1,7]) print(v.shape)
Transpose
To transpose a vector (or matrix) is to switch the rows with the columns. In the vector examples above the row and column vector are the transpose of each other. Using Mathematical notation the transpose is written with a super-scripted T or t. For example:
\begin{equation} b = \left[2, 3, 17 \right] \end{equation} \begin{equation} b^T = \left[ \begin{array}{c} 2 \\ 3 \\ 17 \end{array} \right] \end{equation}In python the transpose of a vector (or matrix) is a method in np.matrix. Here is an example:
import numpy as np v = np.matrix([4,4,1,7]) print('v = ',v) print('the shape of v is a vector = ', v.shape) print('the transpose of v is ', v.T) print('the shape of the transpose is also a vector', v.T.shape)
Matrix
A matrix is like a vector except that it has rows and columns of numbers. In fact, it can have as many levels ≥ 2, 3, 4, etc. of numbers as are imaginable. These levels are called the rank of the matrix. Sometimes the rank of a matrix is referred to the number of dimensions (or ndims in python) of the matrix. For this tutorial we will only use the rank to prevent confusion with N-dimensional vectors mentioned above. In python M.shape yields (N,M), (N,M,R), etc. In this tutorial matrices are typeset like M with upper case and bold typeface. In python there is a special datatype called a np.matrix. We have spent a little time with them before. Here is a python example for the creation of a python np.matrix.
import numpy as np M = np.matrix([[2,2], [2,3]]) print('M is a matrix = ',M) print('the shape of M is a vector = ', M.shape) print('the transpose of M is ', M.T) print('the shape of the transpose of is also a vector', M.T.shape)
Vector and Matrix Multiplication
(video) Here are some handwritten notes from a video. ./figures/matrix_multiplication_examples.pdf
Here is some example code to do matrix multiplication in Python.
import numpy as np B = np.matrix([[2,2], [2,3]]) D = np.matrix([ [5,5,5],[4,4,4] ] ) G = B*D print('The matrix G is \n {:} \n '.format(G)) print('the shape of B is ', B.shape) print('the shape of D is ', D.shape) print('the shape of G is ', G.shape)
dot product
(video) Here are some handwritten notes from a video. ./figures/vector_dot_product.pdf
Here is some code demonstrating how to take the dot product of 2 vectors.
import numpy as np ## these two vector can not be multiplied directly b = np.matrix( [2 , 1 ]) ## make a row vector a = np.matrix( [5,5]) ## make another row vector g = a*b.T ## needs to be row times column so use transpose print('The inner product of g is \n {:} \n Notice that it is a np.matrix'.format(g)) print('the shape of b is ', b.shape) print('the shape of a is ', a.shape) print('the shape of g is ', g.shape) ## This inner product is so commonly used that there is a command for this ## this short cut command really is only benificial if your vectors are np.arrays ## if your vectors are np.matrix then the output of this function (np.dot) is the same as above b = np.array( [2 , 1 ]) a = np.array( [5,5]) g = np.dot(a,b) print('The dot product of g is \n {:} \n Notice that it is a NOT matrix'.format(g)) print('the shape of b is ', b.shape) print('the shape of a is ', a.shape) print('the shape of g is ', g.shape, 'because it is not an np.array or np.matrix' )
- So what is the dot product of 2 vectors?
The dot product (or scalar product or inner product) of two vectors is a scalar number that is geometrically the product of the magnitudes or lengths of the two vectors times the cosine of the angle between them. Algebraically, the dot product is the sum of the products of the elements of the two sequences of numbers. What does it mean for Chemometrics? It is a kind of "measure" of the overlap of the two vectors. The more similar the bigger the overlap. This issue with soley using the dot product to care the numerically compare two vectors is that the dot product is not normalized. So you can not know what the "best" value is. However you can compare a limited set of data.
Figure 17: Wikipedia example of scalar projection consept of dot product
What is the HQI
It is a normalized "measure" or "metric" of the overlap of the two vectors u and r. If the vectors r and u are identical then the HQI should be 1. If the vectors r and u are completely unrelated, and thus have no overlap (orthogonal in math lingo) then the HQI should be 0.
What are these vectors? First we need to understand that a spectrum is an N-dimensional vector, where N is the number of wavelengths or wavenumbers in the spectrum. To calculate the HQI you need to perform some vector math.
Code for calculating the HQI
Here is some python code for this equation above.
import numpy as np HQI = np.dot(u,r)**2/(np.dot(u,u) * np.dot(r,r) )
Below is simple code to search though a library of spectra and find the closest match.
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np import pandas as pd # data organization module import matplotlib.pyplot as plt # plotting module import scipy.io as sio #open matlab files # define useful functions for re-use def hqicalc(r,u): ''' function to calculate the hqi between r and u vectors r is the reference u is the unknown ''' import numpy as np HQI = np.dot(u,r)**2/(np.dot(u,u) * np.dot(r,r) ) return HQI ## get data ifile = 'data/libdata.mat' mdict = sio.loadmat(ifile) ## read the file output into a dictionary print(mdict.keys()) ##print the names of the contents of the matfile x = mdict['x'] datalist = ['silk', 'wool', 'nylon', 'cotton', 'polyester'] ## list of known spectra specunk = np.squeeze(mdict['silk']) #an artificial unknown spectrum hqidict = {} ## empty dictionary to store results for polymername in datalist: ## step through the datalist r = np.squeeze(mdict[polymername]) ## extract known spectrum by name one at a time hqi = hqicalc(r,specunk) ## call the hqi calculating function hqidict[polymername] = hqi ## store results in dictionary print(polymername,hqi) ## print nice table for reading ## store into a dataframe df = pd.DataFrame.from_dict(hqidict,orient='index',columns=['HQI']) fig = plt.figure(figsize=(12,8)) ax1 = fig.add_subplot(211) df.plot(kind='barh',ax=ax1,legend=False) ax1.set_xlabel('HQI') x = np.squeeze(mdict['x']) #fig, ax = plt.subplots(figsize=(12,9)) ax2 = fig.add_subplot(212) ax2.plot(x,specunk+0.01,c='k',label='unknown') for polymername in datalist: ## step through the datalist r = np.squeeze(mdict[polymername]) ax2.plot(x,r,label=polymername) ## plot one spectrum at a time ax2.set_xlim((400,2000)) ax2.invert_xaxis() ax2.legend() ax2.set_xlabel('Wavenumber /$cm^{-1}$') ax2.set_ylabel('Absorbance') ofile = 'figures/libsearch1.png' fig.savefig(ofile,dpi=100) ## dpi is dots per inch and is a figure resotution thing ### This part is for making this website, so you don't need it return ofile
Assignment 12 (ass12)
- A
create a function that will
- take as inputs 2 vectors
- calculate the HQI of the 2 vectors
- round the HQI (using the np.around function) to 3 decimal places
- return the HQI
- B
load the dataset file lotsofspectra.mat Use the function from ass12 part A to find hqi of all the known spectra vs the unknown spectrum Report your answer as a Markdown cell with the closest known spectrum label for example: known1 Show your work for credit. No code/work = no credit
Other similarity Measurements (metrics of distance)
One of the problems with the hit quality index is shown in the example above is that all of the values are jammed near 1 (the perfect match score). The primary reasons for this is that we are using the whole spectrum for our library search, and most of the spectrum is in fact similar to the unknown. We will cover dealing with this kind of situation in the next section.
There are other metrics that could be used to search a library for similar spectra. This whole set of metrics are called either metrics of distance or their inverse similarity metrics.
In the figure below is depicted 2 additional metrics that can be used to compare 2 vectors.
Figure 18: Distance Mearures between vectors
Euclidean Distance
If you remember your high school geometry, you may remember learning about the Pythagorean theorem.
\begin{equation} c^2 = a^2 + b^2 \end{equation}where c, b, and a are all the length of a triangle's sides. In a more generalized sense, this is also called the Euclidean distance (ED). In the figure above, points A and B are points in a 2D coordinate system. The coordinates are listed in the ( ) beside the label. If you were asked what is the distance between 2 points. The hypotenuse of the right triangle is what you would draw. For a 2D coordinate system the Euclidean distance is the length of the hypotenuse and is solved for by this equation:
\begin{equation} ED = \sqrt{\Delta X^2 + \Delta Y^2} \end{equation}where Δ X and Δ Y are the length of the sides of the triangle formed along the coordinate axes (X and Y).
You may be asking yourself how this relates to spectroscopic or chromatographic data. Remember that these data (spectra or chromatograms) are vectors. Also remember that the coordinates for points A and B are also vectors. So the math for solving for the ED is the same, only the number of dimensions in the vector are different. So if we can generalized the math to deal with any number of dimension in vectors then these two situations become the same.
In linear algebra, equations are represented as matrices/vectors. We will learn about this in a later section of this tutorial. For now just accept that this is true. The Euclidean distance is represented by the following equation.
\begin{equation} ED = \sqrt{\Delta X \cdot \Delta Y} \end{equation}and the python code to calculate he Euclidean distance is
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np import pandas as pd # data organization module import matplotlib.pyplot as plt # plotting module import scipy.io as sio #open matlab files # define useful functions for re-use def hqicalc(r,u): ''' function to calculate the hqi between r and u vectors r is the reference u is the unknown ''' import numpy as np HQI = np.dot(u,r)**2/(np.dot(u,u) * np.dot(r,r) ) return HQI def edcalc(r,u): ''' calculate the Euclidean distance between 2 vectors ''' import numpy as np #ED = euclidean(r,u) ED = (np.dot(r-u,r-u))**.5 return ED #ofile = 'figures/libsearch1.png' #fig.savefig(ofile,dpi=100) ## dpi is dots per inch and is a figure resotution thing ### This part is for making this website, so you don't need it #return ofile
One nice feature about the Euclidean distance is that it does a better job of not bunching all of the distances together as in the HQI example above. One "disadvantage" of ED is that as a distance measurement two similar vectors (spectra) have a small distance (the same vectors would have a distance of zero. Conversely, vectors (spectra) that are very different have a larger number. This behavior is the opposite of the HQI discussed previously.
To combat this difference between the HQI scale and ED there is a concept called similarity.
Cosine metric
add words here
Pearson's correlation coefficient
add word here
Similarity
Similarity is a re-scaling of any distance measurement between 0 and 1, where 0 is unsimilar and 1 is similar. Here is the most common way to calculate a similarity score.
\begin{equation} S_{d} = 1- \frac{d}{d_{max}} \end{equation}where
- Sd is the similarity score from some distance metric (Euclidean, Manhattan, Hamming, etc.)
- d is the distance metric (Euclidean, Manhattan, Hamming, etc.)
- dmax is the maximum distance metric (Euclidean, Manhattan, Hamming, etc.) of all known data
This concept can be useful for comparing distances and overlap type of metrics such as HQI, Pearson's correlation coefficient and ED.
Here is some python code calculating several similarity scores for a polymer library search. You can see that ED is better than HQI at not bunching all of the spectra near perfect (1).
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np import pandas as pd # data organization module import matplotlib.pyplot as plt # plotting module import scipy.io as sio #open matlab files # define useful functions for re-use def hqicalc(r,u): ''' function to calculate the hqi between r and u vectors r is the reference u is the unknown ''' import numpy as np HQI = np.dot(u,r)**2/(np.dot(u,u) * np.dot(r,r) ) return HQI def edcalc(r,u): ''' calculate the Euclidean distance between 2 vectors ''' import numpy as np #ED = euclidean(r,u) ED = (np.dot(r-u,r-u))**.5 return ED ifile = 'data/libdata.mat' mdict = sio.loadmat(ifile) ## read the file output into a dictionary from scipy.spatial.distance import cosine datalist = ['silk', 'wool', 'nylon', 'cotton', 'polyester'] ## list of known spectra specunk = np.squeeze(mdict['silk'])+0.001 #an artificial unknown spectrum hqidict = {} ## empty dictionary to store results EDdict = {} dotdict = {} cordict = {} for polymername in datalist: ## step through the datalist r = np.squeeze(mdict[polymername]) ## extract known spectrum by name one at a time hqi = hqicalc(r,specunk) ## call the hqi calculating function hqidict[polymername] = hqi ## store results in dictionary ed = edcalc(r,specunk) EDdict[polymername] = ed #dotdict[polymername] = np.dot(r,specunk) cordict[polymername] = np.corrcoef(r,specunk)[0,1] ## Pearson's correlation coefficient #print(polymername,ed) ## print nice table for reading df = pd.DataFrame.from_dict(hqidict,orient='index',columns=['HQI']) ## similarity df['ED'] = pd.DataFrame.from_dict(EDdict,orient='index',columns=['ED']) ## distance df['cor'] = pd.DataFrame.from_dict(cordict,orient='index',columns=['cor']) ## similarity ## convert to similarity (0 - 1) df['ED'] = 1 - df['ED']/df['ED'].max() ## calculate simularity based upon ED ##make graph fig = plt.figure(figsize=(12,8)) ax1 = fig.add_subplot(211) df.plot(kind='barh',ax=ax1,legend=False) ax1.set_xlabel('Similarity') ax1.legend() x = np.squeeze(mdict['x']) #fig, ax = plt.subplots(figsize=(12,9)) ax2 = fig.add_subplot(212) ax2.plot(x,specunk+0.01,c='k',label='unknown') for polymername in datalist: ## step through the datalist r = np.squeeze(mdict[polymername]) ax2.plot(x,r,label=polymername) ## plot one spectrum at a time ax2.set_xlim((400,2000)) ax2.invert_xaxis() ax2.legend() ax2.set_xlabel('Wavenumber /$cm^{-1}$') ax2.set_ylabel('Absorbance') ofile = 'figures/libsearch2.png' fig.savefig(ofile,dpi=100) ## dpi is dots per inch and is a figure resotution thing ### This part is for making this website, so you don't need it return ofile
ass13 (assignment 13)
The purpose of this assignment is to test how you might implement a new distance measurement. write a function that will
- take 2 inputs that are vectors (numpy arrays with only 1D)
- calculate the cosine distance between the 2 vectors
- return the distance rounded to 3 decimal places
- HINT: look at this webpage link for guidance
- due date is in the directory
Mixture Identity
This area of chemometrics is still actively being pursued and represents the cutting edge of chemometrics research. Why is this important and what does it mean? Most real samples are mixtures. To determine what the components in this mixture are requires one of three techniques.
- Physical Separation: chromatography, extraction, selective precipitation
- Spectral Separation: using non-overlaping bands in spectra to determine the presence
and concentration of the mixture's components. (i.e. ICP-OES, MS, etc.)
- Chemometrics: We will cover this latter after we learn some more advanced matrix math.
Pre-Processing data
Often the raw data that is measured must be modified, before you can process it into chemical information. We will refer to this type of data manipulation as preprocessing data. (Getting the data ready to process.)
Here are a set of pre-processing techniques. Part of your goal as a data user is to determine what the best set of pre-processing tools are for a given problem.
cropping and zapping and truncating
Sometimes data contains information this is not important or even detrimental for a calculation. Here is an example:
Figure 19: Infrared spectrum of ethanol (neat). The arrow indicates a non-ethanol band.
In this figure is an infrared spectrum of neat ethanol. The arrow indicates a band that is not attributable to ethanol, but rather CO2.
- If you were to use this spectrum in a calculation (i.e. library search, ethanol concentration in gin, etc.), how would this CO2 spectral contribution effect your calculation results?
- HINT: Is the CO2 level constant?
- Should you include this portion of the spectrum in your calculations? (If you sneeze into your beaker during a titration should you use the data from that titration in your results?)
In the previous section on graphing data, we talked about not showing this part of the data in figures making. (See the ax.setxlim command in python/matplotlib). In this section of this course, we are going to learn about strategies for not using portions of your data.
cropping data and truncating data
These are the same thing. You are dropping useless, contaminated, interfering, non-linearly correlating parts of the data. You simply slice the array. I usually create a new array instead of deleting part of the original array. That way I have the original vector still in memory for latter use. Here is how I do it.
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np import pandas as pd # data organization module import matplotlib.pyplot as plt # plotting module import scipy.io as sio #open matlab files # load and organize the data ifile = 'data/libdata.mat' mdict = sio.loadmat(ifile) datalist = ['silk', 'wool', 'nylon', 'cotton', 'polyester'] ## list of known spectra Au = np.squeeze(mdict['silk'])+0.001 #an artificial unknown spectrum x = mdict['x'].T ## extract the wavenumber values out of the dictionary pe = np.squeeze(mdict['polyester']) ## extract the absorbance values of the polyester, np.squeeze() drops dems = 1 fig = plt.figure(figsize=(12,12)) ax1 = fig.add_subplot(211) ax1.plot(x,pe,label='polyester (whole)') ax1.plot(x,Au,label='unknown (whole)') ax1.set_xlim((500,4000)) ax1.invert_xaxis() ax1.legend() ax2 = fig.add_subplot(212) rng = np.arange(1159,len(Au)) ## indices of awl Auc = Au[rng] ## slice the org spectrum with the awl indices pec = pe[rng] ## slice the org spectrum with the awl indices xc = x[rng] ## also slice the wavenumber/wavelengths for plotting ax2.plot(xc,pec,label='polyester (cropped)') ax2.plot(xc,Auc,label='unknown (cropped)') ax2.set_xlim((500,4000)) ax2.invert_xaxis() ## vibrational spectra need this ax2.legend() ax2.set_xlabel('Wavelength /$cm^{-1}$') ## latex works here ax2.set_ylabel('Absorbance') ax1.set_ylabel('Absorbance') ofile = 'figures/cropspec.png' fig.savefig(ofile,dpi=100) ## dpi is dots per inch and is a figure resotution thing ### This part is for making this website, so you don't need it return ofile
- How did we determine the position or index of the analytical wavelengths (awl) in the code above?
For a quick and dirty way to determine the slice or crop range of the awl, simply plot the spectra without the x vector (plot(wool.T)) and use the cursor to hover over the graph. Use your chemical intuition to pick the "best" region. We might also call this first set of awl as a first approximation of the awl. Later, in this tutorial we will talk about how to optimize the choices for awl to achieve some analytical goal, but for now a visual inspection will work fine.
zapping data
This means to set a particular area to zero. I don't really like this as it can add artifacts to the data near the edge of zapped area. For the unknown data in the library searching problem you might be tempted to zap the region between 1800 and 2500 cm-1 where you know their is no useful data. Here is what that might look like.
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np import pandas as pd # data organization module import matplotlib.pyplot as plt # plotting module import scipy.io as sio #open matlab files ## get data ifile = 'data/libdata.mat' mat_contents = sio.loadmat(ifile) ## read the file output into a dictionary print(mat_contents.keys()) ##print the names of the contents of the matfile woolorg = mat_contents['wool'] ## retrieve the Spectra xorg = mat_contents['x'] ## retrieve the x values print('the original shape of the wool is ',woolorg.shape) xorg = xorg.T ## transpose x for future less typing ## pre-process data nspec,npts = woolorg.shape ## get the shape of org wool vector zapwl = np.arange(1,1161) ## set the zapped wavelengths wool = woolorg.copy() ##make a copy of woolorg into wool, otherwize these are linked wool[0,zapwl] = 0.0 ## crop: slice org wool vector into smaller vector x = xorg ## for plotting xorg needs to be cropped ## plot data using subplots fig = plt.figure(figsize=(12,8)) ax1 = fig.add_subplot(211) ax1.plot(xorg,woolorg.T,label='original data') ax2 = fig.add_subplot(212) ax2.plot(x,wool.T,label='zapped data') ax1.set_xlim((500,4000)) ## set graph's limits ax2.set_xlim((500,4000)) ## set graph's limits ax2.invert_xaxis() ## vibrational spectra need this ax1.invert_xaxis() ax2.set_xlabel('Wavelength /$cm^{-1}$') ## latex works here ax2.set_ylabel('Absorbance') ax1.set_ylabel('Absorbance') ofile = 'figures/zappedspec.png' fig.savefig(ofile,dpi=100) ## dpi is dots per inch and is a figure resotution thing ### This part is for making this website, so you don't need it return ofile
Scaling Data
autoscale or z-transformation
This is only possible if you have multiple measurements or spectra. This transformation will produce odd looking spectra that are difficult to interpret (see figure below), but it accentuates the portions of the spectra that are changing over the whole dataset. Notice that the region where the diamond ATR absorbs light also gets accentuated.
\begin{equation} Z = \frac{A - A_{mean}}{A_{std}} \end{equation}Here is how to do this z-transformation. We will only graph one so you can see the difference.
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np import pandas as pd # data organization module import matplotlib.pyplot as plt # plotting module import scipy.io as sio #open matlab files ## example autoscaling spectrum within whole dataset # load and organize the data ifile = 'data/libdata.mat' mdict = sio.loadmat(ifile) x = np.squeeze(mdict['x']) ## extract the wavenumber values out of the dictionary datalist = ['silk', 'wool', 'nylon', 'cotton', 'polyester'] ## list of known spectra datadict = {} for name in datalist: ## for loop through the filelist spec = np.squeeze(mdict[name]) ## store spectrum in dictiorary datadict[name] = spec ## store original data for comparison df = pd.DataFrame.from_dict(datadict,orient='index',columns=x) ## convert the dictionary to a dataframe dfas = (df - df.mean())/df.std(ddof=1) ## autoscale fig,(ax1,ax2) = plt.subplots(figsize=(12,9),nrows=2, ncols=1, sharex=True) fig.subplots_adjust(hspace=0) df.T.plot(ax=ax1) ## plot the dataframe with index as labels on the axis ax ax1.text(3500,0.3,'Original') dfas.T.plot(ax=ax2) ax2.set_xlim((650,4000)) ## set graph's limits ax2.invert_xaxis() ## vibrational spectra need this ax2.legend() ## call the legond for the axis to show it ax2.set_xlabel('Wavenumber /cm$^{-1}$') ## latex works here ax2.set_ylabel('Absorbance') ax2.text(3500,1.57,'autoscaled') ofile = 'figures/ztransspec.png' fig.savefig(ofile,dpi=100) ## dpi is dots per inch and is a figure resotution thing ### This part is for making this website, so you don't need it return ofile
normalization, by band height
Normalization will produce data that is between 0 and 1. If you have a band that you know is not changing in a dataset then you can set it to 1 and look at the rest of these bands. This plus the band area normalization is a kind of internal normalization.
set specific band's absorbance = 1
\begin{equation} Anorm = \frac{A}{A(450)} \end{equation}Also you can normalize a spectrum by setting the band with the maximum height to 1.
\begin{equation} Anorm = \frac{A}{max(A)} \end{equation}where A is the unnormalized absorbance vector.
python code to normalize by max band height
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np import pandas as pd # data organization module import matplotlib.pyplot as plt # plotting module import scipy.io as sio #open matlab files ## function to normalize data def normbyh(spec): ''' normalize by max band height ''' tmp = spec - np.min(spec) An = tmp/np.max(tmp) return An # load and organize the data ifile = 'data/libdata.mat' mdict = sio.loadmat(ifile) x = np.squeeze(mdict['x']) datalist = ['silk', 'wool', 'nylon', 'cotton', 'polyester'] ## list of known spectra datadict = {} datadictpp = {} for name in datalist: ## for loop through the filelist spec = np.squeeze(mdict[name]) ## store spectrum in dictiorary datadict[name] = spec ## store original data for comparison datadictpp[name] = normbyh(spec) #normalize the data as you read it in df = pd.DataFrame.from_dict(datadict,orient='index',columns=x) ## convert the dictionary to a dataframe dfpp = pd.DataFrame.from_dict(datadictpp,orient='index',columns=x) ## convert the dictionary to a dataframe fig,(ax1,ax2) = plt.subplots(figsize=(12,9),nrows=2, ncols=1, sharex=True) fig.subplots_adjust(hspace=0) df.T.plot(ax=ax1) ## plot the dataframe with index as labels on the axis ax dfpp.T.plot(ax=ax2) ax2.set_xlim((650,4000)) ## set graph's limits ax2.invert_xaxis() ## vibrational spectra need this ax2.legend() ## call the legond for the axis to show it ax2.set_xlabel('Wavenumber /cm$^{-1}$') ## latex works here ax2.set_ylabel('Absorbance') ofile = 'figures/bandnormspec.png' fig.savefig(ofile,dpi=100) ## dpi is dots per inch and is a figure resotution thing ### This part is for making this website, so you don't need it return ofile
python code to normalize by specific band height
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np import pandas as pd # data organization module import matplotlib.pyplot as plt # plotting module import scipy.io as sio #open matlab files def normbysh(spec,bindex): ''' normalize by specific band height ''' tmp = spec - np.min(spec) An = tmp/tmp[bindex] return An ## example normalize by specific band height # load and organize the data ifile = 'data/libdata.mat' mdict = sio.loadmat(ifile) x = np.squeeze(mdict['x']) ## extract the wavenumber values out of the dictionary datalist = ['silk', 'wool', 'nylon', 'cotton', 'polyester'] ## list of known spectra datadict = {} datadictpp = {} bindex = 559 ## C-H band print(x[559]) for name in datalist: ## for loop through the filelist spec = np.squeeze(mdict[name]) ## store spectrum in dictiorary datadict[name] = spec ## store original data for comparison datadictpp[name] = normbysh(spec,bindex) #normalize the data as you read it in df = pd.DataFrame.from_dict(datadict,orient='index',columns=x) ## convert the dictionary to a dataframe dfpp = pd.DataFrame.from_dict(datadictpp,orient='index',columns=x) ## convert the dictionary to a dataframe fig,(ax1,ax2) = plt.subplots(figsize=(12,9),nrows=2, ncols=1, sharex=True) fig.subplots_adjust(hspace=0) df.T.plot(ax=ax1) ## plot the dataframe with index as labels on the axis ax dfpp.T.plot(ax=ax2) ax2.set_xlim((650,4000)) ## set graph's limits ax2.invert_xaxis() ## vibrational spectra need this ax2.legend() ## call the legond for the axis to show it ax2.set_xlabel('Wavenumber /cm$^{-1}$') ## latex works here ax2.set_ylabel('Absorbance') ax2.axhline(y=1.0,linestyle='--',color='k',alpha=0.5) ax2.axvline(x=x[559],linestyle='--',color='g',alpha=0.5) ofile = 'figures/bandnormspecspecific.png' fig.savefig(ofile,dpi=100) ## dpi is dots per inch and is a figure resotution thing ### This part is for making this website, so you don't need it return ofile
normalization, by band area
Later.
set total area under the spectrum = 1
\begin{equation} Anorm = \frac{A}{areaunder(A)} \end{equation}normalization, by total spectral area
This type of normalization is generally a foolproof way of getting all of your data on to a common y-scale. You divide the whole spectrum by the magnitude of the vector (spectrum) or the total area under the spectrum.
set total area under the spectrum = 1
\begin{equation} Anorm = \frac{A}{areaunder(A)} \end{equation}# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np import pandas as pd # data organization module import matplotlib.pyplot as plt # plotting module import scipy.io as sio #open matlab files def normbya(spec): ''' normalize spectra by total area of spectrum ''' import numpy as np tmp = spec - np.min(spec) area = np.linalg.norm(tmp) specn = tmp/area return specn # load and organize the data ifile = 'data/libdata.mat' mdict = sio.loadmat(ifile) x = np.squeeze(mdict['x']) ## extract the wavenumber values out of the dictionary datalist = ['silk', 'wool', 'nylon', 'cotton', 'polyester'] ## list of known spectra datadict = {} datadictpp = {} for name in datalist: ## for loop through the filelist spec = np.squeeze(mdict[name]) ## store spectrum in dictiorary datadict[name] = spec ## store original data for comparison datadictpp[name] = normbya(spec) #normalize the data as you read it in df = pd.DataFrame.from_dict(datadict,orient='index',columns=x) ## convert the dictionary to a dataframe dfpp = pd.DataFrame.from_dict(datadictpp,orient='index',columns=x) ## convert the dictionary to a dataframe fig,(ax1,ax2) = plt.subplots(figsize=(12,9),nrows=2, ncols=1, sharex=True) fig.subplots_adjust(hspace=0) df.T.plot(ax=ax1) ## plot the dataframe with index as labels on the axis ax ax1.text(3500,0.3,'Original') dfpp.T.plot(ax=ax2) ax2.set_xlim((650,4000)) ## set graph's limits ax2.invert_xaxis() ## vibrational spectra need this ax2.legend() ## call the legond for the axis to show it ax2.set_xlabel('Wavenumber /cm$^{-1}$') ## latex works here ax2.set_ylabel('Absorbance') ax2.text(3500,0.08,'area normalized') ofile = 'figures/areanormspec.png' fig.savefig(ofile,dpi=100) ## dpi is dots per inch and is a figure resotution thing ### This part is for making this website, so you don't need it return ofile
Noisy data, "high frequency"
Some times the SNR of spectral data makes it difficult to understand the data. In an ideal situation you should acquire better data, but sometimes that is not possible. So you might have to smooth the data to extract the information that you need.
What do we mean by "high frequency" noise? As you look (left-to-right) at the spectrum below you will see "features" that move rapidly up and down. Some "features" rise and fall slower than others. The "features" that you care about are called the signal. The "features" that you don't care about we will call "noise" (some people define noise as unwanted signal). There are many sources of noise: electrical, sample geometry, scan-time, detector, environmental (humans bumping into your spectrometer during the measurement), Sometimes the unwanted signal can be other components in the sample generating a signal that you don't care about. We call this interference. In every case, the unwanted features can impede our ability to extract information from the data.
The way you deal with these unwanted "features" depend upon their characteristics compared to the wanted "features".
Here is an example of a noisy spectrum. As you scan across this spectrum from left to right the noise moves up and down more rapidly than the signal. So we will call this "high frequency" noise.
Figure 20: Noisy spectrum
There are two methods to "remove" the high frequency noise. One is using the Savitzky-Golay method or convolution.
The other is to use Fourier filtering.
convolution smoothing
This explanation is pretty good (link).
Here is animation that also might help (from wikipedia).
Figure 21: Demonstration of convolution
Here is how you smooth the data using convolution.
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np import pandas as pd # data organization module import matplotlib.pyplot as plt # plotting module import scipy.io as sio #open matlab files def dataloader(ifile): rawdata = np.genfromtxt(ifile,delimiter=",") ## read the file output into an array x = rawdata[:,0] ## extract/slice the wavenumbers (x values) spec = rawdata[:,1] return x,spec # get data ifile = './data/toluene.csv' x,A = dataloader(ifile) ## pre-process data window = np.array([1, 2, 3, 4, 5, 4, 3, 2, 1 ],dtype=float) #define the smoothing function, triangle #window =np.array([1, 2, 3, 2, 1],dtype=float) ## shorter triangle function windowsize = len(window); window = window/np.sum(window) # normalize it so that the smoothed spectra has the same area As = np.convolve(A,window,mode='same') # convolution of the smoothing function and the data ## plot data using subplots fig = plt.figure(figsize=(8,8)) ax1 = fig.add_subplot(211) ax1.plot(x,A.T,label='original data') ax1.text(280,0.5,'Original') ax2 = fig.add_subplot(212) ax2.plot(x,As.T,label='area normalized data') ax2.text(280,0.5,'homemade smoothed') ax1.set_xlim((220,300)) ## set graph's limits ax2.set_xlim((220,300)) ## set graph's limits ax2.set_xlabel('Wavelength /nm') ## latex works here ax2.set_ylabel('Absorbance') ax1.set_ylabel('Absorbance') ofile = 'figures/gapsmoothed.png' fig.savefig(ofile,dpi=100) ## dpi is dots per inch and is a figure resotution thing ### This part is for making this website, so you don't need it return ofile
Compare different gap sizes to see what the effect is.
Here are is an example of Savitzky-Golay smoothing using a function from a module.
Here is the paper.
There are better ways to smooth spectra than convolving a triangular function across it. The most common is the Savitzky-Golay algorithm. In this algorithm a more sophisticated smoothing function is used that better matches the shape of spectral bands. Fortunately, this function is accessible from the scipy module.
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np import pandas as pd # data organization module import matplotlib.pyplot as plt # plotting module #import scipy.io as sio #open matlab files from scipy.signal import savgol_filter as sg ## new module from scipy def dataloader(ifile): rawdata = np.genfromtxt(ifile,delimiter=",") ## read the file output into an array x = rawdata[:,0] ## extract/slice the wavenumbers (x values) spec = rawdata[:,1] return x,spec # get data ifile = './data/toluene.csv' x,A = dataloader(ifile) ## pre-process data windowsize = 11 polyorder = 2 dorder = 2 As = sg(A, windowsize, polyorder, deriv=0) ## plot data using subplots fig,(ax1,ax2) = plt.subplots(figsize=(12,9),nrows=2, ncols=1, sharex=True) fig.subplots_adjust(hspace=0) ax1.plot(x,A,label='original data') ax1.text(280,0.5,'Original') ax2.plot(x,As,label='area normalized data') ax2.text(280,0.5,'SG smoothed') #ax1.set_xlim((220,300)) ## set graph's limits ax2.set_xlim((220,300)) ## set graph's limits ax2.set_xlabel('Wavelength /nm') ## latex works here ax2.set_ylabel('Absorbance') ax1.set_ylabel('Absorbance') ofile = 'figures/sgsmoothed.png' fig.savefig(ofile,dpi=100) ## dpi is dots per inch and is a figure resotution thing ### This part is for making this website, so you don't need it return ofile
smoothing, FFT
Here is another way of filtering high frequency noise. Do it with Fourier Transforms. Remember Fourier transform is the mathematical tool utilized in FT-IR and NMR (even though we don't called it FT-NMR) to convert the time-domain measured data (interferograms) into frequency-domain spectra. Here is a video from a company https://www.youtube.com/watch?v=qz0MLVh7Gok first part, and here is another more detailed video https://www.youtube.com/watch?v=spUNpyF58BY.
The key to filtering data with FFT is that features in the "spectral-domain" (this is the spectra) that are high frequency will be grouped together and hopefully far apart from signal in the "Fourier-domain" after you FFT the spectrum.
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np import pandas as pd # data organization module import matplotlib.pyplot as plt # plotting module import scipy.io as sio #open matlab files from numpy.fft import fft, ifft ## import the fft and ifft functions from numpy.fft def dataloader(ifile): rawdata = np.genfromtxt(ifile,delimiter=",") ## read the file output into an array x = rawdata[:,0] ## extract/slice the wavenumbers (x values) spec = rawdata[:,1] return x,spec # get data ifile = './data/toluene.csv' x,A = dataloader(ifile) ## pre-process data percent = 35 ## 0-50 this is how much of the time domain size that you want to smooth T = fft(A) ## convert spectral-domain data to Fourier-domain data npts = len(T) ## calc. how long is it ## this next line generates a set of indices in the middle of the T vector see plots mid = np.arange((np.round(npts/2-percent/100*npts)),(round(npts/2+percent/100*npts)),dtype=int) ## this is the filter coordinates ## setup filter filt = np.ones((T.shape)) ### array of all ones filt[mid] = 0 ## set mid part of the filter to zeros Tfiltered = T*filt ## element-wise multiply the filter by the Fourier-domain data As = np.real(ifft(Tfiltered)) ## inverse the filtered Fourier-domain data back into spectral-domain data (only keep the real part) ## plot the data and intermediate steps fig = plt.figure(figsize=(8,8)) ax1 = fig.add_subplot(411) ax1.plot(x,A.T,label='original data') ax2 = fig.add_subplot(412) ax2.plot(np.real(T)/np.max(np.real(T)),label='real part of fft of data') ax2.plot(filt,label='filter') ax3 = fig.add_subplot(413) ax3.plot(np.real(Tfiltered/np.max(Tfiltered)),label='filtered fft of data') ax4 = fig.add_subplot(414) ax4.plot(x,As.T,label='FFT smoothed') ax1.legend() ax2.legend() ax3.legend() ax4.legend() ax4.annotate('notice the ringing \n at the broundries', xy=(719, 0.3), xytext=(500, 0.31), arrowprops=dict(facecolor='black', shrink=0.005,width=1,headwidth=5), ) #ax2.set_xlabel('Wavelength /nm') ## latex works here #ax2.set_ylabel('Absorbance') #ax1.set_ylabel('Absorbance') ofile = 'figures/fftmoothed.png' fig.savefig(ofile,dpi=100) ## dpi is dots per inch and is a figure resotution thing ### This part is for making this website, so you don't need it return ofile
Noisy data, "low frequency" baseline problems
Sometimes data has a non-zero baseline. This is often either a sample preparation problem (scattering), a background measurement problem, or an interfering chemical contributing to the spectrum. So the best solution with this type of data situation is to remeasure your data. Unfortunately, this can not always happen.
Here is some example data.
Figure 22: scattering type of data
fit and baseline subtraction
This is the most common way to deal with this problem (other than remeasuring your data).
Select some of the points on the spectrum that are clearly not part of the chemistry but optical in nature.
Figure 23: selected points along baseline
Fit these points with a polynomial.
Figure 24: selected points along baseline are fitted with a polynomial
and subtract that polynomial from the spectral data.
Figure 25: crazy baseline fixed
Here is some code to correct this kind of data.
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np import pandas as pd # data organization module import matplotlib.pyplot as plt # plotting module import scipy.io as sio #open matlab files def dataloader(ifile): rawdata = np.genfromtxt(ifile,delimiter=",") ## read the file output into an array x = rawdata[:,0] ## extract/slice the wavenumbers (x values) spec = rawdata[:,1] return x,spec # get data ifile = 'data/toluenebl.csv' x,A = dataloader(ifile) ####### DATA inpsection ##plot the absorbance values; #inspect the data # find indices where you think there is not chemical information #plt.plot(A) ## fit baseline to function cr = [0,50 ,275, 300, 350,400] #these are the indices order = 5 #this is the order of the polynomial that you are using to fit the data p = np.polyfit(x[cr],A[cr],order) #get the fitting parameters (polynominal coefficents) fity = np.polyval(p,x) #produce a new set of y data based upon the fit and your x Abl = A - fity #subtract the fitted background from you spectrum. ## plot data using subplots fig,(ax1,ax2) = plt.subplots(figsize=(12,9),nrows=2, ncols=1, sharex=True) fig.subplots_adjust(hspace=0) ax1.plot(x,A,label='original data') ax1.text(280,0.5,'Original') ax1.plot(x,fity.T,label='baseline fit') ax2.plot(x,Abl,label='area normalized data') ax2.text(280,0.5,'baseline corrected') #ax1.set_xlim((220,300)) ## set graph's limits #ax2.set_xlim((220,300)) ## set graph's limits ax2.set_xlabel('Wavelength /nm') ## latex works here ax2.set_ylabel('Absorbance') ax1.set_ylabel('Absorbance') ofile = 'figures/baselinecorrected.png' fig.savefig(ofile,dpi=100) ## dpi is dots per inch and is a figure resotution thing ### This part is for making this website, so you don't need it return ofile
Spectral derivatives
Sometimes it is difficult to remove the non-zero baseline. One way to minimize the impact on data analysis of low frequency baseline (spectral drift, fluorescence, scattering), is to just leave it in the spectrum, but de-emphasize its effects on any future data analysis.
The simplest way to do this is to use derivative spectra. There are two major issues with using derivative spectra.
- by de-emphasizing the "low frequency noise", the derivatization of spectra emphasizes the high frequency portions of the data, or amplifies the noise. This is usually unwanted.
- derivative spectra don't look like normal spectra and are therefore more difficult to interpret.
Fortunately, by simply changing the gap or window function, you can use the Savitzky-Golay (SG) algorithm to take the derivative of a spectrum and smooth counteract the amplification of the noise.
I like the second derivative best, because band position is preserved.
Here is some code for implementation.
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np import pandas as pd # data organization module import matplotlib.pyplot as plt # plotting module import scipy.io as sio #open matlab files from scipy.signal import savgol_filter as sg # scipy version of the Savitzky Golay smoothing and derivatives def dataloader(ifile): rawdata = np.genfromtxt(ifile,delimiter=",") ## read the file output into an array x = rawdata[:,0] ## extract/slice the wavenumbers (x values) spec = rawdata[:,1] return x,spec # get data ifile = 'data/toluenebl.csv' x,A = dataloader(ifile) ## pre-process data windowsize = 11 ## smoothing function size porder = 3 ##polynomial order must be larger than derivative dorder = 2 ## derivative order 0 = smooth, 1 = 1st derivative, 2 = 2nd derivative Ad = sg(A,windowsize,polyorder=porder,deriv=dorder) fig,(ax1,ax2) = plt.subplots(figsize=(12,9),nrows=2, ncols=1, sharex=True) fig.subplots_adjust(hspace=0) ax1.plot(x,A,label='original data') ax1.text(280,0.5,'Original') ax2.plot(x,Ad) ax2.text(280,0.01,'2$^{nd}$ Derivative') ax2.set_xlabel('Wavelength /nm') ## latex works here ax2.set_ylabel('2$^{nd}$ Derivative Absorbance') ax1.set_ylabel('Absorbance') ofile = 'figures/derivativespectra.png' fig.savefig(ofile,dpi=100) ## dpi is dots per inch and is a figure resotution thing ### This part is for making this website, so you don't need it return ofile
FFT
Maybe we will add this later. For now here is an article that my friend Bu wrote.
https://doi.org/10.1366/0003702053946010
Assignment 14 (ass14)
part A
- Create a jupyter notebook in this assignment directory.
- In this notebook, load the data (data.mat) provided in the ass14/data/ directory.
- Preprocess the unknown spectra.
- You choose which preprocessing routines to do and in which order in order to improve the "match" metric between the known and unknown.
- Create a graph similar to
showing your results. Only use Euclidean distance (or its Similarity, your choice.) for this graph.
Part B
Write a short description of your preprocessing choices, and why you chose them using Markdown.
Introduction to Linear Algebra
We will refer to linear algebra as a set of techniques for solving systems of linear equations.
Here is a working set of equations:
\begin{equation} \begin{aligned} x + y = 2\\ 2x + 3y = 7 \end{aligned} \end{equation}These equations can be re-written as a set of matrices.
\begin{equation} \underbrace{ \begin{bmatrix} 1 & 1 \\ 2 & 3 \end{bmatrix} }_\text{C} \underbrace{ \begin{bmatrix} x \\ y \end{bmatrix} }_\text{v} = \underbrace{ \begin{bmatrix} 2\\ 7 \end{bmatrix} }_\text{b} \end{equation}and more compactly as
\begin{equation} Cv = b \end{equation}Some rules and definitions for matrices:
python arrays and matrices
In python, there is a difference between an array and a matrix. A matrix is a special type of array for which matrix operations are the default.
- In python arrays that are matrices, mathematical operations on them are matrix operations.
- In arrays that are NOT matrices, mathematical operations on them are element-wise.
- We will sometimes use the matrix form of arrays in this course to keep the linear algebraic operations easier to read
import numpy as np #this command loads all of the functions in numpy and labels them np from numpy.linalg import inv # setup matrices (with data this would be the step where you load the data) C = np.array([[1,1],[2,3]],dtype=float) # C is an array Cm = np.matrix(C) # C is a matrix b = np.array([2,7],dtype=float) # b is an array bm = np.matrix(b).T # b is a matrix print('b = ',b) print('b*2 = ',b*2) print('b*b = ',b*b) print('bm = ',bm) print('bm*2 = ',bm*2) print('bm.T*bm = ',bm.T*bm)
- matrix dimensions are listed (in matlab and octave and python, and therefore here) as (rows, columns)
- transpose of a matrix means to flip the rows and columns (in python/numpy to take the transpose of a matrix where C is the matrix C.T is the transpose)
- in matrix multiplication, matrices don't commute.
- matrix times a constant is an element-wise operation
- the identity matrix times any other square matrix results in the other matrix (python/numpy code for identity matrix is np.eye(5) where 5 is the number of diagonal elements)
\begin{equation} \underbrace{ \begin{bmatrix} 1 & 1 \\ 2 & 3 \end{bmatrix} }_\text{C} \underbrace{ \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix} }_\text{E} = \underbrace{ \begin{bmatrix} 1 & 1 \\ 2 & 3 \end{bmatrix} }_\text{C} \end{equation} - there is no such thing as division by a matrix
- only square matrices have inverses
In python you can take the inverse of a matrix using the numpy module's function inv(C) where C is the matrix
### load the inv library into python's memory for use from numpy.linalg import inv
- you can calculate the inverse of a matrix only if its determinant does not equal zero. Using the C matrix from above the two vertical lines around the C mean determinant in mathematical notation
\begin{equation} \left|C\right| \ne 0 \end{equation} - A matrix times its inverse is equal to the identity matrix
\begin{equation} C C^{-1} = E \end{equation}
Solving a system of linear equations, where the coefficient matrix is square
The corresponding python/numpy code for this calculation is:
import numpy as np #this command loads all of the functions in numpy and labels them np from numpy.linalg import inv # setup matrices (with data this would be the step where you load the data C = np.array([[1,1],[2,3]],dtype=float) # C is an array #C = np.matrix(C) # C is a matrix b = np.array([2,7],dtype=float) # b is an array #b = np.matrix(b).T # b is a matrix v = inv(C)@b # you don't need the rest of this return v
Solving a system of linear equations, where the coefficient matrix is not square
To solve this type of regression you take the Moore-Penrose pseudo-inverse
\begin{equation} \begin{aligned} Cv &=\;b\\ (C^TC)^{-1} C^T C v &= (C^TC)^{-1} C^T b\\ Ev = v &= (C^TC)^{-1} C^T b \end{aligned} \end{equation}The corresponding Matlab and octave code for this calculation is:
import numpy as np #this command loads all of the functions in numpy and labels them np from numpy.linalg import inv, pinv # setup matrices (with data this would be the step where you load the data C = np.array([[1,1],[2,3]],dtype=float) # C is an array #C = np.matrix(C) # C is a matrix b = np.array([2,7],dtype=float) # b is an array #b = np.matrix(b).T # b is a matrix v = inv(C.T@C)@C.T@b print('pseudo-inverse method hand-coded: ', v) # or because python is cool there is already the function for the pseudo-inverse of a matrix v = pinv(C)@b print('pseudo-inverse method buildin to numpy: ', v)
Practice
Convert this set of equations to Matrix Notation
solve for K using matrix notation
solve for C using matrix notation
K-matrix Method
All chemometric analyzes usually involve 3 stages:
- Data collection, selection and preprocessing.
- Calculation and evaluation of the results.
- Presentation of the results
Given all of the concepts and tools that we have learned before this point we are ready to tackle the second of these chemometric stages. (Unfortunately, this stage is the easiest.)
In this section of the course, we will explore the applications and limitations of the Beer-Lambert-Bouguer Law (you probably know this law as Beer's Law).
We will use the term, K-matrix method, to refer to application of linear algebra to solving Beer's law problems.
The Basic Procedure
Prepare calibration standards
Contains known concentrations
Prepare unknown samples
Contains unknown concentrations
measure standards and samples
Store data from calibration standards in to matrix
Akn[nspec,npts]
Store data from samples in to matrix
Aunk[nspec,npts]
Store known concentrations into matrix
C[nspec,ncomp]
Fit the data
This procedure is also called training the model.
Spreadsheet (Excel, LibreOffice Calc, Gnumeric)
use linest Here is an example ./figures/univariate_anal.xlsx
univaritate analysis, used in samples with a single unknown or multiple unknowns with non-overlapping data (e.g. ICP-OES, etc.)
- use np.polyfit to get the fitting parameters (slope and intercept for linear fit)
- fitting parameters is another phrase for the model that represents this data
- remember what the model means for absorbance spectroscopy (absorptivity)
- See Example 1.A above
import numpy as np #this command loads all of the functions in numpy and labels them np from numpy.linalg import inv, pinv # setup matrices (with data this would be the step where you load the data from a file) x = np.arange(1,16) y = np.array([3.95, 5.58,6.66, 9.84, 11.19,13.18,15.61,16.83,18.47,21.01, 22.11, 24.37, 27.67, 29.88, 30.54]) nspec = len(y) ncomp = 1 #training the model (linear) p =np.polyfit(x,y,1) #begin validation setup yp =np.polyval(p,x) # predict the y values at the dataset's x values R = y - yp; # calculate the residuals SEC = np.sqrt(np.sum(R**2,axis=0)/np.abs(nspec-ncomp)) ## calc SEC (standard error of calibration print('How well did the model fit? SEC = +/-',SEC) ## begin K-matrix regression #K = pinv(C)*Akn ## solve for the model #Cu = Au*pinv(K) ## determine conc. of samples of unknown concentration #print('The concentrations in Cu are ',Cu)
multivariate analysis, overlapping bands or using the whole spectrum
- See example 1.B above for this kind of problem
- use matrix math to get the K matrix,
- the K matrix is the model for this chemical system.
- it is the absorptivity × pathlength.
Use the model (K) or fitting parameters (slope and intercept) to calculate the concentration of the unknown spectra
Cunk[nspec,ncomp]
The math for K-matrix method
The matrix math involved in the K-matrix method is the same we saw in the section on Linear Algebra.
\begin{equation} A_{kn} = CK \end{equation}Where Akn is the calibration dataset, C is the known concentrations for spectra in Akn, and K is the model that linearly relates the A and C.
Solve for K (fit the data).
\begin{equation} K = (C^TC)^{-1}C^TA \end{equation}K is the model that describes the data (Akn and C). If the samples of unknown concentration chemically behave the same way as the calibration standards, then K can be used to determine the concentrations of samples from their spectra.
Solve for Cu.
\begin{equation} C_{u} = A_{u}K^T(KK^T)^{-1} \end{equation}Where Cu is the determined unknown concentration, Au is the measured unknown spectrum, and K is the previously determined model for this chemical system.
Code for K-matrix method
For this example below we will not use whole spectra, but rather absorbance values at specific analytical wavelengths
import numpy as np #this command loads all of the functions in numpy and labels them np from numpy.linalg import inv, pinv # setup matrices (with data this would be the step where you load the data from a file) Akn = np.array([ ## spectra of calibration standards [0.157706, 0.052752 , 0.075142], [0.177411 , 0.051542 , 0.065700], [0.187398 , 0.038855 , 0.066837], [0.200147 , 0.047255 , 0.062150], [0.221439 , 0.041524 , 0.057156], ]) Au = np.array([0.197056 , 0.041629 , 0.068137]) ## spectra of samples of unknown concentration Ckn = np.array([[60, 15 , 25], ## concentration of calibration standards [70 , 15, 15], [75 , 5 , 20], [80, 10, 10], [90 , 5 , 5]]) ## begin K-matrix regression K = pinv(Ckn)@Akn ## solve for the model Cu = Au@pinv(K) ## determine conc. of samples of unknown concentration print('The concentrations in Cu are ',Cu)
Here is the code to do the K-matrix method with the whole spectrum.
import numpy as np #this command loads all of the functions in numpy and labels them np from numpy.linalg import inv, pinv import scipy.io as sio # setup matrices (with data this would be the step where you load the data from a file) ifile = 'data/kmatdata1.mat' ##name of datafile to load dic = sio.loadmat(ifile) ## load the data in to a dictionary print(dic.keys()) ## see how the data is stored in the dictionary Akn = dic['Akn'] ## extact the data from the dictionary Ckn = dic['Ckn'] ## extact the data from the dictionary Au = dic['Au'] ## extact the data from the dictionary x = dic['x'].T ## x is for graphing not for calc ## begin K-matrix regression K = pinv(Ckn)@Akn ## solve for the model Cu = Au@pinv(K) ## determine conc. of samples of unknown concentration print('The concentrations in Cu are ',Cu)
Assignment 15
- Part 1: Write a function that will
- Take 2 inputs
- a set of spectra
- a set of concentration data
- calculate the K-matrix
- return the K-matrix
- Take 2 inputs
- Part 2: Write a function that will
- Take 2 inputs
- a set of Unknown spectra
- the K-matrix
- calculate the concentrations of the components in the Unknown spectra using the K-matrix method
- return the Concentration matrix
Due date in directory
- Take 2 inputs
Model evaluation
This section of this tutorial is all about model evaluation for a any generic model. The examples given herein only apply to the to the K-matrix method (because that is all that we have learn so far).
It is important to have confidence in your results. Generally, this is the purpose of statistics. Whenever you determine the concentrations of components in an unknown solution you need to report your uncertainty. There are several ways to do this, but in my opinion the best ways are those that directly address the ability of your model to predict the concentrations. It is not possible to compare true concentrations of unknown samples and a model's predicted concentrations. The next best thing is to compare true and predicted concentrations of known samples. These known samples are called validation samples.
There are two model evaluations that we will learn in this course.
Standard Error of Calibration (SEC)
Sometimes this is also called Standard Error of Estimation (SEE). The SEC tries to estimate how well does the model predict the training set, and is a measure of the quality of the calibration fit of the calibration data.
Here is the equation.
\begin{equation} SEC=\sqrt{\frac{ \sum(c_i - c_{pi})^2}{\left|N-K\right|}} \end{equation}where ci is the known concentration, cpi is the predicted concentration, N is the number of measurements (nspec), K is the number of fitting terms in the model (linear K = 2 (slope,intercept), quadratic K = 3, etc.)
\begin{equation} \left|N-K\right| \end{equation}is the degrees of freedom for the model.
Note: SEC gets better the more fitting terms in the model (K) and the fewer the number of data (N). So it is susceptible to bias when comparing different calibration models and data that are different levels of complexity and size.
Also
Here is the code for univariate modeling like you might use in MS Excel. for SEC
import numpy as np #this command loads all of the functions in numpy and labels them np from numpy.linalg import inv, pinv # setup matrices (with data this would be the step where you load the data from a file) x = np.arange(1,16) y = np.array([3.95, 5.58,6.66, 9.84, 11.19,13.18,15.61,16.83,18.47,21.01, 22.11, 24.37, 27.67, 29.88, 30.54]) nspec = len(y) ncomp = 1 #training the model (linear) p =np.polyfit(x,y,1) #begin validation setup yp =np.polyval(p,x) # predict the y values at the dataset's x values R = y - yp; # calculate the residuals SEC = np.sqrt(np.sum(R**2,axis=0)/np.abs(nspec-ncomp)) ## calc SEC print('How well did the model fit the clalibration data? SEC = +/-',SEC) ## begin K-matrix regression #K = pinv(C)*Akn ## solve for the model #Cu = Au*pinv(K) ## determine conc. of samples of unknown concentration #print('The concentrations in Cu are ',Cu)
Here is the code for K-matrix multivariate data analysis:
##estimate the uncertainty using PRESScv import numpy as np #this command loads all of the functions in numpy and labels them np from numpy.linalg import inv, pinv import scipy.io as sio # setup matrices (with data this would be the step where you load the data from a file) ifile = 'data/kmatdata1.mat' ##name of datafile to load dic = sio.loadmat(ifile) ## load the data in to a dictionary print(dic.keys()) ## see how the data is stored in the dictionary Akn = dic['Akn'] ## extact the data from the dictionary Ckn = dic['Ckn'] ## extact the data from the dictionary Au = dic['Au'] ## extact the data from the dictionary x = dic['x'].T ## x is for graphing not for calc. so no need to matrix nspec,npts = Akn.shape nspec,ncomp = Ckn.shape print(nspec,npts) K = pinv(Ckn)@Akn ## fit calibration spectra and concentrations Cp = Akn@pinv(K) ## predict conc of Au original data R = np.array(Ckn - Cp) #calculate the residuals print(R.shape) SEC = np.sqrt(np.sum(R**2,axis=0)/np.abs(nspec-ncomp)) ## calc SEC K = pinv(Ckn)@Akn ## fit calibration spectra and concentrations Cup = Au@pinv(K) ## predict conc of Au original data print('Concentrations: ', Cup) print('Uncertainties (SEC): ', SEC)
Standard Error of Prediction using cross validation (SEPcv)
A still better estimate of the predictivity (is this even a word?) of your model than SEC is to have the model predict samples not in the training set.
If you have good data SEC and SEP will be similar (the same order of magnitude).
There are basically two ways to do this:
- separate validation
Split the known data in to a training set and a validation set This technique is called separate validation and works best with large data sets. One limitation of separate validation is that a bad spectrum (or data point) can be either in the Training set or the Validation set. This can result in either a bad model or poor performance of the model that is in really a result of poor validation data.
- cross-validation
or leave one known spectrum out of the training set, predict(s) its concentration(s), put it back in and drop the next one. This method iterates over the entire training set, and works really well if you have a smaller set of knowns. One advantage is that it is good at identifying poor calibration data.
SEPcv is sometimes called Predicted Error Sum of Squares or PRESS
Here is the equation.
\begin{equation} SEP=\sqrt{\frac{ \sum(c_i - c_{pi})^2}{\left|M\right|}} \end{equation}where ci is the known concentration, cpi is the predicted concentration, M is the number of measurements (nspec) in the validation set. There are zero degrees of freedom lost because there is no a priori knowledge about the validation set.
Here is the code for univariate SEPcv:
TODO there is a problem with this code
import numpy as np #this command loads all of the functions in numpy and labels them np from numpy.linalg import inv, pinv import matplotlib.pyplot as plt ## this loads the moduule matplotlib.pyplot as plt # setup matrices (with data this would be the step where you load the data from a file) Ckn = np.arange(1,16,dtype=float) Akn = np.array([3.95, 5.58,6.66, 9.84, 11.19,13.18,15.61,16.83,18.47,21.01, 22.11, 24.37, 27.67, 29.88, 30.54]) nspec = len(Akn) ncomp = 1 R = np.zeros((Ckn.shape),dtype=float) # prefill R with zeros #this loop will iterated through each sample and leave it out for i in range(nspec): Cu = Ckn[i] # drop the ith x value Au = Akn[i] # drop the ith y value A = np.delete(Akn,i,axis=0) C = np.delete(Ckn,i,axis=0) p = np.polyfit(C,A,1) # train the model with all the remaining data #This is the validation step that is within the loop Cup = np.polyval(p,Cu) # predict left out y value R[i] = Cu - Cup # calculate the residual for the left out y and store it in an array #calculate the SEP using the R vector SEPcv = np.sqrt(np.sum(R**2,axis=0)/nspec) ##same units as x p = np.polyfit(Ckn,Akn,1) # train the model with all the remaining data # Cup = np.polyval(p,Ckn) # predict left out y value fig, ax = plt.subplots(figsize=(12,9)) ax.plot(Ckn,Akn,'*',label='data') ax.plot(Ckn,Ckn*p[0]+p[0],label='fit') ax.set_ylabel('Intensity') ax.set_xlabel('Concentration /ppm') ax.legend() print('How well did the model fit? SEP = +/-',SEPcv) ofile = 'figures/univariate_calibration_example_sepcv.png' fig.savefig(ofile,transparent=False,papertype='letter',orientation='bottom') return ofile
In the example below several aspects are different from the above example. Below you will find a function that performs the concentration prediction. This is a useful procedure because it allows you do drop-in a new modeling system as a new function. (i.e. K-matrix, PCAR, PLSR, ANNR, etc). Here is the code of implementation:
##estimate the uncertainty using PRESScv import numpy as np #this command loads all of the functions in numpy and labels them np from numpy.linalg import inv, pinv import matplotlib.pyplot as plt ## this loads the moduule matplotlib.pyplot as plt import scipy.io as sio # setup matrices (with data this would be the step where you load the data from a file) ifile = 'data/kmatdata1.mat' ##name of datafile to load mdict = sio.loadmat(ifile) ## load the data in to a dictionary #print(mdict.keys()) ## see how the data is stored in the dictionary Aknorg = mdict['Akn'] ## extact the data from the dictionary and convert to matrix Cknorg = mdict['Ckn'] ## extact the data from the dictionary and convert to matrix Auorg = mdict['Au'] ## extact the data from the dictionary and convert to matrix x = mdict['x'].T ## x is for graphing not for calc. so no need to matrix ## conc calc functions here def kmatconccalc(Akn,Ckn, Au): ''' calculate the concentration of unknown. usage: Cup = kmatconccalc(Akn,Ckn, Au) Akn must be matrix, known spectra Au must be a matrix, unknown spectra Ckn must be matrix, conconcentation of known spectra ''' import numpy as np ##this loads the module numpy as np from numpy.linalg import pinv K = pinv(Ckn)@Akn Cup = Au@pinv(K) return Cup def svdpcaf(A,nlv): ''' calculate the scores and loading vectors of a set of data usage: s,L = svdpcaf(A,nlv) where A = the matrix of data A.shape = (nspec,npts) nlv = the number of loading vectors (integer) s = scores matrix s.shape = (nspec,nlv) L = is the loading vector matrix L.shape = (nlv, npts) requires ''' import numpy as np from numpy.linalg import inv from numpy.linalg import svd U,W,Vh = svd(A,full_matrices=1) V = Vh.T L = -1*V[:,:nlv].T s = -1*np.dot(A,inv(V.T))[:,:nlv] return s,L def pcarcalc(Akn,Ckn,Au,nlv): ''' calculate the concentration of unknown. usage: Cup = pcarcalc(Akn,Ckn,Au,nlv) Akn must be matrix, known spectra Au must be a matrix, unknown spectra Ckn must be matrix, conconcentation of known spectra ''' sk,L = svdpcaf(Akn,nlv) ## calculate the Score and Loading vectors P = pinv(sk)@Ckn ## calculate the rotational matrix (P) from known scores and conc. ## PCAR (testing or validating) su = Au@pinv(L) ## calculate the scores for the unknown spectra in (Au) Cup = su@P ## rotate the scores of the unknown sample in to concentrations return Cup ## preprocess the original data here ## preprocess the data rng = np.arange(len(x)) ## crop the data to wavelengths that better obey Beer's Law Aknc = Aknorg[:,rng] ## drop outlier dindex = [] ## list of indices to drop [] means drop none Akn = np.delete(Aknc,dindex,axis=0) Ckn = np.delete(Cknorg,dindex,axis=0) nspec,npts = Akn.shape ## get the shape of the data (do this after preprocessing) R = np.zeros((Ckn.shape),dtype=float) ## create a residuals array # create org matrices for future referencing for i in range(nspec): Au = Akn[i,:] ## set ith to unknown spec Cu = Ckn[i,:] ## set ith to unknown conc C = np.delete(Ckn,i,axis=0) ## drop ith from calibration conc A = np.delete(Akn,i,axis=0) ## drop ith from calibration spectra Cup = kmatconccalc(A,C,Au) ## predict the concentration of the ith unknown spectrum #Cup = pcarcalc(A,C,Au,3) ## predict the concentration of the ith unknown spectrum R[i,:] = Cu - Cup ## calc the residuals (difference from known and prediction) SEPcv = np.sqrt(np.sum(R**2,axis=0)/nspec)##same units as C ## inspect the residuals fig, ax = plt.subplots(figsize=(12,9)) ## figsize is the aspect ratio of the figure #ax.bar(np.array([1.0,2.0]),R.T) sampn = np.arange(nspec) ax.plot(sampn,R/Ckn*100,'*') labels = ['acetaminophen','stearic acid', 'dextrose'] ax.legend(labels) ax.set_xlabel('Sample number') ax.set_ylabel('Percent error /%') ax.axhline(y=0,linestyle='--',color='g',alpha=0.6) ## drop data if outliers ## final concentration prediction of unknown after optimization Cup = kmatconccalc(Akn,Ckn,Auorg) print('Concentrations: ', Cup) print('Uncertainties SEPcv: ', SEPcv)
Assignment 16 (ass16)
Write a function that will determine the SEP\(_{cv}\) for univariate data
- Take 2 inputs
- a set of univariate absorbance values
- a corresponding set of concentration data
- perform cross validation of a set of known spectral absorbances and their concentrations
- using the np.polyfit, and np.polyval functions
- return the SEP
Due date in directory
Principal Component Analysis and Regression
Here are some lecture slides: ./figures/PCA_lecture_deck.pdf Here is a diagram for your learning:
For this part of the tutorial we will use a set of FT-IR spectra of textiles. Here is a figure of the data:
These textile spectra contain X different polymers.
Just looking at the spectra, you can see several differences. For example, the band near 728 cm-1 does not appear in every spectrum. One simple way to inspect the data is to plot the absorbance as 2 different wavenumber locations. In the bottom of the figure above are two examples. The ratios of these 2 pairs of spectral absorbances show some separation or clustering in the lower 2 graphs. If you play around with multiple pairs of spectral absorbance, you will see that lots of combinations of spectral absorbances produce clustering of the coordinates.
If you incorporate more than 2 spectral absorbances into a composite feature, you might be able to generate better separation between the different clusters. This is what PCA does for you.
Here is code to calculate the scores and loading vectors of spectra:
import numpy as np #this command loads all of the functions in numpy and labels them np import matplotlib.pyplot as plt ## this loads the moduule matplotlib.pyplot as plt from numpy.linalg import inv, pinv from scipy.spatial.distance import pdist,squareform import scipy.cluster.hierarchy as sch import scipy.io as sio from scipy.signal import savgol_filter as sg #import sys ## this loads the sys module #sys.path.append('lib/') ## this adds the directory lib in the current directory into the path #from chemo import svdpcaf ## code to get the scores and loading vecotrs fighole = 'figures' ## this is where you write/save your figures datahole = 'data/' ## this is where you store/read you data def svdpcaf2(A,nlv): ''' calculate the scores and loading vectors of a set of data usage: s,L,F = svdpcaf(A,nlv) where A = the matrix of data A.shape = (nspec,npts) nlv = the number of loading vectors (integer) s = scores matrix s.shape = (nspec,nlv) L = is the loading vector matrix L.shape = (nlv, npts) F is the fraction of variance requires ''' import numpy as np from numpy.linalg import svd U,SG, Vt = svd(A,full_matrices=True) S = -1*(U@np.diag(SG)) L = -1*Vt nspec,npts = A.shape F = (SG**2) / (nspec-1) #eig_val = sigma**2 / (A_.shape[0]-1) F = F / F.sum() return S[:,:nlv], L[:nlv,:],F ## setup color dict # use a color tuple # this is a dictionary of colors usage cl[0] yields blue (0,0,1) # you don't have to use this dictionary, I just like it cl = {0:(0, 0, 1),#; %blue 4:(0, 1, 0),#; % bright green 6:(0, 1, 1),#; % cyan 3:(0, 0, 0),#; %black 1:(1, 0, 0),#; % red 7:(1, 1, 0),# % yellow 2:(1, 0, 1),#; % magenta 5:(1, 0.5, 0.5),#; % pink brown 8:(0.75, 1, 0.75),# % dull light green 9:(0.5, 0.5, 1),# % greyblue 10:(1, 0.5, 0),#; % orange 11:(0, 0.8, 0.5),# % bluegreen 12:(0.5, 0, 0.6),# % purple 13:(1,0.75,0),## orange 14:(0.5,0,1), ##also purple, dark 15:(0.78,1,0), ##ugly green 16:(0,0.5,1), #blue ski 17:(1,0,0.25), #pink red 18:(0.39,0.39,0.49), #grey blue 19:(0.02,0.42,0.1), #forest green 20:(0.75,0.26,0.36) #dark red } # load the data from a file ifile = 'data/numclass_activity.mat' ddict = sio.loadmat(ifile) print(ddict.keys()) Aorg = ddict['Aorg'] xorg = ddict['xorg'].T #gl = ddict['gl'] ##group label ## turn group label in to colors tuple for pretty plotting #gl = np.squeeze(gl) # this makes it iterable in the for loop below #colors = [] #for i in gl: # colors.append(cl[i]) ## pre-processing nspec,npts = Aorg.shape ## derivative spectra Ad = np.zeros(Aorg.shape) windowsize = 11 order = 2 deriv = 2 for i in range(nspec): Ad[i,:] = sg(Aorg[i,:], windowsize, polyorder=order, deriv=2) ##copy last preprocessing step to new matrix for analysis A = Ad.copy() ## calculate the scores nlv = 10 ## number of loading vectors (your guess, later we will learn who to choose this number S,L,F = svdpcaf2(A,nlv) ## graph the scores fig, ax2 = plt.subplots(figsize=(8,6)) ## figsize is the aspect ratio of the figure ax2.scatter(S[:,5],S[:,6]) ## if you don't #ax2.scatter(S[:,5],S[:,6],color=colors) ## if you know the groups ofile = 'figures/scoreplotdemo.png' fig.savefig(ofile,transparent=False,papertype='letter',orientation='bottom') ##### Michael you don't need this part so don't copy it return ofile
If you want to generate all of the score plots at once you can use the scattermatrix command that is part of the pandas module. So an additional step is the conversion of the Scores matrix into a dataframe. Here is the code:
import numpy as np #this command loads all of the functions in numpy and labels them np import matplotlib.pyplot as plt ## this loads the moduule matplotlib.pyplot as plt import pandas as pd from numpy.linalg import inv, pinv from scipy.spatial.distance import pdist,squareform import scipy.cluster.hierarchy as sch import scipy.io as sio #import sys ## this loads the sys module #sys.path.append('lib/') ## this adds the directory lib in the current directory into the path #from chemo import svdpcaf fighole = 'figures' ## this is where you write/save your figures datahole = 'data/' ## this is where you store/read you data def svdpcaf2(A,nlv): ''' calculate the scores and loading vectors of a set of data usage: s,L,F = svdpcaf(A,nlv) where A = the matrix of data A.shape = (nspec,npts) nlv = the number of loading vectors (integer) s = scores matrix s.shape = (nspec,nlv) L = is the loading vector matrix L.shape = (nlv, npts) F is the fraction of variance requires ''' import numpy as np from numpy.linalg import svd U,SG, Vt = svd(A,full_matrices=True) S = -1*(U@np.diag(SG)) L = -1*Vt nspec,npts = A.shape F = (SG**2) / (nspec-1) #eig_val = sigma**2 / (A_.shape[0]-1) F = F / F.sum() return S[:,:nlv], L[:nlv,:],F ## setup color dict # use a color tuple # this is a dictionary of colors usage cl[0] yields blue (0,0,1) # you don't have to use this dictionary, I just like it cl = {0:(0, 0, 1),#; %blue 4:(0, 1, 0),#; % bright green 6:(0, 1, 1),#; % cyan 3:(0, 0, 0),#; %black 1:(1, 0, 0),#; % red 7:(1, 1, 0),# % yellow 2:(1, 0, 1),#; % magenta 5:(1, 0.5, 0.5),#; % pink brown 8:(0.75, 1, 0.75),# % dull light green 9:(0.5, 0.5, 1),# % greyblue 10:(1, 0.5, 0),#; % orange 11:(0, 0.8, 0.5),# % bluegreen 12:(0.5, 0, 0.6),# % purple 13:(1,0.75,0),## orange 14:(0.5,0,1), ##also purple, dark 15:(0.78,1,0), ##ugly green 16:(0,0.5,1), #blue ski 17:(1,0,0.25), #pink red 18:(0.39,0.39,0.49), #grey blue 19:(0.02,0.42,0.1), #forest green 20:(0.75,0.26,0.36) #dark red } # load the data from a file ifile = 'data/numclass_activity.mat' #ifile = 'data/tablets_groups_raman.mat' ddict = sio.loadmat(ifile) print(ddict.keys()) Aorg = ddict['Aorg'] xorg = ddict['xorg'].T ## turn group label in to colors tuple for pretty plotting #gl = np.squeeze(ddict['gl']) ##group label, np.squeeze makes it iterable in the for loop below #colors = [] #for i in gl: # colors.append(cl[i]) ## pre-processing ## calculate the scores nlv = 10 ## number of loading vectors (your guess, later we will learn who to choose this number S,L = svdpcaf(Aorg,nlv) ## graph the scores #fig, ax2 = plt.subplots(figsize=(8,6)) ## figsize is the aspect ratio of the figure #ax2.scatter(S[:,5],S[:,6]) ## if you don't #ax2.scatter(S[:,5],S[:,6],color=colors) ## if you know the groups df = pd.DataFrame(S) ## convert the score matrix into a dataframe #ax3 = pd.plotting.scatter_matrix(df,color=colors) ax3 = pd.plotting.scatter_matrix(df) ofile = 'figures/scattermatrix_demo.png' plt.savefig(ofile,transparent=False,papertype='letter',orientation='bottom') ##### Michael you don't need this part so don't copy it return ofile
PCA-R
One limitation of the K-matrix method is that it requires obedience to Beer's law. Specifically, all components that contribute to the spectra must be included in the model (K). Whenever some components are left out of the training data then the predictive ability of the model is diminished. In the real world, this limitation is not a problem for very controlled systems where all of the mixture components are well known such as in pharmaceutical tablets. However, in complex mixtures (especially those created by nature) such as blood, corn, and beer, establishing a calibration system for all components that might contribute to the spectra is nearly impossible. In these cases you have to use an inverse regression, such as PCA-R or partial least squares regression (PLS-R).
Theory Inverse calibration with PCA-R
First regress A on C; this is called an Inverse calibration.
\begin{equation} C = A P \end{equation}where C is the concentrations, A is the spectra, and P is the regression coefficients (least squares fit parameters).
Procedure
First the known spectra Akn are decomposed into 2 matrices: the scores matrix (S) and the loading vectors (L).
\begin{equation} A_{kn} = S_{kn} L_{kn} \end{equation}The loading vectors in L are a set of vectors that represent all of the spectral features that are found in all of the data (A). There are several important properties of L:
- A has dimensions (nspec, npts) where nspec are the number of spectra, npts is the number of wavelengths
- L has dimensions (nlv, npts) where nlv is the number of loading vectors and is ideally, almost always << nspec
- this nlv << nspec is a benefit for PCA-R, it is a data reduction technique.
- L is the model that represents the data, models are simpler, and smaller than real-life
- as you step through the loading vectors in L [0 \(\rightarrow\) nlv] each loading vector
- is orthogonal to all of the others:2 vectors that are orthogonal can not be projected onto each other; or do not contain overlapping information
- represent the decreasing variance (set of changes) in the data A
- the 1st loading vector represents the largest set of changing spectral features in that in A
- the 2nd loading vector represents the second largest set of changing spectral feature that are in A and not in the 1st loading vector
- etc until you get to nlv
- S are the weights (coefficients) of the loading vectors and have the following properties
- the dimensions of S are (nspec, nlv)
- Each row in S (score vector) is the weight of the loading vectors in L
- since L represents all of the information in A in a smaller form, each score vector can be a representative (more compact) of each spectrum in A
We can substitute the known Scores (S) for the known spectra (A). This equation defines a relationship between the Concentrations and the scores which is found in this equation.
\begin{equation} C_{kn} = S_{kn} P \end{equation}by solving for P.
\begin{equation} P = S_{kn}^{+} C_{kn} \end{equation}where S+ is the pseudo-inverse of S, and Ckn is the known concentration matrix.
Then we use the model L, which was determined from the known data, to determine the scores for unknown spectra (Au)
\begin{equation} S_{u} = A_{u} L_{kn}^{+} \end{equation}where Lkn+ is the pseudo-inverse of L.
Then finally, the concentration can be calculated from Su and P
\begin{equation} C_{u} = S_{u} P \end{equation}Code
Here is some python code for performing PCA-R and also a comparison of the K-matrix method. In this data set, the two components in the mixture from complexes whose concentrations are not knowable. Compare how the K-matrix and PCA-R methods perform.
import numpy as np ##this loads the module numpy as np import pandas as pd ##this loads the module pandas as pd import matplotlib as mpl ## this loads the moduule matplotlib as mpl import matplotlib.pyplot as plt ## this loads the moduule matplotlib.pyplot as plt import scipy.io as sio from numpy.linalg import pinv,inv import sys ## this loads the sys module sys.path.append('lib/') ## this adds the directory lib in the current directory into the path from chemo import svdpcaf import analtool3 as AT from chemo import svdpcaf fighole = 'figures' ## this is where you write/save your figures datahole = 'data/' ## this is where you store/read you data ## define some functions def svdpcaf2(A,nlv): ''' calculate the scores and loading vectors of a set of data usage: s,L = svdpcaf(A,nlv) where A = the matrix of data A.shape = (nspec,npts) nlv = the number of loading vectors (integer) s = scores matrix s.shape = (nspec,nlv) L = is the loading vector matrix L.shape = (nlv, npts) requires ''' import numpy as np from numpy.linalg import svd U,SG, Vt = svd(A,full_matrices=True) S = -1*(U@np.diag(SG)) L = -1*Vt nspec,npts = A.shape F = (SG**2) / (nspec-1) #eig_val = sigma**2 / (A_.shape[0]-1) F = F / F.sum() return S[:,:nlv], L[:nlv,:],F def pcarcalc(Akn,Ckn,Au,nlv): ''' calculate the concentration of unknown. usage: Cup = pcarcalc(Akn,Ckn,Au,nlv) Akn must be matrix, known spectra Au must be a matrix, unknown spectra Ckn must be matrix, conconcentation of known spectra ''' sk,L,F = svdpcaf2(Akn,nlv) ## calculate the Score and Loading vectors P = pinv(sk)@Ckn ## calculate the rotational matrix (P) from known scores and conc. ## PCAR (testing or validating) su = Au@pinv(L) ## calculate the scores for the unknown spectra in (Au) Cup = su@P ## rotate the scores of the unknown sample in to concentrations return Cup # load the data from a file ifile = 'data/tolpxyl.mat' ddict = sio.loadmat(ifile) print(ddict.keys()) Aorg = ddict['Aorg'] xorg = ddict['xorg'].T Ctrue = ddict['Ctrue'] uidx = 2 ## simulate an unknown spectrum Ckn = np.matrix(Ctrue.copy()) Akn = np.matrix(Aorg.copy()) Ckn = np.delete(Ckn,uidx,axis=0) Akn = np.delete(Akn,uidx,axis=0) #FAKE unknown or separate validation Au = np.matrix(Aorg[uidx,:]) Cknu = np.matrix(Ctrue[uidx,:]) print('Actual unknown conc: ', Cknu) ## pre-processing ## K-matrix for comparision K = pinv(Ckn)@Akn ## solve for the model Cu = Au@pinv(K) ## determine conc. of samples of unknown concentration print('K-matrix predictions: ', Cu) ## PCAR (training) nlv = 2 ## number of loading vectors to use in the model sk,L = svdpcaf(Akn,nlv) ## calculate the Score and Loading vectors P = pinv(sk)@Ckn ## calculate the rotational matrix (P) from known scores and conc. ## PCAR (testing or validating) su = Au@pinv(L) ## calculate the scores for the unknown spectra in (Au) Cu = su@P ## rotate the scores of the unknown sample in to concentrations ## output results print('PCAR predictions: ', Cu)
Cluster Analysis
The goal of cluster analysis is to deduce the relationship between objects or more specifically the structure of these relationships. In biology, this structure is called taxonomy. So we are building the taxonomy of the objects, or a set of groups of objects. In this sentence, object is a vector of data that is characteristic of the sample. Some examples could be an IR, Raman, NMR, MS spectrum of a sample. These 4 spectroscopic techniques are highly information rich, and are therefore good at discriminating between sample that are similar, but different. Another example, might be a vector of concentrations of components (elements, molecules, etc.) in a solution or material.
Exploratory data analysis often includes some kind of cluster analysis. It also usually involves creating visual representations of the data.
In classification, in contrast to cluster analysis, the groups of similar objects are already established, and you (the scientist) are determining into which group the unknown object belongs.
The first thing you should do whenever you suspect that your data has groups is to graph it.
Data for this section
Here is some data that we are going to use in this section. Concentrations of calcium and phosphate in six blood serum samples.
| Object ID | Calcium /mg/100 mL | Phosphate /mg/100 mL |
|---|---|---|
| 1 | 8.0 | 5.5 |
| 2 | 8.25 | 5.75 |
| 3 | 8.7 | 6.3 |
| 4 | 10.0 | 3.0 |
| 5 | 10.25 | 4.0 |
| 6 | 9.75 | 3.5 |
# many functions are avaible in modules or libraries # in this example we will load the numpy module of functions import numpy as np #this command loads all of the functions in numpy and labels them np import pandas as pd # data organization module import matplotlib.pyplot as plt # plotting module ## get data p = np.array([8.0000, 8.2500 , 8.7000, 10.0000 ,10.2500, 9.7500]) ca = np.array([ 5.5000, 5.7500, 6.3000,3.0000,4.0000,3.5000]) labels = [1,2,3,4,5,6] ## process data ## plot data fig, ax = plt.subplots(figsize=(12,9)) ## figsize is the aspect ratio of the figure ax.scatter(p,ca) ## ax is the variable for the axis where you want to work ax.set_xlabel('Concentration of Calcium cation /mg/100 mL') ## latex works here ax.set_ylabel('Concentration of Phosphate /mg/100 mL') ofile = 'figures/scatter_blood.png' fig.savefig(ofile,dpi=100) ## dpi is dots per inch and is a figure resotution thing ### Michael!!!! This part is for making this website, so you don't need it return ofile
Notice how the data has separated into two groups. So it looks like these blood samples are from 2 difference kinds of people. Some questions still remain:
- Is the separation of these two groups statistically significant or is this just a random event?
- How do we know that these two groups are separated from each other?
- How do you look at data that has more than 3 variables? (blood has more than two components, spectra have more than 2 wavelengths, etc)
Distance and Similarity Measurements
The similarity of objects can be established by several distance measurements of either the raw data (especially if the raw is proportional to quantity) or from results (concentrations of specific elements or compounds or groups of compounds e.g. octane number, protein concentration, etc.).
In the library searching section of this tutorial, we used some of these distance and similarity measurements to compare an unknown spectrum to a known spectrum (one at a time), and ranked them by their similarity. That kind of search is a form of classification. In this section we will use these distance measurements for cluster analysis. We will assume that we don't know very much about the data, specifically how many different types of materials are present.
General distance measurement or Minkowski Distance
where K is the number of variables and i, j are the object indices.
One problem or limitation with these general distance measures is that the scale of each metric is impactful for the distance (think of the impact of adding one to a very large number on the result i.e. 10000000+1 ∼ 10000000). So often the data will need to be scaled, or autoscaled or preprocessed in some way.
Also, whenever you have a set of data and you what to know which are similar to each other, you have to measure each data to all other others. This is called a pair-wise comparison and results in an array of pair-wise distances. We can create a heatmap to show the distances.
Manhattan or City block Distance (p=1)
The Minkowski distance when p = 1 is also called the Manhattan or city-block distance. Here is the code for its calculation. Given the blood serum data as an example. This code will also create an image.
import numpy as np #this command loads all of the functions in numpy and labels them np from numpy.linalg import inv, pinv from scipy.spatial.distance import pdist,squareform # setup matrices (with data this would be the step where you load the data from a file) p = np.array([8.0000, 8.2500 , 8.7000, 10.0000 ,10.2500, 9.7500]) #phosphate ca = np.array([ 5.5000, 5.7500, 6.3000,3.0000,4.0000,3.5000]) # calcium raw = np.vstack((p,ca)).T print('Raw data shape: ', raw.shape) ##calculate the pair-wise distantances met = 'cityblock' cd =pdist(raw,metric=met) print(cd) cd = squareform(cd) print('Manhattan Pairwise distances: ', cd)
Notice the diagonal. Does this pattern make sense.
Here is the code to generate the heat map.
import numpy as np #this command loads all of the functions in numpy and labels them np import matplotlib.pyplot as plt ## this loads the moduule matplotlib.pyplot as plt from numpy.linalg import inv, pinv from scipy.spatial.distance import pdist,squareform import scipy.cluster.hierarchy as sch import scipy.io as sio from scipy.signal import savgol_filter as sg ## some helper functions def heatmap(data, row_labels, col_labels, ax=None, cbar_kw={}, cbarlabel="", **kwargs): """ Create a heatmap from a numpy array and two lists of labels. Parameters ---------- data A 2D numpy array of shape (N, M). row_labels A list or array of length N with the labels for the rows. col_labels A list or array of length M with the labels for the columns. ax A `matplotlib.axes.Axes` instance to which the heatmap is plotted. If not provided, use current axes or create a new one. Optional. cbar_kw A dictionary with arguments to `matplotlib.Figure.colorbar`. Optional. cbarlabel The label for the colorbar. Optional. **kwargs All other arguments are forwarded to `imshow`. """ if not ax: ax = plt.gca() # Plot the heatmap im = ax.imshow(data, **kwargs) # Create colorbar cbar = ax.figure.colorbar(im, ax=ax, **cbar_kw) cbar.ax.set_ylabel(cbarlabel, rotation=-90, va="bottom") # We want to show all ticks... ax.set_xticks(np.arange(data.shape[1])) ax.set_yticks(np.arange(data.shape[0])) # ... and label them with the respective list entries. ax.set_xticklabels(col_labels) ax.set_yticklabels(row_labels) # Let the horizontal axes labeling appear on top. ax.tick_params(top=True, bottom=False, labeltop=True, labelbottom=False) # Rotate the tick labels and set their alignment. plt.setp(ax.get_xticklabels(), rotation=-30, ha="right", rotation_mode="anchor") # Turn spines off and create white grid. for edge, spine in ax.spines.items(): spine.set_visible(False) ax.set_xticks(np.arange(data.shape[1]+1)-.5, minor=True) ax.set_yticks(np.arange(data.shape[0]+1)-.5, minor=True) ax.grid(which="minor", color="w", linestyle='-', linewidth=3) ax.tick_params(which="minor", bottom=False, left=False) return ax def annotate_HM(ax,d): ''' This is a quick function to annotate a heat map. ax = annotate_HM(ax,d) where ax is the axis to annotate d is the distance squareform array ''' for i in range(len(labels)): for j in range(len(labels)): text = ax.text(j, i, '{:0.2f}'.format(cd[i, j]),ha="center", va="center", color="w") return ax fighole = 'figures' ## this is where you write/save your figures datahole = 'data/' ## this is where you store/read you data p = np.array([8.0000, 8.2500 , 8.7000, 10.0000 ,10.2500, 9.7500]) #phosphate ca = np.array([ 5.5000, 5.7500, 6.3000,3.0000,4.0000,3.5000]) # calcium labels = [1,2,3,4,5,6] ## labels can be words, filesnames, numbers, letters whatever raw = np.vstack((p,ca)).T ##calculate the pair-wise distances met = 'cityblock' cd =squareform( pdist(raw,metric=met) ) ## graph the scores fig, ax = plt.subplots(figsize=(8,8)) ## figsize is the aspect ratio of the figure fig, ax = plt.subplots() ## figsize is the aspect ratio of the figure ax = heatmap(cd, labels, labels, ax=ax, cmap="jet", cbarlabel="City-block distance") ax = annotate_HM(ax,cd) ofile = 'figures/heatmap_cd.png' fig.savefig(ofile,transparent=False,papertype='letter',orientation='bottom') ##### Michael you don't need this part so don't copy it return ofile
Notice the diagonal. Does this pattern make sense.
Euclidean Distance (p=2)
The Minkowski distance, when p = 2, is also called the Euclidean distance. Here is the code for its calculation. Given the blood serum data as an example.
import numpy as np #this command loads all of the functions in numpy and labels them np from numpy.linalg import inv, pinv from scipy.spatial.distance import pdist,squareform # setup matrices (with data this would be the step where you load the data from a file) p = np.array([8.0000, 8.2500 , 8.7000, 10.0000 ,10.2500, 9.7500]) #phosphate ca = np.array([ 5.5000, 5.7500, 6.3000,3.0000,4.0000,3.5000]) # calcium raw = np.vstack((p,ca)).T print('Raw data shape: ', raw.shape) ##calculate the pair-wise distantances met = 'euclidean' ed =pdist(raw,metric=met) ed = squareform(ed) print('Euclidean Pairwise distances: ', ed)
Notice the diagonal. Does this pattern make sense.
Mahalanobis distance
One distance measurement that is not susceptible to scaling issues the way Minkowski distance methods are and their internal correlations is the Mahalanobis distance.
The main feature difference with Mahalanobis distance is that distortions arising from correlations and co-linearity of features (absorbance values, concentrations, etc.) are avoided. Highly correlating features means that the information represented in the features is similar or the same.
Here is the equation.
\begin{equation} D_{ij}^2 = (x_i - x_j) C^{-1} (x_i - x_j)^T \end{equation}where C is the covariance matrix and xi and xj are the vectors for objects i and j.
Here is the code to do this.
import numpy as np #this command loads all of the functions in numpy and labels them np from numpy.linalg import inv, pinv from scipy.spatial.distance import pdist,squareform # setup matrices (with data this would be the step where you load the data from a file) p = np.array([8.0000, 8.2500 , 8.7000, 10.0000 ,10.2500, 9.7500]) #phosphate ca = np.array([ 5.5000, 5.7500, 6.3000,3.0000,4.0000,3.5000]) # calcium raw = np.vstack((p,ca)).T print('Raw data shape: ', raw.shape) ##calculate the pair-wise distantances md = pdist(raw,metric='mahalanobis') md = squareform(md) print('Mahalanobis Pairwise distances: ', md)
Similarity
This is sort of the opposite of the distance measurements and is often more intuitive for the observer.
\begin{equation} S_{ij} = 1 - \frac{d_{ij}}{d_{max}} \end{equation}where dij is any distance between object i and j and dmax is the maximum distance found in the data set.
Within a data set, completely Similar objects have an Sij of 1 and dissimilar objects have an Sij = 0.
One limitation of similarity measurements is that you have to have the whole data set to generate similarity; you can not just calculate the similarity of 2 spectra.
Hierarchical cluster analysis
This type of analysis is useful if you don't know how many groups you have in your data. The procedure below is generic. If you need you can think of the word object as data (spectrum, concentrations) from a single sample. The algorithm for Hierarchical cluster analysis goes like this:
- calculate every pair-wise distance (see previous section). This is the beginning of the loop
- join the two objects that are have the closest distances into a new object
- Here, join can mean many things, but a common technique is to average their coordinates.
- this joining is also called a linkage
- recalculate every new pair-wise distance with the new set of objects
- join the the two objects that have the closest distances into a new object (loop until you have one object)
- graph these linkages using a dendrogram
Dendrograms are graphs that look a little like a tree or root system. Here is an example
The colors for different leaves are typically chosen by a height/distance threshold. In the example above we tell the algorithm that leaves whose branches are greater than 3 is a called a group and are given a different color.
- different groups can be assigned by the distances between them
There are several strategies for defining groups.
- if you have some a priori knowledge about the structure of the groups, use these as a guide.
- If you use Mahalanobis Distance, then 3 Mahalanobis distances is statistically the same as 3 standard deviations.
some code
import numpy as np #this command loads all of the functions in numpy and labels them np import matplotlib.pyplot as plt ## this loads the moduule matplotlib.pyplot as plt from numpy.linalg import inv, pinv from scipy.spatial.distance import pdist,squareform import scipy.cluster.hierarchy as sch ## scipy's hierarchial classification # setup matrices (with data this would be the step where you load the data from a file) p = np.array([8.0000, 8.2500 , 8.7000, 10.0000 ,10.2500, 9.7500]) #phosphate ca = np.array([ 5.5000, 5.7500, 6.3000,3.0000,4.0000,3.5000]) # calcium labels = ['A','B','C','D','E','F'] ## labels can be words, filesnames, numbers, letters whatever raw = np.vstack((p,ca)).T print('Raw data shape: ', raw.shape) ## setup the figure (creating an axis to graph on (ax) fig = plt.figure(figsize=(8,8)) ax = fig.add_subplot(1,1,1) ax.set_ymargin(1) ######### Compute and plot the dendrogram. ## calculate distances and linkages #Y = sch.linkage(raw, method='ward',metric='euclidean') Y = sch.linkage(raw, method='average',metric='euclidean') #Y = sch.linkage(raw, method='average',metric='mahalanobis') ## graph the linkages into a dendrogram Z2 = sch.dendrogram(Y,labels=labels,color_threshold=1, show_leaf_counts=False, ax=ax) #craft the graph #labels = ax.get_xticklabels() #use this to generate labels from sample index #ax.set_xticklabels(labels,rotation=0, fontsize=14) ax.set_xlabel('Sample label') ax.set_ylabel('Euclidean Distance') ofile = 'figures/hicluster_blood.png' fig.savefig(ofile,transparent=False,papertype='letter',orientation='bottom') ##### Michael you don't need this part so don't copy it return ofile
Combining PCA and Hierarchical Clustering
Whenever your objects (vectors of information about each sample) are spectra that has lots of similar information. In the example below the absorbance values are highly co-linear. A scatter-plot graph of different absorbance values across all spectra in the data shows a general trend from the lower left corner to the upper right corner. This is co-linear data, which means that lots of the absorbance values contain either the same or very similar information about the concentrations of the data. (information overlap)
Figure 26: Why use PCA figure
It is often advisable to apply information techniques that will remove the co-linearity. On strategy for this is to utilize principal component analysis to reduce the dimensionality of the original data. This specifically means to use the scores as a substitution for the original spectral data (intensity, absorbance, etc.). Review the PCAR section for why this works.
Here is some code:
import numpy as np #this command loads all of the functions in numpy and labels them np import matplotlib.pyplot as plt ## this loads the moduule matplotlib.pyplot as plt import pandas as pd from numpy.linalg import inv, pinv from scipy.spatial.distance import pdist,squareform import scipy.cluster.hierarchy as sch import scipy.io as sio import sys ## this loads the sys module sys.path.append('lib/') ## this adds the directory lib in the current directory into the path from chemo import svdpcaf fighole = 'figures' ## this is where you write/save your figures datahole = 'data/' ## this is where you store/read you data ## setup color dict # use a color tuple # this is a dictionary of colors usage cl[0] yields blue (0,0,1) # you don't have to use this dictionary, I just like it cl = {0:(0, 0, 1),#; %blue 4:(0, 1, 0),#; % bright green 6:(0, 1, 1),#; % cyan 3:(0, 0, 0),#; %black 1:(1, 0, 0),#; % red 7:(1, 1, 0),# % yellow 2:(1, 0, 1),#; % magenta 5:(1, 0.5, 0.5),#; % pink brown 8:(0.75, 1, 0.75),# % dull light green 9:(0.5, 0.5, 1),# % greyblue 10:(1, 0.5, 0),#; % orange 11:(0, 0.8, 0.5),# % bluegreen 12:(0.5, 0, 0.6),# % purple 13:(1,0.75,0),## orange 14:(0.5,0,1), ##also purple, dark 15:(0.78,1,0), ##ugly green 16:(0,0.5,1), #blue ski 17:(1,0,0.25), #pink red 18:(0.39,0.39,0.49), #grey blue 19:(0.02,0.42,0.1), #forest green 20:(0.75,0.26,0.36) #dark red } # load the data from a file ifile = 'data/numclass_activity.mat' ddict = sio.loadmat(ifile) print(ddict.keys()) Aorg = ddict['Aorg'] xorg = ddict['xorg'].T #gl = ddict['gl'] ##group label ## turn group label in to colors tuple for pretty plotting #gl = np.squeeze(gl) # this makes it iterable in the for loop below #colors = [] #for i in gl: #colors.append(cl[i]) ## pre-processing ## calculate the scores nlv = 10 ## number of loading vectors (your guess, later we will learn who to choose this number S,L = svdpcaf(Aorg,nlv) #df = pd.DataFrame(S) ## convert the score matrix into a dataframe fig = plt.figure(figsize=(8,8)) ax = fig.add_subplot(1,1,1) ax.set_ymargin(1) # Compute and plot the dendrogram. Y = sch.linkage(S, method='weighted',metric='euclidean') #Y = sch.linkage(df, method='average',metric='mahalanobis') Z2 = sch.dendrogram(Y,color_threshold=1, show_leaf_counts=False,ax=ax) #ax2.set_xticklabels(ids) #ax2.set_xticklabels(sdf.index) plt.xticks(rotation=10) ax.set_xlabel('Sample Number') ax.set_ylabel('Euclidean Distance') ofile = 'figures/scoresplushicluster.png' plt.savefig(ofile,transparent=False,papertype='letter',orientation='bottom') ##### Michael you don't need this part so don't copy it return ofile
Classification
In the Cluster Analysis section we discussed briefly the difference between cluster analysis and classification. In cluster analysis, we are trying to find and define groups of data. In classification, the groups are known and we are trying to assign unknown data into a set group. As a result in this section, we will assume that the group members of known data is already known either through some external means or from cluster analysis.
Here is some code that combines all of the steps to identify an unknown set of data into groups.
import numpy as np ##this loads the module numpy as np import pandas as pd ##this loads the module pandas as pd import matplotlib as mpl ## this loads the moduule matplotlib as mpl import matplotlib.pyplot as plt ## this loads the moduule matplotlib.pyplot as plt import scipy.io as sio from scipy.signal import savgol_filter as sg from scipy.spatial.distance import pdist,squareform import scipy.cluster.hierarchy as sch from numpy.linalg import pinv,inv ## define some functions def svdpcaf2(A,nlv): ''' calculate the scores and loading vectors of a set of data usage: s,L = svdpcaf(A,nlv) where A = the matrix of data A.shape = (nspec,npts) nlv = the number of loading vectors (integer) s = scores matrix s.shape = (nspec,nlv) L = is the loading vector matrix L.shape = (nlv, npts) requires ''' import numpy as np from numpy.linalg import svd U,SG, Vt = svd(A,full_matrices=True) S = -1*(U@np.diag(SG)) L = -1*Vt nspec,npts = A.shape F = (SG**2) / (nspec-1) #eig_val = sigma**2 / (A_.shape[0]-1) F = F / F.sum() return S[:,:nlv], L[:nlv,:],F def heatmap(data, row_labels, col_labels, ax=None, cbar_kw={}, cbarlabel="", **kwargs): """ Create a heatmap from a numpy array and two lists of labels. Parameters ---------- data A 2D numpy array of shape (N, M). row_labels A list or array of length N with the labels for the rows. col_labels A list or array of length M with the labels for the columns. ax A `matplotlib.axes.Axes` instance to which the heatmap is plotted. If not provided, use current axes or create a new one. Optional. cbar_kw A dictionary with arguments to `matplotlib.Figure.colorbar`. Optional. cbarlabel The label for the colorbar. Optional. **kwargs All other arguments are forwarded to `imshow`. """ if not ax: ax = plt.gca() # Plot the heatmap im = ax.imshow(data, **kwargs) # Create colorbar cbar = ax.figure.colorbar(im, ax=ax, **cbar_kw) cbar.ax.set_ylabel(cbarlabel, rotation=-90, va="bottom") # We want to show all ticks... ax.set_xticks(np.arange(data.shape[1])) ax.set_yticks(np.arange(data.shape[0])) # ... and label them with the respective list entries. ax.set_xticklabels(col_labels) ax.set_yticklabels(row_labels) # Let the horizontal axes labeling appear on top. ax.tick_params(top=True, bottom=False, labeltop=True, labelbottom=False) # Rotate the tick labels and set their alignment. plt.setp(ax.get_xticklabels(), rotation=-90, ha="right", rotation_mode="default") # Turn spines off and create white grid. for edge, spine in ax.spines.items(): spine.set_visible(False) ax.set_xticks(np.arange(data.shape[1]+1)-.5, minor=True) ax.set_yticks(np.arange(data.shape[0]+1)-.5, minor=True) ax.grid(which="minor", color="w", linestyle='-', linewidth=3) ax.tick_params(which="minor", bottom=False, left=False) return im, cbar def annotate_HM(ax,labels,d): ''' This is a quick function to annotate a heat map. ax = annotate_HM(ax,d) where ax is the axis to annotate d is the distance squareform array ''' for i in range(len(labels)): for j in range(len(labels)): text = ax.text(j, i, '{:0.2f}'.format(d[i, j]),ha="center", va="center", color="w") return ax # load the data from a file ifile = 'data/IR_which_marzipan_recipe.mat' mdict = sio.loadmat(ifile) print(mdict.keys()) Akn = mdict['Akn'] grp = mdict['grp'] xorg = mdict['xorg'] Au = mdict['Au'] labels= mdict['labels'] ### make color array to color code groups (r,g,b) colordict = {'A':(1,0,0), ## red 'B':(0,0,1) ## blue } colors = [] for g in grp: colors.append(colordict[g]) ## pre-processing ## crop the data rng = np.arange(542,929) ## fingerprint information rich region of spectrum #rng = np.arange(len(Akn[0,:])) ## whole spectrum Ac = Akn[:,rng] ## crop to fingerprint region Auc = Au[:,rng] ## crop unknown to the same region xc = xorg[rng] ## crop the x to the same region for plotting ## 2nd derivative (baseline de-emphasizing) windowsize = 17 ## smoothing function size porder = 3 ##polynomial order must be larger than derivative dorder = 2 ## derivative order 0 = smooth, 1 = 1st derivative, 2 = 2nd derivative Ad = sg(Ac,windowsize,polyorder=porder,deriv=dorder) ## sg known data Aud = sg(Auc,windowsize,polyorder=porder,deriv=dorder) ## sg unknown data same way ## pair-wise Euclidean distance on known spectra,look for sets of spectra that are close in ED distance, ## these will be possible similar data met = 'euclidean' ed =squareform( pdist(Ad,metric=met) ) ## calculate the linkages from the distances Y = sch.linkage(Ad, method='average',metric='euclidean') ## PCA of known data nspec, npts = Ad.shape #calculate the scores nlv = 6 s,L,F = svdpcaf2(Ad,nlv) s = np.array(s) ## use the loading vectors for the known data to generate scores for the unknown data Aud = np.matrix(Aud) ## convert array to matrix su = np.array(Aud*pinv(L)) ## convert to array for plotting purposes ## make a bunch of graphs fig, axs = plt.subplots(3, 2, sharex=False, sharey=False ,gridspec_kw={'hspace': 0.5, 'wspace': 0.5},figsize=(18,18)) (ax1, ax2), (ax3, ax4), (ax5,ax6) = axs ## rawdata crap = ax1.plot(xorg,Akn.T) ax1.invert_xaxis() ax1.set_xlabel('Wavenumber /cm$^{-1}$') ax1.set_ylabel('Absorbance') ax1.set_title('Raw data') ## cropped data crap = ax2.plot(xc,Ac.T) ax2.invert_xaxis() ax2.set_xlabel('Wavenumber /cm$^{-1}$') ax2.set_ylabel('Absorbance') ax2.set_title('Cropped data') ## derivative spectra crap = ax3.plot(xc,Ad.T) ax3.invert_xaxis() ax3.set_xlabel('Wavenumber /cm$^{-1}$') ax3.set_ylabel('2$^{nd}$Derivative \n Absorbance') ax3.set_title('Derivative Spectra') ## heatmap im = heatmap(ed, labels, labels, ax=ax4, cmap="jet", cbarlabel="Euclidean distance") ## HCA Z = sch.dendrogram(Y,labels=labels,color_threshold=0.01, show_leaf_counts=False, ax=ax5) ## scoreplot ## plot the scores r = 0 c = 1 ax6.scatter(s[:,r],s[:,c],25,colors) ## plot the known scores ax6.scatter(su[:,r],su[:,c],25,'k') ## plot the unknown scores ax6.set_xlabel('Score 0') ax6.set_ylabel('Score 1') #add the text labels to the coordinates for i in range(nspec): #ax.text(s[i,r],s[i,c],str(np.arange(nspec)[i])) #ax.text(s[i,r],s[i,c],labels[i][-1]) ax6.text(s[i,r],s[i,c],grp[i][-1]) ## add grp label to points on scatter plot ofile = 'figures/classificaiton_example.png' fig.savefig(ofile,dpi=300) return ofile
Assignment 17 (ass17)
In your ass17 directory is a Jupyter notebook and a data file named ceramicelementaldata.xlsx. In this excel file is a set of concentrations in ppm of elements found in ceramics found from antiquity. Your assignment is to determine how many groups of ceramics there are in this data set. Do all of your work in the provided jupyter notebook. Write in Markdown in the space provided how many groups that you found. Justify you answer with both words and graphics. You may use any of the tools that we have talked about in this tutorial.
Final Assignment
In your final directory is a Jupyter notebook and a data file named ceramicelementaldata.xlsx. In this excel file is a set of concentrations in ppm of elements determined in ceramics found from antiquity. Your assignment is to determine how many groups of ceramics there are in this data set. Do all of your work in the provided jupyter notebook. Write in Markdown in the space provided how many groups that you found. Justify you answer with both words and graphics. You may use any of the tools that we have talked about in this tutorial.