compsci-missing_semester_2020/_subsections/lesson-04.org

Lesson 04 | Data Wrangling

• Links
  • timestamps
• notes
  • intro example
  • SED
    • example
    • sed modifiers
  • regex debugger
  • sort
  • awk
  • paste
  • berkley calculator
  • compute statistics
  • gnuplot
  • xargs

Links

• TOC | Missing Semester
• Playlist: Missing Semester
• class notes
• Curr: https://youtu.be/sz_dsktIjt4?si=XopbHGTFXY-I6Bkh&t=2577

timestamps

• 00:00 - introduction
• 06:55 - Stream Editor
• 07:36 - Replacement Expressions
• 08:58 - Regular Expression
• 09:20 - Regular Expressions
• 10:20 - Square Brackets
• 11:33 - Add Modifiers
• 12:56 - Alternations
• 17:09 - Anchoring the Regular Expression
• 18:58 - Capture Groups
• 20:15 - Regular Expression Debugger
• 24:10 - Regular Sessions
• 26:01 - Match and Email Address
• 29:03 - Sort
• 34:00 - Awk
• 38:44 - Berkeley Calculator
• 40:37 - Computer Statistics over Inputs
• 41:22 - Summary Statistics
• 42:50 - Two sort of special types current
• 41:53 - Plotting
• 45:54 - example where data wrangling is useful
• 46:45 - image captures to standard output
• 47:26 - operate on standard input
• 48:00 - display in an image display

notes

intro example

• using ssh someserver 'somecommand' will run that command on the server

• you could run a series of commands on the server instead of channeling all that info back

  ssh someserver 'journalctl | grep sshd | grep "Disconnected from"' | less

• this will run journalctl on the server, find anything that says 'sshd' and 'disconnec..' in the results
• then send all those results back to our machine where we pipe it through 'less'

SED

• stream editor
• allows you to make changes to the contents of a stream
• full programming langauge
• common task is to run replacement expressions on an input stream

example

sed 's/.*blahblah blah//'

• uses regular expressions

  • way of matching text

sed modifiers

• (ab)* - remove zero or more of 'ab'
• -E use new replacement
• (ab|bc)* - remove 'ab' or 'bc'

regex debugger

• regex101.com

sort

• can sort by column
• sorts ascending by default

awk

• programming language
• focused on columnar data
• can match by pattern

paste

• takes input and puts it together how you want
• '-s' :: single line
• '-d' :: delimiter

berkley calculator

• calculator that reads from stdin

compute statistics

• R language is built for statistical analysis

gnuplot

• plotter
• takes from stdin

xargs

• takes lines of input and puts them into arguments