markdown_files/Data_Preparation_in_the_Shell.md

metadata

title: Data Preparation in the Shell
original_url: >-
  https://tds.s-anand.net/#/data-preparation-in-the-shell?id=data-preparation-in-the-shell
downloaded_at: '2025-06-08T23:26:41.381829'

Data Preparation in the Shell

You’ll learn how to use UNIX tools to process and clean data, covering:

• curl (or wget) to fetch data from websites.
• gzip (or xz) to compress and decompress files.
• wc to count lines, words, and characters in text.
• head and tail to get the start and end of files.
• cut to extract specific columns from text.
• uniq to de-duplicate lines.
• sort to sort lines.
• grep to filter lines containing specific text.
• sed to search and replace text.
• awk for more complex text processing.

Here are the links used in the video:

• Data preparation in the shell - Notebook
• Data Science at the Command Line

[Previous

Data Aggregation in Excel](#/data-aggregation-in-excel)

[Next

Data Preparation in the Editor](#/data-preparation-in-the-editor)