vd(1)                                           Quick Reference Guide                                           vd(1)

NAME
     VisiData — a terminal utility for exploring and arranging tabular data

SYNOPSIS
     vd [options] [input ...]
     vd [options] --play cmdlog [-w waitsecs] [--batch] [-o output] [field=value ...]

DESCRIPTION
     VisiData is a multipurpose tool built on the vdtui platform that can be used to explore, clean, edit, and
     restructure data. Rows can be selected, filtered, and grouped; columns can be rearranged, transformed, and
     derived via regex or Python expressions; and workflows can be saved, documented, and replayed.

   REPLAY MODE
     -p, --play=cmdlog       replay a saved cmdlog within the interface
     -w, --replay-wait=seconds
                             wait seconds between commands
     -b, --batch             replay in batch mode (with no interface)
     -o, --output=file       save final visible sheet to file as .tsv
     --replay-movement=bool  toggle --play to move cursor cell-by-cell
     field=value             replace "{field}" in cmdlog contents with value

   Commands During Replay
        ^U                   pause/resume replay
        Tab                  execute next row in replaying sheet
        ^K                   cancel current replay

   GLOBAL COMMANDS
     In most cases, commands that affect selected rows will affect all rows if no rows are selected.

   Keystrokes for the Cautious
      ^H              view this man page
     z^H              view sheet of commands and keybindings
     ^Q               abort program immediately
      ^C              cancel user input or abort all async threads on current sheet
      q               quit current sheet
     gq               quit all sheets (clean exit)

   Cursor Movement
     Arrow PgUp Home  move as expected
      h   j   k   l   move left/down/up/right
     gh  gj  gk  gl   move all the way to the left/bottom/top/right of sheet
          G  gg       move all the way to the bottom/top of sheet
     ^B  ^F           scroll one page back/forward

     ^^ (Ctrl-^)      jump to previous sheet (swaps with current sheet)

      /   ? regex     search for regex forward/backward in current column
     g/  g? regex     search for regex forward/backward over all visible columns
     z/  z? expr      search by Python expr forward/backward in current column (with column names as variables)
      n   N           move to next/previous match from last search

      <   >           move up/down to next value in current column
     z<  z>           move up/down to next null in current column
      {   }           move up/down to next selected row

      c regex         move to next column with name matching regex
      r regex         move to next row with key matching regex
     zc  zr number    move to column/row number

      H   J   K   L   slide current row/column left/down/up/right
     gH  gJ  gK  gL   slide current row/column all the way to the left/bottom/top/right of sheet

     zh  zj  zk  zl   scroll one left/down/up/right

   Column Manipulation
      _ (underscore)  adjust width of current column
     g_               adjust width of all visible columns
     z_ number        adjust width of current column to number

      - (hyphen)      hide current column
     z-               reduce width of current column by half
     gv               unhide all columns

     ! z!             toggle/unset current column as a key column
     ~  #  %  $  @  z#
                      set type of current column to str/int/float/currency/date/len
       ^              edit name of current column
      g^              set names of all unnamed visible columns to contents of selected rows (or current row)
      z^              set name of current column to contents of current cell
     gz^              set name of current column to combined contents of current column for selected rows (or current
                      row)

       = expr         create new column from Python expr, with column names as variables
      g= expr         set current column for selected rows to result of Python expr
     gz= expr         set current column for selected rows to the items in result of Python sequence expr
      z= expr         evaluate Python expression on current row and show result on status line
      ' (tick)        add a frozen copy of current column with all cells evaluated
     g'               open a frozen copy of current sheet with all visible columns evaluated
     z'  gz'          reset cache for current/all visible column(s)

      : regex         add new columns from regex split; number of columns determined by example row at cursor
      ; regex         add new columns from capture groups of regex (also requires example row)
     z; expr          add new column from bash expr, with $columnNames as variables
      * regex/subst   add column derived from current column, replacing regex with subst (may include \1 backrefs)
     g* regex/subst   modify selected rows in current column, replacing regex with subst

      (   g(          expand current/all visible column(s) of lists (e.g. [3]) or dicts (e.g. {3}) fully
     z(  gz( depth    expand current/all visible column(s) of lists (e.g. [3]) or dicts (e.g. {3}) to given depth (0=
                      fully)
      )               unexpand current column; restore original column and remove other columns at this level

   Row Selection
       s   t   u      select/toggle/unselect current row
      gs  gt  gu      select/toggle/unselect all rows
      zs  zt  zu      select/toggle/unselect rows from top to cursor
     gzs gzt gzu      select/toggle/unselect rows from cursor to bottom
      |   \ regex     select/unselect rows matching regex in current column
     g|  g\ regex     select/unselect rows matching regex in any visible column
     z|  z\ expr      select/unselect rows matching Python expr in any visible column
      , (comma)       select rows matching current cell in current column
     g,               select rows matching current row in all visible columns

   Row Sorting/Filtering
      [   ]           sort ascending/descending by current column
     g[  g]           sort ascending/descending by all key columns
      "               open duplicate sheet with only selected rows
     g"               open duplicate sheet with all rows
     gz"              open duplicate sheet with deepcopy of selected rows

   Editing Rows and Cells
       a   za         append a blank row/column
      ga  gza number  append number blank rows/columns
       d   gd         delete (cut) current/selected row(s) and move to clipboard
       y   gy         yank (copy) current/all selected row(s) to clipboard
      zy  gzy         yank (copy) contents of current column for current/selected row(s) to clipboard
      zd  gzd         delete (cut) contents of current column for current/selected row(s) and move to clipboard
       p    P         paste clipboard rows after/before current row
      zp  gzp         set contents of current column for current/selected row(s) to last clipboard value
       Y   gY         yank (copy) current/all selected row(s) to system clipboard (using options.clipboard_copy_cmd)
      zY  gzY         yank (copy) contents of current column for current/selected row(s) to system clipboard (using
                      options.clipboard_copy_cmd)
       f              fill null cells in current column with contents of non-null cells up the current column
       e text         edit contents of current cell
      ge text         set contents of current column for selected rows to text

     Commands While Editing Input
        Enter  ^C        accept/abort input
        ^O               open external $EDITOR to edit contents
        ^R               reload initial value
        ^A  ^E           move to beginning/end of line
        ^B  ^F           move back/forward one character
        ^H  ^D           delete previous/current character
        ^T               transpose previous and current characters
        ^U  ^K           clear from cursor to beginning/end of line
        Backspace  Del   delete previous/current character
        Insert           toggle insert mode
        Up  Down         set contents to previous/next in history
        Tab  Shift+Tab   autocomplete input (when available)

   Data Toolkit
      o input         open input in VisiData
     ^S g^S filename  save current/all sheet(s) to filename in format determined by extension (default .tsv)
                      Note: if the format does not support multisave, or the filename ends in a /, a directory will
                      be created.
     z^S filename     save key columns and current column only to filename in format determined by extension (default
                      .tsv)
     ^D filename.vd   save CommandLog to filename.vd file
     A number         open new blank sheet with number columns
     R number         pushes sheet with random population subset of number rows
     T                open new sheet with rows and columns transposed

      + aggregator    add aggregator to current column (see Frequency Table)
     z+ aggregator    display result of aggregator over values in selected rows for current column

   Data Visualization
      . (dot)       plot current numeric column vs key columns. The numeric key column is used for the x-axis;
                    categorical key column values determine color.
     g.             plot a graph of all visible numeric columns vs key columns.

     If rows on the current sheet represent plottable coordinates (as in .shp or vector .mbtiles sources),  . plots
     the current row, and g. plots all selected rows (or all rows if none selected).

     Canvas-specific Commands
         +   -              increase/decrease zoom level, centered on cursor
         _ (underscore)     zoom to fit full extent
         s   t   u          select/toggle/unselect rows on source sheet contained within canvas cursor
        gs  gt  gu          select/toggle/unselect rows on source sheet visible on screen
         d                  delete rows on source sheet contained within canvas cursor
        gd                  delete rows on source sheet visible on screen
         Enter              open sheet of source rows contained within canvas cursor
        gEnter              open sheet of source rows visible on screen
         1 - 9              toggle display of layers
        ^L                  redraw all pixels on canvas
         v                  toggle show_graph_labels option
        mouse scrollwheel   zoom in/out of canvas
        left click-drag     set canvas cursor
        right click-drag    scroll canvas

   Other Commands
     Q                quit current sheet and remove it from the CommandLog
     V                view contents of current cell in a new TextSheet
       v              toggle sheet-specific visibility (text wrap on TextSheet, legends/axes on Graph)

     Space longname   execute command by its longname

      ^E              view traceback for most recent error
     g^E              view traceback for most recent errors
     z^E              view traceback for error in current cell

      ^L              refresh screen
      ^R              reload current sheet
     z^R              clear cache for current column
      ^Z              suspend VisiData process
      ^G              show cursor position and bounds of current sheet on status line
      ^V              show version and copyright information on status line
      ^P              open Status History

      ^Y  z^Y  g^Y    open current row/cell/sheet as Python object
      ^X expr         evaluate Python expr and opens result as Python object
     z^X expr         evaluate Python expr on current row and shows result on status line
     g^X stmt         execute Python stmt in the global scope

   Internal Sheets List
      .  Directory Sheet             browse and modify properties of files in a directory

     Metasheets
      .  Columns Sheet (Shift+C)     edit column properties
      .  Sheets Sheet (Shift+S)      jump between sheets or join them together
      .  Options Sheet (Shift+O)     edit all configurable options
      .  Commandlog (Shift+D)        modify and save commands for replay
      .  Error Sheet (^E)            view last error
      .  Status History (^P)         view history of status messages
      .  Threads Sheet (^T)          view, cancel, and profile asynchronous threads

     Derived Sheets
      .  Frequency Table (Shift+F)   group rows by column value, with aggregations of other columns
      .  Describe Sheet (Shift+I)    view summary statistics for each column
      .  Pivot Table (Shift+W)       group rows by key and summarize current column
      .  Melted Sheet (Shift+M)      unpivot non-key columns into variable/value columns

   INTERNAL SHEETS
   Directory Sheet
     (sheet-specific commands)
     Modifying any cell changes the in-memory value. Changes are only applied to the filesystem with ^S
        Enter  gEnter    open current/selected file(s) as new sheet(s)
         ^O  g^O         open current/selected file(s) in external $EDITOR
         d   gd          schedule current/selected file(s) for deletion
         ^R  z^R  gz^R   reload information for all/current/selected file(s), undoing any pending changes
        z^S   ^S         apply all deferred changes to current/all file(s)

   METASHEETS
   Columns Sheet (Shift+C)
     Properties of columns on the source sheet can be changed with standard editing commands (e ge g= Del) on the
     Columns Sheet. Multiple aggregators can be set by listing them (separated by spaces) in the aggregators column.
     The 'g' commands affect the selected rows, which are the literal columns on the source sheet.
     (global commands)
        gC               open Columns Sheet for all visible columns on all sheets
     (sheet-specific commands)
         &               add column from concatenating selected source columns
        g! gz!           toggle/unset selected columns as key columns on source sheet
        g+ aggregator    add Ar aggregator No to selected source columns
        g- (hyphen)      hide selected columns on source sheet
        g~ g# g% g$ g@  gz#
                         set type of selected columns on source sheet to str/int/float/currency/date/len
         Enter           open a Frequency Table sheet grouped by column referenced in current row

   Sheets Sheet (Shift+S)
     (global commands)
        gS               open Sheets Graveyard which includes references to closed sheets
     (sheet-specific commands)
         Enter           jump to sheet referenced in current row
         a               add row to reference a new blank sheet
        gC               open Columns Sheet with all columns from selected sheets
        gI               open Describe Sheet with all columns from selected sheets
        g^R              reload all selected sheets
        z^C  gz^C        abort async threads for current/selected sheets(s)
         & jointype      merge selected sheets with visible columns from all, keeping rows according to jointype:
                         .  inner  keep only rows which match keys on all sheets
                         .  outer   keep all rows from first selected sheet
                         .  full   keep all rows from all sheets (union)
                         .  diff   keep only rows NOT in all sheets
                         .  append keep all rows from all sheets (concatenation)
                         .  extend copy first selected sheet, keeping all rows and sheet type, and extend with
                            columns from other sheets

   Options Sheet (Shift+O)
     (sheet-specific commands)
        Enter  e         edit option

   CommandLog (Shift+D)
     (global commands)
        gD               open Directory Sheet for options.visidata_dir (default: ~/.visidata/), which contains saved
                         commandlogs and macros
     (sheet-specific commands)
          x              replay command in current row
         gx              replay contents of entire CommandLog
         ^C              abort replay
        z^S keystroke    save selected rows to macro mapped to keystroke
                         Macros are saved to .visidata/macro/command-longname.vd. The list of macros is saved at
                         .visidata/macros.vd (keystroke, filename).

   DERIVED SHEETS
   Frequency Table (Shift+F)
     A Frequency Table groups rows by one or more columns, and includes summary columns for those with aggregators.
     (global commands)
        gF               open Frequency Table, grouped by all key columns on source sheet
        zF               open one-line summary for selected rows
     (sheet-specific commands)
         s   t   u       select/toggle/unselect these entries in source sheet
         Enter           open sheet of source rows that are grouped in current cell

   Describe Sheet (Shift+I)
     (global commands)
        gI               open Describe Sheet for all visible columns on all sheets
     (sheet-specific commands)
        zs  zu           select/unselect rows on source sheet that are being described in current cell
         !               toggle/unset current column as a key column on source sheet
         Enter           open a Frequency Table sheet grouped on column referenced in current row
        zEnter           open copy of source sheet with rows described in current cell

   Pivot Table (Shift+W)
     Set key column(s) and aggregators on column(s) before pressing Shift+W on the column to pivot.
     (sheet-specific commands)
         Enter           open sheet of source rows aggregated in current pivot row
        zEnter           open sheet of source rows aggregated in current pivot cell

   Melted Sheet (Shift+M)
     Open melted sheet (unpivot), with key columns retained and all non-key columns reduced to Variable-Value rows.
     (global commands)
        gM regex         open melted sheet (unpivot), with key columns retained and regex capture groups determining
                         how the non-key columns will be reduced to Variable-Value rows.

   Python Object Sheet (^X ^Y g^Y z^Y)
     (sheet-specific commands)
         Enter           dive further into Python object
         e               edit contents of current cell
         v               toggle show/hide for methods and hidden properties
        gv  zv           show/hide methods and hidden properties

OPTIONS
     -f, --filetype=filetype      tsv                set loader to use for filetype instead of file extension
     -y, --confirm-overwrite=F    True               overwrite existing files without confirmation
     --diff=base                  None               add colorizer for all sheets against base
     --encoding=str               utf-8              encoding passed to codecs.open
     --encoding-errors=str        surrogateescape    encoding errors passed to codecs.open
     --regex-flags=str            I                  flags to pass to re.compile() [AILMSUX]
     --default-width=int          20                 default column width
     --wrap=bool                  False              wrap text to fit window width on TextSheet
     --cmd-after-edit=str         go-down            command longname to execute after successful edit
     --col-cache-size=int         0                  max number of cache entries in each cached column
     --quitguard=bool             False              confirm before quitting last sheet
     --null-value=NoneType        None               a value to be counted as null
     --force-valid-colnames=bool  False              clean column names to be valid Python identifiers
     --debug=bool                 False              exit on error and display stacktrace
     --curses-timeout=int         100                curses timeout in ms
     --force-256-colors=bool      False              use 256 colors even if curses reports fewer
     --use-default-colors=bool    False              curses use default terminal colors
     --note-pending=str           ⌛                  note to display for pending cells
     --note-format-exc=str        ?                  cell note for an exception during type conversion or formatting
     --note-getter-exc=str        !                  cell note for an exception during computation
     --note-type-exc=str          !                  cell note for an exception during computation
     --note-unknown-type=str                         cell note for unknown types in anytype column
     --scroll-incr=int            3                  amount to scroll with scrollwheel
     --skip=int                   0                  skip first N lines of text input
     --confirm-overwrite=bool     True               whether to prompt for overwrite confirmation on save
     --save-errors=bool           True               whether to save or discard errors while saving
     --header=int                 1                  parse first N rows of .csv/.tsv as column names
     --delimiter=str                                 delimiter to use for tsv filetype
     --filetype=str                                  specify file type
     --save-filetype=str          tsv                specify default file type to save as
     --tsv-safe-newline=str                          replacement for tab character when saving to tsv
     --tsv-safe-tab=str                              replacement for newline character when saving to tsv
     --clipboard-copy-cmd=str                        command to copy stdin to system clipboard
     --visibility=int             0                  visibility level (0=low, 1=high)
     --min-memory-mb=int          0                  minimum memory to continue loading and async processing
     --replay-wait=float          0.0                time to wait between replayed commands, in seconds
     --replay-movement=bool       False              insert movements during replay
     --visidata-dir=str           ~/.visidata/       directory to load and store macros
     --rowkey-prefix=str          キ                 string prefix for rowkey in the cmdlog
     --cmdlog-histfile=str                           file to autorecord each cmdlog action to
     --regex-maxsplit=int         0                  maxsplit to pass to regex.split
     --show-graph-labels=bool     True               show axes and legend on graph
     --plot-colors=str                               list of distinct colors to use for plotting distinct objects
     --zoom-incr=float            2.0                amount to multiply current zoomlevel when zooming
     --motd-url=str                                  source of randomized startup messages
     --profile=str                                   filename to save binary profiling data
     --csv-dialect=str            excel              dialect passed to csv.reader
     --csv-delimiter=str          ,                  delimiter passed to csv.reader
     --csv-quotechar=str          "                  quotechar passed to csv.reader
     --csv-skipinitialspace=bool  True               skipinitialspace passed to csv.reader
     --csv-escapechar=NoneType    None               escapechar passed to csv.reader
     --safety-first=bool          False              sanitize input/output to handle edge cases, with a performance
                                                     cost
     --json-indent=NoneType       None               indent to use when saving json
     --fixed-rows=int             1000               number of rows to check for fixed width columns
     --pcap-internet=str          n                  (y/s/n) if save_dot includes all internet hosts separately (y),
                                                     combined (s), or does not include the internet (n)
     --graphviz-edge-labels=bool  True               whether to include edge labels on graphviz diagrams

   DISPLAY OPTIONS
     Display options can only be set via the Options Sheet or a .visidatarc (see FILES).

     disp_note_none      ⌀                   visible contents of a cell whose value is None
     disp_truncator      …                   indicator that the contents are only partially visible
     disp_oddspace       ·                   displayable character for odd whitespace
     disp_unprintable    .                   substitute character for unprintables
     disp_column_sep     |                   separator between columns
     disp_keycol_sep     ‖                   separator between key columns and rest of columns
     disp_status_fmt     {sheet.name}|       status line prefix
     disp_lstatus_max    0                   maximum length of left status line
     disp_status_sep      |                  separator between statuses
     disp_edit_fill      _                   edit field fill character
     disp_more_left      <                   header note indicating more columns to the left
     disp_more_right     >                   header note indicating more columns to the right
     disp_error_val                          displayed contents for computation exception
     disp_ambig_width    1                   width to use for unicode chars marked ambiguous
     color_default       normal              the default color
     color_default_hdr   bold underline      color of the column headers
     color_current_row   reverse             color of the cursor row
     color_current_col   bold                color of the cursor column
     color_current_hdr   reverse underline   color of the header for the cursor column
     color_column_sep    246 blue            color of column separators
     color_key_col       81 cyan             color of key columns
     color_hidden_col    8                   color of hidden columns on metasheets
     color_selected_row  215 yellow          color of selected rows
     color_status        bold                status line color
     color_error         red                 error message color
     color_warning       yellow              warning message color
     color_edit_cell     normal              cell color to use when editing cell
     disp_pending                            string to display in pending cells
     color_note_pending  bold magenta        color of note in pending cells
     color_note_type     226 yellow          cell note for numeric types in anytype columns
     disp_date_fmt       %Y-%m-%d            default fmtstr to strftime for date values
     color_change_pending reverse yellow     color for file attributes pending modification
     color_delete_pending red                color for files pending delete
     disp_histogram      *                   histogram element character
     disp_histolen       50                  width of histogram column
     disp_replay_play    ▶                   status indicator for active replay
     disp_replay_pause   ‖                   status indicator for paused replay
     disp_pixel_random   False               randomly choose attr from set of pixels instead of most common
     color_graph_hidden  238 blue            color of legend for hidden attribute
     color_graph_axis    bold                color for graph axis labels
     color_diff          red                 color of values different from --diff source
     color_diff_add      yellow              color of rows/columns added to --diff source

EXAMPLES
           vd foo.tsv

           vd -f sqlite bar.db

           vd -b countries.fixed -o countries.tsv

           vd postgres://username:password@hostname:port/database

           vd --play tests/pivot.vd --replay-wait 1 --output tests/pivot.tsv

           ls -l | vd -f fixed --skip 1 --header 0

           vd -p foo.vd -b -o - > foo.tsv

           vd --diff foo.tsv bar.tsv

FILES
     At the start of every session, VisiData looks for $HOME/.visidatarc, and calls Python exec() on its contents if
     it exists. For example:

        options.min_memory_mb=100  # stop processing without 100MB free

        bindkey('0', 'go-leftmost')   # alias '0' to move to first column, like vim

        def median(values):
            L = sorted(values)
            return L[len(L)//2]

        aggregator('median', median)
     Functions defined in .visidatarc are available in python expressions (e.g. in derived columns).

SUPPORTED SOURCES
     These are the supported sources:

        tsv (tab-separated value)
           Plain and simple. VisiData writes tsv format by default. See the --delimiter option.

        csv (comma-separated value)
           .csv files are a scourge upon the earth, and still regrettably common.
           See the --csv-dialect, --csv-delimiter, --csv-quotechar, and --csv-skipinitialspace options.
           Accepted dialects are excel-tab, unix, and excel.

        fixed (fixed width text)
           Columns are autodetected from the first 1000 rows (adjustable with --fixed-rows).

        json (single object) and jsonl (one object per line).
           Cells containing lists (e.g. [3]) or dicts ({3}) can be expanded into new columns with ( and unexpanded
           with ).

        yaml/yml (requires PyYAML)

        pcap (requires xpkt, dnslib)
           View and investigate captured network traffic in a tabular format.

        png (requires pypng)
           Pixels can be edited and saved in data form. Images can be plotted with . (dot).

     The following URL schemes are supported:
        http (requires requests); can be used as transport for with another filetype
        postgres (requires psycopg2)

     The following sources may include multiple tables. The initial sheet is the table directory; Enter loads the
     entire table into memory.

        sqlite
        xlsx (requires openpyxl)
        xls (requires xlrd)
        hdf5 (requires h5py)
        ttf/otf (requires fonttools)
        mbtiles (requires mapbox-vector-tile)
        htm/html (requires lxml)
        xml (requires lxml)
            v    show only columns in current row attributes
           za    add column for xml attributes
        xpt (SAS; requires xport)
        sas7bdat (SAS; requires sas7bdat)
        sav (SPSS; requires savReaderWriter)
        dta (Stata; requires pandas)
        shp (requires pyshp)

     In addition, .zip, .gz, .bz2, and .xz files are decompressed on the fly.

SUPPORTED OUTPUT FORMATS
     These are the supported savers:

        tsv (tab-separated value)
        csv (comma-separated value)
        json (one object with all rows)
           All expanded subcolumns must be closed (with )) to retain the same structure.
           .shp files can be saved as geoJSON.
        md (org-mode compatible markdown table)
        htm/html (requires lxml)
        png (requires pypng)

     Multisave is supported by html, md, and txt; g^S will save all sheets into a single output file.

AUTHOR
     VisiData was made by Saul Pwanson <vd@saul.pw>.

Linux/MacOS                                        August 19, 2018                                        Linux/MacOS