Supported Formats

Extra notes about formats

tsv (Tab Separated Values), as simple as it gets

• delimiter: field delimiter to use for tsv/usv filetype (default: .)
• row_delimiter: row delimiter to use for tsv/usv filetype (default: \n)
• tsvsafenewline: replacement for newline character when saving to tsv (default: )
• tsvsafetab: replacement for tab character when saving to tsv (default: .)

Use -f usv for Unicode separators U+241F and U+241E. Use -f tsv for awk-like records. Use --delimiter= (an empty string) to make '\0' the value separator. Use --row-delimiter= to make '\0' the row separator.

csv (Comma Separated Values) for maximum compatibility

.csv files are a scourge upon the earth, and still regrettably common. All csv_* options are passed unchanged into csv.reader() and csv.writer().

• csv_dialect: dialect passed to csv.reader (default: excel)
• Accepted dialects are excel-tab, unix, and excel.
• csv_delimiter: delimiter passed to csv.reader (default: ,)
• csv_quotechar: quotechar passed to csv.reader (default: ")
• csv_skipinitialspace: skipinitialspace passed to csv.reader (default: True)
• csv_escapechar: escapechar passed to csv.reader (default: None)
• csv_lineterminator: lineterminator passed to csv.writer (default: \n)

Saving TSV/CSV files

• save_filetype: specify default file type to save as (default: tsv)
• safety_first: sanitize input/output to handle edge cases, with a performance cost (default: False)

Useful options for text formats in general

• regex_skip: regex of lines to skip in text sources (default: )
• save_encoding: encoding passed to codecs.open when saving a file (default: utf8)

fixed

• loader-specific options
  • fixed_rows (default: 1000) number of rows to detect fixed width columns from
  • fixed_maxcols (default: 0) max number of fixed-width columns to create (0 is no max)

json

• loader-specific options
  • json_indent (default: None) indent to use when saving json
  • json_sort_keys (default: False) sort object keys when saving to json
  • default_colname (default: '') column name to use for non-dict rows
• Cells containing lists (e.g. [3]) or dicts (e.g. {3}) can be expanded into new columns with ( and unexpanded with ).
• All expanded subcolumns must be closed (with )) to retain the same structure.
• Support for jsonla was added in 3.0.

xml

• v show only columns in current row attributes
• za add column for xml attributes

pcap

• loader-specific options
  • pcap_internet (default: 'n') (y/s/n) if save_dot includes all internet hosts separately (y), combined (s), or does not include the internet (n)

postgres

• loader-specific options
  • postgres_schema (default: 'public') the desired schema for the Postgres database
• vd postgres://username:password@hostname:port/database opens a connection to the given postgres database.

imap

• vd "imap://user@domain.com:passwordhere@imap-mailserver.com" opens a connection to the IMAP server
  • e.g. vd "imap://someone@hotmail.com:pass123@imap-mail.outlook.com:993"
  • e.g. vd "imap://someone@gmail.com@imap.gmail.com"
    • note that you don't specify a password for gmail here -- instead, you will be prompted to follow some instructions

using VisiData as a pager within psql

In psql:

\pset format csv
\pset pager always
\setenv PSQL_PAGER 'vd -f csv'
\pset pager_min_lines

sqlite

• supports saving for CREATE/INSERT (not wholesale updates)
• z Ctrl+S to commit any add-row/edit-cell/delete-row

mysql

• loader-specific requirements
  • working mysql / mariadb installation or at least the libmysqlclient-dev package (ubuntu; name might be different on other platforms)
  • mysqlclient python module in path or virtual environment (pip install mysqlclient)
• vd mysql://username:password@hostname:port/database opens a connection to the given mysql / mariadb database.

html

• loader-specific options
  • html_title (default: '<h2>{sheet.name}</h2>') table header when saving to html
• load all <table>s in a web page as VisiData sheets.

shp

• Can be edited in raw data form. Images can be plotted with . (dot).
• .shp files can be saved as geoJSON.

mbtiles

• Can be edited in raw data form. Images can be plotted with . (dot).

png

• Can be edited in raw data form. Images can be plotted with . (dot).

ttf

• Can be edited in raw data form. Images can be plotted with . (dot).

pandas

VisiData has an adapter for pandas. To load a file format which is supported by pandas, pass -f pandas data.foo. This will call pandas.read_foo().

For example:

vd -f pandas data.parquet

loads a parquet file. When using the pandas loader, the .fileformat file extension is mandatory.

To load a hierarchy of parquet files located in folder data/, run

vd -f parquet data/

or rename the directory to data.parquet and run

vd data.parquet -f pandas

This should similarly work for any format that has a pandas.read_format() function.

VisiData Internal Formats (.vd, .vdj, .vdx, .vds)

• .vd/vdj/.vdx are command log formats suitable for VisiData scripts and macros
• .vds is a multisheet save format that includes some sheet and column metadata

API (reddit, matrix, zulip, airtable)

• loader-specific requirements
  • require setting authentication information in ~/.visidatarc or on the CLI
  • launch the loader with -f loadername for steps to obtain and configure authentication credentials

Grep

A .grep file is a JSON lines file. It can be in two formats: 1) A simple container with three fields: file - a string with the path to the file where the match was found (absolute or relative path) lineno - an integer with the line number in the file where the match was found, text - a string with the text of the line that matched. 2) ripgrep grep_printer format, described here: https://docs.rs/grep-printer/latest/grepprinter/struct.JSON.html