VisiData version 2.0

A lot of features have been added to VisiData. So many, in fact, that we are calling the next big release v2.0. But with so many changes, we need your help to test and find bugs.

If you can handle the sharp cutting edge of VisiData, then go ahead and install the latest pre-release version:

pip3 install git+https://github.com/saulpw/visidata@v2.-2.1

Then, play with the new features and file any issues you come across.

New Features (v2.-2)

Thanks so much to everyone that is helping us prepare for v2.0. This release will feature a bucketload of bugfixes and additions along with the following new features. Between v2.-1 and v2.-2, 26 issues filed by users had code written for them.

1. Every sheet has its own CommandLog

This endeavor was hinted in this interview:

There is one thing that I do wish that I could solve, and I'm sure that it is solvable, but I haven't managed it yet. The command log reports every action that you take, including all the dead ends that you wind up finding yourself in. Those are sometimes handy to have on there, so you can see what you did and didn't want to keep around. But when you're getting to the end, and you want to do it again, you want to cull all those dead ends out, and just get to the place that you currently are. And so I want to have some kind of graph or tree that would get you from your current state and only show the commands that you took to get there.

and now it is here!

Shift+D is now mapped to cmdlog-sheet, and takes you to the current sheet's commandlog, with all the other loose ends removed. If missed, gShift+D goes to the standard, classic commandlog that contains everything.

2. vls (along with vping and vtop)

It has become apparent that the write-mode of the DirSheet was a conceptually separate tool, even if the interface fits in seamlessly in VisiData. Data sheets and filesystem editing are two radically different use cases.

So from 2.-2 and on, the filesystem manipulation part of VisiData was moved into the entry point vls.

To play with, type

pip3 install git+git://github.com/saulpw/visidata/vsh@develop

and then type vls. Please share any feedback here.

(To learn about the write-mode of the DirSheet, check out the following case study video.)

The package is called vsh because it is going to be a compilation of vd interfaces for Linux utilities.

On top of vls, it also includes:

3. Move deferred changes into its own plugin

Please note, that there are two open bugs for defermods at the time of this release. Subscribe to those issues to get updates on when they have been fixed.

Activating this plugin adds two modes to VisiData: Defermods and Trackmods.

Defermods enables modification for sheets that are deferred (e.g. sqlite and vls). Saving to source is deferred until commit-sheet (Ctrl+S). Deleted rows and modified cells are colored as pending changes until committed. To save to a different file (not source), use save-sheet (g Ctrl+S).

Trackmods tracks changes in sheets whose source has a path. Modifications are colored yellow until changes are committed with commit-sheet (Ctrl+S) or saved with save-sheet (g Ctrl+S).

These modes are not on by default, from plugins.defermods import * must be added to visidatarc (it can also be installed through the PluginsSheet. Sheets must set .defermods and/or .trackmods. By default BaseSheet has .defermods=False and .trackmods set to True when defermods is imported.

defermods is not on by default because: 1) deletes with defermods enabled take longer; especially delete-selected 2) it is easier for users to consciously opt-in to a novel VisiData experience than to opt-out

4. Index into sub-sheets from command line

As requested by aborruso! +:subsheet:col:row as a command line argument now sets subsheet to be the topsheet upon load, with cursor located in cell at row and col. All of the components are optional. For example, +:subsheet:: sets the topsheet to be subsheet with the cursor located at the top left. +::col:row sets the cursor in the cell at row and col in the topsheet. The toplevel source index can be explicitly named, if there is more than one with +toplevel:subsheet::

This currently works for the html, xls, xlsx, xlsb, hdf5, and sqlite loaders.

5. Default startup launches the vdmenu

Previously, if a user launched vd without a source, the DirSheet for the current directory would pop up.

Launching vd without a source file, now opens the vdmenu, a menu of core sheets. Press Enter to open the sheet referenced in the current row.

New Features (v2.-1)

1. Alt+ layer for easier page-switching.

Edit: this feature is currently under revision.

Alt+1 will go to the first loaded sheet, Alt+2 goes to the second etc. A sheet's number is in the lower left of the status line, and in the new shortcut column on the SheetsSheet.

The Alt (or Esc- prefix) layer is otherwise reserved for your personal bindings of builtin and/or custom commands. Use ^[x as the keybinding for Alt+x (or Esc x).

2. Undo/redo for modifying commands

By popular request, VisiData now has undo/redo for certain operations. Press Shift+U to undo the last modification on the current sheet, and then Shift+R to redo what you just undid.

[In general undo will only be available for commands that could change which data is saved, which means that hiding a column is undoable but changing its width is not.]

Please let us know if any command is missing an undo, or if there is anything about an undo or redo operation that feels "weird".

3. Deferred changes

EDIT: This feature has been moved out of VisiData core and into the plugin: defermods.py. See the list of features in 2.-2 for details.

When I made the DirSheet and gave it the power to delete files, and modify metadata, I quickly realized it needed to be a "deferred sheet". Modifications and deletions were staged instead of committed immediately. Then this concept transferred well to vgit, and now it seems like it might be generally useful.

Now, most sheets in VisiData will keep track of their additions/modifications/deletions, and color them accordingly. You can save the changes back to the source with the usual Ctrl+S and revert all changes back to the most recent save with Ctrl+R.

Please let us know how the modify/save/confirm process feels on any sheet, with particular attention to derived sheets and metasheets.

4. Multi-line rows

Previously, VisiData only had single-line rows; to expand long text you had to increase the width of the column.

Now, VisiData v2.x supports optional multi-line rows! Toggle it by pressing v on any sheet with truncated values, and cells in all columns will expand their height and wordwrap to show the full value.

Please let us know how multi-line rows looks and feels for you; in particular how scrolling with multi-line rows feels.

5. Range binning for numeric columns on frequency and pivot tables

Per user request, we have added a feature flag for range binning of numeric columns.

If you set the option numeric_binning to True, Freqency and Pivot tables with numeric columns will bin the values into ranges.

If you play with it, let us know how it feels, and if you find any issues with edge cases.

6. Plugin framework

Did you know that if you want VisiData to do something new or different, you can write a plugin and start using it right away? Some plugins have already been written for 1.5.2, like jsvine's vdnormcol and vddedupe.

2.0 will add a PluginsSheet, which fetches the list of known plugins from visidata.org/plugins.tsv, and allows you to easily install and activate (a), and/or deactivate (d) them. For instructions on how to manually incorporate plugin code, see our docs and dev checklist.

This plugin framework is what makes VisiData a platform, and not just a tool.

The API reworking that we're doing is substantial, so existing plugins and .visidatarc files may need to be changed in a bunch of small ways. With 2.0, we are going to refine and document a stable and reasonable API for plugin authors to rely on.

And we're doing this work, because the plugin and app ecosystem for VisiData very well might become more important than the VisiData core utility itself. Like some people use emacs mostly for magit or org-mode, or jupyter and pandas with python.

7. Lots more changes

There are also lots more changes! Even if you do not end up playing with any of the new features, just using VisiData as you usually do and filing issues as they come would be an enormous help.