We are starting off the New Year with a very small patch for a few bugs.
Ctrl+Z), VisiData will now correctly resume in the editline.
Ctrl+Won an empty value in editline does not error out of the editline.
And that's it.
Happy new year, everyone!
The initial motivation for this patch was to fix an idle performance regression. In the interim, though, we have received a bunch of feedback for improvements and a couple of feature requests.
So thanks for all of your ongoing engagement and for continuing to share VisiData with the world. We hope you enjoy this final release of 2018!
A brief overview:
Improvements to the pandas loader.
When set to True
options.bulk_select_clear bulk select commands (
,, etc) will clear the previously selected rows.
Shift+O once again pushes the global options sheet. You can find the sheet options by pressing
gO now opens the
.visidatarc which can be edited and saved from within the VisiData interface.
Sorting is now asynchronous, per user request. Please let us know how you find it.
gz^ now renames all visible columns to contents of selected rows and
gz* substitutes over all visible columns
As usual, the complete list of changes can be found in the Changelog.
For the most part, this is a bugfix release. A few other improvements are here as well:
~/.visidataadded to sys.path on startup
options.headerto determine number of rows used to set column names (default is 1).
See the CHANGELOG for a more complete list of bugfixes and changes.
This is largely a bugfix and performance release. It was originally planned as a patch release (v1.3.2), but some features and interface changes crept in, so it is officially v1.4.
-f pandasloads file using
As usual, a more complete list of changes and features is in the CHANGELOG.
See the ROADMAP for the planned future direction of VisiData.
We found some issues with 1.3 (aggregators interacting with nulls/errors, primarily) that we didn't want to let sit until 1.4. So we fixed those issues and a couple others, and added a few 'minor' features. This patch version should be a definitive improvement over the base 1.3 version.
The complete list of changes is in the CHANGELOG. Here are the new options and features:
extendjoin type keeps the type of the first sheet, extending it with columns from the other sheets
rename-sheetcommand (thanks to @jsvine for suggestion; what default keybinding should it have?)
gz^R) to undo modifications on all selected rows
It's been a productive 3 months since v1.2. The largest effort in this release was a commands/options reworking, which will hopefully pay dividends in the future. Many other so-called improvements were made as well. Here's a list of most of them:
commands.tsvis an exhaustive list of commands and their attributes and side effects.
Ctrl+H(sysopen-help), which is hopefully its final keybinding.
F1will still open the manpage if the terminal doesn't intercept it.
z?has been repurposed (see below).
Backspace, these help commands are also available by using
Backspace(backspace for help, a new trend).
z Ctrl+Hopens a list of commands available on this sheet.
Space) has been removed.
Space(exec-longname) executes the command for the input longname (tab completion of available commands is supported). (This function was previously bound to
options.wrap(for TextSheet wrapping of lines) now defaults to
R(random-sheet) opens a new sheet instead of selecting random rows (reverting to former behavior).
za(addcol-empty) asks for column name
zd(delete-cell) moves value to clipboard ("cut", like other delete commands)
Shift+Oopens options for the current sheet type, and
g Shift+Oopens the global options sheet.
Spaceand then typing the longname
color_warning(thanks to @jsvine for suggestion)
Falseto keep old behavior) if True, will confirm before quitting last sheet.
gS(sheets-graveyard) command opens a sheet that shows all discarded (but "precious") sheets. These are stored as weak references so they will be garbage collected eventually, but can be resurrected from the graveyard sheet until then.
False) makes loading/saving more robust, likely at the cost of performance which can become significant in large files.
options.tsv_safe_charis split into
addcol-sh) adds new columns for stdout/stderr of a
bashcommand, which uses
$colnameto substitute values from other columns (whole arguments only, so far).
unselect-expr) select/unselect by Python expression (thanks to @jsvine for suggestion).
searchr-expr) to search forward/backward by Python expression.
describe-all) describes all columns in all sheeets (like
(('expand-column') are filled out.
z#sets type of current column to
datesupports adding a number of days (or like
options.motd_urlnow uses https by default (thanks to @jsvine for the warning).
modeis editable (set to octal like
save-sheet) to an existing file gets stuck in an infinite loop when
reload-sheet) is required to refresh the
extcolumn for that row.
prev-search) won't continue a previous
show-aggregatewith mean errors on
)) a previously expanded column on a dup-ed (with
") sheet results it in disappearing on the source sheet.
@deinspanjer (issue #164) discovered that VisiData doesn't work with Python3.7 due to our use of the 'async' identifier, which is now off-limits as it is a formal keyword as of 3.7. The only change in 1.2.1 is changing 'async' to 'asyncthread' in order to work with Python 3.7.
Thanks to the illustrious @anjakefala for getting this release out quickly and without incident (knock on wood).
Here are the major feature enhancements for v1.2. Please see the CHANGELOG for the complete list.
z^Son the commandlog
Tto push derived sheet with transposed rows/columns
zuto select/unselect/toggle rows from top of sheet to cursor (thanks @SteveJSteiner for the suggestion)
gzuto select/unselect/toggle rows from bottom to cursor
gvto unhide all columns
gMto melt into multiple value columns
g*to transform selected rows in place
z</`z>' to move up/down to next null cell
^Ato execute a command by its longname
gDto open the directory at
options.cmdlog_histfileto specify a file to auto-append commandlog to (default empty means disabled)
options.tsv_safe_charto replace tabs and newlines when writing .tsv format (default empty means disabled, for faster saving)
options.error_is_nullto count errors as null (default false)
This is the first release since 1.0. The major additions and changes:
Space. The goal is to make it easier for people to explore the available functionality, and to try commands without knowing their keybindings.
qto back out)
Ycommand series was added to copy ("yank") to the system clipboard (with
options.clipboard_copy_cmd, set to
pbclipfor MacOS by default) to conveniently paste data in any supported text output format.
Ycopies the current row,
zYcopies the current cell, and
gYcopies all selected rows (or all rows).
ycommand series, which yanks to VisiData's internal clipboard.
-now works as a filename to specify stdin/stdout. Useful especially in batch mode to dump final sheet to stdout (
-b -o -).
New supported formats:
md) is now supported for saving (but not loading, yet) to an org-mode compatible table format.
.png files can now be loaded and saved, and crudely viewed on the canvas with an overloaded
.. The pixels can be edited on the source sheet like any other data.
.ttf and .otf (font) files can be loaded and viewed on the canvas. This is super useful for just about no one, but it was a great excuse to implement
These commands have been added or changed, and are expected to stay in future versions:
za command adds an empty, editable column to any sheet.
gza adds N new columns.
) commands will expand/collapse list/dict columns (e.g. in nested json).
In the canvas,
d deletes points from source sheet that are contained within the cursor.
gd deletes all points shown on the screen.
!@#$%-_ special actions on the Columns Sheet and Describe Sheet have been removed. They were sometimes convenient, but more often made it difficult to interact with the Columns Sheet itself.
gforms of these commands are still available and will operate on the source columns (with the exception of
g_, which now works consistently on all sheets as expected).
Shift+Arrows are aliased to
HJKL (though these may not work in all environments).
These additions are more experimental and may not stay in future versions:
Backspace command drops the current sheet (like
q), and also scrubs its history from the cmdlog.
ENTER is now aliased to
modify-edit-cell by default.
Other minor changes:
search and select with no input now uses the most recent input.
N do this already for row search (as before), but the new behavior works more like standard vim/less/etc tools, and also applies to non-row search (like
Many sheets (pivot, describe, melt, and many loaders) have improvements and bugfixes to make them even better.
Finally, some other news:
VisiData has been accepted into Debian for the next release! It is currently available if you've added the
unstable repo; install with
apt install visidata.
Jeremy Singer-Vine has put together a great tutorial for people who want a smoother path to start using VisiData.
Thanks to everyone who contributed to this release! As always, feedback and suggestions are welcome and appreciated.
VisiData 1.0 has been officially released! Changes since the 0.99 release candidate:
$HOME/.visidatarcis used now;
brew install saulpw/vd/visidataon MacOS), and an apt package that is under review for inclusion in Debian.
This is a huge milestone for VisiData. I devoted my sabbatical (the entire year of 2017) to creating an open-source tool that would be useful for myself and other terminal users, and it was very important for me to release a stable and lasting 1.0 so that it could be set aside when I resumed working.
We could have gotten away without certain features and with fewer supported formats, but there is such breadth and depth included in 1.0 (as shown by demos like this one at the PyCascades 2018 conference), that I'm hard-pressed to choose which features should have been omitted.
I owe a great deal to many people who have helped and supported me through this process. VisiData would not be what it is without them. The list of contributors includes everyone who has submitted to the repository, but there are some other groups to whom VisiData and I are eternally indebted:
Finally, a very special thanks to Anja for her incredible help in putting together the website, documentation, tests, and various releases.
Share and enjoy!
This is a 1.0 release candidate.
No changes will be made on stable before 1.0 except for targeted bugfixes with minimal likelihood of regressing anything else. If there are any more bugfixes to be applied, they will be cherry-picked from continuing develop. Either v0.99 or v0.99.1 will become the formal 1.0 in a few weeks.
User-facing changes in this version:
^Wto erase a word in the line editor
gCviews all columns on all sheets
vnow 'visibility toggle' (moved from
--version(thanks to @jsvine)
options.use_default_colors(thanks to @wavexx)
v0.98.1 is a patch release that fixes a couple of minor bugs. The primary change in this release, however, is that the 'visidata' package on PyPI no longer includes all of the loaders' dependencies by default.
This is a minor hassle for first-time users if they want to use certain formats, but since the goal is eventually supporting every possible data format, installing all dependencies for all users by default is not tenable. The base VisiData already includes support for tsv, csv, fixed width, sqlite, graphs and more. Most users will only need one or two additional dependencies.
To install VisiData with all dependencies for all loaders:
pip3 install "visidata[full]"
Of course, you can install just the dependencies you need. See "SUPPORTED SOURCES" in the manpage for which packages to install.
Title: VisiData v0.98, now with graphs and maps (experimental)
When I announced that v0.97 was "feature complete", I must have been talking about vdtui. vdtui has been pretty stable, even as VisiData has added a low-resolution pixel canvas (using Braille unicode characters) to draw graphs and maps. In fact, apart from a few bug fixes and small API changes, the only addition to vdtui in v0.98 is basic mouse support.
Here's a list of what's new in VisiData v0.98:
[canvas] graphs and maps!
g.to push a graph or a map from the current sheet (dot=plot)
uto select/unselect rows at canvas cursor
ENTERto push source sheet with only rows at canvas cursor
_to zoom out to full width
disp_pixel_randomoption chooses pixel attrs at random (weighted), instead of most common
-to zoom in/out via keyboard
Updates to commands
zzmoves cursor to center, uncertain about the future of
ztdue to conflict with
gaadds N new rows
gz=sets value for selected/all rows to a Python sequence in this column
z_sets column width to given value
z-cuts column width in half
Pis now "paste before" (like vim);
Rnow pushes a random sample
^Znow sends SIGSTOP;
^O"opens" the external $EDITOR (from builtin line editor)
~!@#$commands back, to set type of source columns
wis becoming a more universal "visibility toggle"
wtoggles display of the labels
wtoggles hidden properties and methods
Updates to command line args and options
-doption (separate from
--csv-delimiter) sets delimiter for tsv filetype
disp_date_fmtoption for date display format string (default is date-only)
false_is_nullset which values are considered null (previously was
--skiplinesoption renamed to
use_default_colors(at behest of @wavexx)
It's VisiData's 1st birthday! I started working on VisiData 1 year ago today (Oct 29).
v0.97.1 has only a few small patches:
Important If you like VisiData, sign up for the newsletter!
I'm hanging out in #visidata on freenode! Come by and say hi if you use VisiData. I'd love to know how you're using it!
r/visidata (on reddit) is the forum for longer-form discussion.
See the new webpage at visidata.org! We migrated away from RTD, as it was too much hassle and limited our functionality.
We have some example tours generated from our automated tests. See visidata.org/tour. More will be coming with the next release!
The main themes for this release are: 1) commandlog replay fully operational (now running both automated tests and generating tours) 2) a quick ref guide in man page format (available in visidata via F1, and also online at visidata.org/man 3) visidata.org is now the primary home page (we've transitioned away from readthedocs)
There are also the usual suspects: new loaders, new commands, new options, and too many improvements and bugfixes to name. It is getting ever easier to write a new command or loader in a few minutes and have it just work.
With this release, VisiData is feature complete. There will still be many reworkings, interface changes, added commands, tweaks, embellishments, arguments, and explanations, but as far as I can see, all of the functionality is now there. We are inching ever closer to a v1.0 that will hopefully stand the test of time.
The main goals for the next release (v0.98) are 1) a set of great tours, and 2) documentation of the internal architecture: design decisions, implementation guidelines, and API references.
Here are some more detailed descriptions of what comes with this release:
simple remote postgres loader (vd postgres://user:firstname.lastname@example.org:6254/database)
fixed width column detection (ls -al | vd -f fixed --skiplines 1)
F1 launches man page quick ref (gF1 now opens the command sheet)
vd --diff A B shows B with changes from A in red
a Describe Sheet with 'I' shows basic stats for every column!
streamlined commandlog (previously called editlog) functionality
^D to quickly save the current cmdlog; vd -p cmdlog-1.vd to replay it
removed movement commands (use --replay-movement to reinsert movements)
player piano with --delay (used to make the tours on the website)
advanced editing, can now make new sheets from whole cloth!
'A' to add a new sheet with some number of columns
'a' to add a new empty row
'gz=' set this column in selected rows to values from a python list expression
'Del'/gDel to clear a cell (set a value to None)
'y' to yank/delete row; 'p' to paste
'zy' to yank (copy) cell contents; 'zp' to paste ('gzp' to paste to all selected rows)
'f' to fill null cells with the previous non-null value
ctrl-ucommand to toggle profiling of main thread
options.col_statsstill disabled by default)