The process of designing a loader is:
open_foofunction that returns a new Sheet;
When VisiData tries to open a source with filetype
foo, it tries to call
open_foo(path), which should return an instance of
Sheet, or raise an error.
path is a
Path object of some kind.
def open_foo(p): return FooSheet(p.name, source=p) class FooSheet(Sheet): rowtype = 'foobits' # rowdef: foolib.Bar object
Sheetconstructor takes the new sheet name as its first argument. Any other keyword arguments are set as attributes on the new instance.
sourceis sufficient for most loaders, and so the subclass constructor can generally be omitted.
rowtypeis for display purposes only. It should be plural.
reload() is called when the Sheet is first pushed, and thereafter by the user with
Using the Sheet
class FooSheet(Sheet): ... def reload(self): self.rows =  for r in crack_foo(self.source): self.rows.append(r)
rowdefcomment should declare the internal structure of each row.
rowsmust be set to a new list object; do not call
The above code will probably work just fine for smaller datasets, but a large enough dataset will cause the interface to freeze. Fortunately, making an async loader is pretty straightforward:
@asyncthread decorator on the
reload method, which causes it to be launched in a new thread.
Wrap the iterator with
Progress. This updates the progress percentage as it passes each element through.
Append each row one at a time. Do not use a list comprehension; rows should become available as they are loaded.
Do not depend on the order of
rows after they are added; e.g. do not reference
rows[-1]. The order of rows may change during an asynchronous loader.
Exceptions that might be raised while handling a specific row, and add them as the row instead. Never use a bare
except: clause or the loader thread will not be killable.
class FooSheet(Sheet): ... @asyncthread def reload(self): self.rows =  for bar in Progress(foolib.iterfoo(self.source.open_text())): try: r = foolib.parse(bar) except Exception as e: r = e self.rows.append(r)
Test the loader with a large dataset to make sure that:
Each sheet has a unique list of
Column provides a different view into the row.
class FooSheet(Sheet): ... columns = [ ColumnAttr('name'), # foolib.Bar.name Column('bar', getter=lambda col,row: row.inside, setter=lambda col,row,val: row.set_bar(val)), Column('baz', type=int, getter=lambda col,row: row.inside*100) ]
In general, set
columns as a class member. If the columns aren't known until the data is being loaded,
reload() should first call
columns.clear(), and then call
addColumn(col) for each column at the earliest opportunity.
Columns have a few properties, all of which are optional arguments to the constructor except for
name: should be a valid Python identifier and unique among the column names on the sheet. (Otherwise the column cannot be used in an expression.)
type: can be
currency. By default it is
anytype, which passes the original value through unmodified.
width: the initial width for the column.
0 means hidden;
None (default) means calculate on first draw.
getter(col, row) and/or
setter(col, row, value): functions that get or set the value for a given row.
The getter is the essential functionality of a
In general, a
Column constructor is passed a
getter lambda. Columns with more complex functions should be subclasses and override
getter is passed the column instance
col and the
row, and returns the value of the cell. If the sheet itself is needed, it is available as
The default getter returns the entire row.
Column may also be given a
setter lambda, which allows the in-memory row to be modified. The
setter lambda is passed the column instance
row, and the new
value to be set.
In a Column subclass,
Column.setValue(self, row, value) may be overridden instead.
By default there is no
setter, which makes the column read-only.
There are several helpers for constructing
ColumnAttr(colname, attrname, **kwargs) gets/sets the
attrname attribute from the row object using
setattr (as in
attrname defaults to the
ColumnAttr is useful when the rows are Python objects.
ColumnItem(colname, itemkey, **kwargs) uses the builtin
setitem on the row (as in
itemkey also defaults to the
colname itself. This is useful when the rows are Python mappings or sequences, like dicts or lists.
SubrowColumn(newname, origcol, subrowidx, **kwargs) proxies for another Column, in which its row is nested in another sequence or mapping. This is useful on a sheet with augmented rows, like
tuple(orig_index, orig_row); each column on the original sheet would be wrapped in a
SubrowColumn(col.name, col, 1), since
orig_row is now
row. Used in joined sheets.
Recipes for a couple of recurring patterns:
[ColumnItem(name, i) for i, name in enumerate(colnames)]
[ColumnItem(k) for k in self.rows]
globalCommand() have the same signature:
[...]Command(default_keybinding, longname, execstr)
class FooSheet(Sheet): ... FooSheet.addCommand('b', 'reset-bar', 'cursorRow.set_bar(0)')
See the commands design document and commands checklist for more details.
This would be a completely functional read-only viewer for the fictional foolib. For a more realistic example, see the annotated viewtsv or any of the included loaders.
from visidata import * def open_foo(p): return FooSheet(p.name, source=p) class FooSheet(Sheet): rowtype = 'foobits' # rowdef: foolib.Bar object columns = [ ColumnAttr('name'), # foolib.Bar.name Column('bar', getter=lambda col,row: row.inside, setter=lambda col,row,val: row.set_bar(val)), Column('baz', type=int, getter=lambda col,row: row.inside*100) ] @asyncthread def reload(self): import foolib self.rows =  for bar in Progress(foolib.iterfoo(self.source.open_text())): try: r = foolib.parse(bar) except Exception as e: r = e self.rows.append(r) FooSheet.addCommand('b', 'reset-bar', 'cursorRow.set_bar(0)')
A full-duplex loader requires a saver. The saver iterates over all
getTypedValue, and saves the results in the format of that filetype.
@asyncthread def save_foo(path, sheet): with p.open_text(mode='w') as fp: for i, row in enumerate(Progress(sheet.rows)): for col in sheet.visibleCols: foolib.write(fp, i, col.name, col.getValue(row))
The saver should preserve the column names and translate their types into foolib semantics, but other attributes on the Columns should generally not be saved.
When VisiData tries to open a URL with schemetype of
foo (i.e. starting with
foo://), it calls
urlpath is a
UrlPath object, with attributes for each of the elements of the parsed URL.
openurl_foo should return a Sheet or call
error(). If the URL indicates a particular type of Sheet (like
magnet://), then it should construct that Sheet itself. If the URL is just a means to get to another filetype, then it can call
openSource with a Path-like object that knows how to fetch the URL:
def openurl_foo(p, filetype=None): return openSource(FooPath(p.url), filetype=filetype)