Queryverse

VegaLite.jl v2.1 released

2020-04-06T00:00:00+00:00

I just released VegaLite.jl v2.1. While this release just tracks what is happening upstream in the Vega-Lite project, there are some really nice feature additions.

For all the details you can read through the release notes of Vega-Lite:

I’ll use the rest of this post to highlight a few key new features.

Arc marks

Vega-Lite now supports a new arc mark that makes it easy to create pie, donut and radial charts:

For more information, you can read the original Vega-Lite documentation and you can look at a number of Julia examples.

Datum in encoding

The new support for datum in encodings makes it easy to add for example a rule at a given point to a figure.

An example that adds a horizontal rule to a figure is this:

using VegaLite, VegaDatasets

dataset("stocks") |>
@vlplot() +
@vlplot(:line, "date:t", :price, color=:symbol) +
@vlplot(mark={:rule, strokeDash=[2,2], size=2}, y={datum=300})

Note how the y channel here is given a datum, that is a value from the underlying price domain.

For temporal values this has especially nice syntax:

using VegaLite, VegaDatasets

dataset("stocks") |>
@vlplot() +
@vlplot(:line, "date:t", :price, color=:symbol) +
@vlplot(mark={:rule, strokeDash=[2,2], size=2}, x={datum={year=2006}})

Note how we can set a value for a specific year here.

Angle channel for point and text marks

A new angle channel makes it easy to create for example the following wind vector map:

You can find the Julia code for this example in the documentation.

Conclusion

This release was literally zero work on the Julia side of things, all the thanks for these new features should go to the awesome Vega-Lite team that keeps adding new functionality at an amazing speed!

VegaLite.jl v2.0.0 released

2020-03-11T00:00:00+00:00

We released VegaLite.jl v2.0 a couple of days ago. This new version brings a lot of new features along! I’ll try to walk you through some of them in this blog post.

For those of you not familiar with the package: VegaLite.jl is a powerful plotting package for Julia.

Vega-Lite 4

We updated the Julia package to use the latest version of the underlying JavaScript library, Vega-Lite 4. This brings a ton of new features to the table. You can take a look at a detailed description of all these updates by looking at the release notes for all the Vega-Lite versions we now incorporate:

I encourage you to take a look at these release notes, they have examples for each new feature and give a good overview of what is happening in the underlying library. There is a lot of good stuff (regression, loess, density and quantile transforms, a strokeDash encoding channel, lots of new interactive stuff and a ton of other features)!

Support for inline data

We now support inline data, i.e. you can now create a plot from vectors of data directly, without the need to pass a tabular data structure like DataFrame to the plot function. The following example plots a scatter plot of two vectors of random numbers:

using VegaLite

a_vector = rand(10)

@vlplot(:point, x=rand(10), y=a_vector)

You can also pass vectors inside a composite value:

using VegaLite 

@vlplot(:line, x={1:10, title="Timestep"}, y=rand(10))

Support for positional `x` and `y` encodings

The values for the x and y encoding channel can now be passed as positional arguments, thus further reducing the amount of code required for common plots.

In particular, the second positional argument is now interpreted as the x channel, and the third positional argument as the y channel.

A simple example that uses some tabular data is this:

using VegaLite, VegaDatasets

dataset("cars") |> @vlplot(:point, :Acceleration, :Miles_per_Gallon)

The combination of positional arguments and inline data makes the code for a simple scatter plot of two vectors really concise:

using VegaLite

a = rand(10)
b = rand(10)

@vlplot(:point, a, b)

Save plots as HTML files

You can now save a plot directly as a HTML file:

using VegaLite

@vlplot(:point, rand(10), rand(10)) |> save("figure.html")

These HTML files are self contained, and are especially useful for interactive charts (which will fully work if someone opens the generated HTML file in a web browser).

Much better support for Vega specs

The package now exports a new @vgplot macro. It works exactly like @vlplot, except that you can use it to create Vega plots. Vega is the lower-level plotting library that Vega-Lite uses under the hood. It is more verbose than Vega-Lite, but gives you much more control over the types of plots you can create. Take a look at the Vega examples to get a sense.

Better Juno integration

Interactive plots should now work properly in Juno.

Convert plots into Julia code (experimental)

You can now take a plot object and call a function to get some Julia code that would re-create exactly that plot object. Here is a simple example:

using VegaLite, VegaDatasets

p = dataset("cars") |> @vlplot(:point, :Acceleration, :Miles_per_Gallon)

VegaLite.printrepr(stdout, p)

This will output:

@vlplot(mark="point",encoding={x={field="Acceleration"},y={field="Miles_per_Gallon"}})

This standalone example is probably not very useful, but this functionality can be very convenient if you obtain the plot from some other source.

For example, here is an example where you copy a Vega-Lite JSON example and convert it into an equivalent Julia code representation:

using VegaLite

p = vl"""
{
    "mark": "point",
    "encoding": {
        "x": {
            "field": "Miles_per_Gallon"
        },
        "y": {
            "field": "Acceleration"
        }
    }
}
"""

VegaLite.printrepr(stdout, p)

This also outputs the Julia code that would create this spec:

@vlplot(encoding={x={field="Miles_per_Gallon"},y={field="Acceleration"}},mark="point")

Another scenario is that you load a Vega-Lite spec from disc and directly convert it into Julia code:

using VegaLite

VegaLite.printrepr(stdout, load("figure.vegalite"))

Or maybe you used DataVoyager.jl to interactively create a plot and now want to include some code that creates that same figure, but in a non-interactive way:

using DataVoyager, VegaDatasets

# Run Voyager

w = dataset("cars") |> Voyager()

# At this point you would interactively create your plot and proceed once you are done with that

# Extract the plot from the UI
p = w[]

# Convert the interactively created plot into Julia code
VegaLite.printrepr(stdout, p)

New examples

We have a ton of new examples, both for Vega and Vega-Lite plots in the documentation, check them out here!

New expert APIs (experimental)

We now also export a purely non-macro based API. The two functions for that are vlplot and vgplot. Normal users are encouraged to continue to use the @vlplot and @vgplot macros, these new functions are mostly meant for some special situations where other packages can’t use the macro versions of these calls.

We also export new macros and functions called @vlfrag, @vgfrag, vlfrag and vgfrag. These allow you to create spec fragments and then pass these to the main level @vlplot, @vgplot, vlplot and vgplot macros and functions.

For example, in this example I break up the call to @vlplot into two calls by using the @vlfrag macro:

using VegaLite, VegaDatasets

x_frag = @vlfrag(:Miles_per_Gallon, title="Custom title")

dataset("cars") |> @vlplot(:point, x=x_frag)

We again discourage use of this feature in normal user code, this is another feature that is probably most useful for package authors that want to make use of VegaLite.jl.

Bug fixes and performance

There are a ton of bug fixes and performance improvements in this version:

We handle inline data much more efficiently
We auto-encode encoding types in sub specs
We properly apply shorthands in sub specs

Thanks

This release had a lot of folks contributing, I in particular want to thank oheil, mcmcgrath13 and tkf for their help!

ElectronDisplay.jl v1.0.0 Released

2020-02-05T00:00:00+00:00

ElectronDisplay.jl is a package that provides a simple UI based on Electron for displaying plots and tabular data. It works with most plotting packages and almost all data structures that represent tabular data in the Julia ecosystem. The package is a core piece of the Queryverse.

What is new?

There are a bunch of bugfixes as well as some new features in v1.0.0. The main new feature is the plotgallery, an interactive plot display UI that is written in TypeScript and based on React. It supports displaying multiple plots in the same window simultaniously, even if those plots are created with different plotting packages. For example, you can display a Gadfly.jl figure and then plot a VegaLite.jl plot, or multiple VegaLite.jl plots, and they would show in a single window instead of opening a new window each time or replacing the old plots.

Some functionalities in detail

Press CTRL + C on Windows and Linux or Command + C on macOS to copy the current figure to the system clipboard
Press arrow keys to switch between plots quickly and delete/backspace to remove a graph
Supports Vega, VegaLite, and several image formats (such as PNG, JPG, and SVG)
Support viewport-sized plots that resize according to the window size

You can have a look at the detailed release notes here.

Contributions are welcomed

ElectronDisplay.jl is still a new package that needs help from the community to resolve the current issues that it has. In fact, as an apprentice of the Undergraduate Research Apprentice Program supervised by David Anthoff, I have been working on several issues on this project over the past few weeks after I made the initial pull request. If you are interested in making contributions to the project, you can have a look at Issues for ElectronDisplay.jl, or Issues for Plotgallery.

Announcing ElectronDisplay.jl

2019-02-13T00:00:00+00:00

Plot display

Anytime you load ElectronDisplay.jl via using ElectonDisplay, it automatically becomes the default display for you current Julia session. From that moment on, whenever you display a figure from any of the supported plotting packages, it will show up in a window that is managed by ElectronDisplay.jl. Any plotting package that hooks into the standard Julia multimedia I/O API will work with this setup, as long as it supports either the PNG or SVG format with its show method.

ElectronDisplay.jl provides more tightly integrated support for PlotlyJS.jl and VegaLite.jl: plots from both packages support a full range of interactive features inside the window that ElectronDisplay.jl opens for them.

Here is a simple example of how this works. Executing the following example code

julia> using VegaLite, VegaDatasets, ElectronDisplay

julia> dataset("cars") |> @vlplot(:point, x=:Acceleration, y=:Miles_per_Gallon, color=:Origin)

will open a window that displays this plot:

Table display

Thanks to tks’s fantastic recent contribution, ElectronDisplay.jl now also has support for showing tabular data in a grid. To display a table in that way, you simply pass it to the electrondisplay function, and it will show in a new window.

The following example demonstrates how easy it is to display a DataFrame:

julia> using DataFrames, ElectronDisplay

julia> df = DataFrame(a=rand(100), b=rand(100), c=rand(100));

julia> electrondisplay(df)

The table view works with almost any tabular source, not just DataFrames.jl. For example, here we are showing the result of a Query.jl query directly in the grid, without ever materializing it into a DataFrame:

julia> using VegaDatasets, Query, ElectronDisplay

julia> dataset("cars") |>
       @filter(_.Origin=="USA") |>
       @select(-:Miles_per_Gallon) |>
       electrondisplay

And the results are shown in a window like this:

The table display in ElectronDisplay.jl works for any source that either supports a show method for the application/vnd.dataresource+json MIME type, or implements the TableTraits.jl interface. Anything that iterates named tuples automatically is a TableTraits.jl source, so even packages that have never heard of either option might well work with ElectronDisplay.jl.

Summary

ElectronDisplay.jl is probably most useful for people that work directly in the Julia REPL and not in one of the more fully feature IDEs. It is also a very young package, and any help with the package would be most welcome! I have created a few issues with ideas for improvements, but I am sure there are many more. I think this could be a project where someone with web UI experience could really contribute significantly.

Query.jl v0.11 released

2019-02-02T00:00:00+00:00

We just released Query.jl v0.11. The new version adds four new standalone query operators: @select, @rename, @mutate and @unique. We also fixed a number of bugs and wrote a bunch of new documentation. Finally, the standalone versions of the query operators are no longer considered experimental, so feel free to use the versions that use the pipe operator |> without any worries that they might go away in the future. The rest of this post will showcase the new query operators we added.

The `@select` query operator

The standalone @select query operator makes it easier to select a subset of columns from a source table and arrange them in some specified order. Note that the discussion in this section is about the standalone @select query operator, not the LINQ style version of @select.

In the most basic example, you can simply specify all the columns that you want to include in the result table. You can select columns either by their name, or their position.

Lets first create a DataFrame with some example data that we will use as the basis for most examples in this post:

julia> using Query, DataFrames

julia> df = DataFrame(firstCol=rand(10), colA=rand(10), colB=rand(10), colC=rand(10), lastCol=rand(10))
10×5 DataFrame
│ Row │ firstCol    │ colA      │ colB       │ colC      │ lastCol   │
│     │ Float64     │ Float64   │ Float64    │ Float64   │ Float64   │
├─────┼─────────────┼───────────┼────────────┼───────────┼───────────┤
│ 1   │ 0.0815091   │ 0.726354  │ 0.164546   │ 0.419195  │ 0.0685362 │
│ 2   │ 0.984546    │ 0.132284  │ 0.00516908 │ 0.619361  │ 0.847543  │
│ 3   │ 0.452009    │ 0.421744  │ 0.443238   │ 0.581122  │ 0.518726  │
│ 4   │ 0.551547    │ 0.204774  │ 0.80987    │ 0.0977708 │ 0.866429  │
│ 5   │ 0.0537391   │ 0.362015  │ 0.760544   │ 0.775301  │ 0.810517  │
│ 6   │ 0.837029    │ 0.0614884 │ 0.596101   │ 0.0613467 │ 0.230335  │
│ 7   │ 0.701862    │ 0.702575  │ 0.714068   │ 0.283429  │ 0.308963  │
│ 8   │ 0.626259    │ 0.246562  │ 0.573619   │ 0.869932  │ 0.148938  │
│ 9   │ 0.000844883 │ 0.432564  │ 0.289367   │ 0.92641   │ 0.592507  │
│ 10  │ 0.365743    │ 0.666146  │ 0.932312   │ 0.352096  │ 0.684766  │

Now we select the column named colB, the second column and the column named lastCol into a new table:

julia> df |> @select(:colB, 2, :lastCol)
10x3 query result
colB       │ colA      │ lastCol
───────────┼───────────┼──────────
164546   │ 0.726354  │ 0.0685362
00516908 │ 0.132284  │ 0.847543
443238   │ 0.421744  │ 0.518726
80987    │ 0.204774  │ 0.866429
760544   │ 0.362015  │ 0.810517
596101   │ 0.0614884 │ 0.230335
714068   │ 0.702575  │ 0.308963
573619   │ 0.246562  │ 0.148938
289367   │ 0.432564  │ 0.592507
932312   │ 0.666146  │ 0.684766

Note that we are achieving two things at the same time here: we are picking only a few of the columns from the source table, and we are rearranging the order of the columns.

There are a whole number of options available that make it easier to select multiple columns without specifying each individually.

First, we can use the standard julia range syntax to select consecutive columns:

julia> df |> @select(:colB : :lastCol, :colA)
10x4 query result
colB       │ colC      │ lastCol   │ colA
───────────┼───────────┼───────────┼──────────
164546   │ 0.419195  │ 0.0685362 │ 0.726354
00516908 │ 0.619361  │ 0.847543  │ 0.132284
443238   │ 0.581122  │ 0.518726  │ 0.421744
80987    │ 0.0977708 │ 0.866429  │ 0.204774
760544   │ 0.775301  │ 0.810517  │ 0.362015
596101   │ 0.0613467 │ 0.230335  │ 0.0614884
714068   │ 0.283429  │ 0.308963  │ 0.702575
573619   │ 0.869932  │ 0.148938  │ 0.246562
289367   │ 0.92641   │ 0.592507  │ 0.432564
932312   │ 0.352096  │ 0.684766  │ 0.666146

In that example, we first selected all columns between colB and lastCol (so that would be colB, colC and lastCol), and then we also included colA as the second argument to the @select call.

We can also use the range syntax with column position indices:

julia> df |> @select(3:5, :colA)
10x4 query result
colB       │ colC      │ lastCol   │ colA
───────────┼───────────┼───────────┼──────────
164546   │ 0.419195  │ 0.0685362 │ 0.726354
00516908 │ 0.619361  │ 0.847543  │ 0.132284
443238   │ 0.581122  │ 0.518726  │ 0.421744
80987    │ 0.0977708 │ 0.866429  │ 0.204774
760544   │ 0.775301  │ 0.810517  │ 0.362015
596101   │ 0.0613467 │ 0.230335  │ 0.0614884
714068   │ 0.283429  │ 0.308963  │ 0.702575
573619   │ 0.869932  │ 0.148938  │ 0.246562
289367   │ 0.92641   │ 0.592507  │ 0.432564
932312   │ 0.352096  │ 0.684766  │ 0.666146

We also provide a number of convenience functions for bulk column selection. For example, to select all columns that have a name that starts with col, we can write:

julia> df |> @select(startswith("col"))
10x3 query result
colA      │ colB       │ colC
──────────┼────────────┼──────────
726354  │ 0.164546   │ 0.419195
132284  │ 0.00516908 │ 0.619361
421744  │ 0.443238   │ 0.581122
204774  │ 0.80987    │ 0.0977708
362015  │ 0.760544   │ 0.775301
0614884 │ 0.596101   │ 0.0613467
702575  │ 0.714068   │ 0.283429
246562  │ 0.573619   │ 0.869932
432564  │ 0.289367   │ 0.92641
666146  │ 0.932312   │ 0.352096

Or we can select all columns that have a name that ends with Col:

julia> df |> @select(endswith("Col"))
10x2 query result
firstCol    │ lastCol
────────────┼──────────
0815091   │ 0.0685362
984546    │ 0.847543
452009    │ 0.518726
551547    │ 0.866429
0537391   │ 0.810517
837029    │ 0.230335
701862    │ 0.308963
626259    │ 0.148938
000844883 │ 0.592507
365743    │ 0.684766

Lets say we have a situation where we want to select a large number of columns that we can easily select with one of the bulk column options discussed so far, except for one specific column. Here is how we can run such a query:

julia> df |> @select(startswith("col"), -:colB)
10x2 query result
colA      │ colC
──────────┼──────────
726354  │ 0.419195
132284  │ 0.619361
421744  │ 0.581122
204774  │ 0.0977708
362015  │ 0.775301
0614884 │ 0.0613467
702575  │ 0.283429
246562  │ 0.869932
432564  │ 0.92641
666146  │ 0.352096

Here we first select all columns that have a name that starts with col. At this point we have selected the columns colA, colB and colC. Next, we remove colB from this list, by specifying a minus - in front of its name as the next argument.

There is one twist to removing columns: if the first argument to the @select query operator starts with a minus, then we implicitly start out with all columns selected. Here is an example that demonstrates this:

julia> df |> @select(-:colB)
10x4 query result
firstCol    │ colA      │ colC      │ lastCol
────────────┼───────────┼───────────┼──────────
0815091   │ 0.726354  │ 0.419195  │ 0.0685362
984546    │ 0.132284  │ 0.619361  │ 0.847543
452009    │ 0.421744  │ 0.581122  │ 0.518726
551547    │ 0.204774  │ 0.0977708 │ 0.866429
0537391   │ 0.362015  │ 0.775301  │ 0.810517
837029    │ 0.0614884 │ 0.0613467 │ 0.230335
701862    │ 0.702575  │ 0.283429  │ 0.308963
626259    │ 0.246562  │ 0.869932  │ 0.148938
000844883 │ 0.432564  │ 0.92641   │ 0.592507
365743    │ 0.666146  │ 0.352096  │ 0.684766

Because we start this @select query operator with an argument that has a minus - in front, we start out with all columns, and then remove the colB column.

The minus - can of course also be combined with all the bulk column options. The next example selects all columns from the source table, except if they have a column name that starts with col:

julia> df |> @select(-startswith("col"))
10x2 query result
firstCol    │ lastCol
────────────┼──────────
0815091   │ 0.0685362
984546    │ 0.847543
452009    │ 0.518726
551547    │ 0.866429
0537391   │ 0.810517
837029    │ 0.230335
701862    │ 0.308963
626259    │ 0.148938
000844883 │ 0.592507
365743    │ 0.684766

If you are familiar with dplyr all of this will be highly familiar: we largely copied the semantics from that package.

The `@rename` query operator

The @rename query operator renames one or more columns from a table. Each rename is specified as a Pair, where the first element is either the old name of the column that should be renamed, or the position of the column. The second element must be a Symbol with the new name for the column. Here is an example:

julia> df |> @rename(:colB=>:foo, 1=>:bar)
10x5 query result
bar         │ colA      │ foo        │ colC      │ lastCol
────────────┼───────────┼────────────┼───────────┼──────────
0815091   │ 0.726354  │ 0.164546   │ 0.419195  │ 0.0685362
984546    │ 0.132284  │ 0.00516908 │ 0.619361  │ 0.847543
452009    │ 0.421744  │ 0.443238   │ 0.581122  │ 0.518726
551547    │ 0.204774  │ 0.80987    │ 0.0977708 │ 0.866429
0537391   │ 0.362015  │ 0.760544   │ 0.775301  │ 0.810517
837029    │ 0.0614884 │ 0.596101   │ 0.0613467 │ 0.230335
701862    │ 0.702575  │ 0.714068   │ 0.283429  │ 0.308963
626259    │ 0.246562  │ 0.573619   │ 0.869932  │ 0.148938
000844883 │ 0.432564  │ 0.289367   │ 0.92641   │ 0.592507
365743    │ 0.666146  │ 0.932312   │ 0.352096  │ 0.684766

Here we are renaming the colB column to foo, and the first column to bar. This example highlights that one can rename multiple columns in a single call to @rename.

All other columns in a source table are left unmodified by the @rename query operator, and the order of columns is also not changed.

The semantics of this query operator once again follow the equivalent dplyr function.

The `@mutate` query operator

The @mutate query operator can be used to either add columns to a table, or mutate the content of some columns in a table. Lets say we want to add a new column to our table that is the sum of column colA and the log of column colB:

julia> df |> @mutate(newCol = _.colA + log(_.colB))
10x6 query result
firstCol    │ colA      │ colB       │ colC      │ lastCol   │ newCol
────────────┼───────────┼────────────┼───────────┼───────────┼────────────
0815091   │ 0.726354  │ 0.164546   │ 0.419195  │ 0.0685362 │ -1.07821
984546    │ 0.132284  │ 0.00516908 │ 0.619361  │ 0.847543  │ -5.13278
452009    │ 0.421744  │ 0.443238   │ 0.581122  │ 0.518726  │ -0.391904
551547    │ 0.204774  │ 0.80987    │ 0.0977708 │ 0.866429  │ -0.00610799
0537391   │ 0.362015  │ 0.760544   │ 0.775301  │ 0.810517  │ 0.0882928
837029    │ 0.0614884 │ 0.596101   │ 0.0613467 │ 0.230335  │ -0.455857
701862    │ 0.702575  │ 0.714068   │ 0.283429  │ 0.308963  │ 0.365798
626259    │ 0.246562  │ 0.573619   │ 0.869932  │ 0.148938  │ -0.309229
000844883 │ 0.432564  │ 0.289367   │ 0.92641   │ 0.592507  │ -0.807497
365743    │ 0.666146  │ 0.932312   │ 0.352096  │ 0.684766  │ 0.596058

We pass each column we want compute for the output table as a keyword argument to the @mutate query operator. The name of the keyword argument will become the name of the column that we are computing. To the right of the equal sign = we pass an expression that computes the value for the current row for the new column. We have access to the current row via the _ syntax (in the same way it is used in all the other standalone query operators). In our example this allows us for example to reference the value of colA and colB in the current row _. If the output name for the column (in our case newCol) does not exist as a column in the input table, a new column with that name will be added at the end of the table.

The next example uses an output name for the column that already exists, in which case the content of that column will be replaced with the new, computed values:

julia> df |> @mutate(firstCol = _.colA + log(_.colB))
10x5 query result
firstCol    │ colA      │ colB       │ colC      │ lastCol
────────────┼───────────┼────────────┼───────────┼──────────
-1.07821    │ 0.726354  │ 0.164546   │ 0.419195  │ 0.0685362
-5.13278    │ 0.132284  │ 0.00516908 │ 0.619361  │ 0.847543
-0.391904   │ 0.421744  │ 0.443238   │ 0.581122  │ 0.518726
-0.00610799 │ 0.204774  │ 0.80987    │ 0.0977708 │ 0.866429
0.0882928   │ 0.362015  │ 0.760544   │ 0.775301  │ 0.810517
-0.455857   │ 0.0614884 │ 0.596101   │ 0.0613467 │ 0.230335
0.365798    │ 0.702575  │ 0.714068   │ 0.283429  │ 0.308963
-0.309229   │ 0.246562  │ 0.573619   │ 0.869932  │ 0.148938
-0.807497   │ 0.432564  │ 0.289367   │ 0.92641   │ 0.592507
0.596058    │ 0.666146  │ 0.932312   │ 0.352096  │ 0.684766

One can also specify multiple output columns in a single call to @mutate, for example the following code replaces the values in one existing column (named firstCol) and adds a new column named newCol.

julia> df |> @mutate(firstCol = _.colA + log(_.colB), newCol=_.colC * 2)
10x6 query result
firstCol    │ colA      │ colB       │ colC      │ lastCol   │ newCol
────────────┼───────────┼────────────┼───────────┼───────────┼─────────
-1.07821    │ 0.726354  │ 0.164546   │ 0.419195  │ 0.0685362 │ 0.838391
-5.13278    │ 0.132284  │ 0.00516908 │ 0.619361  │ 0.847543  │ 1.23872
-0.391904   │ 0.421744  │ 0.443238   │ 0.581122  │ 0.518726  │ 1.16224
-0.00610799 │ 0.204774  │ 0.80987    │ 0.0977708 │ 0.866429  │ 0.195542
0.0882928   │ 0.362015  │ 0.760544   │ 0.775301  │ 0.810517  │ 1.5506
-0.455857   │ 0.0614884 │ 0.596101   │ 0.0613467 │ 0.230335  │ 0.122693
0.365798    │ 0.702575  │ 0.714068   │ 0.283429  │ 0.308963  │ 0.566858
-0.309229   │ 0.246562  │ 0.573619   │ 0.869932  │ 0.148938  │ 1.73986
-0.807497   │ 0.432564  │ 0.289367   │ 0.92641   │ 0.592507  │ 1.85282
0.596058    │ 0.666146  │ 0.932312   │ 0.352096  │ 0.684766  │ 0.704192

Note that in all of these examples, @mutate leaves any column that is not named explicitly as an output column alone.

And it might not come as a surprise at this point: we again followed the lead of the equivalent dplyr function in the design.

The `@unique` query operator

The @unique query operator is simple: it removes any duplicate values from an input sequence. Here is a simple example where we remove duplicate values from an array of Int values:

julia> [1,2,3,4,5,2,4] |> @unique()
?-element query result
 1
 2
 3
 4
 5

One can of course also use this with tabular data:

 julia> df2 = DataFrame(a=[1,2,3,2], b=[4,3,2,3])
4×2 DataFrame
│ Row │ a     │ b     │
│     │ Int64 │ Int64 │
├─────┼───────┼───────┤
│ 1   │ 1     │ 4     │
│ 2   │ 2     │ 3     │
│ 3   │ 3     │ 2     │
│ 4   │ 2     │ 3     │

julia> df2 |> @unique()
3x2 query result
a │ b
──┼──
1 │ 4
2 │ 3
3 │ 2

Conclusion

As always, please let us know about any issues or problems that you might run into while using Query.jl over in the github repository.

Welcome to the new Queryverse homepage

2019-01-19T00:00:00+00:00

Welcome to the new homepage for all things Queryverse! Going forward we will use this blog to announce new features and generally explain the packages that make up the Queryverse.

Query.jl v0.10 released

2018-08-29T00:00:00+00:00

I just released Query.jl v0.10. This is the first version that is compatible with julia 1.0, and at the same time it drops support for older julia versions.

This version comes with two breaking changes, other than that there are no new features.

New way to select group key

In previous versions one selected the value of a group key via the .key syntax, like in this example:

@from i in df begin
    @group i by i.children into g
    @select {Key=g.key,Count=length(g)}
    @collect DataFrame
end

This syntax is no longer support, instead you have to use the key function to access the key of a group:

@from i in df begin
    @group i by i.children into g
    @select {Key=key(g),Count=length(g)}
    @collect DataFrame
end

`..` syntax removed

In previous versions one could use the a..b syntax as a shortcut for map(i->i.b, a). This syntax has been removed, and instead groups of table rows now support access to a whole column via a simple a.b syntax. For example, a typical groupby query might now look like this:

@from i in df begin
    @group i by i.state into g
    @select {group=key(g),mage=mean(g.age), oldest=maximum(g.age), youngest=minimum(g.age)}
    @collect DataFrame
end

Youtube tutorial - Introduction to Queryverse.jl

2018-06-18T00:00:00+00:00

I recorded an online tutorial to the Queryverse.jl last week. You can watch the tutorial here.

The tutorial covers tabular file IO, data manipulation with Query.jl and visual data exploration with VegaLite.jl and DataVoyager.jl.

Any feedback would be most welcome!

Query.jl v0.9.x released

2017-12-20T00:00:00+00:00

I just released Query.jl v0.9.0. The new version adds the @take and @drop standalone query operators and brings pretty printing to uncollected queries.

Pretty printing

In previous versions queries displayed a really awful mess of internal data when they were displayed in the REPL. In practice one always had to collect a query into something like a DataFrame to get a nice view of the query result. The new version changes that and provides a nice output for any query, even an uncollected one. Here is an example:

julia> using FileIO, Query, CSVFiles

julia> filename = "https://gist.githubusercontent.com/davidanthoff/bebfd24c1a3f32f576eb61bee77f5944/raw/dd9233ad860037a2155f3a9ca3c37eb2d5572573/testdata2.csv";

julia> load(filename) |> @map({_.Year, _.Cause_Name})
15028x2 query result
Year │ Cause_Name
─────┼───────────────────────
1999 │ Unintentional Injuries
1999 │ Unintentional Injuries
1999 │ Unintentional Injuries
1999 │ Unintentional Injuries
1999 │ Unintentional Injuries
1999 │ Unintentional Injuries
1999 │ Unintentional Injuries
1999 │ Unintentional Injuries
1999 │ Unintentional Injuries
1999 │ Unintentional Injuries
... with 15018 more rows

The pretty printing should work for the values returned from any of the query operators. The output format is heavily inspired by R’s tibbles.

I hope this will make interactive work much more pleasant because it should be easier to build up more complicated queries step by step, while periodically running a query to check intermediate results.

I also plan to add this to the whole tabular file IO of the iterable tables ecosystem at a later date (e.g. CSVFiles.jl, FeatherFiles.jl, ExcelFiles.jl, StatFiles.jl etc.).

The @take and @drop query commands

Those are fairly straightforward: both of these filter elements out of a sequence. @take limits the number of elements to some upper maximum, and @drop skips a number of elements. Here is an example of how one can use these:

using FileIO, Query, CSVFiles

filename = "https://gist.githubusercontent.com/davidanthoff/bebfd24c1a3f32f576eb61bee77f5944/raw/dd9233ad860037a2155f3a9ca3c37eb2d5572573/testdata2.csv"

load(filename) |>
    @filter(_.Cause_Name!="All Causes" && !isnull(_.Age_adjusted_Death_Rate)) |>
    @groupby(_.Cause_Name) |>
    @map({cause=_.key, death_rate=sum(_..Age_adjusted_Death_Rate)}) |>
    @orderby_descending(_.death_rate) |>
    @drop(2) |>
    @take(3) |>
    save("output.feather")

This example showcases a whole range of features, including the use of the @drop and @take operations. The official documentation for these two new operators is in the “Experimental Features” section in the Query.jl documentation.

Any feedback on these new features (and old ones) is most welcome, and of course any help with the overall package would also be fantastic!

This post is being discussed here.

Query.jl v0.8.x released

2017-11-21T00:00:00+00:00

I just released Query.jl v0.8.0. The new version has some breaking renames in the experimental parts of the package (that is why they are experimental!), extends the set of experimental standalone query commands, adds a slight twist to the experimental anonymous function syntax and ships with a whole bunch of package refactoring under the hood.

Renamed experimental standalone query commands

The @select standalone command was renamed to @map, and the @where command to @filter. Those names are more in line with julia conventions.

This change only applies to the experimental standalone versions of the query commands that you would use with the pipe operator. Nothing has changed about the LINQ syntax, i.e. that part will retain the SQL-like terms from LINQ. I have no plans to change those terms going forward, i.e. this does not indicate that there are any breaking changes planned for the stable part of LINQ.

New experimental standalone query commands

This release adds support for the @groupjoin, @join and @mapmany standalone query commands, i.e. you can now use those with the pipe syntax. This part of the documentation describes the arguments to those commands.

Support for two arguments in the experimental anonymous function syntax

Some of new standalone query commands require an anonymous function that takes two arguments. The existing experimental shortcut syntax for creating anonymous functions has been extended to support that scenario. To create a two argument anonymous function, you simply have to use both _ and __ (double underscore) in the expression that should be turned into an anonymous function. For example {a=_, b=__} will be translated into (i1,i2)->{a=i1, b=i2}.

Package refactoring

This release continues the breakup of the very large package Query.jl into smaller packages that do specific things. The current situation now is this:

IteratorInterfaceExtensions.jl defines a number of small extensions to the base julia iterator interface that are used by both Query.jl and the iterable tables universe.
QueryOperators.jl contains the definition of the query operators and the default iterator based backend implementation.
Query.jl now contains the syntax for the two supported front-ends: the traditional LINQ style syntax and the new experimental standalone commands.
TableTraits.jl defines a very minimal interface for tabular data interop.
TableTraitsUtils.jl provides some helper functions that make it easier to implement the interface defined in TableTraits.jl. Some packages use this to implement the table traits interface, but others don’t need it.
IterableTables.jl contains all the integrations for various packages with the iterable tables ecosystem that have not yet moved into those packages themselves.

This refactoring significantly simplifies the dependency situation with these packages. The dependency graph now looks roughly like this:

                                             IterableTables
                                            /
                                           /- TableTraitUtils
                                          /
                              TableTraits
                            /
IteratorInterfaceExtensions
                            \
                              QueryOperators
                                            \
                                             Query

Thanks

Thanks as always to all the folks that contributed to this effort with bug reports, suggestions and PRs. This release also got some new benchmarks that were contributed by floswald that will hopefully trigger some performance improvements going forward.

Please do report any bugs and suggestions back! And help with this whole effort is of course also always most welcome.

This post is being discussed here.

Queryverse

VegaLite.jl v2.1 released

Arc marks

Datum in encoding

Angle channel for point and text marks

Conclusion

VegaLite.jl v2.0.0 released

Vega-Lite 4

Support for inline data

Support for positional x and y encodings

Save plots as HTML files

Much better support for Vega specs

Better Juno integration

Convert plots into Julia code (experimental)

New examples

New expert APIs (experimental)

Bug fixes and performance

Thanks

ElectronDisplay.jl v1.0.0 Released

What is new?

Some functionalities in detail

Contributions are welcomed

Announcing ElectronDisplay.jl

Plot display

Table display

Summary

Query.jl v0.11 released

The @select query operator

The @rename query operator

The @mutate query operator

The @unique query operator

Conclusion

Welcome to the new Queryverse homepage

Query.jl v0.10 released

New way to select group key

.. syntax removed

Youtube tutorial - Introduction to Queryverse.jl

Query.jl v0.9.x released

Pretty printing

The @take and @drop query commands

Query.jl v0.8.x released

Renamed experimental standalone query commands

New experimental standalone query commands

Support for two arguments in the experimental anonymous function syntax

Package refactoring

Thanks

Support for positional `x` and `y` encodings

The `@select` query operator

The `@rename` query operator

The `@mutate` query operator

The `@unique` query operator

`..` syntax removed