
Suppose you're developing a Powerpipe mod, called `my-aws-insights`, to visualize AWS resources. It includes a dashboard for S3 buckets, with two charts and a table.

<div>
<img src="/images/blog/2024-05-flexible-mod-dependencies/initial-dashboard.png" />
</div>

The source code for the mod includes HCL widgets to define the charts and the table, and also includes the queries that fill those widgets with data.

Now suppose you'd like to source those queries from a separate mod. A couple of new features in Powerpipe — installing from the local filesystem, or from a remote branch — make it much easier to manage that separation of concerns. 

## Install and use a local dependency

Our mod lives in the local directory `~/my-aws-insights`. Now we'll create a new directory, `~/aws-queries`, initialize it with [powerpipe init](https://powerpipe.io/docs/reference/cli/mod#powerpipe-mod-init), copy the queries into it, and remove them from `my-aws-insights`. 

Then we'll adjust the charts and table in `my-aws-insights` to refer to the queries in the `aws-queries` mod.

<div>
<img src="/images/blog/2024-05-flexible-mod-dependencies/adjusted-query-refs.png" />
</div>

After completing the search-and-replace, query references like `query.s3_bucket_month_created` are fully-qualified with the name of the dependency (`aws_queries.query.s3_bucket_month_created`). Now we can install the mod from the local filesystem.

```
$ pwd
/home/jon/my-aws-insights
$ powerpipe mod install ~/aws-queries

Installed 1 mod:

local
└── /home/jon/aws-queries
```
The dashboard runs as before, but sources the queries from this new library mod.

## Install from a branch

We can now iterate rapidly on this local library mod but, at some point, we'll want to move it to GitHub so others can use it. So let's do that.

First we uninstall the local mod.

```
powerpipe mod uninstall `~/aws-queries`
```

Then we reinstall from GitHub. If the repo is tagged with a semver constraint, this command installs the latest version.

```
powerpipe mod install github.com/judell/aws-queries
```

And if it isn't tagged? No problem! You can now also install from a branch.

```
powerpipe mod install github.com/judell/aws-queries#main
```

Again the dashboard runs as before, sourcing queries from the mod that was installed from GitHub. Now others can depend on the library mod, perhaps even on alternate or experimental versions. And as we continue to evolve the library mod, we can switch back to a local install to rapidly iterate on it.

## Flexible update strategies

You've also now got a lot more flexibility when installing or updating mods, using the `--pull` argument to specify various update strategies.

<div>
<img src="/images/blog/2024-05-flexible-mod-dependencies/update-strategies.png" />
</div>

## See it in action

<div className="flex justify-center">
<iframe
    class="youtube-video"
    src="https://www.youtube-nocookie.com/embed/rxkp0Pj5t8c"
    frameBorder="0"
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Powerpipe: Use files, branches, or tags for mod dependencies"
>
</iframe>
</div>

## Reuse, remix, refactor

You could always [reuse and remix](https://steampipe.io/blog/remixing-dashboards) Powerpipe mods. Now it's easy to refactor them too, so you can extract shared components into dependent mods that you can rapidly develop and test, and easily share. If you haven't tried writing your own mods, now's a great time to get started. Go build something useful, then [tell us](https://turbot.com/community/join) about it!



Install a mod dependency locally, iterate on it rapidly, then share it to others who can install from a remote branch or tag.

Use files, branches, or tags for mod dependencies


## Introducing Powerpipe

Powerpipe is an open-source tool from [Turbot](https://turbot.com) that supports DevOps with:

**Benchmarks** - [5,000+ open-source controls](https://hub.powerpipe.io) from CIS, NIST, PCI, HIPAA, FedRamp and more. [Run instantly on your machine](/docs#run-security-and-compliance-benchmarks) or as part of your deployment pipeline.

**Relationship Diagrams** - The only dashboarding tool designed from the ground up to [visualize DevOps data](/docs#visualize-cloud-infrastructure). Explore your cloud, understand relationships, drill down to the details.

**Dashboards & Reports** - High-level dashboards provide a quick overview. Use them to highlight misconfigurations and hotspots. [Filter](/docs/run/benchmark/benchmark-dashboard#filtering--grouping), pivot, and [snapshot](/docs/run/snapshots) results.

**Code, not clicks** - Our dashboards are [code](/docs/powerpipe-hcl): version-controlled, composable, shareable, easy to edit — designed for the way you work. [Join](https://turbot.com/community/join) our open-source community!

## About Powerpipe dashboards and benchmarks

Powerpipe supports an ecosystem of [dashboards and benchmarks](https://hub.powerpipe.io/mods). These open-source components use HCL to define and compose elements that include [tables](/docs/powerpipe-hcl/table), [charts](/docs/powerpipe-hcl/chart), [relationship graphs](/docs/powerpipe-hcl/graph), [benchmarks](/docs/powerpipe-hcl/benchmark), and [controls](/docs/powerpipe-hcl/control). And they use SQL to query for the data that animates them, typically by way of a parallel ecosystem of [plugins](https://hub.steampipe.io/plugins) that translate cloud APIs to database tables.

For example, the [AWS Insights](https://hub.powerpipe.io/mods/turbot/aws_insights) mod provides 135+ overview and detail dashboards for AWS services: EBS, ECS, EC2, IAM, and more.

## Use the Powerpipe server to view dashboards

Likewise the [AWS Compliance](https://hub.powerpipe.io/mods/turbot/aws_compliance) mod provides 1100+ controls for Audit Manager Control Tower, AWS Foundational Security Best Practices, CIS, and many more.

Powerpipe [does not require Steampipe](/docs/run#selecting-a-database), but [mods](/docs/build) like `AWS Insights` rely on it. So to run the `AWS Insights` dashboards:

- Install [Powerpipe](https://powerpipe.io/downloads)
- Install [Steampipe](https://steampipe.io/downloads)
- Install the [AWS plugin for Steampipe](https://hub.steampipe.io/plugins/turbot/aws)

Then, in a fresh directory:

- `powerpipe mod init` (Powerpipe always runs in the context of a [mod](/docs/build).)

- `powerpipe mod install github.com/turbot/steampipe-mod-aws-insights` (There are commands for [mod management](/docs/build).)

- `steampipe service start` (Give Powerpipe access to Steampipe plugins.)

- `powerpipe server` (Many commands run client-only but dashboards use a server.)
- browse to http://localhost:9033

Dashboards are grouped by AWS service, here's the overview for EC2 instances.

<div>
<img src="/images/blog/2024-03-introducing-powerpipe/aws_ec2_dashboard.png" />
</div>

And here's the detail view for an instance, featuring a relationship graph that shows what it mounts, runs as, and connects to.

<div>
<img src="/images/blog/2024-03-introducing-powerpipe/aws_ec2_instance_detail.png" />
</div>

Although these dashboards use [Steampipe plugins](https://hub.steampipe.io/plugins), you can also connect Powerpipe to Postgres, SQLite, MySQL, or DuckDB. [This mod](https://hub.powerpipe.io/mods/turbot/rnacentral/dashboards), for example, visualizes biomedical data in a hosted instance of Postgres.

## Use the Powerpipe server to run benchmarks

To run the AWS Compliance benchmarks, follow the same recipe but use https://hub.powerpipe.io/mods/turbot/aws_compliance. These dashboards are grouped by benchmark, here's the CIS v300 benchmark.

<div>
<img src="/images/blog/2024-03-introducing-powerpipe/cis_v300_dashboard.png" />
</div>

You can adjust how results are filtered and grouped. Here we filter to just section 1, and group by `account_id`.

<div>
<img src="/images/blog/2024-03-introducing-powerpipe/cis_v300_dashboard_group_and_filter.png" />
</div>

## Use the Powerpipe CLI to run benchmarks

You can alternatively run that same benchmark, from the same folder. List the benchmarks.

```
powerpipe benchmark list
```

Run a benchmark.

```
powerpipe benchmark run cis_v300
```

The [documentation](/docs/reference/cli/benchmark#powerpipe-benchmark-run) covers a range of options including `--export` to save results in `asff`, `csv`, and other file formats, and `--snapshot` to create a snapshot [in Turbot Pipes](https://steampipe.io/blog/release-0-17-0#share-snapshots-with-steampipe-cloud).

## Compose and create dashboards and benchmarks

The thousands of pre-built dashboards and benchmarks on the hub are all available for [reuse and remixing](https://steampipe.io/blog/remixing-dashboards). You can use [mod management commands](/docs/reference/cli/mod) to [create your own mod](/docs/build/create-mod), install a local copy of one or more published mods, rearrange the resources they provide, and add your dashboards or benchmarks. Or you can create your own [dashboards](/docs/build/writing-dashboards) and [benchmarks](/docs/build/writing-dashboards) from scratch. In either case, you use HCL to define these elements, and SQL to fill them with data.

Here's a simple dashboard to chart Turbot blog posts by month. It uses the [RSS plugin](https://hub.steampipe.io/plugins/turbot/rss) to aggregate posts and the [chart](/docs/powerpipe-hcl/chart) resource to visualize the data.

```hcl
dashboard "blog_posts" {

  title = "Blog posts"

  chart {
    type = "column"
    title = "posts by month"
    sql = <<EOQ
      select
        to_char(published, 'YYYY-MM') as published,
        count(*)
      from
        rss_item
      where
        feed_link = 'https://turbot.com/blog/feed.xml'
     group by published
     order by published
    EOQ
    }

}
```

To replicate this example: install Powerpipe and Steampipe as above, install the RSS plugin (`steampipe plugin install rss`), start the Steampipe service, save this HCL code in a file (e.g. `dashboard.pp`), run `powerpipe server`, and visit http://localhost:9093. Note that Powerpipe watches its source files and reacts to edits of HCL and/or SQL. For example, in an auto-saving editor, try changing the chart type from `column` to `bar`. Then change the aggregation from `YYYY-MM` to `YYYY`. You'll see the chart reflect both changes in realtime.

And here's a benchmark that defines a single control to ensure that each post has at least one category.

```hcl
benchmark  "rss" {
  title    = "RSS compliance"
  children = [
    control.has_categories
  ]
}

control "has_categories" {
  title = "Ensure each item has one or more categories."
  sql = <<EOT
    select
      title as resource,
      case
        when jsonb_array_length(categories) > 0 then 'ok'
        else 'alarm'
      end as status,
      case
        when jsonb_array_length(categories) > 0 then 'Has ' || jsonb_array_length(categories) || ' categories'
        else 'Must have at least one category'
      end as reason,
      title,
      categories::text
    from
      rss_item
    where
      feed_link = 'https://turbot.com/blog/feed.xml'
    EOT
}
```

A [benchmark](/docs/powerpipe-hcl/benchmark) lists one or more child [controls](/docs/powerpipe-hcl/control). Each control runs a query that reports the [required columns](/docs/powerpipe-hcl/control#required-control-columns) `status`, `reason`, and `resource`, along with optional dimensions — in this case, `title` and `categories`.

## See it in action

<div className="flex justify-center">
<iframe
    class="youtube-video"
    src="https://www.youtube-nocookie.com/embed/-h6RSpvR0FE"
    frameBorder="0"
    allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"
    allowFullScreen
    title="Powerpipe: Dashboards for DevOps"
>
</iframe>
</div>

## Data visualizations and compliance controls as code

Powerpipe is a flexible open-source platform to visualize and monitor your cloud infrastructure, ensure security compliance, and even build your own custom dashboards and benchmarks. With pre-built modules connecting to AWS, Azure, GCP, and more, along with easy ways to create and share your own mods, Powerpipe is an essential GitOps-friendly tool for DevOps teams wrangling cloud complexity. To get started, [download](/downloads) the tool, follow the [tutorial](/docs), explore the [mods](https://hub.powerpipe.io), then [let us know](https://turbot.com/community/join) how it goes!


An open-source dashboard server and CLI to visualize your data, manage your security posture, and deliver actionable DevOps intelligence.

Powerpipe: Dashboards for DevOps


Steampipe users have enjoyed the built-in dashboards-as-code features since [Steampipe CLI v0.13](https://steampipe.io/blog/release-0-13-0). With Powerpipe we wanted to make those same features available for other databases. That's one reason we've [decoupled](https://steampipe.io/blog/steampipe-unbundled) dashboards from the Steampipe CLI and dashboard server: so you can use Powerpipe with your own database.

Powerpipe includes the [43 Steampipe mods](https://hub.powerpipe.io/?engines=steampipe) to visualize AWS, Azure, GCP, GitHub, Terraform and more. And nine ready-to-use Powerpipe mods show you how to visualize data in [Postgres](https://hub.powerpipe.io/?engines=postgres), [SQLite](https://hub.powerpipe.io/?engines=sqlite), [DuckDB](https://hub.powerpipe.io/?engines=duckdb), and [MySQL](https://hub.powerpipe.io/?engines=mysql)!

If you're using Steampipe today to run dashboards and/or benchmarks, that all still works, but over time we'll deprecate these uses of Steampipe in favor of Powerpipe.

Powerpipe extends your Steampipe superpowers: it's database-agnostic, with dashboards that filter and group. Of course it also does what you've always been doing.

## Run Steampipe as usual

Let's assume you've [installed Steampipe](https://steampipe.io/downloads) along with [AWS plugin](https://hub.steampipe.io/plugins/turbot/aws) and are running Steampipe as a service (`steampipe service start`).

## Add Powerpipe

Now you [install Powerpipe](https://powerpipe.io/downloads).

Here are some common operations in Steampipe along with their Powerpipe counterparts.

## Run a dashboard

| action | Steampipe | Powerpipe |
| --- | --- | --- |
| Install AWS Insights | `steampipe mod install github.com/turbot/steampipe-mod-aws-insights` | `powerpipe mod install github.com/turbot/steampipe-mod-aws-insights` |
| Start dashboard server | `steampipe dashboard` | `powerpipe server` |
| Browse to dashboards | http://localhost:9194 | http://localhost:9033 |

See [Powerpipe dashboards](/docs/run/dashboard) for details.

## Run a benchmark

| action | Steampipe | Powerpipe |
| --- | --- | --- |
| Install AWS Compliance | `steampipe mod install github.com/turbot/steampipe-mod-aws-compliance` | `powerpipe mod install github.com/turbot/steampipe-mod-aws-compliance` |
| Start dashboard server | `steampipe dashboard` | `powerpipe server` |
| Browse to dashboards | http://localhost:9194 | http://localhost:9033 |
| Run a benchmark in your terminal | `steampipe check benchmark.cis_v300` | `powerpipe benchmark run cis_v300` |

See [Running benchmarks](/docs/run/benchmark) for details.

## Run a query

| action | Steampipe | Powerpipe |
| --- | --- | --- |
| List queries | `steampipe query list` | `powerpipe query list` |
| Run named query | `steampipe query query_name` | `powerpipe query query_name` |
| Run literal query | `steampipe query "sql statement"` | `powerpipe query run "sql statement"` |

See [Powerpipe query](/docs/reference/cli/query) for details.

## File extensions

| extension          | Steampipe | Powerpipe |
| ------------------ | --------- | --------- |
| HCL code for a mod | `.sp`     | `.pp`     |
| Config info        | `.spc`    | `.ppc`    |
| Snapshots          | `.sps`    | `.pps`    |
| Variables          | `.spvars` | `.ppvars` |

See [dashboards](/docs/reference/cli/dashboard), [benchmarks](/docs/reference/cli/benchmark), and [config files](/docs/reference/config-files) for details.

## Environment variables

| Steampipe                | Powerpipe                |
| ------------------------ | ------------------------ |
| `STEAMPIPE_LOG_LEVEL`    | `POWERPIPE_LOG_LEVEL`    |
| `STEAMPIPE_MAX_PARALLEL` | `POWERPIPE_MAX_PARALLEL` |
| `PIPES_TOKEN`            | `PIPES_TOKEN`            |

Many variables carry over from Steampipe, but Powerpipe adds new ones like `POWERPIPE_DATABASE`. See [Powerpipe Environment Variables](/docs/reference/env-vars) for details.

## Passing input variables

| Steampipe | Powerpipe |
| --- | --- |
| `steampipe query --var=instance_state="running"` | `powerpipe query --var=instance_state="running"` |
| `steampipe check all --var-file='tags.spvars'` | `powerpipe benchmark run all --var-file='tags.ppvars'` |

See [Passing Input Variables](/docs/build/mod-variables#passing-input-variables) for more examples.

## Change databases

Powerpipe defaults to a local Steampipe instance (`postgres://steampipe@127.0.0.1:9193/steampipe`) but you can change to another database in several ways.

### Using `--database`

```hcl
powerpipe benchmark run cis_v120 --database postgres://myusername:passworrd@mydbserver.mydomain.com:9193/steampipe
```

or

```hcl
powerpipe server --database  postgres://myusername:passworrd@mydbserver.mydomain.com:9193/steampipe
```

See [Selecting a database](/docs/run/#selecting-a-database) for more examples.

### Using a workspace

```hcl
workspace "my_workspace" {
  database = "postgres://my_username:passsord@db1.jpeterman.com:5432/steampipe"
}
```

or

```hcl
workspace "my_workspace" {
  database = "sqlite:./my_sqlite.db"
}
```

See [Managing workspaces](/docs/run/workspaces) for more examples.

## Use the Powerpipe Hub

The Steampipe Hub will redirect you to the [Powerpipe Hub](https://hub.powerpipe.io) where, under a fresh coat of paint, you'll find all your favorite mods.

## Read the Powerpipe docs

The Steampipe docs are also spruced up at [powerpipe.io/docs](https://powerpipe.io/docs).

## Enjoy the upgrade!

Rest assured that all v0.x versions of Steampipe mods will work in both Steampipe and Powerpipe. But note that from v1.0.0 onward, mods will be in Powerpipe format only.

Once you're settled in, we encourage you to try a few things. If you run benchmarks, turn on filters and grouping to slice and dice your benchmark results. If you build dashboards, check out some of the [new mods](https://hub.powerpipe.io/mods/turbot/rnacentral) that work with non-Steampipe databases and schemas. And as always, [let us know](https://turbot.com/community/join) your thoughts.


Play your favorite Steampipe hits, and enjoy some new ones.

Migrating from Steampipe to Powerpipe

Articles about Powerpipe published elsewhere

Categories

Powerpipe

Turbot

Use files, branches, or tags for mod dependencies

Powerpipe: Dashboards for DevOps

Migrating from Steampipe to Powerpipe

Powerpipe news and reviews