Card Folio

Demo here: https://youtu.be/aFVkEZIRvKw

Prerequisites

• geth full-node
• docker
• node.js (version 8 or greater)

The Portfolio project requires a full-node geth as it will build a crypto portfolio from arbitrary Ethereum addresses by retrieving transaction receipts from transactions that can live at any block height. When running geth make sure to enable the web socket protocol, as the Cardstack Hub leverages web3.js event subscription.

The simplest way to use geth is via Docker. Instructions are available here: https://hub.docker.com/r/ethereum/client-go/. Make sure to use a mounted volume with the -v option so that you don't lose your chain data when you stop your geth's docker container. You can also use homebrew to install geth if you are using a Mac. Instructions are available here: https://github.com/ethereum/go-ethereum/wiki/Installation-Instructions-for-Mac.

The following is an example of the command line options to use for running geth locally for the Rinkeby test network in a manner that is compatible with the Portfolio project:

$ geth --rinkeby --syncmode "full" \
     --rpc --rpcapi eth,net,web3 --rpcaddr 0.0.0.0 \
     --ws --wsaddr 0.0.0.0 --wsorigins '*' --wsapi eth,net,web3 \
     --cache 4096

Note that it will take about a day for the geth node to sync on the Rinkeby network since it is a full-node. (Mainnet will take a few days to sync). Infura is also a perfectly valid option, although you may find that the performance is not quite as fast as a locally hosted geth.

Additionally, if you do not wish to run geth, you can also use mocked Ethereum data instead. Instructions around how to manipulate the mocked Ethereum data are described in the sections below.

Installation

• git clone <repository-url> this repository
• cd portfolio
• yarn install (we use Yarn workspaces, so use Yarn and not NPM)

Preparing the Ethereum Index

The Cardstack Hub maintains a secondary index of the Ethereum blockchain that is a representation of all the transactions in the blockchain using postgres. When the Cardstack hub starts it updates its Ethereum index by populating the index with any new Ethereum transactions since the last time it was running. In the case of the very first time the Cardstack hub starts, this process will take quite awhile. There are a few options for how you can speed this up:

Using AWS to Construct an Ethereum Index

This is probably the fastest way to assemble the Ethereum Index. We have authored Terraform modules that can be used spin up a pool of EC2 instances that can assemble the Ethereum index by distributing the work amongst the instances in the pool. The approach requires that you have an RDS postgres DB available in AWS. Ideally this approach would be used if you are deploying the Portfolio app into AWS, but you could also just extract the assembled index from your RDS DB and import into your local DB.

TODO add Terraform examples

Assembling an Ethereum Index Locally

We have authored a nodejs script that is available in the @cardstack/ethereum module that will assemble an Ethereum Index into your local DB. This script will split the work to assemble the Ethereum index into a configurable number of forked processes. Additionally, we have extracted an already built Ethereum Index into a CSV file, so that you don't have to build the Ethereum Index from the genesis block. Using your favorite postgres client, you can import this CSV into your Ethereum Index, and then use the script for adding the transactions from blocks that were mined after the CSV was generated.

From your portfolio project folder you can execute the following script, where the --jsonRpcUrl is the URL of your geth node's web socket interface:

$ HUB_ENVIRONMENT=development node ./node_modules/@cardstack/ethereum/scripts/build-index.js --jsonRpcUrl=ws://localhost:8546

(Note that this script just uses the PGHOST, PGPORT, PGUSER, and PGPASSWORD environment variables to connect to your postgres database.)

Additionally, if you only want to index specific blocks you can add:

--start=<block number> --end=<block number>

When --start is not specified, the script assumes you want to start at block #0. If --end is not specified the script assumes you want to end at the current block.

You may also configure the amount of worker child processes to fork. The default amount of workers is 10.

--workerCount=20

For a list of options use the --help option.

Ethereum Index CSV Files

In order to speed up the the process of building your Ethereum Index, we have extracted an already built Ethereum Index into a CSV file. You can use your favorite Postgres client to upload the CSV into your database.

First make sure your DB is running. From the instructions in the "Running" section below, run:

yarn start-prereqs

Then you can create the database by executing the script:

$ HUB_ENVIRONMENT=development node ./node_modules/@cardstack/ethereum/scripts/build-index.js --createDb

(Note that this script just uses the PGHOST, PGPORT, PGUSER, and PGPASSWORD environment variables to connect to your postgres database.)

Then using your Postgres DB client (I like using Postico), you can upload the CSV into the newly created ethereum_index database. The CSV files are very large. You may need to use the split UNIX command to split up the CSV into smaller chunks based on the what your Postgres client can handle.

We have made the following CSV files available:

• Rinkeby up to block #3915387 https://cardstack.com/csv/rinkeby_transactions_3915387.csv (8.5GB) SHA-256 Checksum 47a561b12ecc58f8708248b86db6c392c02faf0652fa20d8e5dde4ed74020e62
• TODO add Mainnet CSV files and their SHA256 checksums

(feel free to open an issue if you need a CSV file that is not listed above)

Running

There are docker container(s) that are prerequisites for running the portfolio app. You can start and stop them with:

yarn start-prereqs
yarn stop-prereqs

yarn start-prereqs will bind the PostgreSQL running in the Docker container to port 5432 which is the standard Postgres port. That might conflict with a local PostgreSQL you might have running, which would actually be used by the Cardstack hub in that case as opposed to the one running in the Docker container.

Once the prerequisites are running, you can run both the Hub and the Ember CLI development server like this:

yarn start

Alternatively, you can run the Hub and Ember CLI separately with:

yarn start-hub
yarn start-ember

Environment Variables:

HUB_ENVIRONMENT

The HUB_ENVIRONMENT environment variable is used to tell the Cardstack hub which environment it is running within. The possible values are development, test, and production. Generally the development environment is used for doing development on your local machine. The test environment is used for running the automated tests. The production environment is used for running a hosted application. If this environment variable is not specified, then the HUB_ENVIRONMENT=development is assumed.

JSON_RPC_URLS

Use the environment variable JSON_RPC_URLS to specify a comma separated list of geth web socket URL's to use for the Portfolio application. If this environment variable is not used, then the mock Ethereum data will be used instead.

CRYPTO_COMPARE_API_KEY

Use the environment variable CRYPTO_COMPARE_API_KEY to specify your crypto compare API key in order to get live and historical crypto conversion dates. If this environment variable is not used, then a randomish number is substituted instead that hovers around $100 USD for 1 unit of crypto currency. (When HUB_ENVIRONMENT=test then exactly $100 USD is used, as random numbers dont make for good tests.)

GIT_REPO

Use the environment variable GIT_REPO to specify the git data source SSH URL. Note that this is only used when HUB_ENVIRONMENT=production is also specified. If this variable is not used, then the ephemeral data source will be used instead.

GIT_PRIVATE_KEY

Use the environment variable GIT_PRIVATE_KEY to specify the private key for the SSH user used in the GIT_REPO environment variable. Note that this value will have newlines within it. Please see https://github.com/cardstack/cardstack/blob/master/packages/git/README.md for more details around specying this environment variable.

GIT_BRANCH_PREFIX

Use the environment variable GIT_BRANCH_PREFIX when you wish to use a branch other than master for your git data source. The value of GIT_BRANCH_PREFIX will be prepended to the string master when deriving the name of the branch to use for reading and writing documents to the git data source.

LOG_LEVELS

Use the environment variable LOG_LEVELS to adjust the logging on the Cardstack hub process. More information around how to use this can be found here: https://github.com/cardstack/logger

Postgres

The standard Postgres environment variables PGHOST, PGPORT, PGUSER, PGPASSWORD, etc are supported. You can control the postgres instance that the Cardstack hub uses by setting these environment variables.

Configuring Users, Portfolios, Wallets, and Assets

The Cardstack Hub can connected to many different types of data sources. Some data sources are read-only (like the Ethereum data source, Crypto Compare data source, and the asset history data sources) and others are writable for holding content types that are controlled by the hub, including users, portfolios, wallets, and asset content types. The way we have arranged our writable data sources for the Portfolio project is to use an "ephemeral" data sour