Xg2xg

A handy lookup table of similar technology and services to help ex-googlers survive the real world :) pull-requests are very welcomed. Please do not list any confidential projects!

For a working example of (some) of these technologies integrated together, see: https://github.com/google/startup-os

See also: System Design Primer, The Hadoop Ecosystem Table, Map AWS services to GCP products, Database of Databases, Google Open Source Glossary

Technology

Core Technology

| Google Internal | Google External | Open Source / Real-World | | --------------- | ---------------------------------------- | ---------------------------------------- | | MapReduce | | Apache Hadoop, Spark | | Flume | DataFlow | Apache Beam | | Protocol Buffer | Protobuf, FlatBuffers | Cap'n Proto, Thrift, Avro, Amazon Ion, CBOR, kryo | | Stubby | gRPC | Thrift, Bolt | | Chubby | | Apache Zookeeper, etcd, HashiCorp Consul | | Goops / PubSub | Cloud Pub/Sub | Apache Kafka, Apache Pulsar, Facebook LogDevice | | //base | | abseil |

Infrastructure

| Google Internal | Google External | Open Source / Real-World | | -------------------- | ---------------------------------------- | ---------------------------------------- | | Borg | Kubernetes | Apache Mesos, Apache Aurora, HashiCorp Nomad | | GSLB | Cloud Load Balancing - Internal | AWS ELB, Istio, linkerd | OnePlatform | API Gateway | Swagger | | GFE, Maglev, uberproxy | Cloud Load Balancing - HTTPS / External | envoy, AWS ALB, HAProxy, nginx, F5 | | uberproxy (sso proxy) | Identity-Aware Proxy | buzzfeed-sso, Pomerium, Envoy Gateway | | Zanzibar | Zanzibar Research Paper | SpiceDB/authzed, Ory Keto, topaz, Opal, (iam)Keycloak, Warrant, OpenFGA | | data center hardware | open compute | | | Jupiter, Starblaze | | | | B4, Stargate, TE | | | | USPS, Andromeda | | | | ESDN | | Faucet | broccoli man | ||

Storage

| Google Internal | Google External | Open Source / Real-World | | ---------------------------------------- | ---------------------------------------- | ---------------------------------------- | | GFS/Colossus | | HDFS, Ceph, GlusterFS, MooseFS | | Bigtable | Cloud Bigtable | PrestoDB, Cassandra, HBase, Accumulo, DynamoDB, ScyllaDB | | Spanner | Cloud Spanner | Vitess, CockroachDB, TiDB | | ColumnIO / Capacitor | BigQuery storage | Apache Parquet, ORC, Apache Iceberg | | sstable | levelDB | RocksDB, pebble | | zippy | Snappy | lz4 | | RecordIO | Riegeli, TFRecords, & in OR-Tools, szl | stuffed-record-stream |

Services

| Google Internal | Google External | Open Source | SaaS | | ---------------------------------------- | ---------------------------------------- | ---------------------------------------- | ---------------------------------------- | | Dremel | BigQuery | Apache Drill, Presto, Spark(sort-of), | AWS Athena, Redshift Spectrum, Snowflake | | Dremel UI | | Redash, Metabase, Apache Superset | | | Search (Mustang, Alexandria) | | Elasticsearch, OpenSearch, Solr, Lucene | algolia | | pubsub | pubsub | NATS.io, RabbitMQ, PubNub | AWS SQS/SNS, AWS AppSync | | MillWheel | Cloud Dataflow | Apache Flink, beam | | | Colab | Colaboratory | Jupyter | Observable | | PLX | Google Data Studio | | Mode | | Monarch | paper | | | | Napa | paper | | | | MakerSuite| MakerSuite | | |

DevOps

| Google Internal | Google External | Real-World | | ------------------------------- | ---------------------------------------- | ---------------------------------------- | | Assimilator | | Dominator | | Blaze | Bazel | Buck, Pants, please.build, Blade, recc, EngFlow, BuildBuddy, flare.build | | Oncall | | PagerDuty, [OpsGenie](https://www.opsgenie.co