Skip to content
This repository has been archived by the owner on May 22, 2024. It is now read-only.

Home

Jump to bottom
Jimmy Yih edited this page Jun 4, 2019 · 30 revisions

PostgreSQL Merge

Greenplum's core is Postgres so we need to continuously sync with upstream Postgres. Currently, we are still in CATCHUP mode so our merge process is not solidified. Once in STREAMING, we should see this process be much more easy and possibly automated.

What is the definition of done?

Spirit is: are we realizing the value of the code that we have merged

  • Master CI is green
    • Tests run both with and without ORCA optimizer
    • Existing tests are green
    • All test brought in from upstream Postgres are green
    • Listed here to be explicit but is included in the CI being green
      • Upgradable gpupgrade from 5.x to Master -- pg_upgrade as a proxy until gpupgrade exists
      • No platform regressions -- RHEL, SUSE, Ubuntu(?)
      • Output a build at the end of each merge back into master
  • If it works for HEAP, it needs to work for AO/AOCO (can also disable feature and/or log a story as future work)
  • CVEs are identified and addressed -- to be addressed at the very end of the cycle
  • Benchmarking performance monthly
  • Fast/slow retro -- +/∆

Resources:

How to create a merge iteration branch and submit to master finally:
How to create merge iteration branch

Some advice on dealing with merge conflicts:
Dealing with merge conflicts

Check against Greenplum historical repository:
Pivotal employees have access to a private repository which has Greenplum's history from GPDB 4.3 and older. Checking the historical records on why something was changed or introduced in Greenplum can be important for certain merge conflicts. For non-Pivotal contributors, please ask in the Greenplum Developers Google Group for help in seeing into Greenplum's past.

How to run tests:
Running tests against merge branch

How to Concourse CI:
The Concourse CI is hosted publicly at https://pg-merge.ci.gpdb.pivotal.io. To add/update a new merge pipeline, currently, a Pivotal employee must update the secrets yaml file (prefixed with gpdb_merge) in a private Pivotal continuous integration repository and follow the comments in that yaml file. Check the Github file history on the secrets yaml file for an example of what to update. The file may also need to be updated (e.g. add some variables) to adapt to new pipeline changes from GPDB master.

Spreadsheet that lists all the commits between 8.3 and 9.5:
Commit Spreadsheet

Earlier plans when we used to break down into smaller iterations:
8.4 breakdown
9.0 breakdown
9.1 breakdown

Tips & Tricks

  • Here's a little perl script that auto-resolves the most trivial merge conflicts:

  • The upstream sources will be consulted regularly: for example, clone a PostgreSQL tree into a directory named 8.4 and check out REL8_4_STABLE, and another one in a 8.3 directory with REL8_3_STABLE. That will make it easy to compare versions of files.

  • After compilation is green, you'll want to try initializing the master segment before going big with an actual Greenplum cluster. This should be as easy as running the following:
    initdb -E UNICODE -D /tmp/demoDataDir-1 --locale=en_US.utf-8 --max_connections=150 --shared_buffers=128000kB --data-checksums

  • When ready for the first set of tests (e.g. src/test/regress/parallel_schedule), initialize the cluster without mirrors so that you can stay focused on test issues instead of replication issues:
    WITH_MIRRORS=false make create-demo-cluster

For any questions, please ask at [email protected].

Clone this wiki locally