Skip to content

Commit

Permalink
sgm-gharchive: high level summery doc(let).
Browse files Browse the repository at this point in the history 
Signed-off-by: Matt Young <[email protected]>
  • Loading branch information
@halcyondude
halcyondude committed Dec 8, 2024
1 parent e0a8ae3 commit b38c45e
Showing 1 changed file with 307 additions and 0 deletions.
307 changes: 307 additions & 0 deletions db/sgm-gharchive/docs/entities-gharchive.ipynb
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,307 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "e94d7db9-1c73-4e9d-b3cb-45adde512204",
"metadata": {},
"source": [
"# Github Archive (gharchive.org)"
]
},
{
"cell_type": "markdown",
"id": "fe206ad5-280e-499c-a929-270bc39b163c",
"metadata": {
"jp-MarkdownHeadingCollapsed": true,
"tags": []
},
"source": [
"## What is the Github Archive?"
]
},
{
"cell_type": "markdown",
"id": "c82f0d0c-b41b-4127-8fff-c33db28983cb",
"metadata": {
"jp-MarkdownHeadingCollapsed": true,
"tags": []
},
"source": [
"https://www.gharchive.org\n",
"\n",
"> Open-source developers all over the world are working on millions of projects: writing code & documentation, fixing & submitting bugs, and so forth. GH Archive is a project to record the public GitHub timeline, archive it, and make it easily accessible for further analysis.\n",
">\n",
">GitHub provides 15+ event types, which range from new commits and fork events, to opening new tickets, commenting, and adding members to a project. These events are aggregated into hourly archives, which you can access with any HTTP client\n",
"\n",
"---\n",
"\n",
"* Activity archives are available starting 2/12/2011.\n",
"* Activity archives for dates between 2/12/2011-12/31/2014 was recorded from the (now deprecated) Timeline API.\n",
"* **Activity archives for dates starting 1/1/2015 is recorded from the Events API.**\n",
"\n",
"---\n",
"\n",
"* Each archive is a compressed (.gz) JSON Lines formatted file.\n",
"* Each line of the file is a JSON object representing an event that happened, and is one of 20 types."
]
},
{
"cell_type": "markdown",
"id": "8752c03c-2095-4df1-ba67-22d7f4a5e190",
"metadata": {
"tags": []
},
"source": [
"## Design: Github Events"
]
},
{
"cell_type": "markdown",
"id": "cb6b8400-a059-4a60-83ad-bd38977048bd",
"metadata": {
"tags": []
},
"source": [
"### How Github Event types are modeled\n",
"\n",
"> The Events API can return different types of events triggered by activity on GitHub. Each event response contains shared properties, but has a unique payload object determined by its event type. The Event object common properties describes the properties shared by all events, and each event type describes the payload properties that are unique to the specific event.\n",
"\n",
"_More info: https://docs.github.com/en/webhooks-and-events/events/github-event-types_"
]
},
{
"cell_type": "markdown",
"id": "aaecd7fe-ec61-4285-af84-e28f1834555b",
"metadata": {},
"source": [
"We use python3's [support for enumerations](https://docs.python.org/3/library/enum.html) to capture the **kinds of things** that exist.\n",
"\n",
"* code completion in editors reduces iterations due to typos in config, and RSI\n",
"* the functions provided by the `enum` class are powerful. The HOWTO is detailed: https://docs.python.org/3/howto/enum.html\n",
"* we leverage it for configuration\n",
"\n",
"\n",
"```python\n",
"class EventType(Enum):\n",
" '''https://docs.github.com/en/webhooks-and-events/events/github-event-types '''\n",
"\n",
" def __str__(self):\n",
" return str(self.value)\n",
"\n",
" CommitCommentEvent = 'CommitCommentEvent'\n",
" CreateEvent = 'CreateEvent'\n",
" DeleteEvent = 'DeleteEvent' \n",
" ForkEvent = 'ForkEvent'\n",
" GollumEvent = 'GollumEvent'\n",
" IssueCommentEvent = 'IssueCommentEvent'\n",
" IssuesEvent = 'IssuesEvent'\n",
" MemberEvent = 'MemberEvent'\n",
" PublicEvent = 'PublicEvent'\n",
" PullRequestEvent = 'PullRequestEvent'\n",
" PullRequestReviewEvent = 'PullRequestReviewEvent'\n",
" PullRequestReviewCommentEvent = 'PullRequestReviewCommentEvent'\n",
" PullRequestReviewThreadEvent = 'PullRequestReviewThreadEvent'\n",
" PushEvent = 'PushEvent'\n",
" ReleaseEvent = 'ReleaseEvent'\n",
" SponsorshipEvent = 'SponsorshipEvent'\n",
" WatchEvent = 'WatchEvent'\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "a4743376-6525-42cc-9842-8ba5fb3527bb",
"metadata": {
"tags": []
},
"source": [
"### Entities"
]
},
{
"cell_type": "markdown",
"id": "123b176f-274b-457e-b8b7-d077f8b9b1cd",
"metadata": {
"jp-MarkdownHeadingCollapsed": true,
"tags": []
},
"source": [
"#### Common properties on each event"
]
},
{
"cell_type": "markdown",
"id": "f4b38465-4794-4eb2-b45d-c798cdca0c2b",
"metadata": {
"jp-MarkdownHeadingCollapsed": true,
"tags": []
},
"source": [
"\n",
"```json\n",
"[\n",
" {\n",
" \"type\": \"WatchEvent\",\n",
" \"public\": false,\n",
" \"payload\": {\n",
" },\n",
" \"repository\": {\n",
" \"id\": 3,\n",
" \"name\": \"octocat/Hello-World\",\n",
" \"url\": \"https://api.github.com/repos/octocat/Hello-World\"\n",
" },\n",
" \"actor\": {\n",
" \"id\": 1,\n",
" \"login\": \"octocat\",\n",
" \"gravatar_id\": \"\",\n",
" \"avatar_url\": \"https://github.com/images/error/octocat_happy.gif\",\n",
" \"url\": \"https://api.github.com/users/octocat\"\n",
" },\n",
" \"org\": {\n",
" \"id\": 1,\n",
" \"login\": \"github\",\n",
" \"gravatar_id\": \"\",\n",
" \"url\": \"https://api.github.com/orgs/github\",\n",
" \"avatar_url\": \"https://github.com/images/error/octocat_happy.gif\"\n",
" },\n",
" \"created_at\": \"2011-09-06T17:26:27Z\",\n",
" \"id\": \"12345\"\n",
" }\n",
"]\n",
"```\n",
"\n"
]
},
{
"cell_type": "markdown",
"id": "5fdfc851-c465-4794-a612-a8bdefe8ae3a",
"metadata": {},
"source": [
"### Event Example: WatchEvent (GitHub \"Star\")1"
]
},
{
"cell_type": "markdown",
"id": "c3f09632-29ac-464e-a857-c402e6715e3f",
"metadata": {
"tags": []
},
"source": [
"Recall that the header looks like this, note \"id\" is used as the a primary key as convention. The helper library fixes up `id` columns to be named `{entity}_github_id`. \n",
"\n",
"```json\n",
"[\n",
" {\n",
" \"type\": \"WatchEvent\",\n",
" \"public\": false,\n",
" \"payload\": {\n",
" },\n",
" \"repository\": {\n",
" \"id\": 3,\n",
" \"name\": \"octocat/Hello-World\",\n",
" \"url\": \"https://api.github.com/repos/octocat/Hello-World\"\n",
" },\n",
" \"actor\": {\n",
" \"id\": 1,\n",
" \"login\": \"octocat\",\n",
" \"gravatar_id\": \"\",\n",
" \"avatar_url\": \"https://github.com/images/error/octocat_happy.gif\",\n",
" \"url\": \"https://api.github.com/users/octocat\"\n",
" },\n",
" \"org\": {\n",
" \"id\": 1,\n",
" \"login\": \"github\",\n",
" \"gravatar_id\": \"\",\n",
" \"url\": \"https://api.github.com/orgs/github\",\n",
" \"avatar_url\": \"https://github.com/images/error/octocat_happy.gif\"\n",
" },\n",
" \"created_at\": \"2011-09-06T17:26:27Z\",\n",
" \"id\": \"12345\"\n",
" }\n",
"]\n",
"```\n",
"\n",
"The `name` and `url` fields are specified as `category_cols` - indicating that we should handle them as a [Pandas Category](https://pandas.pydata.org/docs/user_guide/categorical.html).\n",
"\n",
">The categorical data type is useful in the following cases:\n",
">\n",
"> * A string variable consisting of only a few different values. Converting such a string variable to a categorical variable will save some memory, see here.\n",
">\n",
"> * The lexical order of a variable is not the same as the logical order (“one”, “two”, “three”). By converting to a categorical and specifying an order on the categories, sorting and min/max will use the logical order instead of the lexical order.\n",
">\n",
"> * As a signal to other Python libraries that this column should be treated as a categorical variable (e.g. to use suitable statistical methods or plot types).\n",
"\n",
"This is similar to the \"dictionary compression\" used in the Arrow projects. The value is stored once per unique value, rather than being duplicated per event. This produces a dramatic reduction in memory usage, making it viable to be used by an engineer with a MacBook Pro.\n",
"\n",
"More info: https://pandas.pydata.org/docs/user_guide/categorical.html#categorical-memory\n",
"\n",
"Configuration for the `repo` entity:\n",
"\n",
"```json\n",
"str(EntityType.Repo).lower(): {\n",
" 'description': 'ENTITY: repo',\n",
" 'dtypes_write': {\n",
" 'repo_github_id': sa_dialect_postgresql.BIGINT,\n",
" 'name': sa_dialect_postgresql.TEXT,\n",
" 'org_name': sa_dialect_postgresql.TEXT,\n",
" 'repo_name': sa_dialect_postgresql.TEXT,\n",
" 'url': sa_dialect_postgresql.TEXT\n",
" },\n",
" 'category_cols': ['name', 'url'],\n",
" 'embedded_json_cols': [],\n",
" 'natural_keys': ['id'],\n",
" 'db_table_name': 'repo'\n",
" },\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "befee230-c26d-46b9-8807-9e1bc0086bb0",
"metadata": {},
"source": [
"### Resources (arrow/feather/pandas)"
]
},
{
"cell_type": "markdown",
"id": "4e093032-a3f9-45b7-aa7e-58f59ad418d6",
"metadata": {
"tags": []
},
"source": [
"* https://arrow.apache.org/docs/python/pandas.html\n",
"* https://arrow.apache.org/docs/python/feather.html#feather-file-format\n",
"* https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_feather.html#pandas-dataframe-to-feather\n",
"* https://arrow.apache.org/docs/python/generated/pyarrow.feather.write_feather.html#pyarrow.feather.write_feather\n",
"\n",
"* https://arrow.apache.org/docs/python/data.html#dictionary-arrays\n",
"\n",
"* https://observablehq.com/@uwdata/introducing-arquero\n",
"* https://observablehq.com/@uwdata/arquero-and-apache-arrow\n",
"* https://observablehq.com/@uwdata/an-illustrated-guide-to-arquero-verbs\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.5"
}
},
"nbformat": 4,
"nbformat_minor": 5
}

0 comments on commit b38c45e

Please sign in to comment.