Personal API Keys Discussion

[yakkomajuri]

I was about to dive head-first into this but authentication is such a foundational aspect of everything that I think this needs some discussion first.

Problem

#4713 first raised the issue of feature flags in our server-side libraries being subject to project changes in the UI which was a quite a significant issue. That was fixed in #4764.

However, the same problem also affects:

• The GitHub, GitLab, and Bitbucket plugins
• The PagerDuty plugin
• Anyone with more than 1 project (team) relying on Personal API Keys

In addition, while the integrity issue is fixed with feature flags, it is still far from optimal that we ask people to use a Personal API Key with org-level access in PostHog libs. I wouldn't be comfortable with a key attached to my name in a team project.

So there are a few problems here:

1. Specifying a project when using Personal API Keys
2. Use cases where someone's key is needed for shared purposes (feature flags, plugins)

Potential Solutions

0. In Any Case...

Regardless of what solution we go with below, we probably need to update all relevant view sets to respect the team_id set when specifying a project on an API call. This was done for api/feature_flag on #4764.

1. Forcing Project Specification with Personal API Keys

This ensures people are explicit about what project they want to interact with programmatically. It's probably the simplest solution. Solves problem 1, but not really problem 2. I still would need a personal key on a plugin and feature flags, and that's not great.

2. Scoped Personal API Keys

Almost the same as above, with a little bit more work to shift the project definition to the key creation step rather than the key usage step.

3. Secret Project Key

This can also be done alongside one of the above, but paired with #0, it fixes both problems.

The secret project key would be a shared key among all members of the project, much like the token (project API key), except it would be secret (and prefixed as such) and would allow access to the same things a personal API key does (perhaps with a few exceptions), except limited to a single project.

This would be big for plugins, and also a good thing for feature flags.

Take plugins for example. Currently, to use the PostHog API, they have to ask the user for a Personal API Key. We have some frontend logic to generate and autofill this in some cases, but that's really not optimal, as that plugin is now impersonating someone (same with feature flags).

If plugins want to provide devs with the ability to access the API, they need something better than generating full org access identifiable Personal API Keys in the background. A secret project token would make an implementation for PostHog/plugin-server#401 clean and secure, for instance.

4. Something else?

...

What next?

If you couldn't tell, I'm advocating for #3. Yes it's yet another key, but it solves an important problem. Wanted to get some thoughts before messing a lot with auth though.

This issue would also be split into at least 2 PRs:

• Updating the view sets to respect the project specified
• Adding the secret token

cc @Twixes

[neilkakkar]

This makes sense to me so far. Are you thinking of deprecating personal API keys, or is #3 in addition to personal API keys?

I feel we can do away with personal API keys as things stand. I'm a bit wary of introducing a new key without a migration path / reconciliation step for existing keys.

My assumptions here (which I don't know are correct or not yet):

1. Personal keys don't mean anything without an attached project - i.e. you can't use a personal key without the context of a project
2. People using the PostHog backend collaborate together, create keys together, and don't need to hide keys from other people with access to the same backend.

[Twixes]

Project Secret in addition to Project API Key sounds interesting.
A couple of questions:

• What do we put in created_by and other User foreign key columns when using a project secret?
• What does the HTTP API look like, especially for project-nested nested routes (which all routes will be in some future) like /api/projects/<team_id>/plugin-configs/?

[yakkomajuri]

@Twixes' questions (brainstorming as I go):

1. Couldn't we affiliate keys with a team instead of user (like personal API keys used to be I think)? I guess that would be problematic for places that access request.user so that'd need some work. We could also create a user in the background to associate the key with? A bit hacky perhaps but quite simple.
2. Thanks for the reminder on project nested routes, that might be a substitute for #0. This would need some thinking, and perhaps you actually are better suited to answer than me (you've thought a lot about our API and how it should evolve).

[yakkomajuri]

@neilkakkar's assumptions:

1. I believe this is not actually the case. Personal API Keys are just another way of being you, so they give you access to endpoints like api/users/@me as well.
2. This is also not the case right now. With the key being able to impersonate me, I wouldn't be comfortable sharing it with my team.

[mariusandra]

Copying "the problem" from here:

It seems that the current posthog-node implementation requires a personal API key with full account access rather than an organizational API key with scoped account access. This seems to imply that API keys are intended to exist for dev/analytics tools for team members and not for production application use.

Organization/team level, and scoped (like in github, screenshot below) keys sound like the way to go somehow.

Regarding request.user, can't we create a dummy "robot" user for each key that's not a personal api key?

[yakkomajuri]

Ok, even worse (and not mentioned above), is that for people with multiple orgs, Personal API keys will transcend those too. Was implied earlier, but not explicitly mentioned in the issue.

[yakkomajuri]

@mariusandra seems like this aligns with solution #2? I think it makes sense, but the problem is that even if scoped, I would rather not use what is called a personal key for a team purpose, hence the project secret.

[yakkomajuri]

Feature flags could actually benefit from a solution like the one I mentioned on #4713, where we do away with Personal API Keys without a new key. Plugins that hit the API on the other hand (at least how I'm envisioning it), using a secret project key would be quite neat.

[yakkomajuri]

Following up here on any further thoughts

@neilkakkar @Twixes @mariusandra

[weyert]

Sorry, for joining this conversation but I am still very much concerned and unclear what the current status of using the posthog non-client modules is.

The above conversation is suggesting to me using feature flag will return the flags of the project I have selected in the app.posthog.com web application?

[yakkomajuri]

Hey @weyert! This here is a deeper discussion about how to restructure auth around Personal API Keys. Any problems that existed with using feature flags in server-side libraries were fixed, as I mentioned on Slack.

See #4764 and PostHog/posthog-node@5a14e6c for posthog-node.

[neilkakkar]

More context / same issue: https://posthog.slack.com/archives/C01F31E9PRD/p1627475465099300

[weyert]

I would like to re-open this discussion. I am having a requirement at work were I would like to define ops or kill-switch feature flags to quickly allow some behaviours to be stopped.

For example, you have data processing jobs which run every few hours which fetches data from third parties. Occasionally the third party might have spike in traffic and ask to temporary postpone our jobs until the high traffic has passed or maybe the third party is having issues.

For this reason, I would like to use feature flags to temporary stop jobs for doing outgoing requests as its easier then needing to redeploy the service to change environment variables etc.

The only current problem I am having is that I can't define server-side feature flags that are only accessible for server-side code and don't get exposed on the client-side e.g. web-applications or mobile applications. For this reason, I would like to propose the following as a first iteration to allow this functionality:

• Introduce a second Project API key (potentially prefixed with phs) which only return feature flags tagged/marked for server-side use
• Add a new property to feature flags to define whether they are a server-side only feature flag or not
• Create a default value on this field in the model so it disabled by default

I think this is a reasonable solution to my problem at hand. I noticed there has been a discussion earlier about scoped personal access tokens but I don't think that would be able to solve the problem and typically it's problematic to use user-specific access tokens e.g. what happens when the user gets disabled or deleted etc?

I am happy to work on the necessary PRs to introduce this functionality

In the future I can imagine we could build upon this by for example support depending feature flags, e.g. only enable front-end feature flags when backend feature flag A and B are enabled etc.

[yakkomajuri]

Hey @weyert! So let's go step-by-step here.

I think a project secret makes sense, so let's start with that. Once that's done we can look into the best way of having "hidden" flags.

So for a project secret:

• This would be a key that can access most (probably not all, see second point) endpoints that are currently accessible via Personal API Key, but it would be scoped by project
• There's some nuance here as you'd need to handle endpoints that currently rely on a user calling them. Obvious ones are things like @me but probably anything accessing request.user would need some thought
• A new prefix like phs makes sense

I wonder if @Twixes has any thoughts here given you're probably the person with the most context on our authentication mechanisms.

[weyert]

I think a project secret makes sense, so let's start with that. Once that's done we can look into the best way of having "hidden" flags.

So for a project secret:

• This would be a key that can access most (probably not all, see second point) endpoints that are currently accessible via Personal API Key, but it would be scoped by project

Ah okay, but it should still show up together with the other project api key, correct?

• There's some nuance here as you'd need to handle endpoints that currently rely on a user calling them. Obvious ones are things like @me but probably anything accessing request.user would need some thought

Hmm, maybe for now we could limit the change behind a feature flag and only limit it to the feature flag related endpoints?

• A new prefix like phs makes sense

Great

I wonder if @Twixes has any thoughts here given you're probably the person with the most context on our authentication mechanisms.

Looking forward to hear from @Twixes :)

[yakkomajuri]

We could do this iteratively, yes. However, I do worry about this endpoint-specific code sitting there and for a long time (ever?) we don't actually implement it everywhere else but we've now rolled out this change to people like yourself who want to depend on it.

In general I'm fine with starting with one endpoint though

[weyert]

Awesome, I will have a crack at it :)

[aurovrata]

How difficult would it be to scope API keys to end-points?
If we need to query cohorts in a client-side js script, then it would expose the API key to the public, which could be used to query everything else on the server.
Restricting an API key to one or more end-point would solve this.

[Twixes]

We don't currently have plans to implement scoping for personal API keys, but regardless of that, we very strongly advocate against exposing any personal API key on the client side. A personal API key is tied to a specific user and their identity even if strictly scoped, so it should be almost as private as a password. @aurovrata

[NikoFoX]

Hey,
I would love to use 1 api_key for both read and write.
I use posthog on frontend and backend.
On frontend - where I only capture events - using only write api key feels great, because I don't want to expose read functionality for users.
On backend I use posthog to capture events and read insights from posthog in 1 project. I use project api key to capture events (using posthog python module), but personal api key to read from API. It seems weird for me to use my own api key (personal) in a whole project for the whole team to let backend read from API.
I just think that we should have option to create new api keys for project (more than 1) which we could configure to be read and/or write.
That way I could use 1 key for my frontend, which would be write only and other 1 key for backend, which would be both read and write.

[mariusandra]

That way I could use 1 key for my frontend, which would be write only and other 1 key for backend, which would be both read and write.

@NikoFoX that's exactly what we recommend doing, except the read API key is tied to someone's account, not to the project in general. To really have a "project read API key", you can create a new fake user who will only have access to the project. Your suggestion makes sense though - multiple configurable project API keys sounds like a reasonable ask.