2013.05.16 Weekly Check in

• 13:31 <demory> hey folks, should we do a quick check in?
• 13:32 <demory> ^ FrankP grant_h mattwigway abyrd
• 13:32 <mattwigway> sounds good
• 13:33 <FrankP> hey demory...up.
• 13:33 <FrankP> yup.
• 13:33 <demory> hey
• 13:33 <demory> so, i've been working thru various items we discussed last week for the calltaker and field trip modules
• 13:34 <demory> should have a new update ready for use by early next week
• 13:35 <demory> that will include the online request form for calltakers, the mailables manager, etc.
• 13:35 <demory> abyrd, you around? i see there's been some progress on the routing bugs too
• 13:37 <demory> guess he's not around
• 13:37 <demory> FrankP, anything to report on your end?
• 13:39 <abyrd> by demory
• 13:39 <abyrd> *hi
• 13:39 <demory> hey abyrd
• 13:40 <abyrd> apparently my irc notifications are not working... again
• 13:41 <abyrd> I worked through many of the routing bugs last week, comments are in the tickets
• 13:42 <abyrd> in sum, a few nasty bugs coupled with OSM data errors
• 13:42 <demory> yeah
• 13:42 <demory> FrankP, how do the tests look w/ the latest code?
• 13:43 <abyrd> FrankP sent me a link to a bunch of failing tests, but I think I know the reasons for those
• 13:43 <demory> ok
• 13:43 <abyrd> need to make the heuristic code symmetric but that's coupled with another refactor I'm doing
• 13:43 <FrankP> tests are all passing now...haven't had time to dig in and verify / close bugs. the only tests not passing are preferred transfers
• 13:43 <abyrd> oh great, good to know
• 13:43 <abyrd> take your time on the tickets, I'm still tying up loose ends
• 13:44 <FrankP> http://maps5.trimet.org/otp_osm_report.html
• 13:44 <abyrd> this week I got started trying to embed jetty in OTP
• 13:44 <mattwigway> I thought you were embedding grizzly?
• 13:44 <abyrd> after really studying up on the details of the Java EE server and Servlet/JSP specs it is obvious that they are truly overkill for what we are doing
• 13:45 <abyrd> mattwigway, well I started with jetty since it aims to be an embeddable servlet container but quickly realized we don't even use most of the servlet spec
• 13:45 <abyrd> yeah, you can just connect Grizzly straight to Jersey... bypassing the servlet container stuff entirely
• 13:46 <abyrd> so I'm continuing in that direction
• 13:46 <abyrd> we have a bunch of new devs starting with otp and the servlet/spring etc. learning curve is a major barrier
• 13:47 <FrankP> hey andrew ... will we still be packaging things into a .war? (thanks btw for all the bug fixes ... hopefully done there, at least after preferred transfers)
• 13:47 <abyrd> FrankP, yes we'd still provide a WAR for legacy systems
• 13:48 <abyrd> basically all the OTP functionality would be in a JAR, and there would be a couple of different modules that are thin wrappers around that main module
• 13:48 <abyrd> one is a WAR, another is a runnable JAR
• 13:48 <abyrd> you can combine the two but fancy tricks with Maven are unreliable/brittle
• 13:49 <mattwigway> was there a decision on whether to hang on to maven?
• 13:49 <abyrd> ideally the standalone jar would do both server and graph building to avoid duplicated dependencies
• 13:49 <abyrd> mattwigway, I talked at length about this with other developers
• 13:49 <FrankP> Cool...I have to say (after working with python / buildout / virtualenv / etc...), there is nothing better than dropping a .war file into the webapp directory.
• 13:50 <abyrd> maven dependency management is quite convenient, and I think we shouldn't toss it out wholesale without trying to sidestep our specific problems
• 13:50 <abyrd> so I want to move toward using Maven in a completely standard way: just dependency management, compile, and package
• 13:51 <abyrd> by making OTP self-contained, the integration tests can just become another module of Junint tests
• 13:51 <abyrd> no fancy downloading servlet containers from the internet, assembling custom WARs and starting them with shell commands
• 13:52 <abyrd> somy goal is to have OTP running as a plain old java app from the command line soon, self contained, and even a single step graph-build/serve mode
• 13:52 <abyrd> which would be phenomenal for tests and first-time users
• 13:53 <abyrd> I reworked the OTP.zip build scripts and was about to trigger per-build deployments from the CI server but realized it was within reach to just make OTP a "normal" java JAR
• 13:54 <FrankP> I actually delete the starting of tomcat (or whatever it is) from integration-test's .pom file...I just need a command line that builds the 3 .war files, then builds a graph (and I could do the graph building as a separate action outside of mvn...no problem there)
• 13:54 <abyrd> by the way ci.opentripplanner.org is back up (but now located in France :)
• 13:54 <demory> abyrd, so how would graph-builder configuration work for the self-contained java app? would there still be a spring config file to edit?
• 13:55 <abyrd> FrankP, my goal is to get all the small unit test fixtures (GTFS/OSM) into the repo to make the build less brittle, and move all "big" GTFS/OSM operations to outside scripts.
• 13:55 <abyrd> demory, in the short term yes
• 13:56 <abyrd> as long as we're using spring. i want to phase out the heavyweight frameworks slowly, evaluating the advantages before making the final steps. the elimination order would be
• 13:56 <abyrd> first external servlet container
• 13:56 <abyrd> second user-facing spring config
• 13:56 <abyrd> and fancy Maven actions
• 13:57 <abyrd> last Spring itself (experimenting with Guice)
• 13:58 <demory> how do you see GB config working post-Spring?
• 13:58 <abyrd> demory, the way this would work is an in-memory graph service. You point OTP to a graph builder config, it builds the graph but does not (necessarily) serialize to disk, then starts up an OTP server and hands over the graph
• 13:58 <demory> ah
• 13:59 <demory> i've also been thinking about some kind of "quick-start" gb config utility that would perhaps run as a web service
• 13:59 <abyrd> demory, pre-determined module order, maybe 1-2 alternate configurations for unusual cases (no OSM data)
• 13:59 <demory> where you just point it to your gtfs, etc.
• 13:59 <abyrd> point the graphbuilder to a directory full of files, or use a properties file to change a few parameters
• 14:00 <abyrd> yeah I have long wanted to just make a 99th percentile graphbuilder configuration and let people just point it at a directory of GTFS and OSM files
• 14:00 <abyrd> we can do that with the standalone OTP
• 14:01 <abyrd> ...combine with some disk caching of graphs and you could start OTP every time by just pointing it at a graph builder config. it looks for the graph in disk cache, if no input files have been modified it just loads the graph
• 14:01 <abyrd> if they have been modified it rebuilds
• 14:01 <abyrd> not everyone needs that and i'm not sure it's worth the complexity but it's an option
• 14:02 <demory> yeah
• 14:02 <abyrd> but starting up without a servlet container and without a separate graph builder step would save us loads of pain
• 14:02 <abyrd> let alone new users
• 14:02 <abyrd> imagine not even having to integrate tomcat with eclipse
• 14:02 <demory> yeah, sounds good. also, what about config options that currently go in app-context -- gtfs-RT feeds, etc?
• 14:02 <abyrd> directly starting OTP as a daemon on startup
• 14:03 <abyrd> (see jenkins' /etc/init.d/jenkins )
• 14:04 <abyrd> demory, usually there is "one right way" to set these things up. so we'd just have a couple of main profiles, and load the appropriate context for the profile
• 14:04 <abyrd> I think most of the relevant config could fit in a 1-page properties file
• 14:04 <abyrd> if not 2-3 lines
• 14:05 <demory> ok
• 14:05 <abyrd> anyway, none of this precludes the "old approach"
• 14:05 <mattwigway> what about additional custom graphbuilders (e.g. shapefile?)
• 14:06 <abyrd> mattwigway, presumably if you put a shapefile in the directory it will be loaded
• 14:06 <mattwigway> but what if tomorrow I write a TIGER/Line loader?
• 14:06 <mattwigway> also, shapefiles require schema specification
• 14:06 <abyrd> OTP will remain perfectly modular
• 14:07 <abyrd> I'm talking about making it usable for 99% of new users and usual use cases
• 14:07 <abyrd> when you want to do something custom, you just do it the same way you do now: modify the code
• 14:07 <abyrd> but I want it to work as a pre-packaged JAR for almost everyone, out of the box
• 14:08 <mattwigway> I totally agree on that.
• 14:08 <abyrd> if you want to plug the modules together differently, you can just change the injection code / XML (depending on which framework you use)
• 14:09 <abyrd> generally the people writing TIGER/Line import modules would also be comfortable adding a line to a Guice module :)
• 14:09 <mattwigway> that seems fair :)
• 14:10 <abyrd> I just think there's an obvious massive improvement to be made here for the majority of users/developers, which doesn't break any legacy systems
• 14:10 <abyrd> so I'm going to attack that first and we can evaluate response to that change
• 14:11 <abyrd> BTW I want to do some housecleaning in the repo... is anyone still using api-extended or can we remove it?
• 14:12 <abyrd> and I think we should agree on a naming scheme for modules that clarifies the distinction between clients, servers, and libraries
• 14:12 <abyrd> the term "webapp" means too many different things
• 14:14 <demory> abyrd, api-extended is the old system map module, right?
• 14:14 <demory> old, as in, not touched in ~3 years
• 14:15 <demory> i have some thoughts about system mapping but it will be completely different from that
• 14:15 <demory> so i have so problem w/ it going away
• 14:16 <abyrd> demory, system mapping is definitely a good thing but yes the api-extended is in disuse at this point
• 14:17 <abyrd> and I have to wonder how many people spend time attempting to deploy it to their servers (like I did at first)
• 14:18 <abyrd> it will still be available in the old releases and branches
• 14:18 <demory> right
• 14:18 <demory> also, i agree about the confusing usage of "webapp"
• 14:19 <demory> could otp-api-webapp just be otp-api?
• 14:19 <abyrd> As long as we're speculatively reinventing the world, I also find the opentripplanner- prefixes excessive
• 14:19 <demory> and maybe use "frontend" or "ui" in naming the front-end modules instead of "webapp"
• 14:20 <abyrd> otp- would make for a heck of a lot less typing and fit into widgets/file listings better
• 14:20 <abyrd> and we have the org.opentripplanner scope to prevent collisions with the other OTPs of the world.
• 14:20 <abyrd> demory, what about just client/server?
• 14:21 <mattwigway> do we need any prefix at all then?
• 14:21 <abyrd> I think we'd have otp-api-grizzly and otp-api-war
• 14:21 <abyrd> mattwigway, I was wondering that myself
• 14:21 <abyrd> let's check what the maven convention is
• 14:21 <mattwigway> I guess having server.war would be bad if there are multiple projects running on a single tomcat instance
• 14:22 <demory> yeah, keeping the otp- prefix could be helpful for that situation
• 14:22 <abyrd> mattwigway, certainly but I don't know if the war name is necessarily just the subproject artifact name
• 14:22 <abyrd> let's look into that
• 14:24 <demory> abyrd, i like "client" for the front-end implementations
• 14:24 <abyrd> mattwigway, looks like the convention is to include the parent package name in the submodules
• 14:24 <demory> e.g. otp-leaflet-client, otp-openlayers-client, etc.
• 14:24 <abyrd> but in our case I see no harm in shortening it to OTP
• 14:24 <abyrd> going to find a precedent before actually changing it though
• 14:25 <abyrd> demory, yes agreed
• 14:25 <demory> so would otp-server be the api module?
• 14:25 <abyrd> I think there would be several otp-server modules just like there are several otp-client modules
• 14:25 <demory> yeah
• 14:26 <abyrd> I was calling them otp-grizzly-api before
• 14:26 <abyrd> but they could also be otp-grizzly-api-server
• 14:26 <abyrd> but does it even make sense to call it an "api server"?
• 14:26 <abyrd> I mean, an API is just a standard you use to talk to a server
• 14:26 <mattwigway> API is perhaps better; for instance I think the Thrift API is better called an API than a server.
• 14:27 <abyrd> mattwigway, it depends on interpretation... my main goal is to have a coherent use of terminology
• 14:28 <abyrd> a module is a server if it loads its own graphs and routing code, its own protocol handling layers
• 14:28 <abyrd> a module called something-api sounds like a plugin to a server that provides an API
• 14:29 <abyrd> (which incidentally we will eventually want to implement -- just one OTP server with pluggable protocol modules for REST, Thrift, Protobuf, whatever)
• 14:29 <abyrd> ^ the above interpretation is in no way definitive, I was just giving a possible distinction between the terms server and api
• 14:32 <mattwigway> on another topic, abyrd, I'm curious if there are plans to expand batch analyst beyond location-based measures?
• 14:32 <abyrd> "If you intend for it to be used externally, remember the module = artifact ID = filename (in best practice), so it needs to be recognisable (maven-model-2.0.jar is clear, model-2.0.jar is not)."
• 14:32 <abyrd> http://stackoverflow.com/questions/3509690/what-are-best-practices-for-maven-modules-naming
• 14:33 <abyrd> "'Internal' modules may omit it for brevity"
• 14:33 <abyrd> so the otp- prefix should do the job
• 14:33 <abyrd> mattwigway, thanks for bringing that back up.
• 14:34 <abyrd> I will respond to your mailing list email this evening
• 14:34 <mattwigway> Well, the balancing-factors model is not really possible with the data I have anyhow, so I wouldn't worry too much about that specifically
• 14:34 <abyrd> balancing factors are extremely interesting as indicators
• 14:35 <abyrd> their main disadvantage being that they have no identifiable units
• 14:35 <mattwigway> but I'm just thinking that many people are using schedule/time-space models now and in OTP we have a unique multimodal engine
• 14:35 <mattwigway> for implementing logit models or time-space models, for example
• 14:35 <abyrd> but are good for relative comparisons, which is exactly what we want to do with service/infastructure changes and impact analysis
• 14:36 <mattwigway> right, well, most of the more academic/newer models have no units (even the gravity model)
• 14:36 <abyrd> whether or not you want to do balancing factors, they bring up a need for pre-computing an entire distance matrix as an input to a second phase
• 14:36 <abyrd> lots of people need this for other projects
• 14:37 <mattwigway> And time-space requires even more data
• 14:37 <abyrd> we need both types of indicators: relative/unitless for comparative work, and simple clear indicators like cumulative accessibility for communications/consultation/activism purposes
• 14:38 <abyrd> mattwigway, lots of interesting things to try here. Can you list them on one of the wiki pages, then let's pick one specific option and try to do an implementation/study.
• 14:39 <abyrd> But we'll have to pick one specific thing beyond cumulative accessibility lest we get overwhelmed with all the options
• 14:40 <mattwigway> I was trying to think of a way to implement an abstract framework that would then allow us to implement other things one-by-one
• 14:40 <mattwigway> because we don't want to have to restructure everything later on
• 14:41 <abyrd> sounds like a good idea
• 14:41 <abyrd> let me know if you want to discuss some time, or feel free to add speculative pages to the wiki
• 14:42 <mattwigway> I'm pretty buried right now, but maybe sometime next month?
• 14:42 <abyrd> yeah no hurry, there are plenty of other things to work on
• 14:42 <abyrd> well, anything else for the check-in?
• 14:43 <abyrd> I will archive this discussion on the wiki
• 14:43 <demory> hey, sorry, had to step away quickly
• 14:44 <abyrd> np
• 14:44 <demory> abyrd, i actually have a backlog of these check-ins i've been meaning to upload to the wiki
• 14:44 <demory> i'll do that now
• 14:44 <abyrd> oh good
• 14:44 <demory> including this one
• 14:44 <abyrd> thanks